If you can’t adequately improve the screen matte using the clip levels, you can create a matte in Nuke round the pixels you definitely want to be foreground or background and use this as a mask input. The inside mask makes the foreground less transparent and the outside mask is used to clean up the background that might have bits of the foreground showing through. It is sometimes referred to as the hold out mask.
The outside mask (garbage mask) is often used to clean up screens that are not a constant color or have lighting rigs in shot by forcing the alpha transparent.
The inside mask can be used to keep elements in the foreground that you don't want to lose (an actor’s blue eyes in front of a blue screen). These masks should normally be softened externally to blend into the screen matte.
The below image shows the Bezier spline drawn around the lighting rig on the left side of the screen.
Connect the mask to the OutM input of Keylight and switch the parameter OutM Component to Alpha. The outside mask forces that part of the image to be in the background, thus keying out the rig.