O_ColourMatcher

Welcome to Ocula from The Foundry. My name’s Jon, and in this tutorial we are going to take a look at setting up and reviewing the ColourMatcher node in Ocula 3.0. We are going to look at colour correcting this footage. If I switch between left and right eye, here, you can see there are some large changes in reflections on the floor. We want to match those reflections to the image, to ease the stereo viewing experience. In my script, I am setting up and reviewing disparity. We are rendering that out for use, and I am pulling it in here to do a colour correction, using the ColourMatcher node in Ocula, and we are going to look at quality checking that later on.

So, first of all, let’s have a look at the setup and review. I have set the analysis keys in the Solver node, and I am using the default parameters in the DisparityGenerator node. There are some tutorials available on how to set up the Solver and DisparityGenerator nodes. Here I also have an OcclusionDetector node, and that’s adding in a new occlusion mask channel in Nuke. This defines the pixels that do not match between the left and right view, the pixels that are obscured or revealed between the views. The ColourMatcher is going to ignore those occluded pixels when it pulls the colour from one view onto the other. So, if we look at the OcclusionDetector, you can tune how the occlusions are defined and we’ll see how to do that later on. Here, you can see that the image borders are defined as occluded between left and right view. I am going to render out disparity and occlusion and then pull that back in to use with the ColourMatcher node.

In the ColourMatcher, you can define the left and right views, whether to match the left to the right, or update the right to match the left. There are three different methods to calculate the colour: update a Basic lift and gain, a more complicated 3D LUT, which you can then export for use outside Ocula or Nuke, and there is also an option to do a local colour correction (Local Matching). This matches small blocks between the left and right images, so it will match the subtle variations between the views, which is the reflection we are seeing here. You can choose whether to treat the occluded regions, defined by the occlusion detector, separately or not, and there are some parameters to tune. We will see how to tune those updates later. There are also some Multi-scale Options for the blocks. You can choose whether to use one or more block sizes, you can set the maximum block size to use, and also define how the update is applied. These options help to stabilise the colour update over time. The ColourMatcher also has a Mask input. You can choose how the mask is defined with the basic and 3D LUT methods, then the update is calculated in the mask region and applied to the whole image. In Local Matching, the update is calculated for the masked region and applied to that masked region. So, here’s the update that’s been calculated using the local correction method; you can see that reflections have been updated so they match between the left and right view.

Now, let’s have a look at how we can quality check the colour correction. Down here, I am re-calculating the disparity for the corrected plates. Notice I have to pull the Solver information from upstream. I could have just re-calculated it here in the new Solver node. I am using that disparity with the Ocula StereoReviewGizmo. I have got it set to show a CheckerBoard key mix between the left and right view. If I switch the correction, I can pick up the converge point on the gizmo and move it around to check between any differences on the left and right on different parts of the image. If we take a look at the original footage, you can see the difference that’s been corrected. So, again, if we moved around, you could see the different colours in the left and right view in the key mix. Another way to quality check is to use the NewView node to pull the corrected view onto the ‘hero’ view, and then you can check for differences. I have the NewView set to pull the Left view, which we updated across onto the right using an Interpolate Position of 1. I can then do a per-pixel difference, so we can check for errors on the whole image at once. The difference is here, only on the edges, which may have a sub-pixel shift disparity calculation. If we look at the original plates, you can see the colour difference that’s been corrected by the ColourMatcher node.

Let’s have a look at the colour correction for another shot. Here’s the left and right view, and there is a large colour shift we want to correct here. I have keyed the Solver, and we are calculating the disparity and occlusions as before. If we look at those occlusions, these are the pixels that are marked as different between left and right views. Now there are two ways in which occlusions are calculated, first by looking at differences in Depth Occlusions, where one part of the image is obscured by the other, and the other by looking at differences in image content, using a threshold on the colour difference between the views (Colour Threshold). So, we can switch these thresholds off and tune them to the shot. I am going to start by pulling up the Depth Occlusions until we just pick out the occlusions between the depth layers. I don’t want this to become too large and blobby. I can then work with the Colour Threshold so, again, I can start to pull this up until we just pick out the differences at these changes between depth and the scene. I tend to be quite conservative here; you don’t want to pick up lots of isolated regions that are going to flicker between frames. We want it to be stable over time. There are also some options here to refine the occlusion regions, and there’s some presets you can choose from, according to how much of the image is different between left and right views. So, here we are, seeing an extreme difference, so it’s much larger. One extra thing I have done for this shot is to add a roto to block out the bit at the top of the image, which is different between left and right. So, you are free to edit the occlusion masks with Roto or RotoPaint.

So, let’s have a look at the results of the colour matching. This is the Basic method doing the lift and gain. You can improve this using the 3D LUT method, which calculates a global update based on pixel matches, based on the left and right views. So, the result is much better, but it still has some variations in the reflections on the water. Now, I can export this and use it in a VectorField node in Nuke, so I can apply that 3D LUT without Ocula, or you can use it outside Nuke. Here’s the ColourMatcher result, and here’s the result from the VectorField node, and they match. We can also do a local correction to match the reflections on the water in this footage. Here, you can see the results: the left and right view match. Now we can use our QC technique again by looking at the colour difference within NewView. So, here’s the difference for the local colour correction, now we can go back and switch the methods, and compare those differences. There’s large changes with Basic. It’s better with 3D LUT, but you can see the differences with the reflections on the water. And, finally, the Local Matching, which gives the best result.

Let’s have a look at one more example. In this shot there are some extreme changes between the views. Large areas are not visible when you switch from one to the other. Here, you can see the shop front is obscured when I switch from the left to the right. Let’s review the footage with NewView. I am going to colour match the left view. So, here, I want to pull the right onto the left, and then compare that to the original left. You can see it highlights that shop front. We want to pick up this region with the OcclusionDetector. Now let’s set that up. I am going to switch to the Mat overlay (M) and you can see it picks up the shop fronts. There are some holes though, and some isolated regions, and these can flicker, so we want to remove them and create a stable mask. I am going to switch to the Extreme setting. This expands the mask and fills the holes to make sure that they are stable. Now let’s set up the thresholds, as we did before. We just want to pick out the shop front here. I can switch to set Colour Threshold next, and I want to make sure that there are no isolated patches left in the mask that will flicker, causing the colour update to flicker. So, this looks good. Now, one extra thing I have done here is to add a roto mask, just to make sure that occlusion is masked for the shot. I could have actually switched the thresholds off in the OcclusionDetector, and just used a roto instead.

Now, let’s have a look at the colour correction for the left view. So, here, we have the basic lift and gain. Let’s switch to see the local block-based colour match. You can now see a yellow band on the wall. This is where the ColourMatcher is compensating for the occlusions. It looks at the pixels outside of the mask and it pulls the colour update inside of the mask. The problem here is that it’s pulled the update from the black region into the adjacent bricks. I just need to expand the Region Size, where it searches for the colour update so it can pull the update from other bricks. You can also restrict this to using regions of similar colour by reducing the Colour Sigma. Now, finally, you will notice there is some dark banding inside the teapot. This happens because the blocks are spanning the light and dark region. I need to use a smaller Block Size, so the update covers a smaller area. This looks good. You can try tuning the multi-sampling options, particularly if you are having any issues with the correction changing between frames; multi-sampling can improve the stability. Now, there are still some isolated regions inside of the occlusion mask I’m not happy with. What I am going to do now is switch to using the 3D LUT result in this region. This is the 3D LUT correction, and it looks pretty good. So, what I have done is shuffled the occlusion mask into alpha, and I am going to apply that as a matte to pick out the 3D LUT correction inside the occlusion mask, and the local correction outside. Here’s switching between the original and our final results, and you can see it corrects some shadowing and reflections in the shot.

So, that wraps up this tutorial on the ColourMatcher. We have looked at rendering out the occlusion mask with disparity for use in colour matching. We have looked at the local colour matching and quality checking the result, in particular using NewView to rebuild one view on top of the other to look at colour differences. We then looked at a more detailed example, and how to set up the occlusions, how to edit them with roto, and finally, we looked at an extreme occlusion example and how to tune the colour matching parameters, and also switch between different colour matching results.