Apply and Improve the Results from Your Network
For more details on Machine Learning with CopyCat in NukeX, see https://learn.foundry.com/nuke#machine-learning.
Apply the Model to the Sequence
Trained networks are applied to sequences using the Inference node. The node reads in a .cat file and applies the effect learned from the data set to the whole sequence.
Improve the Results Your Model Produces
One of the simplest errors to correct is when you have connected the Input and Ground Truth images in a different order. You can look at the AppendClip nodes in the Node Graph to check the numbering, but this can be difficult if you have a large data set. An easy way to check is by connecting a Merge node in difference mode to compare the Input and Ground Truth images.
If the result still isn't what you expected, it could be that the image pairs in the data set don't contain enough diverse examples or that elements such as shadows and defocus are not represented in the data set.
Augment Image Pairs in the Data Set to Improve Results
You can fake the elements not represented in an existing data set, which requires less effort than creating brand new image pairs. This can work particularly well for shadow elements or areas where the matte passes over different backgrounds in the source sequence.
Playing through the example source sequence doesn't provide any examples of the where the actor passes directly over the car in the background, with the red color on either side of where the matte passes. This could cause problems for the neural network as it may only expect the actor against predominantly sand, mountains, and sky colors.
|
|
|
|
You can fake new foreground and background combinations by copying a Ground Truth matte into an Input image and then transforming the matte of the actor over the car body. You can also crop the new image pair to focus on just the areas where the actor and car overlap.
Tip: It's a good idea to add your additional image pairs using new AppendClip nodes. This way, you can easily isolate your original data set from faked image pairs if the new pairs aren't working to improve the output.
This simple example shows one possible workflow to produce any number of fake image pairs to add to the data set.
Add Image Pairs to the Data Set for Variety
Adding new image pairs to the data set is another solution, but can be very time consuming. For example, the sequence shown in this topic contains a few frames where the actor crosses into the shadow of the car.
Retrain Your Model Using Checkpoints and Weights
Training a network may not always produce the results you're looking for on the first run, so CopyCat allows you to resume training from the latest checkpoint in the Data Directory or restart from a previously trained model. You can select the required option from the Advanced > Initial Weights dropdown:
• Checkpoint - the training uses weighting from an existing .cat file. Enter the path to the .cat file in the Checkpoint File control or browse to its location.
• Deblur - the training starts, or resumes from the most recent .cat file, with a model weighted towards deblur effects, which can improve the training results for similar operations. For example, you could use the Deblur node on a sequence, and then try to improve the result on frames where it didn't work particularly well using Copycat.
• Upscale - the training starts, or resumes from the most recent .cat file, with a model weighted towards upscaling, which can improve the training results for similar operations. For example, you could use the Upscale node on a sequence, and then try to improve the result on frames where it didn't work particularly well using Copycat.
Thresholding and Filling the Matte
If there are still some artifacts in the background or holes in the foreground matte, you can use the Grade node or RotoPaint nodes to complete the matte. Thresholding refers to adjusting the black point and white point in the Grade node's controls to erode artifacts while leaving the edges of your matte intact. Filling describes manually filling holes in your matte using a RotoPaint node or similar.
In this exaggerated example, the background contains a lot of artifacts of the car and the mountains.