Creating a Data Set for Training

Video:  Creating Mattes Using CopyCat in Nuke

A CopyCat data set is a number of image pairs used to train a network to perform a specific task. Image pairs consist of one Input image and one Ground Truth image, before and after the effect is applied. The network is attempting to learn how to create the Ground Truth from the Input image.

Note:  The Input and Ground Truth image pairs must be the same format size.

CopyCat can ingest sequential formats, such as .jpg and .exr, and container formats, such as .mov and .mxf, but the set up is slightly different in each case.

Sequential Format Data Sets

Sequences of .jpg and .exr files are the simplest data sets to create because they're already broken down into the individual frames required for training. This example shows a data set you could use to teach a network to apply a matte, but the principle is the same for clean up, deblurring, or any other manual work.

1. Play through your sequence and select a number of frames to use for the data set.

A diverse selection of images tends to produce the best results and the more frames you use, the better the results are likely to be. For example, if you're training a network to mask an object, try to pick frames that represent a wide variety of mask shapes.

2. Create a Read node for each frame selected and set the File control to point to the required frame number. For example:

../assets/input/input_v01_0211.png

3. Create an AppendClip node and connect all the Read nodes.
4. Connect the AppendClip node to the CopyCat node's Input pipe.



5. Using the same frames, roto out the masks for the Ground Truth.
   See Using RotoPaint for more information on Roto and RotoPaint.

Note:  In this example, the alpha channel was written to disk separately, but CopyCat can read the alpha from the Roto shapes in the Node Graph.

6. Create a Read node for each of the masks and set the File control to point to the required frame number. For example:

../assets/groundtruth/truth_v01_0211.png

7. Create an AppendClip node and connect all the Read nodes.

Warning:  Ensure that your Ground Truth images are connected to the AppendClip node in the same order as your Input images. Image pairs that don't match affect the quality of the resulting model.

8. Connect the AppendClip node to the CopyCat node's Ground Truth pipe.



9. Proceed to Training and Monitoring the Network for more details on CopyCat settings and training.

Container Format Data Sets

Container formats like .mov and .mxf files require a bit more work to separate out the individual frames required for training. This example shows a data set you could use to teach a network to apply a matte, but the principle is the same for clean up, deblurring, or any other manual work.

1. Play through your sequence and select a number of frames to use for the data set.

A diverse selection of images tends to produce the best results and the more frames you use, the better the results are likely to be. For example, if you're training a network to mask an object, try to pick frames that represent a wide variety of mask shapes.

2. Create a Read node and set the File control to point to the full sequence. For example:

../assets/input/input_v01.mov

3. Create a FrameHold and FrameRange node for each frame selected in the data set.
4. Set each FrameHold node's first frame control to the required frame and the FrameRange node's frame range control to 1-1.

Note:  The FrameRange nodes ensure that only 1 frame is ingested by CopyCat per Read node, the frame specified by the FrameHold node.

5. Create an AppendClip node and connect all the FrameRange nodes.
6. Connect the AppendClip node to the CopyCat node's Input pipe.



7. Using the same frames, roto out the masks for the Ground Truth and write the alpha channel to disk.
   See Using RotoPaint for more information on Roto and RotoPaint.
8. Create a Read node for each of the masks and set the File control to point to the required frame number. For example:

../assets/groundtruth/truth_v01_0211.png

9. Create an AppendClip node and connect all the Read nodes.

10. Warning:  Ensure that your Ground Truth images are connected to the AppendClip node in the same order as your Input images. Image pairs that don't match affect the quality of the resulting model.

11. Connect the AppendClip node to the CopyCat node's Ground Truth pipe.