Apply Pre-Trained Models to Shots Using Inference

Nuke Studio and Hiero's soft effect collection also includes an Inference effect in the timeline. The Inference effect applies a pre-trained .cat file to create the effect modeled by a neural network across shots on the timeline. The Inference effect is designed to maximize artist efficiency by eliminating the need for round-tripping between the timeline and comp environments in order to see the results of CopyCat training on multiple shots. You can also convert your own models to .cat files using the CatFileCreator, or download open-source models from Cattery for use on the timeline

See Train Neural Networks to Replicate Effects Using Machine Learning and Import Pre-Trained PyTorch Models for more information on creating and sourcing .cat files for use in the Inference effect.

Warning:  The timeline Inference effect can apply models trained on RGB input images only. .cat files requiring additional input channels (motion, position etc) are not supported.

Add an Inference Effect to a Shot

So you've trained your model and it's ready to go. In this example, we've trained a model to remove a bruise from an actor's face. Here's how to apply the model to a shot on the timeline.

1. Either select the shot on the timeline manually or use the handy auto-select feature to choose a shot for you depending on the current playhead's position.
2. Click the soft effect menu icon on the left-hand side of the timeline.
3. Select Inference to add the effect to the shot.

An error is displayed in the Viewer because no .cat file is selected by default.

4. In the Properties panel, enter a Model File path to the .cat file you want to apply to the shot.

The model is applied to the shot and you'll see the result in the Viewer.



Note:  Depending on the model size and complexity, there may be a delay while the effect is applied to the shot.

5. You can enable Optimize for Speed and Memory if you want a faster preview of the results. The Properties panel also displays the Channels In and Channels Out for reference.

Tip:  You can also apply Inference soft effects to an entire sequence by creating a new track and adding the effect to the track, instead of individual shots. See Adding Sequence-Level Soft Effects for more information.

Apply an Inference Effect to Multiple Shots

You can easily test how well the Inference effect works on other shots by simply copying and pasting the effect onto those shots. However, once you intend to use an Inference effect on multiple shots, it's a good idea to apply it to one shot first and then clone the effect to the other shots in case you update the .cat file later on with further training. Another useful method is to apply the soft effect to a track above the clip, then extend it to cover as many clips on the tracks below as needed. You can also cut the soft effect into pieces for more control over which clips it applies to.

1. Select the shot you want to use as the source of the cloned Inference effect.
2. Click the soft effect menu icon on the left-hand side of the timeline.
3. Select Inference to add the effect to the shot.

An error is displayed in the Viewer because no .cat file is selected by default.

4. In the Properties panel, enter a Model File path to the .cat file you want to apply to the shot.

The model is applied to the shot and you'll see the result in the Viewer.



Note:  Depending on the model size and complexity, there may be a delay while the effect is applied to the shot.

5. Select the Inference effect, right-click and choose Edit > Copy As Clones or press the Ctrl/Cmd+K keyboard shortcut.
6. Select the other shots in the timeline to which you want to add the cloned effect.
7. Right-click the selection and choose Edit > Paste or press the Ctrl/Cmd+V keyboard shortcut.

The effect is cloned to the selected shots and the Inference effect is applied. Cloned effects are clearly marked with a C and retain the same effect name, in this case Inference2, so you can quickly locate them.



If you train the model further to improve the results, simply update the clone source Model File control with the new .cat file to apply it to all the clones on the timeline.