Import Pre-Trained PyTorch Models

The CatFileCreator node (NukeX and Nuke Studio only) creates custom .cat files from PyTorch TorchScript .pt files within Nuke, opening up a wide range of pre-existing, trained models. You can import TorchScript files from GitHub, Model Zoo, or your own internal R&D department and slot them into your existing pipeline through NukeX's CatFileCreator.

For more details on Machine Learning with CopyCat in NukeX, see https://learn.foundry.com/nuke#machine-learning.

CatFileCreator allows you to add user knobs to your .cat files that are automatically passed to NukeX's Inference node so that you can have complete control over your pre-trained machine learning models. To create .cat files with exposed user knobs, you need a valid TorchScript .pt file containing correctly defined variables that link to the user knobs you define in Nuke.

Warning:  It is important to note that not all PyTorch models work without modification. In order to correctly construct a TorchScript model from a PyTorch model please follow the guidelines laid out in the PyTorch user guide at https://pytorch.org/docs/stable/jit.html and the information provided by Foundry in the Nuke .cat File Creation Guide here: https://learn.foundry.com/nuke/developers/14.1/catfilecreationreferenceguide/index.html

Converting .pt Files into .cat Files

PyTorch models must be converted to TorchScript .pt file so that CatFileCreator can convert them to Inference-readable .cat files. In order to create a custom .cat file with exposed user knobs, you will first need a valid TorchScript .pt file that contains correctly defined variables. These variables will be linked to the user knobs you define in Nuke. The combination of a custom TorchScript and user knobs enables you to create and use a wide variety of different PyTorch architectures in the Inference node, as well as the ability to control their function within Nuke using the user knob definitions.

As noted previously, not all PyTorch models can be converted without modification, so have a look at the Introduction to TorchScript guide provided by PyTorch.

To convert a valid .pt file into a .cat file:

Adding User Knobs

CatFileCreator allows you to add user knobs to your .cat files that are automatically passed to NukeX's Inference node so that you can have complete control over your pre-trained machine learning models.

Note:  Any user knobs you add must be associated with variables in the TorchScript file and have the same name.

In this example, we'll be adding a dropdown to allow artists to choose from a number of objects in a semantic segmentation model, but there are several other types of user knob available. See Adding Knobs Using Drag-and-Drop for more information.

To add user knobs to CatFileCreator:

Applying the Model to Your Footage

To apply the result of your model conversion, either: