Render Farm Plug-ins

✨ New in Katana 4

Introduction to Render Farm Plug-ins

The FarmAPI Python package provides a base class for a type of plug-in that allows developers to closely integrate a specific render farm system with Katana: FarmAPI.BaseFarmPlugin

The BaseFarmPlugin class provides an interface that allow Katana to:

  • Send Preview Render, Live Render, and Disk Render jobs to a render farm to be executed remotely.
  • View the list of jobs running on the render farm.
  • Retrieve information about a particular render farm job, such as the current state and the job’s render log.
  • Stop/restart previously submitted jobs.

In general terms, a BaseFarmPlugin is an expression of the adapter pattern: render farm-specific functionality can be expressed within the implementation of the various methods of the interface.

Render farm plug-ins are managed using the FarmAPI.FarmPluginManager module.

Terminology

Render Farm - term given to the collective hardware and software that extend the local compute resources of an artist for the purposes of rendering images.

Render Job - a series of stages/commands that describe the work required to produce a rendered image.

Queue Management Server - a software component that manages job requests and schedules their execution based on priority or resource requirements.

Render Node - a compute resource managed by the queue management server. Jobs consume the resources of a render node.

Quick Start

This section describes how to implement a render farm plug-in. It assumes you have a render farm accessible from your network, and that you can communicate with the queue management server in order to submit and retrieve information about jobs.

  1. Subclass FarmAPI.BaseFarmPlugin and register your plug-in with Katana.

    The first stage requires creating a class derived from FarmAPI.BaseFarmPlugin which describes the interface of methods you need to implement to integrate your render farm with Katana:

    from Katana import FarmAPI


class MyFarmPlugin(FarmAPI.BaseFarmPlugin):
    pass


PluginRegistry = [("FarmPlugin", 2.0, "My Farm Plugin", MyFarmPlugin)]

  2. Add the plug-in to your KATANA_RESOURCES path in a Plugins subfolder. For example:

    # For a plug-in at /home/john/katana_resources/Plugins/MyFarmPlugin.py:

KATANA_RESOURCES=/home/john/katana_resources

  3. Implement the necessary methods (see below) to allow Katana to communicate with your render farm. Methods can be grouped into three categories. Some methods are optional; not implementing them will disable the corresponding functionality within Katana:

    1. Job Submission - this group of methods handles the preparation of a Job and its submission to your queue management server.
    2. Information Retrieval - this group of methods allows Katana to fetch information about a previously submitted Job such as its current state or the contents of its render log.
    3. Job Control - these methods allow artists to start/restart and stop Jobs.

  4. Test. Farm plug-ins are invoked by Katana when the user selects a remote render for a particular render type (Disk Render, Preview Render, or Live Render). The resources (e.g. the Op Tree) are copied to a location accessible to the render farm, and the commands required to generate the rendered image outputs are passed to the render plug-in allowing it to configure a Job in the domain-specific language of the queue management server.

Job Submission

The methods in this group provide the opportunity to configure a Render Job and submit it to the farm’s queue management server. Job submission is a three-stage process:

  1. Creation of temporary render files: The Katana render process generates a number of temporary files. An example of these temporary files includes a serialised copy the the Op Tree used by Geolib3 to generate the scene description. This stage is further broken down into the following steps:

    1. Creation of a render context: Katana allocates a globally unique (UUID v4) render context.
    2. Creation of a temporary folder: The farm plug-in is asked to create and return the path to a temporary folder that is accessible by both the artist’s workstation and the render node. This method is also passed the render context which can be used as part of file path to guarantee uniqueness.
    3. Copying of render files: A list of local files will be passed to the farm plug-in which should be copied to a remote location accessible by the render node. Their new (remote) paths should be returned to Katana. Whilst it is not necessary to have copied the files before returning from this method, the copy operation should be complete before the job is started by the queue management server.

  2. Configure environment: The plug-in will be asked to describe the remote Katana installation in a L{BaseFarmPlugin.InstallationMetadata} instance. This includes the path to the Katana installation. The plug-in will also be asked to provide additional environment variables that should be included in the process environment.

  3. Submit Job: The information gathered in the preceding stages is passed to the C{submitJob()} method in a L{FarmAPI.FarmPluginManager.JobRequest} instance. At this point, the plug-in is able to create a job description using the domain-specific language of the render farm, including applying any tags or resource allocation requirements, and then submit it for execution. The C{submitJob()} method should return a string that uniquely identifies the job. This identifier will be used in subsequent interaction.

Information Retrieval

The methods in this group retrieve information about a previously submitted Job. Given the heterogeneity of queue management servers, Katana imposes very few requirements on the data a queue manager provides. One such requirement is that a Job submitted to the farm can be uniquely identified by a number or string. The Job identifier should be convertible to string and returned when the C{submitJob()} method is called. Subsequent metadata about a Job will be requested by referencing a job by its identifier.

Minimum requirements of a Job:

  • It should be possible to describe a Job’s current state in terms of the enumeration C{Job.State}.
  • It must be possible to execute an ordered list of commands.
  • The assets specified by RenderOutputDefine nodes (for Disk Renders only) should be accessible by both the render node and the artist’s workstation.

Optional requirements of a Job:

  • It should be possible to retrieve the filename of the render log that corresponds to a Job. If supplied, this render log will be loaded into Katana’s B{Render Log} tab and monitored as it updates.

Job Control

Notes on Network Topology

Katana’s farm plug-in infrastructure makes a number of network topology assumptions:

  • The plug-in is capable of copying/moving/publishing files from the artist’s workstation (where the plug-in is running) to an asset/file or object store that is also accessible by the render node.
  • Conversely, the plug-in also assumes that files such as the render log (produced by the render node) and any rendered images (in the case of a Disk Render) are accessible from the artist’s file system.
  • The render node is able to connect via TCP/IP to the artist’s workstation running the Catalog Server which is used to receive pixel data (in the case of Preview Renders and Live Renders).

If users are required to authenticate with the queue management server via credentials, this should be handled either on first contact with the farm management software, or on plug-in construction.

Render Farm Plug-in Reference Documentation

class FarmAPI.BaseFarmPlugin

Bases: object

Base class for farm plug-ins.

Farm plug-ins are used by Katana to dispatch jobs to a render farm (or other remote compute resource). Each render farm must provide the following functionality, or the farm plug-in must emulate that behavior:

  • Submit Jobs: Ability to configure and submit a job to the farm based on a number of parameters such as project file, frame range and render outputs.
  • List Jobs: Ability to list current or past jobs for the current Katana session. Plug-in authors are free to define the filtering, for example only show jobs related to the current show or shot the artist is working on.
  • Job Detail: Ability to retrieve metadata about a previously submitted job, such as its start time and contents of the render log.
  • Job Control: Ability to start or stop a particular job.
  • Render Asset Management: Some assets must be made available to the render farm. A farm plug-in is required to implement some basic asset management functionality (or utilise the Katana Asset API). Examples include creating a temporary I{workspace} for the render job and moving files to that location.

@since: Katana 4.0v1

class InstallationMetadata(katanaRoot, renderPluginDsoPaths, renderbootCommand=None)

Bases: object

Describes the installation of Katana running on a render node.

__init__(katanaRoot, renderPluginDsoPaths, renderbootCommand=None)

Initializes an instance of the class.

The given renderPluginDsoPaths dictionary is expected to contain names of registered render plug-ins as keys, and absolute paths to the folders containing the renderers` DSOs as values, for example:

{
    'dl': ('/usr/local/3delight-2.1.4/Linux-x86_64/'
           '3DelightForKatana/Libs'),
    'prman': ('/opt/pixar/RenderManForKatana-23.3-katana3.5/'
              'plugins/Resources/PRMan23/Libs'),
    'customRenderer': '/path/to/customRenderer/Libs'
}
Parameters:
  • katanaRoot (str) – Absolute path to the root of the Katana installation.
  • renderPluginDsoPaths (dict of str → str) – Mapping with names of registered render plug-ins as keys, and absolute paths to the folders containing the renderers` DSOs as values.
  • renderbootCommand (str or None) – If not None, the absolute path to the command that should be used to invoke renderboot. If None it will be deduced from the given katanaRoot.
BaseFarmPlugin.copyRenderFiles(filenames, renderContextName)

Copies the list of files to a location accessible by the render nodes.

Typically the files will be copied to a subfolder of the temporary directory created by createTempDirectory.

Return type:

map of str to str
Parameters:
  • filenames (list of str) – A list of names of files to be copied.
  • renderContextName (str) – A globally unique ID for the render job for which to copy the files.
Returns:

A map containing each of the given filenames and the absolute path of its respective copy.
BaseFarmPlugin.createTemporaryDirectory(renderContextName)

Creates a temporary directory accessible by the render nodes to be used by the renderboot process.

Typically an NFS share or other object store would be used.

Return type:str
Parameters:renderContextName (str) – The render context for which to create the temporary directory.
Returns:The file system location path of the temporary directory that was created.
BaseFarmPlugin.getEnvironmentForJob(renderContextName)
Return type:dict of str
Parameters:renderContextName (str) – The render context for which to return an environment.
Returns:Dictionary of environment variables to be available to the process environment during the render job.
BaseFarmPlugin.getJobState(jobId)

Returns the state of the job specified by jobId.

Return type:str
Parameters:jobId (str) – The unique identifier of the job whose state to return.
Returns:The state of the job with the given ID as a value from the Job.State enumeration.
Raises:JobNotFoundException – If the specified job does not exist on the render farm.
BaseFarmPlugin.getJobs(filters)
Return type:list
Parameters:filters (list) – The filters to apply for obtaining the list of jobs.
Returns:A list of Jobs based on the specified filter criteria.
BaseFarmPlugin.getRemoteInstallationMetadata()
Return type:BaseFarmPlugin.InstallationMetadata
Returns:An instance of InstallationMetadata containing the details of the remote installation of Katana.
BaseFarmPlugin.getRenderLogFilename(jobId)

Returns the filename of the render log for the specified jobId, or None if it cannot be accessed from this host.

Return type:str or None
Parameters:jobId (str) – The unique identifier of the job whose render log filename to return.
Returns:File path to render log or None.
Raises:JobNotFoundException – If the specified job does not exist on the render farm.
BaseFarmPlugin.startJob(jobId)

Starts the Job specified by the job ID, if it is not already running.

Parameters:jobId (str) – The ID of the job to start.
BaseFarmPlugin.stopJob(jobId)

Stops the Job specified by the job ID, if it is not already stopped.

Parameters:jobId (str) – The ID of the job to stop.
BaseFarmPlugin.submitJob(jobRequest)

Creates and submits a job to the render farm.

This method will be called to allow the farm plug-in to create a description of the render job (using the render farm’s own description language). This job should then be submitted to the queue management server.

The commands that should be run are pre-processed based on the information provided by BaseFarmPlugin.InstallationMetadata.

Return type:str
Parameters:jobRequest (FarmAPI.FarmPluginManager.JobRequest) – Contains details of the job that should be submitted to the farm, such as the commands to be run, frame range, process environment, etc.
Returns:A unique identifier for the Job on the farm.
class FarmAPI.Job(jobId, name, startTime, endTime, state, renderLogFilename)

Bases: object

Minimal Plain Old Python Object (POPO) that encapsulates information about a Job.

@since: Katana 4.0v1

class State

Bases: object

Describes the state of a job.

Plug-in authors should map their farm-specific Job states to this set of states.

kAllStates = ('Waiting', 'Running', 'Completed', 'Cancelled', 'Failed')
kCancelled = 'Cancelled'
kCompleted = 'Completed'
kFailed = 'Failed'
kRunning = 'Running'
kWaiting = 'Waiting'
Job.__init__(jobId, name, startTime, endTime, state, renderLogFilename)

Initializes an instance of this class.

Parameters:
  • jobId (str) – The ID of the job represented by this object.
  • name (str) – The name of the job represented by this object.
  • startTime (int) – The time in milliseconds when the job represented by this object was started.
  • endTime (int) – The time in milliseconds when the job represented by this object finished.
  • state (str) – The state of the job represented by this object.
  • renderLogFilename (str) – The name of the render log file for the job represented by this instance.
Raises:

ValueError – If the given state is not one of the states that are defined as constants in the State class.
class FarmAPI.FarmPluginException

Bases: exceptions.Exception

Class implementing a type of exception that can be raised by subclasses of L{FarmAPI.BaseFarmPlugin} in response to exceptional circumstances.

@since: Katana 4.0v1

class FarmAPI.JobNotFoundException(jobId, contextMessage)

Bases: PyUtilModule.FarmAPI.BaseFarmPlugin.FarmPluginException

Specialisation of FarmAPI.FarmPluginException to be raised when a requested job cannot be found.

Since:Katana 4.0v1
__init__(jobId, contextMessage)
Parameters:
  • jobId (str) – String representation (or convertible to) of the ID of the Job that could not be found.
  • contextMessage (str) – Message describing the context in which the error occurred.