Render Farm Plug-ins
====================
|sparkles| **New in Katana 4**
Introduction to Render Farm Plug-ins
------------------------------------
The :doc:`FarmAPI</Scripting/RenderingAScene/FarmAPI>` Python package provides
a base class for a type of plug-in that allows developers to closely integrate
a specific render farm system with Katana:
:py:class:`FarmAPI.BaseFarmPlugin`
The ``BaseFarmPlugin`` class provides an interface that allow Katana to:
- Send Preview Render, Live Render, and Disk Render jobs to a render farm
to be executed remotely.
- View the list of jobs running on the render farm.
- Retrieve information about a particular render farm job, such as the
current state and the job's render log.
- Stop/restart previously submitted jobs.
In general terms, a ``BaseFarmPlugin`` is an expression of the adapter pattern:
render farm-specific functionality can be expressed within the implementation
of the various methods of the interface.
Render farm plug-ins are managed using the :py:mod:`FarmAPI.FarmPluginManager`
module.
Terminology
-----------
`Render Farm` - term given to the collective hardware and software that extend
the local compute resources of an artist for the purposes of rendering images.
`Render Job` - a series of stages/commands that describe the work required to
produce a rendered image.
`Queue Management Server` - a software component that manages job requests and
schedules their execution based on priority or resource requirements.
`Render Node` - a compute resource managed by the queue management server.
Jobs consume the resources of a render node.
Quick Start
-----------
This section describes how to implement a render farm plug-in. It assumes you
have a render farm accessible from your network, and that you can communicate
with the queue management server in order to submit and retrieve information
about jobs.
1. Subclass :py:class:`FarmAPI.BaseFarmPlugin` and register your plug-in with
Katana.
The first stage requires creating a class derived from
:py:class:`FarmAPI.BaseFarmPlugin` which describes the interface of methods
you need to implement to integrate your render farm with Katana::
from Katana import FarmAPI
class MyFarmPlugin(FarmAPI.BaseFarmPlugin):
pass
PluginRegistry = [("FarmPlugin", 2.0, "My Farm Plugin", MyFarmPlugin)]
2. Add the plug-in to your ``KATANA_RESOURCES`` path in a ``Plugins``
subfolder. For example::
# For a plug-in at /home/john/katana_resources/Plugins/MyFarmPlugin.py:
KATANA_RESOURCES=/home/john/katana_resources
3. Implement the necessary methods (see below) to allow Katana to communicate
with your render farm. Methods can be grouped into three categories. Some
methods are optional; not implementing them will disable the corresponding
functionality within Katana:
1. Job Submission - this group of methods handles the preparation of a
Job and its submission to your queue management server.
2. Information Retrieval - this group of methods allows Katana to fetch
information about a previously submitted Job such as its current state
or the contents of its render log.
3. Job Control - these methods allow artists to start/restart and stop
Jobs.
4. Test. Farm plug-ins are invoked by Katana when the user selects a remote
render for a particular render type (Disk Render, Preview Render, or Live
Render). The resources (e.g. the Op Tree) are copied to a location
accessible to the render farm, and the commands required to generate the
rendered image outputs are passed to the render plug-in allowing it to
configure a Job in the domain-specific language of the queue management
server.
Job Submission
--------------
The methods in this group provide the opportunity to configure a Render Job and
submit it to the farm's queue management server. Job submission is a
three-stage process:
1. Creation of temporary render files: The Katana render process generates a
number of temporary files. An example of these temporary files includes a
serialised copy the the Op Tree used by Geolib3 to generate the scene
description.
This stage is further broken down into the following steps:
1. Creation of a render context: Katana allocates a globally unique (UUID
v4) render context.
2. Creation of a temporary folder: The farm plug-in is asked to create
and return the path to a temporary folder that is accessible by both
the artist's workstation and the render node. This method is also
passed the render context which can be used as part of file path to
guarantee uniqueness.
3. Copying of render files: A list of local files will be passed to the
farm plug-in which should be copied to a remote location accessible by
the render node. Their new (remote) paths should be returned to
Katana.
Whilst it is not necessary to have copied the files before returning
from this method, the copy operation should be complete before the job
is started by the queue management server.
2. Configure environment: The plug-in will be asked to describe the remote
Katana installation in a `BaseFarmPlugin.InstallationMetadata` instance.
This includes the path to the Katana installation. The plug-in will also be
asked to provide additional environment variables that should be included
in the process environment.
3. Submit Job: The information gathered in the preceding stages is passed to
the `submitJob()` method in a `FarmAPI.FarmPluginManager.JobRequest`
instance. At this point, the plug-in is able to create a job description
using the domain-specific language of the render farm, including applying
any tags or resource allocation requirements, and then submit it for
execution. The `submitJob()` method should return a string that uniquely
identifies the job. This identifier will be used in subsequent interaction.
Information Retrieval
---------------------
The methods in this group retrieve information about a previously submitted
Job. Given the heterogeneity of queue management servers, Katana imposes very
few requirements on the data a queue manager provides. One such requirement is
that a Job submitted to the farm can be uniquely identified by a number or
string. The Job identifier should be convertible to string and returned when
the `submitJob()` method is called. Subsequent metadata about a Job will be
requested by referencing a job by its identifier.
Minimum requirements of a Job:
- It should be possible to describe a Job's current state in terms of the
enumeration `Job.State`.
- It must be possible to execute an ordered list of commands.
- The assets specified by RenderOutputDefine nodes (for Disk Renders only)
should be accessible by both the render node and the artist's workstation.
Optional requirements of a Job:
- It should be possible to retrieve the filename of the render log that
corresponds to a Job. If supplied, this render log will be loaded into
Katana's **Render Log** tab and monitored as it updates.
Job Control
-----------
Notes on Network Topology
`````````````````````````
Katana's farm plug-in infrastructure makes a number of network topology
assumptions:
- The plug-in is capable of copying/moving/publishing files from the artist's
workstation (where the plug-in is running) to an asset/file or object store
that is also accessible by the render node.
- Conversely, the plug-in also assumes that files such as the render log
(produced by the render node) and any rendered images (in the case of a
Disk Render) are accessible from the artist's file system.
- The render node is able to connect via TCP/IP to the artist's workstation
running the Catalog Server which is used to receive pixel data (in the
case of Preview Renders and Live Renders).
If users are required to authenticate with the queue management server via
credentials, this should be handled either on first contact with the farm
management software, or on plug-in construction.
Render Farm Plug-in Reference Documentation
-------------------------------------------
.. autoclass:: FarmAPI.BaseFarmPlugin
.. autoclass:: FarmAPI.Job
.. autoclass:: FarmAPI.FarmPluginException
.. autoclass:: FarmAPI.JobNotFoundException
.. |->| unicode:: U+2192
.. |sparkles| unicode:: U+2728