Render Farm Plug-ins ==================== |sparkles| **New in Katana 4** Introduction to Render Farm Plug-ins ------------------------------------ The :doc:`FarmAPI` Python package provides a base class for a type of plug-in that allows developers to closely integrate a specific render farm system with Katana: :py:class:`FarmAPI.BaseFarmPlugin` The ``BaseFarmPlugin`` class provides an interface that allow Katana to: - Send Preview Render, Live Render, and Disk Render jobs to a render farm to be executed remotely. - View the list of jobs running on the render farm. - Retrieve information about a particular render farm job, such as the current state and the job's render log. - Stop/restart previously submitted jobs. In general terms, a ``BaseFarmPlugin`` is an expression of the adapter pattern: render farm-specific functionality can be expressed within the implementation of the various methods of the interface. Render farm plug-ins are managed using the :py:mod:`FarmAPI.FarmPluginManager` module. Terminology ----------- `Render Farm` - term given to the collective hardware and software that extend the local compute resources of an artist for the purposes of rendering images. `Render Job` - a series of stages/commands that describe the work required to produce a rendered image. `Queue Management Server` - a software component that manages job requests and schedules their execution based on priority or resource requirements. `Render Node` - a compute resource managed by the queue management server. Jobs consume the resources of a render node. Quick Start ----------- This section describes how to implement a render farm plug-in. It assumes you have a render farm accessible from your network, and that you can communicate with the queue management server in order to submit and retrieve information about jobs. 1. Subclass :py:class:`FarmAPI.BaseFarmPlugin` and register your plug-in with Katana. The first stage requires creating a class derived from :py:class:`FarmAPI.BaseFarmPlugin` which describes the interface of methods you need to implement to integrate your render farm with Katana:: from Katana import FarmAPI class MyFarmPlugin(FarmAPI.BaseFarmPlugin): pass PluginRegistry = [("FarmPlugin", 2.0, "My Farm Plugin", MyFarmPlugin)] 2. Add the plug-in to your ``KATANA_RESOURCES`` path in a ``Plugins`` subfolder. For example:: # For a plug-in at /home/john/katana_resources/Plugins/MyFarmPlugin.py: KATANA_RESOURCES=/home/john/katana_resources 3. Implement the necessary methods (see below) to allow Katana to communicate with your render farm. Methods can be grouped into three categories. Some methods are optional; not implementing them will disable the corresponding functionality within Katana: 1. Job Submission - this group of methods handles the preparation of a Job and its submission to your queue management server. 2. Information Retrieval - this group of methods allows Katana to fetch information about a previously submitted Job such as its current state or the contents of its render log. 3. Job Control - these methods allow artists to start/restart and stop Jobs. 4. Test. Farm plug-ins are invoked by Katana when the user selects a remote render for a particular render type (Disk Render, Preview Render, or Live Render). The resources (e.g. the Op Tree) are copied to a location accessible to the render farm, and the commands required to generate the rendered image outputs are passed to the render plug-in allowing it to configure a Job in the domain-specific language of the queue management server. Job Submission -------------- The methods in this group provide the opportunity to configure a Render Job and submit it to the farm's queue management server. Job submission is a three-stage process: 1. Creation of temporary render files: The Katana render process generates a number of temporary files. An example of these temporary files includes a serialised copy the the Op Tree used by Geolib3 to generate the scene description. This stage is further broken down into the following steps: 1. Creation of a render context: Katana allocates a globally unique (UUID v4) render context. 2. Creation of a temporary folder: The farm plug-in is asked to create and return the path to a temporary folder that is accessible by both the artist's workstation and the render node. This method is also passed the render context which can be used as part of file path to guarantee uniqueness. 3. Copying of render files: A list of local files will be passed to the farm plug-in which should be copied to a remote location accessible by the render node. Their new (remote) paths should be returned to Katana. Whilst it is not necessary to have copied the files before returning from this method, the copy operation should be complete before the job is started by the queue management server. 2. Configure environment: The plug-in will be asked to describe the remote Katana installation in a L{BaseFarmPlugin.InstallationMetadata} instance. This includes the path to the Katana installation. The plug-in will also be asked to provide additional environment variables that should be included in the process environment. 3. Submit Job: The information gathered in the preceding stages is passed to the C{submitJob()} method in a L{FarmAPI.FarmPluginManager.JobRequest} instance. At this point, the plug-in is able to create a job description using the domain-specific language of the render farm, including applying any tags or resource allocation requirements, and then submit it for execution. The C{submitJob()} method should return a string that uniquely identifies the job. This identifier will be used in subsequent interaction. Information Retrieval --------------------- The methods in this group retrieve information about a previously submitted Job. Given the heterogeneity of queue management servers, Katana imposes very few requirements on the data a queue manager provides. One such requirement is that a Job submitted to the farm can be uniquely identified by a number or string. The Job identifier should be convertible to string and returned when the C{submitJob()} method is called. Subsequent metadata about a Job will be requested by referencing a job by its identifier. Minimum requirements of a Job: - It should be possible to describe a Job's current state in terms of the enumeration C{Job.State}. - It must be possible to execute an ordered list of commands. - The assets specified by RenderOutputDefine nodes (for Disk Renders only) should be accessible by both the render node and the artist's workstation. Optional requirements of a Job: - It should be possible to retrieve the filename of the render log that corresponds to a Job. If supplied, this render log will be loaded into Katana's B{Render Log} tab and monitored as it updates. Job Control ----------- Notes on Network Topology ````````````````````````` Katana's farm plug-in infrastructure makes a number of network topology assumptions: - The plug-in is capable of copying/moving/publishing files from the artist's workstation (where the plug-in is running) to an asset/file or object store that is also accessible by the render node. - Conversely, the plug-in also assumes that files such as the render log (produced by the render node) and any rendered images (in the case of a Disk Render) are accessible from the artist's file system. - The render node is able to connect via TCP/IP to the artist's workstation running the Catalog Server which is used to receive pixel data (in the case of Preview Renders and Live Renders). If users are required to authenticate with the queue management server via credentials, this should be handled either on first contact with the farm management software, or on plug-in construction. Render Farm Plug-in Reference Documentation ------------------------------------------- .. autoclass:: FarmAPI.BaseFarmPlugin .. autoclass:: FarmAPI.Job .. autoclass:: FarmAPI.FarmPluginException .. autoclass:: FarmAPI.JobNotFoundException .. |->| unicode:: U+2192 .. |sparkles| unicode:: U+2728