Katana Queue ============ |sparkles| **New in Katana 4** Introduction to `Katana Queue` ------------------------------ Katana ships with a minimal render farm implementation, named `Katana Queue` (or ``kq`` for short), which is integrated with Katana using a :doc:`custom render farm plug-in` via Katana's :doc:`FarmAPI`. The Katana Queue system uses a certain number of `Agent` processes on the local workstation to host ``renderboot`` processes for performing renders. It is possible to configure the number of Agents running on the local machine. It is also possible to launch Agents on another machine that is accessible on the network (`Remote Agents`), and to add them to the pool of Agents that are managed by the local Katana Queue. To start a render on Katana Queue, right-click a 3D node in the :kat:ui:`Node Graph` tab, and choose :kat:ui:`Katana Queue > Preview Render`. A :kat:ui:`Katana Queue` tab is provided, which lists jobs that are managed by the local Katana Queue, with the ability to select jobs, display their render log, and stop and restart them. Both the Katana Queue farm plug-in and the :kat:ui:`Katana Queue` tab are shipped as source, providing developers with :ref:`farmapi-example-code`. ``kq`` and Agent Lifetime ------------------------- The lifetime of the local ``kq`` is tied to the lifetime of a Katana UI session: ``kq`` and the Agents that it launches are started when Katana starts up, and are shut down when Katana shuts down. If a render fails, and the Katana render process is terminated, the Agent that is running the process is typically kept alive. In case an Agent is terminated, ``kq`` launches another Agent to replace it, so that there are always the specified number of Agents running and available to process render jobs. Configuring the Number of Local Agents -------------------------------------- You can set the ``KQ_NUM_AGENTS`` environment variable to a numeric value to configure the number of Agents that ``kq`` launches when Katana starts up. .. code-block:: bash export KQ_NUM_AGENTS=10 By default, ``kq`` launches and maintains ``2`` agents. Adding Remote Agents -------------------- This section provides a step-by-step guide for adding additional `Remote Agents` to your local Katana Queue, allowing you to expand the compute capacity of your local workstation to other machines on your network. The simple process management technique behind Remote Agents uses a collection of scripts and SSH to start and stop agents on a remote workstation. Whilst this technique will work well for one or two artists or developers testing Katana Queue and Katana's :doc:`FarmAPI`, additional tools would be needed to fully manage this in a production environment. We therefore recommended using one of the existing, commercially-available render farm systems, and connecting to it via Katana's ``FarmAPI``. Step 1: Preparing the Remote Machine ```````````````````````````````````` The first step is to prepare the remote machine to enable it to run the agent. 1. Install Katana (4.0v1 or later) and any render plug-ins you would like to use. .. note:: Ensure the installation paths for Katana and any render plug-ins on the remote machine match the local machine. 2. Ensure an SSH server is installed and running: CentOS: .. code-block:: bash sudo yum -y install openssh-server Ubuntu: .. code-block:: bash sudo apt-get install openssh-server sudo systemctl start sshd sudo systemctl enable sshd 3. Download and install the (Linux: :download:`StartAgents.sh` and :download:`StopAgents.sh`, Windows: :download:`StartAgents.ps1` and :download:`StopAgents.ps1`) shell scripts to the home directory of the SSH user that has SSH access to the remote machine. .. literalinclude:: StartAgents.sh :caption: :download:`StartAgents.sh` :language: bash .. literalinclude:: StopAgents.sh :caption: :download:`StopAgents.sh` :language: bash .. literalinclude:: StartAgents.ps1 :caption: :download:`StartAgents.ps1` :language: powershell .. literalinclude:: StopAgents.ps1 :caption: :download:`StopAgents.ps1` :language: powershell 4. (Linux only) Modify the file permissions to add executable permissions to the two shell scripts. CentOS & Ubuntu: .. code-block:: bash cd $HOME chmod +x StartAgents.sh StopAgents.sh 5. Add the public SSH key from the local Katana workstation to the authorized keys of the remote machine. .. caution:: The steps here may vary. The below assumes that you have never generated an SSH key pair on your local machine. You should verify this by first listing the contents of ``$HOME/.ssh/`` (to ensure you don't delete or overwrite an existing key). CentOS & Ubuntu: .. code-block:: bash # Create the keypair (Local Machine) ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key ($HOME/.ssh/id_rsa): [Enter key name] Enter passphrase (empty for no passphrase): [Leave blank] Enter same passphrase again: ... # Copy the PUBLIC key to the $HOME/.ssh/authorized_keys on the remote machine. (local machine): cat $HOME/.ssh/[key pair name].pub (remote machine): copy paste the contents of the .pub file into a new line in $HOME/.ssh/authorized_keys # Test the connection (Local Machine) ssh -i $HOME/.ssh/[key pair name] username@remotehost Step 2: Launching Remote Agents ``````````````````````````````` To launch an agent on the remote machine, you can use the :kat:ui:`Launch Remote Agents` shelf item from the built-in :kat:ui:`KatanaQueue` shelf. The shelf item opens a :kat:ui:`Launch Remote Agents` dialog that let's you configure how to access a remote machine and run the ``StartAgents.sh`` shell script on it via SSH. .. image:: AgentManager_screenshot_420x529.png :alt: Screenshot of the *Launch Remote Agents* dialog The majority of the fields in the dialog will be filled in for you, but key fields to double check are: :kat:ui:`SSH` Ensure the SSH details correspond to the remote user and the key pair that you set up in the previous step. :kat:ui:`Agent` - The :kat:ui:`Host` field refers to the host name or IP address of the remote workstation. - :kat:ui:`Num Agents` controls how many agents will be launched on the remote machine. - :kat:ui:`Katana Install` provides the path to the installation directory of Katana on the remote machine. Click the :kat:ui:`Start Agents` button to launch the specified number of Agents on the remote machine, and have them connect to the local Katana Queue. Step 3: Stopping Remote Agents `````````````````````````````` The `StopAgents` script can be used to stop any remote Agents that have been started. Note: on Windows this script requires elevated privileges to run correctly. Troubleshooting ``````````````` Use the following checklist to troubleshoot the most common problems encountered when adding remote agents to your local Katana Queue. +-----------------------------------------------------------------------------+-------------------------+ | ``KQ_DATA_ROOT`` environment variable set to common network share location. | |ballot_box_with_check| | +-----------------------------------------------------------------------------+-------------------------+ | File & folder permissions allow read and write access from local and remote | |ballot_box_with_check| | | agent to ``KQ_DATA_ROOT``. | | +-----------------------------------------------------------------------------+-------------------------+ | Remote agent can communicate with local workstation: | |ballot_box_with_check| | | | | | - (remote) ping [local IP address] | | | - (remote) ping [local host name] | | | | | | If the remote host can ping the IP address but not resolve the host name | | | you may need to configure an entry in the ``/etc/hosts`` file of the remote | | | machine, or contact your system administrator to correctly configure DNS | | | resolution to the local machine. | | +-----------------------------------------------------------------------------+-------------------------+ | Local firewall allows connections from the remote host. | |ballot_box_with_check| | +-----------------------------------------------------------------------------+-------------------------+ .. |sparkles| unicode:: U+2728 .. |ballot_box_with_check| unicode:: U+2611