Katana Queue¶

✨ New in Katana 4

Introduction to `Katana Queue`¶

Katana ships with a minimal render farm implementation, named Katana Queue (or kq for short), which is integrated with Katana using a custom render farm plug-in via Katana’s FarmAPI.

The Katana Queue system uses a certain number of Agent processes on the local workstation to host renderboot processes for performing renders. It is possible to configure the number of Agents running on the local machine. It is also possible to launch Agents on another machine that is accessible on the network (Remote Agents), and to add them to the pool of Agents that are managed by the local Katana Queue.

To start a render on Katana Queue, right-click a 3D node in the Node Graph tab, and choose Katana Queue > Preview Render.

A Katana Queue tab is provided, which lists jobs that are managed by the local Katana Queue, with the ability to select jobs, display their render log, and stop and restart them.

Both the Katana Queue farm plug-in and the Katana Queue tab are shipped as source, providing developers with FarmAPI Example Code.

`kq` and Agent Lifetime¶

The lifetime of the local kq is tied to the lifetime of a Katana UI session: kq and the Agents that it launches are started when Katana starts up, and are shut down when Katana shuts down.

If a render fails, and the Katana render process is terminated, the Agent that is running the process is typically kept alive. In case an Agent is terminated, kq launches another Agent to replace it, so that there are always the specified number of Agents running and available to process render jobs.

Configuring the Number of Local Agents¶

You can set the KQ_NUM_AGENTS environment variable to a numeric value to configure the number of Agents that kq launches when Katana starts up.

export KQ_NUM_AGENTS=10

By default, kq launches and maintains 2 agents.

Adding Remote Agents¶

This section provides a step-by-step guide for adding additional Remote Agents to your local Katana Queue, allowing you to expand the compute capacity of your local workstation to other machines on your network.

The simple process management technique behind Remote Agents uses a collection of scripts and SSH to start and stop agents on a remote workstation. Whilst this technique will work well for one or two artists or developers testing Katana Queue and Katana’s FarmAPI, additional tools would be needed to fully manage this in a production environment. We therefore recommended using one of the existing, commercially-available render farm systems, and connecting to it via Katana’s FarmAPI.

Step 1: Preparing the Remote Machine¶

The first step is to prepare the remote machine to enable it to run the agent.

Install Katana (4.0v1 or later) and any render plug-ins you would like to use.

Note

Ensure the installation paths for Katana and any render plug-ins on the remote machine match the local machine.

Ensure an SSH server is installed and running:

CentOS:

sudo yum -y install openssh-server

Ubuntu:

sudo apt-get install openssh-server
sudo systemctl start sshd
sudo systemctl enable sshd

Download and install the (Linux: StartAgents.sh and StopAgents.sh, Windows: StartAgents.ps1 and StopAgents.ps1) shell scripts to the home directory of the SSH user that has SSH access to the remote machine.

StartAgents.sh¶

#!/usr/bin/env bash

set -eux

# Command-line argument format
if [ "$#" -lt 7 ]; then
  echo Insufficient arguments to start agent.
  exit 1
fi

KATANA_ROOT=$1
KQ_DATA_ROOT=$2
KQ_HOST=$3
KQ_REQUEST_PORT=$4
KQ_UPDATE_PORT=$5
KQ_BROADCAST_PORT=$6
NUM_AGENTS=$7

echo Starting Katana Agent on $(hostname) to "${KQ_HOST}"

# Determine interpreter path
PYTHON_BIN="${KATANA_ROOT}/bin/python.sh"
AGENT_SCRIPT_FOLDER="${KATANA_ROOT}/plugins/Resources/Core/Plugins/kq/Agent/"
AGENT_SCRIPT="${AGENT_SCRIPT_FOLDER}/Agent.pyc"
if [ ! -f "${AGENT_SCRIPT}" ]; then
  # For debug builds
  AGENT_SCRIPT="${AGENT_SCRIPT_FOLDER}/Agent.py"
fi

# Additional python path
PYTHONPATH_INTERNAL=${PYTHONPATH:-}:"${KATANA_ROOT}/bin/python"
PYTHONPATH_INTERNAL=$PYTHONPATH_INTERNAL:"${KATANA_ROOT}/plugins/Resources/Core/Plugins"

# LD_LIBRARY_PATH for correct OpenSSL & PyZMQ
LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-}:"${KATANA_ROOT}/bin:${KATANA_ROOT}/bin/OpenSSL/"

# Run the script
cd "${KATANA_ROOT}"

for (( agent = 0; agent < ${NUM_AGENTS}; agent++ )); do
  bash -c "setsid env PYTHONPATH=${PYTHONPATH_INTERNAL} \\
      KQ_DATA_ROOT=${KQ_DATA_ROOT} \\
      LD_LIBRARY_PATH=${LD_LIBRARY_PATH} \\
      ${PYTHON_BIN} ${AGENT_SCRIPT} \\
      --minimal-env True \\
      --request-port ${KQ_REQUEST_PORT} \\
      --update-port ${KQ_UPDATE_PORT} \\
      --broadcast-port ${KQ_BROADCAST_PORT} \\
      ${KQ_HOST} \\
      </dev/null \\
      >"${KQ_DATA_ROOT}/logs/remote_agent_${agent}.log" \\
      2>&1 \\
      &"

  echo "Agent ${agent} Launched"
done

StopAgents.sh¶

#!/usr/bin/env bash

echo Stopping Agents on $(hostname)

# TODO(DH): Determine if we can use pgrep in a cross-Linux compatible way.
ps -ef | grep 'Agent.py' | grep -v grep | awk '{print $2}' | xargs -r kill -9

StartAgents.ps1¶

# Copyright (c) 2021 The Foundry Visionmongers Ltd. All Rights Reserved.
# Start the Katana Queue Agent

$KATANA_ROOT = $args[1]
$KQ_DATA_ROOT = $args[2]
$KQ_HOST = $args[3]
$KQ_REQUEST_PORT = $args[4]
$KQ_UPDATE_PORT = $args[5]
$KQ_BROADCAST_PORT = $args[6]
$NUM_AGENTS = $args[7]

Write-Host "Starting Katana Agent on $Env:ComputerName to $KQ_HOST"

$PYTHON_BIN = "$KATANA_ROOT\bin\python.exe"

$AGENT_SCRIPT_FOLDER = "$KATANA_ROOT\plugins\Resources\Core\Plugins\kq\Agent"
$AGENT_SCRIPT = "$AGENT_SCRIPT_FOLDER\Agent.pyc"

$PYTHONPATH = "$KATANA_ROOT\bin\python;"
$PYTHONPATH += "$KATANA_ROOT\plugins\Resources\Core\Plugins"
$PYTHONPATH += $Env:PYTHONPATH + ";"

# Build an environment for the Agent processes
$EnvCopy = [System.Environment]::GetEnvironmentVariables().GetEnumerator() | sort name
$NewEnvironment = @{}
$NewEnvironment.Add("PYTHONPATH", "$PYTHONPATH")
$NewEnvironment.Add("KQ_DATA_ROOT", $KQ_DATA_ROOT)

foreach ($v in $EnvCopy)
{
    if (!($NewEnvironment.ContainsKey($v.Name)))
    {
        $NewEnvironment.Add($v.Name, $v.Value)
    }
}

$EnvAsList = $NewEnvironment.GetEnumerator() | % { "$($_.Name)=$($_.Value)" }

$startupProperties=[wmiclass]"Win32_ProcessStartup"
$startupProperties.Properties["EnvironmentVariables"].Value = $EnvAsList

# Start the Agents
# See https://stackoverflow.com/questions/355988/how-do-i-deal-with-quote-characters-when-using-cmd-exe for cmd.exe escape rules
for ($agent = 0; $agent -lt $NUM_AGENTS; $agent++) {
    $commandString = "cmd /s /k """"$PYTHON_BIN"" ""$AGENT_SCRIPT"" "
    $commandString += "--request-port $KQ_REQUEST_PORT "
    $commandString += "--update-port $KQ_UPDATE_PORT "
    $commandString += "--broadcast-port $KQ_BROADCAST_PORT "
    $commandString += "$KQ_HOST "
    $commandString += "1> ""$KQ_DATA_ROOT\logs\remote_agent_$agent.log"" 2>&1"""

    Write-Host "Launching $commandString"
    ([wmiclass]"win32_Process").create($commandString, "$KATANA_ROOT\bin", $startupProperties)
}

StopAgents.ps1¶

# Copyright (c) 2021 The Foundry Visionmongers Ltd. All Rights Reserved.
# Stops all Katana Queue Agents running on this host
$agents = Get-WmiObject Win32_Process -Filter "name = 'cmd.exe'"

foreach($process in $agents)
{
    if ($process.CommandLine -ne $null -and $process.CommandLine.Contains('Agent.py'))
    {
        taskkill /f /pid $process.ProcessId /T
        Write-Host $process.CommandLine
    }
}

(Linux only) Modify the file permissions to add executable permissions to the two shell scripts.

CentOS & Ubuntu:
```
cd $HOME
chmod +x StartAgents.sh StopAgents.sh
```

Add the public SSH key from the local Katana workstation to the authorized keys of the remote machine.

Caution

The steps here may vary. The below assumes that you have never generated an SSH key pair on your local machine. You should verify this by first listing the contents of $HOME/.ssh/ (to ensure you don’t delete or overwrite an existing key).

CentOS & Ubuntu:

# Create the keypair (Local Machine)
ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key ($HOME/.ssh/id_rsa): [Enter key name]
Enter passphrase (empty for no passphrase): [Leave blank]
Enter same passphrase again:
...

# Copy the PUBLIC key to the $HOME/.ssh/authorized_keys on the remote machine.

(local machine): cat $HOME/.ssh/[key pair name].pub
(remote machine): copy paste the contents of the .pub file into a new line in $HOME/.ssh/authorized_keys

# Test the connection (Local Machine)
ssh -i $HOME/.ssh/[key pair name] username@remotehost

Step 2: Launching Remote Agents¶

To launch an agent on the remote machine, you can use the Launch Remote Agents shelf item from the built-in KatanaQueue shelf. The shelf item opens a Launch Remote Agents dialog that let’s you configure how to access a remote machine and run the StartAgents.sh shell script on it via SSH.

Screenshot of the *Launch Remote Agents* dialog

The majority of the fields in the dialog will be filled in for you, but key fields to double check are:

SSH

Ensure the SSH details correspond to the remote user and the key pair that you set up in the previous step.

Agent

The Host field refers to the host name or IP address of the remote workstation.
Num Agents controls how many agents will be launched on the remote machine.
Katana Install provides the path to the installation directory of Katana on the remote machine.

Click the Start Agents button to launch the specified number of Agents on the remote machine, and have them connect to the local Katana Queue.

Step 3: Stopping Remote Agents¶

The StopAgents script can be used to stop any remote Agents that have been started. Note: on Windows this script requires elevated privileges to run correctly.

Troubleshooting¶

Use the following checklist to troubleshoot the most common problems encountered when adding remote agents to your local Katana Queue.

`KQ_DATA_ROOT` environment variable set to common network share location.	☑
File & folder permissions allow read and write access from local and remote agent to `KQ_DATA_ROOT`.	☑
Remote agent can communicate with local workstation: (remote) ping [local IP address] (remote) ping [local host name] If the remote host can ping the IP address but not resolve the host name you may need to configure an entry in the `/etc/hosts` file of the remote machine, or contact your system administrator to correctly configure DNS resolution to the local machine.	☑
Local firewall allows connections from the remote host.	☑