Katana Queue¶

Step 1: Preparing the Remote Machine¶

The first step is to prepare the remote machine to enable it to run the agent.

1. Install Katana (4.0v1 or later) and any render plug-ins you would like to use.

   Note

   Ensure the installation paths for Katana and any render plug-ins on the remote machine match the local machine.

2. Ensure an SSH server is installed and running:

   CentOS:

   sudo yum -y install openssh-server
   

   Ubuntu:

   sudo apt-get install openssh-server
   sudo systemctl start sshd
   sudo systemctl enable sshd
   

3. Download and install the (Linux: StartAgents.sh and StopAgents.sh, Windows: StartAgents.ps1 and StopAgents.ps1) shell scripts to the home directory of the SSH user that has SSH access to the remote machine.

   StartAgents.sh¶

   #!/usr/bin/env bash
   
   set -eux
   
   # Command-line argument format
   if [ "$#" -lt 7 ]; then
     echo Insufficient arguments to start agent.
     exit 1
   fi
   
   KATANA_ROOT=$1
   KQ_DATA_ROOT=$2
   KQ_HOST=$3
   KQ_REQUEST_PORT=$4
   KQ_UPDATE_PORT=$5
   KQ_BROADCAST_PORT=$6
   NUM_AGENTS=$7
   
   echo Starting Katana Agent on $(hostname) to "${KQ_HOST}"
   
   # Determine interpreter path
   PYTHON_BIN="${KATANA_ROOT}/bin/python2.7/bin/pythonWrapper.sh"
   AGENT_SCRIPT_FOLDER="${KATANA_ROOT}/plugins/Resources/Core/Plugins/kq/Agent/"
   AGENT_SCRIPT="${AGENT_SCRIPT_FOLDER}/Agent.pyc"
   if [ ! -f "${AGENT_SCRIPT}" ]; then
     # For debug builds
     AGENT_SCRIPT="${AGENT_SCRIPT_FOLDER}/Agent.py"
   fi
   
   # Additional python path
   PYTHONPATH_INTERNAL=${PYTHONPATH:-}:"${KATANA_ROOT}/bin/python"
   PYTHONPATH_INTERNAL=$PYTHONPATH_INTERNAL:"${KATANA_ROOT}/plugins/Resources/Core/Plugins"
   
   # LD_LIBRARY_PATH for correct OpenSSL & PyZMQ
   LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-}:"${KATANA_ROOT}/bin:${KATANA_ROOT}/bin/OpenSSL/"
   
   # Run the script
   cd "${KATANA_ROOT}"
   
   for (( agent = 0; agent < ${NUM_AGENTS}; agent++ )); do
     bash -c "setsid env PYTHONPATH=${PYTHONPATH_INTERNAL} \\
         KQ_DATA_ROOT=${KQ_DATA_ROOT} \\
         LD_LIBRARY_PATH=${LD_LIBRARY_PATH} \\
         ${PYTHON_BIN} ${AGENT_SCRIPT} \\
         --minimal-env True \\
         --request-port ${KQ_REQUEST_PORT} \\
         --update-port ${KQ_UPDATE_PORT} \\
         --broadcast-port ${KQ_BROADCAST_PORT} \\
         ${KQ_HOST} \\
         </dev/null \\
         >"${KQ_DATA_ROOT}/logs/remote_agent_${agent}.log" \\
         2>&1 \\
         &"
   
     echo "Agent ${agent} Launched"
   done
   

   StopAgents.sh¶

   #!/usr/bin/env bash
   
   echo Stopping Agents on $(hostname)
   
   # TODO(DH): Determine if we can use pgrep in a cross-Linux compatible way.
   ps -ef | grep 'Agent.py' | grep -v grep | awk '{print $2}' | xargs -r kill -9
   

   StartAgents.ps1¶

   # Copyright (c) 2021 The Foundry Visionmongers Ltd. All Rights Reserved.
   # Start the Katana Queue Agent
   
   $KATANA_ROOT = $args[1]
   $KQ_DATA_ROOT = $args[2]
   $KQ_HOST = $args[3]
   $KQ_REQUEST_PORT = $args[4]
   $KQ_UPDATE_PORT = $args[5]
   $KQ_BROADCAST_PORT = $args[6]
   $NUM_AGENTS = $args[7]
   
   Write-Host "Starting Katana Agent on $Env:ComputerName to $KQ_HOST"
   
   $PYTHON_BIN = "$KATANA_ROOT\bin\python.exe"
   
   $AGENT_SCRIPT_FOLDER = "$KATANA_ROOT\plugins\Resources\Core\Plugins\kq\Agent"
   $AGENT_SCRIPT = "$AGENT_SCRIPT_FOLDER\Agent.pyc"
   
   $PYTHONPATH = "$KATANA_ROOT\bin\python;"
   $PYTHONPATH += "$KATANA_ROOT\plugins\Resources\Core\Plugins"
   $PYTHONPATH += $Env:PYTHONPATH + ";"
   
   # Build an environment for the Agent processes
   $EnvCopy = [System.Environment]::GetEnvironmentVariables().GetEnumerator() | sort name
   $NewEnvironment = @{}
   $NewEnvironment.Add("PYTHONPATH", "$PYTHONPATH")
   $NewEnvironment.Add("KQ_DATA_ROOT", $KQ_DATA_ROOT)
   
   foreach ($v in $EnvCopy)
   {
       if (!($NewEnvironment.ContainsKey($v.Name)))
       {
           $NewEnvironment.Add($v.Name, $v.Value)
       }
   }
   
   $EnvAsList = $NewEnvironment.GetEnumerator() | % { "$($_.Name)=$($_.Value)" }
   
   $startupProperties=[wmiclass]"Win32_ProcessStartup"
   $startupProperties.Properties["EnvironmentVariables"].Value = $EnvAsList
   
   # Start the Agents
   # See https://stackoverflow.com/questions/355988/how-do-i-deal-with-quote-characters-when-using-cmd-exe for cmd.exe escape rules
   for ($agent = 0; $agent -lt $NUM_AGENTS; $agent++) {
       $commandString = "cmd /s /k """"$PYTHON_BIN"" ""$AGENT_SCRIPT"" "
       $commandString += "--request-port $KQ_REQUEST_PORT "
       $commandString += "--update-port $KQ_UPDATE_PORT "
       $commandString += "--broadcast-port $KQ_BROADCAST_PORT "
       $commandString += "$KQ_HOST "
       $commandString += "1> ""$KQ_DATA_ROOT\logs\remote_agent_$agent.log"" 2>&1"""
   
       Write-Host "Launching $commandString"
       ([wmiclass]"win32_Process").create($commandString, "$KATANA_ROOT\bin", $startupProperties)
   }
   

   StopAgents.ps1¶

   # Copyright (c) 2021 The Foundry Visionmongers Ltd. All Rights Reserved.
   # Stops all Katana Queue Agents running on this host
   $agents = Get-WmiObject Win32_Process -Filter "name = 'cmd.exe'"
   
   foreach($process in $agents)
   {
       if ($process.CommandLine -ne $null -and $process.CommandLine.Contains('Agent.py'))
       {
           taskkill /f /pid $process.ProcessId /T
           Write-Host $process.CommandLine
       }
   }
   

4. (Linux only) Modify the file permissions to add executable permissions to the two shell scripts.

   CentOS & Ubuntu:

   cd $HOME
   chmod +x StartAgents.sh StopAgents.sh
   

5. Add the public SSH key from the local Katana workstation to the authorized keys of the remote machine.

   Caution

   The steps here may vary. The below assumes that you have never generated an SSH key pair on your local machine. You should verify this by first listing the contents of $HOME/.ssh/ (to ensure you don’t delete or overwrite an existing key).

   CentOS & Ubuntu:

   # Create the keypair (Local Machine)
   ssh-keygen
   
   Generating public/private rsa key pair.
   Enter file in which to save the key ($HOME/.ssh/id_rsa): [Enter key name]
   Enter passphrase (empty for no passphrase): [Leave blank]
   Enter same passphrase again:
   ...
   
   # Copy the PUBLIC key to the $HOME/.ssh/authorized_keys on the remote machine.
   
   (local machine): cat $HOME/.ssh/[key pair name].pub
   (remote machine): copy paste the contents of the .pub file into a new line in $HOME/.ssh/authorized_keys
   
   # Test the connection (Local Machine)
   ssh -i $HOME/.ssh/[key pair name] username@remotehost
   

Step 2: Launching Remote Agents¶

To launch an agent on the remote machine, you can use the Launch Remote Agents shelf item from the built-in KatanaQueue shelf. The shelf item opens a Launch Remote Agents dialog that let’s you configure how to access a remote machine and run the StartAgents.sh shell script on it via SSH.

The majority of the fields in the dialog will be filled in for you, but key fields to double check are:

SSH

Ensure the SSH details correspond to the remote user and the key pair that you set up in the previous step.

Agent

• The Host field refers to the host name or IP address of the remote workstation.
• Num Agents controls how many agents will be launched on the remote machine.
• Katana Install provides the path to the installation directory of Katana on the remote machine.

Click the Start Agents button to launch the specified number of Agents on the remote machine, and have them connect to the local Katana Queue.

Step 3: Stopping Remote Agents¶

The StopAgents script can be used to stop any remote Agents that have been started. Note: on Windows this script requires elevated privileges to run correctly.

Troubleshooting¶

Use the following checklist to troubleshoot the most common problems encountered when adding remote agents to your local Katana Queue.