3 Job Submission

_images/the_basics.jpg

To understand SquidNet’s basic concepts, make sure to checkout these power point slides (pdf document).

SquidNet is an advanced render farm management system that controls the distribution and processing of single-frame, multi-frame, video generation and workflow management requests. A SquidNet render farm consists of submitting workstations (clients), a workflow manager (master) and rendering/processing engines (slaves). Jobs are submitted from clients using SquidNet’s graphical user (GUI) and command line (CLI) interfaces. Once received, the master persistently stores these jobs in its local render queue and intelligently breaks up the requests into smaller segments that get evenly distributed to available render farm processing (rendering) nodes. These small segments, called job slices, contain rendering instructions for the processing nodes. While render nodes are processing requests, the master idly stands by waiting for job completion responses from these nodes or for new incoming job requests from submitting client nodes.

The Master can be run on any networked computer (Windows, Mac, Linux), uses very little system resources and is heavily multi-threaded to support thousands of queued jobs and hundreds of client workstations and render nodes. Client nodes can also run on any computer platform and can co-exist with your 3D applications. Slave nodes are the render farm work horses and can reside on any platform as well.

Although SquidNet’s primary role is to manage render farm processing operations, it’s also a powerful tool for workflow management. With SquidNet’s transaction framework, you can schedule batch jobs that contain user-defined scripts than can be executed on any render node at any time of the day. Transactions also allow you to queue job request chains, video generation, output folder archiving and image transfer operations to remote locations using secure communications channels (SFTP/FTP, SSH, OpenSSL, etc…).

3.1 How things work

3.1.1 Rendering Workflow

_images/workflow.png

The figure above provides a high-level description of SquidNet’s processing pipeline:

  1. Client UI configuration information is stored in configuration folder that’s accessible by all job submitting nodes (ie; networked).
  2. Clients (GUI or CLI) submit jobs to the render farm. GUI configuration information and settings are read from configuration folder.
  3. All jobs are submitted to Master node which:
  1. Receives jobs requests in it’s input queue.
  2. Validates job quest (job type, rendering parameters, etc…) .
  3. Adds job to main processing queue (job queue).
  4. Evenly divides job request into slices (as instructed by job request).
  5. Checks availability and status of render nodes.
  6. Distributes job slices to render nodes.
  7. Waits for responses from render nodes. Render node responses added to output queue.
  8. Waits for new requests from clients or for completion messages from rendering nodes.
  1. Clients monitor job queue activity from GUI/CLI interfaces.
  2. Upon job completion, Master node removes job entries from job queue.
  3. Images/frames are rendered to output folder.

3.1.2 Job Workflow

_images/user_workflow.png

The figure above provides a high-level description of SquidNet’s user pipeline:

  1. Named projects are stored in configuration folder (preferably on a network share).
  2. Within the named projects, job profiles are stored. Job profiles contain application type (template), scene path, output folder and other processing parameters required for render farm processing.
  3. Users open job profiles stored in project folders and submit them to the render farm (ie; master node).
  4. The Master divides up the job request into smaller more-managable fragments called slices. Each slice contains processing instructions for the rendering nodes.
  5. The Master distributes slices to available render nodes.
  6. Each node renders scene frames (images) to specified network storage device.

3.2 Project Manager

_images/project_manager.jpg

Project Manager

The SquidNet user interface manages all jobs using project folders. Use the Project Manager to arrange all your customer projects in a meaningful manner.

Open the Project Manager (Project->Project Manager…) and create a new project by selecting the “New Project” folder. By default, projects are stored under the configuration path defined in the Preferences dialog. Once created, double-click on project to open the project window.

_images/project_jobs.jpg

Job profiles in project folder

3.3 Submitting a Job

To submit a job to the farm, do the following:

  1. Assuming you have your project folder open, press the “New Job” button.
  2. In the “Template” dropdown list, select the desired application template.
  3. On the left, set rendering parameters.
  4. On the right, select the “Application” profile then fill the rest of the fields.
  5. Press “Submit” button to queue to the farm.