Toolserver:Job scheduling

We are currently testing a new way to run tools on the Toolserver: batch job scheduling. Batch jobs will be familiar to many people who have used university, research or mainframe systems. The main difference between interactive jobs (which we currently use) and batch jobs is that when you run an interactive job, it starts immediately, runs to completion, then exits. A batch job is submitted to the job server; when sufficient system resources are available, the job server starts the job on a suitable (idle) host. The job might be suspended during execution if load is too high, and will resume when resources are available again. After submitting a batch job, you can log out and come back later when the job has finished to examine its output. If you like, you can ask to receive mail when the job starts or finished.

For users, the main advantage of batch jobs is that you do not need to worry about where to start a job, or whether the job needs to use a particular nice level, and so on; the job server will handle that for you. As we add more job execution hosts to the cluster, your jobs will automatically take advantage of the new resources with no changes needed from you.

For admins, batch jobs give us tighter control over resources allocation and usage, and allow us to see more clearly how the Toolserver is being used.

The batch job software we have chosen is Sun Grid Engine. Full documentation for users is available here; some common examples are described below.

Queues and jobs
The basic method by which jobs are scheduled is the queue. A queue is a list of jobs which have been submitted. Each queue has a certain number of execution slots; when free slots are available, jobs are moved into them and begin running. When no more slots are available, jobs are queued, and will start when an execution slot is available.

Submitting a job
To submit a job, use the qsub command:

% qsub -N test test.sh

This submits the shell script "test.sh" as a job with the name "test". Giving the job a name is not required, but is recommended so you can easily identify the job. If you want to run a binary instead of a shell script, use qsub -b y.

If you want to receive mail when a job completes, use the -m e option:

% qsub -N test -m e test.sh

To receive mail when a job starts and when it finishes, use -m be.

When your job is finished, its output (stdout) will be written to the file test.sh.oX, where X is increased for each run of the job. The job's standard error is written to test.sh.eX.

Displaying jobs
To display your jobs, use the qstat command:

% qstat job-ID prior   name       user         state submit/start at     queue                          slots ja-task-ID -      8 0.55500 test       rriver       r     09/15/2009 16:15:51 all.q@willow.toolserver.org        1

Here, the job test is running (state r), in the all.q queue (the default) on willow.

Interactive jobs
An interactive job is a special kind of job which, instead of running a command, requests a shell on an idle system. To start an interactive job, use qlogin:

% qlogin Your job 11 ("QLOGIN") has been submitted waiting for interactive job to be scheduled ... Your interactive job 11 has been successfully scheduled. Establishing builtin session to host willow ... Sun Microsystems Inc.  SunOS 5.10      Generic January 2005 %

To exit the interactive session, exit the shell (e.g. by typing CTRL+D).

Special queues
The default queue, all.q, includes all login servers in the cluster. If you have a job which can only be run on a particular type of host, you can request that it only be executed on a particular operating system.

To run a job on Solaris only:

% qsub -l arch=sol-amd64 -N test test.sh

To run a job on Linux only:

% qsub -l arch=lx24-amd64 -N test test.sh

We strongly recommend that you write jobs which can run on either kind of server, and use the default queue. This will provide the most flexibility when scheduling your job. (In particular, we are unlikely to add more Linux servers, so writing jobs which also run on Solaris will increase the resources available to your jobs.)