Using Hypnotoad

 

Welcome to Hypnotoad.  

Hypnotoad uses the SLURM job queueing system.   It's the same system used by the much larger Midway cluster that the Research Computing Center runs.  Hypnotoad, aside from being a good [free!] place to run some jobs, is a good place to 'cut your teeth' when learning how to use a cluster.

Clusters:

In very general terms, there are two classes of computers you need to be aware of, the head node and the compute nodes.

Hypnotoad:    The cluster "head node".   You'll interact with the cluster by ssh'ing to hypnotoad.uchicago.edu.   You'll use your CNet credentials to log in after we activate your access.  From here you'll work on your code [edit, upload, download, compile..].  You'll also submit your compute jobs to the SLURM queueing system from hypnotoad.

Hypnonodes:   These are the workers behind hypnotoad, also generically called the compute nodes.  SLURM keeps track of the hypnonodes and assigns them compute jobs based on their availability and their abiltiy to run a particular job. 

There are, currently, 3 classes of hypnonodes..

hypnonode-0-0 ... hypnonode-0-7    Included in the default node "partition".  Your job may run here if you'd don't specify a partition of nodes to use.   If you run MPI jobs, you should specify these nodes with the #SBATCH --constraint=ib flag [details below].

hypnonode-1-0 ... hypnonode-1-4    The CUDA/OpenCL nodes.   The 1-x hypnonodes are not grouped with the 0-x or 2-x nodes in the default node partition.   To send jobs [preferably only GP-GPU accellerated jobs] to these nodes, you will use the   #SBATCH --partition="cuda"  directive in your job scripts [details below]

hypnonode-2-0 ... hypnonode-2-5    Dual-Socket Dual-Core AMD Opteron 2.6GHz, 8GB RAM.    These nodes are grouped in the default partition.  If you don't specify where to run your jobs in your job script, they may run on these machines.

You can see what Hypnotoad and the HypnotNodes are up to at the Ganglia monitoring page. http://hypnotoad.uchicago.edu/ganglia.

Getting Stuff Done..

Though it's OK to do development and related functions [upload/download..] on hypnotoad, the real work is farmed out to the hypnonodes. 

You'll need a couple things to get to get going.

  • The first is the actual work you want to do.  That can be an application you wish to run on a node or nodes or it could be data that another application [like Mathematica] requires to run.

  • After you have work to do, you need to create a submission script that tells SLURM how to run your job.  You'll pass this script to SLURM and SLURM will use the information in that script to run your compute job.

 

SUBMISSION SCRIPT..

A simple script is required to submit your computation job to the hypnonodes.   This submission script describes a number of requirements and behaviors for a compute job.   You can specify what resources the job requires to run so you'll be matched up with the appropriate hardware and how many of those resources will need.  You'll also tell the cluster what to run and where to deliver the results of those compuations in this submission script.

 

Inside the SLURM script, most directives to SLURM are preceded by "#SBATCH" .

Comments are also preceded by "#" but they aren't follow by "SBATCH" and shouldn't be in all caps.  For clarity I prefer using multiple hash marks to make the comment lines easy to identify.      

Here's an script example Richard Dahringer [the GeoSci sys admin] provided to me with some comments added.  Comments precede directives.

## like any shell script, we start by providing the path to a shell which will execute the script #!/bin/sh  

##We follow by using the #SBATCH command to control SLURM ##First we'll set the number of cores you'll require to One.  

##All systems except 0-1 have 4-cores.

#SBATCH --ntasks=1

##job-name defines a name for your job. In this example, a job named "test" would appear in the job queue when you type "squeue scriptname" at the command prompt

#SBATCH --job-name=test

##define where your output should be written.

##In this example your job output is written to a file named "test.out" in the directory you're running this from.

##You can specfy a path with the filename if you chose.

#SBATCH --output=test.out

##the --time directive sets the Wall Time for the job.

##In the following example we tell SLURM the job can not run for more than one hour.  

## if it's still running after one hour it will be terminated.

#SBATCH --time=1:00:00

##"module" is an exception to the "#SBATCH" rule.   Where #SBATCH tells SLURM how to run the job, module tells slurm what extra stuff we'll need from the system to get the job to execute. ##below we're telling slurm to unload the standard openmpi libraries   ##..and then load the faster Intel MPI libraries in their place.

module unload openmpi

module load intelmpi

##last bit.

##You need to give SLURM some work to do.

##In this example we're telling SLURM to run the Super_Duper_Code application that I've compiled. ~/my_research_code/Super_Duper_Code

Once we've typed that script up, we'll save it and make sure it's executable.  For the purposes of this document, let's assume the script has been saved as "SLURM_job_script.scp"

Now that we have the SLURM_job_script.scp file, we need to set it to be executable.   To do that, I'll type    "chmod +x SLURM_job_script.scp" into the shell.   You may need to add the path to the script if you're not in the same directory.

USE OTHER APPS TO PROCESS MY WORK..

Your job need not be a compiled executable.  You can also tell pre-compiled applications to process input that you provide.    For example, If you wanted to run calculations in Mathematica you would tell SLURM you'll be using Mathematica by loading a module with Mathematica-specific information and then you'll tell SLURM what to do by providing the command to run the Mathematica job at the end of your script.

 

##The following "module" directive tells SLURM you'll need to load the module for Mathematica

##Other pre-compiled applications have similar modules available

module load Mathematica   

##You still have to tell SLURM what you want to run

##At the end of the script we would invoke Mathematica in place of "~/my_research_code/Super_Duper_Code" from the previous example   ##This is same syntax you would use to run Mathematica interactively from the commandline

math -noprompt -script test.m

 

 

MATHEMATICA NOTE**.. though you can load a workbook in the Mathematica gui, to use the command line you have to create a *.m file from the workbook and pass that to the Mathematica "math" executable.

 

GET YOUR WORK RUNNING..

Once you have your submission script, you'll be ready to submit your job.   SLURM includes some command-line utilities to allow you to summit, monitor, and controll jobs.  For your convenience, they very cleverly start with "s".  These commands are invoked by you in a shell on hypnotoad.uchicago.edu.

sbatch <script_name>   --the command to submit a job to the job queues.  This is the more automated approach

srun <script_name>    --the interactive job command.  This is the "do this now" command

squeue       --see what jobs are sitting in the job queue [what's waiting and what's currently running on the nodes].  You can also use the qstat command, which formats the output slightly diferently.  

The SLURM script that you'll create is submitted to the scheduler with the "sbatch" or "srun" commands.  Generally, we prefer you stick with "sbatch".

 

THAT'S IT?..

There are a lot of other directives you can feed in with the SBATCH command.  

You can tell it how many compute nodes to run a job on [if you software supports message passing and can run on multiple nodes], how many CPUs [not cores] to use, how much RAM to reserve, which nodes or groups of nodes to use….  If you don't use those directives, you get the system defaults.

All these options may see a bit obsessive but there are instances when they can be important.  Imagine you have an application that utilizes multiple cores efficiently and which requires 8GB of RAM to avoid paging.  If you don't specify these minimum requirements, you might get stuck on a node with insufficient resources.

SLURM will back-fill under-utilized nodes.   Suppose I submit a number of jobs and I specify one CPU core per job.  Most of the hypnonodes have 4 cores and SLURM is aware of this.  It will run up to four single-core jobs on each node simultaneously if needed to get them out of the queue.  It's possible that your SMP-friendly, RAM-hungry job could get stuck on a 4-core hypnonode that's already running 3 jobs that have used up most of the available RAM.  :-(

TELL ME MORE

That's hopefully enough to get you started.  We'll continue to work on our documentation but in the meantime I've provided links to some more thorough resources below.

Some common slurm directives.. http://www.lrz.de/services/compute/linux-cluster/batch_parallel/

A bunch of example scripts.. https://wiki.uio.no/usit/suf/vd/hpc/index.php/SLURM_example_scripts

Info about running Mathematica jobs.. http://www.tchpc.tcd.ie/node/1086