clusterlib.scheduler.submit

clusterlib.scheduler.submit(job_command, job_name='job', time='24:00:00', memory=4000, email=None, email_options=None, log_directory=None, backend='auto', shell_script='#!/bin/bash')

Write the submission query (without script).

Parameters:

job_command : str,

Command associated to the job, e.g. ‘python main.py’.

job_name : str, optional (default=”job”)

Name of the job.

time : str, optional (default=”24:00:00”)

Maximum time format “HH:MM:SS”.

memory : str, optional (default=4000)

Maximum virtual memory in mega-bytes

email : str, optional (default=None)

Email where job information is sent. If None, no email is asked to be sent.

email_options : str, optional (default=None)

Specify email options:
  • SGE : Format char from beas (begin,end,abort,stop) for SGE.
  • SLURM : either BEGIN, END, FAIL, REQUEUE or ALL.

See the documenation for more information

log_directory : str, optional (default=None)

Specify the log directory. If None, no log directory is specified. Job logs will be at log_directory with the name job_name.job_id.txt where the job_id is given by the scheduler.

backend : {‘auto’, ‘slurm’, ‘sge’}, optional (default=”auto”)

Backend where the job will be submitted. If ‘auto’, try detect the backend to use based on the commands available in the PATH variable looking first for ‘slurm’ and then for ‘sge’ if slurm is not found. The default backend selected when backend=’auto’ can also be fixed by setting the “CLUSTERLIB_BACKEND” environment variable.

shell_script : str, optional (default=”#!/bin/bash”)

Specify shell that is used by the script.
Returns:

submission_query : str,

Return the submission query in the appropriate format. The obtained query could be directly launch using os.subprocess. Further options could be appended at the end of the string.

Examples

First, let’s generate a command for SLURM to launch the program main.py.

>>> from clusterlib.scheduler import submit
>>> script = submit("python main.py --args 1", backend='slurm')
>>> print(script)
echo '#!/bin/bash
python main.py --args 1' | sbatch --job-name=job --time=24:00:00 --mem=4000

The job can be latter launched using for instance os.system(script).