diff --git a/doc/source/admin/cluster.md b/doc/source/admin/cluster.md index 5f40c91440b..3ff9eaff053 100644 --- a/doc/source/admin/cluster.md +++ b/doc/source/admin/cluster.md @@ -195,7 +195,7 @@ Most options available to `qsub(1b)` and `pbs_submit(3b)` are supported. Except ``` -The value of *ppn=* is used by PBS to define the environment variable $PBS_NCPUS which in turn is used by galaxy for [GALAXY_SLOTS](https://galaxyproject.org/admin/config/galaxy_slots/). +The value of *ppn=* is used by PBS to define the environment variable `$PBS_NCPUS` which in turn is used by galaxy for [GALAXY_SLOTS](https://galaxyproject.org/admin/config/galaxy_slots/). ### Condor @@ -355,6 +355,30 @@ It is also a good idea to make sure that only trusted users, e.g. root, have wri Some maintenance and support of this code will be provided via the usual [Support](https://galaxyproject.org/support/) channels, but improvements and fixes would be greatly welcomed, as this is a complex feature which is not used by the Galaxy Development Team. +## Special environment variables for job resources + +Galaxy *tries* to define special environment variables for each job that contain +the information on the number of available slots and the amount of available +memory: + +* `GALAXY_SLOTS`: number of available slots +* `GALAXY_MEMORY_MB`: total amount of available memory in MB +* `GALAXY_MEMORY_MB_PER_SLOT`: amount of memory that is available for each slot in MB + +More precisely Galaxy inserts bash code in the job submit script that +tries to determine these values. This bash code is defined here: + +* lib/galaxy/jobs/runners/util/job_script/CLUSTER_SLOTS_STATEMENT.sh +* lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh + +If this code is unable to determine the variables, then they will not be set. +Therefore in the tool XML files the variables should be used with a default, +e.g. `\${GALAXY_SLOTS:-1}` (see also https://planemo.readthedocs.io/en/latest/writing_advanced.html#cluster-usage). + +In particular `GALAXY_MEMORY_MB` and `GALAXY_MEMORY_MB_PER_SLOT` are currently +defined only for a few cluster types. Contributions are very welcome, e.g. let +the Galaxy developers know how to modify that file to support your cluster. + ## Contributors * **Oleksandr Moskalenko**, debugged a number of problems related to running jobs as the real user and using DRMAA with TORQUE. diff --git a/lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh b/lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh index fef76e5fff9..5d58124365c 100644 --- a/lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh +++ b/lib/galaxy/jobs/runners/util/job_script/MEMORY_STATEMENT.sh @@ -1,4 +1,11 @@ if [ -n "$SLURM_JOB_ID" ]; then GALAXY_MEMORY_MB=`scontrol -do show job "$SLURM_JOB_ID" | sed 's/.*\( \|^\)Mem=\([0-9][0-9]*\)\( \|$\).*/\2/p;d'` 2>memory_statement.log fi + +if [ -z "$GALAXY_MEMORY_MB_PER_SLOT" -a -n "$GALAXY_MEMORY_MB" ]; then + GALAXY_MEMORY_MB_PER_SLOT=$(($GALAXY_MEMORY_MB / $GALAXY_SLOTS)) +elif [ -z "$GALAXY_MEMORY_MB" -a -n "$GALAXY_MEMORY_MB_PER_SLOT" ]; then + GALAXY_MEMORY_MB=$(($GALAXY_MEMORY_MB_PER_SLOT * $GALAXY_SLOTS)) +fi [ "${GALAXY_MEMORY_MB--1}" -gt 0 ] 2>>memory_statement.log && export GALAXY_MEMORY_MB || unset GALAXY_MEMORY_MB +[ "${GALAXY_MEMORY_MB_PER_SLOT--1}" -gt 0 ] 2>>memory_statement.log && export GALAXY_MEMORY_MB_PER_SLOT || unset GALAXY_MEMORY_MB_PER_SLOT diff --git a/lib/galaxy/tools/xsd/galaxy.xsd b/lib/galaxy/tools/xsd/galaxy.xsd index 2497b7ad974..0f7d4af05fb 100644 --- a/lib/galaxy/tools/xsd/galaxy.xsd +++ b/lib/galaxy/tools/xsd/galaxy.xsd @@ -2656,7 +2656,8 @@ be escaped with a backslash (``\``) when appearing in ``command`` or ``configfil Name | Description ---- | ----------- ``\${GALAXY_SLOTS:-4}`` | Number of cores/threads allocated by the job runner or resource manager to the tool for the given job (here 4 is the default number of threads to use if running via custom runner that does not configure GALAXY_SLOTS or in an older Galaxy runtime). -``\$GALAXY_MEMORY_MB`` | Amount of memory in megabytes (1024^2 bytes) allocated by the administrator (via the resource manager) to the tool for the given job. If unset, tools should not attempt to limit memory usage. +``\$GALAXY_MEMORY_MB`` | Total amount of memory in megabytes (1024^2 bytes) allocated by the administrator (via the resource manager) to the tool for the given job. If unset, tools should not attempt to limit memory usage. +``\$GALAXY_MEMORY_MB_PER_SLOT`` | Amount of memory per slot in megabytes (1024^2 bytes) allocated by the administrator (via the resource manager) to the tool for the given job. If unset, tools should not attempt to limit memory usage. See the [Planemo docs](https://planemo.readthedocs.io/en/latest/writing_advanced.html#cluster-usage) on the topic of ``GALAXY_SLOTS`` for more information and examples.