From b6fcf2d51eb6514504098484bc48660cac4b3b7f Mon Sep 17 00:00:00 2001 From: John Chilton Date: Tue, 6 Oct 2026 20:50:26 -0400 Subject: [PATCH 1/3] Document Slurm epilog for orphaned InteractiveTool containers Galaxy's EXIT trap should kill IT containers on stop, but SIGKILL'd job scripts leave them running (#13511). Document the safety-net epilog usegalaxy.org runs, reading container_name from configs/container_config.json. Co-Authored-By: Claude Opus 5.5 --- .../admin/special_topics/interactivetools.rst | 54 +++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/doc/source/admin/special_topics/interactivetools.rst b/doc/source/admin/special_topics/interactivetools.rst index 029794ffaa8..42602af0348 100644 --- a/doc/source/admin/special_topics/interactivetools.rst +++ b/doc/source/admin/special_topics/interactivetools.rst @@ -413,3 +413,57 @@ user desires, this may not be advisable and an admin may want to restrict the runtime of InteractiveTools *(and jobs in general)*. However, if the job is killed by the DRM, the user is not informed beforehand and data in the container could be discarded. + + +Cleaning up orphaned containers +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +With ``docker run`` the container is started by the Docker daemon, not by the +job, so it is outside the job's process tree and the DRM cannot kill it +directly. To handle this, Galaxy's job script sets an ``EXIT`` trap that runs +``docker kill`` on the container when the job script exits - including when +the job is stopped by the user or terminated by the DRM with ``SIGTERM``. +This should be sufficient and no extra configuration ought to be needed. + +If the job script is killed with ``SIGKILL`` before it can run the trap, +however, the container keeps running after Galaxy considers the job finished +(see `#13511 `__). If +``docker ps`` on your compute nodes shows InteractiveTool containers (named +with a 32 character hex string) whose jobs are no longer running, you can +clean them up from the DRM's side. usegalaxy.org runs a Slurm epilog script +as a safety net for exactly this reason. + +For InteractiveTools Galaxy writes ``configs/container_config.json`` to the +job directory, and its ``container_name`` field holds the name the container +was started with. A Slurm epilog script can use this to kill the container +when the job ends: + +.. code-block:: bash + + #!/bin/bash + # Kill any InteractiveTool container left behind by a Galaxy job. + [ -n "${SLURM_JOB_ID:-}" ] || exit 0 + ( + workdir=$(scontrol show job "$SLURM_JOB_ID" | grep -o 'WorkDir=[^ ]*' | cut -d= -f2-) + # Galaxy submits from the job directory; Pulsar may submit from its working subdirectory. + for container_config in "${workdir}/configs/container_config.json" "${workdir}/../configs/container_config.json"; do + if [ -f "$container_config" ]; then + docker kill "$(jq -r '.container_name' "$container_config")" + break + fi + done + ) >/dev/null 2>&1 || true + exit 0 + +Install it on each compute node (e.g. as ``/etc/slurm/epilog.sh``, executable +by root) and enable it in ``slurm.conf``: + +.. code-block:: ini + + Epilog=/etc/slurm/epilog.sh + PrologEpilogTimeout=90 + +The epilog runs as root on the node after every job, so ``jq`` and ``docker`` +must be available there. Jobs without a ``container_config.json`` are left +alone. If your destination sets ``docker_host`` or ``docker_sudo``, adjust the +``docker kill`` command to match. Other DRMs offer similar post-job hooks. From f0811cce6a4a531015391d7edd7d24a9a8ba94af Mon Sep 17 00:00:00 2001 From: John Chilton Date: Tue, 6 Oct 2026 21:22:45 -0400 Subject: [PATCH 2/3] Scope InteractiveTool epilog docs to runners it works with Note which runners set a usable Slurm WorkDir, that only Slurm-run jobs trigger the epilog, container_monitor and docker_* caveats; only kill Galaxy-generated container names; drop PrologEpilogTimeout. Co-Authored-By: Claude Opus 5.5 --- .../admin/special_topics/interactivetools.rst | 39 +++++++++++++------ 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/doc/source/admin/special_topics/interactivetools.rst b/doc/source/admin/special_topics/interactivetools.rst index 42602af0348..401925ef870 100644 --- a/doc/source/admin/special_topics/interactivetools.rst +++ b/doc/source/admin/special_topics/interactivetools.rst @@ -423,15 +423,15 @@ job, so it is outside the job's process tree and the DRM cannot kill it directly. To handle this, Galaxy's job script sets an ``EXIT`` trap that runs ``docker kill`` on the container when the job script exits - including when the job is stopped by the user or terminated by the DRM with ``SIGTERM``. -This should be sufficient and no extra configuration ought to be needed. +Normally no extra configuration is needed. If the job script is killed with ``SIGKILL`` before it can run the trap, however, the container keeps running after Galaxy considers the job finished (see `#13511 `__). If -``docker ps`` on your compute nodes shows InteractiveTool containers (named -with a 32 character hex string) whose jobs are no longer running, you can -clean them up from the DRM's side. usegalaxy.org runs a Slurm epilog script -as a safety net for exactly this reason. +``docker ps`` on your compute nodes shows containers named with a 32 character +hex string (the names Galaxy gives job containers) whose jobs are no longer +running, you can clean them up from the DRM's side. usegalaxy.org runs a Slurm +epilog script as a safety net for exactly this reason. For InteractiveTools Galaxy writes ``configs/container_config.json`` to the job directory, and its ``container_name`` field holds the name the container @@ -445,10 +445,12 @@ when the job ends: [ -n "${SLURM_JOB_ID:-}" ] || exit 0 ( workdir=$(scontrol show job "$SLURM_JOB_ID" | grep -o 'WorkDir=[^ ]*' | cut -d= -f2-) - # Galaxy submits from the job directory; Pulsar may submit from its working subdirectory. + # Galaxy submits from the job directory; Pulsar submits from its working subdirectory. for container_config in "${workdir}/configs/container_config.json" "${workdir}/../configs/container_config.json"; do if [ -f "$container_config" ]; then - docker kill "$(jq -r '.container_name' "$container_config")" + container_name=$(jq -r '.container_name' "$container_config") + # Only kill names Galaxy generates. + [[ "$container_name" =~ ^[0-9a-f]{32}$ ]] && docker kill "$container_name" break fi done @@ -461,9 +463,24 @@ by root) and enable it in ``slurm.conf``: .. code-block:: ini Epilog=/etc/slurm/epilog.sh - PrologEpilogTimeout=90 The epilog runs as root on the node after every job, so ``jq`` and ``docker`` -must be available there. Jobs without a ``container_config.json`` are left -alone. If your destination sets ``docker_host`` or ``docker_sudo``, adjust the -``docker kill`` command to match. Other DRMs offer similar post-job hooks. +must be available there. It always exits 0 and discards its output, because a +failing epilog drains the node. Jobs without a ``container_config.json`` are +left alone. Keep in mind: + +- The epilog finds the job directory from the job's Slurm ``WorkDir``, so it + works with runners that submit from the job directory: Galaxy's ``slurm`` and + ``drmaa`` runners and Pulsar's DRMAA-based managers. The CLI runners submit + with plain ``sbatch`` and leave ``WorkDir`` wherever the submitting process + was running. +- It only covers jobs Slurm runs. Jobs run directly by Galaxy's local runner or + by Pulsar's non-DRM managers never trigger it. +- ``container_config.json`` is not written when the destination sets + ``container_monitor`` to ``false``. +- If the destination changes how Docker is invoked (``docker_host``, + ``docker_sudo``, ``docker_sudo_cmd`` or ``docker_cmd``), adjust the + ``docker kill`` command to match. ``container_config.json`` records these + settings under ``connection_configuration``. + +Other DRMs offer similar post-job hooks. From 28d78493d29a3b25a2571a17d00252abcfafcf34 Mon Sep 17 00:00:00 2001 From: John Chilton Date: Wed, 14 Oct 2026 23:14:24 -0400 Subject: [PATCH 3/3] Link usegalaxy.org's Slurm epilog from InteractiveTool docs Co-Authored-By: Claude Opus 5.5 --- doc/source/admin/special_topics/interactivetools.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/doc/source/admin/special_topics/interactivetools.rst b/doc/source/admin/special_topics/interactivetools.rst index 401925ef870..73d4f3d0e3d 100644 --- a/doc/source/admin/special_topics/interactivetools.rst +++ b/doc/source/admin/special_topics/interactivetools.rst @@ -430,8 +430,10 @@ however, the container keeps running after Galaxy considers the job finished (see `#13511 `__). If ``docker ps`` on your compute nodes shows containers named with a 32 character hex string (the names Galaxy gives job containers) whose jobs are no longer -running, you can clean them up from the DRM's side. usegalaxy.org runs a Slurm -epilog script as a safety net for exactly this reason. +running, you can clean them up from the DRM's side. usegalaxy.org runs a +`Slurm epilog script +`__ +as a safety net for exactly this reason; the script below is adapted from it. For InteractiveTools Galaxy writes ``configs/container_config.json`` to the job directory, and its ``container_name`` field holds the name the container