Merge pull request #24143 from jmchilton/it_container_epilog_docs

Document a Slurm epilog for cleaning up orphaned InteractiveTool containers

doc/source/admin/special_topics/interactivetools.rst

@@ -413,3 +413,76 @@ user desires, this may not be advisable and an admin may want to restrict the
 runtime of InteractiveTools *(and jobs in general)*. However, if the job is
 killed by the DRM, the user is not informed beforehand and data in the container
 could be discarded.
+
+
+Cleaning up orphaned containers
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+With ``docker run`` the container is started by the Docker daemon, not by the
+job, so it is outside the job's process tree and the DRM cannot kill it
+directly. To handle this, Galaxy's job script sets an ``EXIT`` trap that runs
+``docker kill`` on the container when the job script exits - including when
+the job is stopped by the user or terminated by the DRM with ``SIGTERM``.
+Normally no extra configuration is needed.
+
+If the job script is killed with ``SIGKILL`` before it can run the trap,
+however, the container keeps running after Galaxy considers the job finished
+(see `#13511 <https://github.com/galaxyproject/galaxy/issues/13511>`__). If
+``docker ps`` on your compute nodes shows containers named with a 32 character
+hex string (the names Galaxy gives job containers) whose jobs are no longer
+running, you can clean them up from the DRM's side. usegalaxy.org runs a
+`Slurm epilog script
+<https://github.com/galaxyproject/infrastructure-playbook/blob/8e96583f146bb79c0d5e3d8b8c3f4f975163c7af/files/slurm/epilog.sh>`__
+as a safety net for exactly this reason; the script below is adapted from it.
+
+For InteractiveTools Galaxy writes ``configs/container_config.json`` to the
+job directory, and its ``container_name`` field holds the name the container
+was started with. A Slurm epilog script can use this to kill the container
+when the job ends:
+
+.. code-block:: bash
+
+    #!/bin/bash
+    # Kill any InteractiveTool container left behind by a Galaxy job.
+    [ -n "${SLURM_JOB_ID:-}" ] || exit 0
+    (
+        workdir=$(scontrol show job "$SLURM_JOB_ID" | grep -o 'WorkDir=[^ ]*' | cut -d= -f2-)
+        # Galaxy submits from the job directory; Pulsar submits from its working subdirectory.
+        for container_config in "${workdir}/configs/container_config.json" "${workdir}/../configs/container_config.json"; do
+            if [ -f "$container_config" ]; then
+                container_name=$(jq -r '.container_name' "$container_config")
+                # Only kill names Galaxy generates.
+                [[ "$container_name" =~ ^[0-9a-f]{32}$ ]] && docker kill "$container_name"
+                break
+            fi
+        done
+    ) >/dev/null 2>&1 || true
+    exit 0
+
+Install it on each compute node (e.g. as ``/etc/slurm/epilog.sh``, executable
+by root) and enable it in ``slurm.conf``:
+
+.. code-block:: ini
+
+    Epilog=/etc/slurm/epilog.sh
+
+The epilog runs as root on the node after every job, so ``jq`` and ``docker``
+must be available there. It always exits 0 and discards its output, because a
+failing epilog drains the node. Jobs without a ``container_config.json`` are
+left alone. Keep in mind:
+
+- The epilog finds the job directory from the job's Slurm ``WorkDir``, so it
+  works with runners that submit from the job directory: Galaxy's ``slurm`` and
+  ``drmaa`` runners and Pulsar's DRMAA-based managers. The CLI runners submit
+  with plain ``sbatch`` and leave ``WorkDir`` wherever the submitting process
+  was running.
+- It only covers jobs Slurm runs. Jobs run directly by Galaxy's local runner or
+  by Pulsar's non-DRM managers never trigger it.
+- ``container_config.json`` is not written when the destination sets
+  ``container_monitor`` to ``false``.
+- If the destination changes how Docker is invoked (``docker_host``,
+  ``docker_sudo``, ``docker_sudo_cmd`` or ``docker_cmd``), adjust the
+  ``docker kill`` command to match. ``container_config.json`` records these
+  settings under ``connection_configuration``.
+
+Other DRMs offer similar post-job hooks.