diff --git a/doc/source/dev/debugging_galaxy_slurm.md b/doc/source/dev/debugging_galaxy_slurm.md new file mode 100644 index 00000000000..fd0965a7da5 --- /dev/null +++ b/doc/source/dev/debugging_galaxy_slurm.md @@ -0,0 +1,76 @@ + +# Debugging Galaxy: Slurm Compute Cluster + +This document explains how to debug a Galaxy instance that uses a Slurm compute cluster (not the +default LocalRunner). If you have a MacOS machine, you need to create a Linux VM using Oracle's +VirtualBoxVM and run the instructions in the VM. For reference, I created a 64-bit Linux VM with +4096 MB of RAM and 32 GB of disk space (all other parameters were left as default) and managed to +successfully run the instructions. If using a Linux VM, you may need to install Git, Python, VSCode, +and Docker (if using Irods storage). Finally, These instructions are intended for developers who +want to debug their Galaxy instance and are not meant for deployment purposes (We have nice Ansible +playbooks for that). + +## Debugging Galaxy in VS Code + +1. Install libslurm-dev + * `sudo apt install libslurm-dev` + * On my VM, the above command puts slurm/slurm.h in /usr/include and libslurm.a in /usr/lib/x86_64-linux-gnu + * You need these header and library files to compile slurm-drmaa in the next step + +2. Install slurm-drmaa + * Download slurm-drmaa tar file from https://github.com/natefoo/slurm-drmaa/releases/download/1.1.2/slurm-drmaa-1.1.2.tar.gz + * Extract the downloaded tar file into home directory: `tar -xzvf slurm-drmaa-1.1.2.tar.gz -C ~` + * To compile and install slurm-drmaa + * `cd ~/slurm-drmaa-1.1.2` + * `./configure --with-slurm-inc=/usr/include --with-slurm-lib=/usr/lib/x86_64-linux-gpu` + * `sudo make install` + * On my VM, the above command puts libdrmaa.so in /usr/local/lib. You need the path to this file in the next step + +3. Replace the contents of ~/galaxy/config/job_conf.xml with the following code snippet (create the file if it does not already exist) + ``` + + + + + /usr/local/lib/libdrmaa.so + + + + + + + ``` + +4. Uncomment the following line in ~/galaxy/config/galaxy.yml (If the file does not exist, create it by copying ~/galaxy/config/galaxy.yml.sample) + * `job_config_file: job_conf.xml` + +5. Install Slurm (Instructions borrowed from https://gist.github.com/ckandoth/2acef6310041244a690e4c08d2610423) + * Install the necessary packages: `sudo apt install -y slurm-wlm slurm-wlm-doc` + * Load /usr/share/doc/slurm-wlm/html/configurator.html in a browser + * Set your VM's hostname in `SlurmctldHost` and `NodeName` + * Set `StateSaveLocation` to `/var/spool/slurmctld` + * Set `SlurmdSpoolDir` to `/var/spool/slurmd` + * Set `ProctrackType` to `LinuxProc` + * Set `SelectType` to `Cons_res` + * Set `SelectTypeParameters` to `CR_Core_Memory` + * Set `JobAcctGatherType` to `Linux` + * Hit Submit and save the resulting text into `/etc/slurm-llnl/slurm.conf` + * I.e., the configuration file referred to in /lib/systemd/system/slurmctld.service and /lib/systemd/system/slurmd.service + * Create /var/spool/slurmd and /var/spool/slurmctld directories and /var/log/slurm_jobacct.log file and set ownership appropriately + * `sudo mkdir -p /var/spool/slurmd` + * `sudo mkdir -p /var/spool/slurmctld` + * `sudo touch /var/log/slurm_jobacct.log` + * `sudo chown slurm:slurm /var/spool/slurmd /var/spool/slurmctld /var/log/slurm_jobacct.log` + * Install mailutils so that Slurm won't complain about /bin/mail missing + * `sudo apt install -y mailutils` + * Make sure munge is installed/running, and a munge.key was created with user-only read-only permissions, owned by munge:munge + * `sudo apt install munge` + * `sudo service munge start` + * `sudo ls -l /etc/munge/munge.key` + * Start slurmd and slurmctld services + * `sudo service slurmd start` + * `sudo service slurmctld start` + +6. Follow the instructions [here](./debuging_galaxy.md) to setup VSCode for debugging. + +Enjoy debugging session your Galaxy instance backed by a Slurm cluster! diff --git a/doc/source/dev/index.rst b/doc/source/dev/index.rst index 74f48f5be70..4f363522ce2 100644 --- a/doc/source/dev/index.rst +++ b/doc/source/dev/index.rst @@ -22,5 +22,6 @@ A multi-hour long video playlist covering these slides can be found at writing_tests debugging_tests debugging_galaxy + debugging_galaxy_slurm starting_galaxy file_upload