From 96ef23a79ba9b8ec47dc4d8705a602d8950e28cd Mon Sep 17 00:00:00 2001 From: Kaivan Kamali Date: Thu, 2 Sep 2021 16:10:41 -0400 Subject: [PATCH 1/4] Added instructions on how to debug Galaxy backed by a Slurm cluster --- doc/source/dev/debugging_galaxy_slurm.md | 78 ++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 doc/source/dev/debugging_galaxy_slurm.md diff --git a/doc/source/dev/debugging_galaxy_slurm.md b/doc/source/dev/debugging_galaxy_slurm.md new file mode 100644 index 00000000000..e3963cc8e76 --- /dev/null +++ b/doc/source/dev/debugging_galaxy_slurm.md @@ -0,0 +1,78 @@ + +# Debugging Galaxy: Slurm Compute Cluster + +This document explains how to debug a Galaxy instance that uses a Slurm compute cluster (not the +default LocalRunner). If you have a MacOS machine, you need to create a Linux VM using Oracle's +VirtualBoxVM and run the instructions in the VM. For reference, I created a 64-bit Linux VM with +4096 MB of RAM and 32 GB of disk space (all other parameters were left as default) and managed to +successfully run the instructions. If using a Linux VM, you may need to install Git, Python, VSCode, +and Docker (if using Irods storage). Finally, These instructions are intended for developers who +want to debug their Galaxy instance and are not meant for deployment purposes (We have nice Ansible +playbooks for that). + +## Debugging Galaxy in VS Code + +1. Install libslurm-dev + * `sudo apt install libslurm-dev` + * On my VM, the above command puts `slurm/slurm.h` in `/usr/include` and `libslurm.a` in `/usr/lib/x86_64-linux-gnu` + * You need these header and library files to compile slurm-drmaa in the next step + +2. Install slurm-drmaa + * Download slurm-drmaa tar file from `https://github.com/natefoo/slurm-drmaa/releases/download/1.1.2/slurm-drmaa-1.1.2.tar.gz` + * Extract the downloaded tar file into home directory: `tar -xzvf slurm-drmaa-1.1.2.tar.gz -C ~` + * To compile and install slurm-drmaa + * `cd ~/slurm-drmaa-1.1.2` + * `./configure --with-slurm-inc=/usr/include --with-slurm-lib=/usr/lib/x86_64-linux-gpu` + * `sudo make install` + * On my VM, the above command puts `libdrmaa.so` in `/usr/local/lib`. You need the path to this file in the next step + +3. Replace the contents of `~/galaxy/config/job_conf.xml` with the following code snippet (create the file if it does not already exist) + ``` + + + + + /usr/local/lib/libdrmaa.so + + + + + + + + memory + processors,memory,time,project + + + ``` + +4. Install Slurm (Instructions borrowed from `https://gist.github.com/ckandoth/2acef6310041244a690e4c08d2610423`) + * Install the necessary packages: `sudo apt install -y slurm-wlm slurm-wlm-doc` + * Load `/usr/share/doc/slurm-wlm/html/configurator.html` in a browser + * Set your VM's hostname in `SlurmctldHost` and `NodeName` + * Set `StateSaveLocation` to `/var/spool/slurmctld` + * Set `SlurmdSpoolDir` to `/var/spool/slurmd` + * Set `ProctrackType` to `LinuxProc` + * Set `SelectType` to `Cons_res` + * Set `SelectTypeParameters` to `CR_Core_Memory` + * Set `JobAcctGatherType` to `Linux` + * Hit `Submit` and save the resulting text into `/etc/slurm-llnl/slurm.conf` + * I.e., the configuration file referred to in `/lib/systemd/system/slurmctld.service` and `/lib/systemd/system/slurmd.service` + * Create `/var/spool/slurmd` and `/var/spool/slurmctld` and `/var/log/slurm_jobacct.log` file and set ownership appropriately + * `sudo mkdir -p /var/spool/slurmd` + * `sudo mkdir -p /var/spool/slurmctld` + * `sudo touch /var/log/slurm_jobacct.log` + * `sudo chown slurm:slurm /var/spool/slurmd /var/spool/slurmctld /var/log/slurm_jobacct.log` + * Install mailutils so that Slurm won't complain about `/bin/mail` missing + * `sudo apt install -y mailutils` + * Make sure munge is installed/running, and a munge.key was created with user-only read-only permissions, owned by munge:munge + * `sudo apt install munge` + * `sudo service munge start` + * `sudo ls -l /etc/munge/munge.key` + * Start slurmd and slurmctld services + * `sudo service slurmd start` + * `sudo service slurmctld start` + +5. Follow the instructions [here](./debuging_galaxy.md) to setup VSCode for debugging. + +Enjoy debugging session your Galaxy instance backed by a Slurm cluster! From b47374aef7adc848ca4ebea8f8265c60147a0eea Mon Sep 17 00:00:00 2001 From: Kaivan Kamali Date: Thu, 2 Sep 2021 17:10:15 -0400 Subject: [PATCH 2/4] Added instructions for debuuging Galaxy using a Slurm cluster --- doc/source/dev/debugging_galaxy_slurm.md | 29 +++++++++++++----------- doc/source/dev/index.rst | 1 + 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/doc/source/dev/debugging_galaxy_slurm.md b/doc/source/dev/debugging_galaxy_slurm.md index e3963cc8e76..ffd8ec6b742 100644 --- a/doc/source/dev/debugging_galaxy_slurm.md +++ b/doc/source/dev/debugging_galaxy_slurm.md @@ -14,19 +14,19 @@ playbooks for that). 1. Install libslurm-dev * `sudo apt install libslurm-dev` - * On my VM, the above command puts `slurm/slurm.h` in `/usr/include` and `libslurm.a` in `/usr/lib/x86_64-linux-gnu` + * On my VM, the above command puts slurm/slurm.h in /usr/include and libslurm.a in /usr/lib/x86_64-linux-gnu * You need these header and library files to compile slurm-drmaa in the next step 2. Install slurm-drmaa - * Download slurm-drmaa tar file from `https://github.com/natefoo/slurm-drmaa/releases/download/1.1.2/slurm-drmaa-1.1.2.tar.gz` + * Download slurm-drmaa tar file from https://github.com/natefoo/slurm-drmaa/releases/download/1.1.2/slurm-drmaa-1.1.2.tar.gz * Extract the downloaded tar file into home directory: `tar -xzvf slurm-drmaa-1.1.2.tar.gz -C ~` * To compile and install slurm-drmaa * `cd ~/slurm-drmaa-1.1.2` * `./configure --with-slurm-inc=/usr/include --with-slurm-lib=/usr/lib/x86_64-linux-gpu` * `sudo make install` - * On my VM, the above command puts `libdrmaa.so` in `/usr/local/lib`. You need the path to this file in the next step + * On my VM, the above command puts libdrmaa.so in /usr/local/lib. You need the path to this file in the next step -3. Replace the contents of `~/galaxy/config/job_conf.xml` with the following code snippet (create the file if it does not already exist) +3. Replace the contents of ~/galaxy/config/job_conf.xml with the following code snippet (create the file if it does not already exist) ``` @@ -46,24 +46,27 @@ playbooks for that). ``` -4. Install Slurm (Instructions borrowed from `https://gist.github.com/ckandoth/2acef6310041244a690e4c08d2610423`) +4. Uncomment the following line in ~/galaxy/config/galaxy.yml + * `job_config_file: job_conf.xml` + +5. Install Slurm (Instructions borrowed from https://gist.github.com/ckandoth/2acef6310041244a690e4c08d2610423) * Install the necessary packages: `sudo apt install -y slurm-wlm slurm-wlm-doc` - * Load `/usr/share/doc/slurm-wlm/html/configurator.html` in a browser + * Load /usr/share/doc/slurm-wlm/html/configurator.html in a browser * Set your VM's hostname in `SlurmctldHost` and `NodeName` * Set `StateSaveLocation` to `/var/spool/slurmctld` * Set `SlurmdSpoolDir` to `/var/spool/slurmd` * Set `ProctrackType` to `LinuxProc` - * Set `SelectType` to `Cons_res` - * Set `SelectTypeParameters` to `CR_Core_Memory` + * Set `SelectType` to `Cons_res` + * Set `SelectTypeParameters` to `CR_Core_Memory` * Set `JobAcctGatherType` to `Linux` - * Hit `Submit` and save the resulting text into `/etc/slurm-llnl/slurm.conf` - * I.e., the configuration file referred to in `/lib/systemd/system/slurmctld.service` and `/lib/systemd/system/slurmd.service` - * Create `/var/spool/slurmd` and `/var/spool/slurmctld` and `/var/log/slurm_jobacct.log` file and set ownership appropriately + * Hit Submit and save the resulting text into `/etc/slurm-llnl/slurm.conf` + * I.e., the configuration file referred to in /lib/systemd/system/slurmctld.service and /lib/systemd/system/slurmd.service + * Create /var/spool/slurmd and /var/spool/slurmctld directories and /var/log/slurm_jobacct.log file and set ownership appropriately * `sudo mkdir -p /var/spool/slurmd` * `sudo mkdir -p /var/spool/slurmctld` * `sudo touch /var/log/slurm_jobacct.log` * `sudo chown slurm:slurm /var/spool/slurmd /var/spool/slurmctld /var/log/slurm_jobacct.log` - * Install mailutils so that Slurm won't complain about `/bin/mail` missing + * Install mailutils so that Slurm won't complain about /bin/mail missing * `sudo apt install -y mailutils` * Make sure munge is installed/running, and a munge.key was created with user-only read-only permissions, owned by munge:munge * `sudo apt install munge` @@ -73,6 +76,6 @@ playbooks for that). * `sudo service slurmd start` * `sudo service slurmctld start` -5. Follow the instructions [here](./debuging_galaxy.md) to setup VSCode for debugging. +6. Follow the instructions [here](./debuging_galaxy.md) to setup VSCode for debugging. Enjoy debugging session your Galaxy instance backed by a Slurm cluster! diff --git a/doc/source/dev/index.rst b/doc/source/dev/index.rst index 74f48f5be70..4f363522ce2 100644 --- a/doc/source/dev/index.rst +++ b/doc/source/dev/index.rst @@ -22,5 +22,6 @@ A multi-hour long video playlist covering these slides can be found at writing_tests debugging_tests debugging_galaxy + debugging_galaxy_slurm starting_galaxy file_upload From 99880294c6b7bdead170a04d5ad6aad7e37d838d Mon Sep 17 00:00:00 2001 From: Kaivan Kamali Date: Fri, 3 Sep 2021 09:27:29 -0400 Subject: [PATCH 3/4] Minor revision --- doc/source/dev/debugging_galaxy_slurm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/source/dev/debugging_galaxy_slurm.md b/doc/source/dev/debugging_galaxy_slurm.md index ffd8ec6b742..961bc34dc96 100644 --- a/doc/source/dev/debugging_galaxy_slurm.md +++ b/doc/source/dev/debugging_galaxy_slurm.md @@ -46,7 +46,7 @@ playbooks for that). ``` -4. Uncomment the following line in ~/galaxy/config/galaxy.yml +4. Uncomment the following line in ~/galaxy/config/galaxy.yml (If the file does not exist, create it by copying ~/galaxy/config/galaxy.yml.sample) * `job_config_file: job_conf.xml` 5. Install Slurm (Instructions borrowed from https://gist.github.com/ckandoth/2acef6310041244a690e4c08d2610423) From 952d947087d11373e6dfc8bbbb3910a047fae445 Mon Sep 17 00:00:00 2001 From: Kaivan Kamali Date: Fri, 3 Sep 2021 12:49:36 -0400 Subject: [PATCH 4/4] Removed resources section per John's comment --- doc/source/dev/debugging_galaxy_slurm.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/doc/source/dev/debugging_galaxy_slurm.md b/doc/source/dev/debugging_galaxy_slurm.md index 961bc34dc96..fd0965a7da5 100644 --- a/doc/source/dev/debugging_galaxy_slurm.md +++ b/doc/source/dev/debugging_galaxy_slurm.md @@ -38,11 +38,6 @@ playbooks for that). - - - memory - processors,memory,time,project - ```