From 3cca45b2bb570bb213d9a77764d606633aea7cfc Mon Sep 17 00:00:00 2001 From: John Chilton Date: Mon, 23 Jan 2017 16:14:59 -0500 Subject: [PATCH] Dependency resolver documentation revisions. - Add documentation for cached dependency management as solution to LOCKERRORS. - Add documentation for overcoming problems with hard links. (Different approaches) - Add documentation on upgrading Conda as needed. - Correct and fill out documentation of defaults. - Fix some bugs in the Galaxy packages documentation. - Expand Galaxy package documentation to include a simpler example. - Update Conda configuration option documentation to reflect new defaults. - Update Conda configuration option documentation to discuss copy_dependencies. - Cross link Conda and dependency resolver documentation as it makes sense. - Add a link to the Bioconda channel order documentation for future reference. - Make things a bit more concise in places - start with the implicit tl;dr. --- doc/source/admin/conda_faq.rst | 208 +++++++++++++++------- doc/source/admin/dependency_resolvers.rst | 102 ++++++++--- 2 files changed, 215 insertions(+), 95 deletions(-) diff --git a/doc/source/admin/conda_faq.rst b/doc/source/admin/conda_faq.rst index 4fe67d155b6..2167aa69da2 100644 --- a/doc/source/admin/conda_faq.rst +++ b/doc/source/admin/conda_faq.rst @@ -1,22 +1,18 @@ +.. _conda_faq: + =========================== Conda for Tool Dependencies =========================== -Galaxy tools (also called wrappers) traditionally use Tool Shed package -recipes to install their dependencies. At the tool's installation time -the recipe is downloaded and executed in order to provide the underlying -software executables. Introduction of these Galaxy-specific recipes was -a necessary step at the time, however nowadays there are other more -mature and stable options to install software in a similar manner. The +Galaxy tools (also called wrappers) have tradionally used Tool Shed package +recipes to install their dependencies. These were too tightly tied to Galaxy +and to the Tool Shed and so have been replaced with Conda as the package +management solution of choice for newer best practice tools. The Galaxy community has taken steps to improve the tool dependency system -in order to enable new features and expand its reach. This document aims -to describe these and answer the FAQ. - -Galaxy has adopted a new standard for tool dependencies: Conda packages! - -Not only do Conda packages make tool dependencies more reliable and -stable, they are also easier to test and faster to develop than the -traditional Tool Shed package recipes. +in order to enable new features and expand its reach. Not only do Conda packages +make tool dependencies more reliable and stable, they are also easier to test +and faster to develop than the traditional Tool Shed package recipes. This +document aims to describe these and answer frequently asked questions. Conda is a package manager like ``apt-get``, ``yum``, ``pip``, ``brew`` or ``guix``. We don't want to argue about the relative merits of various package @@ -25,8 +21,8 @@ community contributions (such as implementing a Guix package manager or enhancing the existing brew support to bring it on par with Conda). As a community, we have decided that Conda is the one that best fulfills -community's needs. The following are some of the crucial Conda features that led -to this decision: +tje community's current needs. The following are some of the crucial Conda +features that led to this decision: - Installation of packages does not require *root* privileges (installation at any location the Galaxy user has write access to) @@ -48,14 +44,19 @@ Below we answer some common questions (collected by Lance Parsons): 1. How do I enable Conda dependency resolution for Galaxy jobs? *************************************************************** -Galaxy's dependency job resolution is managed via -``dependency_resolvers_conf.xml`` configuration file. Most Galaxy administrators -should be using Galaxy's default dependency resolvers configuration file -( ``dependency_resolvers_conf.xml.sample`` ). With +The short answer is that as of 17.01, Galaxy should install Conda the first +time it starts up and be configured to use it by default. + +The long answer is that Galaxy's dependency job resolution is managed via +``dependency_resolvers_conf.xml`` configuration file. This configuration +file is discussed in detail in the :ref:`Dependency Resolvers ` +documentation. Most Galaxy administrators will be using Galaxy's default dependency +resolvers configuration file (``config/dependency_resolvers_conf.xml.sample``). With release 16.04, Galaxy has enabled Conda dependency resolution by default when -Conda was already installed on the system. Having Conda enabled in -``dependency_resolvers_conf.xml`` means that Galaxy can look for job -dependencies using the Conda system when it attempts to run tools. +Conda was already installed on the system. As of 17.01, Galaxy will also install +Conda as needed when starting up. Having Conda enabled in ``dependency_resolvers_conf.xml`` +means that Galaxy can look for job dependencies using the Conda system when it +attempts to run tools. Note that the order of resolvers in the file matters and the ```` entry should remain first. This means that tools that have specified Tool Shed packages @@ -67,24 +68,7 @@ See `galaxy.ini.sample`_ for the complete list. +--------------------------+--------------------------+---------------------------+ | Setting | Default setting | Meaning | +--------------------------+--------------------------+---------------------------+ -| ``conda_prefix`` | / | the location | -| | \_conda | on the | -| | | filesystem where Conda | -| | | packages and | -| | | environments are | -| | | installed | -| | | | -| | | IMPORTANT : Due to a | -| | | current limitation in | -| | | Conda, the total length | -| | | of the | -| | | | -| | | ``conda_prefix`` and the | -| | | ``job_working_directory`` | -| | | path should be less | -| | | than 50 characters! | -+--------------------------+--------------------------+---------------------------+ -| ``conda_auto_init`` | False | Set to True to instruct | +| ``conda_auto_init`` | True | Set to True to instruct | | | | Galaxy to install Conda | | | | (the package manager) | | | | automatically if it | @@ -98,6 +82,13 @@ See `galaxy.ini.sample`_ for the complete list. | | | dependencies before | | | | running a job. | +--------------------------+--------------------------+---------------------------+ +| ``conda_prefix`` | / | the location | +| | \_conda | on the | +| | | filesystem where Conda | +| | | packages and | +| | | environments are | +| | | installed | ++--------------------------+--------------------------+---------------------------+ *Table 1: Commonly used configuration options for Conda in Galaxy.* @@ -125,6 +116,11 @@ To summarize, there are four ways to manage Conda dependencies for use with Galaxy. For all of these options, Conda dependency management must be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` file. +#. Galaxy Admin Interface (>= 16.07) - Galaxy will install Conda tool + dependencies when tools are installed from the Tool Shed if the + option “When available, install externally managed dependencies (e.g. + Conda)? Beta” is checked. Admins may also view and manage Conda + dependencies via the Admin interface. #. Manual Install - Conda dependencies may be installed by administrators from the command line. Conda (and thus the Conda environments) should be installed in the location specified by the @@ -141,11 +137,6 @@ be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` fi Tools that require samtools version 0.1.19 will then be able to find and use the installed Conda package. -#. Galaxy Admin Interface (>= 16.07) - Galaxy will install Conda tool - dependencies when tools are installed from the Tool Shed if the - option “When available, install externally managed dependencies (e.g. - Conda)? Beta” is checked. Admins may also view and manage Conda - dependencies via the Admin interface. #. Automatically at tool run time - When a tool is run and a dependency is not found, Galaxy will attempt to install the dependency using Conda if ``conda_auto_install`` is activated in the configuration. @@ -157,15 +148,13 @@ be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` fi ************************************************************************************************** The minimum required version of Galaxy to use Conda is 16.01, however -version 16.07 or greater is recommended. The 16.07 release of Galaxy has +version 17.01 or greater is recommended. The 16.07 release of Galaxy has a graphical user interface to manage packages, but this is not required to have Conda dependencies managed and used by Galaxy. Conda packages should work on all compatible operating systems with -*glibc* version 2.5 or newer (this includes Centos 5). We will most -likely switch soon to *glibc* version 2.12 as a minimum requirement (this -includes CentOS 6). So all packages will run on all \*nix operating -systems newer than 2007. +*glibc* version 2.12 or newer (this includes Centos 6). So all packages +will run on all major \*nix operating systems newer than 2007. 4. If I have Conda enabled, what do I need to do to install tools using it? For example, how can I install the latest Trinity? And how will I know the dependencies are installed? @@ -214,6 +203,11 @@ The order in which resolvers are tried is listed in the The first system that satisfies a requirement will be used. See `resolver docs`_ for detailed documentation. +This however is not recommended, ideally tools will target and test +against Conda for all dependencies. Also resolving all requirements +with Conda gives Conda a chance to select compatibile versions of +dependencies. Read more about selecting compatible versions on +`Issue #3299`_ and `Pull Request #3391`_. 6. How do I know what system is being used by a given tool? *********************************************************** @@ -301,35 +295,110 @@ leave the old versions as they are – simply because of time. Old tools will use the traditional installation system; this system will stay and will be supported for installing old tools to guarantee sustainability -and reproducibility. New tools from the IUC, may be Conda only. +and reproducibility. New tools from the IUC and other best practices sources +are Conda only. -13. What can I do if Conda doesn't work for me? -*********************************************** +13. What can I do about this placehold error? +********************************************* -There is currently a limitation in the way Conda packages are being -built. This limitation will be addressed shortly by the Conda community, -however this requires all packages to be rebuilt. - -To work around this limitation, please make sure that the total length -of the ``conda_prefix`` and ``job_working_directory`` path is less than 50 -characters long. - -If this is your problem, you should see a warning similar to the -following in your galaxy log files: +If you see a warning similar to the following in your galaxy log files: .. code-block:: bash ERROR: placeholder '/home/ray/r_3_3_1-x64-3.5/envs/_build_placehold_placehold_placehold_placehold_pl' too short +This means you are very likely using an older version of Conda. This +bug has been fixed with the Conda release that is targeted by Galaxy +17.01 or newer. + +In the past, the work around for this limitation, was to make sure that the total length +of the ``conda_prefix`` and ``job_working_directory`` path was less than 50 +characters long. + + +14. What can I do about this LOCKERROR error? +*********************************************** + +This question addresses work arounds for Conda if something like the following +message appears in your logs: + +.. code-block:: bash + + Error: LOCKERROR: It looks like conda is already doing something. + The lock ['/galaxy/galaxy-app/tool-dependencies/_conda/pkgs/.conda_lock-119903'] was found. Wait for it to finish before continuing. + If you are sure that conda is not running, remove it and try again. + You can also use: $ conda clean --lock + +First, you may wish to enable cached dependencies. This can be done by setting +``use_cached_dependency_manager`` in ``galaxy.ini``. Many jobs will create a +per job Conda environment with just the dependencies needed for that job installed. +This will be placed on the filesystem containg the job working directory. This +is an expensive operation and Conda doesn't always link environments correctly +accross filesystems. Enabling this job caching will create a cache for each required +combination of requirements in the directory specified by ``tool_dependency_cache_dir`` +in ``galaxy.ini`` (defaulting to ``/_cache``). + +The cached dependency manager was added to the 16.10 release of Galaxy (see +`Pull Request #3106`_). In 17.01 Galaxy was updated to build the cached dependencies +as needed if the caching is in fact enabled (see `Pull Request #3348`_) and reduced +the number of jobs that would require such caching (see `Pull Request #3391`_). + + +15. What can I do about linking errors? +*************************************** + +If Galaxy jobs run on filesystems that cannot hardlink Conda packages managed +by Galaxy, linking errors may occur when building environment to execute jobs. +There are a few ways to potentially work around this discussed below. + +The most straight forward and efficient work around is probably just to enable the cached +dependency manager as described in the previous question. Notice the default location +of the cache is right next to the default Conda directory - so hardlinks should +lie on the same file system as the default Conda installation. + +If this still doesn't work, perhaps the underlying file system does not support hard +linking at all. In this case it is best to add ``always_softlink: True`` to Galaxy's +YAML ``condarc`` file, this should be created by Galaxy and placed in +``/bin`` folder. In that case you can delete the ``conda_prefix`` folder and restart Galaxy, which will again attempt to install Conda. If this does not solve your problem or you have any trouble following -the instructions, please ask on the Galaxy mailing list or the Galaxy -IRC channel. +the instructions, please ask on the Galaxy developing mailing list or the Galaxy +Gitter or IRC channel. + +17. How can I upgrade Conda? +**************************** + +Many potential issues with Conda have been resolved with fixes in Conda itself. If +you let Galaxy install Conda prior to the release of 17.01 you probably have version +3.19.3. This can be updated to 4.2.13 with the following command: + +.. code-block:: bash + + $ bedtools -The requirement turn into inputs to the dependency resolver. Each dependency resolver is thus given given one or -two inputs: the name of the dependency to resolve and, in most cases, the version string of the -dependency. +These declared requirements are passed as inputs to the dependency resolver. Default Dependency Resolvers ---------------------------- @@ -33,32 +34,50 @@ The default configuration of dependency resolvers is equivalent to the following .. code-block:: xml - + - + + -This default dependency resolver configuration contains three items. First, the *tool shed dependency resolver* is used, +This default dependency resolver configuration contains five items. First, the *tool shed dependency resolver* is used, then the *Galaxy packages dependency resolver* is used, first looking for packages by name and version string and then -finally looking for the package just by name. The default configuration thus prefers packages installed from the Galaxy -Tool Shed, before trying to find a "Galaxy package" satisfying the specific version the dependency requires before -finally falling back to looking for a Galaxy package with merely the correct name. If any of the dependency +finally looking for the package just by name, and finally likewise checks Conda for a versioned or unversioned match. +The default configuration thus prefers packages installed from the Galaxy Tool Shed using legacy ``tool_dependencies.xml`` +files, before trying to find a "Galaxy package" satisfying the specific version the dependency requires before +falling back to looking for a Galaxy package with merely the correct name, and then looking for Conda recipes with +matching name and version, and finally just for a Conda package with the correct name. If any of the dependency resolvers succeeds a dependency resolution object is returned and no more resolvers are called. This dependency resolution object provides shell commands to prepend to the shell script that runs the tool. +This order can be thought of as a descending order of deliberation. Tool Shed dependencies must be declared next to the +tool by the tool author and must be selected for installation at tool installation time - this requires specific actions +by both the tool author and the deployer who isntalled the tools. The dependency is therefore expected to highly craft +to the individual tool. If Galaxy packages have been setup, the deployer of a Galaxy tool has purposely crafted tool +dependency statements for a specific installation - this is slightly less deliberate than tool shed packages but +such requirements are less likely to be incidentally resolved than Conda packages. Conda recipes are niether tied to +tools or a specific installation and are maintained in Conda channels such as Bioconda. + +So while tool shed packages are first - they are also somewhat deprecated. Maintaining Conda recipes makes it easier +to describe software dependencies both inside of Galaxy and outside. + Tool Shed Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``tool_shed_packages`` dependency resolver works with packages installed from the Galaxy Tool Shed. When a package -is installed from the Tool Shed it creates a directory structure under the directory that is specified as the -``tool_dependency_dir`` in Galaxy's configuration. This directory structure contains references to the tool's name, -owner (in the Tool Shed) and version string (amongst other things) and ultimately contains a file named ``env.sh`` -that contains commands to make the dependency runnable. This is installed, along with the packaged tool, by the tool -package and doesn't require any configuration by the Galaxy administrator. +The ``tool_shed_packages`` dependency resolver works with explicit software packages installed from the Galaxy Tool +Shed as described by legacy ``tool_dependency.xml`` files. When such a package is installed from the Tool Shed it +creates a directory structure under the directory that is specified as the ``tool_dependency_dir`` in Galaxy's +configuration. This directory structure contains references to the tool's ID, owner (in the Tool Shed) and version +string (amongst other things) and ultimately contains a file named ``env.sh`` that contains commands to make the +dependency runnable. This is installed, along with the packaged tool, by the tool package and doesn't require any +configuration by the Galaxy administrator. + +Tools installed from the Tool Shed may also install Conda recipes and most new best practice tools do this +by default now. The Tool Shed dependency resolver is not able to resolve package requirements that do not have a version string, like the `bedtools` example above. @@ -80,7 +99,20 @@ needs ``bedtools`` version 2.20.1, the dependency resolver will look for a direc If the Galaxy Package dependency resolver finds a ``bin`` directory in this directory, it adds it to the ``PATH`` used by the scripts Galaxy uses to run tools. If, however, it finds an ``env.sh`` script, it sources this script before running the tool that requires this dependency. This can be used to set up the environment -needed for the tool to run. For example, this ``env.sh`` uses `Environment Modules `_ +needed for the tool to run. + +A simple example might be to assume that a collection of bioinformatics software is manually installed in various +directories under ``/opt/biosoftware``. In this case a ``/bedtools/2.20.1/env.sh`` could be +setup to add the corresponding bedtools installation to the Galaxy tool execution's ``PATH``. + +.. code-block:: bash + + #!/bin/sh + + export PATH=$PATH:/opt/biosoftware/bedtools/2.20.1/bin + + +As another example, this ``env.sh`` uses `Environment Modules `_ to setup the environment for ``bedtools`` .. code-block:: bash @@ -94,10 +126,10 @@ to setup the environment for ``bedtools`` module add bedtools/bedtools-2.20.1 The Galaxy Package dependency resolver operates quite similarly when used in versionless module. Instead of looking -for a directory named after a version, it looks for a directory ending in ``default``. For example -``bedtools/default``. It then looks for a `bin` subdirectory or ``envh.sh`` and incorporates these in the tool -script that finally gets run. This versionless (i.e. default) lookup is also used if the package requirement -does not specify a version string. +for a directory named after a version, it looks for a directory symbolic link named ``default`` that links to a +concrete version such as the ``2.20.1`` example above. For example if ``bedtools/default`` links to ``bedtools/2.20.1``. +It then looks for a `bin` subdirectory or ``envh.sh`` and incorporates these in the tool script that finally gets run. +This versionless (i.e. default) lookup is also used if the package requirement does not specify a version string. Environment Modules Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -149,7 +181,8 @@ used, they'll be used in the ``load`` command e.g. ``modulecmd sh load bwa/0.7.1 Homebrew Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This dependency resolver uses homebrew packages to resolve requirements. +This dependency resolver uses homebrew packages to resolve requirements. It is highly experimental +and undocumented. Brew Tool Shed Package Resolver @@ -163,32 +196,43 @@ and will almost certainy be removed from the code base. Conda Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The conda XML tag can be used to configure a conda dependency resolver. +The ``conda`` directive can be used to configure a conda dependency resolver. This resolver can be configured with the following options. +For a very detailed discussion of Conda dependency resolution, check out the +:ref:`Conda FAQ `. + prefix - The conda_prefix used to locate dependencies in (default: ``/_conda``). + The conda_prefix used to locate dependencies in (defaults to ``/_conda``). exec The conda executable to use, it will default to the one on the PATH (if available) and then to ``/bin/conda``. versionless - whether to resolve tools using a version string or not (default: *false*) + whether to resolve tools using a version string or not (defaults to *False*). debug - Pass debug flag to conda commands (default: false). + Pass debug flag to conda commands (default: *false*). ensure_channels conda channels to enable by default. See http://conda.pydata.org/docs/custom-channels.html for more - information about channels. (default: iuc,bioconda,r,defaults,conda-forge). + information about channels. This defaults to ``iuc,bioconda,r,defaults,conda-forge``. + This order should be consistent with `Bioconda perscribed order `__ + if it includes ``bioconda``. auto_install Set to True to instruct Galaxy to look for and install missing tool - dependencies before each job runs. (default: False) + dependencies before each job runs (defaults to *False*). auto_init Set to True to instruct Galaxy to install conda from the web automatically if it cannot find a local copy and conda_exec is not - configured. + configured. This defaults to *True* as of Galaxy 17.01. + +copy_dependencies + Set to ``True`` to instruct Galaxy to copy dependencies over instead of symbolically + linking them when creating per job environments. This should be considered somewhat + deprecated because Conda will do this as needed for newer versions of Conda - such + as the version targeted with Galaxy 17.01+.