@@ -5,43 +5,54 @@ Dependency Resolvers in Galaxy
==============================
There are two parts to building a link between Galaxy and command line bioinformatics tools: (1) the tool XML that
specifies a mapping between the Galaxy web user interface and the tool command line, and (2) system tool dependencies that
specify how to source the actual packages that implement the tool’
includes commands (such as changes to the `` PATH `` environment variable) that are generated by *dependency
resolvers*. These same dependency resolvers are used by the Galaxy administrative UI to display whether an installed
tool's dependencies have been installed on the Galaxy server, and to show how they will be resolved at job runtime.
There is a default dependency resolver configuration but administrators can provide their own configuration using the
`` dependency_resolvers_conf.xml `` file in the Galaxy `` config/ `` directory.
specifies a mapping between the Galaxy web user interface and the tool command line, and (2) the actual command-line
tools, known as Galaxy *tool dependencies* , which must be installed and available on the system(s) where Galaxy is
configured to run those tools. The job script that Galaxy uses to run a job includes commands (such as changes to the
`` PATH `` environment variable) that are generated by *dependency resolvers*
the Galaxy administrative UI to display whether an installed tool's dependencies have been installed on the Galaxy
server, and to show how they will be resolved at job runtime. There is a default dependency resolver configuration but
administrators can provide their own configuration us ing the `` dependency_resolution `` configuration option in Galaxy's
configuration file, `` galaxy.yml `` . Previously this configuration was stored in a separate XML file,
`` dependency_resolvers_conf.xml `` . Loading the dependency resolvers configuration from that XML file is deprecated but
still supported, however, the documentation and sample configuration file for the XML format can only be found in Galaxy
releases prior to 21.05.
The binding between tool XML and the system tools they need to run is specified in the tool XML using `` <requirement> ``
tags, for example
.. note ::
The *tool XML* referred to below is different from the deprecated *dependency resolvers XML* referred to above.
The binding between tool XML and the command-line tools they need to run is specified in the tool XML using
`` <requirement> `` tags, for example:
.. code-block :: xml
<requirement type= "package" version= "0.7.10.039ea20639" > bwa</requirement>
In some cases these requirement tags can be specified without a version
In some cases these requirement tags can be specified without a version:
.. code-block :: xml
<requirement type= "package" > bedtools</requirement>
These declared requirements are passed as inputs to the dependency resolver.
These declared requirements are passed as inputs to the dependency resolver in order to generate the environmental setup
in the job script so that the correct tool dependencies required by the tool are found on the `` $PATH `` .
Default Dependency Resolvers
----------------------------
The default configuration of dependency resolvers is equivalent to the following `` dependency_resolvers_conf.xml ``
The default configuration of dependency resolvers is equivalent to the following configuration in `` galaxy.yml `` :
.. code-block :: xml
.. code-block :: yaml
<dependency_resolvers >
<tool_shed_packages />
<galaxy_packages />
<conda />
<galaxy_packages versionless= "true" />
<conda versionless= "true" />
</dependency_resolvers>
galaxy :
dependency_resolvers :
- type : tool_shed_packages
- type : galaxy_packages
- type : conda
- type : galaxy_packages
versionless : true
- type : conda
versionless : true
This default dependency resolver configuration contains five items:
@@ -54,7 +65,8 @@ This default dependency resolver configuration contains five items:
5. finally the *Conda dependency resolver* is checked for a package matching the required name only.
If any of the dependency resolvers succeed, a dependency resolution object is returned and no more resolvers are
called. This dependency resolution object provides shell commands to prepend to the shell script that runs the system tool.
called. This dependency resolution object provides shell commands to prepend to the shell script that runs the
command-line tool.
This order can be thought of as a descending order of deliberation. Tool Shed dependencies must be declared next to the
tool by the tool author and must be selected for installation at tool installation time - this requires specific actions
@@ -70,33 +82,52 @@ to describe software dependencies both inside of Galaxy and outside.
Tool Shed Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block :: yaml
- type : tool_shed_packages
The `` tool_shed_packages `` dependency resolver works with explicit software packages installed from the Galaxy Tool
Shed as described by legacy `` tool_dependencies.xml `` files. When such a package is installed from the Tool Shed it
creates a directory structure under the directory that is specified as the `` tool_dependency_dir `` in Galaxy's
configuration. This directory structure contains references to the tool's ID, owner (in the Tool Shed) and version
string (amongst other things) and ultimately contains a file named `` env.sh `` that contains commands to make the
dependency runnable. This is installed, along with the packaged tool, by the tool package and doesn't require any
configuration by the Galaxy administrator.
dependency runnable. This `` env.sh `` file is installed, along with the packaged tool, by the tool package and doesn't
require any configuration by the Galaxy administrator. The Tool Shed-specific components of the path come from
Galaxy's install database (see the `` install_database_connection `` option in the Galaxy configuration) and are not
configured by hand.
Tools installed from the Tool Shed may also install Conda recipes and most new best practice tools do thi s
by default now.
All new and updated tools in the Tool Shed that follow `Galaxy IUC <https://galaxyproject.org/iuc/> `_
no longer use Tool Shed dependencies, and must have dependencies resolvable via Conda. Because of this, the Tool Shed
resolver is largely only relevant to older Galaxy servers that have old tools installed.
The Tool Shed dependency resolver is not able to resolve package requirements that do not have a version string,
like the `bedtools`
like the `` bedtools ``
Galaxy Packages Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The `` galaxy_packages `` dependency resolver allows Galaxy admins to specify how Galaxy should load manually
installed packages. This resolver can be configured either to use the version string or in *versionless* mode.
.. code-block :: yaml
The Galaxy Packages dependency resolver takes a `` base_path `` argument that specifies the path under which
it starts looking for the files it requires. The default value for this `` base_path `` is the
`` tool_dependency_dir `` configured in Galaxy's `` config/galaxy.yml `` . Below the base path, the Galaxy Packages
resolver looks for directories named after tools, e.g. `` bedtools `` . As mentioned before, this resolver
works in versioned and versionless mode. The default mode is versioned, where the dependency resolver looks for a
directory named after the dependency's version string. For example, if the Galaxy tool specifies that it
needs `` bedtools `` version 2.20.1, the dependency resolver will look for a directory `` bedtools/2.20.1 `` .
- type : galaxy_packages
versionless : <true|false>
base_path : <filesystem path>
The `` galaxy_packages `` dependency resolver allows Galaxy admins to specify how Galaxy should load manually
installed packages.
This resolver can be configured with the following parameters, all of which are optional:
base_path
The path under which the resolver looks for packages matching the tool's specified requirements. The default value
is the value of the `` tool_dependency_dir `` option in Galaxy's configuration file.
versionless
Ignore requirement versions and use the "default" version instead (see below).
Below the base path, the Galaxy Packages resolver looks for a directory matching the requirement name, e.g.
`` bedtools `` . Inside the name directory, the resolver looks for a directory matching the requirement version. For
example, if the Galaxy tool specifies that it needs `` bedtools `` version 2.20.1, the dependency resolver will look for
a directory `` <base_path>/bedtools/2.20.1 `` .
If the Galaxy Package dependency resolver finds a `` bin `` directory in this directory, it adds it to the `` PATH ``
used by the scripts Galaxy uses to run tools. If, however, it finds an `` env.sh `` script, it sources this
@@ -127,37 +158,176 @@ to setup the environment for ``bedtools``
module add bedtools/bedtools-2.20.1
The Galaxy Package dependency resolver operates quite similarly when used in versionless modul e. Instead of looking
The Galaxy Package dependency resolver operates quite similarly when used in versionless mode. Instead of looking
for a directory named after a version, it looks for a directory symbolic link named `` default `` that links to a
concrete version such as the `` 2.20.1 `` example above. For example if `` bedtools/default `` links to `` bedtools/2.20.1 `` .
It then looks for a `bin` `` env.sh `` and incorporates these in the tool script that finally gets run.
This versionless (i.e. default) lookup is also used if the package requirement does not specify a version string.
Conda Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block :: yaml
- type : conda
versionless : <true|false>
prefix : <filesystem path>
exec : <filesystem path>
debug : <true|false>
ensure_channels : [ channel, channel...]
auto_install : <true|false>
auto_init : <true|false>
copy_dependencies : <true|false>
read_only : <true|false>
The `` conda `` dependency resolver is used to find (and optionally install-on-demand) dependencies using the `Conda
Package Manager <https://conda.io/>`__. For a very detailed discussion of Conda dependency resolution, check out the
:doc: `Conda FAQ <conda_faq>` .
Additionally, the conda resolver makes use of *mulled* dependencies, where all of the tool's specified requirements
are installed into a single Conda environment. More details about mulled dependencies can be found in in the
:doc: `Mulled Containers <mulled_containers>` documentation.
This resolver can be configured with the following parameters, all of which are optional:
prefix
The root of the conda installation used to locate dependencies in (default: value of global `` conda_prefix ``
option or `` <tool_dependency_dir>/_conda `` otherwise).
exec
The conda executable to use, it will default to the one on `` $PATH `` (if available) and then to
`` <conda_prefix>/bin/conda `` .
versionless
Whether to resolve tools using a version string or not (default: `` false `` ).
debug
Pass debug flag to conda commands (default: `` false `` ).
ensure_channels
Conda channels to enable by default. See https://conda.io/docs/user-guide/tasks/manage-channels.html for more
information about channels. This defaults to the value of the global `` conda_ensure_channels `` option or
`` iuc,conda-forge,bioconda,defaults `` otherwise. This order should be consistent with the `Bioconda prescribed
order <https://github.com/bioconda/bioconda-recipes/blob/master/config.yml>`__ if it includes ` `bioconda`
auto_install
If `` true `` , Galaxy will look for and install missing tool dependencies before running a job (default: value of
the global `` conda_auto_install `` option or `` false `` otherwise).
auto_init
If `` true `` , Galaxy will try to install Conda from the web automatically if it cannot find a local copy and
`` conda_exec `` is not configured (default: the value of the global `` conda_auto_init `` option or `` ` true ``
otherwise).
copy_dependencies
If `` true `` , Galaxy will copy dependencies over instead of symbolically linking them when creating per job
environments. This is deprecated because Conda will do this as needed for newer versions of Conda - such as the
versions targeted with Galaxy 17.01 and later.
read_only
If `` true `` , Galaxy will not attempt to install or uninstall requirement sets into this environment.
The conda resolver will search for Conda environments named::
__<requirement_name>@<requirement_version>
in the case that a tool only has one requirement tag, or::
mulled-v1-<hash>
when a tool has multiple requirement tags, where `` <hash> `` is a hash derived from the requirements' names and
versions.
For example, to try an administrator-maintained read-only Conda installation at `` /hpc/conda `` first and then a
Galaxy-maintained writable Conda installation at `` /galaxy/conda `` second (where any missing dependencies will
be automatically installed at tool runtime), use the following:
.. code-block :: yaml
- type : conda
auto_init : false
auto_install : false
prefix : /hpc/conda
- type : conda
auto_init : true
auto_install : true
prefix : /galaxy/conda
Lmod Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block :: yaml
- type : lmod
versionless : <true|false>
lmodexec <filesystem path>
settargexec : <filesystem path>
modulepath : <filesystem path[:filesystem path:...]>
mapping_files : <filesystem path>
The `` lmod `` dependency resolver interacts with the `Lmod environment modules system <https://lmod.readthedocs.io/> `__
commonly found on HPC systems.
This resolver can be configured with the following parameters, all of which are optional:
lmodexec
Path to the Lmod executable on your system. This cannot be just "module" because module is actually a bash
function and not the real Lmod binary (see the result of the "type module" command). Default: value of the
`` $LMOD_CMD `` environment variable.
settargexec
Path to the settarg executable on your system. Default: value of the `` $LMOD_SETTARG_CMD `` environment variable.
modulepath
Path to the folder that contains the LMOD module files on your system. This can be a single path or a
semicolon-separated list of paths. Default: value of the `` $MODULEPATH `` environment variable.
versionless
Set to `` true `` to resolve a dependency based on its name only (the version number is ignored). Only modules
marked as Default will be listed by the "avail" command (The -d option is used). Default: `` false `` .
mapping_files
Path to a YAML configuration file that can be used to link tools requirements with existing Lmod modules. Default:
`` config/lmod_modules_mapping.yml ``
Environment Modules Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The example above used Environment Modules to set the `` PATH `` (and other settings) for `` bedtools `` . With
the `` modules `` dependency resolver it is possible to use Environment Modules directory. This resolver
takes these parameters:
.. code-block :: yaml
- type : modules
versionless : <true|false>
modulecmd <filesystem path>
modulepath : <filesystem path[:filesystem path:...]>
find_by : <directory|avail>
prefetch : <true|false>
default_indicator : <string>
The `` modules `` dependency resolver interacts with the `Environment Modules system <https://modules.sourceforge.net/> `__
commonly found on HPC systems.
This resolver can be configured with the following parameters, all of which are optional:
modulecmd
p ath to Environment Modules' `` modulecmd `` tool
P ath to Environment Modules' `` modulecmd `` tool.
modulepath
v alue used for MODULEPATH environment variable, used to locate modules
V alue used for `` $ MODULEPATH`` .
versionless
w hether to resolve tools using a version string or not (default: `` false `` )
W hether to resolve tools using a version string or not (default: `` false `` ).
find_by
w hether to use the `` DirectoryModuleChecker `` or `` AvailModuleChecker `` (permissable values are `` directory `` or `` avail `` ,
default is `` avail `` )
W hether to use the `` DirectoryModuleChecker `` or `` AvailModuleChecker `` (permissable values are `` directory `` or `` avail `` ,
default is `` avail `` ).
prefetch
i n the AvailModuleChecker prefetch module info with `` module avail `` (default: `` true `` )
I n the AvailModuleChecker, prefetch module info with `` module avail `` (default: `` true `` ).
default_indicator
w hat indicate to the AvailModuleChecker that a module is the default version (default: `` (default) `` ). Note
W hat indicates to the AvailModuleChecker that a module is the default version (default: `` (default) `` ). Note
that the first module found is considered the default when no version is used by the resolver, so
the sort order of modules matters.
@@ -183,60 +353,13 @@ used, they'll be used in the ``load`` command e.g. ``modulecmd sh load bwa/0.7.1
Homebrew Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This dependency resolver uses h omebrew p ackages to resolve requirements. It is highly experimental
and undocumented .
The `` homebrew `` the `H omebrew P ackage Manager <https://brew.sh/> `__ to resolve requirements.
It is highly experimental, undocumented, and unmaintained, and likely to be dropped from the code base .
Brew Tool Shed Package Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Brewed Tool Shed Package Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This dependency resolver would resolve tool shed packages that had been
auto converted to the tool shed. It is highly experimental, undocumented,
and will almost certainy be removed from the code base.
Conda Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~
The `` conda `` directive can be used to configure a conda dependency resolver.
This resolver can be configured with the following options. For a very detailed
discussion of Conda dependency resolution, check out the :doc: `Conda FAQ <conda_faq>` .
prefix
The conda_prefix used to locate dependencies in (default: `` <tool_dependency_dir>/_conda `` ).
exec
The conda executable to use, it will default to the one on the
PATH (if available) and then to `` <conda_prefix>/bin/conda `` .
versionless
whether to resolve tools using a version string or not (default: `` False `` ).
debug
Pass debug flag to conda commands (default: `` False `` ).
ensure_channels
conda channels to enable by default. See
https://conda.io/docs/user-guide/tasks/manage-channels.html for more
information about channels. This defaults to `` iuc,conda-forge,bioconda,defaults `` .
This order should be consistent with the `Bioconda prescribed order <https://github.com/bioconda/bioconda-recipes/blob/master/config.yml> `__
if it includes `` bioconda `` .
auto_install
If `` True `` , Galaxy will look for and install missing tool
dependencies before running a job (default: `` False `` ).
auto_init
If `` True `` , Galaxy will try to install Conda from the web
automatically if it cannot find a local copy and `` conda_exec `` is not
configured. This defaults to `` True `` as of Galaxy 17.01.
copy_dependencies
If `` True `` , Galaxy will copy dependencies over instead of symbolically
linking them when creating per job environments. This should be considered somewhat
deprecated because Conda will do this as needed for newer versions of Conda - such
as the version targeted with Galaxy 17.01+.
read_only
If `` True `` , Galaxy will not attempt to install or uninstall requirement sets into
this environment.
The `` brewed_tool_shed `` as an attmept to resolve tool shed packages that had been auto converted
to the tool shed. It is highly experimental, undocumented, unmaintained, and will almost certainly be removed from the
code base.
Nate Coraor