opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-09-24 16:30:27 +08:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #3483 from jmchilton/doc_revision_0

Browse Source 
[17.01] Dependency resolver documentation revisions.
This commit is contained in:
Martin Cech
2017-01-25 10:44:06 -05:00
committed by GitHub
parent 71876f6de7 133dc79d3e
commit 3eaad9a84e
2 changed files with 215 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/admin/conda_faq.rst
+142 -66
View File
@@ -1,22 +1,18 @@
.. _conda_faq:
===========================
Conda for Tool Dependencies
===========================
Galaxy tools (also called wrappers) traditionally use Tool Shed package
recipes to install their dependencies. At the tool's installation time
the recipe is downloaded and executed in order to provide the underlying
software executables. Introduction of these Galaxy-specific recipes was
a necessary step at the time, however nowadays there are other more
mature and stable options to install software in a similar manner. The
Galaxy tools (also called wrappers) have tradionally used Tool Shed package
recipes to install their dependencies. These were too tightly tied to Galaxy
and to the Tool Shed and so have been replaced with Conda as the package
management solution of choice for newer best practice tools. The
Galaxy community has taken steps to improve the tool dependency system
in order to enable new features and expand its reach. This document aims
to describe these and answer the FAQ.
Galaxy has adopted a new standard for tool dependencies: Conda packages!
Not only do Conda packages make tool dependencies more reliable and
stable, they are also easier to test and faster to develop than the
traditional Tool Shed package recipes.
in order to enable new features and expand its reach. Not only do Conda packages
make tool dependencies more reliable and stable, they are also easier to test
and faster to develop than the traditional Tool Shed package recipes. This
document aims to describe these and answer frequently asked questions.
Conda is a package manager like ``apt-get``, ``yum``, ``pip``, ``brew`` or
``guix``. We don't want to argue about the relative merits of various package
@@ -25,8 +21,8 @@ community contributions (such as implementing a Guix package manager or
enhancing the existing brew support to bring it on par with Conda).
As a community, we have decided that Conda is the one that best fulfills
community's needs. The following are some of the crucial Conda features that led
to this decision:
the community's current needs. The following are some of the crucial Conda
features that led to this decision:
- Installation of packages does not require *root* privileges
(installation at any location the Galaxy user has write access to)
@@ -48,14 +44,19 @@ Below we answer some common questions (collected by Lance Parsons):
1. How do I enable Conda dependency resolution for Galaxy jobs?
***************************************************************
Galaxy's dependency job resolution is managed via
``dependency_resolvers_conf.xml`` configuration file. Most Galaxy administrators
should be using Galaxy's default dependency resolvers configuration file
( ``dependency_resolvers_conf.xml.sample`` ). With
The short answer is that as of 17.01, Galaxy should install Conda the first
time it starts up and be configured to use it by default.
The long answer is that Galaxy's dependency job resolution is managed via
``dependency_resolvers_conf.xml`` configuration file. This configuration
file is discussed in detail in the :ref:`Dependency Resolvers <dependency_resolvers>`
documentation. Most Galaxy administrators will be using Galaxy's default dependency
resolvers configuration file (``config/dependency_resolvers_conf.xml.sample``). With
release 16.04, Galaxy has enabled Conda dependency resolution by default when
Conda was already installed on the system. Having Conda enabled in
``dependency_resolvers_conf.xml`` means that Galaxy can look for job
dependencies using the Conda system when it attempts to run tools.
Conda was already installed on the system. As of 17.01, Galaxy will also install
Conda as needed when starting up. Having Conda enabled in ``dependency_resolvers_conf.xml``
means that Galaxy can look for job dependencies using the Conda system when it
attempts to run tools.
Note that the order of resolvers in the file matters and the ``<tool_shed_packages />``
entry should remain first. This means that tools that have specified Tool Shed packages
@@ -67,24 +68,7 @@ See `galaxy.ini.sample`_ for the complete list.
+--------------------------+--------------------------+---------------------------+
| Setting | Default setting | Meaning |
+--------------------------+--------------------------+---------------------------+
| ``conda_prefix`` | <tool\_dependency\_dir>/ | the location |
| | \_conda | on the |
| | | filesystem where Conda |
| | | packages and |
| | | environments are |
| | | installed |
| | | |
| | | IMPORTANT : Due to a |
| | | current limitation in |
| | | Conda, the total length |
| | | of the |
| | | |
| | | ``conda_prefix`` and the |
| | | ``job_working_directory`` |
| | | path should be less |
| | | than 50 characters! |
+--------------------------+--------------------------+---------------------------+
| ``conda_auto_init`` | False | Set to True to instruct |
| ``conda_auto_init`` | True | Set to True to instruct |
| | | Galaxy to install Conda |
| | | (the package manager) |
| | | automatically if it |
@@ -98,6 +82,13 @@ See `galaxy.ini.sample`_ for the complete list.
| | | dependencies before |
| | | running a job. |
+--------------------------+--------------------------+---------------------------+
| ``conda_prefix`` | <tool\_dependency\_dir>/ | the location |
| | \_conda | on the |
| | | filesystem where Conda |
| | | packages and |
| | | environments are |
| | | installed |
+--------------------------+--------------------------+---------------------------+
*Table 1: Commonly used configuration options for Conda in Galaxy.*
@@ -125,6 +116,11 @@ To summarize, there are four ways to manage Conda dependencies for use
with Galaxy. For all of these options, Conda dependency management must
be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` file.
#. Galaxy Admin Interface (>= 16.07) - Galaxy will install Conda tool
dependencies when tools are installed from the Tool Shed if the
option “When available, install externally managed dependencies (e.g.
Conda)? Beta” is checked. Admins may also view and manage Conda
dependencies via the Admin interface.
#. Manual Install - Conda dependencies may be installed by
administrators from the command line. Conda (and thus the Conda
environments) should be installed in the location specified by the
@@ -141,11 +137,6 @@ be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` fi
Tools that require samtools version 0.1.19 will then be able to find
and use the installed Conda package.
#. Galaxy Admin Interface (>= 16.07) - Galaxy will install Conda tool
dependencies when tools are installed from the Tool Shed if the
option “When available, install externally managed dependencies (e.g.
Conda)? Beta” is checked. Admins may also view and manage Conda
dependencies via the Admin interface.
#. Automatically at tool run time - When a tool is run and a dependency
is not found, Galaxy will attempt to install the dependency using
Conda if ``conda_auto_install`` is activated in the configuration.
@@ -157,15 +148,13 @@ be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` fi
**************************************************************************************************
The minimum required version of Galaxy to use Conda is 16.01, however
version 16.07 or greater is recommended. The 16.07 release of Galaxy has
version 17.01 or greater is recommended. The 16.07 release of Galaxy has
a graphical user interface to manage packages, but this is not
required to have Conda dependencies managed and used by Galaxy.
Conda packages should work on all compatible operating systems with
*glibc* version 2.5 or newer (this includes Centos 5). We will most
likely switch soon to *glibc* version 2.12 as a minimum requirement (this
includes CentOS 6). So all packages will run on all \*nix operating
systems newer than 2007.
*glibc* version 2.12 or newer (this includes Centos 6). So all packages
will run on all major \*nix operating systems newer than 2007.
4. If I have Conda enabled, what do I need to do to install tools using it? For example, how can I install the latest Trinity? And how will I know the dependencies are installed?
@@ -214,6 +203,11 @@ The order in which resolvers are tried is listed in the
The first system that satisfies a requirement will be used. See
`resolver docs`_ for detailed documentation.
This however is not recommended, ideally tools will target and test
against Conda for all dependencies. Also resolving all requirements
with Conda gives Conda a chance to select compatible versions of
dependencies. Read more about selecting compatible versions on
`Issue #3299`_ and `Pull Request #3391`_.
6. How do I know what system is being used by a given tool?
***********************************************************
@@ -301,35 +295,110 @@ leave the old versions as they are – simply because of time.
Old tools will use the traditional installation system; this system will
stay and will be supported for installing old tools to guarantee sustainability
and reproducibility. New tools from the IUC, may be Conda only.
and reproducibility. New tools from the IUC and other best practices sources
are Conda only.
13. What can I do if Conda doesn't work for me?
***********************************************
13. What can I do about this placehold error?
*********************************************
There is currently a limitation in the way Conda packages are being
built. This limitation will be addressed shortly by the Conda community,
however this requires all packages to be rebuilt.
To work around this limitation, please make sure that the total length
of the ``conda_prefix`` and ``job_working_directory`` path is less than 50
characters long.
If this is your problem, you should see a warning similar to the
following in your galaxy log files:
If you see a warning similar to the following in your galaxy log files:
.. code-block:: bash
ERROR: placeholder '/home/ray/r_3_3_1-x64-3.5/envs/_build_placehold_placehold_placehold_placehold_pl' too short
This means you are very likely using an older version of Conda. This
bug has been fixed with the Conda release that is targeted by Galaxy
17.01 or newer.
In the past, the work around for this limitation, was to make sure that the total length
of the ``conda_prefix`` and ``job_working_directory`` path was less than 50
characters long.
14. What can I do about this LOCKERROR error?
***********************************************
This question addresses work arounds for Conda if something like the following
message appears in your logs:
.. code-block:: bash
Error: LOCKERROR: It looks like conda is already doing something.
The lock ['/galaxy/galaxy-app/tool-dependencies/_conda/pkgs/.conda_lock-119903'] was found. Wait for it to finish before continuing.
If you are sure that conda is not running, remove it and try again.
You can also use: $ conda clean --lock
First, you may wish to enable cached dependencies. This can be done by setting
``use_cached_dependency_manager`` in ``galaxy.ini``. Many jobs will create a
per job Conda environment with just the dependencies needed for that job installed.
This will be placed on the filesystem containg the job working directory. This
is an expensive operation and Conda doesn't always link environments correctly
across filesystems. Enabling this job caching will create a cache for each required
combination of requirements in the directory specified by ``tool_dependency_cache_dir``
in ``galaxy.ini`` (defaulting to ``<tool_dependency_dir>/_cache``).
The cached dependency manager was added to the 16.10 release of Galaxy (see
`Pull Request #3106`_). In 17.01 Galaxy was updated to build the cached dependencies
as needed if the caching is in fact enabled (see `Pull Request #3348`_) and reduced
the number of jobs that would require such caching (see `Pull Request #3391`_).
15. What can I do about linking errors?
***************************************
If Galaxy jobs run on filesystems that cannot hardlink Conda packages managed
by Galaxy, linking errors may occur when building environment to execute jobs.
There are a few ways to potentially work around this.
The most straight forward and efficient work around is probably just to enable the cached
dependency manager as described in the previous question. Notice the default location
of the cache is right next to the default Conda directory - so hardlinks should
lie on the same file system as the default Conda installation.
If this still doesn't work, perhaps the underlying file system does not support hard
linking at all. In this case it is best to add ``always_softlink: True`` to Galaxy's
YAML ``condarc`` file, this should be created by Galaxy and placed in
``<tool_dependency_dir/_condarc``. This requires Conda 4.3 or newer. Note this is a
newer version of Conda than shipped with Galaxy as of 17.01. See the question below
on upgrading Conda if you must use this trick.
Alternatively, copying can be used when creating environments instead links (either
symbolic or hard). To enable this set ``conda_copy_dependencies`` to ``True`` in
``galaxy.ini``. This requires at least version 16.07 of Galaxy.
More reading on this can be found at `Conda Pull Request #3870`_, `Conda Issue #3308`,
and Galaxy `Issue #3193`_.
16. What can I do if Conda doesn't work for me?
***********************************************
Please review the common problems covered in the previous few questions, if your
problem is different more investigation will be needed.
In rare cases Conda may not have been properly installed by Galaxy.
A symptom for this is if there is no activate script in
``<conda_prefix>/bin`` folder. In that case you can delete the ``conda_prefix`` folder
and restart Galaxy, which will again attempt to install Conda.
If this does not solve your problem or you have any trouble following
the instructions, please ask on the Galaxy mailing list or the Galaxy
IRC channel.
the instructions, please ask on the Galaxy developing mailing list or the Galaxy
Gitter or IRC channel.
17. How can I upgrade Conda?
****************************
Many potential issues with Conda have been resolved with fixes in Conda itself. If
you let Galaxy install Conda prior to the release of 17.01 you probably have version
3.19.3. This can be updated to 4.2.13 with the following command:
.. code-block:: bash
$ <tool_dependency_dir/_conda/bin/conda update -y conda==4.2.13
The command can obviously be adapted to install any version of Conda.
.. _Conda documentation: http://conda.pydata.org/docs/building/build.html
.. _Conda quick-start: http://conda.pydata.org/docs/get-started.html
@@ -342,3 +411,10 @@ IRC channel.
.. _BioConda: https://bioconda.github.io
.. _contact with the IUC: https://gitter.im/galaxy-iuc/iuc
.. _galaxy.ini.sample: https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample
.. _Pull Request #3106: https://github.com/galaxyproject/galaxy/pull/3106
.. _Pull Request #3348: https://github.com/galaxyproject/galaxy/pull/3348
.. _Pull Request #3391: https://github.com/galaxyproject/galaxy/pull/3391
.. _Issue #3193: https://github.com/galaxyproject/galaxy/issues/3193
.. _Conda Pull Request #3870: https://github.com/conda/conda/pull/3870
.. _Conda Issue #3308: https://github.com/conda/conda/issues/3308
.. _Issue #3299: https://github.com/galaxyproject/galaxy/issues/3299
doc/source/admin/dependency_resolvers.rst
+73 -30
View File
@@ -1,3 +1,6 @@
.. _dependency_resolvers:
Dependency Resolvers in Galaxy
==============================
@@ -8,7 +11,7 @@ job uses includes commands, such as changes to the ``PATH`` environment variable
resolvers*. There is a default dependency resolver configuration but administrators can provide their own configuration
using the ``dependency_resolvers_conf.xml`` configuration file in the Galaxy ``config/`` directory.
The binding between tool XML and the tools they need to run is specified in the tool XML using *requirements*
The binding between tool XML and the tools they need to run is specified in the tool XML using ``requirement``
tags, for example
.. code-block:: xml
@@ -21,9 +24,7 @@ In some cases these requirement tags can be specified without a version
<requirement type="package">bedtools</requirement>
The requirement turn into inputs to the dependency resolver. Each dependency resolver is thus given given one or
two inputs: the name of the dependency to resolve and, in most cases, the version string of the
dependency.
These declared requirements are passed as inputs to the dependency resolver.
Default Dependency Resolvers
----------------------------
@@ -33,32 +34,49 @@ The default configuration of dependency resolvers is equivalent to the following
.. code-block:: xml
<dependency_resolvers>
<!-- the default configuration, first look for dependencies installed from the toolshed -->
<!-- the default configuration, first look for legacy dependencies installed from the toolshed -->
<tool_shed_packages />
<!-- then look for env.sh files profile according to the "galaxy packages" schema -->
<!-- then look for env.sh files profile according to the "galaxy packages" schema -->
<galaxy_packages />
<galaxy_packages versionless="true" />
<!-- finally look for Conda dependencies. -->
<conda />
<conda versionless="true" />
</dependency_resolvers>
This default dependency resolver configuration contains three items. First, the *tool shed dependency resolver* is used,
then the *Galaxy packages dependency resolver* is used, first looking for packages by name and version string and then
finally looking for the package just by name. The default configuration thus prefers packages installed from the Galaxy
Tool Shed, before trying to find a "Galaxy package" satisfying the specific version the dependency requires before
finally falling back to looking for a Galaxy package with merely the correct name. If any of the dependency
This default dependency resolver configuration contains five items. First, the *tool shed dependency resolver* is used,
then the *Galaxy packages dependency resolver* is used (initially looking for packages by name and version string and then looking for the package just by name), and finally it checks Conda for a versioned or unversioned match.
The default configuration thus prefers packages installed from the Galaxy Tool Shed using legacy ``tool_dependencies.xml``
files, before trying to find a "Galaxy package" satisfying the specific version the dependency requires before
falling back to looking for a Galaxy package with merely the correct name, and then looking for Conda recipes with
matching name and version, and finally just for a Conda package with the correct name. If any of the dependency
resolvers succeeds a dependency resolution object is returned and no more resolvers are called. This dependency
resolution object provides shell commands to prepend to the shell script that runs the tool.
This order can be thought of as a descending order of deliberation. Tool Shed dependencies must be declared next to the
tool by the tool author and must be selected for installation at tool installation time - this requires specific actions
by both the tool author and the deployer who installed the tools. The dependency is therefore expected to highly craft
to the individual tool. If Galaxy packages have been setup, the deployer of a Galaxy tool has purposely crafted tool
dependency statements for a specific installation - this is slightly less deliberate than tool shed packages but
such requirements are less likely to be incidentally resolved than Conda packages. Conda recipes are neither tied to
tools or a specific installation and are maintained in Conda channels such as Bioconda.
So while tool shed packages are first - they are also somewhat deprecated. Maintaining Conda recipes makes it easier
to describe software dependencies both inside of Galaxy and outside.
Tool Shed Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``tool_shed_packages`` dependency resolver works with packages installed from the Galaxy Tool Shed. When a package
is installed from the Tool Shed it creates a directory structure under the directory that is specified as the
``tool_dependency_dir`` in Galaxy's configuration. This directory structure contains references to the tool's name,
owner (in the Tool Shed) and version string (amongst other things) and ultimately contains a file named ``env.sh``
that contains commands to make the dependency runnable. This is installed, along with the packaged tool, by the tool
package and doesn't require any configuration by the Galaxy administrator.
The ``tool_shed_packages`` dependency resolver works with explicit software packages installed from the Galaxy Tool
Shed as described by legacy ``tool_dependencies.xml`` files. When such a package is installed from the Tool Shed it
creates a directory structure under the directory that is specified as the ``tool_dependency_dir`` in Galaxy's
configuration. This directory structure contains references to the tool's ID, owner (in the Tool Shed) and version
string (amongst other things) and ultimately contains a file named ``env.sh`` that contains commands to make the
dependency runnable. This is installed, along with the packaged tool, by the tool package and doesn't require any
configuration by the Galaxy administrator.
Tools installed from the Tool Shed may also install Conda recipes and most new best practice tools do this
by default now.
The Tool Shed dependency resolver is not able to resolve package requirements that do not have a version string,
like the `bedtools` example above.
@@ -80,7 +98,20 @@ needs ``bedtools`` version 2.20.1, the dependency resolver will look for a direc
If the Galaxy Package dependency resolver finds a ``bin`` directory in this directory, it adds it to the ``PATH``
used by the scripts Galaxy uses to run tools. If, however, it finds an ``env.sh`` script, it sources this
script before running the tool that requires this dependency. This can be used to set up the environment
needed for the tool to run. For example, this ``env.sh`` uses `Environment Modules <http://modules.sourceforge.net/>`_
needed for the tool to run.
A simple example might be to assume that a collection of bioinformatics software is manually installed in various
directories under ``/opt/biosoftware``. In this case a ``<tool_dependency_dir>/bedtools/2.20.1/env.sh`` could be
setup to add the corresponding bedtools installation to the Galaxy tool execution's ``PATH``.
.. code-block:: bash
#!/bin/sh
export PATH=$PATH:/opt/biosoftware/bedtools/2.20.1/bin
As another example, this ``env.sh`` uses `Environment Modules <http://modules.sourceforge.net/>`_
to setup the environment for ``bedtools``
.. code-block:: bash
@@ -94,10 +125,10 @@ to setup the environment for ``bedtools``
module add bedtools/bedtools-2.20.1
The Galaxy Package dependency resolver operates quite similarly when used in versionless module. Instead of looking
for a directory named after a version, it looks for a directory ending in ``default``. For example
``bedtools/default``. It then looks for a `bin` subdirectory or ``envh.sh`` and incorporates these in the tool
script that finally gets run. This versionless (i.e. default) lookup is also used if the package requirement
does not specify a version string.
for a directory named after a version, it looks for a directory symbolic link named ``default`` that links to a
concrete version such as the ``2.20.1`` example above. For example if ``bedtools/default`` links to ``bedtools/2.20.1``.
It then looks for a `bin` subdirectory or ``envh.sh`` and incorporates these in the tool script that finally gets run.
This versionless (i.e. default) lookup is also used if the package requirement does not specify a version string.
Environment Modules Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -149,7 +180,8 @@ used, they'll be used in the ``load`` command e.g. ``modulecmd sh load bwa/0.7.1
Homebrew Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This dependency resolver uses homebrew packages to resolve requirements.
This dependency resolver uses homebrew packages to resolve requirements. It is highly experimental
and undocumented.
Brew Tool Shed Package Resolver
@@ -163,32 +195,43 @@ and will almost certainy be removed from the code base.
Conda Dependency Resolver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The conda XML tag can be used to configure a conda dependency resolver.
The ``conda`` directive can be used to configure a conda dependency resolver.
This resolver can be configured with the following options.
For a very detailed discussion of Conda dependency resolution, check out the
:ref:`Conda FAQ <conda_faq>`.
prefix
The conda_prefix used to locate dependencies in (default: ``<tool_dependency_dir>/_conda``).
The conda_prefix used to locate dependencies in (defaults to ``<tool_dependency_dir>/_conda``).
exec
The conda executable to use, it will default to the one on the
PATH (if available) and then to ``<conda_prefix>/bin/conda``.
versionless
whether to resolve tools using a version string or not (default: *false*)
whether to resolve tools using a version string or not (defaults to *False*).
debug
Pass debug flag to conda commands (default: false).
Pass debug flag to conda commands (default: *False*).
ensure_channels
conda channels to enable by default. See
http://conda.pydata.org/docs/custom-channels.html for more
information about channels. (default: iuc,bioconda,r,defaults,conda-forge).
information about channels. This defaults to ``iuc,bioconda,r,defaults,conda-forge``.
This order should be consistent with `Bioconda perscribed order <https://github.com/bioconda/bioconda-recipes/blob/master/config.yml#L8>`__
if it includes ``bioconda``.
auto_install
Set to True to instruct Galaxy to look for and install missing tool
dependencies before each job runs. (default: False)
dependencies before each job runs (defaults to *False*).
auto_init
Set to True to instruct Galaxy to install conda from the web
automatically if it cannot find a local copy and conda_exec is not
configured.
configured. This defaults to *True* as of Galaxy 17.01.
copy_dependencies
Set to ``True`` to instruct Galaxy to copy dependencies over instead of symbolically
linking them when creating per job environments. This should be considered somewhat
deprecated because Conda will do this as needed for newer versions of Conda - such
as the version targeted with Galaxy 17.01+.