opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-10-24 02:04:04 +08:00
Code Issues Packages Projects Releases Wiki Activity

Update Conda FAQ

Browse Source
This commit is contained in:
Enis Afgan
2016-09-01 15:55:54 -04:00
parent 0d35f9435a
commit 07cf96c260
1 changed files with 109 additions and 112 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/admin/conda_faq.rst
+109 -112
View File
@@ -18,25 +18,24 @@ Not only do Conda packages make tool dependencies more reliable and
stable, they are also easier to test and faster to develop than the
traditional Tool Shed package recipes.
Conda is a package manager like apt-get, yum, pip, brew or guix. We
don't want to argue about the relative merits of various package
managers here, in fact Galaxy supports multiple package managers and we
welcome community contributions (such as implementing a Guix package
manager or enhancing the existing brew support to bring it on par with
Conda).
Conda is a package manager like ``apt-get``, ``yum``, ``pip``, ``brew`` or
``guix``. We don't want to argue about the relative merits of various package
managers here, in fact Galaxy supports multiple package managers and we welcome
community contributions (such as implementing a Guix package manager or
enhancing the existing brew support to bring it on par with Conda).
As a community, we have decided that Conda is the one that best fulfills
our needs. The following are some of the crucial Conda features that led
As a community, we have decided that Conda is the one that best fulfills the
project's needs. The following are some of the crucial Conda features that led
to this decision:
- Installation of packages does not require root privileges
(Installation at any location the Galaxy user has write access to)
- Multiple versions of software can be installed in parallel
- Installation of packages does not require *root* privileges
(installation at any location the Galaxy user has write access to)
- Multiple versions of software can be installed on the system
- HPC-ready
- Faster and more robust package installations through p re-compiled
- Faster and more robust package installations through pre-compiled
packages (no build environment complications)
- Independent of programming language ( R, Perl, Python, Julia, Java,
pre-compiled binaries, and more )
- Independent of programming language (R, Perl, Python, Julia, Java,
pre-compiled binaries, and more)
- Easy to write recipes (1 YAML description file + 1 Bash install
script)
- An active, large and growing community (with more and more software
@@ -46,23 +45,20 @@ to this decision:
Below we answer some common questions (collected by Lance Parsons):
1. How do I enable Conda dependency resolution for existing Galaxy installations?
1. How do I enable Conda dependency resolution for Galaxy jobs?
*********************************************************************************
Most Galaxy administrators have not set up a dependency resolvers
configuration file ( ``dependency_resolvers_conf.xml`` ) which means they
are using Galaxy's default ( ``dependency_resolvers_conf.xml.sample`` ).
Galaxy has enabled Conda dependency resolution by default since release
16.04 (if Conda was installed already), so many existing installations
can use Conda dependencies. Having Conda enabled in ``dependency_resolvers_conf.xml`` means that Galaxy will
look for dependencies using the Conda system when it attempts to run
tools. If ``conda_auto_install`` is True , and a dependency is not found,
Galaxy will attempt to install it using the configured Conda channels. A
graphical user interface that allows administrators install Conda
packages directly from Galaxy when tools are installed from the Tool
Shed is available since the 16.07 version of Galaxy.
Galaxy's dependency job resolution is managed via
``dependency_resolvers_conf.xml`` configuration file. Most Galaxy administrators
have not set up a dependency resolvers configuration file, which means they are
using Galaxy's default ( ``dependency_resolvers_conf.xml.sample`` ). With
release 16.04, Galaxy has enabled Conda dependency resolution by default when
Conda was already installed on the system. Having Conda enabled in
``dependency_resolvers_conf.xml`` means that Galaxy will look for job
dependencies using the Conda system when it attempts to run tools.
The most important configuration settings related to Conda are listed in Table 1.
The most common configuration settings related to Conda are listed in Table 1.
See `galaxy.ini.sample`_ for the complete list.
+--------------------------+--------------------------+---------------------------+
| Setting | Default setting | Meaning |
@@ -84,41 +80,42 @@ The most important configuration settings related to Conda are listed in Table 1
| | | path should be less |
| | | than 50 characters! |
+--------------------------+--------------------------+---------------------------+
| ``conda_auto_init`` | False | Set to True to instruct |
| | | Galaxy to install Conda |
| | | (the package manager) |
| | | automatically if it |
| | | cannot find a local copy |
| | | already on the system. |
+--------------------------+--------------------------+---------------------------+
| ``conda_auto_install`` | False | Set to True to instruct |
| | | Galaxy to look for and |
| | | install Conda packages |
| | | for missing tool |
| | | dependencies before |
| | | running a job |
+--------------------------+--------------------------+---------------------------+
| ``conda_auto_init`` | False | Set to True to instruct |
| | | Galaxy to install Conda |
| | | automatically if it |
| | | cannot find a local copy |
| | | running a job. |
+--------------------------+--------------------------+---------------------------+
*Table 1: Important configuration options for Conda in Galaxy*
*Table 1: Commonly used configuration options for Conda in Galaxy.*
2. How do Conda dependencies work? Where do things get installed?
*********************************************************************************
In contrast to the old dependency system, which was used exclusively by
Galaxy, Conda is a pre-existing, independent project. With Conda it
is possible for an admin to install packages without touching Galaxy at
all – managing your dependencies independently from Galaxy. Galaxy can
handle these dependencies for you, but admins are not required to use
Galaxy for dependency management.
In contrast to the old dependency system, which was used exclusively by Galaxy,
Conda is a pre-existing, independent project. With Conda, it is possible for an
admin to install and manage packages without touching Galaxy at all. Galaxy can
handle these dependencies for you, but admins are not required to use Galaxy for
dependency management.
There are a few new config options in the galaxy.ini file (see Table
1 or ``galaxy.ini.sample`` for more information), but by default Galaxy
will install Conda (the package manager) and the required packages in
the ``<tool_dependency_dir>/_conda/`` directory. In this directory,
Galaxy will create an ``envs`` folder with all of the environments managed
by Galaxy. Every environment contains a ``lib``, ``bin``, ``share``, and ``include``
subdirectory, depending on the tool, and is sufficient to get a Galaxy
tool up and running. Galaxy simply sources this folder via Conda and
makes everything available before the tool is executed on your cluster.
There are a few new config options in the ``galaxy.ini`` file (see Table 1 or
`galaxy.ini.sample`_ for more information), but by default Galaxy will install
Conda (the package manager) and the required packages in the
``<tool_dependency_dir>/_conda/`` directory. In this directory, Galaxy will
create an ``envs`` folder with all of the environments managed by Galaxy. Each
environment contains a ``lib``, ``bin``, ``share``, and ``include``
subdirectory, depending on the tool, and is sufficient to get a Galaxy tool up
and running. Galaxy simply sources this folder via Conda and makes everything
available before the tool is executed on your system.
To summarize, there are four ways to manage Conda dependencies for use
with Galaxy. For all of these options, Conda dependency management must
@@ -148,9 +145,8 @@ be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` fi
#. Automatically at tool run time - When a tool is run and a dependency
is not found, Galaxy will attempt to install the dependency using
Conda if ``conda_auto_install`` is activated in the configuration.
#. Via the API , upon tool installation (>= 16.07) - The Galaxy
community maintains an `ansible role`_ that uses
BioBlend_ and the Galaxy API to install tools.
#. Via the API (>= 16.07) - The Galaxy community maintains an `ansible role`_
that uses BioBlend_ and the Galaxy API to install tools.
3. What is required to make use of this? Any specific packages, Galaxy revision, OS version, etc.?
@@ -158,29 +154,64 @@ be configured in the ``dependency_resolvers_conf.xml`` and the ``galaxy.ini`` fi
The minimum required version of Galaxy to use Conda is 16.01, however
version 16.07 or greater is recommended. The 16.07 release of Galaxy has
a nice graphical user interface to manage packages, but this is not
a graphical user interface to manage packages, but this is not
required to have Conda dependencies managed and used by Galaxy.
Conda packages should work on all compatible operating systems with
glibc version (2.5) or newer (this includes Centos 5). We will most
likely switch soon to glibc version 2.12 as minimum requirement (this
*glibc* version 2.5 or newer (this includes Centos 5). We will most
likely switch soon to *glibc* version 2.12 as a minimum requirement (this
includes CentOS 6). So all packages will run on all \*nix operating
systems newer than 2007.
4. How do I know what system is being used by a given tool?
4. If I have Conda enabled, what do I need to do to install tools using it? For example, how can I install the latest Trinity? And how will I know the dependencies are installed?
*********************************************************************************
The Galaxy log will tell you which dependency resolution system is used
This depends on your ``galaxy.ini`` setting. Galaxy will install the Conda
package manager for you if you have enabled ``conda_auto_init``. If
``conda_auto_install`` is enabled, Galaxy will install Trinity via Conda only
when a Trinity job is launched and Trinity is not yet installed. With release
16.07 you can see which dependencies are being used in the “Manage installed
tools” section of the Admin panel and you can select whether or not to install
Conda packages or Tool Shed package recipes when you install new tools there,
even if ``conda_auto_install`` is disabled.
More improvements to the UI will be coming in future releases. To see if Galaxy
has created a Trinity environment for you have a look at folder under
``<tool_dependency_dir>/_conda/envs/``(or ``<conda_prefix>/envs`` if you have changed `conda_prefix` in your galaxy.ini file).
5. Can I mix traditional Galaxy packages and Conda packages?
*********************************************************************************
Yes, the way this works is that Galaxy goes through the list of
requirements for a tool, and then determines for each requirement if it
can be satisfied by any of the active resolver systems.
The order in which resolvers are tried is listed in the
``dependency_resolvers_conf.xml`` file. The default order is
- Tool Shed packages
- Packages manually installed by administrators
- Conda packages
The first system that satisfies a requirement will be used.
6. How do I know what system is being used by a given tool?
*********************************************************************************
The Galaxy log will show which dependency resolution system is used
to satisfy each tool dependency and you can specify priorities using the
``config/dependency_resolution_conf.xml`` file. If you put Conda on
top, Galaxy will at first try to use Conda to resolve a tool dependency;
if this does not work, the following dependency resolvers is used, as
specified. See `resolver docs`_ for detailed documentation. Starting from galaxy release 16.07, you can
see which dependency will be used (“resolved”) in the admin panel.
``dependency_resolution_conf.xml`` file. If you put Conda on
top, Galaxy will first try to use Conda to resolve a tool dependency;
if this does not work, the following dependency resolver is used, as
listed. See `resolver docs`_ for detailed documentation. Starting from Galaxy
release 16.07, you can see which dependency will be used (“resolved”) in the
Admin panel.
5. How do I go about specifying Conda dependencies for a tool? All the docs still seem to recommend (or exclusively discuss) the ``tool_dependencies.xml`` method.
7. How do I go about specifying Conda dependencies for a tool? All the docs still seem to recommend (or exclusively discuss) the ``tool_dependencies.xml`` method.
*********************************************************************************
The simple answer is: you don't need to do much to make Conda work for a tool.
@@ -197,11 +228,11 @@ We will gradually adjust the documentation about ``tool_dependencies.xml`` and
deprecate it everywhere.
6. During tool installation what if there is no Conda package available for given requirement? What if the requirement is resolved in a different software than the original wrapper author meant to use?
8. During tool installation what if there is no Conda package available for a given requirement? What if the requirement is resolved in a different software than the original wrapper author meant to use?
*********************************************************************************
If there is no Conda package available during tool installation the tool
will install successfully, and can be used if its dependencies are
will install automatically, and can be used if its dependencies are
satisfied by another dependency system such as Tool Shed package
recipes, Docker containers or modules.
@@ -210,21 +241,21 @@ is no equivalent of the “owner” concept used in Galaxy packages
installed from the Tool Shed.
7. Where can I find a list of existing Conda packages that I can point to, so I don't have to reinvent the wheel for common dependencies?
9. Where can I find a list of existing Conda packages that I can point to, so I don't have to reinvent the wheel for common dependencies?
*********************************************************************************
With Conda installed run:
With Conda package manager installed on your system, run:
.. code-block:: bash
$ conda search <package_name> -c bioconda -c iuc
This will search in all channels that are activated by default in
Galaxy. If you find your package your are ready to go. If not please
Galaxy. If you find your package, you are ready to go. If not please
`create a Conda package`_ and submit_ it to BioConda_ or get in `contact with the IUC`_.
8. How can I create a new Conda package for a dependency?
10. How can I create a new Conda package for a dependency?
*********************************************************************************
Adding a package to the BioConda or IUC Conda channels will make it
@@ -234,11 +265,11 @@ and assist with all development. You will also see a few of us at this
project to get you started :)
Don't be scared! Conda recipes are really simple to write. Conda also
offers you, so called \`skeleton\` generators, generating recipes from
pypi, cran, cpan for you (mostly) automatically.
offers so called \`skeleton\` generators that generate recipes from
pypi, cran, or cpan for you (mostly) automatically.
9. Is there a way to convert traditional Tool Shed package recipes that are not yet in a Conda channel?
11. Is there a way to convert traditional Tool Shed package recipes that are not yet in a Conda channel?
*********************************************************************************
First, you do not need to do anything to your wrapper as long as the
@@ -254,47 +285,12 @@ leave the old versions as they are – simply because of time.
10. What is the recommendation for existing installations? Will I continue to maintain both systems or migrate to the new Conda system eventually?
*********************************************************************************
Old tools will use the traditional installation system, this system will
stay and will be supported to install old tools. This we promised to our
community to guarantee sustainability and reproducibility. New tools
from the IUC, may be Conda only.
Old tools will use the traditional installation system; this system will
stay and will be supported for installing old tools to guarantee sustainability
and reproducibility. New tools from the IUC, may be Conda only.
11. If I have Conda enabled, what do I need to do to install tools using it? For example, how can I install the latest Trinity? And how will I know dependencies are installed?
*********************************************************************************
This depends on your ``galaxy.ini`` setting. Galaxy will install Conda for
you if you have enabled ``conda_auto_init``. If ``conda_auto_install`` is
enabled, Galaxy will install Trinity via Conda when a Trinity job is
launched and Trinity is not yet installed . With the latest 16.07
release you can see which dependencies are being used in the “Manage
installed tools” section of the admin panel and you can select whether
or not to install Conda packages or Tool Shed package recipes when you
install new tools, even if ``conda_auto_install`` is disabled.
More improvements to the UI will be coming in future releases. To see if Galaxy
has created a Trinity environment for you have a look at folder under
``<tool_dependency_dir>/_conda/envs/``(or ``<conda_prefix>/envs`` if you have changed `conda_prefix` in your galaxy.ini file).
12. Can I mix traditional Galaxy packages and Conda packages?
*********************************************************************************
Yes, the way this works is that Galaxy goes through the list of
requirements for a tool, and then determines for each requirement if it
can be satisfied by any of the active resolver systems.
The order in which resolvers are tried is listed in the
``dependency_resolvers_conf.xml`` file, and the default order is
- Tool Shed packages
- Packages manually installed by administrators
- Conda packages
The first system that satisfies a requirement will be used.
13. What can I do if Conda doesn't work for me?
12. What can I do if Conda doesn't work for me?
*********************************************************************************
There is currently a limitation in the way Conda packages are being
@@ -331,3 +327,4 @@ IRC channel.
.. _submit: https://bioconda.github.io/#step-4-join-the-team
.. _BioConda: https://bioconda.github.io
.. _contact with the IUC: https://gitter.im/galaxy-iuc/iuc
.. _galaxy.ini.sample: https://github.com/galaxyproject/galaxy/blob/dev/config/galaxy.ini.sample