diff --git a/config/dependency_resolvers_conf.xml.sample b/config/dependency_resolvers_conf.xml.sample deleted file mode 120000 index 7a876ca5938..00000000000 --- a/config/dependency_resolvers_conf.xml.sample +++ /dev/null @@ -1 +0,0 @@ -../lib/galaxy/config/sample/dependency_resolvers_conf.xml.sample \ No newline at end of file diff --git a/doc/source/admin/dependency_resolvers.rst b/doc/source/admin/dependency_resolvers.rst index 2c601f5acdb..c4be94d322b 100644 --- a/doc/source/admin/dependency_resolvers.rst +++ b/doc/source/admin/dependency_resolvers.rst @@ -5,43 +5,54 @@ Dependency Resolvers in Galaxy ============================== There are two parts to building a link between Galaxy and command line bioinformatics tools: (1) the tool XML that -specifies a mapping between the Galaxy web user interface and the tool command line, and (2) system tool dependencies that -specify how to source the actual packages that implement the tool’s commands. The job script that Galaxy uses to run a job -includes commands (such as changes to the ``PATH`` environment variable) that are generated by *dependency -resolvers*. These same dependency resolvers are used by the Galaxy administrative UI to display whether an installed -tool's dependencies have been installed on the Galaxy server, and to show how they will be resolved at job runtime. -There is a default dependency resolver configuration but administrators can provide their own configuration using the -``dependency_resolvers_conf.xml`` configuration file in the Galaxy ``config/`` directory. +specifies a mapping between the Galaxy web user interface and the tool command line, and (2) the actual command-line +tools, known as Galaxy *tool dependencies*, which must be installed and available on the system(s) where Galaxy is +configured to run those tools. The job script that Galaxy uses to run a job includes commands (such as changes to the +``PATH`` environment variable) that are generated by *dependency resolvers*. These same dependency resolvers are used by +the Galaxy administrative UI to display whether an installed tool's dependencies have been installed on the Galaxy +server, and to show how they will be resolved at job runtime. There is a default dependency resolver configuration but +administrators can provide their own configuration using the ``dependency_resolution`` configuration option in Galaxy's +configuration file, ``galaxy.yml``. Previously this configuration was stored in a separate XML file, +``dependency_resolvers_conf.xml``. Loading the dependency resolvers configuration from that XML file is deprecated but +still supported, however, the documentation and sample configuration file for the XML format can only be found in Galaxy +releases prior to 21.05. -The binding between tool XML and the system tools they need to run is specified in the tool XML using ```` -tags, for example +.. note:: + + The *tool XML* referred to below is different from the deprecated *dependency resolvers XML* referred to above. + +The binding between tool XML and the command-line tools they need to run is specified in the tool XML using +```` tags, for example: .. code-block:: xml bwa -In some cases these requirement tags can be specified without a version +In some cases these requirement tags can be specified without a version: .. code-block:: xml bedtools -These declared requirements are passed as inputs to the dependency resolver. +These declared requirements are passed as inputs to the dependency resolver in order to generate the environmental setup +in the job script so that the correct tool dependencies required by the tool are found on the ``$PATH``. Default Dependency Resolvers ---------------------------- -The default configuration of dependency resolvers is equivalent to the following ``dependency_resolvers_conf.xml`` +The default configuration of dependency resolvers is equivalent to the following configuration in ``galaxy.yml``: -.. code-block:: xml +.. code-block:: yaml - - - - - - - + galaxy: + dependency_resolvers: + - type: tool_shed_packages + - type: galaxy_packages + - type: conda + - type: galaxy_packages + versionless: true + - type: conda + versionless: true This default dependency resolver configuration contains five items: @@ -54,7 +65,8 @@ This default dependency resolver configuration contains five items: 5. finally the *Conda dependency resolver* is checked for a package matching the required name only. If any of the dependency resolvers succeed, a dependency resolution object is returned and no more resolvers are -called. This dependency resolution object provides shell commands to prepend to the shell script that runs the system tool. +called. This dependency resolution object provides shell commands to prepend to the shell script that runs the +command-line tool. This order can be thought of as a descending order of deliberation. Tool Shed dependencies must be declared next to the tool by the tool author and must be selected for installation at tool installation time - this requires specific actions @@ -70,33 +82,52 @@ to describe software dependencies both inside of Galaxy and outside. Tool Shed Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +.. code-block:: yaml + + - type: tool_shed_packages + The ``tool_shed_packages`` dependency resolver works with explicit software packages installed from the Galaxy Tool Shed as described by legacy ``tool_dependencies.xml`` files. When such a package is installed from the Tool Shed it creates a directory structure under the directory that is specified as the ``tool_dependency_dir`` in Galaxy's configuration. This directory structure contains references to the tool's ID, owner (in the Tool Shed) and version string (amongst other things) and ultimately contains a file named ``env.sh`` that contains commands to make the -dependency runnable. This is installed, along with the packaged tool, by the tool package and doesn't require any -configuration by the Galaxy administrator. +dependency runnable. This ``env.sh`` file is installed, along with the packaged tool, by the tool package and doesn't +require any configuration by the Galaxy administrator. The Tool Shed-specific components of the path come from +Galaxy's install database (see the ``install_database_connection`` option in the Galaxy configuration) and are not +configured by hand. -Tools installed from the Tool Shed may also install Conda recipes and most new best practice tools do this -by default now. +All new and updated tools in the Tool Shed that follow `Galaxy IUC `_ best practices +no longer use Tool Shed dependencies, and must have dependencies resolvable via Conda. Because of this, the Tool Shed +resolver is largely only relevant to older Galaxy servers that have old tools installed. The Tool Shed dependency resolver is not able to resolve package requirements that do not have a version string, -like the `bedtools` example above. +like the ``bedtools`` example above. Galaxy Packages Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The ``galaxy_packages`` dependency resolver allows Galaxy admins to specify how Galaxy should load manually -installed packages. This resolver can be configured either to use the version string or in *versionless* mode. +.. code-block:: yaml -The Galaxy Packages dependency resolver takes a ``base_path`` argument that specifies the path under which -it starts looking for the files it requires. The default value for this ``base_path`` is the -``tool_dependency_dir`` configured in Galaxy's ``config/galaxy.yml``. Below the base path, the Galaxy Packages -resolver looks for directories named after tools, e.g. ``bedtools``. As mentioned before, this resolver -works in versioned and versionless mode. The default mode is versioned, where the dependency resolver looks for a -directory named after the dependency's version string. For example, if the Galaxy tool specifies that it -needs ``bedtools`` version 2.20.1, the dependency resolver will look for a directory ``bedtools/2.20.1``. + - type: galaxy_packages + versionless: + base_path: + +The ``galaxy_packages`` dependency resolver allows Galaxy admins to specify how Galaxy should load manually +installed packages. + +This resolver can be configured with the following parameters, all of which are optional: + +base_path + The path under which the resolver looks for packages matching the tool's specified requirements. The default value + is the value of the ``tool_dependency_dir`` option in Galaxy's configuration file. + +versionless + Ignore requirement versions and use the "default" version instead (see below). + +Below the base path, the Galaxy Packages resolver looks for a directory matching the requirement name, e.g. +``bedtools``. Inside the name directory, the resolver looks for a directory matching the requirement version. For +example, if the Galaxy tool specifies that it needs ``bedtools`` version 2.20.1, the dependency resolver will look for +a directory ``/bedtools/2.20.1``. If the Galaxy Package dependency resolver finds a ``bin`` directory in this directory, it adds it to the ``PATH`` used by the scripts Galaxy uses to run tools. If, however, it finds an ``env.sh`` script, it sources this @@ -127,37 +158,176 @@ to setup the environment for ``bedtools`` module add bedtools/bedtools-2.20.1 -The Galaxy Package dependency resolver operates quite similarly when used in versionless module. Instead of looking +The Galaxy Package dependency resolver operates quite similarly when used in versionless mode. Instead of looking for a directory named after a version, it looks for a directory symbolic link named ``default`` that links to a concrete version such as the ``2.20.1`` example above. For example if ``bedtools/default`` links to ``bedtools/2.20.1``. It then looks for a `bin` subdirectory or ``env.sh`` and incorporates these in the tool script that finally gets run. This versionless (i.e. default) lookup is also used if the package requirement does not specify a version string. +Conda Dependency Resolver +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + - type: conda + versionless: + prefix: + exec: + debug: + ensure_channels: [channel, channel...] + auto_install: + auto_init: + copy_dependencies: + read_only: + +The ``conda`` dependency resolver is used to find (and optionally install-on-demand) dependencies using the `Conda +Package Manager `__. For a very detailed discussion of Conda dependency resolution, check out the +:doc:`Conda FAQ `. + +Additionally, the conda resolver makes use of *mulled* dependencies, where all of the tool's specified requirements +are installed into a single Conda environment. More details about mulled dependencies can be found in in the +:doc:`Mulled Containers ` documentation. + +This resolver can be configured with the following parameters, all of which are optional: + +prefix + The root of the conda installation used to locate dependencies in (default: value of global ``conda_prefix`` + option or ``/_conda`` otherwise). + +exec + The conda executable to use, it will default to the one on ``$PATH`` (if available) and then to + ``/bin/conda``. + +versionless + Whether to resolve tools using a version string or not (default: ``false``). + +debug + Pass debug flag to conda commands (default: ``false``). + +ensure_channels + Conda channels to enable by default. See https://conda.io/docs/user-guide/tasks/manage-channels.html for more + information about channels. This defaults to the value of the global ``conda_ensure_channels`` option or + ``iuc,conda-forge,bioconda,defaults`` otherwise. This order should be consistent with the `Bioconda prescribed + order `__ if it includes ``bioconda``. + +auto_install + If ``true``, Galaxy will look for and install missing tool dependencies before running a job (default: value of + the global ``conda_auto_install`` option or ``false`` otherwise). + +auto_init + If ``true``, Galaxy will try to install Conda from the web automatically if it cannot find a local copy and + ``conda_exec`` is not configured (default: the value of the global ``conda_auto_init`` option or ```true`` + otherwise). + +copy_dependencies + If ``true``, Galaxy will copy dependencies over instead of symbolically linking them when creating per job + environments. This is deprecated because Conda will do this as needed for newer versions of Conda - such as the + versions targeted with Galaxy 17.01 and later. + +read_only + If ``true``, Galaxy will not attempt to install or uninstall requirement sets into this environment. + +The conda resolver will search for Conda environments named:: + + __@ + +in the case that a tool only has one requirement tag, or:: + + mulled-v1- + +when a tool has multiple requirement tags, where ```` is a hash derived from the requirements' names and +versions. + +For example, to try an administrator-maintained read-only Conda installation at ``/hpc/conda`` first and then a +Galaxy-maintained writable Conda installation at ``/galaxy/conda`` second (where any missing dependencies will +be automatically installed at tool runtime), use the following: + +.. code-block:: yaml + + - type: conda + auto_init: false + auto_install: false + prefix: /hpc/conda + - type: conda + auto_init: true + auto_install: true + prefix: /galaxy/conda + +Lmod Dependency Resolver +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + - type: lmod + versionless: + lmodexec + settargexec: + modulepath: + mapping_files: + + +The ``lmod`` dependency resolver interacts with the `Lmod environment modules system `__ +commonly found on HPC systems. + +This resolver can be configured with the following parameters, all of which are optional: + +lmodexec + Path to the Lmod executable on your system. This cannot be just "module" because module is actually a bash + function and not the real Lmod binary (see the result of the "type module" command). Default: value of the + ``$LMOD_CMD`` environment variable. + +settargexec + Path to the settarg executable on your system. Default: value of the ``$LMOD_SETTARG_CMD`` environment variable. + +modulepath + Path to the folder that contains the LMOD module files on your system. This can be a single path or a + semicolon-separated list of paths. Default: value of the ``$MODULEPATH`` environment variable. + +versionless + Set to ``true`` to resolve a dependency based on its name only (the version number is ignored). Only modules + marked as Default will be listed by the "avail" command (The -d option is used). Default: ``false``. + +mapping_files + Path to a YAML configuration file that can be used to link tools requirements with existing Lmod modules. Default: + ``config/lmod_modules_mapping.yml`` + Environment Modules Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The example above used Environment Modules to set the ``PATH`` (and other settings) for ``bedtools``. With -the ``modules`` dependency resolver it is possible to use Environment Modules directory. This resolver -takes these parameters: +.. code-block:: yaml + + - type: modules + versionless: + modulecmd + modulepath: + find_by: + prefetch: + default_indicator: + + +The ``modules`` dependency resolver interacts with the `Environment Modules system `__ +commonly found on HPC systems. + +This resolver can be configured with the following parameters, all of which are optional: modulecmd - path to Environment Modules' ``modulecmd`` tool + Path to Environment Modules' ``modulecmd`` tool. modulepath - value used for MODULEPATH environment variable, used to locate modules + Value used for ``$MODULEPATH`` environment variable, used to locate modules. versionless - whether to resolve tools using a version string or not (default: ``false``) + Whether to resolve tools using a version string or not (default: ``false``). find_by - whether to use the ``DirectoryModuleChecker`` or ``AvailModuleChecker`` (permissable values are ``directory`` or ``avail``, - default is ``avail``) + Whether to use the ``DirectoryModuleChecker`` or ``AvailModuleChecker`` (permissable values are ``directory`` or ``avail``, + default is ``avail``). prefetch - in the AvailModuleChecker prefetch module info with ``module avail`` (default: ``true``) + In the AvailModuleChecker, prefetch module info with ``module avail`` (default: ``true``). default_indicator - what indicate to the AvailModuleChecker that a module is the default version (default: ``(default)``). Note + What indicates to the AvailModuleChecker that a module is the default version (default: ``(default)``). Note that the first module found is considered the default when no version is used by the resolver, so the sort order of modules matters. @@ -183,60 +353,13 @@ used, they'll be used in the ``load`` command e.g. ``modulecmd sh load bwa/0.7.1 Homebrew Dependency Resolver ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This dependency resolver uses homebrew packages to resolve requirements. It is highly experimental -and undocumented. +The ``homebrew`` dependency resolver uses the `Homebrew Package Manager `__ to resolve requirements. +It is highly experimental, undocumented, and unmaintained, and likely to be dropped from the code base. -Brew Tool Shed Package Resolver -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Brewed Tool Shed Package Resolver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This dependency resolver would resolve tool shed packages that had been -auto converted to the tool shed. It is highly experimental, undocumented, -and will almost certainy be removed from the code base. - - -Conda Dependency Resolver -~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``conda`` directive can be used to configure a conda dependency resolver. -This resolver can be configured with the following options. For a very detailed -discussion of Conda dependency resolution, check out the :doc:`Conda FAQ `. - -prefix - The conda_prefix used to locate dependencies in (default: ``/_conda``). - -exec - The conda executable to use, it will default to the one on the - PATH (if available) and then to ``/bin/conda``. - -versionless - whether to resolve tools using a version string or not (default: ``False``). - -debug - Pass debug flag to conda commands (default: ``False``). - -ensure_channels - conda channels to enable by default. See - https://conda.io/docs/user-guide/tasks/manage-channels.html for more - information about channels. This defaults to ``iuc,conda-forge,bioconda,defaults``. - This order should be consistent with the `Bioconda prescribed order `__ - if it includes ``bioconda``. - -auto_install - If ``True``, Galaxy will look for and install missing tool - dependencies before running a job (default: ``False``). - -auto_init - If ``True``, Galaxy will try to install Conda from the web - automatically if it cannot find a local copy and ``conda_exec`` is not - configured. This defaults to ``True`` as of Galaxy 17.01. - -copy_dependencies - If ``True``, Galaxy will copy dependencies over instead of symbolically - linking them when creating per job environments. This should be considered somewhat - deprecated because Conda will do this as needed for newer versions of Conda - such - as the version targeted with Galaxy 17.01+. - -read_only - If ``True``, Galaxy will not attempt to install or uninstall requirement sets into - this environment. +The ``brewed_tool_shed`` dependency resolver was an attmept to resolve tool shed packages that had been auto converted +to the tool shed. It is highly experimental, undocumented, unmaintained, and will almost certainly be removed from the +code base. diff --git a/doc/source/admin/galaxy_options.rst b/doc/source/admin/galaxy_options.rst index 895ad01274b..bcf22549870 100644 --- a/doc/source/admin/galaxy_options.rst +++ b/doc/source/admin/galaxy_options.rst @@ -407,13 +407,10 @@ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :Description: - The dependency resolvers config file specifies an ordering and - options for how Galaxy resolves tool dependencies (requirement - tags in Tool XML). The default ordering is to the use the Tool - Shed for tools installed that way, use local Galaxy packages, and - then use Conda if available. See - https://github.com/galaxyproject/galaxy/blob/dev/doc/source/admin/dependency_resolvers.rst - for more information on these options. + Specifies the path to the standalone dependency resolvers + configuration file. This configuration can now be specified + directly in the Galaxy configuration, see the description of the + 'dependency_resolvers' option for details. The value of this option will be resolved with respect to . :Default: ``dependency_resolvers_conf.xml`` @@ -3861,6 +3858,9 @@ definition of the resolvers to enable can be embedded into Galaxy's config with this option. This has no effect if a dependency_resolvers_config_file is used. + The syntax, available resolvers, and documentation of their + options is explained in detail in the documentation: + https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html :Default: ``None`` :Type: seq diff --git a/lib/galaxy/config/sample/dependency_resolvers_conf.xml.sample b/lib/galaxy/config/sample/dependency_resolvers_conf.xml.sample deleted file mode 100644 index f111e535818..00000000000 --- a/lib/galaxy/config/sample/dependency_resolvers_conf.xml.sample +++ /dev/null @@ -1,53 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - diff --git a/lib/galaxy/config/sample/galaxy.yml.sample b/lib/galaxy/config/sample/galaxy.yml.sample index ae2da998746..78a7fe6683f 100644 --- a/lib/galaxy/config/sample/galaxy.yml.sample +++ b/lib/galaxy/config/sample/galaxy.yml.sample @@ -307,13 +307,10 @@ galaxy: # . #tool_dependency_dir: dependencies - # The dependency resolvers config file specifies an ordering and - # options for how Galaxy resolves tool dependencies (requirement tags - # in Tool XML). The default ordering is to the use the Tool Shed for - # tools installed that way, use local Galaxy packages, and then use - # Conda if available. See - # https://github.com/galaxyproject/galaxy/blob/dev/doc/source/admin/dependency_resolvers.rst - # for more information on these options. + # Specifies the path to the standalone dependency resolvers + # configuration file. This configuration can now be specified directly + # in the Galaxy configuration, see the description of the + # 'dependency_resolvers' option for details. # The value of this option will be resolved with respect to # . #dependency_resolvers_config_file: dependency_resolvers_conf.xml @@ -1898,6 +1895,9 @@ galaxy: # definition of the resolvers to enable can be embedded into Galaxy's # config with this option. This has no effect if a # dependency_resolvers_config_file is used. + # The syntax, available resolvers, and documentation of their options + # is explained in detail in the documentation: + # https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html #dependency_resolvers: null # Alternative representation of various dependency resolution diff --git a/lib/galaxy/webapps/galaxy/config_schema.yml b/lib/galaxy/webapps/galaxy/config_schema.yml index c8540a33011..0097aea618d 100644 --- a/lib/galaxy/webapps/galaxy/config_schema.yml +++ b/lib/galaxy/webapps/galaxy/config_schema.yml @@ -316,12 +316,9 @@ mapping: path_resolves_to: config_dir required: false desc: | - The dependency resolvers config file specifies an ordering and options for how - Galaxy resolves tool dependencies (requirement tags in Tool XML). The default - ordering is to the use the Tool Shed for tools installed that way, use local - Galaxy packages, and then use Conda if available. - See https://github.com/galaxyproject/galaxy/blob/dev/doc/source/admin/dependency_resolvers.rst - for more information on these options. + Specifies the path to the standalone dependency resolvers configuration file. This + configuration can now be specified directly in the Galaxy configuration, see the + description of the 'dependency_resolvers' option for details. conda_prefix: type: str @@ -2803,6 +2800,11 @@ mapping: resolvers to enable can be embedded into Galaxy's config with this option. This has no effect if a dependency_resolvers_config_file is used. + The syntax, available resolvers, and documentation of their options is explained in detail in the + documentation: + + https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html + dependency_resolution: type: map desc: |