Relative 'tool_path' directories are resolved relative to GALAXY_ROOT (or really working directory). This is probably the expected behavior - but I think it is advantageous to be able to find tools relative to the tool_conf file also. This can now be done using string.Template resolution of ${tool_conf_dir}.
Allows testing both how URLs are going to map to controllers and actions as well as testing what paths will be produced by url_for.
Requires some refactoring of Galaxy framework classes and buildapp to allow stubbing out actual instantiation of controllers, etc....
Testing it out:
---------------
- Install [docker](http://docker.io) (tough, but getting easier).
- Copy `test/functional/tools/catDocker.xml` to somewhere in `tools/` and add to `tool_conf.xml`.
- Add `<param id="docker_enabled">true</param>` to your favorite job destination.
- Run the tool.
Description and Configuration:
------------------------------
Works with all stock job runners including remote jobs with the LWR.
Supports file system isolation allowing deployer to determine what paths are exposed to container and optionally allowing these to be read-only. They can be overridden or extended but defaults are provided that attempt to guess what should be read-only and what should be writable based on Galaxy's configuration and the job destination. Lots of details and discussion in job_conf.xml.sample_advanced.
`$GALAXY_SLOTS` (however it is configured for the given runner) is passed into the container at runtime and will be available.
Tools are allowed to explicitly annotate what container should be used to run the tool. I added in hooks to allow a more expansive approach where containers could be linked to requirements and resolved that way. To be clear, this is not implemented at all but the class ContainerRegistry is instantiated, passed the list of requirements, and given the chance to return a list of potential containers... if someone wants to implement this someday.
From a reproducibility stand-point it makes sense for tool author's to have control over which container is selected, but there is this security and isolation aspect to these enhancements as well. So there are some more advanced options that allow deployers (instead of tool authors) to decide which containers are selected for jobs. `docker_default_container_id` can be added to a destination to cause that container to be used for all un-mapped tools - which will result in every job on that destination being run in a docker container. If the deployer does not even trust those tools annotated with image ids - they can go a step further and set `docker_container_id_override` instead. This will likewise cause all jobs to run in a container - but the tool details themselves will be ignored and *EVERY* tool will use the specified container.
Additional advanced docker options are available to control memory, enable network access (disabled by default), where docker is found, if and how sudo is used, etc.... These are all documented in `job_conf.xml.sample_advanced`.
Implementation Details:
-----------------------
Metadata is set outside the container - so the container itself only needs to supply the underlying application and doesn't need to be configured with Galaxy for instance. Likewise - traditional `tool_dependency_dir` based dependency resolution is disabled when job is run in a container - for now it is assumed the container will supply these dependencies.
What's Next:
------------
If implementation is merged, much is left to be discussed and worked through - how to fetch and control what images are fetched (right now the code just assumes if you have docker enabled all referenced images are available), where to fetch images from, tool shed integration (host a tool shed docker repository?, services to build docker images preconfigured with tool shed depedencies?), etc.... This is meant as more of a foundation for the dependency resolution and job runner portions of this.
This is based on work by Hector del Risco at the University of Florida.
Various tool-style parameters can be defined in job_resource_params_conf.xml (see sample) and these define the pool of all possible resource parameters that can be injected into a tool. These can be groupped together and assigned to tools in job_conf.xml (see documented changes to job_conf.xml.sample_advanced).
A conditional is then injected into each tool that gives the end-user the ability to select these parameters. The values of these parameters (as a dict) can then be obtained by dynamic job destination that define a 'resource_params' argument.
Lots here still needs to be tested/verified - can workflows exported from Galaxies without resource parameters (or different ones, be reused in Galaxies with resource parameters) (I think yes?)? can workflows exported with resource parameters be used in other Galaxies (I think yes?), how best to present UI while running workflows? (maybe nothing needs to be changed), should exported job parameter data be swept clean (say when exporting a workflow for instance).
A great extension to this would be to support conditional when defining parameters.
Finally, the Galaxy team will make a reasonable effort to support this syntax for job_conf.xml, job_resource_params_conf.xml, and internal interface for job parameters for sometime once this changeset has reached -stable but until these interfaces may change.
Thanks to Bjoern for the bug report.
This attribute had been DatasetCollection in earlier versions of this code and the update code was only partially cut-over to use the new location for name (on HistoryDatasetCollectionAssociation).
Unit tests verifing the fixed, correct behavior included. Bug introduced in e8c84dd715782e7c1d709d8068e6033b835f7f39 as an unintended consequence of duplicating the first dataset in the dictionary of input datasets generated by the tool action code.
Fixes https://trello.com/c/LCEKxImR.
Previously LWR jut serialized tool requirements, by transitioning to this class LWR should be able to resolve tool shed installed packages as well - requiring the additional repository and tool dependency context.
John Chilton