Including same test with workflow parameter substition - gotta admit I thought that workflow test was going to fail - so this week is looking pretty good :).
Add new example based on Anton's bamtools split work and label all the outputs as visible - this should likely be the default but until it is might as well demonstrate them as visible since that is how they are most useful.
Directly setting format attribute, setting to format to 'input' (ambigious, non-deterministic and should be deprecated IMO), using format_source, and using change_format actions.
Add data labels so the tools works in the workflow editor and added a conditional switches to some with collection params and multiple input data parameters to test some state-y logic in workflow editor.
Allow tool authors to specify citations using a DOI or BibTeX. BibTeX can be specified either by pointing at a BibTeX file in the tool directory or by embedding bibtex entries right in tool citation blocks. If referencing a file parallel to the tool, the file should contain only a single BibTeX entry, this restriction can be easily lifted by adding a BibTeX parser as a Python dependency for Galaxy - but I do not have permission to do this.
These citations will appear at the bottom of the tool form in a formatted way but the user will have to option to select RAW BibTeX for copying and pasting. Likewise, the history menu now has an option allowing users to aggregatesuch citations across an analysis a comparable list of citations.
UI interactions are implemented using a Backbone model and view and data is fetched from the Galaxy server as BibTeX using the API. Two API entry points have been added - one to fetch the BibTeX entries for a tool and another for a history.
BibTeX entries for citations annotated with DOIs will be fetched from http://dx.doi.org/ and cached using Beaker.
Additional Limitations:
- I am not super happy with a few different GUI elements of this. It is ugly and I didn't write the BibTeX parser but I did write the code that takes parsed BibTeX and converts it to a formatted entry. If merged, I will outline a Trello card to follow up and improve the UI and find some more standard way to build a formatted HTML citation from a parsed BibTeX entry.
- BibTeX Limitations: LaTeX embedded in the BibTeX entries doesn't render properly when producing a "pretty" citation in the GUI (should still be exported to citation managers properly though). Cross references aren't supported at this time.
Alternative Implementations:
BibTex/DOI vs. PROV:
There was some discussion of PROV encoding citation information on the development mailing list. The citation tags on tools are typed so this could certainly be added - but it was discussed at the BOSC 2014 codefest and there was some conensus that tool authors are more likely to already have BibTeX or DOIs available for the tool's references and the major reference managers end users will likely plug these citations into while writing papers are more likely to be able to consume BibTeX than anything else.
The bench biologist using Galaxy is the consumer of this work I most concerned with - if we need to convert citations into other formats such as EndNote or Word's bibliography support there is a suite of tools we could optionally plug into Galaxy (http://sourceforge.net/p/bibutils/home/Bibutils/) to enable this down the road. PROV seems to have such an ecosystem to leverage - I could not even find a tool to convert it BibTeX.
Parse BibTeX Client vs Server:
As mentioned above it would be nice in some ways to be able to parse and reason about BibTeX on the backend - but it would require adding a new dependency to Python. Since we have to ship BibTeX to the browser anyway to allow users to copy and paste it - I decided it was easier to start with parsing and formatting BibTeX on the client side. I therefore added the following library BSD JavaScript dependency https://github.com/mayanklahiri/bib2json to enable this. I would be happy to revisit this decision and produce formatted entries server side - there seem to be more Python options for doing this than JavaScript.
Relative 'tool_path' directories are resolved relative to GALAXY_ROOT (or really working directory). This is probably the expected behavior - but I think it is advantageous to be able to find tools relative to the tool_conf file also. This can now be done using string.Template resolution of ${tool_conf_dir}.
Was already parsing optional attribute but I put exactly zero thought into the implementation so these didn't work at all I don't think. This fills out the implementation, adds a test tool, and some cheetah helpers to facilitate this: "#if $collect_param" will fail if input not supplied or collection is empty and "#if $collect_param.is_input_supplied" will fail is input not supplied (i.e. empty collections will pass this check).
Testing it out:
---------------
- Install [docker](http://docker.io) (tough, but getting easier).
- Copy `test/functional/tools/catDocker.xml` to somewhere in `tools/` and add to `tool_conf.xml`.
- Add `<param id="docker_enabled">true</param>` to your favorite job destination.
- Run the tool.
Description and Configuration:
------------------------------
Works with all stock job runners including remote jobs with the LWR.
Supports file system isolation allowing deployer to determine what paths are exposed to container and optionally allowing these to be read-only. They can be overridden or extended but defaults are provided that attempt to guess what should be read-only and what should be writable based on Galaxy's configuration and the job destination. Lots of details and discussion in job_conf.xml.sample_advanced.
`$GALAXY_SLOTS` (however it is configured for the given runner) is passed into the container at runtime and will be available.
Tools are allowed to explicitly annotate what container should be used to run the tool. I added in hooks to allow a more expansive approach where containers could be linked to requirements and resolved that way. To be clear, this is not implemented at all but the class ContainerRegistry is instantiated, passed the list of requirements, and given the chance to return a list of potential containers... if someone wants to implement this someday.
From a reproducibility stand-point it makes sense for tool author's to have control over which container is selected, but there is this security and isolation aspect to these enhancements as well. So there are some more advanced options that allow deployers (instead of tool authors) to decide which containers are selected for jobs. `docker_default_container_id` can be added to a destination to cause that container to be used for all un-mapped tools - which will result in every job on that destination being run in a docker container. If the deployer does not even trust those tools annotated with image ids - they can go a step further and set `docker_container_id_override` instead. This will likewise cause all jobs to run in a container - but the tool details themselves will be ignored and *EVERY* tool will use the specified container.
Additional advanced docker options are available to control memory, enable network access (disabled by default), where docker is found, if and how sudo is used, etc.... These are all documented in `job_conf.xml.sample_advanced`.
Implementation Details:
-----------------------
Metadata is set outside the container - so the container itself only needs to supply the underlying application and doesn't need to be configured with Galaxy for instance. Likewise - traditional `tool_dependency_dir` based dependency resolution is disabled when job is run in a container - for now it is assumed the container will supply these dependencies.
What's Next:
------------
If implementation is merged, much is left to be discussed and worked through - how to fetch and control what images are fetched (right now the code just assumes if you have docker enabled all referenced images are available), where to fetch images from, tool shed integration (host a tool shed docker repository?, services to build docker images preconfigured with tool shed depedencies?), etc.... This is meant as more of a foundation for the dependency resolution and job runner portions of this.
Tool test is mildly useful to verify this works for single execution - but more useful as basis for future changesets testing mixed collection and subcollection mapping tool executions.
Allow users to select dataset collections in place of individual datasets for data tool parameters with multiple="true" enabled (if all elements of collection would be valid as input to this parameter).
Restrict collection reductions to flat collections. If a user wanted to reduce a nested collection they probably want to map of the subcollections reducing each and building a collection of the reductions. TODO: The sentence is probably unintelligiable, need to provide a concrete example.
A functional test demonstrating these reductions in included.
-framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see.
Dave Bouvier