If present, it can be one of
- "default", no-op fallback to stdio tags and erroring on standard error output.
- "exit_code", error if tool exit code is not 0. (The @jmchilton recommendation).
- "aggressive", error if tool exit code is not 0 or either Exception: or Error: appears in standard error/output. (The @bgruening recommendation).
Refactoring and unit/functional tests to support and demonstrate this.
Run functional test with:
./run_tests.sh -framework -id detect_errors_aggressive
Run relevant unit tests:
nosetests test/unit/tools/test_parsing.py
Updated from original version to reflect comments on pull request #117 - in particular the ``detect_errors`` tag was moved from ``tool`` to ``command``.
Like done for Docker to isolate metadata commands from the environment modifications required to resolve tool dependencies. Should allow for Python 3 dependencies (originally also allowed samtools - but Nate other commit resolved that problem also).
Just meant as a config option for now - it will become the default once tested more thoroughly. For now enable it by setting enable_beta_tool_command_isolation to True in galaxy.ini.
By "static" I mean tools such as a FASTQ de-interlacer that would produce a "paired" collection with two datasets everytime. By "determinable" I mean tools that perform N->N operations within the same job - such as a tool that needs to normalize a bunch of datasets all at once and not in separate jobs. (For N->N collection operations that should or can be done in N separate jobs tool authors should just write tools that operate over a dataset and produce a dataset and let the end-user 'map over' that operation.)
There are still large classes of operations where the structure of the output collection cannot be pre-determined - such as splitting files (e.g. bam files by read group) - that are not implemented in this commit.
Model:
The models have been updated to do a more thorough job of tracking collection outputs. Jobs just producing HistoryDatasetCollectionAssociations works fine for simple jobs producing collections - but you don't want to map a list over a tool that produces a pair and produce a bunch of pairs HDCAs and a list:pair HDCA- you just want a bunch of pieces and the one list:pair at that the top.
Workflow:
Workflows containing such operations can be executed - but the workflow editor has not been updated to handle this complexity (and it will require a significant overhaul) so such tools are not available in the workflow editor.
Tool Testing:
This commit also introduces a new tool XML syntax for describing tests on output collections. See files test/functional/tools/collection_creates_list.xml and test/functional/tools/collection_creates_pair.xml for examples.
Tests:
Includes two tools to test this - one that uses explicit pair output names and one that iterates over the structure of input list to produce an output list.
Includes several new tools API tests that test the tools described above via the API and implicit mapping over such tools. Includes two new workflow API tests - one that verifies a simple workflow with output collections works and one that verifies mapping over workflow steps in collections works.
How to use:
1.) Place multiple tools with different IDs in your tool conf.
2.) ... ummm ... no step 2 - just use the tools.
Implementation:
The Tool Shed allows tool lineages by assigning each tool version a GUID and tracking versions in a database. This
implementation works by simply allowing the ToolBox to contain multiple tools with the same ID and orders them by the version specified by the tool author.
To track enable this a second tool lineage has been introduced that just uses tool versions instead of a database (non-toolshed installed tools are not longer placed into the Tool Shed install database). The ToolBox has been updated to allow multiple versions per tool id (defaulting to the 'latest' version for all operations which do not specify a version). Both jobs and workflow steps would track tool versions but did not use that version when fetching tools from the Toolbox - these components have been updated to try to use the tool version.
Unit tests working through most of the ToolBox and tool panel have been added, as well as functional tests exercising the tools API and to ensure workflows now at least attempt to respect tool versions (still kind of silently switches versions in some cases). Manual tests against the new tool form seem to demonstrate the tool switching and tool re-running work with only minor changes to the tools API and the job handler.
Leave empty module in galaxy.tools.filters and modify config to ensure backward compatibility (filters in this old directory will continue to work for now).
Update config/galaxy.ini.sample with more discussion of ToolBox filter.
- Allow overriding the base module location for ToolBox filters (needed for OS packages, etc...).
- Allow scanning multiple base modules for filters.
- Unit tests for module loading functionality, custom tool, label, and section filters, default hidden and require_login filters.
Using ToolBox._xxx instead of ToolBox.__xxx because realistically ToolBox is still much to large to grok and so I imagine it will need to be broken up even more (base class focused on just the panel details perhaps - or mixins - etc...).
None of the components using this method are using the index any more - so eliminate it. Also introduce a new ToolBox method to hide the details of ToolBox.shed_tool_confs from this method.
Carl Eberhard