This should not be considered the best practice for most text based files, but for binary files we cannot produce diffs anyway so this is probably better than storing the pointless binary blobs in repositories for instance. In particular Dan and I thought this would be useful for testing index files for data managers.
Includes test tool demonstrating this functionality, run test with
./run_tests.sh -framework -id md5sum
If present, it can be one of
- "default", no-op fallback to stdio tags and erroring on standard error output.
- "exit_code", error if tool exit code is not 0. (The @jmchilton recommendation).
- "aggressive", error if tool exit code is not 0 or either Exception: or Error: appears in standard error/output. (The @bgruening recommendation).
Refactoring and unit/functional tests to support and demonstrate this.
Run functional test with:
./run_tests.sh -framework -id detect_errors_aggressive
Run relevant unit tests:
nosetests test/unit/tools/test_parsing.py
Updated from original version to reflect comments on pull request #117 - in particular the ``detect_errors`` tag was moved from ``tool`` to ``command``.
- Add example tool demonstrating/testing specifing format via conditional output actions.
- Add API test testing mapping collections over tools without output action formatting.
- Add API test testing more complex actions using the Cut1 tool.
- Add script to hit the API with workflows requests.
- Update testing Dockerfile to allow just booting up with a fixed master API key for this test to leverage.
- WIP: on updating test Dockerfile to work with slurm, multiple handlers, uwsgi, etc...
- Add example tool for testing (section.xml).
- Implement popular flat group style test cases for section elements (first example in section.xml).
- Implement nested groupping style test cases for section elements (section example in section.xml).
Repeatedly one wants to do things like create symbolic links and then call a helper script - the 'interpreter' tag doesn't allow shell commands before calling a helper script so they could not be used in this fashion. The previous pattern for doing this was then to use the ToolShed only 'set_environment' requirement tag. These were onerous to setup and the resulting tools were no longer portable to non-ToolShed installed contexts - I believe this variant is more robust and elegant.
More information https://trello.com/c/0pgF5PBQ and here https://trello.com/c/XK5SqE1i.
Tools may now use $input.element_identifier during tool evalution for input 'data' parameters with the following semantics:
- If the input was specified as a single dataset by the user - this just fallbacks to providing the $input.name.
- If the input was mapped over a collection (to produce many jobs) or if the input is a 'multiple="true"' input that was provided a collection - the $input.element_identifier will be the element identifier for the corresponding collection item (generally much more useful the dataset name - since if preserved throughout workflows).
'data_collection' parameters already can access this kind of information - but it is something of a best practice to use simple 'data' parameters since they are compatible with more traditional un-collected datasets.
This commit really needs more comments - but Philip Mabon has been patiently waiting for this functionality for a long time.
Models:
Track whether dataset collections have been populated yet.
Dataset collections are still effectively immutable once populated - but dynamic output collections require them to be sort of like `final` fields in Java (analogy courtesy of JJ) - allowing them to be declared before they are initialized or populated. This is tracked by the `populated_state` field.
Tools:
Output collections can now describe `discover_datasets` elements just like datasets - except in this case instead of dynamically populating new datasets in the history - they will comprise the collection. `designation` has been reused to serve as the element_identifier for the collection element corresponding to the dataset.
See Pull Request 356 for more information on the discover_datasets tag https://bitbucket.org/galaxy/galaxy-central/pull-request/356/enhancements-for-runtime-discovered.
Workflows:
Update workflow execution and recovery for dynamic output collections.
Galaxy workflow data flow before collections
* - * - * - * - * - *
Galaxy worfklow data flow after collections (iteration 1)
* - * - * \
* - * - *
* - * - * / \
* - * - *
* - * - * \ /
* - * - *
* - * - * /
Galaxy worfklow data flow after this commit
/ * - * \
* - * * - *
/ \ * - * / \
/ \
/ \
/ / * - * \ \
* - * -- * - * * - * -- * - *
\ \ * - * / /
\ /
\ /
\ / * - * \ /
* - * * - *
\ * - * /
By "static" I mean tools such as a FASTQ de-interlacer that would produce a "paired" collection with two datasets everytime. By "determinable" I mean tools that perform N->N operations within the same job - such as a tool that needs to normalize a bunch of datasets all at once and not in separate jobs. (For N->N collection operations that should or can be done in N separate jobs tool authors should just write tools that operate over a dataset and produce a dataset and let the end-user 'map over' that operation.)
There are still large classes of operations where the structure of the output collection cannot be pre-determined - such as splitting files (e.g. bam files by read group) - that are not implemented in this commit.
Model:
The models have been updated to do a more thorough job of tracking collection outputs. Jobs just producing HistoryDatasetCollectionAssociations works fine for simple jobs producing collections - but you don't want to map a list over a tool that produces a pair and produce a bunch of pairs HDCAs and a list:pair HDCA- you just want a bunch of pieces and the one list:pair at that the top.
Workflow:
Workflows containing such operations can be executed - but the workflow editor has not been updated to handle this complexity (and it will require a significant overhaul) so such tools are not available in the workflow editor.
Tool Testing:
This commit also introduces a new tool XML syntax for describing tests on output collections. See files test/functional/tools/collection_creates_list.xml and test/functional/tools/collection_creates_pair.xml for examples.
Tests:
Includes two tools to test this - one that uses explicit pair output names and one that iterates over the structure of input list to produce an output list.
Includes several new tools API tests that test the tools described above via the API and implicit mapping over such tools. Includes two new workflow API tests - one that verifies a simple workflow with output collections works and one that verifies mapping over workflow steps in collections works.
How to use:
1.) Place multiple tools with different IDs in your tool conf.
2.) ... ummm ... no step 2 - just use the tools.
Implementation:
The Tool Shed allows tool lineages by assigning each tool version a GUID and tracking versions in a database. This
implementation works by simply allowing the ToolBox to contain multiple tools with the same ID and orders them by the version specified by the tool author.
To track enable this a second tool lineage has been introduced that just uses tool versions instead of a database (non-toolshed installed tools are not longer placed into the Tool Shed install database). The ToolBox has been updated to allow multiple versions per tool id (defaulting to the 'latest' version for all operations which do not specify a version). Both jobs and workflow steps would track tool versions but did not use that version when fetching tools from the Toolbox - these components have been updated to try to use the tool version.
Unit tests working through most of the ToolBox and tool panel have been added, as well as functional tests exercising the tools API and to ensure workflows now at least attempt to respect tool versions (still kind of silently switches versions in some cases). Manual tests against the new tool form seem to demonstrate the tool switching and tool re-running work with only minor changes to the tools API and the job handler.
See test/functional/tools/job_properties.xml for example. Provides full access to assertion based XML tags - tabular and XML based assertions probably not so interesting so these include - has_text, has_line, not_has_text, has_text_matching, has_line_matching.
With test case to ensure they don't break again. Problem was due to a small switch of an if to an elif in 7114d15a6d22 which had been sort of masking over an older bug.
Thanks to Peter Cock for reporting the issue.
Long term this could allow Galaxy to support - multiple tooling formats (Galaxy-like YAML, CWL http://bit.ly/cwltooldesc, etc...). But I think it is also important from a purely design perspective - this is a core logic class integrating different components - they should not also be doing XML parsing.
To verify the interface for parsing tools is expressive enough to allow multiple useful implementations, I built a test YAML tool description that implements many of the same features as Galaxy but smooths out rough edges (uses exit codes for job failure by default for instance). Loading these tools is disabled by default and it is not documented how to enable them because they are not intended to be part of Galaxy's public API.
Rather than relying solely on exceptions back to nose/test framework - add option (--structured_data_report_file) to run_tests.sh that causes a bunch of detailed data to be dumped to the specified file in a very structured way. Includes full to_dict of the job from the API which in turn includes job metrics, command-line, job's standard error and outputs (instead of the test frameworks), as well as the tool inputs, and exceptions broken out for tool execution versus output checking.
Its all indexed in the file by the test id (without the actual test toolbox depending on knowing the test id) - so one could pair this information with the XUnit output to produce much more detailed breakdowns of the tests.
Gives more details to aid testing (especially for tools with lots of outputs) and lets developer collect all correct test data files at once (when GALAXY_TEST_SAVE is set or using `planemo test` with --job_output_files argument) instead of needing to rerun the tool repeatedly.
This will prevent run_tests.sh from picking up the extra test when running full test tool box - the way planemo does (i.e. this fixes the problem where planemo always runs one extra passing test).
guerler