By "static" I mean tools such as a FASTQ de-interlacer that would produce a "paired" collection with two datasets everytime. By "determinable" I mean tools that perform N->N operations within the same job - such as a tool that needs to normalize a bunch of datasets all at once and not in separate jobs. (For N->N collection operations that should or can be done in N separate jobs tool authors should just write tools that operate over a dataset and produce a dataset and let the end-user 'map over' that operation.)
There are still large classes of operations where the structure of the output collection cannot be pre-determined - such as splitting files (e.g. bam files by read group) - that are not implemented in this commit.
Model:
The models have been updated to do a more thorough job of tracking collection outputs. Jobs just producing HistoryDatasetCollectionAssociations works fine for simple jobs producing collections - but you don't want to map a list over a tool that produces a pair and produce a bunch of pairs HDCAs and a list:pair HDCA- you just want a bunch of pieces and the one list:pair at that the top.
Workflow:
Workflows containing such operations can be executed - but the workflow editor has not been updated to handle this complexity (and it will require a significant overhaul) so such tools are not available in the workflow editor.
Tool Testing:
This commit also introduces a new tool XML syntax for describing tests on output collections. See files test/functional/tools/collection_creates_list.xml and test/functional/tools/collection_creates_pair.xml for examples.
Tests:
Includes two tools to test this - one that uses explicit pair output names and one that iterates over the structure of input list to produce an output list.
Includes several new tools API tests that test the tools described above via the API and implicit mapping over such tools. Includes two new workflow API tests - one that verifies a simple workflow with output collections works and one that verifies mapping over workflow steps in collections works.
See test/functional/tools/job_properties.xml for example. Provides full access to assertion based XML tags - tabular and XML based assertions probably not so interesting so these include - has_text, has_line, not_has_text, has_text_matching, has_line_matching.
Rather than relying solely on exceptions back to nose/test framework - add option (--structured_data_report_file) to run_tests.sh that causes a bunch of detailed data to be dumped to the specified file in a very structured way. Includes full to_dict of the job from the API which in turn includes job metrics, command-line, job's standard error and outputs (instead of the test frameworks), as well as the tool inputs, and exceptions broken out for tool execution versus output checking.
Its all indexed in the file by the test id (without the actual test toolbox depending on knowing the test id) - so one could pair this information with the XUnit output to produce much more detailed breakdowns of the tests.
Gives more details to aid testing (especially for tools with lots of outputs) and lets developer collect all correct test data files at once (when GALAXY_TEST_SAVE is set or using `planemo test` with --job_output_files argument) instead of needing to rerun the tool repeatedly.
This will prevent run_tests.sh from picking up the extra test when running full test tool box - the way planemo does (i.e. this fixes the problem where planemo always runs one extra passing test).
E.g. composite data types. API should probably return the original output name since that is the purpose of the parameter but I am not sure how much work that would be.
Imagine a tool with a repeat statement like:
<repeat name="rinst" title="Repeat Parameter">
<param name="int_param" type="integer" value="1" />
<param name="float_param" type="float" value="2.0" />
</repeat>
3 test instances overidding int_param default in all three, but leaving the float_param default in place for the first and last can specified as follows:
<param name="rinst_0|int_param" value="4" />
<param name="rinst_1|int_param" value="5" />
<param name="rtnst_2|int_param" value="6" />
<param name="rinst_1|float_param" value="4.0" />
This syntax can be mixed and matched with specifing conditionals and nested repeats, etc....
One could imagine an alternative syntax like:
<param name="rinst|int_param" value="4" />
<param name="rinst|int_param" value="5" />
<param name="rtnst|int_param" value="6" />
<param name="rinst|float_param" value="4.0" />
Upon consideration, I have determined that this ambigious syntax is more difficult to reason about, leaves so much ambigiouity in place especially in the context of default and nested elements, and is harder to support. So the plan is to not support it at this time.
The following changeset will add "the right" way to do this anyway:
<repeat name="rinst">
<param name="rinst|int_param" value="4" />
</repeat>
<repeat name="rinst">
<param name="rinst|int_param" value="5" />
<param name="rinst|float_param" value="4.0" />
</repeat>
<repeat name="rinst">
<param name="rinst|int_param" value="6" />
</repeat>
Turns out previous changesets attempting to separately disambiguate and supply implicit defaults for conditionals do not really work with data inputs - because they are not recognized as dataset input params during parsing. Additionally, this change should eliminate any ordering requirements of tool test params - i.e. you shouldn't need to provide the test parameters in an order that makes it clear what branch you are on before suppling a dataset in order for the parser to realize it is a dataset.
Finally, this changeset allows deletion of a lot of code and creates some more robust abstractions.
Outputs must specify name and must be specified in the same order with the Twill variant. This second restriction is entirely arbitrary using API so dropping it here.
Adding tool demonstrating this functionality (test/functional/tools/output_order.xml) which fails with twill interactor but works fine with API interactor.
Adding file test/functional/tools/metadata.xml demonstrating how to check output metadata - this file also demonstrates setting metadata on uploaded datasets and verifies both of these functionalities. Checking output metadata is only available for new API driven tool testing.
Add example tool test for composite data (worked with Twill based test framework prior to this commit, works with API based framework as a result of this commit).
- Report formatted standard error and standard output of tools like Twill interactor.
- Respect testdef.maxseconds to timeout test cases.
- Check history item - like Twill interactor do not verify 'error' datasets.
- Properly raise exceptions with reported API error messages when running tools.
Use requests' response.content instead of response.text for reading results.
Fixes the following error 'UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-3: ordinal not in range(128)' when dealing with ASN.1 binary files.
Expand 'inputs' tree when building tool inputs for API functional tests.
Code is in there for both flat inputs (the way the UI would submit them), and a nested tree structure. The nested tree structure will not work with the Galaxy API currently - but it is the way the API should work :).
Implement GalaxyInteractorApi, is working for simple tests.
Lots left to do likely - conditionals, repeats, composite uploads, pages(?).
Introduces dependency on Python requests package - subsequent changeset will provide urllib2 based fallback if requests unavailable.
Add mechanism to tool test parser code to determine which Galaxy interactor should be used ('twill' or 'api' are current options). This maybe a stop gap to force usage of the API interactor until that becomes the default or may prove in the long term to be essential if there are certain tools that will always require Twill-specific functionality or if new browser-based JavaScript (e.g. w/selenium) are implemented. An interactor value can be specified at the tests level and/or at the level of individual test elements in the tool XML (using the 'interactor' attribute on either element).
The current default interactor is 'twill'. The default interactor app-wide can be overridden with the GALAXY_TEST_DEFAULT_INTERACTOR environment variable.
both from the dataset info page. The functional tests will now provide output
of these on tool test failure. The functional tests can now use
tool_dependency_dir as well.
John Chilton