Allow multiple collections to be fed to a multi data parameter in one reduction step. Fixes#750 and will really simplify certain classes of tools.
Rebased original with fixes for rerun of such reductions.
Manually tested workflow execution and everything seems fine. The workflow editor already thought this was possible, so that is another bug corrected by this enhancement.
To run the associated API test, execute the following command:
./run_tests.sh -with_framework_test_tools -api test/api/test_tools.py:ToolsTestCase.test_reduce_multiple_lists_on_multi_data
Conflicts:
static/maps/mvc/dataset/dataset-choice.js.map
static/maps/mvc/form/form-select-content.js.map
static/scripts/mvc/form/form-select-content.js
Copy CWL style of specifying a outputs specification block at the top-level of the workflow specification and within that use the "source" attribute to specify the output. Reuse the specification used by "$link"s to describe this output - namely <label_or_order_index>[#<output_name=output>].
In downstream work this is used for subworkflows and nested tools. Not sure which of these would potentially hit main line Galaxy first so setting this all up in its own commit.
Lots of other tests would fail if simple uploads aren't working, but it is better to be direct and very explicit about why this particular test is failing.
This strategy proved to work around certain race conditions in tool testing so hopefully it will solve the transiently failing job searching and filtering test cases.
This is a slight tightening up of the ad-hoc, experimental workflow YAML definition used by the test framework (and by Kyle, but lets just admit Kyle is part of Galaxy's test framework).
This format is still defined entirely client side by transcoding the YAML or python object description into real (or format 1) Galaxy workflows - so these cannot realisticaly be declared part of Galaxy's public interface and can remain experimental and subject to change.
In addition to refactoring the code implementing these workflows for upstream modifications to explore new features, the format itself has been made slightly more stringent in two ways:
- 'steps' must now be explicit (previously converted auto-convert list to dict).
- Ensure the workflow declared a 'class' is defined and the class is 'GalaxyWorkflow'.
This will help with discovery of workflow objects with downstream tooling (planemo for workflows?) and brings the Galaxy definition slightly more inline with the CWL definition for workflows (very slightly).
Continuation of work started by @nsoranzo here 92a152d89c. In addition to increasing the max timeout it will now print the timeout in the assertion error so we can see if it is being overridden and attempt to adjust timeouts too small.
For tools which dynamically discover datasets for an output collection, keep the collection yellow until the structure of the HDCA has been fully defined. HDCAs still have the problem that they will always be green even if datasets contained within are in error, running, or queued states - but at least the structure is now reflected in the color.
By default if single data parameters are mapped over by collections, the collections are matched up (e.g. for two collections the items are paired off and two collections of dimension (N) will result in running N jobs and producing an (N) dimensional output for each define output). In the parameter meta value wrapper (where batch mode can be set with the ``batch`` flag), the linked flag is now respected for collection operations. If ``linked`` is ``False``, then the cross product of the inputs will be used to map over the tool. In the above simplest case, this would cause an NxN (``list:list``) collection to be created for each tool output.
This operation was previously available for individual datasets, but when supplied a collection it would not create an implicit collection pulling together all the relevant datasets with the correct structure - it would just run the jobs and leave the datasets uncollected. Now a collection with the correct structure and identifiers is created.
Limitations:
-----------------
This does not enable tool form support but the API for collections matches that for doing cross product operations over sets of individual datasets, so once support is added to the tool form for that collection support should be trivial.
Testing:
-----------------
The following test case has been extended to now ensure that an implicit collection is created and that it has the right dimensionality/structure and contents.
./run_tests.sh -with_framework_test_tools -api test/api/test_tools.py:ToolsTestCase.test_map_over_two_collections_unlinked
Runtime post job actions are post job actions inserted when the workflow is invoked instead of being part of the workflow object in the database. The bug noticed by @kellrott was that these actions were being appended to the original workflow post job actions instead of being transient things just attached to the jobs themselves.
This fixes that problem and adds a test to try to prevent regressions.
John Chilton