Convert workflow API to use new-style API decorator throughout.
Improved workflow exception handling - now more explicit status code setting and returning of raw error strings in the API. Most obvious exceptional paths out of the API are now coming through MessageExceptions of things even more specific. Add API tests for many of these exceptional conditions.
Use more helper methods to reduce method length, duplication.
Previous route was "POST /api/workflows/upload", this still works but should be condisdered deprecated in favor
of POSTing to /api/workflows with 'workflow' in the payload.
The old 'ds_map' parameter remains in place for backward compatibility and with the same behavior. A new parameter 'inputs' can now be specified instead however, and its keys corresponding to the steps 'order_index' instead of the raw unencoded database ids 'ds_map' uses. This variant has the nice property that this map can be constructed without prior knowledge of how Galaxy will assign ids during import - hence it is easier to use and more portable.
Additionally, 'inputs' is more flexiable and can revert to the old behavior by specifying a new parameter 'inputs_by' as 'step_id'. 'inputs_by' can also be 'name' - this is even more human friendly because it will assign the ids based on data input names (this is what I intend to use mostly, but I have not made it the default because not all workflows will have data inputs with distinct names).
Finally, the new name 'inputs' will be more appropriate once data collection inputs can be explicitly mapped via the API.
Probably should do something like this server side so workflow API can be used more deterministically (wouldn't need to import a workflow and then hit the API to know how to use it).
Thanks to Bjoern for the bug report.
This attribute had been DatasetCollection in earlier versions of this code and the update code was only partially cut-over to use the new location for name (on HistoryDatasetCollectionAssociation).
With expanded test case that checks such a connection. This takes care of of subsequent connections by iteratively updating hid_to_output_pair correctly (... I think, test works with at least two).
Collections can be mapped over 'data' parameters and sufficiently nested collections can be mapped over 'data_collection' parameters (for instance a list of 5 pairs can be supplied to a tool taking in a pair and 5 jobs will be executed). I (perhaps poorly) term these concepts collection mapping and subcollection mapping.
Prior to this changeset - the API for doing collection 'mapping' and 'subcollection mapping' was somewhat more divergent and the tool execution code explicitly forbid doing both kinds of mappings in the same tool execution even if the effective collections could be matched (e.g. it could not map a 'data' parameter over a list of 5 datasets and a pair parameter a list of 5 pairs in the same execution).
This changeset should remedy this - as long as the effective collection mappings can match up such jobs should be possible. The workflow editor (and I think runner) already thought this was possible, so this changeset reduces the tool-workflow impedance mismatch - an existing problem exacerbated by recent dataset collections introduction.
This all needs much more testing - test workflows execute this way, functional test of a tool execution that combines collection mapping and subcollection mapping, etc....
Replace ad-hoc tools API test method for skipping tests with a more general purpose decorator. Use new decorator to specify required tools for workflow tests.
Add functional test to verify the correctness of this tracking.
TODO: refactor __skip_unless_tool from tools test so this new workflow test can be skipped if not run with -with_framework_test_tools.
API now properly expects hids. That was unfortunate - need to figure out why the existing functional test did not break because of this and improve it.
Allow users to select dataset collections in place of individual datasets for data tool parameters with multiple="true" enabled (if all elements of collection would be valid as input to this parameter).
Restrict collection reductions to flat collections. If a user wanted to reduce a nested collection they probably want to map of the subcollections reducing each and building a collection of the reductions. TODO: The sentence is probably unintelligiable, need to provide a concrete example.
A functional test demonstrating these reductions in included.
Allow replacing data parameter inputs with collections - this will cause the tool to produce multiple jobs for the submission - one for each combination of input parameters after being matched up (linked). In addition to various unit tests, functional tests demonstrate the API usage in `test/functional/api/test_tools.py`.
Each data tool parameter can be specified via a similar named parameter but with the suffix |__multirun__. This second parameter variant should be a list of datasets - one job will be created for each such dataset. In addition to various unit tests, various functional tests demonstrates this functionality in `test/functional/api/test_tools.py`.
John Chilton