Thanks to Bjoern for the bug report.
This attribute had been DatasetCollection in earlier versions of this code and the update code was only partially cut-over to use the new location for name (on HistoryDatasetCollectionAssociation).
Unit tests verifing the fixed, correct behavior included. Bug introduced in e8c84dd715782e7c1d709d8068e6033b835f7f39 as an unintended consequence of duplicating the first dataset in the dictionary of input datasets generated by the tool action code.
Fixes https://trello.com/c/LCEKxImR.
Previously LWR jut serialized tool requirements, by transitioning to this class LWR should be able to resolve tool shed installed packages as well - requiring the additional repository and tool dependency context.
With expanded test case that checks such a connection. This takes care of of subsequent connections by iteratively updating hid_to_output_pair correctly (... I think, test works with at least two).
Tool test is mildly useful to verify this works for single execution - but more useful as basis for future changesets testing mixed collection and subcollection mapping tool executions.
Collections can be mapped over 'data' parameters and sufficiently nested collections can be mapped over 'data_collection' parameters (for instance a list of 5 pairs can be supplied to a tool taking in a pair and 5 jobs will be executed). I (perhaps poorly) term these concepts collection mapping and subcollection mapping.
Prior to this changeset - the API for doing collection 'mapping' and 'subcollection mapping' was somewhat more divergent and the tool execution code explicitly forbid doing both kinds of mappings in the same tool execution even if the effective collections could be matched (e.g. it could not map a 'data' parameter over a list of 5 datasets and a pair parameter a list of 5 pairs in the same execution).
This changeset should remedy this - as long as the effective collection mappings can match up such jobs should be possible. The workflow editor (and I think runner) already thought this was possible, so this changeset reduces the tool-workflow impedance mismatch - an existing problem exacerbated by recent dataset collections introduction.
This all needs much more testing - test workflows execute this way, functional test of a tool execution that combines collection mapping and subcollection mapping, etc....
Replace ad-hoc tools API test method for skipping tests with a more general purpose decorator. Use new decorator to specify required tools for workflow tests.
Add functional test to verify the correctness of this tracking.
TODO: refactor __skip_unless_tool from tools test so this new workflow test can be skipped if not run with -with_framework_test_tools.
API now properly expects hids. That was unfortunate - need to figure out why the existing functional test did not break because of this and improve it.
This commit enables the workflow editor to deal with input collection data parameter types and inputs (easy) and much more complicatedly reason about mappings and reductions over inputs and collection inputs. Not sure I can really describe the new interface in a changeset - essentially it is more complicated to determine if a input can be connected to by an output - and that connection affects what are valid other inputs and what the outputs are.
Allow users to select dataset collections in place of individual datasets for data tool parameters with multiple="true" enabled (if all elements of collection would be valid as input to this parameter).
Restrict collection reductions to flat collections. If a user wanted to reduce a nested collection they probably want to map of the subcollections reducing each and building a collection of the reductions. TODO: The sentence is probably unintelligiable, need to provide a concrete example.
A functional test demonstrating these reductions in included.
When running normal tools with normal data inputs across dataset collections in parallel - a collection will be created for each output with a "structure" matching that of the input (for instance pairs will create pairs, lists will create lists with the same identifiers, etc...). A previous changeset added the ability to run the tool in parallel - this changeset extends that functionality to create analogous collections from these parallel runs.
For example, if one filters a pair of FASTQ files a pair is created out of the result. Likewise, if one filters a list of FASTQ files - a list collection with same cardinality is built from the results.
There is a lot left TODO here - for one a lot of this logic should be moved into the dataset_collection module. The matching needs to be exact right now - not a problem for pairs (every 'element' has name 'left' or 'right') but for lists with element names - these have to match exactly - but a 'list' like samp1_l, samp2_l, samp3_l should be able to match against samp1_r, samp2_r, samp3_r and create a new list samp1, samp2, and samp3. Even if there is no matching prefixes a new 'unlabeled' list should be able to be created.
Allow replacing data parameter inputs with collections - this will cause the tool to produce multiple jobs for the submission - one for each combination of input parameters after being matched up (linked). In addition to various unit tests, functional tests demonstrate the API usage in `test/functional/api/test_tools.py`.
Each data tool parameter can be specified via a similar named parameter but with the suffix |__multirun__. This second parameter variant should be a list of datasets - one job will be created for each such dataset. In addition to various unit tests, various functional tests demonstrates this functionality in `test/functional/api/test_tools.py`.
(Like it works for all other runners except the LWR). This is essentially scaffolding toward elimination of extra metadata step in local runner.
Got Dan's blessing to kill this off a while ago but I still don't quite understand why it is separated out - so doing this in two parts. This changeset changes nothing by default but adds an option to local runner destinations (embed_metadata_in_job) to allow them operate in the new fashion (embed the metadata setting step in the job itself). At some later point this option should just be removed and all local jobs should operate this way. Use this new option to speed up the local job runner or to test out how this runner will work in the future.
Test running, capturing standard output and exit code, killing job, setting metadata, adding Galaxy to jobs PYTHONPATH, GALAXY_SLOTS logic (default and overidding).
John Chilton