Case made well by @erasche on #1746, fixes#1746.
To resolve this more correctly (and many other potential type problems) - the workflow editor should be significantly reworked to track the datatype (or a range of datatypes) on each terminal (the way it does for collection mapping information). The change would be a weeks worth of development effort and inappropriate to apply to a release branch.
- Drop version requirement per commends by @bgruening.
- Add examples for conditionals, repeats, sections, colors, and booleans specified truevalue/falsevalue.
- Various bug fixes unearthed by new test cases.
```
<configfiles>
<inputs name="inputs" format="json" version="1" />
</configfiles>
```
Version is specified as 1, because the Galaxy tool plumbing doesn't really distinguish between parameter is '', parameter was explicitly supplied as None, parameter was absent, etc... - everything is a just a string in some ways. This distinction may be more important with JSON and for instance the CWL will require more percision, so we may want to revise Galaxy's tool handling in a backward compatible way someday.
Inherit collection_type from an input, the way metadata and format can be.
This commit includes a test tool demonstrating this functionality. The tool test for this tool can be executed using the following command:
```
./run_tests.sh -framework -id collection_type_source
```
- maxseconds on test tag stopped being respected at some point, fix that.
- maxseconds on test tag stopped being parsed at some later point, fix that.
- Revise default behavior so it can be overridden with yet another testing environment variable GALAXY_TEST_DEFAULT_WAIT.
- Add test case to ensure there is no future regression of these behaviors.
This is a slight tightening up of the ad-hoc, experimental workflow YAML definition used by the test framework (and by Kyle, but lets just admit Kyle is part of Galaxy's test framework).
This format is still defined entirely client side by transcoding the YAML or python object description into real (or format 1) Galaxy workflows - so these cannot realisticaly be declared part of Galaxy's public interface and can remain experimental and subject to change.
In addition to refactoring the code implementing these workflows for upstream modifications to explore new features, the format itself has been made slightly more stringent in two ways:
- 'steps' must now be explicit (previously converted auto-convert list to dict).
- Ensure the workflow declared a 'class' is defined and the class is 'GalaxyWorkflow'.
This will help with discovery of workflow objects with downstream tooling (planemo for workflows?) and brings the Galaxy definition slightly more inline with the CWL definition for workflows (very slightly).
Can do it by numeric index or element name, e.g. input[0] or input["forward"]. Only works for 1-D collections.
Implements #699.
Run new and most relevant existing tests with the following commands:
```
nosetests test/unit/tools/test_actions.py
./run_tests.sh -framework -id output_format_collection
./run_tests.sh -framework -id output_format
```
Expose and test through experimental YAML representation of tools. This doesn't represent official support of (vastly superior) YAML tools, this representation continues to serve as an easy way to exercise the abstract tool parsing interface.
Brings simple_constructs.yml and simple_constructs.xml further in line also.
Test:
./run_tests.sh -framework -id simple_constructs_y
It never passed, it was originally used by me to test failing tests compared output of bams as sam files. Now that we want all these tests to pass we need to rethink how to test that aspect of the testing framework.
In this context dynamic structure means the number of elements and identifiers in lists (at some or any level) are not known ahead of time. In these cases, tool authors must resort to using <discover_dataset> tags.
The generic example tool demonstrating this is test/functional/tools/collection_creates_dynamic_nested.xml - which demonstrates building a list of lists from a tool.
A common use case will probably be building a list of pairs, a tool demonstrating this can be found at test/functional/tools/collection_creates_dynamic_list_of_pairs.xml. The <discover_datasets> tag infers dynamic information from the filename - so this tool demonstrates mass renaming of a common file name pattern for pairs (_1/_2 suffixes) into a format more readily consumed by <discover_datasets> in a portable shell fashion.
Previously, these <discover_dataset> tags would just use the "designation" concept as the element identifier when building collections. This was insufficient for nested collections because an identifier must be specified at each level of the collection. Now <discover_dataset> tags may define multiple identifier_N patterns - e.g. identifier_0, identifier_1, etc.... If these are specified, a desigination no longer must be specified and will just be interpreted as <value_0>:<value_1>:.... This scheme is completely backward compatible and when there is only a single identifier - desigination and identifier_0 can be used interchangeably.
Testing:
The simple list of lists tool test case can be executed using:
```
./run_tests.sh -framework -id collection_creates_dynamic_list_of_pairs
```
This verifies the nested datasets are setup properly.
The list of pairs example tool test case can be executed using:
```
./run_tests.sh -framework -id collection_creates_dynamic_list_of_pairs
```
This additionally verifies paired dataset collections can be created this way and that the shell commands in the example work properly for renaming the files for easy discovery.
Finally, the following existing test case verifies that designation can be used in lieu of identifier_0 for flat collections (i.e. this generalization is backward compatible):
```
./run_tests.sh -framework -id collection_split_on_column
```
https://bitbucket.org/galaxy/galaxy-central/pull-request/634/allow-tools-to-explicitly-produce-dataset added the ability for tool to create simple collections (lists and pairs). It also outlined three creation scenarios - fixed collections, pre-determinable collection structures (like output lists based on input lists), and fully dynamic output collections. This pull request allows tool to output nested collections for these first two.
Fully dynamic nested collections (using <discover_datasets> tags) is not implemented in this commit.
Additionally, the tool test syntax has been extended to allow testing nested collection elements and an example tool is included that demonstrates this functionality - test/functional/tools/collection_creates_list_of_pairs.xml.
An ``<environment_variables>`` top-level XML block can be used to define these.
```
<environment_variables>
<environment_variable name="FOO">$bar</environment_variable>
<environment_variable name="FOO2">$bar2</environment_variable
</environment_variables>
```
The expressions are contained in the text of the block.
Implementation Detials:
In order to avoid odd shell expression stuff - the evaluated expressions are written to files and environment variables are read from these files at runtime.
Testing:
A demo tool ``test/functional/tools/environment_variables.xml`` with a test case that can be executed with the following expression.
```
./run_tests.sh -framework -id environment_variables
```
John Chilton