Update tool XSD to encourage using fully qualified inputs.
Reorganized the existing tests for consistency with other test tools I think.
(Rebased with fixes, including XSD fixes from @nsoranzo.)
Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Re-use test_data specification used by many of the API workflow tests and a workflow used by another test. Add abstractions and DOM element for mapping labeled test data to tool form inputs.
- More stuff tested:
- Connecting and disconnecting simple terminals as well as mapping terminals (ribbons).
- Rendering simple workflows, workflow with output collections, workflow with mapping, and workflow with a subworkflow.
- Collapsing side panels in the workflow editor.
- Small tweaks to editor DOM.
- More screenshots.
Should only affect unzip I think - fixes#5780 - probably need to rebuild collections though, collections built in the meantime are going to be broken in the DB.
Includes test case that fails against 18.01.
I did a few of these test conversions with 0ff639a29c but this covers many more the workflow API tests. The idea is to switch to this with statement and context manager that does a better job managing test histories including summarizing the history when problems occur (this summary will include job information after #5697).
Allows describing hierarchical data in JSON or inferring structure from archives or directories.
Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories.
Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads).
The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render.
This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_libraries.py with one that converts a simple YAML file into an API call and allows many new options for free.
In future PRs I'll add filtering options to this and it will serve as the backend to 4733.
This would affect for example the mapping over of fastq-dump (and all other
tools in the sra-toolkit).
If one had attempted this previously the discovery phase would fail with:
```
galaxy.tools.parameters.output_collect ERROR 2018-01-30 08:56:46,369 Problem gathering output collection.
Traceback (most recent call last):
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 169, in collect_dynamic_collections
collection
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/managers/collections.py", line 216, in collection_builder_for
return builder.BoundCollectionBuilder(dataset_collection, collection_type_description)
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/dataset_collections/builder.py", line 84, in __init__
raise Exception("Cannot reset elements of an already populated dataset collection.")
Exception: Cannot reset elements of an already populated dataset collection.
```
Instead we force the collection state to be new when we are discovering output collection datasets,
which seems reasonable to me. This includes an API testcase that would have failed previously.
This is using the simplest of workflows. I had to
remove the chromInfo part from the workflow, because the
path referenced would be changed dynamically and so the
job search wouldn't recognize this as being equivalent.
Also includes a simple test that runs cat1 three times,
the second time with `use_cached_job=False` (the default),
and the third time with `use_cached_job=True` and verifies
that the second output is different from the third, and that the
first and third output are identical.
This does not work at the moment, since we need to verify a jobs' input name,
dataset and metadata, for which we don't have an explicit trail. The only trace
is via the update_time (i.e if a input dataset hasn't been updated since the
job was created we know these were the actual parameters in use during the
execution of this job).
Nicola Soranzo