This specifically addresses the problem where some jobs of a mapped-over
collection have failed. Instead of filtering the failed collection and
restarting the workflow at this position (involving a lot of copy-paste ...)
the user can now limit the rerun to the problematic jobs and the workflow
should resume from there.
Should fix https://github.com/galaxyproject/galaxy/issues/2235.
This is one possible implementation, it would also be feasible to not
manipulate the original collection, but to copy the HDCA and then to replace
collection elements and replace all references for jobs that depend on the HDCA,
as we do for HDAs. This implementation seems simpler, but let me know if you
see problems with this approach.
- Update explicit stdio definitions as well as detect_errors-based definitions to allow this.
- Update runner code to allow resubmission in such cases.
If one maps a tool over an empty collection, one should get an empty output collection for each of the tool's outputs.
This should have always been a valid operation I think, the code just wasn't quite structured right when the collection work was merged - there were too many places I was assuming the existence of at least one actual output or input that was mapped over to continue going. This reworks all of that and seems to handle empty collections appropriately.
Fixes#4025 (not being informing the user of the operation, but actually by allowing the operation - which it should be allowed I think).
Checked through tools-iuc and I couldn't find any tools that use the actual input data parameters in filter statements so I think it is a fairly good approximation to assume that all mapped jobs will either filter or not at least with current execution model.
Increase the timeout - this test fails on Jenkins every now and then and reports the DM job is still queued. I assume this is because of conda_auto_install setting up the requirements for this data manager and the logs seem consistent with that assumption.
I also added data manager json type to the sample so this warning goes away:
```
galaxy.model WARNING 2017-11-30 06:39:59,335 Datatype class not found for extension 'data_manager_json'
```
- Replace test using Cut1 with two tests each using framework test tools loaded with Selenium tests by default.
- Create a symbolic link so that tour generator is configured with default test webhooks.
- Establish and use components and navigates_galaxy.py abstractions for tour generation and navigation.
- In one of the tool tests actually walk the tour (previously we only verified the initial tour popover).
- Add flake8-import-order to flake8 Pipfile and remove py27-lint-imports
and py27-lint-imports-include-list tox envs
- Fix most E201 and E202 errors reported by flake8-import-order v0.15,
but pin flake8-import-order to v0.14.3 until
https://github.com/PyCQA/flake8-import-order/issues/123
is fixed
This let us drop 2 jobs on Travis per each job.
- Verify a simple data source works when responding via the API.
- Verify a data source sending back a file:// parameter is blocked (GX-2017-0004/GX-2017-0003 - it is tracked different ways in the commit message and in the security issue tracker).
Rather than depend on something in ``functional`` test module from ``base.driver_util``, just set these global hacks up in ``driver_util`` itself. This eliminates a dependency of ``functional`` on ``base`` - ideally ``base`` would not depend on any other test modules.
- Add an API test for datatype-defined composite uploads - including exercising newline conversion and the space_to_tab parameter.
- Add a test decorator skip_without_datatype to mirror skip_without_tool for this test, improve both decorators.
- The ftype parameter in the composite test tools does nothing - drop it and drop it from the XSD spec.
- Slightly improve the documentation for these composite_data elements in the XSD.
In https://github.com/galaxyproject/galaxy/pull/4254 `vcf_bgzip` was made
a proper datatype with the introduction of the `VcfGz` class, but was
still described as a subclass in `datatypes_conf.xml` .
Also fix `file_ext` attribute in `BaseVcf` and subclasses.
Allow override by specifying "provided_metadata_style=" attribute on outputs in tool XML - this should allow newer profile tools to use the older style JSON providing a clear path for upgrades.
without needing to specify a discovered dataset pattern. If the ``discovered_datasets`` tag includes ``from_tool_provided_metadata="true"`` the datsets listed in galaxy.json will just be used directly without needing to be "discovered" using a dataset pattern. Metadata and such can continue to be specified either in that file or on the discovered_dataset element except things like pattern (which is not used) and sort_by (since the json should describe the order I suppose).
- Add another test to demonstrate referencing entries in galaxy.json for datasets by dataset path basename instead of ID.
- Add a bit more documentation to test tools.
- Redo the structure of galaxy.json for profile >= 17.09 tools with two new test tools to demonstrate the new structure - for both output dataset metadata and discovered dataset metadata.
- Improved abstractions for interaction between tools and jobs to enable different tool provided metadata collection depending on tool profile version.
John Chilton