This would affect for example the mapping over of fastq-dump (and all other
tools in the sra-toolkit).
If one had attempted this previously the discovery phase would fail with:
```
galaxy.tools.parameters.output_collect ERROR 2018-01-30 08:56:46,369 Problem gathering output collection.
Traceback (most recent call last):
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 169, in collect_dynamic_collections
collection
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/managers/collections.py", line 216, in collection_builder_for
return builder.BoundCollectionBuilder(dataset_collection, collection_type_description)
File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/dataset_collections/builder.py", line 84, in __init__
raise Exception("Cannot reset elements of an already populated dataset collection.")
Exception: Cannot reset elements of an already populated dataset collection.
```
Instead we force the collection state to be new when we are discovering output collection datasets,
which seems reasonable to me. This includes an API testcase that would have failed previously.
Starting with 16344a6 we attempt to set metadata also for files that have no
content. BAM files need to contain at least a header. To avoid failing tests we
only create the appropriate output based on the input.
Home Directory Handling
-----------------------
- For profile < 18.01 tools:
- If use_shared_home="false" is set on the command block - the tool will be given a clean $HOME directory.
- If ``shared_home_dir`` is set in Galaxy's config or the job destination configuration, Galaxy will set $HOME in the tool's job environment to point at this directory.
- For profile >= 18.01 tools, jobs will be given a clean home directory by default unless ``use_shared_home="true"`` is set. If that is set, the profile < 18.01 behavior is used.
In addition to these changes for Galaxy tools, the tool framework itself has been updated to allow other potential behaviors including the CWL defaults.
Integration tests for each of these cases has been added.
Upgrading to 18.01 - we recommend dropping overriding ``HOME`` in any Galaxy environment configuration (e.g. env directives in job_conf.xml). If any such configuration is found, we recommend instead setting "shared_home_dir" for destinations to the that cluster destination's shared HOME directory. If all destinations share a single HOME directory, this can be set in galaxy.ini instead of job_conf.xml.
Temp Directory Handling
-----------------------
There were serious disagreements with how to proceed here. I felt we should aggressively isolate tool TMP directories and provide deployers options to tweak these through structured arguments. Nicola felt we should defer to the environment variables already being set job_conf.xml - but tweak their meanings by default. Nate fell somewhere in between. I think all sides have merit and could make sense depending on what you want to improve (easing cognative load, reducing out-of-box errors, easing advanced deployer configs, structured reasonsing of paths by Galaxy, etc...).
I've navigated a path here that doesn't particularly reduce out of the box errors as I wanted by setting up a per-job temp directory by default but does make it possible and shouldn't break any existing configurations and doesn't make it anything more difficult to manage these things via environment variables.
The new approach doesn't change any thing by default for Galaxy tools (regardless of tool profile version). The only way to achieve new behaviors is for the deployer to set a job_conf parameter called "tmp_dir". If this is set, it can be set to "True" (to default to a new, clean directory below the job directory) or to a shell expression to allow setting this dynamically at runtime to paths or paths beneath directories as needed on a per-destination basis. If a temp directory is set this way all three variables TMP, TMPDIR, and TEMP are set.
Docker Environment Handling
---------------------------
- Mirror the CWL behavior of mounting an external /tmp by default.
- For newer tools, pass through the HOME, TMP, TMPDIR, and TEMP environment variables into the Docker container in addition to GALAXY_SLOTS.
- Fix the pass through of environment variables into Docker containers.
This test is still failing because `CONVERTER_sam_to_bam` will be invoked instead
of CONVERTER_sam_to_bam_native. Possibly becasue Bam inherits from BamNative, so
this isn't strictly speaking wrong, though not what one would expect.
This specifically addresses the problem where some jobs of a mapped-over
collection have failed. Instead of filtering the failed collection and
restarting the workflow at this position (involving a lot of copy-paste ...)
the user can now limit the rerun to the problematic jobs and the workflow
should resume from there.
Should fix https://github.com/galaxyproject/galaxy/issues/2235.
This is one possible implementation, it would also be feasible to not
manipulate the original collection, but to copy the HDCA and then to replace
collection elements and replace all references for jobs that depend on the HDCA,
as we do for HDAs. This implementation seems simpler, but let me know if you
see problems with this approach.
- Update explicit stdio definitions as well as detect_errors-based definitions to allow this.
- Update runner code to allow resubmission in such cases.
If one maps a tool over an empty collection, one should get an empty output collection for each of the tool's outputs.
This should have always been a valid operation I think, the code just wasn't quite structured right when the collection work was merged - there were too many places I was assuming the existence of at least one actual output or input that was mapped over to continue going. This reworks all of that and seems to handle empty collections appropriately.
Fixes#4025 (not being informing the user of the operation, but actually by allowing the operation - which it should be allowed I think).
Checked through tools-iuc and I couldn't find any tools that use the actual input data parameters in filter statements so I think it is a fairly good approximation to assume that all mapped jobs will either filter or not at least with current execution model.
Increase the timeout - this test fails on Jenkins every now and then and reports the DM job is still queued. I assume this is because of conda_auto_install setting up the requirements for this data manager and the logs seem consistent with that assumption.
I also added data manager json type to the sample so this warning goes away:
```
galaxy.model WARNING 2017-11-30 06:39:59,335 Datatype class not found for extension 'data_manager_json'
```
- Replace test using Cut1 with two tests each using framework test tools loaded with Selenium tests by default.
- Create a symbolic link so that tour generator is configured with default test webhooks.
- Establish and use components and navigates_galaxy.py abstractions for tour generation and navigation.
- In one of the tool tests actually walk the tour (previously we only verified the initial tour popover).
- Add flake8-import-order to flake8 Pipfile and remove py27-lint-imports
and py27-lint-imports-include-list tox envs
- Fix most E201 and E202 errors reported by flake8-import-order v0.15,
but pin flake8-import-order to v0.14.3 until
https://github.com/PyCQA/flake8-import-order/issues/123
is fixed
This let us drop 2 jobs on Travis per each job.
- Verify a simple data source works when responding via the API.
- Verify a data source sending back a file:// parameter is blocked (GX-2017-0004/GX-2017-0003 - it is tracked different ways in the commit message and in the security issue tracker).
John Chilton