Only fetch visible datasets into big, cached list of history datasets under consideration. Hidden datasets don't seem to be used by the fetcher or initial value stuff so it seems fine to exclude them. The advantage should be clear for histories with a large number of datasets hidden below a signficantly smaller number collections.
Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Cleanup up internal representations used to bridge the tool parser and the tool testing modules to produce more sensible dictified representations when exported via the API. This is mostly replacing tuples (which get converted to JSON lists) to dicts but there are some other small cleanups.
Home Directory Handling
-----------------------
- For profile < 18.01 tools:
- If use_shared_home="false" is set on the command block - the tool will be given a clean $HOME directory.
- If ``shared_home_dir`` is set in Galaxy's config or the job destination configuration, Galaxy will set $HOME in the tool's job environment to point at this directory.
- For profile >= 18.01 tools, jobs will be given a clean home directory by default unless ``use_shared_home="true"`` is set. If that is set, the profile < 18.01 behavior is used.
In addition to these changes for Galaxy tools, the tool framework itself has been updated to allow other potential behaviors including the CWL defaults.
Integration tests for each of these cases has been added.
Upgrading to 18.01 - we recommend dropping overriding ``HOME`` in any Galaxy environment configuration (e.g. env directives in job_conf.xml). If any such configuration is found, we recommend instead setting "shared_home_dir" for destinations to the that cluster destination's shared HOME directory. If all destinations share a single HOME directory, this can be set in galaxy.ini instead of job_conf.xml.
Temp Directory Handling
-----------------------
There were serious disagreements with how to proceed here. I felt we should aggressively isolate tool TMP directories and provide deployers options to tweak these through structured arguments. Nicola felt we should defer to the environment variables already being set job_conf.xml - but tweak their meanings by default. Nate fell somewhere in between. I think all sides have merit and could make sense depending on what you want to improve (easing cognative load, reducing out-of-box errors, easing advanced deployer configs, structured reasonsing of paths by Galaxy, etc...).
I've navigated a path here that doesn't particularly reduce out of the box errors as I wanted by setting up a per-job temp directory by default but does make it possible and shouldn't break any existing configurations and doesn't make it anything more difficult to manage these things via environment variables.
The new approach doesn't change any thing by default for Galaxy tools (regardless of tool profile version). The only way to achieve new behaviors is for the deployer to set a job_conf parameter called "tmp_dir". If this is set, it can be set to "True" (to default to a new, clean directory below the job directory) or to a shell expression to allow setting this dynamically at runtime to paths or paths beneath directories as needed on a per-destination basis. If a temp directory is set this way all three variables TMP, TMPDIR, and TEMP are set.
Docker Environment Handling
---------------------------
- Mirror the CWL behavior of mounting an external /tmp by default.
- For newer tools, pass through the HOME, TMP, TMPDIR, and TEMP environment variables into the Docker container in addition to GALAXY_SLOTS.
- Fix the pass through of environment variables into Docker containers.
- Update explicit stdio definitions as well as detect_errors-based definitions to allow this.
- Update runner code to allow resubmission in such cases.
- Add flake8-import-order to flake8 Pipfile and remove py27-lint-imports
and py27-lint-imports-include-list tox envs
- Fix most E201 and E202 errors reported by flake8-import-order v0.15,
but pin flake8-import-order to v0.14.3 until
https://github.com/PyCQA/flake8-import-order/issues/123
is fixed
This let us drop 2 jobs on Travis per each job.
We do this by recording any macros that are required to load a tool. When
loading a tool we register the macro to be watched, and if the macro changes we
reload all corresponding tools.
On a computer where LMOD is not installed, the MODULEPATH variable is not defined and the modulepath attributes of the LMOD dependency resolver must be used.
- Add another test to demonstrate referencing entries in galaxy.json for datasets by dataset path basename instead of ID.
- Add a bit more documentation to test tools.
- Redo the structure of galaxy.json for profile >= 17.09 tools with two new test tools to demonstrate the new structure - for both output dataset metadata and discovered dataset metadata.
- Improved abstractions for interaction between tools and jobs to enable different tool provided metadata collection depending on tool profile version.
Martin Cech