We're now just copying the current python path. Also the handler names
have actually changed, I think this hasn't been updated after a late
change (it is now as it should be, based on the documentation).
Setup a hierarchical objectstore with ids - these ids can be zero weight for objectstores that should only be used on a per-job-destination basis. Job destinations can then set object_store_id as a param to force a particular object store from the hierarchical configuration to be used. This can be set in job_conf.xml directly for static mappings or combined with dynamic job destinations for dynamic mapping to object stores. Dynamic job destinations can dispatch on cluster conditions, user, user preferences, etc.. Additionally, job resource parameters can be used with dynamic job destinations to allow user input on the object store to use.
An integration test case is included that tests and demonstrates all these different pieces. It sets up a Galaxy with a job and object store configuration that routes one tool to a specific disk store (called "static"), and routes another tool dynamically using job resource parameters that allow the user to pick between "slow, cheap" and "fast, expensive" storage options (for the test it is all just disk stores used - but you can imagine choosing between EBS and S3 or something this way).
I'd be happy to see something more formal than job resource parameters or user preferences developed to help users route job outputs to particular object stores - this is just a demonstration of what can be done today after these changes using extension points already being used by admins and users.
- Test hierarchical store uses its first disk store for new files and not the remaining entries.
- Test basic functionality of writing to the distributed object store.
- Test all job outputs get placed in the same object store.
Trying to improve the user experience of this rule based uploader by placing HDAs and HDCAs in the history at the outset that the history panel can poll and that we can turn red if the upload fails.
From Marius' PR review:
> I can see that a job launched in my logs, but it failed and there were no visual indications of this in the UI
Not every HDA for instance can be created, for example if reading them from a zip file for instance that happens on the backend still. Likewise if HDCAs don't define a collection type up front they cannot be pre-created (if for instance that is inferred from a folder structure). Library things aren't precreated at all in this commit. There is room to pre-create more but I think this is an atomic commit as it is now and it will hopefully improve the user experience for the rule based uploader considerably.
Allows describing hierarchical data in JSON or inferring structure from archives or directories.
Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories.
Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads).
The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render.
This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_libraries.py with one that converts a simple YAML file into an API call and allows many new options for free.
In future PRs I'll add filtering options to this and it will serve as the backend to 4733.
Home Directory Handling
-----------------------
- For profile < 18.01 tools:
- If use_shared_home="false" is set on the command block - the tool will be given a clean $HOME directory.
- If ``shared_home_dir`` is set in Galaxy's config or the job destination configuration, Galaxy will set $HOME in the tool's job environment to point at this directory.
- For profile >= 18.01 tools, jobs will be given a clean home directory by default unless ``use_shared_home="true"`` is set. If that is set, the profile < 18.01 behavior is used.
In addition to these changes for Galaxy tools, the tool framework itself has been updated to allow other potential behaviors including the CWL defaults.
Integration tests for each of these cases has been added.
Upgrading to 18.01 - we recommend dropping overriding ``HOME`` in any Galaxy environment configuration (e.g. env directives in job_conf.xml). If any such configuration is found, we recommend instead setting "shared_home_dir" for destinations to the that cluster destination's shared HOME directory. If all destinations share a single HOME directory, this can be set in galaxy.ini instead of job_conf.xml.
Temp Directory Handling
-----------------------
There were serious disagreements with how to proceed here. I felt we should aggressively isolate tool TMP directories and provide deployers options to tweak these through structured arguments. Nicola felt we should defer to the environment variables already being set job_conf.xml - but tweak their meanings by default. Nate fell somewhere in between. I think all sides have merit and could make sense depending on what you want to improve (easing cognative load, reducing out-of-box errors, easing advanced deployer configs, structured reasonsing of paths by Galaxy, etc...).
I've navigated a path here that doesn't particularly reduce out of the box errors as I wanted by setting up a per-job temp directory by default but does make it possible and shouldn't break any existing configurations and doesn't make it anything more difficult to manage these things via environment variables.
The new approach doesn't change any thing by default for Galaxy tools (regardless of tool profile version). The only way to achieve new behaviors is for the deployer to set a job_conf parameter called "tmp_dir". If this is set, it can be set to "True" (to default to a new, clean directory below the job directory) or to a shell expression to allow setting this dynamically at runtime to paths or paths beneath directories as needed on a per-destination basis. If a temp directory is set this way all three variables TMP, TMPDIR, and TEMP are set.
Docker Environment Handling
---------------------------
- Mirror the CWL behavior of mounting an external /tmp by default.
- For newer tools, pass through the HOME, TMP, TMPDIR, and TEMP environment variables into the Docker container in addition to GALAXY_SLOTS.
- Fix the pass through of environment variables into Docker containers.
- Update explicit stdio definitions as well as detect_errors-based definitions to allow this.
- Update runner code to allow resubmission in such cases.
Nicola Soranzo