Upgrade syntax using `pyupgrade --py36-plus` .
Manually drop several `six` imports.
Also:
- Remove broken pr_cache in scripts/bootstrap_history.py
- Fix broken prefix removal in lib/galaxy/tool_util/deps/mulled/mulled_build.py
Implement both datatype driven composite files and free-style extra files - both implemented in much more hacky ways in current upload.xml API driven endpoint.
*Overview*
This work defines an interface for interacting with "filesystem"-like entities during "upload". In addition to being a pluggable framework adding important new capabilities to Galaxy, this is a generalization and formalization of existing file sources (e.g. the directories described by `library_import_dir`, `user_library_dir`, and `ftp_upload_dir`).
*Plugin Infrastructure*
This introduces a new plugin `FilesSource` to represent sources of directories and files during "upload". A `FilesSource` plugin should be able to index directories and download (called 'realize' to be generic) files to local posix directories. Indexing is used by the remote_files API to provide the client with hierarchies to navigate and to build URIs for the files. The 'realize' operation is used by the 'upload1' and '__DATA_FETCH__' and tools during upload to bring the files into Galaxy as datasets.
An instance of the `ConfiguredFileSources` class is responsible for managing individual instances of `FilesSource` plugins. It has methods to map URIs to the appropriate plugin instance.
The `ConfiguredFileSources` class tracks the loaded plugins and reuses the go to `galaxy.util.plugin_config` module for loading YAML (or XML) definitions of plugins (the same dependency resolvers, job metrics, auth backends, etc. do). A `ConfiguredFileSources` object can serialize itself to a file and re-materialize it during job execution to allow using this abstraction during uploads.
When operating within the Galaxy app, the `ConfiguredFileSources` uses an adapter pattern to parse user-level information from Galaxy's `trans` object. During serialization, the `ConfiguredFileSources` object is expected to encode all the required information about the user that is needed into the output JSON description of the file sources. This is because the web transaction won't be available remotely during the upload job. These objects working in such different ways between the Galaxy process and in the remote job is mildly jarring - so unit tests have been written to ensure this all functions properly.
*Plugin Implementations*
The `FilesSource` interface has a helper implementation base class `BaseFilesSource` that provides some assistance for plugin development. Additionally, the base class `PyFilesystem2FilesSource` extends `BaseFilesSource` but assumes a PyFilesystem2 implementation exists to target the file source of interest - so the plugin author need only provide a PyFilesystem `FS` object describing the target. This commit includes three concrete implementations - posix, webdav, and dropbox. `posix` extends `BaseFilesSource` while the others are light-weight extensions of `PyFilesystem2FilesSource`.
**posix**
While one could imagine a very lightweight implementation based on `PyFilesystem2FilesSource` this fully worked through plugin is implemented directly to ensure we respect Galaxy's strong security checks on paths containing symlinks and preserve the semantics `user_library_import_symlink_allowlist`.
**webdav**
Galaxy tools for integrating OwnCloud exist - see https://github.com/shiltemann/Galaxy-Owncloud-Integration, part of the driver for this work was extending that idea to provide more integrated UX for uploading that data. So this work includes a WebDav plugin (and associated test cases) that could potentially target OwnCloud.
This plugin was a good exercise in flushing and testing the PyFilesystem2 interface but the PyFilesystem2 WebDAV implementation seems a bit fragile... we might want to replace it with more direct APIs but we can take a wait and see approach.
The config YAML for a webdav plugin that lets user's target their own OwnCloud servers configured via user preferences might look something like:
```
- type: webdav
id: owncloud1
label: OwnCloud
doc: User-configured OwnCloud files
url: ${user.preferences['owncloud|url']}
login: ${user.preferences['owncloud|username']}
password: ${user.preferences['webdav|password']}
```
The configuration would provide a user's OwnCloud files at `gxfiles://owncloud1/`.
If instead, a big centralized WebDav server is made available with public data for all users (mirroring use cases of `library_import_dir`) - a simpler configuration not requiring user preferences might be something like:
```
- type: webdav
id: lab
label: Lab WebDAV server
doc: Our lab's research files managed at ourlab.org.
url: http://ourlab.org:7083
login: ${environ.get('WEBDAV_LOGIN')}
password: ${environ.get('WEBDAV_PASSWORD')}
```
The configuration would provide a these WebDAV files at `gxfiles://lab/`.
These two examples demonstrate basic templating is allowed inside the YAML configuration. These are Cheetah templates exposing very specific views of the 'user', 'config', and the whole 'environ' available to the Galaxy server.
**dropbox**
The Dropbox PyFilesystem2 plugin is even easier to configure, all that is needed is a Dropbox access token (this can be configured from the settings menu and may be isolated to a specific app specific folder for added security on the user's part).
An example of such a plugin might be:
```
- type: dropbox
id: dropbox1
label: Dropbox Files
doc: Your Dropbox files - configure an access token via the user preferences
accessToken: ${user.preferences['dropbox|access_token']}
```
The configuration would provide a user's Dropbox files at `gxfiles://dropbox1/`.
**gxftp**
This is an automatically populated plugin (if `ftp_upload_dir` is configured in Galaxy) that provides the user's FTP files at `gxftp://`.
**gximport**
This is an automatically populated plugin (if `library_import_dir` is configured in Galaxy) that provides Galaxy's library import files at `gximport://`.
**gxuserimport**
This is an automatically populated plugin (if `user_library_import_dir` is configured in Galaxy) that provides the requesting user's Galaxy's user library import files at `gximportfiles://`.
*Why not a tool?*
One could imagine a tool - but the upload dialog has many advanced options for selecting how to ingest files (convert tabs and newlines, select format vs. detect, select dbkey, organize into collections, organize via rules, etc...). It would be next to impossible to provide all these same options via a normal tool and the user experience would be very different than using the upload components in Galaxy - which have been optimized and designed for this task.
That said - one future direction I would like to take this is to be able to mark plugins as writable and implement a new tool form input type "export_directory" or something like that. This could then be used to write data export tools. This could be used to write generalizations of the the cloud send tool.
*`ObjectStore` vs `FilesSource`*
ObjectStores provide datasets not files, the files are organized logically in a very flat way around a dataset. `FilesSource` s instead provide files and directories, not datasets. A `FilesSource` is meant to be browsed in hierarchical fashion - and also has no concept of extra files, etc..
*Future Work*
- This is hopefully going to serve as the basis of a first pass at Terra integration with Galaxy using the FISS lib. Having an implementation based on `PyFilesytem2` means we could potentially integrate support for S3, Basespace, Google Drive, OneDrive, etc..
- Tool form support for selecting files for import and directories for export.
- Allow writing collection archives, history export, etc.. to the `FilesSource` - this would really enhance the UI around getting big stuff out of Galaxy potentially I think.
Rebase into galaxy.files...
Prior to this, composite uploads were only allowed for select data types. As far as I can tell, the framework itself allows arbitrary datatypes to have extra files. This allows the upload API to take in extra files for datatypes that aren't explicitly annotated has having extra files.
These may be specified one at a time or in directory structures via tar files.
`logging.exception()` already adds the the exception information after
the supplied message.
Also: the `message` attribute of `Exception` class has been dropped in
Python 3, always use `str(e)`.
xref. https://github.com/galaxyproject/galaxy/issues/1715
This does undo an attempt that Nate made to auto sniff fastq.gz files when they can't be converted anyway, but that was kind of broken in its own way.
From Gitter:
jmchilton: natefoo "# Link mode can't decompress anyway, so enable sniffing for keep-compressed datatypes even when auto_decompress is enabled" Did you test this hack? I get the thought process but it seems like it results in uploaded fastq.gz files defaulting to fastqcssanger.gz? Maybe it didn't at some point though?
Nate Coraor: ummmmmmmmmmmm maybe probably not
John Chilton: I have four fixes for compressed datatypes that all fall apart because of this hack... can I just use the configured sniff order or do you want me to try to preserve this and write a test case
Nate Coraor: nah go ahead and de-hack
John Chilton: There is a good argument to be made for making the behavior more consistent anyway right? Cool thanks
Previously sniffing would happen on the original file (before carriage returns and tabular spaces were converted) if in_place was false and on the converted file if it was true.
Allows describing hierarchical data in JSON or inferring structure from archives or directories.
Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories.
Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads).
The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render.
This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_libraries.py with one that converts a simple YAML file into an API call and allows many new options for free.
In future PRs I'll add filtering options to this and it will serve as the backend to 4733.
I like this better for three reasons:
- Since usually it is scripts producing this JSON - we have the most control at that point for determining the failure and we don't have to deal with an artificial dependency between the tool's stdio and the output.
- At some point we could potentially allow some datasets to be ok now even though the job fails.
- It is a cleaner interface at the Python level between job finish and output collection IMO (no need for isinstance checking).
I think setting each of these variables once and simplifing the context they are used in (in the case of purge_upload) makes it more clear what each variable is and how it is set. I also think one, more detailed comment for each variable helps.
Note: This will break run-as-user uploads started prior to the upgrade to 18.XX and executed after the upgrade. It is a small switch to restore the old behavior but I'm not sure it is worth the complexity it adds to the file.
```
run_as_real_user = dataset.get('run_as_real_user', False) or dataset_get.('in_place', True)
```
Remove the need to call `Binary.register_unsniffable_binary_ext()` for
each unsniffable binary datatype.
Fix https://github.com/galaxyproject/galaxy/issues/3441 , where the upload
of files of a datatype defined in datatypes_conf.xml as subclass of an
unsniffable binary datatype ended up with "The uploaded binary file
contains inappropriate content" because it was not possible to register the
subclassed datatype as unsniffable.
Also remove unused `stop_err()` function in upload.py .
Also, when sniffing binary files, sniff images together with the other
formats and respect sniff order.
Also remove `is_multi_byte` from:
- `stream_to_open_named_file()`
- `stream_to_file()`
Remove the now unused `get_image_ext()` and `Binary.is_sniffable_binary()`
and all the calls to `Binary.register_sniffable_binary_format()`.
Also, merge its 3 parameters `gzip_only`, `bz2_only`, `zip_only` into
`compressed_formats` (a list of allowed formats).
As a consequence of the changes in `get_fileobj()`, update:
- `files_diff()`
- `get_file_peek()`, which now determines that a file is binary when a
`UnicodeDecodeError` exception is raised and doesn't need
`is_multi_byte` any more
- `iter_headers()` and `get_headers()`, which now return Unicode and don't
need `is_multi_byte` parameter any more
As a consequence of the changes in `get_file_peek()`, update:
- `set_peek()`, which now doesn't need `is_multi_byte` any more
As a consequence of the changes in `get_headers()`, update:
- `guess_ext` and `is_column_based()`, which now determine that a file is
binary when a `UnicodeDecodeError` exception is raised and don't need
`is_multi_byte` any more
As a consequence of the changes to `guess_ext`, update:
- `handle_uploaded_dataset_file() doesn't need `is_multi_byte` any more
Also, remove duplicated calls to `get_file_peek()` in
lib/galaxy/datatypes/molecules.py and lib/galaxy/datatypes/msa.py
The `is_multi_byte` was not removed from the signature of `get_file_peek()`
and `set_peek()` in order to preserve compatibility for ToolShed datatypes,
thanks @jmchilton for the review.
mvdbeek