Allows describing hierarchical data in JSON or inferring structure from archives or directories.
Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories.
Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads).
The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render.
This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_libraries.py with one that converts a simple YAML file into an API call and allows many new options for free.
In future PRs I'll add filtering options to this and it will serve as the backend to 4733.
- Move api.helpers to base.populators.
- Refactor helper subclasses adapted to use bioblend instead of test framework stuff into base.populators for greater visibility.
- Refactor workflow format 2 testing stuff for reuse by Selenium.
mvdbeek