The included script takes in an object store definition to write to, model store output target (e.g. tar file or directory or bagit file/directory format matching the new history exports added in 7367), and a bunch of files in defined via an upload-2.0-style YAML - and writes the physical files into the object store and writes persisted metadata into the model store. The supplied script describes the input format as do the new unit tests in test_model_discover.py.
Right now datasets for a history and whole libraries (including recursive folders and library datasets) can be described this way. These concepts have been extended to collections in #7058 but only in the context of discovering job outputs and writing them to an export model store current.
This script was created mostly by refactoring stuff out of output_collect.py that is used to create datasets during output discovery (i.e. parsing the same data out of galaxy.json files).
- Allow import and export collections.
- Fix job I/O for separated tool and job script streams (merged in #7095 during 19.05)
- Fix job state handling so that all imported jobs have terminal state.
- Fixes to eliminate HID handling for associations (objects were previously tracked by HID but inputs/outputs in history may not be in history - so HID is not at all appropriate, problem was amplified by collections).
- Fix job output mapping - was using the HDA name instead of job output name during import (old exports don't have this info - only new ones do).
- Fixes assigning HIDs to datasets during import and fixes to hid_counter - in many ways this was very broken for imported histories.
- Introduce cohesive model object import/export abstractions including a lot of new, well tested functionality.
- Allow import/export of datasets and collections stand-alone regardless of history.
- Allow export and re-import of datasets and collections (so for instance datasets can be created for a job and read in at the end of a job and merged back into the database). This is needed for extended metadata collection.
- Separate implementation for import abstraction that allow import of legacy archives without complicating logic for current archive format.
- Implementation of import and export abstraction for [BagIt](https://datatracker.ietf.org/doc/rfc8493/) using [bdbag](https://github.com/fair-research/bdbag).
- Context manager support and clean APIs for all of this including import and export of directories, archives, and bags.
- A lot more unit testing.
mvdbeek