Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
Allows describing hierarchical data in JSON or inferring structure from archives or directories.
Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories.
Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads).
The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render.
This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_libraries.py with one that converts a simple YAML file into an API call and allows many new options for free.
In future PRs I'll add filtering options to this and it will serve as the backend to 4733.
The queryname sorted datatype (BamQuerynameSorted) ensures that the file is
queryname ordered. The BamInputSorted datatype can be used to describe the
output of aligners, which usually keep mate pairs adjacent. The BamInputSorted
datatype allows using tools that require mates to be adjacent, without
requiring an explicit sorting step. This can reduce the time and space
required for duplicate marking tools, HiC tools and structural variant
detection tools that require mates to be grouped together in an alignment file.
The BamQuerynameSorted has a converter that works for all datatypes that can be
consumed by samtools. There is no converter for BamInputSorted, since
BamQuerynameSorted is valid input for tools requiring BamInputSorted
input.
I'm targeting 18.01 since I think it would be very unfortunate if we start
annotating aligners with the very loose BamNative datatype, since this datatype
does not make any promises about sort order at all.
This closes https://github.com/galaxyproject/galaxy/issues/5497, goes
a long way towards the problems mentioned in https://github.com/galaxyproject/galaxy/issues/5496
and would make https://github.com/galaxyproject/tools-iuc/pull/1732 and https://github.com/galaxyproject/tools-iuc/pull/1591 better
Not sure this is the right thing to do (maybe there is a better way to get an ID?), but it avoids
```
IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "tag_name_key"
DETAIL: Key (name)=(group) already exists.
[SQL: 'INSERT INTO tag (type, parent_id, name) VALUES (%(type)s, %(parent_id)s, %(name)s) RETURNING tag.id'] [parameters: {'parent_id': None, 'type': 0, 'name': u'group'}]
galaxy.tools.execute WARNING 2018-02-06 15:22:38,808 [p:2122,w:1,m:0] [uWSGIWorker1Core1] There was a failure executing a job for tool [__TAG_FROM_FILE__] - Error executing tool: (psycopg2.IntegrityError) duplicate key value violates unique constraint "tag_name_key"
```
Given a collection with element_identifiers:
```yml
element_1
element_2
```
we can arbitrarily re-order collection elements using a dataset
that specifies the new order, e.g
```
element_2
element_1
```
I think this nicely complements numeric and alphanumeric sorts that are
already possible with this tool.
It gets added anyway when the file is uploaded to Galaxy due to the
default "Use POSIX standard" (to_posix_lines) option. Many tests which
simply copy them with cat and compare them would fail a strict diff test.
- Better error handling (check for bad characters when creating collections).
- Implement a strict mode parameter to do even more validation.
- Rework tabular vs txt mode to be explicit user choice.
- Mirror fix in release_17.01 for datasets not having a history,
I think it needs to be touched up but the basic operation seems to work so far. I think what remains to be done is:
- Validate uniqueness of identifiers and provide nice messages if they are not unique.
- Validate that at least the required number of lines are present in the file and provide a nice message if not.
- Add strict mode to ensure exactly the correct number of lines is added.
- Find where validation of identifiers happens in the API and apply same validation here - try not to let unsafe identifiers be created.
- Consider more advanced modes - selecting a column, apply a regex replace, pick two columns for nested lists, etc.... None of this may need to be needed in the first iteration.
- Consider another mode where a collection is labelled against an existing collection - should that be a separate tool of the same tool.
Having a BaseFastq class allows Fastq and FastqGz classes to inherit from
BaseFastq. FastqGz is therfor not a Fastq datatype and goes through the
converter.
Also add 2 tools to demonstrate that compressed fastq will be converted to
uncompressed fastq if the tool specifies `format="fastq"`, while
`format="fastq.gz"` leaves files compressed.
The tool test can be run with:
```
planemo test --galaxy_root . test/functional/tools/compressed_fastq_no_conversion.xml
planemo test --galaxy_root . test/functional/tools/compressed_fastq_conversion.xml
```
Note that fastq conversion appears to required a galaxy user session, and
planemo testing fails. Interactive testing with `planemo serve` is not affected
and works fine.
- Rename expected output Log from input Log. This relieves a concern I had that the tests were just working because the inputs and outputs match.
- Add another "unregistered" output file to the output directory and add a test for it. This verifies that such files are infact tested.
- Add a nested "unregistered" output file to the output directory and add a test for it. This verifies that nested files are tested.
John Chilton