Fix `pysam.index()` calls which now use samtools 1.16 where
`samtools index input output` doesn't work any more if `output` is the path
of an already existing file, but `samtools index -o output input` works
fine.
Also:
- Pre-create the (empty) index file for mock test datasets as a unit test
reproducer of the above issue.
- Fix `test_cram` unit test that wasn't checking the return value of
`CRAM.set_index_file()`, which was in fact returning False because
`test-data/2.cram` was not sorted (updated now).
Somewhere between file/libmagic 5.36 and 5.39, the output of `file` on a
`.csi` file changed, breaking the ``metadata_bcf`` framework test with
the following error:
```
if found_exceptions:
> raise JobOutputsError(found_exceptions, job_stdio)
E galaxy.tool_util.verify.interactor.JobOutputsError: Output output_of_input_metadata: different than expected, difference (using contains):
E ( /tmp/tmpu1a3dgjcbcf_index_metadata_test.txt v. /tmp/tmpkt78tzggbcf_index_metadata_test.txt )
E Failed to find 'gzip compressed data, extra field' in history data. (lines_diff=0):
lib/galaxy/tool_util/verify/interactor.py:1222: JobOutputsError
```
So add a requirement for the latest `file` version and update the test
accordingly.
dont compare the json file content but rely completely on the assertions on
the loaded json contents.
the json content test was superficial and did not work anymore since the order
of the keys differed. it worked before since the inputs contained only a
single key.
- in particular the case of an empty select did yield "None" before
now its []
Problem was that the ValueWrapper is not None if nothing is selected
- but also non empty selects were "option1,option2" now its
["option1","option2"]
- renamed variable value -> value_wrapper
Demonstrates using the Apply Rules collection operation tool against existing collections to:
- Use grouping to build nested lists from flat ones.
- Invert nested collections.
- Filter collections.
- Filter while also grouping to build nested collections.
Based on count data used in the DESeq2 tutorial (http://bioconductor.org/packages/devel/bioc/vignettes/DESeq2/inst/doc/DESeq2.html), originally from the Bioconductor Pasilla package (http://bioconductor.org/packages/release/data/experiment/html/pasilla.html). Citation: Huber W, Reyes A (2018). pasilla: Data package with per-exon and per-gene read counts of RNA-seq samples of Pasilla knock-down by Brooks et al., Genome Research 2011.. R package version 1.8.0.
Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
Allows describing hierarchical data in JSON or inferring structure from archives or directories.
Datasets or archive sources can be specified via uploads, URLs, paths (if admin && allow_path_paste), library_import_dir/user_library_import_dir, and/or FTP imports. Unlike existing API endpoints, a mix of these on a per file basis is allowed and they work seemlessly between libraries and histories.
Supported "archives" include gzip, zip, bagit directories, bagit achives (with fetching and validations of downloads).
The existing upload API endpoint is quite rough to work with both in terms of adding parameters (e.g. the file type and dbkey hanlding in 4563 was difficult to implement, terribly hacky, and should seemingly have been trivial) and in terms of building requests (one needs to build a tool form - not describe sensible inputs in JSON). This API is built to be intelligable from an API standpoint instead of being constrained to the older style tool form. Additionally it built with hierarchical data in mind in a way that would not be easy at all enhancing the tool form components we don't even render.
This implements 5159 though much simpler YAML descriptions of data libraries should be possible basically as the API descriptions. We can replace the data library script in Ephemeris https://github.com/galaxyproject/ephemeris/blob/master/ephemeris/setup_data_libraries.py with one that converts a simple YAML file into an API call and allows many new options for free.
In future PRs I'll add filtering options to this and it will serve as the backend to 4733.
The queryname sorted datatype (BamQuerynameSorted) ensures that the file is
queryname ordered. The BamInputSorted datatype can be used to describe the
output of aligners, which usually keep mate pairs adjacent. The BamInputSorted
datatype allows using tools that require mates to be adjacent, without
requiring an explicit sorting step. This can reduce the time and space
required for duplicate marking tools, HiC tools and structural variant
detection tools that require mates to be grouped together in an alignment file.
The BamQuerynameSorted has a converter that works for all datatypes that can be
consumed by samtools. There is no converter for BamInputSorted, since
BamQuerynameSorted is valid input for tools requiring BamInputSorted
input.
I'm targeting 18.01 since I think it would be very unfortunate if we start
annotating aligners with the very loose BamNative datatype, since this datatype
does not make any promises about sort order at all.
This closes https://github.com/galaxyproject/galaxy/issues/5497, goes
a long way towards the problems mentioned in https://github.com/galaxyproject/galaxy/issues/5496
and would make https://github.com/galaxyproject/tools-iuc/pull/1732 and https://github.com/galaxyproject/tools-iuc/pull/1591 better
Not sure this is the right thing to do (maybe there is a better way to get an ID?), but it avoids
```
IntegrityError: (psycopg2.IntegrityError) duplicate key value violates unique constraint "tag_name_key"
DETAIL: Key (name)=(group) already exists.
[SQL: 'INSERT INTO tag (type, parent_id, name) VALUES (%(type)s, %(parent_id)s, %(name)s) RETURNING tag.id'] [parameters: {'parent_id': None, 'type': 0, 'name': u'group'}]
galaxy.tools.execute WARNING 2018-02-06 15:22:38,808 [p:2122,w:1,m:0] [uWSGIWorker1Core1] There was a failure executing a job for tool [__TAG_FROM_FILE__] - Error executing tool: (psycopg2.IntegrityError) duplicate key value violates unique constraint "tag_name_key"
```
John Chilton