Just annotate auto_compressed_types="gz|bz2|gz,bz2" in datatypes_conf.xml and get for free:
- The compressed datatype with correct extension, no need to have a class definition.
- Converters from the compressed datatype to the uncompressed datatype.
- A sniffer in the correct order.
Additionally, the hacky way we were controlling whether types would be sniffed or not has been improved. This can now be set in the datatypes_conf.xml file or read from a global Galaxy configuration option for options that haven't been explicitly set on a per-datatype basis in the configuration (sniff_compressed_dynamic_datatypes_default).
This should simplify all future datatypes moving in this direction and introduces the possibility of also just loading them in dynamically as needed based on what the tools in the tool panel annotate for inputs and outputs. See a tool with an input of type "genbank.gz" should be enough to trigger this someday I hope.
It was already allowed in `test/functional/tools/sample_datatypes_conf.xml`
This will stop the skipping of tests requiring the `velvet` datatype, like
`ToolsUploadTestCase.test_composite_datatype` in
`test/api/test_tools_upload.py` .
We handle other bam types this way and other more specific variants of an extension (e.g. thermo.raw) so this feels like the more consistent thing to do.
The queryname sorted datatype (BamQuerynameSorted) ensures that the file is
queryname ordered. The BamInputSorted datatype can be used to describe the
output of aligners, which usually keep mate pairs adjacent. The BamInputSorted
datatype allows using tools that require mates to be adjacent, without
requiring an explicit sorting step. This can reduce the time and space
required for duplicate marking tools, HiC tools and structural variant
detection tools that require mates to be grouped together in an alignment file.
The BamQuerynameSorted has a converter that works for all datatypes that can be
consumed by samtools. There is no converter for BamInputSorted, since
BamQuerynameSorted is valid input for tools requiring BamInputSorted
input.
I'm targeting 18.01 since I think it would be very unfortunate if we start
annotating aligners with the very loose BamNative datatype, since this datatype
does not make any promises about sort order at all.
This closes https://github.com/galaxyproject/galaxy/issues/5497, goes
a long way towards the problems mentioned in https://github.com/galaxyproject/galaxy/issues/5496
and would make https://github.com/galaxyproject/tools-iuc/pull/1732 and https://github.com/galaxyproject/tools-iuc/pull/1591 better
Helena Rasche