- Require a URL or FTP column for raw pasted data.
- Require list identifiers if creating collections.
- Require name in more scenerios that require a name.
A mistake was made in the rush to rule-builder-all-the-stuffs - libraries, hdas, and FTP all initialized a different set of columns from the data (which is probably fine) but with no indication this was happening in the underlying rules (which is definitely not fine).
To understand why this was a mistake and see why in the long term this hurts reproducibility, take loading rules for HDAs from the history panel as an example. In the initial implementation it would pre-initialize two columns - one for "hid" and one for "name" - any rules that were generated from there would start with an implicit assumption of two columns. This is a problem because if someone saves some rules and we (Galaxy developers) decide later on that name tag should be in the initial list but HID doesn't make sense and make that change - we basically are breaking all existing rules to support a very logical change in behavior. Likewise, Galaxy produces a hash with last modified date and size for each FTP entry - this is data we have and could stick in the rule builder by default - but if we change that over time existing FTP rules will break.
It would be much more sustainable to instead include the initial column generation information in the initial rules generated - so that each of these modalities start in reality with an empty table and the initial set of rules describe what metadata to pull initial columns from. This way if you save a set of rules it will include the initial column choices, and if they are pasted in at some later point where columns are picked differently - the rules will still work because you are replacing the initial column definitions.
The initial rules implementation didn't have this concept of loading columns from metadata - but this was added to support loading existing identifiers as part of the "Apply Rules to Existing Collection" tool. This commit extends that idea to do the same thing (but with different columns) when loading data from HDAs, FTP, and data libraries.
Present users with the job standard error and also check for binary sniffing problems and report them with a pretty message suggesting specifying a file type.
Resolving the default when unspecified in the XML before writing out datatype confs fixes this. This makes this option always explicit in the datatypes conf that is shared with the upload tool.
This does undo an attempt that Nate made to auto sniff fastq.gz files when they can't be converted anyway, but that was kind of broken in its own way.
From Gitter:
jmchilton: natefoo "# Link mode can't decompress anyway, so enable sniffing for keep-compressed datatypes even when auto_decompress is enabled" Did you test this hack? I get the thought process but it seems like it results in uploaded fastq.gz files defaulting to fastqcssanger.gz? Maybe it didn't at some point though?
Nate Coraor: ummmmmmmmmmmm maybe probably not
John Chilton: I have four fixes for compressed datatypes that all fall apart because of this hack... can I just use the configured sniff order or do you want me to try to preserve this and write a test case
Nate Coraor: nah go ahead and de-hack
John Chilton: There is a good argument to be made for making the behavior more consistent anyway right? Cool thanks
- Incorrect parsing of default auto sniff flag value.
- Not respecting flag value when adding to dict of compressed sniffers.
- Not checking the right value when checking for compressed datatype instances to add to sniff order.
- Not filtering out the sniffers from being automatically added during the default stage of sniffer handling where all types with sniff are added at the end.
This last point is the most complicated. I added a hack with #6091 so that the compressed sniffers would only return True if in validation mode or explicitly enabled. I'm not convinced that approached worked and I'm definitely convinced it was a bad approach.
The better, cleaner approach is to just always let the sniffer return its truth regardless of validation mode or not and instead only add it to the list of sniffers if it should be enabled. This is what I was assuming would happen but I found this method append_to_sniff_order that was adding sniffers back in that I didn't want added.
Dannon