This fixes the following traceback:
```
( unicode_stream ) > Test-1 ... galaxy.tools DEBUG 2018-08-21 09:40:59,052 Validated and populated state for tool request (0.098 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,076 Handled output named out_file1 for tool unicode_stream (1.337 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,080 Added output datasets to history (4.399 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,084 Verified access to datasets for Job[unflushed,tool_id=unicode_stream] (0.058 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,086 Setup for job Job[unflushed,tool_id=unicode_stream] complete, ready to flush (5.220 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,127 Flushed transaction for job Job[id=354,tool_id=unicode_stream] (40.627 ms)
galaxy.tools.execute DEBUG 2018-08-21 09:40:59,127 Tool [unicode_stream] created job [354] (56.506 ms)
galaxy.tools.execute DEBUG 2018-08-21 09:40:59,132 Executed 1 job(s) for tool unicode_stream request: (79.434 ms)
galaxy.jobs.mapper DEBUG 2018-08-21 09:40:59,556 (354) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2018-08-21 09:40:59,561 (354) Dispatching to local runner
galaxy.jobs DEBUG 2018-08-21 09:40:59,571 (354) Working directory for job is: /tmp/tmp3drfmr6x/tmp_ywd33ar/tmpbzo4a1d9/database/job_working_directory_azry9a1y/000/354
galaxy.jobs DEBUG 2018-08-21 09:40:59,574 (354) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-08-21 09:40:59,589 Job [354] queued (27.361 ms)
galaxy.jobs.handler INFO 2018-08-21 09:40:59,604 (354) Job dispatched
galaxy.jobs.runners ERROR 2018-08-21 09:40:59,715 (354) Failure preparing job
Traceback (most recent call last):
File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 187, in prepare_job
job_wrapper.prepare()
File "/galaxy/lib/galaxy/jobs/__init__.py", line 874, in prepare
self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build()
File "/galaxy/lib/galaxy/tools/evaluation.py", line 435, in build
raise e
File "/galaxy/lib/galaxy/tools/evaluation.py", line 431, in build
self.__build_config_files()
File "/galaxy/lib/galaxy/tools/evaluation.py", line 504, in __build_config_files
self.__write_workdir_file(config_filename, config_text, param_dict, is_template=is_template)
File "/galaxy/lib/galaxy/tools/evaluation.py", line 578, in __write_workdir_file
f.write(value)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)
galaxy.tools.error_reports DEBUG 2018-08-21 09:40:59,869 Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7f92efb81668> generated response None
FAIL
```
which is due to the fact that the default encoding in the Docker image
is `ANSI_X3.4-1968`.
Also update `unicode_stream` test tool to IUC best practices.
- Switch select_tag to group_tag as a parameter name - since it is restricted to group_tags and we don't call data parameters select_data.
- Add minimal XSD documentation for this new parameter type.
- Add example tools - for single and multiple tags.
- Add API tests for these tools and the tag handling.
This doesn't quite work because of the dynamic output collection,
which fails with:
```
galaxy.tools.parameters.output_collect DEBUG 2018-06-06 18:40:38,784 (3) Add dynamic collection datasets to history for output [reverse] (171.970 ms)
galaxy.tools.parameters.output_collect ERROR 2018-06-06 18:40:38,839 Problem gathering output collection.
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 340, in collect_dynamic_outputs
collection_builder.populate()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 88, in populate
elements = self.build_elements()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 59, in build_elements
new_elements[identifier] = element.build()
AttributeError: 'HistoryDatasetAssociation' object has no attribute 'build'
```
This seems to happen because the inner collection is wrongly detected as
nested. In general I doubt that mapping over colleciton output works
with dynamically discovered output collections.
This doesn't quite work because of the dynamic output collection,
which fails with:
```
galaxy.tools.parameters.output_collect DEBUG 2018-06-06 18:40:38,784 (3) Add dynamic collection datasets to history for output [reverse] (171.970 ms)
galaxy.tools.parameters.output_collect ERROR 2018-06-06 18:40:38,839 Problem gathering output collection.
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 340, in collect_dynamic_outputs
collection_builder.populate()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 88, in populate
elements = self.build_elements()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 59, in build_elements
new_elements[identifier] = element.build()
AttributeError: 'HistoryDatasetAssociation' object has no attribute 'build'
```
This seems to happen because the inner collection is wrongly detected as
nested. In general I doubt that mapping over colleciton output works
with dynamically discovered output collections.
Update tool XSD to encourage using fully qualified inputs.
Reorganized the existing tests for consistency with other test tools I think.
(Rebased with fixes, including XSD fixes from @nsoranzo.)
Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Just annotate auto_compressed_types="gz|bz2|gz,bz2" in datatypes_conf.xml and get for free:
- The compressed datatype with correct extension, no need to have a class definition.
- Converters from the compressed datatype to the uncompressed datatype.
- A sniffer in the correct order.
Additionally, the hacky way we were controlling whether types would be sniffed or not has been improved. This can now be set in the datatypes_conf.xml file or read from a global Galaxy configuration option for options that haven't been explicitly set on a per-datatype basis in the configuration (sniff_compressed_dynamic_datatypes_default).
This should simplify all future datatypes moving in this direction and introduces the possibility of also just loading them in dynamically as needed based on what the tools in the tool panel annotate for inputs and outputs. See a tool with an input of type "genbank.gz" should be enough to trigger this someday I hope.
Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
We handle other bam types this way and other more specific variants of an extension (e.g. thermo.raw) so this feels like the more consistent thing to do.
This test correctly fails with:
```
galaxy.jobs.runners.local ERROR 2018-03-02 17:55:32,396 Job wrapper finish method failed
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/jobs/runners/local.py", line 152, in queue_job
self._finish_or_resubmit_job(job_state, stdout, stderr, exit_code)
File "/Users/mvandenb/src/galaxy/lib/galaxy/jobs/runners/__init__.py", line 432, in _finish_or_resubmit_job
job_state.job_wrapper.finish(stdout, stderr, exit_code, check_output_detected_state=check_output_detected_state)
File "/Users/mvandenb/src/galaxy/lib/galaxy/jobs/__init__.py", line 1381, in finish
'primary': self.tool.collect_primary_datasets(out_data, self.get_tool_provided_job_metadata(), tool_working_directory, input_ext, input_dbkey)
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/__init__.py", line 1614, in collect_primary_datasets
return output_collect.collect_primary_datasets(self, output, tool_provided_metadata, job_working_directory, input_ext, input_dbkey=input_dbkey)
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 372, in collect_primary_datasets
for discovered_file in discover_files(name, tool_provided_metadata, dataset_collectors, job_working_directory, outdata):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 490, in discover_files
for (match, collector) in walk_over_file_collectors(extra_file_collectors, job_working_directory, matchable):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 507, in walk_over_file_collectors
for match in walk_over_extra_files(extra_file_collector.directory, extra_file_collector, job_working_directory, matchable):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 519, in walk_over_extra_files
for filename in os.listdir(directory):
OSError: [Errno 2] No such file or directory: '/private/var/folders/df/6xqpqpcd7h73b6jpx9t6cwhw0000gn/T/tmp5Tz0m9/tmpfUHHl6/tmpVBcnfI/database/job_working_directory_LJioiW/000/2/working/i_do_not_exist'
```
The queryname sorted datatype (BamQuerynameSorted) ensures that the file is
queryname ordered. The BamInputSorted datatype can be used to describe the
output of aligners, which usually keep mate pairs adjacent. The BamInputSorted
datatype allows using tools that require mates to be adjacent, without
requiring an explicit sorting step. This can reduce the time and space
required for duplicate marking tools, HiC tools and structural variant
detection tools that require mates to be grouped together in an alignment file.
The BamQuerynameSorted has a converter that works for all datatypes that can be
consumed by samtools. There is no converter for BamInputSorted, since
BamQuerynameSorted is valid input for tools requiring BamInputSorted
input.
I'm targeting 18.01 since I think it would be very unfortunate if we start
annotating aligners with the very loose BamNative datatype, since this datatype
does not make any promises about sort order at all.
This closes https://github.com/galaxyproject/galaxy/issues/5497, goes
a long way towards the problems mentioned in https://github.com/galaxyproject/galaxy/issues/5496
and would make https://github.com/galaxyproject/tools-iuc/pull/1732 and https://github.com/galaxyproject/tools-iuc/pull/1591 better
Marius van den Beek