This fixes the following traceback:
```
( unicode_stream ) > Test-1 ... galaxy.tools DEBUG 2018-08-21 09:40:59,052 Validated and populated state for tool request (0.098 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,076 Handled output named out_file1 for tool unicode_stream (1.337 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,080 Added output datasets to history (4.399 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,084 Verified access to datasets for Job[unflushed,tool_id=unicode_stream] (0.058 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,086 Setup for job Job[unflushed,tool_id=unicode_stream] complete, ready to flush (5.220 ms)
galaxy.tools.actions INFO 2018-08-21 09:40:59,127 Flushed transaction for job Job[id=354,tool_id=unicode_stream] (40.627 ms)
galaxy.tools.execute DEBUG 2018-08-21 09:40:59,127 Tool [unicode_stream] created job [354] (56.506 ms)
galaxy.tools.execute DEBUG 2018-08-21 09:40:59,132 Executed 1 job(s) for tool unicode_stream request: (79.434 ms)
galaxy.jobs.mapper DEBUG 2018-08-21 09:40:59,556 (354) Mapped job to destination id: local
galaxy.jobs.handler DEBUG 2018-08-21 09:40:59,561 (354) Dispatching to local runner
galaxy.jobs DEBUG 2018-08-21 09:40:59,571 (354) Working directory for job is: /tmp/tmp3drfmr6x/tmp_ywd33ar/tmpbzo4a1d9/database/job_working_directory_azry9a1y/000/354
galaxy.jobs DEBUG 2018-08-21 09:40:59,574 (354) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-08-21 09:40:59,589 Job [354] queued (27.361 ms)
galaxy.jobs.handler INFO 2018-08-21 09:40:59,604 (354) Job dispatched
galaxy.jobs.runners ERROR 2018-08-21 09:40:59,715 (354) Failure preparing job
Traceback (most recent call last):
File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 187, in prepare_job
job_wrapper.prepare()
File "/galaxy/lib/galaxy/jobs/__init__.py", line 874, in prepare
self.command_line, self.extra_filenames, self.environment_variables = tool_evaluator.build()
File "/galaxy/lib/galaxy/tools/evaluation.py", line 435, in build
raise e
File "/galaxy/lib/galaxy/tools/evaluation.py", line 431, in build
self.__build_config_files()
File "/galaxy/lib/galaxy/tools/evaluation.py", line 504, in __build_config_files
self.__write_workdir_file(config_filename, config_text, param_dict, is_template=is_template)
File "/galaxy/lib/galaxy/tools/evaluation.py", line 578, in __write_workdir_file
f.write(value)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)
galaxy.tools.error_reports DEBUG 2018-08-21 09:40:59,869 Bug report plugin <galaxy.tools.error_reports.plugins.sentry.SentryPlugin object at 0x7f92efb81668> generated response None
FAIL
```
which is due to the fact that the default encoding in the Docker image
is `ANSI_X3.4-1968`.
Also update `unicode_stream` test tool to IUC best practices.
This is pretty hackish as fixtures can't be used during test discovery.
To properly shut down the driver I've added a conftest.py file that will
be called when quitting pytest. We might be able to handle this a
little better if we wrote a full plugin, but for now this works.
Also now reports all test indexes, so you can select e.g test 2 of a
tool.
We don't rely on TwillTestCase anymore, that is now exclusively used by
ShedTwillTestCase. Then I've just re-used the infrastructure of
ToolsTestCase for discovering tests, while making sure that we do use an
admin API key during testing.
- Switch select_tag to group_tag as a parameter name - since it is restricted to group_tags and we don't call data parameters select_data.
- Add minimal XSD documentation for this new parameter type.
- Add example tools - for single and multiple tags.
- Add API tests for these tools and the tag handling.
This doesn't quite work because of the dynamic output collection,
which fails with:
```
galaxy.tools.parameters.output_collect DEBUG 2018-06-06 18:40:38,784 (3) Add dynamic collection datasets to history for output [reverse] (171.970 ms)
galaxy.tools.parameters.output_collect ERROR 2018-06-06 18:40:38,839 Problem gathering output collection.
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 340, in collect_dynamic_outputs
collection_builder.populate()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 88, in populate
elements = self.build_elements()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 59, in build_elements
new_elements[identifier] = element.build()
AttributeError: 'HistoryDatasetAssociation' object has no attribute 'build'
```
This seems to happen because the inner collection is wrongly detected as
nested. In general I doubt that mapping over colleciton output works
with dynamically discovered output collections.
This doesn't quite work because of the dynamic output collection,
which fails with:
```
galaxy.tools.parameters.output_collect DEBUG 2018-06-06 18:40:38,784 (3) Add dynamic collection datasets to history for output [reverse] (171.970 ms)
galaxy.tools.parameters.output_collect ERROR 2018-06-06 18:40:38,839 Problem gathering output collection.
Traceback (most recent call last):
File "/Users/mvandenb/src/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 340, in collect_dynamic_outputs
collection_builder.populate()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 88, in populate
elements = self.build_elements()
File "/Users/mvandenb/src/galaxy/lib/galaxy/dataset_collections/builder.py", line 59, in build_elements
new_elements[identifier] = element.build()
AttributeError: 'HistoryDatasetAssociation' object has no attribute 'build'
```
This seems to happen because the inner collection is wrongly detected as
nested. In general I doubt that mapping over colleciton output works
with dynamically discovered output collections.
Update tool XSD to encourage using fully qualified inputs.
Reorganized the existing tests for consistency with other test tools I think.
(Rebased with fixes, including XSD fixes from @nsoranzo.)
Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Just annotate auto_compressed_types="gz|bz2|gz,bz2" in datatypes_conf.xml and get for free:
- The compressed datatype with correct extension, no need to have a class definition.
- Converters from the compressed datatype to the uncompressed datatype.
- A sniffer in the correct order.
Additionally, the hacky way we were controlling whether types would be sniffed or not has been improved. This can now be set in the datatypes_conf.xml file or read from a global Galaxy configuration option for options that haven't been explicitly set on a per-datatype basis in the configuration (sniff_compressed_dynamic_datatypes_default).
This should simplify all future datatypes moving in this direction and introduces the possibility of also just loading them in dynamically as needed based on what the tools in the tool panel annotate for inputs and outputs. See a tool with an input of type "genbank.gz" should be enough to trigger this someday I hope.
Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
Marius van den Beek