PR #6925 introduced a GUI for connecting non-data (e.g. integer, boolean, color, etc..) workflow input parameter to tool input parameters (the backend for this was originally added in #1306). That ideally was just the beginning of work toward using such values in structured ways in workflows.
This PR extends tool output handling to allow producing of non-data parameters. These can serve as a source for non-data values in workflows the same work workflow input parameters can. To make such values more easy to produce, this PR also introduces Galaxy expression tools - mirroring functionality regularly used in CWL. These small JavaScript-based tools that consume inputs just like a regular Galaxy tool but that produce dictionary of non-data values.
I think these expressions will be maximally useful when paired with format 2 workflows once we allow users to load arbitrary tools (I make the case more in full here https://github.com/galaxyproject/galaxy/pull/7545#issuecomment-473894424), but I outline some potential uses there as well.
Because there is always a checklist in my PR descriptions:
- Tool definition language and plumbing and datatype for expressing expressions as jobs.
- Allow connecting expression tools to parameters in workflows, will delay evaluation of workflow so calculated value
- Example test expression tools for testing and demonstration.
Test case for indexing dynamic tools and deactivating dynamic tools. Verify deactivation via attributes in resulting object but also by trying to run a deactivated dynamic tool and getting a 404.
- Introduce models and a API for creating tools dynamically.
- Use Galaxy's testing-only YAML based representation of tools to prototype this.
- Extend Format 2 workflow definitions to allow embedding tools directly into workflows, either directly or using a CWL-style @import syntax.
Testing:
Test cases demonstrating tools can be imported (only by admins) and are runnable are included with this commit. More test cases regarding workflow use of dynamic tools and Format 2 workflow definition extensions are also included.
These tests can be run with the following commands:
```
./run_tests.sh -api test/api/test_tools.py:ToolsTestCase.test_nonadmin_users_cannot_create_tools
./run_tests.sh -api test/api/test_tools.py:ToolsTestCase.test_dynamic_tool_1
./run_tests.sh -api test/api/test_workflows.py:WorkflowsApiTestCase.test_import_export_dynamic
./run_tests.sh -api test/api/test_workflows_from_yaml.py:WorkflowsFromYamlApiTestCase.test_workflow_embed_tool
./run_tests.sh -api test/api/test_workflows_from_yaml.py:WorkflowsFromYamlApiTestCase.test_workflow_import_tool
```
Prior to this, composite uploads were only allowed for select data types. As far as I can tell, the framework itself allows arbitrary datatypes to have extra files. This allows the upload API to take in extra files for datatypes that aren't explicitly annotated has having extra files.
These may be specified one at a time or in directory structures via tar files.
Fix warnings during py34-unit tests like:
```
/home/travis/build/galaxyproject/galaxy/test/unit/workflows/test_extract_summary.py:46: DeprecationWarning: Please use assertEqual instead.
self.assertEquals(job_dict[hda.job], [('out1', derived_hda_2)])
```
See e.g. https://travis-ci.org/galaxyproject/galaxy/jobs/438829277
There was an existing library dataset permission API and one proposed for HDAs in #6461. This tries to reconcile the two approaches and move toward managers.
- Proper treatment of "master_api_key" throughout the manager layers related to this.
- Use PUT instead of POST, since that is more appropriate for updates I believe.
- Use a similar URL pattern for library datasets, history contents, and dataset APIs.
- Generalize the dataset API to allow HDA or LDDA inputs (the proposal for the new API in #6461 added an HDA-only option to the dataset API - this isn't how that API layer is meant to be used I believe).
- Add an HDA only endpoint in the history contents API for this functionality to mirror the library dataset API.
- Use consistent and backward compatible payload parsing for these endpoints.
- Use consistent serialization for the result of these changes.
- Generalize the HDA variant of this to allow different actions - such as make public and make private that were available in the library dataset variant.
- Test cases for the new API endpoints, the roles API, tests for permissions during collection creation and running tools with both dataset and collection inputs.
- Switch select_tag to group_tag as a parameter name - since it is restricted to group_tags and we don't call data parameters select_data.
- Add minimal XSD documentation for this new parameter type.
- Add example tools - for single and multiple tags.
- Add API tests for these tools and the tag handling.
Unlike upload 1.0, use different 'src' types to distinguish between URLs and pasted content - in the abstract this is better because it is more structured and practically this allows 'pasting' spreadsheets that start with URLs for instance (that is sometimes useful).
It is jarring to users that works for collection inputs but not multiple data inputs, especially because the IUC discourages collection inputs when multiple data inputs would work.
It is jarring to users that works for collection inputs but not multiple data inputs, especially because the IUC discourages collection inputs when multiple data inputs would work.
Martin Cech