Tests overridding various output properties (ext, dbkey, name, visible) with file name and galaxy.json, collecting from new_file_path versus job_working_directory, setting job output associations, and logic related to adding outputs to copied histories.
Workflow run template was causing these methods to be called wrong (and has been for a LONG time), this in turn caused me to misunderstand what the spec for how other_values in tool parameters should operate when writing the tests.
The real underlying bug should be fixed with 55cf8bb.
This reduces code duplication related dataset_collectors now and abstracts out important functionality I reuse further to collect dataset collections downstream.
Was originally added in 28d43f4 as DatasetParamContext and backed out of right away. I have reworked it so that it no longer breaks implicit conversion, the relevant classes and methods have less generic names, and it has a healthy set of test cases.
In addition to basic tests on matching datasets to parameters, selections, and implicit conversions - these tests include testing of dataset security inconjuction with data_destination tools as well as filtering data parameters on other data parameters.
Test optional datasets can be used in tool evaluation (in test_evaluation.py).
Add test_data_parameters.py which test many random DataToolParameter behaviors. Test various paths to DataToolParameter.to_python - including recently enhanced ability to use optional dataset with 'multiple=True' data parameters. Test filtering on datatypes, implicit conversion options (both existing conversions and new ones) both when building HTML forms and picking intial values for workflows. Test special handling of hidden datasets. Tests picking intial datasets when optional and without repeats when used in subsequent calls.
Previous changes enabled targetted rewriting of specific kinds of paths - working directory, inputs, outputs, extra files, version path, etc.... This change allows rewriting remaining 'unstructured' paths - namely data indices.
Right now tool evaluation framework uses this capability only for SelectParameter values and fields - which is where these paths will be for data indices. Changeset lays out the recipe for doing this and the functionality could easily be extended for arbitrary parameters or other specific kinds of inputs.
The default ComputeEnvironment does not rewrite any paths obviously, but the abstract base class docstring lays out how to extend a ComputeEnvironment to do this:
def unstructured_path_rewriter( self ):
""" Return a function that takes in a value, determines if it is path
to be rewritten (will be passed non-path values as well - onus is on
this function to determine both if its input is a path and if it should
be rewritten.)
"""
The LwrComputeEnviroment has been updated to provide such a rewriter - it will rewrite such paths, and create a dict of paths that need to be transferred, etc.... The LWR server and client side infrastructure that enables this can be found in this changeset - https://bitbucket.org/jmchilton/lwr/commits/63981e79696337399edb42be5614bc7218cdf95f.
This changeset includes tests for changes to wrappers and the tool evaluation module to enable this.
Pull code out of job wrapper and tool for building and evaluating against template environments and move them into a new ToolEvaluator class (in galaxy/tools/evalution.py). Introduce an abstraction (ComputeEnvironment) for various paths that get evaluated that may be different on a remote server (inputs, outputs, working directory, tools and config directory) and evaluate the template against an instance of this class. Created a default instance of this class (SharedComputeEnvironment). The idea will be that the LWR should be able to an LwrComputeEnvironment and send this to the JobWrapper when building up job inputs - nothing in this commit is LWR specific though so other runners should be able to remotely stage jobs using other mechanisms as well.
This commit adds extensive unit tests of this tool evaluation - testing many different branches through the code, with and without path rewriting, testing job hooks, config files, testing the cheetah evaluation of simple parameters, conditionals, repeats, and non-job stuff like $__app__ and $__root_dir__. As well as a new test case class for JobWrapper and TaskWrapper - though this just tests the relevant portions of that class - namely prepare and version handling.
For clarity - more work toward reduces tools/__init__.py to a more managable size. This also includes an initial suite of test cases for these wrappers - testing simple select wrapper, select wrapper with file options, select wrapper with drilldown widget, raw object wrapper, input value wrapper, and the dataset file name wrapper with and without false paths.
Want to refactor some stuff around in DefaultToolAction so can be reused when dealing with dataset collections downstream in https://github.com/jmchilton/galaxy-central/tree/collections_1 - so creating unit tests to ensure functionality is not changing.
This changeset also reworks test_execution.py moving more stuff to test/unit/tools_support.py to share between test files.
Slightly confusing that state params and raw incoming passed to that method, so pull out rerun_remap_job_id sooner and just pass that along (it was the only incoming was used for). Use the oppertunity to isolate potential errors with decoding rerun_remap_job_id and include more informative error message.
Add unit test to test invalid rerun_remap_job_ids.
Going to be doing some more work on tool state stuff so it will be good to have a way to test that. This also brings in test/unit/tools_support.py from Pull Request #287 (would be overkill for just these tests, but it is useful for future tests coming down the pipe.)
John Chilton