Functional testing of workflows (for Galaxy).

... tool shed integration will be more challenging, but this can be used for directly configured Galaxy tools and outlines a syntax that could be shared with a tool shed driven approach.

The one tricky part is how to match workflow outputs to things to check. Right now it is based on the index of the output across the workflow. This is both difficult to determine and very brittle to workflow modifications - how to proceed - require annontation string to be setup? Modify workflow data model to all assigning names to outputs the way inputs havenames? At any rate this current syntax should be considered beta and may change - if it does Galaxy will not continue to support this syntax.

To run the sample test execute the following test command:

sh run_functional_tests.sh -workflow test-data/workflows/1.xml

scripts/functional_tests.py

@@ -420,7 +420,7 @@ def main():
     new_path = [ os.path.join( cwd, "test" ) ]
     new_path.extend( sys.path[1:] )
     sys.path = new_path
-    import functional.test_toolbox
+
     # ---- Find tests ---------------------------------------------------------
     if galaxy_test_proxy_port:
         log.info( "Functional tests will be run against %s:%s" % ( galaxy_test_host, galaxy_test_proxy_port ) )
@@ -436,6 +436,13 @@ def main():
         os.environ[ 'GALAXY_TEST_HOST' ] = galaxy_test_host
 
         def _run_functional_test( testing_shed_tools=None ):
+            workflow_test = __check_arg( '-workflow', param=True )
+            if workflow_test:
+                import functional.test_workflow
+                functional.test_workflow.WorkflowTestCase.workflow_test_file = workflow_test
+                functional.test_workflow.WorkflowTestCase.master_api_key = master_api_key
+                functional.test_workflow.WorkflowTestCase.user_api_key = os.environ.get( "GALAXY_TEST_USER_API_KEY", default_galaxy_user_key )
+            import functional.test_toolbox
             functional.test_toolbox.toolbox = app.toolbox
             functional.test_toolbox.build_tests(
                 testing_shed_tools=testing_shed_tools,