There has been for some time a scary warning when testing (http://dev.list.galaxyproject.org/run-test-sh-ImportError-cannot-import-name-args-from-interpreter-flags-td4666277.html) that I now understand is the result of Twill being packaged with its own custom subprocess module that conflicts with modules using multiprocessing (e.g. kombu).
After the #198 this scary warning has become an exception that prevents exception of tests (see https://travis-ci.org/galaxyproject/planemo/jobs/61547827 for instance). Importing subprocess seems to force Python to use the system module instead of twills - which seems to prevent these problems.
There have been other reports of this problem with Twill in other projects and removing the subprocess module didn't seem to break anything for them either.
... for tool tests. Longer term this functionality should be dropped (i.e. after all the tools are out of Galaxy) and the test-data always lives next to the tool - but for now it decreases the size of the repository ahead of a potential move to github.
Long term this could allow Galaxy to support - multiple tooling formats (Galaxy-like YAML, CWL http://bit.ly/cwltooldesc, etc...). But I think it is also important from a purely design perspective - this is a core logic class integrating different components - they should not also be doing XML parsing.
To verify the interface for parsing tools is expressive enough to allow multiple useful implementations, I built a test YAML tool description that implements many of the same features as Galaxy but smooths out rough edges (uses exit codes for job failure by default for instance). Loading these tools is disabled by default and it is not documented how to enable them because they are not intended to be part of Galaxy's public API.
Rather than relying solely on exceptions back to nose/test framework - add option (--structured_data_report_file) to run_tests.sh that causes a bunch of detailed data to be dumped to the specified file in a very structured way. Includes full to_dict of the job from the API which in turn includes job metrics, command-line, job's standard error and outputs (instead of the test frameworks), as well as the tool inputs, and exceptions broken out for tool execution versus output checking.
Its all indexed in the file by the test id (without the actual test toolbox depending on knowing the test id) - so one could pair this information with the XUnit output to produce much more detailed breakdowns of the tests.
Functional tests don't work without the tweak to imports in cloudlaunch - not sure if that has always been a problem and I am just running the tests in a different Galaxy instance or if the earlier import of util caused the problem.
Regardless I guess we shoud update to a version of bioblend that doesn't require simplejson.
Now appends test tools to sample tools instead of to default tools Galaxy would other wise load (config/tool_conf.xml or tool_conf.xml if these exist instead of sample) - so this is different behavior but I am unsure if it is worse. --with_framework_test_tools was added for API tests and these should probably target the distribution and not random tools someone might have configured so perhaps this new behavior is superior.
-framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see.
Enable task splitting in functional tests to test this. Specify tool_dir in samples tool_conf.xml so it doesn't need to be explicitly set in scripts/functional_tests.py.
... if you can call 1 new class a framework. Includes a few test cases to exercise/drive it. These examples include a histories API test (a typical API test) and a general test of the API framework itself (mostly just the run_as functionality).
This includes changes to interactor.py to make it more useful outside the context of tool/workflow testing as well as a tweak to test Galaxy that gets started to allow testing of the run_as feature.
Pull logic for how master and user API keys are determinined for testing out of functional_tests.py for reuse elsewhere.
This refactoring will help the creation of an API test framework.
John Chilton