Long term this could allow Galaxy to support - multiple tooling formats (Galaxy-like YAML, CWL http://bit.ly/cwltooldesc, etc...). But I think it is also important from a purely design perspective - this is a core logic class integrating different components - they should not also be doing XML parsing.
To verify the interface for parsing tools is expressive enough to allow multiple useful implementations, I built a test YAML tool description that implements many of the same features as Galaxy but smooths out rough edges (uses exit codes for job failure by default for instance). Loading these tools is disabled by default and it is not documented how to enable them because they are not intended to be part of Galaxy's public API.
Rather than relying solely on exceptions back to nose/test framework - add option (--structured_data_report_file) to run_tests.sh that causes a bunch of detailed data to be dumped to the specified file in a very structured way. Includes full to_dict of the job from the API which in turn includes job metrics, command-line, job's standard error and outputs (instead of the test frameworks), as well as the tool inputs, and exceptions broken out for tool execution versus output checking.
Its all indexed in the file by the test id (without the actual test toolbox depending on knowing the test id) - so one could pair this information with the XUnit output to produce much more detailed breakdowns of the tests.
Functional tests don't work without the tweak to imports in cloudlaunch - not sure if that has always been a problem and I am just running the tests in a different Galaxy instance or if the earlier import of util caused the problem.
Regardless I guess we shoud update to a version of bioblend that doesn't require simplejson.
Now appends test tools to sample tools instead of to default tools Galaxy would other wise load (config/tool_conf.xml or tool_conf.xml if these exist instead of sample) - so this is different behavior but I am unsure if it is worse. --with_framework_test_tools was added for API tests and these should probably target the distribution and not random tools someone might have configured so perhaps this new behavior is superior.
-framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see.
Enable task splitting in functional tests to test this. Specify tool_dir in samples tool_conf.xml so it doesn't need to be explicitly set in scripts/functional_tests.py.
... if you can call 1 new class a framework. Includes a few test cases to exercise/drive it. These examples include a histories API test (a typical API test) and a general test of the API framework itself (mostly just the run_as functionality).
This includes changes to interactor.py to make it more useful outside the context of tool/workflow testing as well as a tweak to test Galaxy that gets started to allow testing of the run_as feature.
Pull logic for how master and user API keys are determinined for testing out of functional_tests.py for reuse elsewhere.
This refactoring will help the creation of an API test framework.
Shared across Tool Shed and Galaxy to reduce code duplication. Previously this was just a bunch, making it a real class allows placing some shared logic in there.
This refactoring also eliminates some global variables galaxy.model.mapping.Session, galaxy.model.mapping.context, and same for tool shed. This may break things, but these are things that should probably be fixed anyway.
In particular small changes to db_shell.py and a unit test that depended on these global variables have been updated. The functional test frameworks likewise needed to be updated to not depend on these - these changes were more substantial.
To fix these functional tests, I essentially replace old references to global variables in Galaxy with references to global variables just defined in the test framework in test/functional/database_contexts.py (a slight improvement).
... tool shed integration will be more challenging, but this can be used for directly configured Galaxy tools and outlines a syntax that could be shared with a tool shed driven approach.
The one tricky part is how to match workflow outputs to things to check. Right now it is based on the index of the output across the workflow. This is both difficult to determine and very brittle to workflow modifications - how to proceed - require annontation string to be setup? Modify workflow data model to all assigning names to outputs the way inputs havenames? At any rate this current syntax should be considered beta and may change - if it does Galaxy will not continue to support this syntax.
To run the sample test execute the following test command:
sh run_functional_tests.sh -workflow test-data/workflows/1.xml
Middle ground between recreating a completely new database and pointing at existing database with GALAXY_TEST_DBURI. The former requires a lot of setup time, the latter results in test failures in certain cases (namely tool shed tests expecting clean database).
GALAXY_TEST_DB_TEMPLATE can be either a file path (absolute) or URL.
In order to facilitate this, a new Galaxy config option (database_auto_migrate) has been added. If this option is enabled, when Galaxy starts up and points at an existing database, if that database is not at the newest version it will be automatically migrated. This option defaults to False, but is enabled in testing if GALAXY_TEST_DB_TEMPLATE is set.
I think we should go a step further and make this (database_auto_migrate) default to True if database_connection references an sqlite database - unless we believe there are Galaxy instances out there based on sqlite that are REALLY old or that are production enough to warrent requiring admins to do that database migration in a separate step (presumably encouraging them to make a backup pre-migration). Thoughts?
John Chilton