-framework will still cause just a small set of non-default, non-user facing tools to be tested and exerise various aspects of the tool and test frameworks, but now an argument can be passed in -with_framework_test_tools to allow these tools to be used as part of the normal test framework (i.e. when not using -framework). This serves two purposes - to ensure the feature coverage of existing tests remains high as the tests are migrated out of the distribution and to allow a set of tests to be available for functional (e.g. API tests) that users don't need to see.
Enable task splitting in functional tests to test this. Specify tool_dir in samples tool_conf.xml so it doesn't need to be explicitly set in scripts/functional_tests.py.
... if you can call 1 new class a framework. Includes a few test cases to exercise/drive it. These examples include a histories API test (a typical API test) and a general test of the API framework itself (mostly just the run_as functionality).
This includes changes to interactor.py to make it more useful outside the context of tool/workflow testing as well as a tweak to test Galaxy that gets started to allow testing of the run_as feature.
Pull logic for how master and user API keys are determinined for testing out of functional_tests.py for reuse elsewhere.
This refactoring will help the creation of an API test framework.
Shared across Tool Shed and Galaxy to reduce code duplication. Previously this was just a bunch, making it a real class allows placing some shared logic in there.
This refactoring also eliminates some global variables galaxy.model.mapping.Session, galaxy.model.mapping.context, and same for tool shed. This may break things, but these are things that should probably be fixed anyway.
In particular small changes to db_shell.py and a unit test that depended on these global variables have been updated. The functional test frameworks likewise needed to be updated to not depend on these - these changes were more substantial.
To fix these functional tests, I essentially replace old references to global variables in Galaxy with references to global variables just defined in the test framework in test/functional/database_contexts.py (a slight improvement).
... tool shed integration will be more challenging, but this can be used for directly configured Galaxy tools and outlines a syntax that could be shared with a tool shed driven approach.
The one tricky part is how to match workflow outputs to things to check. Right now it is based on the index of the output across the workflow. This is both difficult to determine and very brittle to workflow modifications - how to proceed - require annontation string to be setup? Modify workflow data model to all assigning names to outputs the way inputs havenames? At any rate this current syntax should be considered beta and may change - if it does Galaxy will not continue to support this syntax.
To run the sample test execute the following test command:
sh run_functional_tests.sh -workflow test-data/workflows/1.xml
Middle ground between recreating a completely new database and pointing at existing database with GALAXY_TEST_DBURI. The former requires a lot of setup time, the latter results in test failures in certain cases (namely tool shed tests expecting clean database).
GALAXY_TEST_DB_TEMPLATE can be either a file path (absolute) or URL.
In order to facilitate this, a new Galaxy config option (database_auto_migrate) has been added. If this option is enabled, when Galaxy starts up and points at an existing database, if that database is not at the newest version it will be automatically migrated. This option defaults to False, but is enabled in testing if GALAXY_TEST_DB_TEMPLATE is set.
I think we should go a step further and make this (database_auto_migrate) default to True if database_connection references an sqlite database - unless we believe there are Galaxy instances out there based on sqlite that are REALLY old or that are production enough to warrent requiring admins to do that database migration in a separate step (presumably encouraging them to make a backup pre-migration). Thoughts?
John Chilton