- Rename get_master_api_key to get_admin_api_key - less offesnive and more accurate since it doesn't need to be the bootstrap key.
- Add types and cleanup docstrings.
I've been refactoring toward this for at least half a decade. Being able to pip install the API and Selenium tests without bringing in the whole app is important to make them usable outside the context of the app - for testing remote, running servers for instance.
If galaxy-test-driver is available the test cases will still spawn a Galaxy instance if needed, but this is not longer required for import or use.
Split test/base into lib/galaxy_test/base and lib/galaxy_test/driver. Dividing things on whether they import galaxy-app code or not - with the goal of a galaxy-test-base package that doesn't depend on galaxy-app and a galaxy-test-driver package that does.
I've been refactoring around this division for a long time. Certain tests (namely the integration tests) always need to bootstrap Galaxy - but the API tests and the Selenium tests should be installable and runnable without Galaxy in the Python environment. So future galaxy-test-api and galaxy-test-selenium packages should have only an optional dependency on galaxy-test-driver.
Properly sturcturing these namespaces is a precursor for real packages on PyPI, clean separation of the tool shed testing code, and cleans up all the flake8 noqa hacks in test/.
We don't rely on TwillTestCase anymore, that is now exclusively used by
ShedTwillTestCase. Then I've just re-used the infrastructure of
ToolsTestCase for discovering tests, while making sure that we do use an
admin API key during testing.
- Add flake8-import-order to flake8 Pipfile and remove py27-lint-imports
and py27-lint-imports-include-list tox envs
- Fix most E201 and E202 errors reported by flake8-import-order v0.15,
but pin flake8-import-order to v0.14.3 until
https://github.com/PyCQA/flake8-import-order/issues/123
is fixed
This let us drop 2 jobs on Travis per each job.
See documentation added to run_tests.sh. This can be configured to target local web browsers or Selenium remote services as detailed in that script.
Individual tests can be executed with nosetests directly or groups of tests can share a common test Galaxy spin up when using ./run_tests.sh. If GALAXY_TEST_EXTERNAL=<url> is set - it will be respected and Galaxy will not be spun up (remember the URL needs to be reachable from the Selenium server inside the Docker container).
Every test failure writes a current screenshot of Galaxy to database/test_errors.
I will be honest that the desire to move toward selenium is based solely on failing to get the Casper tests to pass consistently due to bugs causing segfaults in the underlying tools (probably casperjs or phantomjs?). Over the past year I have tried multiple versions of dependencies, etc... and it never works out for me. Likewise Martin has never been able to get the tests to run consistently under Jenkins. I don't think Python or JavaScript is inherently better, this wasn't based on a personal preference about what kind of test I want to write.
Despite this being the primary reason, there are clear benefits to Selenium. It tests the actual web browsers we support, generates screenshots, has better documented and more robust tooling, can scale across clusters. Less clear, but certainly still a benefit of being Python based is that it fits with the rest of the test framework more cleanly than CasperJS.
Finally Carl's last words on CasperJS were "Ditch it".
The following specific tests were added or replaced:
- Implemented tour testing via Selenium (walk the two working stock tours and verify elements avaiable and clicks are valid). This is how I discovered #3206.
- Added completely new workflow GUI test (basic creation from URL and in editor).
- Replaced broken CasperJS registeration tests (text and expectations now wrong) with a Selenium variant that works against dev.
- Replaced broken anonymous history CasperJS tests with working Selenium tests.
- Replaced broken upload CasperJS tests with working Selenium tests.
- Replaced history options CasperJS tests with Selenium tests.
- Replaced login CasperJS tests with Selenium tests.
- Replaced history-share-tests.js with test_history_options.py
- Replaced history-panel-tests.js with test_history_panel.py
- Replaced a big part of hda-state-tests.js with test_history_dataset_state.py
I'm confident these utilities represent a sharable and higher-level abstraction around functional testing of Galaxy that can be used outside of Galaxy's testing framework. So a subset of the functional test stuff is in a separate module with minimal dependencies that I intend to make stand alone and pip installable like galaxy-lib.
This module consists of:
- Sizzle stuff in its own package. This code allows Selenium to reason with jQuery selectors instead of vanilla CSS selectors.
- Functionality for creating a Selenium driver and virtual display.
- The ``HasDriver`` mixin - this provides higher level navigation utilities not specifically tied to Galaxy.
- A package with abstractions for navigating Galaxy. This provides a higher-level interfactor for things such as logging in and out, registering a user, navigating menus, fetching Galaxy style tooltips and error messages, and walking Galaxy tours.
In order to demonstrate this new module is useful outside of Galaxy tests - I've included a CLI package to ease building argparse utilities around these abstractions and implemented a simple demonstration script that walks a Galaxy tour and dumps screenshots of Galaxy at each step to a folder. This is more of a demonstration of the utilities than an actual polished end user tool - though it might be helpful in linting and debugging tours.
In the future I hope to extend this scripting to implement
- Periodic deployment testing to ensure things like Jupyter work in production.
- A best practice recipe for Galaxy QA testing by deployers.
- An utility to generate dozens of screenshots of various aspects of Galaxy so we can quickly visually inspect ever part of the GUI before big releases.
xref https://github.com/galaxyproject/starforge/pull/115 xref #1419
IntegrationTestCases differ from functional tests in that they start up a Galaxy instance configured on a per test class basis - so this new class of "integration tests" can test configuration modalities of Galaxy that cannot be tested with traditional Galaxy functional tests.
Classes allow de-duplicating logic and state organization in a different way than methods. Use a common setup, run, tear down paradigm for these tests.
Instead of writing it below GALAXY_ROOT (which it shouldn't do at all) - add to the temp directory managed for the test so that special clean up logic isn't required for it. The global switch GALAXY_TEST_NO_CLEANUP can be used to recover the state of this file if needed now also.
- Sync up the logic for reading and setting GALAXY_TEST_HOST, GALAXY_TEST_PORT, TOOL_SHED_TEST_HOST, and TOOL_SHED_TEST_PORT.
- Setup one high level method for taking a Galaxy-style "app" and creating a webapp from a factory method and then launching a paste server for that entity respecting the above variables.
Previously this was entering a conditional on the same condition twice. This simplifies this to just do that once and makes the galaxy and tool shed driver code look more similar with respect to launching a Galaxy webapp.
into driver_util. This greatly simplifies the code structure of scripts/functional_tests.py's main method.
Add description of GALAXY_TEST_SHED_TOOL_CONF and some documentation to all of this.
Dannon Baker