- Use localStorage to enable the logs at the start of new connections during tests.
- Add new flatten console logging options in the client to hack around problem recovering multiple argument style logging the client uses with only getting back first argument when using Selenium API (at least with Chrome).
- Break out the browser log into a sort of default and verbose mode (only the latter includes DEBUG and INFO messages).
- Use json.dump with the indent call to get more visually parsable logging messages in the Selenium test failure dump.
This continues a thread started in 0650978a91 and #4732 of switching to smarter selectors from a more structured YAML description of the Galaxy DOM. This continues that by:
- Eliminating any use of the old navigation YAML file (navigation-data.yml).
- Replace href selectors with other things since they break when testing Galaxy with a proxy-prefix.
- Extending the raw selectors to allow concept of nesting selectors.
- Building even smarter "components" that wrap these raw selectors generated from the YAML directly in a way that allows them to utilize the actual Selenium session and helper class.
So the old ``self.navigation_data`` is gone, ``self.navigation`` still yields the raw selectors, and ``self.components`` yields the smarter variant. Hopefully all will agree the code that uses ``self.components`` is more compact and readable.
The following a very basic example of this:
```diff
def click_masthead_user(self):
- self.wait_for_and_click(self.navigation.masthead.selectors.user)
+ self.components.masthead.user.wait_for_and_click()
```
Here is an example of combining it with the new child selector syntax:
```diff
def assert_item_dbkey_displayed_as(self, hid, dbkey):
- item_body_selector = self.history_panel_item_body_selector(hid=hid, wait=True)
- dbkey_selector = item_body_selector + ' ' + self.test_data["historyPanel"]["selectors"]["hda"]["dbkey"]
- dbkey_element = self.wait_for_selector_visible(dbkey_selector)
- assert dbkey in dbkey_element.text
+ item_body = self.history_panel_item_component(hid=hid)
+ dbkey_text = item_body.dbkey.wait_for_text()
+ assert dbkey in dbkey_text
```
- Test importing datasets from histories.
- Test basic importing of a dataset from a path.
- Test creating a folder.
- Test create library, renaming a library, filtering libraries by name, and sorting of libraries.
- Test various buttons (e.g. details, help).
Creates and deletes a history for tests - good for tests meant to aim at production servers that need to have useful metadata and need to be cleaned up.
Implement different "WAIT_TYPES" and try to sleep for different amounts of time based on what is being waited for.
Replace sleep statements in addition to Selenium wait on statements.
I don't get why when we click submit the user does not actually get logged in, but based on the last round of improved error messages this seems to be the case. You might think there is some callback in the login form that doesn't get registered by the time Selenium clicks the submit button - but this doesn't seem to be the case - I don't see any jquery magic happening in login.mako.
Should fix failures like this:
https://jenkins.galaxyproject.org/job/selenium/482/testReport/junit/selenium_tests.test_saved_histories/SavedHistoriesTestCase/test_history_publish/
I'd say at this point this is the most common problem in the Selenium tests.
This also introduces a framework for taking state snapshots of the Galaxy interface during tests that will only get written out if the tests fail. We now take screenshots before and after submitting the login form but this is a general purpose debugging mechanism that could be used other places.
These tests aren't idealized unittest.TestCase because they initialize class level data in instance level methods. While this isn't ideal, it is seems an entirely fair workaround given that SeleniumTestCase setups up the Selenium connection itself in an instance method - so class-level initializers would not be able to setup Galaxy data. Since I think we will continue using this pattern then, probably best to formalize it a bit and improve error handling.
This provides a formal super class for these test cases that provides a uniform method for setting up the class level data and tracks whether this is successful or not. This serves a couple purposes beyond simple uniformity. First, it tracks if the state has actually been setup or not and will skip subsequent tests if it hasn't. Some of these tests aren't passing very consistently on Jenkins and so we get a bunch of extra noise for tests that are attempting to run without their preconditions met - this will fix that and make the original errors much more clear. Moving the "hacky" part of this into the framework itself also means the tests themselves don't have to repeat hacks like seeing if variables are set with ``getattr`` and such - I always prefer one framework hack to a dozen application hacks.
- Fix rename test for recent change to using alert.
- Test basic adding a tag.
- Outline of a test for downloading workflows.
- Add test for publishing display on index.
- Add test for using the search box on the workflow index page.
- By default this won't occur locally, but you can set GALAXY_TEST_SELENIUM_RETRIES to a non-zero number to enable auto retrying tests that many times.
- Capture the stack trace in the Selenium test error report directory - this will be useful for debugging problems that may fail once but pass on a subsequence attempt. Jenkins now captures these directories and includes their content in the test reports.
- Document the Selenium test error report directory in run_tests.sh as well as this new retry variable.
- Update the Jenkins test script to set this new variable to 1 so transient failures break the build much less frequently.
Expand and then break out workflow management tests into its own file since these aren't related to the workflow editor per se. Includes new tests of workflow "viewing" and "renaming".
Add a new file for testing workflow execution. This currently contains tests for simple execution with a single input and a test for running workflows with tool version upgrades.
This includes a refactoring of some existing tests related to checking various things in the history panel - as part of building up good history panel abstractions. The previous constructs were ported fairly literally from older CasperJS tests. These abstractions therefore referred to everything has "hda"s instead of history items. The newer abstractions therefore allow for collections and are built around HIDs (a visual thing exposed to the user) instead of HDA IDs. I think this is higher-level and more appropriate for a web functional test.
Rebase into workflow GUI tests.
See documentation added to run_tests.sh. This can be configured to target local web browsers or Selenium remote services as detailed in that script.
Individual tests can be executed with nosetests directly or groups of tests can share a common test Galaxy spin up when using ./run_tests.sh. If GALAXY_TEST_EXTERNAL=<url> is set - it will be respected and Galaxy will not be spun up (remember the URL needs to be reachable from the Selenium server inside the Docker container).
Every test failure writes a current screenshot of Galaxy to database/test_errors.
I will be honest that the desire to move toward selenium is based solely on failing to get the Casper tests to pass consistently due to bugs causing segfaults in the underlying tools (probably casperjs or phantomjs?). Over the past year I have tried multiple versions of dependencies, etc... and it never works out for me. Likewise Martin has never been able to get the tests to run consistently under Jenkins. I don't think Python or JavaScript is inherently better, this wasn't based on a personal preference about what kind of test I want to write.
Despite this being the primary reason, there are clear benefits to Selenium. It tests the actual web browsers we support, generates screenshots, has better documented and more robust tooling, can scale across clusters. Less clear, but certainly still a benefit of being Python based is that it fits with the rest of the test framework more cleanly than CasperJS.
Finally Carl's last words on CasperJS were "Ditch it".
The following specific tests were added or replaced:
- Implemented tour testing via Selenium (walk the two working stock tours and verify elements avaiable and clicks are valid). This is how I discovered #3206.
- Added completely new workflow GUI test (basic creation from URL and in editor).
- Replaced broken CasperJS registeration tests (text and expectations now wrong) with a Selenium variant that works against dev.
- Replaced broken anonymous history CasperJS tests with working Selenium tests.
- Replaced broken upload CasperJS tests with working Selenium tests.
- Replaced history options CasperJS tests with Selenium tests.
- Replaced login CasperJS tests with Selenium tests.
- Replaced history-share-tests.js with test_history_options.py
- Replaced history-panel-tests.js with test_history_panel.py
- Replaced a big part of hda-state-tests.js with test_history_dataset_state.py
I'm confident these utilities represent a sharable and higher-level abstraction around functional testing of Galaxy that can be used outside of Galaxy's testing framework. So a subset of the functional test stuff is in a separate module with minimal dependencies that I intend to make stand alone and pip installable like galaxy-lib.
This module consists of:
- Sizzle stuff in its own package. This code allows Selenium to reason with jQuery selectors instead of vanilla CSS selectors.
- Functionality for creating a Selenium driver and virtual display.
- The ``HasDriver`` mixin - this provides higher level navigation utilities not specifically tied to Galaxy.
- A package with abstractions for navigating Galaxy. This provides a higher-level interfactor for things such as logging in and out, registering a user, navigating menus, fetching Galaxy style tooltips and error messages, and walking Galaxy tours.
In order to demonstrate this new module is useful outside of Galaxy tests - I've included a CLI package to ease building argparse utilities around these abstractions and implemented a simple demonstration script that walks a Galaxy tour and dumps screenshots of Galaxy at each step to a folder. This is more of a demonstration of the utilities than an actual polished end user tool - though it might be helpful in linting and debugging tours.
In the future I hope to extend this scripting to implement
- Periodic deployment testing to ensure things like Jupyter work in production.
- A best practice recipe for Galaxy QA testing by deployers.
- An utility to generate dozens of screenshots of various aspects of Galaxy so we can quickly visually inspect ever part of the GUI before big releases.
xref https://github.com/galaxyproject/starforge/pull/115 xref #1419
Nicola Soranzo