The automatic retrying of tests only runs for methods - not for the full test case (if that makes sense) - so setUp() isn't rerun. By refactoring the login action out of the setUp method and into the test function for these cases - when the test is retried the user should be logged back in again as well.
Previously we only detected transitions for retrying actions based on stale element exceptions, this expands that to include popups that may be fading out for instance.
I think this is what is happening with the transiently failing test here https://jenkins.galaxyproject.org/job/selenium/528/artifact/528-test-errors/test_save_as2017092203291506065342/stacktrace.txt. The exception indicates that a click was not clickable because a modal element that was fading out - though it had been previously clickable. I think this should fix that.
I don't get why when we click submit the user does not actually get logged in, but based on the last round of improved error messages this seems to be the case. You might think there is some callback in the login form that doesn't get registered by the time Selenium clicks the submit button - but this doesn't seem to be the case - I don't see any jquery magic happening in login.mako.
Should fix failures like this:
https://jenkins.galaxyproject.org/job/selenium/482/testReport/junit/selenium_tests.test_saved_histories/SavedHistoriesTestCase/test_history_publish/
I'd say at this point this is the most common problem in the Selenium tests.
This also introduces a framework for taking state snapshots of the Galaxy interface during tests that will only get written out if the tests fail. We now take screenshots before and after submitting the login form but this is a general purpose debugging mechanism that could be used other places.
These tests aren't idealized unittest.TestCase because they initialize class level data in instance level methods. While this isn't ideal, it is seems an entirely fair workaround given that SeleniumTestCase setups up the Selenium connection itself in an instance method - so class-level initializers would not be able to setup Galaxy data. Since I think we will continue using this pattern then, probably best to formalize it a bit and improve error handling.
This provides a formal super class for these test cases that provides a uniform method for setting up the class level data and tracks whether this is successful or not. This serves a couple purposes beyond simple uniformity. First, it tracks if the state has actually been setup or not and will skip subsequent tests if it hasn't. Some of these tests aren't passing very consistently on Jenkins and so we get a bunch of extra noise for tests that are attempting to run without their preconditions met - this will fix that and make the original errors much more clear. Moving the "hacky" part of this into the framework itself also means the tests themselves don't have to repeat hacks like seeing if variables are set with ``getattr`` and such - I always prefer one framework hack to a dozen application hacks.
We do this by recording any macros that are required to load a tool. When
loading a tool we register the macro to be watched, and if the macro changes we
reload all corresponding tools.
At the insistence of @nsoranzo, be more explicit about waiting for workflow test in API tests. While I was in there I also setup new abstractions for waiting on state and updated various workflow cancelling tests to use test_history context for better error reporting.
I saw a failed test earlier that would have been fixed by a switch to wait_for_and_click in workflow_editor_click_options. This makes that switch and moves some of these workflow editor helper functions into navigates_galaxy.py for reuse by other test cases and by library users.
John Chilton