The automatic retrying of tests only runs for methods - not for the full test case (if that makes sense) - so setUp() isn't rerun. By refactoring the login action out of the setUp method and into the test function for these cases - when the test is retried the user should be logged back in again as well.
I don't get why when we click submit the user does not actually get logged in, but based on the last round of improved error messages this seems to be the case. You might think there is some callback in the login form that doesn't get registered by the time Selenium clicks the submit button - but this doesn't seem to be the case - I don't see any jquery magic happening in login.mako.
Should fix failures like this:
https://jenkins.galaxyproject.org/job/selenium/482/testReport/junit/selenium_tests.test_saved_histories/SavedHistoriesTestCase/test_history_publish/
I'd say at this point this is the most common problem in the Selenium tests.
This also introduces a framework for taking state snapshots of the Galaxy interface during tests that will only get written out if the tests fail. We now take screenshots before and after submitting the login form but this is a general purpose debugging mechanism that could be used other places.
These tests aren't idealized unittest.TestCase because they initialize class level data in instance level methods. While this isn't ideal, it is seems an entirely fair workaround given that SeleniumTestCase setups up the Selenium connection itself in an instance method - so class-level initializers would not be able to setup Galaxy data. Since I think we will continue using this pattern then, probably best to formalize it a bit and improve error handling.
This provides a formal super class for these test cases that provides a uniform method for setting up the class level data and tracks whether this is successful or not. This serves a couple purposes beyond simple uniformity. First, it tracks if the state has actually been setup or not and will skip subsequent tests if it hasn't. Some of these tests aren't passing very consistently on Jenkins and so we get a bunch of extra noise for tests that are attempting to run without their preconditions met - this will fix that and make the original errors much more clear. Moving the "hacky" part of this into the framework itself also means the tests themselves don't have to repeat hacks like seeing if variables are set with ``getattr`` and such - I always prefer one framework hack to a dozen application hacks.
I saw a failed test earlier that would have been fixed by a switch to wait_for_and_click in workflow_editor_click_options. This makes that switch and moves some of these workflow editor helper functions into navigates_galaxy.py for reuse by other test cases and by library users.
- set_tags() appeared in both published and saved history tests, refactored into navigates_galaxy with function name history_panel_add_tags
- is_displayed() appeared in both published and saved history tests, refactored into has_driver as selector_is_displayed
- Introduced history_panel_rename to reduce duplication across history panel tests and newer published and saved history tests.
- Removed custom history click option helper in saved and published history tests and just used the variant in navigates_galaxy.
- use wait_for_and_click a couple more places...
- Be sure when sharing a history with a user we wait to actually see that user's e-mail address appear in the history's sharing list after submission.
- Print something for the failed history sharing assertions (I'm pretty sure they are 403 errors because some aspect of the sharing or login didn't work - but might as well verify).
Locally this never happens, but remotely sometimes when sorting by owner the histories are sorted by owner but not in the same order as the test was asserting. I would assume this is likely a postgres vs. sqlite difference - perhaps the former doesn't sort the rows by the order they were created within equal sort classifications (i.e. same owner) but the latter does. I've fixed this - the test should not have been asserting such a strong condition I don't think.
I switched to randomly generated usernames without actually ensure the random usernames would be generated in the same lexiographic order - that wasn't super bright of me. That sort by owner test should pass as frequently as the other tests in that test case class now.
There are a few different transiently failing tests on Jenkins that seem to fail because the test thinks Galaxy is logged in but it is not. So now after we click the login button in Selenium - we will wait for to see the "Register or Login" button turn into the "User" button.
This is a similar problem and fix to what was done in #4562 - which seemed to help.
John Chilton