From 1a7215b040fdd3cfc4a5615f596ce3fb97d1d9ef Mon Sep 17 00:00:00 2001 From: John Chilton Date: Sun, 10 Jan 2021 18:44:20 -0500 Subject: [PATCH] Documentation on writing Galaxy tests. --- doc/Makefile | 3 + doc/source/dev/run_tests_help.txt | 246 +++++++++++++++++++++ doc/source/dev/writing_tests.md | 356 +++++++++++++++++++++++++++++- 3 files changed, 604 insertions(+), 1 deletion(-) create mode 100644 doc/source/dev/run_tests_help.txt diff --git a/doc/Makefile b/doc/Makefile index bcb01bcf577..66e184a4e1b 100644 --- a/doc/Makefile +++ b/doc/Makefile @@ -42,6 +42,9 @@ help: @echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " updaterst to update Sphinx RST files for lib/ to reflect code structure changes" +source/dev/run_tests_help.txt: + bash ../run_tests.sh --help >> source/dev/run_tests_help.txt + source/dev/schema.md: parse_gx_xsd.py schema_template.md ../lib/galaxy/tool_util/xsd/galaxy.xsd ## Build Github-flavored Markdown from Galaxy Tool XSD (expects lxml in environment) python parse_gx_xsd.py schema_template.md ../lib/galaxy/tool_util/xsd/galaxy.xsd > $@ diff --git a/doc/source/dev/run_tests_help.txt b/doc/source/dev/run_tests_help.txt new file mode 100644 index 00000000000..f33850097ce --- /dev/null +++ b/doc/source/dev/run_tests_help.txt @@ -0,0 +1,246 @@ +'run_tests.sh -id bbb' for testing one tool with id 'bbb' ('bbb' is the tool id) +'run_tests.sh -sid ccc' for testing one section with sid 'ccc' ('ccc' is the string after 'section::') +'run_tests.sh -list' for listing all the tool ids +'run_tests.sh -api (test_path)' for running all the test scripts in the ./lib/galaxy_test/api directory, test_path + can be pytest selector +'run_tests.sh -integration (test_path)' for running all integration test scripts in the ./test/integration directory, test_path + can be pytest selector +'run_tests.sh -toolshed (test_path)' for running all the test scripts in the ./lib/tool_shed/test directory +'run_tests.sh -installed' for running tests of Tool Shed installed tools +'run_tests.sh -main' for running tests of tools shipped with Galaxy +'run_tests.sh -framework' for running through example tool tests testing framework features in test/functional/tools" +'run_tests.sh -framework -id toolid' for testing one framework tool (in test/functional/tools/) with id 'toolid' +'run_tests.sh -data_managers -id data_manager_id' for testing one Data Manager with id 'data_manager_id' +'run_tests.sh -unit' for running all unit tests (doctests and tests in test/unit) +'run_tests.sh -unit (test_selector)' for running unit tests on specified test path (using pytest selector syntax) +'run_tests.sh -selenium' for running all selenium web tests (in lib/galaxy_test/selenium) +'run_tests.sh -selenium (test_selector)' for running specified selenium web tests (using pytest selector syntax) + +This wrapper script largely serves as a point documentation and convenience for +running Galaxy's Python tests. Most Python tests shipped with Galaxy can be run with +pytest directly. Galaxy's client unit tests can be run with make client-test +or yarn directly as documented in detail in client/README.md. + +The main test types are as follows: + +- API: These tests are located in lib/galaxy_test/api and test various aspects of the Galaxy + API and test general backend aspects of Galaxy using the API. +- Integration: These tests are located in test/integration and test special + configurations of Galaxy. All API tests assume a particular Galaxy configuration + defined by test/base/driver_util.py and integration tests can be used to + launch and test Galaxy in other configurations. +- Framework: These tests are all Galaxy tool tests and can be found in + test/functional/tools. These are for the most part meant to test and + demonstrate features of the tool evaluation environment and of Galaxy tool XML + files. +- Unit: These are Python unit tests either defined as doctests or inside of + test/unit. These should generally not require a Galaxy instance and should + quickly test just a component or a few components of Galaxy's backend code. +- Selenium: These are full stack tests meant to test the Galaxy UI with real + browsers and are located in lib/galaxy_test/selenium. +- ToolShed: These are web tests that use the older Python web testing + framework twill to test ToolShed related functionality. These are + located in lib/tool_shed/test. + +Python testing is mostly done via pytest. Specific tests can be selected +using the pytest selector syntax is described at https://docs.pytest.org/en/latest/usage.html. + +The spots these selectors can be used is described in the above usage documentation +as test_path. A few examples are shown below. + +Run all API tests: + ./run_tests.sh -api + +The same test as above can be run using nosetests directly as follows: + pytest lib/galaxy_test/api + +However when using pytest directly output options defined in this +file aren't respected and a new Galaxy instance will be created for each +TestCase class (this scripts optimizes it so all tests can share a Galaxy +instance). + +Run a full class of API tests: + ./run_tests.sh -api lib/galaxy_test/api/test_tools.py::ToolsTestCase + +Run a specific API test: + ./run_tests.sh -api lib/galaxy_test/api/test_tools.py::ToolsTestCase::test_map_over_with_output_format_actions + +Run all selenium tests (Under Linux using Docker): + # Start selenium chrome Docker container + docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:3.0.1-aluminum + GALAXY_TEST_SELENIUM_REMOTE=1 ./run_tests.sh -selenium + +Run a specific selenium test (under Linux or Mac OS X after installing geckodriver or chromedriver): + ./run_tests.sh -selenium lib/galaxy_test/selenium/test_registration.py::RegistrationTestCase::test_reregister_username_fails + +Run a selenium test against a running server while watching client (fastest iterating on client tests): + ./run.sh & # run Galaxy on 8080 + make client-watch & # watch for client changes + export GALAXY_TEST_EXTERNAL=http://localhost:8080/ # Target tests at server. + . .venv/bin/activate # source the virtualenv so can skip run_tests.sh. + pytest lib/galaxy_test/selenium/test_workflow_editor.py::WorkflowEditorTestCase::test_data_input + +Note About Selenium Tests: + +If using a local selenium driver such as a Chrome or Firefox based one +either chromedriver or geckodriver needs to be installed an placed on +the PATH. + +More information on geckodriver can be found at +https://github.com/mozilla/geckodriver and more information on +chromedriver can be found at +https://sites.google.com/a/chromium.org/chromedriver/. + +By default Galaxy will check the PATH for these and pick +whichever it finds. This can be overridden by setting +GALAXY_TEST_SELENIUM_BROWSER to either FIREFOX, CHROME, or something +more esoteric (including OPERA and PHANTOMJS). + +If PyVirtualDisplay is installed Galaxy will attempt to run this +browser in a headless mode. This can be disabled by setting +GALAXY_TEST_SELENIUM_HEADLESS to 0 however. + +Selenium can also be setup a remote service - to target a service set +GALAXY_TEST_SELENIUM_REMOTE to 1. The target service may be configured +with GALAXY_TEST_SELENIUM_REMOTE_PORT and +GALAXY_TEST_SELENIUM_REMOTE_HOST. By default Galaxy will assume the +remote service being targetted is CHROME - but this can be overridden +with GALAXY_TEST_SELENIUM_BROWSER. + +In this remote mode, please ensure that GALAXY_TEST_HOST is set to a +host that is accessible from the Selenium host. By default under Linux +if GALAXY_TEST_SELENIUM_REMOTE is set, Galaxy will set this to be the IP +address Docker exposes localhost on to its child containers. This trick +doesn't work on Mac OS X and so GALAXY_TEST_HOST will need to be crafted +carefully ahead of time. + +For Selenium test cases a stack trace is usually insufficient to diagnose +problems. For this reason, GALAXY_TEST_ERRORS_DIRECTORY is populated with +a new directory of information for each failing test case. This information +includes a screenshot, a stack trace, and the DOM of the currently rendered +Galaxy instance. The new directories are created with names that include +information about the failed test method name and the timestamp. By default, +GALAXY_TEST_ERRORS_DIRECTORY will be set to database/errors. + +The Selenium tests seem to be subject to transient failures at a higher +rate than the rest of the tests in Galaxy. Though this is unfortunate, +they have more moving pieces so this is perhaps not surprising. One can +set the GALAXY_TEST_SELENIUM_RETRIES to a number greater than 0 to +automatically retry every failed test case the specified number of times. + +External Tests: + +A small subset of tests can be run against an existing Galaxy +instance. The external Galaxy instance URL can be configured with +--external_url. If this is set, either --external_master_key or +--external_user_key must be set as well - more tests can be executed +with --external_master_key than with a user key. + +Extra options: + + --verbose_errors Force some tests produce more verbose error reporting. + --no_cleanup Do not delete temp files for Python functional tests + (-toolshed, -framework, etc...) + --debug On python test error or failure invoke a pdb shell for + interactive debugging of the test + --report_file Path of HTML report to produce (for Python Galaxy + functional tests). If not given, a default filename will + be used, and reported on stderr at the end of the run. + --xunit_report_file Path of XUnit report to produce (for Python Galaxy + functional tests). + --skip-venv Do not create .venv (passes this flag to + common_startup.sh) + --dockerize Run tests in a pre-configured Docker container (must be + first argument if present). + --db For use with --dockerize, run tests using partially + migrated 'postgres', 'mysql', or 'sqlite' databases. + --external_url External URL to use for Galaxy testing (only certain + tests). + --external_master_key Master API key used to configure external tests. + --external_user_key User API used for external tests - not required if + external_master_key is specified. + --skip_flakey_fails Skip flakey tests on error (sets + GALAXY_TEST_SKIP_FLAKEY_TESTS_ON_ERROR=1). + +Environment Variables: + +In addition to the above command-line options, many environment variables +can be used to control the Galaxy functional testing processing. Command-line +options above like (--external_url) will set environment variables - in such +cases the command line argument takes precedent over environment variables set +at the time of running this script. + +Functional Test Environment Variables + +GALAXY_TEST_DBURI Database connection string used for functional + test database for Galaxy. +GALAXY_TEST_INSTALL_DBURI Database connection string used for functional + test database for Galaxy's install framework. +GALAXY_TEST_INSTALL_DB_MERGED Set to use same database for Galaxy and install + framework, this defaults to True for Galaxy + tests an False for shed tests. +GALAXY_TEST_DB_TEMPLATE If GALAXY_TEST_DBURI is unset, this URL can be + retrieved and should be an sqlite database that + will be upgraded and tested against. +GALAXY_TEST_TMP_DIR Temp directory used for files required by + Galaxy server setup for Galaxy functional tests. +GALAXY_TEST_SAVE Location to save certain test files (such as + tool outputs). +GALAXY_TEST_EXTERNAL Target an external Galaxy as part of testing. +GALAXY_TEST_JOB_CONFIG_FILE Job config file to use for the test. +GALAXY_CONFIG_MASTER_API_KEY Master or admin API key to use as part of + testing with GALAXY_TEST_EXTERNAL. +GALAXY_TEST_USER_API_KEY User API key to use as part of testing with + GALAXY_TEST_EXTERNAL. +GALAXY_TEST_VERBOSE_ERRORS Enable more verbose errors during API tests. +GALAXY_TEST_UPLOAD_ASYNC Upload tool test inputs asynchronously (may + overwhelm sqlite database). +GALAXY_TEST_RAW_DIFF Don't slice up tool test diffs to keep output + managable - print all output. (default off) +GALAXY_TEST_DEFAULT_WAIT Max time allowed for a tool test before Galaxy + gives up (default 86400) - tools may define a + maxseconds attribute to extend this. +GALAXY_TEST_TOOL_DEPENDENCY_DIR tool dependency dir to use for Galaxy during + functional tests. +GALAXY_TEST_FILE_DIR Test data sources (default to + test-data,https://github.com/galaxyproject/galaxy-test-data.git) +GALAXY_TEST_DIRECTORY /test +GALAXY_TEST_TOOL_DATA_PATH Set to override tool data path during tool + shed tests. +GALAXY_TEST_FETCH_DATA Fetch remote test data to + GALAXY_TEST_DATA_REPO_CACHE as part of tool + tests if it is not available locally (default + to True). Requires git to be available on the + command-line. +GALAXY_TEST_DATA_REPO_CACHE Where to cache remote test data to (default to + test-data-cache). +GALAXY_TEST_SKIP_FLAKEY_TESTS_ON_ERROR + Skip tests annotated with @flakey on test errors. +HTTP_ACCEPT_LANGUAGE Defaults to 'en' +GALAXY_TEST_NO_CLEANUP Do not cleanup main test directory after tests, + the deprecated option TOOL_SHED_TEST_NO_CLEANUP + does the same thing. +GALAXY_TEST_HOST Host to use for Galaxy server setup for + testing. +GALAXY_TEST_PORT Port to use for Galaxy server setup for + testing. +GALAXY_TEST_TOOL_PATH Path defaulting to 'tools'. +GALAXY_TEST_SHED_TOOL_CONF Shed toolbox conf (defaults to + config/shed_tool_conf.xml) used when testing + installed to tools with -installed. +GALAXY_TEST_HISTORY_ID Some tests can target existing history ids, this option + is fairly limited and not compatible with parrallel testing + so should be limited to debugging one off tests. +TOOL_SHED_TEST_HOST Host to use for shed server setup for testing. +TOOL_SHED_TEST_PORT Port to use for shed server setup for testing. +TOOL_SHED_TEST_FILE_DIR Defaults to lib/tool_shed/test/test_data. +TOOL_SHED_TEST_TMP_DIR Defaults to random /tmp directory - place for + tool shed test server files to be placed. +TOOL_SHED_TEST_OMIT_GALAXY Do not launch a Galaxy server for tool shed + testing. + +Unit Test Environment Variables + +GALAXY_TEST_INCLUDE_SLOW - Used in unit tests to trigger slower tests that + aren't included by default with --unit/-u. + diff --git a/doc/source/dev/writing_tests.md b/doc/source/dev/writing_tests.md index 9a6c7d3fd28..9e3122855da 100644 --- a/doc/source/dev/writing_tests.md +++ b/doc/source/dev/writing_tests.md @@ -1,6 +1,6 @@ # Writing Tests for Galaxy -## Sources of Documentation +## Other Sources of Documentation Over the last several years, the most up-to-date documentation on the structure and running of Galaxy tests has been in the help text for @@ -10,3 +10,357 @@ High-level information on Galaxy's CI process can be found in the and the corresponding [YouTube playlist](https://bit.ly/gx-arch-vids). Some more specifics on running and writing Galaxy client unit tests can be found in ``client/README.md`` of the Galaxy codebase. + +## An Overview of Galaxy Tests + +Galaxy has many test suites and frameworks. A potentially overwhelming +question at first is, *where does a given test belong?* What testing suite +or framework should it be added to. There few questions that may +be able to help find the right documentation for a given test one wishes +to write. + +***Does this test require a running server and database to execute?*** + + - **No** + + If no, this test should probably be implemented as a Galaxy unit test. Unit tests generally, and Galaxy ones specifically, are especially useful + for complex components that are well architected to be tested in isolation. + The best unit tests are unit tests that shield a lot of their potential complexity from their + consumers and components that do not have a lot of dependencies - especially + on the database or a web server. + + ***Is the component under test a client (ES6) or backend (Python) component?*** + + - **Client/ES6** + + These tests should be placed in ``client/src` directly and executed + via Jest, check out [Frontend/ES6 Unit Tests](#es6_unit) below for more + information. + + - **Backend/Python** + + These tests should be placed in ``test/unit` or doctests and + executed via pytest, check out [Backend/Python Unit Tests](#python_unit) below for more + information. + + - **Yes** + + In this case you're looking at some sort of functional test that requires + a running Galaxy server and the Galaxy database. All of these tests are + currently implemented in Python. + + ***Does this test require the Galaxy web interface?*** + + - **No** + + Most of the time, we've found that these tests are best when + they exercise the Galaxy API to drive the test. These tests + are all Python tests executed by pytest. The testing frameworks + provide everything you need to spin up a Galaxy instance and + communicate with its API to invoke the component under test + and write expectations about what should result from the API. + There are two different (but very related) frameworks to do this + and the choice between which is appropriate comes down to + the following question. + + ***Does this test require a special configuration of Galaxy?*** + + - **No** + + In this case simple Galaxy API tests are likely the most + appropriate way to implement the desired test, these tests + are located in ``lib/galaxy_test/api``. Checkout + [API Tests](#api) below for more information. + + - **Yes** + + Tests that require a custom Galaxy with a very specific + configuration to be spun up for the tests are called + Galaxy integration tests and are located in ``test/integration``. + Checkout [Integration Tests](#integration) below for more + information. + + - **Yes** + + The tests that exercise the Galaxy user interface and require + a functional Galaxy server are all use Selenium to drive interaction + with the Galaxy web interface. There are two frameworks or suites + available for building tests like this and they both provide high + level access to the Galaxy API like the tests above. The frameworks + also take care of starting the Galaxy server. + + Like above, the choice between these two different frameworks + comes down to the answer to the following question. + + ***Does this test require a special configuration of Galaxy?*** + + - **No** + + These tests should be placed into ``lib/galaxy_test/selenium`` + and implemented using the [Selenium Tests](#selenium) + framework describe below. + + - **Yes** + + Tests that require both a very specific Galaxy configuration + as well as the ability to drive a running Galaxy web interface + should be place into ``test/integration_selenium``. Checkout + the [Selenium Integration Tests](#selenium_integration) + information below for more information. + +## Backend/Python Unit Tests + +These are Python unit tests either defined inside of ``test/unit`` or +via doctests within a Python component. These should generally not require +a Galaxy instance and should quickly test just a component or a few +components of Galaxy's backend code. + +### doctests to stand-alone tests? + +doctests tend to be more brittle and more restrictive. I (@jmchilton) +would strongly suggest writing stand-alone unit testing files separate +from the code itself unless the tests are so clean and so isolated they +serve as high-quality documentation for the component under test. + +### Slow 'Unit' Tests + +There are tests in Galaxy that test integration with external sources +that do not require a full Galaxy server. While these aren't really +"unit" tests in a traditional sense, they are unit tests from a Galaxy +perspective because they do not depend on a Galaxy server. + +These tests should be marked as requiring the environment variable +``GALAXY_TEST_INCLUDE_SLOW`` to run. + +### Continuous Integration + +The Python unit tests are ran against each pull request to Galaxy using +CircleCI. If any of these tests fail, the pull request will be marked +red. This test suite is moderately prone to having tests fail that are +unrelated to the pull request being tested, if this test suite fails on +a pull request with changes that seem to be unrelated to the pull request - +ping the Galaxy committers on the pull request and request a re-run. The +CircleCI test definition for these tests is located in ``.circleci/config.yml`` +below Galaxy's root. + +## Frontend/ES6 Unit Tests + +Detailed information on writing Galaxy client tests can be found in +[client/README.md](https://github.com/galaxyproject/galaxy/blob/dev/client/README.md#client-side-unit-testing). + +### Continuous Integration + +The client tests are ran against each pull request to Galaxy using +GitHub actions. If any of these tests fail, the pull request will be marked +red. This test suite is moderately prone to having tests fail that are +unrelated to the pull request being tested, if this test suite fails on +a pull request with changes that seem to be unrelated to the pull request - +ping the Galaxy committers on the pull request and request a re-run. The +Github actions workflow definition for these tests is located in +``.github/workflows/jest.yaml`` below Galaxy's root. + +## Tool Framework Tests + +A great deal of the complexity and interface exposed to Galaxy plugin +developers, comes in the form of Galaxy tool wrapper definition files. +Likewise, a lot of the legacy behavior Galaxy needs to maintain is +maintained for older tool definitions. For this reason, a lot of Galaxy's +complex internals can just be tested by simply running a tool test. +Obviously, Galaxy is much more complex than this but a surprising amount +of Galaxy's tests are simply tool tests. This suite of tools that +run and have their tests exercised is called the "Tool Framework Tests" +or simply "Framework Tests". + +Adding a tool test is as simple as finding a related tool in the sample +tools (``test/functional/tools``) and adding a test block to that file +or adding a new tool to this directory and referencing it in the +sample tool configuration XML (``test/functional/tools/samples_tool_conf.xml``). + +For information on writing Galaxy Tool Tests can be found in Planemo's +documentation - for instance in the [Test-Driven Development](https://planemo.readthedocs.io/en/latest/writing_advanced.html#test-driven-development) +section. + +### Continuous Integration + +The Tool framework tests are ran against each pull request to Galaxy using +GitHub actions. If any of these tests fail, the pull request will be marked +red. This test suite is fairly stable and typically there are not +transiently failed tests unrelated to the pull request being tested. The +Github actions workflow definition for these tests is located in +``.github/workflows/framework.yaml`` below Galaxy's root. + +## API Tests + +These tests are located in ``lib/galaxy_test/api`` and test various aspects +of the Galaxy API and test general backend aspects of Galaxy using the API. + +### An Example ``lib/galaxy_test/api/test_roles.py`` + +This test file shows a fairly typical API test. It demonstrates the +basic structure. It demonstrates how to ``GET`` and ``POST`` against the +API. It demonstrates how to use both a typical user and admin user-only + functionality. + +### Populating Test Data with ``lib/galaxy_test/base/populators.py`` + +The ``test_roles.py`` example above also creates a ``DatasetPopulator`` +object that it uses to get some common information from the configured +Galaxy server under test. Populators are used extensively throughout +API tests as well as integration and Selenium tests to both populate +data to test (histories, workflows, collections, libraries, etc..) +as well as access information from the Galaxy server (e.g. fetch +information from datasets, users, Galaxy's configuration, etc..). + +Populators and API tests in general make heavy use of the [requests +library](https://requests.readthedocs.io/en/master/) for Python. + +### API Test Assertions + +There is a module of common assertions ``galaxy_test.base.api_asserts`` +used to check API request status codes, dictionary content, and Galaxy +specific error messages. + +### Continuous Integration + +The API tests are ran against each pull request to Galaxy using +GitHub actions. If any of these tests fail, the pull request will be marked +red. This test suite is fairly stable and typically there are not +transiently failed tests unrelated to the pull request being tested. The +Github actions workflow definition for these tests is located in +``.github/workflows/api.yaml`` below Galaxy's root. + +## Integration Tests + +These tests are located in ``test/integration``. These tests have access +to all the same API utilities as API tests described above, but can access +Galaxy internals and may define hooks for configuring Galaxy in certain ways +during startup. + +Galaxy integration tests in some ways are more powerful than API tests - +they can both control Galaxy's configuration and can access Galaxy's +internals. However, this power comes at a real cost - each test case must +spin up its own Galaxy server (a relatively expensive operation) and the +tests cannot be executed against external Galaxy servers (it wouldn't make +sense to given these custom hooks during configuration of the server). +For these reasons, we bundle up Galaxy API tests for use in deployment +testing of production setups for instance and Galaxy API tests should be +implemented whenever possible and integration tests implemented only when +an API test is not possible or practical. + +Integration tests can make use of dataset populators and API assertions +as described above in the API test documentation. It is worth reviewing +that documentation before digging into integration examples. + +### An Example ``test/integration/test_quotas.py`` + +This is a really simple example that does some testing with the Quotas +API of Galaxy. This API is off by default so it must be enabled for the +the test. The top of the test file demonstrates both how to create an +integration test and how to modify Galaxy's configuration for the test. + +```python +#... +from galaxy_test.driver import integration_util + + +class QuotaIntegrationTestCase(integration_util.IntegrationTestCase): + require_admin_user = True + + @classmethod + def handle_galaxy_config_kwds(cls, config): + config["enable_quotas"] = True + + #... +``` + +Integration test cases extend the ``IntegrationTestCase`` class defined in +the ``galaxy_test.driver.integration_util`` module (located in +``lib/galaxy_test/driver/integration_uti.py`` below Galaxy's root). + +The ``require_admin_user`` option above tell the test framework that the +default user configured for API interactions must be an admin user. + +This example overrides Galaxy's configuration using the +``handle_galaxy_config_kwds`` class method. This method is called before +a Galaxy server is created and passed the testing server's default +configuration as the ``config`` argument to that class method. This +``config`` object is effectively the Python representation of the Galaxy +configuration file (``galaxy.yml``) used to start the Python server. +Almost anything you can do in ``galaxy.yml``, you can modify the Galaxy +server to do using the same keys. Examples of various ways integration tests +have modified this dictionary include setting up custom object stores +(e.g. ``objectstore/test_mixed_store_by.py``), +setting up non-local job runners (e.g. ``test_cli_runners.py``), setting +up custom job destinations (e.g. ``test_job_recovery.py``), and configuring +Galaxy for tool shed operations (e.g. ``test_repository_operations.py``). + +There may be cases where an integration test is used not to allow +some custom configuration of Galaxy but to access Galaxy's internals. +Integration tests have direct access to Galaxy's ``app`` object via +``self._app`` and direct access to the database as a result. An example of +a test that uses these is ``test_workflow_refactoring.py``. This test +required accessing the way workflow steps are stored in the database and +not just how they are serialized by the API and so tests database models +directly. Generally though this type of usage should be avoided. + +### Continuous Integration + +The Integration tests are ran against each pull request to Galaxy using +GitHub actions. If any of these tests fail, the pull request will be marked +red. This test suite is moderately prone to having tests fail that are +unrelated to the pull request being tested, if this test suite fails on +a pull request with changes that seem to be unrelated to the pull request - +ping the Galaxy committers on the pull request and request a re-run. The +GitHub actions workflow definition for these tests is located in +``.github/workflows/integration.yaml`` below Galaxy's root. + +## Selenium Tests + +These are full stack tests meant to test the Galaxy UI with real +browsers and are located in ``lib/galaxy_test/selenium``. + +### Continuous Integration + +The Selenium tests are ran against each pull request to Galaxy using +GitHub actions. If any of these tests fail, the pull request will be marked +red. This test suite is moderately prone to having tests fail that are +unrelated to the pull request being tested, if this test suite fails on +a pull request with changes that seem to be unrelated to the pull request - +ping the Galaxy committers on the pull request and request a re-run. The +GitHub actions workflow definition for these tests is located in +``.github/workflows/selenium.yaml`` below Galaxy's root. + +## Selenium Integration Tests + +These tests are located ``test/integration_selenium`` and simply +combine the capabilities of Selenium tests and Integration tests +(both described above) into test cases that can do both. There +are no new capabilities or gotchas of this test suite beyond +what is described above these sections. + +A quintessential example is ``test/integration_selenium/test_upload_ftp.py``. +Testing the FTP capabilities of the user interface requires both +Selenium to drive the test case and a custom Galaxy configuration +that mocks out an FTP directory and points the Galaxy server at it +with various options (``ftp_upload_dir``, ``ftp_upload_site``). + +### Continuous Integration + +The Selenium integration tests are ran against each pull request to Galaxy using +GitHub actions. If any of these tests fail, the pull request will be marked +red. This test suite is moderately prone to having tests fail that are +unrelated to the pull request being tested, if this test suite fails on +a pull request with changes that seem to be unrelated to the pull request - +ping the Galaxy committers on the pull request and request a re-run. The +GitHub actions workflow definition for these tests is located in +``.github/workflows/selenium_integration.yaml`` below Galaxy's root. + +## Running Python Tests + +The best information about how to run Galaxy's Python tests can be +found in the help output of ``run_tests.sh --help``. + +```eval_rst +.. include:: run_tests_help.txt + :literal: +```