Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
- New upload tab for "Collections".
- This has a new option to select "Collection Type" in footer.
- Collection upload geared toward homogeneous datasets - must specify genome and file format for all files.
- New button "Build" after all individual datasets have finished their upload to the server.
- Clicking "Build" launches the appropriate collection builder for selected collection type.
- By default, uploading a collection hides the original uploaded datasets (can be unchecked).
Implement tests for all uploaded collection types.
- Use newer abstraction in the simplest upload test - verify the default extension (auto -> sam) and dbkey (?) also.
- Add a test case that explicitly sets the extension of an upload and verifies.
- Add a test case that explicitly sets a genome and verifies.
- Add a test case that tests setting ext for all uploads and verifies.
- Add a test case that tests setting genome for all uploads and verifies.
See documentation added to run_tests.sh. This can be configured to target local web browsers or Selenium remote services as detailed in that script.
Individual tests can be executed with nosetests directly or groups of tests can share a common test Galaxy spin up when using ./run_tests.sh. If GALAXY_TEST_EXTERNAL=<url> is set - it will be respected and Galaxy will not be spun up (remember the URL needs to be reachable from the Selenium server inside the Docker container).
Every test failure writes a current screenshot of Galaxy to database/test_errors.
I will be honest that the desire to move toward selenium is based solely on failing to get the Casper tests to pass consistently due to bugs causing segfaults in the underlying tools (probably casperjs or phantomjs?). Over the past year I have tried multiple versions of dependencies, etc... and it never works out for me. Likewise Martin has never been able to get the tests to run consistently under Jenkins. I don't think Python or JavaScript is inherently better, this wasn't based on a personal preference about what kind of test I want to write.
Despite this being the primary reason, there are clear benefits to Selenium. It tests the actual web browsers we support, generates screenshots, has better documented and more robust tooling, can scale across clusters. Less clear, but certainly still a benefit of being Python based is that it fits with the rest of the test framework more cleanly than CasperJS.
Finally Carl's last words on CasperJS were "Ditch it".
The following specific tests were added or replaced:
- Implemented tour testing via Selenium (walk the two working stock tours and verify elements avaiable and clicks are valid). This is how I discovered #3206.
- Added completely new workflow GUI test (basic creation from URL and in editor).
- Replaced broken CasperJS registeration tests (text and expectations now wrong) with a Selenium variant that works against dev.
- Replaced broken anonymous history CasperJS tests with working Selenium tests.
- Replaced broken upload CasperJS tests with working Selenium tests.
- Replaced history options CasperJS tests with Selenium tests.
- Replaced login CasperJS tests with Selenium tests.
- Replaced history-share-tests.js with test_history_options.py
- Replaced history-panel-tests.js with test_history_panel.py
- Replaced a big part of hda-state-tests.js with test_history_dataset_state.py
I'm confident these utilities represent a sharable and higher-level abstraction around functional testing of Galaxy that can be used outside of Galaxy's testing framework. So a subset of the functional test stuff is in a separate module with minimal dependencies that I intend to make stand alone and pip installable like galaxy-lib.
This module consists of:
- Sizzle stuff in its own package. This code allows Selenium to reason with jQuery selectors instead of vanilla CSS selectors.
- Functionality for creating a Selenium driver and virtual display.
- The ``HasDriver`` mixin - this provides higher level navigation utilities not specifically tied to Galaxy.
- A package with abstractions for navigating Galaxy. This provides a higher-level interfactor for things such as logging in and out, registering a user, navigating menus, fetching Galaxy style tooltips and error messages, and walking Galaxy tours.
In order to demonstrate this new module is useful outside of Galaxy tests - I've included a CLI package to ease building argparse utilities around these abstractions and implemented a simple demonstration script that walks a Galaxy tour and dumps screenshots of Galaxy at each step to a folder. This is more of a demonstration of the utilities than an actual polished end user tool - though it might be helpful in linting and debugging tours.
In the future I hope to extend this scripting to implement
- Periodic deployment testing to ensure things like Jupyter work in production.
- A best practice recipe for Galaxy QA testing by deployers.
- An utility to generate dozens of screenshots of various aspects of Galaxy so we can quickly visually inspect ever part of the GUI before big releases.
xref https://github.com/galaxyproject/starforge/pull/115 xref #1419
Nicola Soranzo