Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
- Flakey was throwing nose's SkipTest but the Selenium framework was catching unittest's, fix that to use unittest version in both places.
- There problematic test is still throwing exceptions in the tearDown which is still causing build failures, skip the whole test.
Now that #5701 has been merged we have some more clues on the problem with this test. It is dying on the second upload being changed here. I suspect what is happening is that we are waiting on a history to become okay but we make that check before the first upload has even finished - so there is some chance the history is okay because it is empty - so we are reopening the home page in the middle of the first upload and this is causing something odd to happen - something is popping up an empty alert. I'm not sure what is causing that alert but it isn't what we are testing with this test so it is best to really just wait on the uploads - as we do in other tests (see test_uploads.py for instance).
history_panel_wait_for_hid_ok is a stronger assertion and is generally more robust and has better error reporting anyway so this should be a stability win all around.
Prefetch shared user information as counts instead of objects and do so in initial query to eliminate an extra 15 SQL queries per page and load less data related to sharing.
Replace columns that would cause HDA information to be joined into the query (history size and HDA state counts) with a spinner that will be fetched in subsequent queries on the client end. There can hundreds of thousands of datasets per history - this information shouldn't be summarized to get the initial page to render - it can be fetched one history at a time once the page is rendered.
This commit also adds a new column "Items" that corresponds to the next HID - and that I think is a good summary of the "history size" before the dataset state information is loaded and even gives additional information because that count includes collections. This also renders state information for deleted and hidden datasets that was previously missing and could cause confusion. That said I don't like the dataset summaries - I'd rather just have the item count and then job state summaries.
- Add tests for creating a collection from a library.
- Add more class names to library GUI to improve selectors for tests.
- Automatically select whichever radio box makes sense in library collection builder dialog based on whether there is a selection or not.
- Replace test using Cut1 with two tests each using framework test tools loaded with Selenium tests by default.
- Create a symbolic link so that tour generator is configured with default test webhooks.
- Establish and use components and navigates_galaxy.py abstractions for tour generation and navigation.
- In one of the tool tests actually walk the tour (previously we only verified the initial tour popover).
Dannon Baker