Allow the rules DSL & GUI component to operate on existing collections to allow filtering, sorting, modifying identifiers and general re-organization of existing collections (e.g. the outputs of tools). Implementing this as a collection operation tool so that it should be executable interactively in the tool form and in a batch fashion as part of workflow executions.
For this to be tracked properly as a tool execution and to work properly in the tool form, I've implemented a new tool framework and tool form parameter type called "rules".
This can thought of as a more GUI friendly alternative to my proposed collection operations that consumed JavaScript expressions.
This includes API tests for both tool and workflow execution of the new tool as well as Selenium tests for tool form execution, workflow editor interactions, and workflow running.
Re-use test_data specification used by many of the API workflow tests and a workflow used by another test. Add abstractions and DOM element for mapping labeled test data to tool form inputs.
- More stuff tested:
- Connecting and disconnecting simple terminals as well as mapping terminals (ribbons).
- Rendering simple workflows, workflow with output collections, workflow with mapping, and workflow with a subworkflow.
- Collapsing side panels in the workflow editor.
- Small tweaks to editor DOM.
- More screenshots.
- More stuff tested:
- Setting input name and annotation, saving, reloading.
- Simple tests for collection inputs in addition to data inputs.
- A simple workflow renders its connections.
- More screenshots.
- Expanded use of navigation.yml for abstraction description of elements.
- More abstractions for probing the workflow in the editor - should serve as a basis for additional tests.
- Decorate editor DOM to allow finding elements in reproducible ways (enables testing, but should help with tours as well).
- Make editor tests more robust by:
- Doing more appropriate waiting, instead of adding sleeps for jQuery interactions.
- Not assuming each test occurs with a fresh user (and hence workflow list).
Vue-based component for defining collections by applying rules to a list of files or more general spreadsheet style information (e.g. sample sheets or tabular data from data sources containing URL or FTP file paths for files along with metadata). The widget is fairly complex but very broadly is broken into two panes - one to preview how rules are applied to build up tabular data defining collections (each row corresponding to a file with columns for metadata and such) and one that displays defined rules and allows for editing of these rules and creation of new ones.
The goal behind defining rules this way instead of allowing the user to interact with the spreadsheet display directly is to enable scaling up collection creation. If a user wishes to upload hundreds of datasets - interacting with a widget directly for each input doesn't scale well and would be error prone. If a user wishes to upload hundreds of thousands of datasets - even loading this information in the GUI may not scale (though I've been impressed with the performance so far of this approach) and so we can potentially just display a preview of some of the rows and process the final set of rules on the backend.
Since we can handle an arbitrary number of columns this way, we can define multiple list identifiers per file and so we can easily construct nested lists. Hence this allows creation of not just potentially larger collections but arbitrarily complex lists as well. Paired identifiers via indicator columns are also implemented.
In order to operate over lists of datasets directly - the multi-select history widget now has a new option "Build Collection from Rules" along side the other collection builders. This mode uses the well established dataset collection API to build collections from HDAs.
In order to operate on lists of FTP files or URLs - the upload widget has a new tab "Rule-based" tab that allows users to paste in tabular data or select a history dataset and then send this tabular data to the new builder widget. This will be extended to include FTP directories for instance over time. This mode uses the new data fetch API to build collections and handle uploads of arbitrary collections of files.
The preview of the tabular data generated via rules is done via [Handsontable](https://handsontable.com/) - a JavaScript spreadsheet widget with a VueJS [wrapper component](https://github.com/handsontable/vue-handsontable-official). This turns out to be a fairly nice application for reactive components - as rules are added or modified the spreadsheet just naturally updates. In my hands the widget scales very nicely - I've uploaded files with tens of thousands of rows and rules modifying the data and changing the spreadsheet do not seem to cause siignificant delays in the web browser.
- Flakey was throwing nose's SkipTest but the Selenium framework was catching unittest's, fix that to use unittest version in both places.
- There problematic test is still throwing exceptions in the tearDown which is still causing build failures, skip the whole test.
Now that #5701 has been merged we have some more clues on the problem with this test. It is dying on the second upload being changed here. I suspect what is happening is that we are waiting on a history to become okay but we make that check before the first upload has even finished - so there is some chance the history is okay because it is empty - so we are reopening the home page in the middle of the first upload and this is causing something odd to happen - something is popping up an empty alert. I'm not sure what is causing that alert but it isn't what we are testing with this test so it is best to really just wait on the uploads - as we do in other tests (see test_uploads.py for instance).
history_panel_wait_for_hid_ok is a stronger assertion and is generally more robust and has better error reporting anyway so this should be a stability win all around.
Prefetch shared user information as counts instead of objects and do so in initial query to eliminate an extra 15 SQL queries per page and load less data related to sharing.
Replace columns that would cause HDA information to be joined into the query (history size and HDA state counts) with a spinner that will be fetched in subsequent queries on the client end. There can hundreds of thousands of datasets per history - this information shouldn't be summarized to get the initial page to render - it can be fetched one history at a time once the page is rendered.
This commit also adds a new column "Items" that corresponds to the next HID - and that I think is a good summary of the "history size" before the dataset state information is loaded and even gives additional information because that count includes collections. This also renders state information for deleted and hidden datasets that was previously missing and could cause confusion. That said I don't like the dataset summaries - I'd rather just have the item count and then job state summaries.
- Add tests for creating a collection from a library.
- Add more class names to library GUI to improve selectors for tests.
- Automatically select whichever radio box makes sense in library collection builder dialog based on whether there is a selection or not.
John Chilton