Before, the code would emit extra queries per each library to
find out whether the current user has permissions on it.
By pre-fetching all of them in one query and passing around as
a dict object we should see a considerable speedup on Galaxies
with many libraries.
Prefetch shared user information as counts instead of objects and do so in initial query to eliminate an extra 15 SQL queries per page and load less data related to sharing.
Replace columns that would cause HDA information to be joined into the query (history size and HDA state counts) with a spinner that will be fetched in subsequent queries on the client end. There can hundreds of thousands of datasets per history - this information shouldn't be summarized to get the initial page to render - it can be fetched one history at a time once the page is rendered.
This commit also adds a new column "Items" that corresponds to the next HID - and that I think is a good summary of the "history size" before the dataset state information is loaded and even gives additional information because that count includes collections. This also renders state information for deleted and hidden datasets that was previously missing and could cause confusion. That said I don't like the dataset summaries - I'd rather just have the item count and then job state summaries.
For all four grid types (histories, viz, workflows, and pages), this computes rating average on the server and prefetches any annotations. This eliminates at least two extra queries for response element in the grid. See comment in the history controller for why I am fairly confident this is a good idea for annotations but tags are less obvious - these grids still go back to the postgres server multiple times per rendered item to render tags.
I'm confident we should either subqueryload, joinedload, or upgrade to sqlalchemy 2.2 and selectinload (http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#sqlalchemy.orm.selectinload) the tags as well - but I'm not sure which without being able to hack on a usegalaxy.org.
This also brings in less of the user model (only username instead of all of it) to reduce over-the-wire transmission of unneeded data from postgres to Galaxy.
For workflows we were LEFT OUTER JOIN-ing on the steps of the latest workflow - so we were bringing back a lot of extra rows for data that was completely unused. In light of this, it makes perfect sense to me why published workflows were the slowest of these and I suspect they will all be equally performant after this change (modulo the number of rows in the tables and the number of rows rendered).
Don't LEFT OUTER JOIN workflow against its steps to produce step count - this causes a lot of duplicated workflow data and fetches step data not needed to just produce a step count.
Tweak pre-fetching for workflows shared with user also.
Nicola Soranzo