The queryname sorted datatype (BamQuerynameSorted) ensures that the file is
queryname ordered. The BamInputSorted datatype can be used to describe the
output of aligners, which usually keep mate pairs adjacent. The BamInputSorted
datatype allows using tools that require mates to be adjacent, without
requiring an explicit sorting step. This can reduce the time and space
required for duplicate marking tools, HiC tools and structural variant
detection tools that require mates to be grouped together in an alignment file.
The BamQuerynameSorted has a converter that works for all datatypes that can be
consumed by samtools. There is no converter for BamInputSorted, since
BamQuerynameSorted is valid input for tools requiring BamInputSorted
input.
I'm targeting 18.01 since I think it would be very unfortunate if we start
annotating aligners with the very loose BamNative datatype, since this datatype
does not make any promises about sort order at all.
This closes https://github.com/galaxyproject/galaxy/issues/5497, goes
a long way towards the problems mentioned in https://github.com/galaxyproject/galaxy/issues/5496
and would make https://github.com/galaxyproject/tools-iuc/pull/1732 and https://github.com/galaxyproject/tools-iuc/pull/1591 better
Before, the code would emit extra queries per each library to
find out whether the current user has permissions on it.
By pre-fetching all of them in one query and passing around as
a dict object we should see a considerable speedup on Galaxies
with many libraries.
Prefetch shared user information as counts instead of objects and do so in initial query to eliminate an extra 15 SQL queries per page and load less data related to sharing.
Replace columns that would cause HDA information to be joined into the query (history size and HDA state counts) with a spinner that will be fetched in subsequent queries on the client end. There can hundreds of thousands of datasets per history - this information shouldn't be summarized to get the initial page to render - it can be fetched one history at a time once the page is rendered.
This commit also adds a new column "Items" that corresponds to the next HID - and that I think is a good summary of the "history size" before the dataset state information is loaded and even gives additional information because that count includes collections. This also renders state information for deleted and hidden datasets that was previously missing and could cause confusion. That said I don't like the dataset summaries - I'd rather just have the item count and then job state summaries.
For all four grid types (histories, viz, workflows, and pages), this computes rating average on the server and prefetches any annotations. This eliminates at least two extra queries for response element in the grid. See comment in the history controller for why I am fairly confident this is a good idea for annotations but tags are less obvious - these grids still go back to the postgres server multiple times per rendered item to render tags.
I'm confident we should either subqueryload, joinedload, or upgrade to sqlalchemy 2.2 and selectinload (http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#sqlalchemy.orm.selectinload) the tags as well - but I'm not sure which without being able to hack on a usegalaxy.org.
This also brings in less of the user model (only username instead of all of it) to reduce over-the-wire transmission of unneeded data from postgres to Galaxy.
For workflows we were LEFT OUTER JOIN-ing on the steps of the latest workflow - so we were bringing back a lot of extra rows for data that was completely unused. In light of this, it makes perfect sense to me why published workflows were the slowest of these and I suspect they will all be equally performant after this change (modulo the number of rows in the tables and the number of rows rendered).
Don't LEFT OUTER JOIN workflow against its steps to produce step count - this causes a lot of duplicated workflow data and fetches step data not needed to just produce a step count.
Tweak pre-fetching for workflows shared with user also.
mvdbeek