Fetch HIDs for these datasets all at once and flush once for all assignments. Creating jobs from 4 threads each job producing 10 outputs - this resulted in reducing the runtime of this portion of the code by 90% and saving around 20 seconds per job.
Before:
galaxy.tools.actions INFO 2015-11-30 15:53:46,235 Add outputs to history (22494.829 ms)
galaxy.tools.actions INFO 2015-11-30 15:53:47,295 Add outputs to history (22239.062 ms)
galaxy.tools.actions INFO 2015-11-30 15:53:48,320 Add outputs to history (23781.079 ms)
galaxy.tools.actions INFO 2015-11-30 15:53:51,490 Add outputs to history (21820.567 ms)
galaxy.tools.actions INFO 2015-11-30 15:54:24,506 Add outputs to history (25353.837 ms)
After:
galaxy.tools.actions INFO 2015-11-30 16:08:47,640 Add outputs to history (2781.675 ms)
galaxy.tools.actions INFO 2015-11-30 16:08:47,860 Add outputs to history (3177.738 ms)
galaxy.tools.actions INFO 2015-11-30 16:08:48,776 Add outputs to history (2425.528 ms)
galaxy.tools.actions INFO 2015-11-30 16:09:02,942 Add outputs to history (2579.022 ms)
Should work when mapping over collections or for big muli-run tool submissions.
Because of database tension with sqlalchemy it is not strictly a linear increase, but the end user walltime experience for a 24 dataset collection being submitted using 4 threads instead 1 drops execution time from 68 seconds to 35.
Rebased with fixes thanks to @nsoranzo - https://github.com/jmchilton/galaxy/commit/7f6514a21222ea787a0676764e4a4fe825891b27#commitcomment-14712999.
- Don't refresh job when this is the only thread fetching it.
- Avoid a bunch of unnecessary flushes, just flush once essentially during process.
Brings this process from taking over 1 second per job on average on sqlite for cat1 on my laptop to around 400 ms on average.
Add proper idpDB sniffing logic (akin to MzSQlite) and test case
Add H5 sniffing based on magic number and test case
Add SQLite/IdpDB/MzSQlite/GeminiSQLite and H5 to default datatypes in registry.py
Continue to delay calculation of this but auto-compute it if requested and module injection code hasn't been called explicitly. Moves logic internal to class where it belongs also.
to cleanly handle the case described here:
https://github.com/galaxyproject/galaxy/issues/667. Specifically, the
following scenario is now cleanly handled:
Example use case: In a toolshed I have a repository of a tool, say
package_new_gene_db_1_2_3, and in revision 1 it depends on
package_sqlite_1_0_0:
package_new_gene_db_1_2_3
package_sqlite_1_0_0
At some point in time I discover that postgres is a better solution for
my gene database, and I upload revision 2:
package_new_gene_db_1_2_3
package_postgres__2_0_0
And I remove the dependecy of sqlite. If I go to my galaxy instance and
I remove revision 1 and install revision 2, the installation is cleanly
handled with theis PR.
It should be noted that this enhancement could adversely impact
reproducibility if best practices are not followed with regard to
defining package dependencies. Since it is now cleanly possible to
eliminate dependencies over time, best practices must be followed to
ensure that elimination of a dependency does not affect the output of a
tool that uses the underlying hierarchy of packages.
John Chilton