opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-10-13 16:57:59 +08:00
Code Issues Packages Projects Releases Wiki Activity
20,416 Commits 61 Branches 243 Tags
102eeef2d7045fa5b8706497f04817cb2f75813a
Commit Graph
20416 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Takao Nakaguchi 102eeef2d7 Replacement should be applied only to first occurrence. 2015-12-04 01:28:23 +00:00
Eric Rasche e76933b557 Merge pull request #1229 from martenson/remove-readthecdocs-badge 
replace the readthedocs badge,
 2015-12-03 16:56:03 -06:00
Martin Cech 6735c9dece replace the readthedocs badge, 
we cannot use it until https://github.com/rtfd/readthedocs.org/issues/1767 is resolved
 2015-12-03 17:11:52 -05:00
Martin Cech 9a1fc647a3 Merge pull request #1212 from jmchilton/tool_opts_6 
Smaller Tool Optimization PR
 2015-12-03 16:23:41 -05:00
John Chilton 14ac3665ee Merge pull request #598 from davebx/async_metadata_reset 
Make bulk repository metadata resets asynchronous.
 2015-12-03 16:10:22 -05:00
John Chilton 92a7b995b0 Merge pull request #1217 from ericenns/collection-tool-input-fixes 
Collection tool input mapping performance fixes
 2015-12-03 15:31:10 -05:00
John Chilton a570956612 Merge pull request #396 from jessevdam/dev 
Added better support for formats that store triples.
 2015-12-03 15:25:13 -05:00
Jesse 4c13c92540 Merge pull request #2 from jmchilton/triples_2 
Remove deprecated datatype stuff
 2015-12-03 21:22:30 +01:00
John Chilton 65e6a1c3b6 Merge pull request #1209 from chambm/feature/idpdb_hdf5_sniffers 
Add idpDB and HDF5 sniffers and fix MzSQlite sniffer
 2015-12-03 15:11:34 -05:00
Eric Enns 362a25e6e6 Updated tool input collection to only add items from a collection which are being used by the tool and not the entire collection. (Thanks to @markiskander for helping me find this). 2015-12-03 10:48:39 -06:00
Eric Rasche c034dc3e33 Merge pull request #1213 from jmchilton/tool_logging_2 
Add debug statement in output checker for why job is failing.
 2015-12-03 10:16:59 -06:00
Nicola Soranzo 20b96d883d Merge pull request #1202 from jmchilton/workflow_tool_refactoring 
Small Tool and Workflow Refactoring and Fixes
 2015-12-03 15:53:22 +00:00
John Chilton a443d99fdd Undo deprecated datatypes stuff, doesn't work with Galaxy well. 2015-12-03 15:15:31 +00:00
John Chilton 3de8579633 Add debug statement in output checker for why job is failing. 
Should help tool testing and other Galaxy testing.
 2015-12-03 10:49:51 +00:00
John Chilton ab2c7c2c07 Tool execution micro optimization... 
... Don't create an extra list to reverse, just iterate over in reversed order.
 2015-12-03 10:16:12 +00:00
John Chilton a190b69e33 Optimize tools that produce multiple outputs. 
Fetch HIDs for these datasets all at once and flush once for all assignments. Creating jobs from 4 threads each job producing 10 outputs - this resulted in reducing the runtime of this portion of the code by 90% and saving around 20 seconds per job.

Before:

galaxy.tools.actions INFO 2015-11-30 15:53:46,235 Add outputs to history (22494.829 ms)
galaxy.tools.actions INFO 2015-11-30 15:53:47,295 Add outputs to history (22239.062 ms)
galaxy.tools.actions INFO 2015-11-30 15:53:48,320 Add outputs to history (23781.079 ms)
galaxy.tools.actions INFO 2015-11-30 15:53:51,490 Add outputs to history (21820.567 ms)
galaxy.tools.actions INFO 2015-11-30 15:54:24,506 Add outputs to history (25353.837 ms)

After:

galaxy.tools.actions INFO 2015-11-30 16:08:47,640 Add outputs to history (2781.675 ms)
galaxy.tools.actions INFO 2015-11-30 16:08:47,860 Add outputs to history (3177.738 ms)
galaxy.tools.actions INFO 2015-11-30 16:08:48,776 Add outputs to history (2425.528 ms)
galaxy.tools.actions INFO 2015-11-30 16:09:02,942 Add outputs to history (2579.022 ms)
 2015-12-03 10:16:12 +00:00
John Chilton 64b7cc9fef Add sample tool that creates 10 outputs. 2015-12-03 10:16:12 +00:00
John Chilton 41d3dc62f7 Parallelize executing tool parameter combinations. 
Should work when mapping over collections or for big muli-run tool submissions.

Because of database tension with sqlalchemy it is not strictly a linear increase, but the end user walltime experience for a 24 dataset collection being submitted using 4 threads instead 1 drops execution time from 68 seconds to 35.

Rebased with fixes thanks to @nsoranzo - https://github.com/jmchilton/galaxy/commit/7f6514a21222ea787a0676764e4a4fe825891b27#commitcomment-14712999.
 2015-12-03 10:16:12 +00:00
John Chilton 876c28f88d Add more job timers. 2015-12-03 10:15:41 +00:00
John Chilton 299fbe99d7 Micro optimization, only fetch tool validation once per multi-run. 2015-12-03 10:15:11 +00:00
John Chilton dd7bddcaa6 Optimize initial queueing of jobs. 
- Don't refresh job when this is the only thread fetching it.
 - Avoid a bunch of unnecessary flushes, just flush once essentially during process.

Brings this process from taking over 1 second per job on average on sqlite for cat1 on my laptop to around 400 ms on average.
 2015-12-03 10:15:11 +00:00
John Chilton dd5f3aec52 Fix workflow extraction test cases broken with 77ed5681f8 2015-12-03 09:55:45 +00:00
chambm 97415de3ce Move idpDB and MzSQlite formats from proteomics.py to binary.py, because the SQLite sniffer lives there, and more specialized formats must be registered for sniffing before the generic format 
Add proper idpDB sniffing logic (akin to MzSQlite) and test case
Add H5 sniffing based on magic number and test case
Add SQLite/IdpDB/MzSQlite/GeminiSQLite and H5 to default datatypes in registry.py
 2015-12-02 18:27:33 -06:00
Martin Cech ffb3de9ede Merge pull request #1153 from jmchilton/collection_type_source 
Implement type_source on output collections.
 2015-12-02 00:14:08 -05:00
John Chilton 67c98f54ca Merge pull request #1200 from nsoranzo/tool_shed_fixes 
Tool shed fixes.
 2015-12-01 23:51:47 -05:00
John Chilton 48340b5829 Add TODO items to workflow progress unit tests. 2015-12-01 23:44:26 +00:00
John Chilton 07c3d10f31 Extend workflow recovery progress test case to include replacement check. 2015-12-01 23:44:26 +00:00
John Chilton 7974c1f334 Add unit tests for workflow progress initialization. 2015-12-01 23:44:26 +00:00
John Chilton cf367ff0c7 Add another workflow progress test case. 
This one for replacing a tool input with an input dataset.
 2015-12-01 23:44:26 +00:00
John Chilton 8a01b42112 Refactor input_by_connections_name logic into model. 
Continue to delay calculation of this but auto-compute it if requested and module injection code hasn't been called explicitly. Moves logic internal to class where it belongs also.
 2015-12-01 23:44:26 +00:00
John Chilton 031a39aba8 Basic unit tests for workflow progress tracking. 2015-12-01 23:44:26 +00:00
John Chilton 21213ef70f Refactor workflow testing code to quickly bootstrap workflows. 2015-12-01 23:44:26 +00:00
John Chilton 4584a579a2 Fix interface declaration for WorkflowModule.get_state. 
Implementers all take this attribute.
 2015-12-01 23:44:25 +00:00
John Chilton 28501ab676 Refactor workflow manager for extension. 2015-12-01 23:44:25 +00:00
John Chilton 42778e2753 Improved error handling in workflow run code. 2015-12-01 20:17:10 +00:00
John Chilton 77ed5681f8 Extend base api workflow _run_jobs helper... 
... to allow loading content from test-data files.
 2015-12-01 20:17:10 +00:00
John Chilton 7ce43a0463 Refactor run_request for downstream diff improvements. 2015-12-01 20:17:10 +00:00
John Chilton ea9f4a0467 Initialize a ToolOutput.from_work_dir. 2015-12-01 20:17:09 +00:00
John Chilton f64d1597b8 Extension point for environment variables. 2015-12-01 20:17:09 +00:00
Nicola Soranzo 194d871e38 Fix Tool Shed export capsule for Mercurial >= 3.4 . 
Fix the following error:
tool_shed.capsule.capsule_manager ERROR 2015-12-01 12:24:37,542 unable to find mercurial command "archive"
Traceback (most recent call last):
  File "/opt/galaxy/lib/tool_shed/capsule/capsule_manager.py", line 88, in export_repository
    work_dir )
  File "/opt/galaxy/lib/tool_shed/capsule/capsule_manager.py", line 148, in generate_repository_archive
    changeset_revision )
  File "/opt/galaxy/lib/tool_shed/util/hg_util.py", line 28, in archive_repository_revision
    options_dict = get_mercurial_default_options_dict( 'archive' )
  File "/opt/galaxy/lib/tool_shed/util/hg_util.py", line 200, in get_mercurial_default_options_dict
    raise Exception('unable to find mercurial command "%s"' % command)
Exception: unable to find mercurial command "archive"
 2015-12-01 16:45:25 +00:00
Nicola Soranzo 657afa89d3 Small fixes. 2015-12-01 16:45:25 +00:00
Nicola Soranzo ff619df394 Downgrade Mercurial wheel to 3.4.2 to restore hg push of Tool Shed repositories. Fix #1183. 2015-12-01 16:45:19 +00:00
Dannon Baker 84464288b1 Merge pull request #1184 from guerler/fix_download_links_002 
Filter download links, avoid warning dialog
 2015-12-01 10:19:22 -05:00
Nicola Soranzo 3524476bc6 Merge branch 'master' into dev 2015-12-01 11:42:16 +00:00
Eric Rasche da2cc588e5 Merge pull request #1194 from martenson/release_15.10 
[15.10] docs - deprecate TS direct hg api
 2015-11-30 13:39:17 -06:00
Martin Cech 5bd898118f deprecate TS direct hg api 2015-11-30 14:35:45 -05:00
Nate Coraor 9e28f2e786 Merge branch 'release_15.10' into dev 2015-11-30 14:13:26 -05:00
John Chilton a03404c04d Merge pull request #1162 from martenson/fix-datalibs 
[15.10] Fix two bugs in datalibs
v15.10 		2015-11-30 13:57:15 -05:00
Martin Cech c532f6808e Merge pull request #1186 from gregvonkuster/release_15.10 
[15.10] Add support for Strand-specific Coordinate Count datatype
 2015-11-30 13:56:24 -05:00
Martin Cech 492fcdab9f Merge pull request #1193 from mvdbeek/gvk_ts_fixes 
Add enhancements to the Galaxy repository install process
 2015-11-30 13:16:06 -05:00