Commit Graph
16742 Commits
Author SHA1 Message Date
mvdbeek 5ff6be93ed Use profile="18.01" for BamNative converters
This is because samtools writes to stderr when sorting/merging large files.
Part of https://github.com/galaxyproject/galaxy/issues/5496#issuecomment-365458089.
2018-02-14 14:10:18 +01:00
Martin Cech ed23cc652d Merge pull request #5514 from jmchilton/optimize_public_grids
[18.01] Optimize public grid database interactions.
2018-02-13 16:29:54 -05:00
John Chilton 80a41f3274 Optimize public grid database interactions.
For all four grid types (histories, viz, workflows, and pages), this computes rating average on the server and prefetches any annotations. This eliminates at least two extra queries for response element in the grid. See comment in the history controller for why I am fairly confident this is a good idea for annotations but tags are less obvious - these grids still go back to the postgres server multiple times per rendered item to render tags.

I'm confident we should either subqueryload, joinedload, or upgrade to sqlalchemy 2.2 and selectinload (http://docs.sqlalchemy.org/en/latest/orm/loading_relationships.html#sqlalchemy.orm.selectinload) the tags as well - but I'm not sure which without being able to hack on a usegalaxy.org.

This also brings in less of the user model (only username instead of all of it) to reduce over-the-wire transmission of unneeded data from postgres to Galaxy.

For workflows we were LEFT OUTER JOIN-ing on the steps of the latest workflow - so we were bringing back a lot of extra rows for data that was completely unused. In light of this, it makes perfect sense to me why published workflows were the slowest of these and I suspect they will all be equally performant after this change (modulo the number of rows in the tables and the number of rows rendered).
2018-02-13 14:29:29 -05:00
John Chilton e73731f552 Optimize database interaction for user workflow list.
Don't LEFT OUTER JOIN workflow against its steps to produce step count - this causes a lot of duplicated workflow data and fetches step data not needed to just produce a step count.

Tweak pre-fetching for workflows shared with user also.
2018-02-13 11:53:10 -05:00
John Chilton 321cd78ad8 Merge pull request #5512 from mvdbeek/fix_new_pysam
[18.01] Make galaxy compatible with pysam 0.14
2018-02-13 11:22:04 -05:00
mvdbeek 3b8a2f9edc Pin pysam to 0.14 2018-02-13 15:22:51 +01:00
mvdbeek 349b25b74a Minimize duplication of BAM set_meta code 2018-02-13 11:49:10 +01:00
mvdbeek b87d57aac7 Make galaxy compatible with pysam 0.14 2018-02-12 22:03:49 +01:00
Martin Cech ba3b3d5c18 do not allow codefiles for TS tool loading 2018-02-12 14:06:50 -05:00
Dannon Baker d91078024b Merge pull request #5491 from nsoranzo/release_18.01_fix_common_startup_for_bare_clone
[18.01] Fix common_startup.sh for galaxy tarballs
2018-02-08 13:20:46 -05:00
Dave B 3f50b45a4e Merge pull request #5445 from gregvonkuster/add-ideaspre
[18.01] Add support for the ideaspre datatype
2018-02-08 13:11:10 -05:00
Nate Coraor b463be78a5 Add nodeenv to dev requirements 2018-02-08 11:33:35 -05:00
John Chilton 64e4e8b572 Merge pull request #5482 from mvdbeek/bam_recursion_bug
[18.01] Fix recursion bug in BamNative datatype
2018-02-08 08:14:48 -05:00
Jelle Scholtalbers db069198af fix #5484 2018-02-08 10:42:31 +01:00
Marius van den Beek 8b0e3bd28f Fix recursion bug in BamNative datatype
We introduced this in
https://github.com/galaxyproject/galaxy/pull/5180/ where we moved a lot
of code form the Bam class into the BamNative class.

Reported by @natefoo
2018-02-07 23:00:44 +01:00
Nate Coraor 11ec4ee93d Revert "Try next dep. resolver if building env failed"
This reverts commit e821d469e6.
2018-02-07 13:16:18 -05:00
Nate Coraor 04e4da0d31 Revert "Avoid duplicate dependency installation"
This reverts commit c0cf41a15c.
2018-02-07 13:16:11 -05:00
Dannon Baker c3ada7f169 Merge pull request #5460 from jmchilton/fix_oom_parse
[18.01] Fix tool stdio level parsing for OOM level.
2018-02-05 15:04:30 -05:00
Dannon Baker 0704d63d0b Merge remote-tracking branch 'jmchilton/1801_hdca_delete' into release_18.01 2018-02-05 13:06:01 -05:00
John Chilton 2f020ef944 Fix tool stdio level parsing for OOM level.
Backend was enhanced to allow this in #5196 and a new entry point to setting exit code for such exit conditions but the custom parsing wasn't updated to allow it. Thanks to @nsoranzo for the catch.
2018-02-05 12:18:22 -05:00
Dannon Baker 2cb3af5d60 Merge pull request #5447 from natefoo/server-name-fix
[18.01] Fix server_name when config is a dict
2018-02-05 10:43:03 -05:00
Dannon Baker 207f5faba0 Merge pull request #5449 from jmchilton/collection_mapping_fixes
[18.01] Collection mapping fixes for regression in 18.01.
2018-02-04 11:52:33 -05:00
John Chilton f99299f833 Pre-compute more collection state during mapping.
Still dealing with fallout from 78babab645. After that commit we were treating structured_like collections like collections with unknown structure and not dealing with collection_type_source information at all when building backbones for new collections during mapping.
2018-02-03 14:59:30 -05:00
mvdbeek 7bbc46658a Fix setting job failure exception
This can happen when the job's tool is unavailble and
outputs_to_working_directory is activated.
Fixes https://github.com/galaxyproject/galaxy/issues/5424.
2018-02-03 18:09:57 +01:00
Nate Coraor dac1ba5812 Fix failing integration tests for workflow handlers. 2018-02-02 15:40:22 -05:00
John Chilton 8e32e93d6d Implement more collection workflow post job actions.
Implement hide, delete, add tag, and remove tag for output collections and mapped over collections.

Fixes https://github.com/galaxyproject/galaxy/issues/1680.
2018-02-02 13:29:57 -05:00
greg 3fdd4edc5c Subclass IdeasPre from Html 2018-02-02 13:07:52 -05:00
Nate Coraor b0162ba543 Fix server_name when config is a dict. 2018-02-02 12:53:39 -05:00
John Chilton 0529153ac0 Fix bug in PJAs for collection mapping.
Reported by @mvdbeek here (https://github.com/galaxyproject/galaxy/pull/5416#issuecomment-362194157).
2018-02-02 09:54:30 -05:00
John Chilton f6f9f9a891 Extend mapped collection output rename PJAs to include # replacements.
Also make sure output collections in non-mapping scenarios are also properly renamed as well.
2018-02-02 09:52:44 -05:00
John Chilton 2b8df110d0 Apply rename PJA to collection outputs including $ replacements.
Update PJA collection test to test HDCA is renamed also.
2018-02-02 09:52:44 -05:00
greg b5945ad3e3 Add IdeasPre datatype subclassed from Rgenetics 2018-02-02 09:22:10 -05:00
John Chilton ab3d53c5ce More options for collection deletion.
Delete button now a drop down menu (like dropdown for downloading metadata files).

- Top option just deletes the collection.
- Second option deletes the collection and attempts to delete the datasets contained within it.
- The third option deletes the collection and attempts to purge the datasets within it.
2018-02-01 23:20:59 -05:00
John Chilton 6022fe7d7e Handle optional inputs when collecting preserved tags.
https://github.com/galaxyproject/galaxy/pull/5403/files#r165283450
2018-02-01 21:03:02 -05:00
John Chilton 6f40f7098d Cleanup collection operation tagging implementation thanks to @mvdbeek comments. 2018-02-01 21:02:01 -05:00
John Chilton 7490fd4520 More robust nametag discovery and propagation.
- Nametags weren't being picked up on collection inputs - so for instance tools that take in a single collection and produce a dataset or file weren't propagating nametags (unless the collection contents had tags). To see this create two unnamed datasets without tags, create a pair from them, add a tag to the pair, and run the pair through the "paired_test" test tool.
- Nametags weren't being propagated to the outputs of collection operations.
2018-02-01 21:02:01 -05:00
Martin Cech ffaabe0a63 Merge pull request #5342 from dannon/integration.webhooks_improvements
[18.01] Webhooks improvements
2018-02-01 15:14:47 -05:00
Martin Cech a37b638491 Merge pull request #5390 from natefoo/framework-dependencies-doc-update
[18.01] Update framework dependencies documentation
2018-02-01 15:00:15 -05:00
Dannon Baker 0ac7b7830d Fix import order in webhooks/__init__.py 2018-02-01 12:18:02 -05:00
Evgeny Anatskiy ada76931d8 add weighted random webhook pick 2018-02-01 12:18:01 -05:00
Evgeny Anatskiy 6070bf07e1 simplify Webhooks' structure 2018-02-01 12:18:01 -05:00
Dave B 30ac9e940a Workaround for extra metadata revision creation
Without this change, uploading non-metadata changes to a toolshed repository will display an error message and create a new metadata revision regardless of whether the code should determine that a new metadata revision is warranted.
2018-02-01 09:21:16 -05:00
Nicola Soranzo 5b23c0e87d Merge branch 'release_17.09' into release_18.01
Conflicts:
	client/galaxy/scripts/mvc/workflow/workflow-node.js
	config/galaxy.ini.sample
	static/maps/mvc/workflow/workflow-node.js.map
	static/scripts/mvc/workflow/workflow-node.js
2018-01-31 10:38:53 +00:00
Nicola Soranzo c0c95548a3 Merge branch 'release_17.05' into release_17.09 2018-01-31 10:16:33 +00:00
John Chilton 9fcac13145 Fix the PJA type issue when mapping renames over nested lists. 2018-01-30 11:16:20 -05:00
mvdbeek eb506b25b9 Allow map-over when discovering dataset collections
This would affect for example the mapping over of fastq-dump (and all other
tools in the sra-toolkit).

If one had attempted this previously the discovery phase would fail with:
```
galaxy.tools.parameters.output_collect ERROR 2018-01-30 08:56:46,369 Problem gathering output collection.
Traceback (most recent call last):
  File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/tools/parameters/output_collect.py", line 169, in collect_dynamic_collections
    collection
  File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/managers/collections.py", line 216, in collection_builder_for
    return builder.BoundCollectionBuilder(dataset_collection, collection_type_description)
  File "/bioinfo/guests/mvandenb/galaxy/lib/galaxy/dataset_collections/builder.py", line 84, in __init__
    raise Exception("Cannot reset elements of an already populated dataset collection.")
Exception: Cannot reset elements of an already populated dataset collection.
```

Instead we force the collection state to be new when we are discovering output collection datasets,
which seems reasonable to me. This includes an API testcase that would have failed previously.
2018-01-30 14:48:42 +01:00
mvdbeek a70f2875b3 Remove r channel from more locations 2018-01-29 18:07:29 +01:00
mvdbeek 414c031247 Follow IUC and Drop r channel from default cond channels 2018-01-29 17:35:49 +01:00
Nicola Soranzo b8873c65ad Install the latest version of conda and conda-build
Conda 4.3.33 contains a fix that should solve some conda env
creation programs:

https://github.com/conda/conda/pull/6766

Also support conda installation on x86 Linux architecture.
2018-01-26 11:34:52 +00:00
Nicola Soranzo 889aa29d02 Conda: pass subprocess arguments as list 2018-01-26 11:31:29 +00:00