opensource/galaxy
3
0
Fork 0
mirror of https://github.com/galaxyproject/galaxy.git synced 2026-09-24 16:30:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
64,495 Commits 69 Branches 243 Tags
4bd80ce6efb3c19d896ed070bbff3f5d74e33bca
Commit Graph
120 Commits
Author SHA1 Message Date
mvdbeek accdea9b14 Always inject all modules, then compute runtime state 2022-10-29 22:11:03 +02:00
mvdbeek 66c143fc12 Inject modules & connections, then build runtime state for all steps 
That means steps can reason about connections that otherwise wouldn't
have modules attached when building runtime inputs.
 2022-10-28 17:05:51 +02:00
Nicola Soranzo c35d8f2a06 Replace `unittest.TestCase` with pytest-based partial re-implementation 
Also:
- Rename `FooBarTestCase` test classes as `TestFooBar`. These were collected
  by pytest only because they were `unittest.TestCase` derived, but normally
  pytest collects only test classes whose name starts with `Test`, see
  https://docs.pytest.org/en/7.1.x/reference/reference.html#confval-python_classes
 2022-09-27 03:34:58 +01:00
Marius van den Beek 955815ca11 Merge pull request #14688 from mvdbeek/dev 
Merge 22.05 into dev
 2022-09-26 15:19:53 +02:00
mvdbeek cbd38554f1 Merge branch 'release_22.05' into dev 2022-09-26 15:15:19 +02:00
John Chilton 550d29493f Merge pull request #14683 from mvdbeek/type_annotation_workflow_run 
Add type hints for galaxy.workflow.{run, run_request}
 2022-09-23 09:14:46 -04:00
mvdbeek 81b2f3e1e1 Add ModuleInjector, avoids test type mismatch 2022-09-23 10:19:02 +02:00
mvdbeek 48c654d1b2 Add type hint for galaxy.workflow.{run, run_request} 
Trying out https://github.com/dropbox/pyannotate here.
It's still a lot of work even with pyannotate, but I think this fixes
some minor oddities, like assigning non-mapped attributes to
WorkflowInvocation instances.
 2022-09-23 09:33:47 +02:00
Nicola Soranzo b5c7e2c86a Replace `self.assert* method calls with plain asserts in test/` 
using https://github.com/pytest-dev/unittest2pytest with the following command:

```
unittest2pytest -f self_assert -w -n test/
```

(excluding ``self.assertRaises*`` method calls).
 2022-09-21 23:50:05 +01:00
mvdbeek 64cb245ddd Fix running subworkflows with disconnected optional/default inputs 2022-09-21 19:31:41 +02:00
mvdbeek 8a8acc625f Fix refactor action dropping outputs on subworkflow connection 
I thought it would be cleaner to make a distinction between the output
name and the output label when reasoning about a subworkflow step in
the context of a parent workflow
(https://github.com/galaxyproject/galaxy/pull/14473/commits/b0727be9e0b096a2dacacb767a1f4ac0fc3e6540#diff-a50bcc5c08ade11249449c4100ea8c286727df601b52a5b9588eb2a385d97da7R543),
in case we'd later want to do something with the label.

I don't think I thought about this correctly though,
and it breaks connections between steps that were made by referencing
the subworkflow's output name.

This wasn't necessary for the main goal of
https://github.com/galaxyproject/galaxy/pull/14473, so I'm partially
reverting that here.

Fixes https://github.com/galaxyproject/planemo/pull/1262#issuecomment-1223874571
 2022-08-23 14:12:29 +02:00
mvdbeek b0727be9e0 Fix change_datatype not reflected in tool and workflow output terminals 2022-08-16 15:46:11 +02:00
John Chilton abb69b552c Implement API endpoints and tests for workflow sharing options. 2022-04-13 15:37:43 -04:00
John Chilton 427847bcc4 Fix up some import "issues". 2022-03-16 14:22:50 -04:00
Dannon Baker 623a5e165a Apply black formatting. 
Apply isort.
 2022-02-03 07:42:20 -05:00
John Chilton 7006e2a7e5 More unit test fixes. 2021-10-08 11:04:00 -04:00
John Chilton 8f9034a414 Typing fixes for various unit tests. 2021-10-05 13:22:41 -04:00
mvdbeek 6d22b6da28 Merge branch 'release_21.05' into dev 2021-06-10 15:37:59 +02:00
mvdbeek f6f4c7e6b7 Fix unit tests after removing get_tool_id 2021-06-10 08:53:49 +02:00
mvdbeek bf78f40bf6 Merge branch 'release_21.05' into dev 2021-06-03 10:23:52 +02:00
Sergey Golitsynskiy fb179aa66d Rename unit test helper classes (workflow support) 2021-06-01 15:08:17 -04:00
Sergey Golitsynskiy 0b094438e0 Enable pytest class-based tests; add flag if n/a 2021-06-01 12:30:25 -04:00
Dannon Baker 16d750be13 Add test for invocation_time 2021-05-28 12:34:18 -04:00
Sergey Golitsynskiy 6f53da3e73 Add backref to JtODAssoc.dataset; add mock 
JtODAssoc mock is required because otherwise on construction the real
JtODAssoc will try to use its internal SQLAlchemy instrumentation to
setup the backref relationship to HDA, which does not exist in the mock;
hense the JtODAssoc itself needs to be a mock.

This fixes the original commit cceb5d2 (reverted in 7fa3313).

Remove unnecessary call to class_mapper(HDA)

Address SAWarning:
> relationship 'HistoryDatasetAssociation.creating_job_associations' will copy
> column history_dataset_association.id to column
> job_to_output_dataset.dataset_id, which conflicts with relationship(s):
> 'JobToOutputDatasetAssociation.dataset' (copies history_dataset_association.id
> to job_to_output_dataset.dataset_id). If this is not the intention, consider
> if these relationships should be linked with back_populates, or if
> viewonly=True should be applied to one or more if they are read-only. For the
> less common case that foreign key constraints are partially overlapping, the
> orm.foreign() annotation can be used to isolate the columns that should be
> written towards.   The 'overlaps' parameter may be used to remove this
> warning. (Background on this error at: http
 2021-05-18 19:02:40 -04:00
Nate Coraor a2d1625780 Merge remote-tracking branch 'upstream/release_21.01' into dev 2021-03-01 16:52:02 -05:00
davelopez 385611d3d7 Add unit test for fake job name guessing 2021-02-26 11:48:04 +01:00
guerler 427594be71 Fix test cases, naming issues 2021-02-19 14:56:49 -05:00
John Chilton 15bde671e7 Fix certain classes of workflow extraction on copied objects. 2021-01-15 23:59:06 -05:00
John Chilton ce9240e921 Rework refactor responses for structured messages... 2021-01-03 19:52:28 -05:00
John Chilton 6be4e2bdbd Rework refactoring API for better separation of concerns. 
Thinner controller, pydantic model to separate options available for manager (which has to render a response) from those of the refactoring executor.
 2021-01-03 19:52:28 -05:00
John Chilton 38fbd87c88 Use existing workflow_manager to load subworkflows in wf modules. 2021-01-03 19:52:28 -05:00
Bjoern Gruening 2cd2777746 adopt to code review 2021-01-02 18:48:32 +00:00
Bjoern Gruening dfba28a443 fix mutable arguments in tests 2021-01-02 18:48:31 +00:00
John Chilton fc2ce45767 API for structured workflow refactoring. 2020-12-29 22:00:32 -05:00
Nicola Soranzo 1ec1f515b3 Merge branch 'release_20.09' into dev 2020-12-09 19:35:39 +00:00
mvdbeek cdad7c79a9 Pass optional subworkflow inputs to editor 
Fixes https://github.com/galaxyproject/galaxy/issues/10864
 2020-12-08 12:29:59 +01:00
mvdbeek 2f5742c7c0 Merge branch 'release_20.09' into dev 2020-11-03 16:04:22 +01:00
mvdbeek 14ced3cbae Set FK on left side of one to many DCE/DC relation 
I think this may fix a circular dependency between dataset_collection
and dataset_collection_element. This doesn't appear to be a problem
if dataset_collection has an id already, but the set of optimizations
that went into 20.09 may get us into the situation where that is not the
case.

I hope this fixes:
```
galaxy.job_execution.output_collect ERROR 2020-11-01 14:58:59,036 Problem gathering output collection.
Traceback (most recent call last):
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
cursor, statement, parameters, context
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.NotNullViolation: null value in column "dataset_collection_id" violates not-null constraint
DETAIL:  Failing row contains (13330967, null, 30637828, null, null, 0, ERR4597396__single).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/galaxy/server/lib/galaxy/job_execution/output_collect.py", line 156, in collect_dynamic_outputs
final_job_state=job_context.final_job_state,
File "/opt/galaxy/server/lib/galaxy/model/store/discover.py", line 286, in populate_collection_elements
self.flush()
File "/opt/galaxy/server/lib/galaxy/job_execution/output_collect.py", line 214, in flush
self.sa_session.flush()
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/scoping.py", line 163, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2523, in flush
self._flush(objects)
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2664, in _flush
transaction.rollback(_capture_exception=True)
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/session.py", line 2624, in _flush
flush_context.execute()
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
uow,
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 236, in save_obj
update,
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 995, in _emit_update_statements
statement, multiparams
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
return meth(self, multiparams, params)
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
distilled_params,
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
e, statement, parameters, cursor, context
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
cursor, statement, parameters, context
File "/opt/galaxy/venv/lib64/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "dataset_collection_id" violates not-null constraint
DETAIL:  Failing row contains (13330967, null, 30637828, null, null, 0, ERR4597396__single).
[SQL: UPDATE dataset_collection_element SET dataset_collection_id=%(dataset_collection_id)s WHERE dataset_collection_element.id = %(dataset_collection_element_id)s]
[parameters: {'dataset_collection_id': None, 'dataset_collection_element_id': 13330967}]
(Background on this error at: http://sqlalche.me/e/gkpj)
```
 2020-11-02 10:16:17 +01:00
Nicola Soranzo 9d74bba7fb Drop support for retired Python 3.5 
Upgrade syntax using `pyupgrade --py36-plus` .

Manually drop several `six` imports.

Also:
- Remove broken pr_cache in scripts/bootstrap_history.py
- Fix broken prefix removal in lib/galaxy/tool_util/deps/mulled/mulled_build.py
 2020-10-07 11:52:13 +01:00
John Chilton fe003d139e GA4GH search API. 2020-08-28 20:29:20 -04:00
John Chilton ed5bc4d1d6 API + UI for importing workflows from a GA4GH TRS server. 2020-08-21 12:06:54 -04:00
guerler 0547ec6cfc Do not automatically overwrite the output labels for subworkflows 2020-06-26 15:29:46 -04:00
mvdbeek 785cfce3fa Fix unit tests 2020-04-25 20:40:26 +02:00
John Chilton 26d5150245 Eliminate extensions=['input_collection']. 
The existing hack of extensions=['input'] is bad enough and should work fine for collections. Not sure what past-John was thinking by adding this.
 2019-12-11 12:26:32 -05:00
John Chilton 20d737cd09 Fix unit tests. 2019-12-10 21:21:31 -05:00
John Chilton fa637518e8 Implement yet another syntax for Galaxy Markdown. 
The community took a vote (https://github.com/galaxyproject/galaxy/pull/8511#issuecomment-525886525) and the overwhelming favorite was this programming style declaration of a vaguely C-ish language embedded in quote fenced blocks.
 2019-08-30 09:46:14 -04:00
John Chilton 664ab71653 Rework toward more RMarkdown-y syntax for Galaxy Markdown. 2019-08-30 09:43:15 -04:00
John Chilton ad650897d6 Implement workflow invocation reports. 
Implement markdown backend and frontend components as well generator plugin framework to allow customizable workflow invocation reports.

Architectural Choices on the Client

Old-style embedded objects vs components

In theory, this could make really nice use of excellent VueJS components for dataset display, dataset collection display, workflow display, etc.. These aren't available yet, we don't even really have Backbone stuff that exist very well outside history panels, so this re-uses "components" from Galaxy Pages for datasets and reuses the collection display for history panels as a stand-alone display for collections (I got this trick from DIsplayStructured.vue). This isn't ideal and I know that but at least reusing things this way will provide a path forward for migrating both pages and this new Markdown language (which could easily replace or co-exist with pages HTML someday) together seamlessly as the real modern components become available and doesn't increase the overall work needed to integrate newer style components. In fact, this might even be the impetus for creating and polishing more of these components.

markdown-it vs markdown-it-vue

I tried this with markdown-it-vue also but it added very little in terms of reducing client code, obscured entirely how to attach plugins to the markdown rendering process, and brought in many, many more extra packages that we don't need or want.

Given there is no representation of invocations in the GUI and not editor for reports config this is a bit challenging still. But here is goes:

- Source Galaxy's virtualenv.
- Login to a user and grab an API key, set in the following command:
- ``GALAXY_TEST_EXTERNAL=http://localhost:8080 GALAXY_TEST_USER_API_KEY=38175dfc69d9009992b69efcca8b209b pytest test/api/test_workflows.py::WorkflowsApiTestCase::test_workflow_invocation_report_custom``
- In the web browser go to http://localhost:8080/api/invocations and grab the latest invocation ID. Replace it in the follow query string.
- Navigate to http://localhost:8080/workflows/invocations/report?id=c887f1d0da42bdfe
 2019-08-30 09:43:01 -04:00
Nicola Soranzo e007cc928f Fix `items() and keys()` not supporting indexing on Py3 2019-08-18 19:25:44 +01:00
Martin Cech e65604bc06 Merge pull request #7556 from jmchilton/expression_tools 
Implement expression tools and non-data tool outputs.
 2019-03-20 18:52:59 -04:00