Commit Graph
58766 Commits
Author SHA1 Message Date
John Chilton b06bdc077e Merge pull request #13560 from mvdbeek/enable_model_store_sources
[22.01] Attach dataset source in model store imports
2022-03-21 10:15:19 -04:00
Marius van den Beek 38509cac8e Merge pull request #13576 from nsoranzo/release_22.01_fix_broken_model_discovery_tests
[22.01] Fix model discovery unit tests
2022-03-18 22:55:39 +01:00
Nicola Soranzo 0075ecb275 Fix model discovery unit tests
that I broke when merging
https://github.com/galaxyproject/galaxy/pull/13552 forward.

Fix:

```
app = <galaxy.model.unittest_utils.data_app.GalaxyDataTestApp object at 0x7f0e00dd3640>, target = '/tmp/tmpyj5d8268', work_directory = '/tmp/tmp_a7e191v'

    def _import_directory_to_history(app, target, work_directory):
        sa_session = app.model.context

        u = model.User(email="collection@example.com", password="password")
        import_history = model.History(name="Test History for Import", user=u)

        sa_session = app.model.context
        sa_session.add_all([u, import_history])
        sa_session.flush()

        assert len(import_history.datasets) == 0

        import_options = store.ImportOptions(allow_dataset_object_edit=True)
>       import_model_store = store.get_import_model_store_for_directory(target, app=app, user=u, import_options=import_options, tag_handler=app.tag_handler.create_tag_handler_session())
E       AttributeError: 'GalaxyDataTestApp' object has no attribute 'tag_handler'

test/unit/data/model/test_model_discovery.py:235: AttributeError
```
2022-03-18 19:12:20 +00:00
Nicola Soranzo 5114b67902 Merge branch 'release_21.09' into release_22.01 2022-03-18 18:28:15 +00:00
Nicola Soranzo 8e4bdb58e4 Merge branch 'release_21.05' into release_21.09 2022-03-18 18:08:10 +00:00
Marius van den Beek 74a51d990a Merge pull request #13552 from mvdbeek/fix_tag_creation_metadata_strategy_extended
[21.09] Fix creating tags within sessionless context
2022-03-18 15:42:43 +01:00
John Chilton 316998dea1 Merge pull request #13573 from mvdbeek/fix_tool_shed_bootstrapping
[22.01] Fix tool shed bootstrapping
2022-03-18 09:11:53 -04:00
mvdbeek 7aae40d1af Include unit test for sessionless export/import with tags
Seems like the only way you could previously include tags
was if you included them in the data fetch tool.
2022-03-18 12:16:17 +01:00
mvdbeek 558cad246e Return tag in get_tag_by_name function 2022-03-18 11:12:06 +01:00
mvdbeek 4c7bfd1032 Fix tool shed bootstrapping script 2022-03-17 19:34:43 +01:00
mvdbeek e9d2d683d6 Add --capture-output when running gunicorn in daemon mode 2022-03-17 19:34:16 +01:00
mvdbeek 3d7dde0f9b Fix tool shed /api/authenticate/baseauth/
Fixes:
```
AttributeError: 'UniverseApplication' object has no attribute 'user_manager'
  File "galaxy/web/framework/decorators.py", line 312, in decorator
    rval = func(self, trans, *args, **kwargs)
  File "galaxy/webapps/galaxy/api/authenticate.py", line 63, in get_api_key
    user = self.app.user_manager.get_user_by_identity(identity)
```
2022-03-17 19:33:18 +01:00
mvdbeek 96934420e8 Attach dataset source in model store imports
Fixes
```
FAILED test/integration_selenium/test_dataset_details_source_transforms.py::DatasetSourceTransformSeleniumIntegrationTestCase::test_displaying_source_transformations_grooming
FAILED test/integration_selenium/test_dataset_details_source_transforms.py::DatasetSourceTransformSeleniumIntegrationTestCase::test_displaying_source_transformations_posixlines
FAILED test/integration_selenium/test_dataset_details_source_transforms.py::DatasetSourceTransformSeleniumIntegrationTestCase::test_displaying_source_transformations_spaces_to_tabs
```
2022-03-17 10:11:13 +01:00
Marius van den Beek ce7d376416 Merge pull request #13553 from mvdbeek/fix_extended_metadata_jobs_with_dce_inputs
[22.01] Fix extended_metadata jobs with dce inputs
2022-03-17 08:45:03 +01:00
Nicola Soranzo bad8ecbda2 Merge pull request #13554 from mvdbeek/fix_sff_extractor_tool 2022-03-16 18:00:25 +00:00
mvdbeek e2d43806dc Fix `sff_extract` tool, drop type annotation
I don't think this tool is really used much, but it is loaded in
`tool_conf.xml`.
https://github.com/galaxyproject/galaxy/pull/11877 added type
annotations, and that's a syntax error on 2.7

The alternative would be to fix the rest of the tool for 3.x,
but I think we're just wasting our time there and we shouldn't
touch this again.
2022-03-16 15:24:55 +01:00
mvdbeek cf359f0b87 Fix extended_metadata jobs with dce inputs
This broke when we started re-exporting jobs to update their command
line in https://github.com/galaxyproject/galaxy/pull/12459/commits/65e0ee213ae574bc0a45dfcc60e997863ff96a7c#diff-bfb854f344e56a25a86761f80a639de09add404a7a53ebde45afb908762fc1eaR161:

```
ERROR    galaxy.jobs:__init__.py:1836 problem importing job outputs. stdout [] stderr [
Traceback (most recent call last):
  File "metadata/set.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/metadata/set_metadata.py", line 121, in set_metadata
    set_metadata_portable()
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/metadata/set_metadata.py", line 307, in set_metadata_portable
    export_store.export_job(job, include_job_data=False)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/__init__.py", line 1320, in export_job
    self.export_jobs([job], include_job_data=include_job_data)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/__init__.py", line 1336, in export_jobs
    job_attrs = job.serialize(self.security, self.serialization_options)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/__init__.py", line 339, in serialize
    return self._serialize(id_encoder, serialization_options)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/__init__.py", line 1447, in _serialize
    params_objects = remap(params_objects, remap_objects)
  File "/home/runner/work/galaxy/galaxy/galaxy root/.venv/lib/python3.7/site-packages/boltons/iterutils.py", line 1128, in remap
    visited_item = visit(path, key, value)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/__init__.py", line 1441, in remap_objects
    new_id = serialization_options.get_identifier_for_id(id_encoder, obj["id"])
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/__init__.py", line 323, in get_identifier_for_id
    raise NotImplementedError()
NotImplementedError
]
```

To test run `GALAXY_CONFIG_METADATA_STRATEGY=extended ./run_tests.sh
-api
lib/galaxy_test/api/test_jobs.py::JobsApiTestCase::test_dce_submission_security`
2022-03-16 13:44:07 +01:00
mvdbeek 880c129fa5 Fix creating tags within sessionless context
xref: https://github.com/galaxyproject/galaxy/issues/13551, fixes
```
galaxy.jobs ERROR 2022-03-15 00:21:29,515 [pN:main,p:2180,tN:LocalRunner.work_thread-3] problem importing job outputs. stdout [] stderr [
Traceback (most recent call last):
  File "metadata/set.py", line 1, in <module>
    from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/metadata/set_metadata.py", line 121, in set_metadata
    set_metadata_portable()
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/metadata/set_metadata.py", line 291, in set_metadata_portable
    collect_dynamic_outputs(job_context, output_collections)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/job_execution/output_collect.py", line 147, in collect_dynamic_outputs
    persist_elements_to_hdca(job_context, elements, hdca, collector=DEFAULT_DATASET_COLLECTOR)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/discover.py", line 719, in persist_elements_to_hdca
    filenames,
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/discover.py", line 292, in populate_collection_elements
    final_job_state=final_job_state,
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/discover.py", line 354, in _populate_elements
    self.add_tags_to_datasets(datasets=element_datasets["datasets"], tag_lists=element_datasets["tag_lists"])
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/discover.py", line 576, in add_tags_to_datasets
    self.tag_handler.add_tags_from_list(user, dataset, tags, flush=False)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 61, in add_tags_from_list
    return self.set_tags_from_list(user, item, new_tags_set, flush=flush)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 75, in set_tags_from_list
    self.apply_item_tags(user, item, unicodify(new_tags_str, "utf-8"), flush=flush)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 207, in apply_item_tags
    self.apply_item_tag(user, item, name, value, flush=flush)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 179, in apply_item_tag
    tag = self._get_or_create_tag(lc_name)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 281, in _get_or_create_tag
    tag = self._create_tag(scrubbed_tag_str)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 246, in _create_tag
    tag = self._create_tag_instance(tag_name)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 428, in _create_tag_instance
    tag = super()._create_tag_instance(tag_name)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/tags.py", line 259, in _create_tag_instance
    Session = sessionmaker(self.sa_session.bind)
AttributeError: 'NoneType' object has no attribute 'bind'
]
Traceback (most recent call last):
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/jobs/__init__.py", line 1828, in finish
    import_model_store.perform_import(history=job.history, job=job)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/__init__.py", line 209, in perform_import
    datasets_attrs = self.datasets_properties()
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/__init__.py", line 930, in datasets_properties
    datasets_attrs = load(open(datasets_attrs_file_name))
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp19s6x278/tmpt0rr8vmw/tmp2mjq6ysx/database/job_working_directory1/000/7/metadata/outputs_populated/datasets_attrs.txt'
galaxy.jobs.runners ERROR 2022-03-15 00:21:29,516 [pN:main,p:2180,tN:LocalRunner.work_thread-3] (7/) Job wrapper finish method failed
Traceback (most recent call last):
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/jobs/runners/__init__.py", line 594, in _finish_or_resubmit_job
    job_stderr=job_stderr,
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/jobs/__init__.py", line 1828, in finish
    import_model_store.perform_import(history=job.history, job=job)
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/__init__.py", line 209, in perform_import
    datasets_attrs = self.datasets_properties()
  File "/home/runner/work/galaxy/galaxy/galaxy root/lib/galaxy/model/store/__init__.py", line 930, in datasets_properties
    datasets_attrs = load(open(datasets_attrs_file_name))
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp19s6x278/tmpt0rr8vmw/tmp2mjq6ysx/database/job_working_directory1/000/7/metadata/outputs_populated/datasets_attrs.txt'
```
2022-03-16 11:32:06 +01:00
Marius van den Beek 1b2c127f65 Merge pull request #13550 from ic4f/release_22.01_releasenotes
Remove duplicate names
2022-03-15 23:09:51 +01:00
Sergey Golitsynskiy cfb8dd1a8c Remove duplicate names 2022-03-15 18:03:00 -04:00
mvdbeek 1cbab39518 Update version to 22.01.1.dev0 2022-03-15 20:46:27 +01:00
mvdbeek 9690680ec7 Update version to 22.01. v22.01 2022-03-15 20:46:24 +01:00
Dannon 86f696cc0b Merge pull request #13546 from mvdbeek/fix_yaml_preview_firefox
[22.01] Fix yaml preview in firefox
2022-03-15 12:47:43 -04:00
Marius van den Beek bfd02136cf Merge pull request #13547 from mvdbeek/add_gravity_to_test_requirements
[22.01] Make gravity import conditional
2022-03-15 17:43:48 +01:00
John Chilton dac867ab8a Merge pull request #13544 from mvdbeek/release_notes_update
[22.01] Update Release notes
2022-03-15 12:32:50 -04:00
mvdbeek 6806a0f995 Make gravity import conditional 2022-03-15 16:59:58 +01:00
mvdbeek 66406ffa72 Fix yaml preview in firefox
by using standard text/plain content-type.
Fixes https://github.com/galaxyproject/galaxy/issues/13295.
2022-03-15 16:04:10 +01:00
mvdbeek 7990527d88 Drop request for feedback 2022-03-15 15:46:54 +01:00
mvdbeek 91fd713fb0 Fix one more case of Python 3.6 2022-03-15 15:38:03 +01:00
mvdbeek b70e231587 Update release notes 2022-03-15 15:33:45 +01:00
mvdbeek 5db8c57731 Also update python.md for minium python version (3.7) 2022-03-15 15:23:39 +01:00
mvdbeek a211132bab Merge branch 'release_21.09' into release_22.01 2022-03-15 15:14:59 +01:00
mvdbeek 525f443222 Merge branch 'release_21.05' into release_21.09 2022-03-15 15:11:26 +01:00
Marius van den Beek 083d94dfca Merge pull request #13531 from mvdbeek/include_gravity_in_sample_config
[22.01] Include gravity in sample config
2022-03-15 14:47:49 +01:00
mvdbeek 90672de88d Rebuild sample for gravity 0.11.0 2022-03-15 12:02:06 +01:00
Björn Grüning 99835a86bf Merge pull request #13537 from davelopez/patch_21.05_allow_admins_access_datasets
[21.05] Allow admin access to datasets API (before refactoring)
2022-03-14 21:30:26 +01:00
Björn Grüning 7bc55b9cb6 Merge pull request #13538 from davelopez/patch_21.09_allow_admin_access_private_datasets
[21.09] Allow admin access to datasets API (before refactoring)
2022-03-14 21:29:37 +01:00
John Chilton ffd916f10c Fixes for LDDA manager - clearly that didn't work at all. 2022-03-14 18:23:15 +01:00
John Chilton 1330da224f Fixes for LDDA manager - clearly that didn't work at all. 2022-03-14 18:21:03 +01:00
Dannon 4e742cdfcf Merge pull request #13536 from mvdbeek/content_id_issue
[22.01] Return actual tool id instead of specified tool id in workflow step
2022-03-14 10:49:54 -04:00
Dannon c8901b1bb1 Merge pull request #13514 from mvdbeek/support_preload
[22.01] Support ``--preload`` and worker_id for server processes
2022-03-14 10:24:29 -04:00
davelopez 248ec4e87b Allow admin access to datasets API (before refactoring)
This patch contains the changes in #13060 and it is meant to be applied only to 21.09 do not merge forward this commit into 22.01 since it is already fixed there
2022-03-14 15:05:17 +01:00
davelopez 71742ac339 Allow admin access to datasets API (before refactoring)
This patch contains the changes in #13060 and it is meant to be applied only to 21.05 do not merge forward this commit into 22.01 since it is already fixed there
2022-03-14 14:50:22 +01:00
mvdbeek 4bb6802d31 Return actual tool id instead of specified tool id in step
We do also return the version changes in the module, so this should be
fine. Fixes https://github.com/galaxyproject/galaxy/issues/13524.
2022-03-14 13:37:50 +01:00
mvdbeek 0f65081127 Rebuild galaxy.yml.sample 2022-03-13 16:44:35 +01:00
mvdbeek 07a763eede Include gravity sample schema in galaxy.yml.sample 2022-03-13 16:44:11 +01:00
Marius van den Beek 68cf3c4754 Merge pull request #13523 from mvdbeek/revert_null_value_handling
[22.01] Revert "Merge pull request #13285 from mvdbeek/null_text_values"
2022-03-11 09:26:50 +01:00
Björn Grüning ed7a303c62 Merge pull request #13522 from mvdbeek/backport_reset_button_fix_rule_builder
[22.01] Backport reset button fix rule builder
2022-03-10 21:03:24 +01:00
mvdbeek c523529411 Revert "Merge pull request #13285 from mvdbeek/null_text_values"
This reverts commit 2549792d42, reversing
changes made to 0223188e59.

Fixes https://github.com/galaxyproject/galaxy/issues/13516, at the cost
of not respecting required text parameters.
2022-03-10 20:56:20 +01:00
Marius van den BeekandNicola Soranzo c98e13befe Skip unnecessary list comprehension
Co-authored-by: Nicola Soranzo <nicola.soranzo@gmail.com>
2022-03-10 19:41:03 +01:00