mirror of
https://github.com/galaxyproject/galaxy.git
synced 2026-09-21 13:50:20 +08:00
source store: drop the resurrected cvmfs probe and the store-watch config surface
The rebase resurrected the cvmfs freshness probe that dev's7876758c12and1577ce6c06deliberately removed (boot trusts read-only stores as published; the stamped revision is populated inside the publish transaction and can never match). Restore freshness/factory/watcher to the dev versions and drop watch_tool_source_stores / tool_source_store_watch_interval plus the toolbox watcher wiring — with no cvmfs probe there is nothing left for the poll loop to watch. This also retires the reload-path stop_watcher() shim; without a watcher thread a superseded toolbox holds no background resources.
This commit is contained in:
@@ -455,22 +455,6 @@
|
||||
URI with ``mode=ro&uri=true``.
|
||||
For details see
|
||||
https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html
|
||||
A ``read_only`` store is trusted at boot whenever its persisted
|
||||
index loads under the current index schema: it is published
|
||||
together with the tools it indexes (the CVMFS model), so a
|
||||
schema-valid index is authoritative by construction and no
|
||||
per-tool coverage scan runs against it.
|
||||
An entry may also declare a freshness probe via ``freshness``.
|
||||
``freshness: cvmfs`` reads the CernVM-FS repository revision (one
|
||||
extended-attribute syscall) from ``freshness_path``, defaulting to
|
||||
the store's SQLite file location; combined with
|
||||
``watch_tool_source_stores`` it lets Galaxy detect and reload a
|
||||
newly published repository revision at runtime. ``freshness:
|
||||
tool_confs`` hashes the local tool conf files plus tool_dir
|
||||
directory mtimes and is compared at boot against the value the
|
||||
populator stamped — the right probe for a writable store populated
|
||||
on the same host; the default
|
||||
(``tool_source_database_connection``) store always uses it.
|
||||
:Default: ``None``
|
||||
:Type: map
|
||||
|
||||
@@ -501,35 +485,6 @@
|
||||
:Type: int
|
||||
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
``watch_tool_source_stores``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
:Description:
|
||||
Poll the freshness probes of read-only tool source stores and
|
||||
reload the lazy toolbox index when a store's token changes (for
|
||||
example, a new CVMFS repository revision was published).
|
||||
Filesystem event watchers do not fire on CVMFS, so polling is the
|
||||
only reliable signal; each poll costs one extended-attribute read
|
||||
per watched store. Only meaningful with ``use_lazy_toolbox`` and a
|
||||
``tool_source_stores`` entry declaring a freshness probe. The
|
||||
admin toolbox-reload API remains available as a manual trigger.
|
||||
:Default: ``false``
|
||||
:Type: bool
|
||||
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
``tool_source_store_watch_interval``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
:Description:
|
||||
Seconds between freshness polls when ``watch_tool_source_stores``
|
||||
is enabled. CVMFS clients only refresh their catalogs every few
|
||||
minutes, so sub-minute polling buys nothing.
|
||||
:Default: ``60.0``
|
||||
:Type: float
|
||||
|
||||
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
``tool_dependency_dir``
|
||||
~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
@@ -41,8 +41,6 @@ class GalaxyAppConfigurationAttributes:
|
||||
tool_source_stores: Any
|
||||
use_lazy_toolbox: bool
|
||||
lazy_toolbox_cache_size: int
|
||||
watch_tool_source_stores: bool
|
||||
tool_source_store_watch_interval: float
|
||||
tool_dependency_dir: str | None
|
||||
dependency_resolvers_config_file: str
|
||||
conda_prefix: str | None
|
||||
|
||||
@@ -611,22 +611,6 @@ galaxy:
|
||||
# with ``mode=ro&uri=true``.
|
||||
# For details see
|
||||
# https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html
|
||||
# A ``read_only`` store is trusted at boot whenever its persisted
|
||||
# index loads under the current index schema: it is published together
|
||||
# with the tools it indexes (the CVMFS model), so a schema-valid index
|
||||
# is authoritative by construction and no per-tool coverage scan runs
|
||||
# against it.
|
||||
# An entry may also declare a freshness probe via ``freshness``.
|
||||
# ``freshness: cvmfs`` reads the CernVM-FS repository revision (one
|
||||
# extended-attribute syscall) from ``freshness_path``, defaulting to
|
||||
# the store's SQLite file location; combined with
|
||||
# ``watch_tool_source_stores`` it lets Galaxy detect and reload a
|
||||
# newly published repository revision at runtime. ``freshness:
|
||||
# tool_confs`` hashes the local tool conf files plus tool_dir
|
||||
# directory mtimes and is compared at boot against the value the
|
||||
# populator stamped — the right probe for a writable store populated
|
||||
# on the same host; the default (``tool_source_database_connection``)
|
||||
# store always uses it.
|
||||
#tool_source_stores: null
|
||||
|
||||
# When true, use the LazyToolBox which loads tools on demand from the
|
||||
@@ -641,21 +625,6 @@ galaxy:
|
||||
# parsing cost for popular tools at the expense of memory.
|
||||
#lazy_toolbox_cache_size: 500
|
||||
|
||||
# Poll the freshness probes of read-only tool source stores and reload
|
||||
# the lazy toolbox index when a store's token changes (for example, a
|
||||
# new CVMFS repository revision was published). Filesystem event
|
||||
# watchers do not fire on CVMFS, so polling is the only reliable
|
||||
# signal; each poll costs one extended-attribute read per watched
|
||||
# store. Only meaningful with ``use_lazy_toolbox`` and a
|
||||
# ``tool_source_stores`` entry declaring a freshness probe. The admin
|
||||
# toolbox-reload API remains available as a manual trigger.
|
||||
#watch_tool_source_stores: false
|
||||
|
||||
# Seconds between freshness polls when ``watch_tool_source_stores`` is
|
||||
# enabled. CVMFS clients only refresh their catalogs every few
|
||||
# minutes, so sub-minute polling buys nothing.
|
||||
#tool_source_store_watch_interval: 60.0
|
||||
|
||||
# Various dependency resolver configuration parameters will have
|
||||
# defaults set relative to this path, such as the default conda
|
||||
# prefix, default Galaxy packages path, legacy tool shed dependencies
|
||||
|
||||
@@ -357,24 +357,6 @@ mapping:
|
||||
For details see
|
||||
https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html
|
||||
|
||||
A ``read_only`` store is trusted at boot whenever its persisted
|
||||
index loads under the current index schema: it is published
|
||||
together with the tools it indexes (the CVMFS model), so a
|
||||
schema-valid index is authoritative by construction and no
|
||||
per-tool coverage scan runs against it.
|
||||
|
||||
An entry may also declare a freshness probe via ``freshness``.
|
||||
``freshness: cvmfs`` reads the CernVM-FS repository revision (one
|
||||
extended-attribute syscall) from ``freshness_path``, defaulting
|
||||
to the store's SQLite file location; combined with
|
||||
``watch_tool_source_stores`` it lets Galaxy detect and reload a
|
||||
newly published repository revision at runtime.
|
||||
``freshness: tool_confs`` hashes the local tool conf files plus
|
||||
tool_dir directory mtimes and is compared at boot against the
|
||||
value the populator stamped — the right probe for a writable
|
||||
store populated on the same host; the default
|
||||
(``tool_source_database_connection``) store always uses it.
|
||||
|
||||
use_lazy_toolbox:
|
||||
type: bool
|
||||
required: false
|
||||
@@ -394,29 +376,6 @@ mapping:
|
||||
in its in-memory LRU cache. Larger values reduce repeat parsing cost
|
||||
for popular tools at the expense of memory.
|
||||
|
||||
watch_tool_source_stores:
|
||||
type: bool
|
||||
default: false
|
||||
required: false
|
||||
desc: |
|
||||
Poll the freshness probes of read-only tool source stores and
|
||||
reload the lazy toolbox index when a store's token changes (for
|
||||
example, a new CVMFS repository revision was published).
|
||||
Filesystem event watchers do not fire on CVMFS, so polling is the
|
||||
only reliable signal; each poll costs one extended-attribute read
|
||||
per watched store. Only meaningful with ``use_lazy_toolbox`` and
|
||||
a ``tool_source_stores`` entry declaring a freshness probe. The
|
||||
admin toolbox-reload API remains available as a manual trigger.
|
||||
|
||||
tool_source_store_watch_interval:
|
||||
type: float
|
||||
default: 60.0
|
||||
required: false
|
||||
desc: |
|
||||
Seconds between freshness polls when ``watch_tool_source_stores``
|
||||
is enabled. CVMFS clients only refresh their catalogs every few
|
||||
minutes, so sub-minute polling buys nothing.
|
||||
|
||||
tool_dependency_dir:
|
||||
type: str
|
||||
default: dependencies
|
||||
|
||||
@@ -296,7 +296,6 @@ def _get_new_toolbox(app: "UniverseApplication", save_integrated_tool_panel: boo
|
||||
tool_configs = app.config.tool_configs
|
||||
|
||||
with app._toolbox_lock:
|
||||
old_toolbox = app._toolbox
|
||||
new_toolbox: ToolBox
|
||||
if getattr(app.config, "use_lazy_toolbox", False) and getattr(app, "tool_source_store", None) is not None:
|
||||
new_toolbox = LazyToolBox(
|
||||
@@ -321,15 +320,6 @@ def _get_new_toolbox(app: "UniverseApplication", save_integrated_tool_panel: boo
|
||||
for tool in new_toolbox.data_manager_tools.values():
|
||||
new_toolbox.register_tool(tool)
|
||||
app._toolbox = new_toolbox
|
||||
# Retire the superseded box's background store watcher. In lazy mode
|
||||
# each toolbox spawns a ToolSourceStoreWatcher daemon bound to itself;
|
||||
# without this every reload leaks a polling thread that keeps grabbing
|
||||
# app._toolbox_lock and disposing the shared store's engines on each
|
||||
# republish. Only the watcher is stopped — in-flight reads on other
|
||||
# threads still work, and the shared app.tool_source_store stays open.
|
||||
# Eager ToolBoxes have no watcher, so guard on the lazy type.
|
||||
if isinstance(old_toolbox, LazyToolBox) and old_toolbox is not new_toolbox:
|
||||
old_toolbox.stop_watcher()
|
||||
|
||||
|
||||
def reload_data_managers(app, **kwargs):
|
||||
|
||||
@@ -50,12 +50,10 @@ from galaxy.tools.source_store.index import (
|
||||
ToolPanelItem,
|
||||
)
|
||||
from galaxy.tools.source_store.populator import (
|
||||
build_whoosh_for_store,
|
||||
conf_to_store_map,
|
||||
populate_for_paths,
|
||||
populate_store_inline,
|
||||
)
|
||||
from galaxy.tools.source_store.watcher import ToolSourceStoreWatcher
|
||||
from galaxy.util import listify
|
||||
from galaxy.util.tool_version import remove_version_from_guid
|
||||
from . import (
|
||||
@@ -503,7 +501,6 @@ class LazyToolBox(ToolBox):
|
||||
# only the guid landed in the panel. Filled after the eager walk
|
||||
# via ``_rebuild_shed_short_id_map``.
|
||||
self._shed_short_id_to_guids: dict[str, set[str]] = {}
|
||||
self._store_watcher: ToolSourceStoreWatcher | None = None
|
||||
# Identity-keyed cache of every indexed ``source_path`` — see
|
||||
# ``_index_source_paths``. Set before ``super().__init__`` because
|
||||
# the eager walk consults it through ``_tool_file_on_disk``.
|
||||
@@ -538,59 +535,6 @@ class LazyToolBox(ToolBox):
|
||||
self._tools_loaded_from_store,
|
||||
)
|
||||
|
||||
self._start_store_watcher()
|
||||
|
||||
def _start_store_watcher(self) -> None:
|
||||
"""Poll externally-published stores for freshness-token changes.
|
||||
|
||||
Only read-only members with a probe are watched: writable stores
|
||||
change through this process's own populate paths, which broadcast
|
||||
their own reloads, and CVMFS (the read-only publishing model)
|
||||
delivers no filesystem events to react to — polling one
|
||||
extended-attribute read per store per tick is the whole cost.
|
||||
"""
|
||||
if not self.app.config.watch_tool_source_stores:
|
||||
return
|
||||
if not isinstance(self._store, CompositeToolSourceStore):
|
||||
log.info("watch_tool_source_stores is enabled but no named tool source stores are configured")
|
||||
return
|
||||
members = [(n, m) for n, m in self._store.members if m.read_only and m.has_freshness_probe]
|
||||
if not members:
|
||||
log.info("watch_tool_source_stores is enabled but no read-only store declares a freshness probe")
|
||||
return
|
||||
self._store_watcher = ToolSourceStoreWatcher(
|
||||
members=members,
|
||||
interval=self.app.config.tool_source_store_watch_interval,
|
||||
on_change=self._on_store_freshness_change,
|
||||
)
|
||||
self._store_watcher.start()
|
||||
log.info(
|
||||
"Watching tool source store(s) %s for freshness changes every %gs",
|
||||
sorted(n for n, _ in members),
|
||||
self.app.config.tool_source_store_watch_interval,
|
||||
)
|
||||
|
||||
def _on_store_freshness_change(self, changed_names: list[str]) -> None:
|
||||
"""A watched store was republished: reload index state and search.
|
||||
|
||||
``invalidate_index_cache`` handles the reload dance (it also
|
||||
disposes read-only members' engines — see there). The whoosh
|
||||
rebuild runs here rather than in the reload path because only a
|
||||
republished store can grow the corpus without a local populate;
|
||||
its corpus-signature check makes re-runs no-ops, and concurrent
|
||||
rebuilds from peer processes degrade to one winner (whoosh lock,
|
||||
errors swallowed and logged by ``build_whoosh_for_store``).
|
||||
"""
|
||||
self.invalidate_index_cache()
|
||||
if not isinstance(self._store, CompositeToolSourceStore):
|
||||
return
|
||||
for name, member in self._store.members:
|
||||
if name not in changed_names:
|
||||
continue
|
||||
index = member.load_index()
|
||||
if index is not None:
|
||||
build_whoosh_for_store(self.app.config, name, index)
|
||||
|
||||
def _init_tools_from_configs(self, config_filenames: list[str]) -> None:
|
||||
"""Load the persistent ``ToolIndex`` before delegating to the eager walk.
|
||||
|
||||
@@ -1792,18 +1736,6 @@ class LazyToolBox(ToolBox):
|
||||
if entry and entry.hidden:
|
||||
tool.hidden = True
|
||||
|
||||
def stop_watcher(self) -> None:
|
||||
"""Stop the background store-freshness watcher, if one is running.
|
||||
|
||||
The reload path uses this to retire a superseded toolbox's watcher
|
||||
thread without the class-level ``ToolLineage.reset()`` and index/store
|
||||
teardown of ``close()`` — the replacement box is already live and the
|
||||
shared ``tool_source_store`` must stay open. Idempotent.
|
||||
"""
|
||||
if self._store_watcher is not None:
|
||||
self._store_watcher.shutdown()
|
||||
self._store_watcher = None
|
||||
|
||||
def close(self) -> None:
|
||||
"""Drop in-memory state at app shutdown.
|
||||
|
||||
@@ -1813,7 +1745,6 @@ class LazyToolBox(ToolBox):
|
||||
``tool_source_store`` before the next boot wires up a fresh
|
||||
toolbox. Idempotent; safe to call more than once.
|
||||
"""
|
||||
self.stop_watcher()
|
||||
with self._cache_lock:
|
||||
self._tool_object_cache.clear()
|
||||
self._tool_index = None
|
||||
|
||||
@@ -3,12 +3,9 @@
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from sqlalchemy.engine import make_url
|
||||
|
||||
from galaxy.tool_util.toolbox.parser import get_toolbox_parser
|
||||
from .composite import CompositeToolSourceStore
|
||||
from .freshness import (
|
||||
cvmfs_probe,
|
||||
FreshnessProbe,
|
||||
tool_confs_probe,
|
||||
)
|
||||
@@ -39,19 +36,6 @@ def _build_default_store(
|
||||
return SqlAlchemyToolSourceStore(url=url, read_only=False, freshness_probe=tool_confs_probe(config))
|
||||
|
||||
|
||||
def _sqlite_database_path(url: str) -> str | None:
|
||||
"""On-disk file behind a sqlite URL, or None for other backends."""
|
||||
parsed = make_url(url)
|
||||
if parsed.drivername.split("+")[0] != "sqlite":
|
||||
return None
|
||||
database = parsed.database
|
||||
if not database or database == ":memory:":
|
||||
return None
|
||||
if database.startswith("file:"):
|
||||
database = database[len("file:") :].split("?", 1)[0]
|
||||
return database or None
|
||||
|
||||
|
||||
def _build_freshness_probe(
|
||||
name: str,
|
||||
spec: dict,
|
||||
@@ -62,15 +46,8 @@ def _build_freshness_probe(
|
||||
return None
|
||||
if freshness == "tool_confs":
|
||||
return tool_confs_probe(config)
|
||||
if freshness == "cvmfs":
|
||||
probe_path = spec.get("freshness_path") or _sqlite_database_path(spec["url"])
|
||||
if not probe_path:
|
||||
raise ConfigurationError(
|
||||
f"tool_source_stores[{name!r}] uses freshness: cvmfs with a non-file url; set freshness_path"
|
||||
)
|
||||
return cvmfs_probe(probe_path)
|
||||
raise ConfigurationError(
|
||||
f"tool_source_stores[{name!r}] freshness must be 'cvmfs', 'tool_confs', or 'none' (got {freshness!r})"
|
||||
f"tool_source_stores[{name!r}] freshness must be 'tool_confs' or 'none' (got {freshness!r})"
|
||||
)
|
||||
|
||||
|
||||
@@ -82,7 +59,7 @@ def build_named_store(
|
||||
"""Build a single named store from a ``tool_source_stores`` entry.
|
||||
|
||||
``spec`` is the dict from galaxy.yml - a SQLAlchemy ``url`` plus
|
||||
optional ``read_only``, ``freshness``, and ``freshness_path`` keys.
|
||||
optional ``read_only`` and ``freshness`` keys.
|
||||
Named stores get no probe unless one is declared: a store populated on
|
||||
a different host (the CVMFS publishing model) would never match a
|
||||
locally-computed conf hash, so ``tool_confs`` cannot be the default
|
||||
|
||||
@@ -7,21 +7,18 @@ compares. A match certifies the store still covers the current tree, so
|
||||
boot skips the per-path coverage scan (and the populate it would trigger).
|
||||
A mismatch is always safe — it only falls back to scanning/populating.
|
||||
|
||||
Two probe kinds exist:
|
||||
The built-in probe is ``tool_confs``: md5 over the tool and data-manager
|
||||
conf file contents, plus the (recursive) directory mtimes of any
|
||||
``tool_dir`` entries they declare. This captures tool
|
||||
additions/removals/renames — the same class of drift the coverage scan
|
||||
detects — without touching individual tool files. In-place edits to a
|
||||
tool's XML are invisible to both, by design: content changes are the
|
||||
incremental populate's job (raw-file md5), not the coverage check's.
|
||||
Wired to the default (writable) store automatically.
|
||||
|
||||
- ``tool_confs``: md5 over the tool and data-manager conf file contents,
|
||||
plus the (recursive) directory mtimes of any ``tool_dir`` entries they
|
||||
declare. This captures tool additions/removals/renames — the same class
|
||||
of drift the coverage scan detects — without touching individual tool
|
||||
files. In-place edits to a tool's XML are invisible to both, by design:
|
||||
content changes are the incremental populate's job (raw-file md5), not
|
||||
the coverage check's. Wired to the default (writable) store
|
||||
automatically.
|
||||
- ``cvmfs``: the CernVM-FS repository revision, read from the
|
||||
``user.revision`` extended attribute the CVMFS client exposes on the
|
||||
mount point. One syscall covers every file in the repository. For a
|
||||
store whose sqlite bundle is published in the same CVMFS transaction as
|
||||
the tools it indexes, a matching revision is a hard consistency proof.
|
||||
Read-only stores need no probe for boot freshness — they are trusted as
|
||||
published (see ``SqlAlchemyToolSourceStore.index_is_fresh``); a probe on
|
||||
such a store only feeds the watcher's change detection.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
@@ -99,38 +96,3 @@ def tool_confs_token(config: "GalaxyAppConfiguration") -> str:
|
||||
|
||||
def tool_confs_probe(config: "GalaxyAppConfiguration") -> FreshnessProbe:
|
||||
return lambda: tool_confs_token(config)
|
||||
|
||||
|
||||
def _os_getxattr(path: str, attribute: str) -> bytes:
|
||||
# ``os.getxattr`` only exists on Linux; CVMFS deployments are Linux.
|
||||
getxattr = getattr(os, "getxattr", None)
|
||||
if getxattr is None:
|
||||
raise FreshnessProbeError("extended attributes are not supported on this platform")
|
||||
return getxattr(path, attribute)
|
||||
|
||||
|
||||
def cvmfs_revision_token(path: str, _getxattr: Callable[[str, str], bytes] = _os_getxattr) -> str:
|
||||
"""CVMFS repository revision token for the repository containing ``path``.
|
||||
|
||||
The CVMFS client exposes repository metadata as extended attributes on
|
||||
the mount point, so ascend from ``path`` until ``user.revision``
|
||||
answers. Raises :class:`FreshnessProbeError` when no ancestor exposes
|
||||
it — ``path`` isn't on CVMFS, or the repository isn't mounted (in
|
||||
which case its tools are unreadable anyway, and "not fresh" is the
|
||||
right verdict).
|
||||
"""
|
||||
probe_path = os.path.abspath(path)
|
||||
while True:
|
||||
try:
|
||||
revision = _getxattr(probe_path, "user.revision")
|
||||
except OSError:
|
||||
parent = os.path.dirname(probe_path)
|
||||
if parent == probe_path:
|
||||
raise FreshnessProbeError(f"no CVMFS revision xattr found on any ancestor of {path}")
|
||||
probe_path = parent
|
||||
continue
|
||||
return f"cvmfs:{os.path.basename(probe_path)}:{revision.decode()}"
|
||||
|
||||
|
||||
def cvmfs_probe(path: str) -> FreshnessProbe:
|
||||
return lambda: cvmfs_revision_token(path)
|
||||
|
||||
@@ -4,8 +4,7 @@ A store published outside this Galaxy process — the CVMFS model, where a
|
||||
publisher repopulates the sqlite bundle in the same transaction that ships
|
||||
new tools — changes without any local filesystem event (inotify does not
|
||||
fire on CVMFS). Polling each store's freshness probe is the only reliable
|
||||
signal, and it is cheap: one probe per store per tick, a single
|
||||
extended-attribute read for CVMFS stores.
|
||||
signal, and it is cheap: one probe per store per tick.
|
||||
|
||||
The watcher itself is deliberately dumb: it detects token transitions and
|
||||
hands the changed store names to ``on_change``. Reload mechanics —
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
import threading
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from galaxy import queue_worker
|
||||
from galaxy.queue_worker import _get_new_toolbox
|
||||
from galaxy.tools.lazy_toolbox import LazyToolBox
|
||||
|
||||
|
||||
class FakeWatcher:
|
||||
def __init__(self):
|
||||
self.stopped = False
|
||||
|
||||
def shutdown(self):
|
||||
self.stopped = True
|
||||
|
||||
|
||||
def _fake_new_toolbox(*args, **kwargs):
|
||||
box = SimpleNamespace(data_manager_tools={})
|
||||
box.register_tool = lambda tool: None
|
||||
return box
|
||||
|
||||
|
||||
def _fake_app(old_toolbox):
|
||||
app = SimpleNamespace()
|
||||
app._toolbox_lock = threading.RLock()
|
||||
app._toolbox = old_toolbox
|
||||
app.toolbox = old_toolbox
|
||||
app.datatypes_registry = MagicMock()
|
||||
app.tool_source_store = None
|
||||
app.config = SimpleNamespace(
|
||||
use_lazy_toolbox=False,
|
||||
tool_configs=[],
|
||||
tool_path="/tmp/tools",
|
||||
lazy_toolbox_cache_size=500,
|
||||
)
|
||||
return app
|
||||
|
||||
|
||||
def _patch_builders(monkeypatch):
|
||||
monkeypatch.setattr(queue_worker, "ToolBox", _fake_new_toolbox)
|
||||
monkeypatch.setattr(queue_worker, "load_lib_tools", lambda toolbox: None)
|
||||
|
||||
|
||||
def test_replacement_stops_superseded_lazy_watcher(monkeypatch):
|
||||
_patch_builders(monkeypatch)
|
||||
old = LazyToolBox.__new__(LazyToolBox)
|
||||
old.data_manager_tools = {}
|
||||
watcher = FakeWatcher()
|
||||
old._store_watcher = watcher # type: ignore[assignment]
|
||||
app = _fake_app(old)
|
||||
|
||||
_get_new_toolbox(app)
|
||||
|
||||
assert watcher.stopped is True
|
||||
assert old._store_watcher is None
|
||||
assert app._toolbox is not old
|
||||
|
||||
|
||||
def test_eager_old_toolbox_is_left_untouched(monkeypatch):
|
||||
_patch_builders(monkeypatch)
|
||||
old = SimpleNamespace(data_manager_tools={})
|
||||
app = _fake_app(old)
|
||||
|
||||
_get_new_toolbox(app)
|
||||
|
||||
assert app._toolbox is not old
|
||||
assert not hasattr(old, "_store_watcher")
|
||||
@@ -2,11 +2,8 @@
|
||||
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
from galaxy.tools.source_store.composite import CompositeToolSourceStore
|
||||
from galaxy.tools.source_store.freshness import (
|
||||
cvmfs_revision_token,
|
||||
FreshnessProbeError,
|
||||
tool_confs_token,
|
||||
)
|
||||
@@ -53,24 +50,6 @@ def test_tool_confs_token_sees_tool_dir_membership_changes(tmp_path):
|
||||
assert tool_confs_token(cfg) != after_top_level
|
||||
|
||||
|
||||
def test_cvmfs_revision_token_ascends_to_the_mount_root():
|
||||
def fake_getxattr(path, attribute):
|
||||
if path == "/cvmfs/main.galaxyproject.org" and attribute == "user.revision":
|
||||
return b"1042"
|
||||
raise OSError(61, "no attribute")
|
||||
|
||||
token = cvmfs_revision_token("/cvmfs/main.galaxyproject.org/galaxy/store.sqlite", _getxattr=fake_getxattr)
|
||||
assert token == "cvmfs:main.galaxyproject.org:1042"
|
||||
|
||||
|
||||
def test_cvmfs_revision_token_raises_off_cvmfs():
|
||||
def fake_getxattr(path, attribute):
|
||||
raise OSError(61, "no attribute")
|
||||
|
||||
with pytest.raises(FreshnessProbeError):
|
||||
cvmfs_revision_token("/plain/local/path", _getxattr=fake_getxattr)
|
||||
|
||||
|
||||
def test_index_is_fresh_tracks_probe(tmp_path):
|
||||
current = {"token": "confs:a"}
|
||||
store = SqlAlchemyToolSourceStore(url=f"sqlite:///{tmp_path}/a.sqlite", freshness_probe=lambda: current["token"])
|
||||
@@ -92,7 +71,7 @@ def test_index_is_fresh_false_when_probe_fails(tmp_path):
|
||||
raise FreshnessProbeError("repo not mounted")
|
||||
|
||||
store = SqlAlchemyToolSourceStore(url=f"sqlite:///{tmp_path}/a.sqlite", freshness_probe=broken_probe)
|
||||
store.store_index(ToolIndex(freshness_token="cvmfs:r:1"))
|
||||
store.store_index(ToolIndex(freshness_token="bundle:1"))
|
||||
assert store.index_is_fresh() is False
|
||||
|
||||
|
||||
@@ -102,14 +81,14 @@ def _stamped_store(path, token, probe_token, read_only=False):
|
||||
|
||||
|
||||
def test_composite_fresh_when_all_members_fresh(tmp_path):
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "cvmfs:r:1", "cvmfs:r:1", read_only=True)
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "bundle:1", "bundle:1", read_only=True)
|
||||
rw = _stamped_store(tmp_path / "rw.sqlite", "confs:x", "confs:x")
|
||||
composite = CompositeToolSourceStore(members=[("ro", ro), ("rw", rw)], default="rw")
|
||||
assert composite.index_is_fresh() is True
|
||||
|
||||
|
||||
def test_composite_stale_writable_member_wins(tmp_path):
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "cvmfs:r:1", "cvmfs:r:1", read_only=True)
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "bundle:1", "bundle:1", read_only=True)
|
||||
rw = _stamped_store(tmp_path / "rw.sqlite", "confs:x", "confs:y")
|
||||
composite = CompositeToolSourceStore(members=[("ro", ro), ("rw", rw)], default="rw")
|
||||
assert composite.index_is_fresh() is False
|
||||
@@ -118,14 +97,14 @@ def test_composite_stale_writable_member_wins(tmp_path):
|
||||
def test_read_only_store_trusts_schema_valid_index_over_probe(tmp_path):
|
||||
# Stamped token and probe value disagree, but a read-only store is
|
||||
# trusted whenever its index loads — the probe only feeds the watcher.
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "cvmfs:r:1", "cvmfs:r:2", read_only=True)
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "bundle:1", "bundle:2", read_only=True)
|
||||
assert ro.index_is_fresh() is True
|
||||
|
||||
|
||||
def test_read_only_store_without_index_is_not_fresh(tmp_path):
|
||||
SqlAlchemyToolSourceStore(url=f"sqlite:///{tmp_path}/ro.sqlite")
|
||||
ro = SqlAlchemyToolSourceStore(
|
||||
url=f"sqlite:///{tmp_path}/ro.sqlite", read_only=True, freshness_probe=lambda: "cvmfs:r:1"
|
||||
url=f"sqlite:///{tmp_path}/ro.sqlite", read_only=True, freshness_probe=lambda: "bundle:1"
|
||||
)
|
||||
assert ro.index_is_fresh() is False
|
||||
|
||||
@@ -141,7 +120,7 @@ def test_composite_read_only_member_without_index_warns_but_stays_fresh(tmp_path
|
||||
|
||||
|
||||
def test_composite_member_without_probe_downgrades_to_none(tmp_path):
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "cvmfs:r:1", "cvmfs:r:1", read_only=True)
|
||||
ro = _stamped_store(tmp_path / "ro.sqlite", "bundle:1", "bundle:1", read_only=True)
|
||||
rw = SqlAlchemyToolSourceStore(url=f"sqlite:///{tmp_path}/rw.sqlite")
|
||||
composite = CompositeToolSourceStore(members=[("ro", ro), ("rw", rw)], default="rw")
|
||||
assert composite.index_is_fresh() is None
|
||||
|
||||
Reference in New Issue
Block a user