diff --git a/doc/source/admin/tool_source_storage.rst b/doc/source/admin/tool_source_storage.rst index 42d6b3bce39..9e7a95fefaa 100644 --- a/doc/source/admin/tool_source_storage.rst +++ b/doc/source/admin/tool_source_storage.rst @@ -157,7 +157,6 @@ Command Line Options --verbose, -v Verbose output --watch, -w Watch tool directories and send reload notifications --watch-polling Use polling observer (for NFS/CVMFS/network FS) - --debounce SECS Debounce time for watch mode (default: 2.0) Examples ^^^^^^^^ @@ -207,13 +206,12 @@ Watch mode options: - ``--watch, -w`` - Enable watch mode - ``--watch-polling`` - Use polling observer (required for network filesystems like NFS/CVMFS) -- ``--debounce SECS`` - Debounce time for file changes (default: 2.0 seconds) Example with polling for network filesystem: .. code-block:: console - $ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling --debounce 5.0 + $ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling **Requirements:** diff --git a/doc/source/dev/tool_source_storage.rst b/doc/source/dev/tool_source_storage.rst index 6d8025482f6..379ca1d6141 100644 --- a/doc/source/dev/tool_source_storage.rst +++ b/doc/source/dev/tool_source_storage.rst @@ -30,14 +30,17 @@ Module Layout :: lib/galaxy/tools/source_store/ - __init__.py ToolSourceStore ABC, StoredToolSource, build_tool_source_store() + __init__.py Public re-exports + interface.py ToolSourceStore ABC and StoredToolSource + factory.py Store construction from Galaxy configuration sqlalchemy.py SqlAlchemyToolSourceStore (any SQLAlchemy URL) composite.py CompositeToolSourceStore (per-conf routing, merged index) index.py ToolIndex, ToolIndexEntry (the lightweight metadata) search.py ToolWhooshIndex (Whoosh search index built from a ToolIndex) discover.py discover_tools() — conf walk without booting a ToolBox populator.py Population + watch logic (parse, store, index, broadcast) - models.py Pydantic response models + freshness.py Optional external freshness probes + watcher.py Filesystem watch support scripts/tool_source/populate_store.py Thin CLI wrapper over populator.main @@ -54,7 +57,8 @@ deployments) — deliberately outside Galaxy's database: the store is a rebuildable cache and does not participate in Galaxy's migrations or session lifecycle. -**ToolIndex** — a single dataclass containing one ``ToolIndexEntry`` per tool, +**ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool +plus its versioned and panel-placement projections, holding everything a store consumer needs (id, name, description, panel section, labels, EDAM, requirements, container info, test counts, hidden/disabled, shed metadata). The index is serialized and gzip-compressed as a blob. @@ -154,10 +158,11 @@ Population Script ``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over ``galaxy.tools.source_store.populator.main``. It loads only the Galaxy -config and calls ``build_tool_source_store(config)`` — the standalone store -needs no datatypes registry or Galaxy model. Tools are parsed in a +config and calls ``build_tool_source_store(config)``. Converter discovery builds +the datatypes registry, but the standalone process does not initialize the Galaxy +model. Tools are parsed in a ``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is -hashed and skipped if an entry with the same hash already exists +matched to its source path and carried forward when its raw file hash is unchanged (``--incremental``, the default). Once the JSON index is committed the populator rebuilds the Whoosh search index (``search.py``) so ranked tool search stays in sync with the stored sources. diff --git a/lib/galaxy/tools/source_store/search.py b/lib/galaxy/tools/source_store/search.py index a7b79002fa3..cda9577df30 100644 --- a/lib/galaxy/tools/source_store/search.py +++ b/lib/galaxy/tools/source_store/search.py @@ -192,8 +192,7 @@ def _entry_to_doc(entry: ToolIndexEntry) -> dict | None: class ToolWhooshIndex: """Build + query a Whoosh index over ``ToolIndexEntry`` rows. - Build at populate time, - query from :meth:`ToolIndex.search`. The on-disk format is plain Whoosh — + Built at populate time and queried by store consumers. The on-disk format is plain Whoosh — no Galaxy-specific encoding, so an operator can reopen it offline. """ diff --git a/test/unit/scripts/tool_source/test_whoosh_dir.py b/test/unit/scripts/tool_source/test_whoosh_dir.py index b12024b3cf9..148cbf6d5cb 100644 --- a/test/unit/scripts/tool_source/test_whoosh_dir.py +++ b/test/unit/scripts/tool_source/test_whoosh_dir.py @@ -11,7 +11,6 @@ from galaxy.tools.source_store.populator import ( def test_default_store_maps_to_store_default_subdir(): - # Backwards-compat: LazyToolBox._get_search_index has been reading from assert ( whoosh_dir_for_store("/var/galaxy/tool_search", DEFAULT_STORE_NAME) == "/var/galaxy/tool_search/_store_default" )