diff --git a/doc/source/admin/galaxy_options.rst b/doc/source/admin/galaxy_options.rst index 9dc56d9bd97..f07797eef2a 100644 --- a/doc/source/admin/galaxy_options.rst +++ b/doc/source/admin/galaxy_options.rst @@ -420,6 +420,45 @@ :Type: str +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +``tool_source_database_connection`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:Description: + SQLAlchemy connection string for the tool source store, a + rebuildable cache of pre-parsed tool sources kept outside Galaxy's + main database. Multi-host deployments should point every Galaxy + process at the same URI, such as a SQLite file on a shared + filesystem. + Sample default ``sqlite:////tool_sources.sqlite``. + Populate the store with: python + scripts/tool_source/populate_store.py + For details see + https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html +:Default: ``None`` +:Type: str + + +~~~~~~~~~~~~~~~~~~~~~~ +``tool_source_stores`` +~~~~~~~~~~~~~~~~~~~~~~ + +:Description: + Optional named tool source stores referenced from individual + tool_conf files via a top-level ``store=""`` attribute (XML) + or ``store: `` key (YAML). When any tool_conf opts in, the + process composes its named store with the default + (``tool_source_database_connection``) store at runtime, with reads + tried in declared order and writes always landing on the default. + Each entry takes a SQLAlchemy ``url`` and an optional ``read_only: + true`` flag. For SQLite connection-level read-only, use a SQLite + URI with ``mode=ro&uri=true``. + For details see + https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html +:Default: ``None`` +:Type: map + + ~~~~~~~~~~~~~~~~~~~~~~~ ``tool_dependency_dir`` ~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index 7b065f21d0f..13cace25439 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -22,6 +22,7 @@ Galaxy Deployment & Administration ai_agents enable_headers_in_fetch_requests tool_panel + tool_source_storage data_tables mq dependency_resolvers diff --git a/doc/source/admin/tool_source_storage.rst b/doc/source/admin/tool_source_storage.rst new file mode 100644 index 00000000000..59348a52f1f --- /dev/null +++ b/doc/source/admin/tool_source_storage.rst @@ -0,0 +1,274 @@ +Tool Source Storage +=================== + +Overview +-------- + +By default, Galaxy parses every tool at startup and keeps all of them in +memory. For installations with many tools this: + +- Slows down Galaxy startup significantly +- Consumes large amounts of memory in every Galaxy process + +Tool source storage addresses this by doing the parsing work once, ahead of +time: + +1. Tool sources are pre-parsed (with macros expanded) and stored in a + configurable database backend +2. A lightweight index over the stored tools supports fast tool listings and + search without touching tool files + +A toolbox that consumes this store to load tools on demand is planned as +follow-up work; this document covers the store, the populator, and the index +that it will build on. + +Configuration +------------- + +Tool source storage is configured in ``galaxy.yml``. The following options are available: + +Default Store +^^^^^^^^^^^^^ + +.. code-block:: yaml + + galaxy: + # SQLAlchemy URI for storing tool sources. + tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite + +The store lives in a standalone database - a SQLite file under +``/tool_sources.sqlite`` by default - separate from Galaxy's main +database. It is a rebuildable cache: it can be deleted at any time and +recreated by re-running the population script. + +Multi-host deployments must point every Galaxy process (web workers *and* +job handlers) at the same store — typically a SQLite file on a shared +filesystem: + +.. code-block:: yaml + + galaxy: + tool_source_database_connection: sqlite:////shared/galaxy/tool_sources.sqlite + +Any other SQLAlchemy-supported database (e.g. PostgreSQL) works as well. + +Per-conf Store Routing (CVMFS Recipe) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Individual ``tool_conf`` files can opt into a *named* tool source store, +distinct from the global default. The typical use case is shipping a +read-only SQLite bundle on CVMFS alongside a tool_conf, so worker +processes can resolve every tool in that conf with local-cached lookups +instead of one network round-trip per JSON file. + +Declare the named stores under the top-level ``tool_source_stores`` key in +``galaxy.yml``. Each entry takes a SQLAlchemy ``url`` and optional +``read_only`` flag. SQLite is the typical choice for CVMFS bundles (single +self-contained file), but any SQLAlchemy-supported database works: + +.. code-block:: yaml + + galaxy: + tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite + tool_source_stores: + cvmfs_main: + url: sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true + read_only: true + site_shared: + url: sqlite:///file:/shared/galaxy/tool_sources.sqlite?mode=ro&uri=true + read_only: true + +Then point the tool_conf at it via the root element's ``store`` attribute +(XML) or top-level key (YAML): + +.. code-block:: xml + + + +
+ + ... +
+ + +At startup, Galaxy inspects every ``tool_conf`` for the attribute, builds +the referenced stores, and wraps them with the writable default in a +composite store. Reads are tried in declared order (first hit wins) and +writes always go to the default store. If no tool_conf opts in, the +default store is used directly with zero overhead. + +**Building the bundle** + +Build the SQLite file from a writable host before shipping it: + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py -c galaxy.yml --target cvmfs_main + +Use ``--target`` to restrict population to a single named store; without +it, ``populate_store.py`` populates **every writable store** referenced +from a tool_conf in the same run. + +Once the bundle is in place on CVMFS (or any read-only mount), restart +Galaxy. The ``read_only: true`` flag prevents Galaxy from writing through that +store. For SQLite connection-level read-only, use ``mode=ro&uri=true`` in the +SQLite URI as shown above. + +Populating the Tool Source Store +-------------------------------- + +After configuring tool source storage, you need to populate it with your tools. +Use the ``populate_store.py`` script: + +Basic Usage +^^^^^^^^^^^ + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py --config /path/to/galaxy.yml + +Deployments that install Galaxy from packages get the same command as the +``galaxy-populate-tool-source-store`` console script (shipped with the +``galaxy-app`` package), so no Galaxy source checkout is needed: + +.. code-block:: console + + $ galaxy-populate-tool-source-store --config /path/to/galaxy.yml + +This will: + +1. Discover tools from your tool configs (uses the same logic as Galaxy startup) +2. Parse each tool (with macro expansion) and compute a content hash +3. Store the tool sources in the configured backend (skipping unchanged tools) + +Note: ``--config`` is required; the script does not assume a default path. + +Command Line Options +^^^^^^^^^^^^^^^^^^^^ + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py --help + + Options: + --config, -c PATH Galaxy configuration file (required) + --dry-run Show what would be stored without storing + --incremental Only store new/changed tools (default) + --full Force re-store of all tools + --tool-id PATTERN Only process tools whose ID contains PATTERN + --parallel, -j N Number of parallel workers (default: 4) + --rebuild-index Rebuild the tool index after population + --target NAME Restrict to a single named store from + tool_source_stores (or '__default__'). Without + this, every writable store is populated. + --verbose, -v Verbose output + --watch, -w Watch tool directories and send reload notifications + --watch-polling Use polling observer (for NFS/CVMFS/network FS) + +Examples +^^^^^^^^ + +**Initial population:** + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py -c /path/to/galaxy.yml + +**Force re-store everything (e.g., after a parser change):** + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py -c galaxy.yml --full + +**Process only a subset of tools:** + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py -c galaxy.yml --tool-id samtools + +Automation with Cron +^^^^^^^^^^^^^^^^^^^^ + +For installations where tools are frequently updated, you can run the population +script on a schedule: + +.. code-block:: cron + + # Update tool source store every hour (incremental is the default) + 0 * * * * /path/to/galaxy/.venv/bin/python /path/to/galaxy/scripts/tool_source/populate_store.py -c /path/to/galaxy.yml >> /var/log/galaxy/tool_source_update.log 2>&1 + +Watch Mode (Live Updates) +^^^^^^^^^^^^^^^^^^^^^^^^^ + +As an alternative to cron, you can run the population script in watch mode to +keep the store continuously up to date. This uses ``watchdog`` to monitor tool +directories for changes and automatically updates the store, then sends a +notification via Kombu to trigger cache reloads in all Galaxy processes. + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py --config galaxy.yml --watch + +Watch mode options: + +- ``--watch, -w`` - Enable watch mode +- ``--watch-polling`` - Use polling observer (required for network filesystems like NFS/CVMFS) + +Example with polling for network filesystem: + +.. code-block:: console + + $ python scripts/tool_source/populate_store.py -c galaxy.yml --watch --watch-polling + +**Requirements:** + +- The ``watchdog`` library must be installed: ``pip install watchdog`` +- Galaxy must have ``amqp_internal_connection`` configured for Kombu notifications +- All Galaxy processes must be connected to the same AMQP broker + +When a tool XML file changes, watch mode will: + +1. Detect the file change (with debouncing to handle rapid edits) +2. Re-parse the tool and update the store +3. Send a ``reload_tool_source_cache`` control message via Kombu +4. All Galaxy processes will invalidate their local caches + +This is useful for: + +- Installations using shared storage where tools may be updated externally +- CI/CD pipelines that deploy tool updates +- Development environments where tools are being actively edited + +Troubleshooting +--------------- + +Tools not appearing in the index +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +1. Re-run the population script with verbose output: + + .. code-block:: console + + $ python scripts/tool_source/populate_store.py -c galaxy.yml -v + +2. Check for parsing errors in the Galaxy log + +Populating an existing installation +----------------------------------- + +To set up tool source storage on an existing Galaxy installation: + +1. Add the configuration to ``galaxy.yml``: + + .. code-block:: yaml + + galaxy: + tool_source_database_connection: sqlite:////srv/galaxy/tool_sources.sqlite + +2. Run the population script: + + .. code-block:: console + + $ python scripts/tool_source/populate_store.py -c /path/to/galaxy.yml + +3. Restart Galaxy diff --git a/doc/source/dev/index.rst b/doc/source/dev/index.rst index 23e212d0a63..067764584d9 100644 --- a/doc/source/dev/index.rst +++ b/doc/source/dev/index.rst @@ -20,6 +20,7 @@ A multi-hour long video playlist covering these slides can be found at data_managers data_source data_types + tool_source_storage faq writing_tests debugging_tests diff --git a/doc/source/dev/tool_source_storage.rst b/doc/source/dev/tool_source_storage.rst new file mode 100644 index 00000000000..fc64e1cb2d5 --- /dev/null +++ b/doc/source/dev/tool_source_storage.rst @@ -0,0 +1,215 @@ +Tool Source Storage Architecture +================================ + +This document describes the architecture of the tool source storage subsystem: +the store backends, the populator, and the index they build. For operator-facing +setup and configuration, see :doc:`/admin/tool_source_storage`. + +A toolbox that consumes this store to load tools on demand is planned as +follow-up work; the pieces documented here are the storage layer it will build on. + +Goals +----- + +The traditional ``ToolBox`` parses every tool XML at startup, builds full +``Tool`` objects, and keeps them all in memory. With thousands of tools that +scales poorly: slow boot, large per-process RSS, and expensive worker reloads. + +The tool source storage subsystem moves that parsing work out of the request +path: + +- A separate process (``populate_store.py``) parses tools once and persists + the canonical, macro-expanded source plus a lightweight metadata index. +- The store and index are laid out so a consumer can load only the index at + startup and materialize ``Tool`` objects on demand, instead of parsing the + full tree in-process. That consumer is the planned follow-up toolbox. + +Module Layout +------------- + +:: + + lib/galaxy/tools/source_store/ + __init__.py Public re-exports + interface.py ToolSourceStore ABC and StoredToolSource + factory.py Store construction from Galaxy configuration + sqlalchemy.py SqlAlchemyToolSourceStore (any SQLAlchemy URL) + composite.py CompositeToolSourceStore (per-conf routing, merged index) + index.py ToolIndex, ToolIndexEntry (the lightweight metadata) + search.py ToolWhooshIndex (Whoosh search index built from a ToolIndex) + discover.py discover_tools() — conf walk without booting a ToolBox + populator.py Population + watch logic (parse, store, index, broadcast) + freshness.py Optional external freshness probes + watcher.py Filesystem watch support + + scripts/tool_source/populate_store.py Thin CLI wrapper over populator.main + +The same ``populator.main`` is registered as the +``galaxy-populate-tool-source-store`` console script in the ``galaxy-app`` +package metadata (``packages/app/pyproject.toml``). + +Data Model +---------- + +Two persistence concepts: + +**StoredToolSource** — the canonical macro-expanded XML/YAML for a tool, +keyed by SHA-256 of the expanded content. Multiple versions of the same +``tool_id`` coexist as separate hashes. The store keeps its own schema in a +standalone database (a SQLite file by default, any SQLAlchemy URL for shared +deployments) — deliberately outside Galaxy's database: the store is a +rebuildable cache and does not participate in Galaxy's migrations or session +lifecycle. + +**ToolIndex** — a Pydantic model containing one default ``ToolIndexEntry`` per tool +plus its versioned and panel-placement projections, +holding everything a store consumer needs (id, name, description, panel section, +labels, EDAM, requirements, container info, test counts, hidden/disabled, +shed metadata). The index is serialized and gzip-compressed as a blob. + +The schema is auto-created on first open; ``tool_index`` holds a single +row per index version. + +Backend Abstraction +------------------- + +``ToolSourceStore`` (in ``tools/source_store/interface.py``) is an ABC defining: + +- ``store/get/exists/delete/list_all/get_by_tool_id/count`` — per-tool source + operations, all keyed by content hash. +- ``store_index/load_index/update_index_entry`` — index operations. +- ``get_stats()`` — backend-specific stats (count, size, backend name). + +``build_tool_source_store(config)`` is the only entry point used +by Galaxy. It builds the default store from +``config.tool_source_database_connection`` and uses the same SQLAlchemy-backed +store implementation for all configured URIs. ``ConfigurationError`` is raised +for missing required settings and is allowed to propagate up so +misconfiguration fails fast at startup. + +The ABC defines a ``read_only: bool`` class attribute (default ``False``). +``ReadOnlyStoreError`` is raised by mutating methods of stores that opted +in. The populator, watch reload, and composite all consult this flag to +route around read-only members rather than crashing. + +Per-conf composition +^^^^^^^^^^^^^^^^^^^^ + +If any tool_conf carries a top-level ``store="..."`` attribute (XML root) +or ``store: ...`` key (YAML), ``build_tool_source_store`` instantiates +the referenced named stores from ``config.tool_source_stores`` and wraps +them with the writable default in a :class:`CompositeToolSourceStore`. + +The composite implements the same ``ToolSourceStore`` interface, so store +consumers stay completely unaware of the multi-store layout: + +- **Reads** iterate ``[per-conf members..., default]`` in order; first + hit wins. ``count`` and ``list_all`` dedupe across members. +- **Writes** always land on the designated default. The default may not + itself be ``read_only``; that's enforced at construction. +- ``load_index()`` calls each member's ``load_index()`` and folds the + entries into a single :class:`ToolIndex`. Earlier members shadow later + ones on tool-id collisions; ``by_section`` is unioned; ``built_at`` + takes the most recent value. +- ``invalidate_index_cache()`` fans out so a single Kombu reload hits + every member. +- ``store_to(name, ...)`` lets the populator address a specific member by + name without going through composite write routing. + +When no tool_conf opts in, ``build_tool_source_store`` returns the +default store directly — the composite path is zero-cost for the common +case. + +The ``sqlalchemy`` backend (``sqlalchemy.py``) was added to make this +useful for CVMFS: a single self-contained ``.sqlite`` file, opened with +its own SQLAlchemy ``MetaData`` (independent of ``galaxy.model``) so the +file is portable, and openable with a SQLite URI such as +``sqlite:///file:/cvmfs/example.org/tools/sources.sqlite?mode=ro&uri=true`` +for read-only mounts. Despite the name, the backend is not sqlite-specific - +pass any SQLAlchemy URL (Postgres, MySQL, ...). Auto schema creation runs on +first open; on remote backends operators may prefer to manage migrations +explicitly. + +Per-conf populator routing +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``scripts/tool_source/populate_store.py`` is per-conf aware. It reads +``parse_store_name()`` for each tool_conf, builds every named store plus +the default, and routes each ``DiscoveredTool.path`` to the store its +conf points at. By default it populates *every* writable store in one +run; ``--target NAME`` restricts to a single store and raises +``ReadOnlyStoreError`` if that store is read-only. Tools whose target is +read-only in default mode are silently skipped (the bundle is treated as +authoritative for those entries). + +Discovery +--------- + +``galaxy.tools.source_store.discover.discover_tools`` walks tool config files +and yields ``DiscoveredTool`` records without booting a full ``ToolBox``. It is +used by: + +- the populator to find tools to parse and store. +- watch mode to know which directories to monitor. +- callers that compare on-disk confs against the indexed tool set. + +Pulling discovery out of ``ToolBox`` was deliberate: the populator must run +*without* a full app (or even a running Galaxy), and the watch mode must run in +a long-lived loop with no Galaxy process at all. + +Population Script +----------------- + +``scripts/tool_source/populate_store.py`` is a thin CLI wrapper over +``galaxy.tools.source_store.populator.main``. It loads only the Galaxy +config and calls ``build_tool_source_store(config)``. Converter discovery builds +the datatypes registry, but the standalone process does not initialize the Galaxy +model. Tools are parsed in a +``ThreadPoolExecutor`` (``--parallel``, default 4 workers); each tool is +matched to its source path and carried forward when its raw file hash is unchanged +(``--incremental``, the default). Once the JSON index is committed the +populator rebuilds the Whoosh search index (``search.py``) so ranked tool +search stays in sync with the stored sources. + +Watch mode (``--watch``) uses ``watchdog`` to monitor every directory yielded +by ``discover_tools``. File events are debounced (default 2 s), the changed +files are re-parsed, the store is updated, and a single +``reload_tool_source_cache`` Kombu control task is published on the Galaxy +exchange. ``--watch-polling`` switches to ``PollingObserver`` for +NFS/CVMFS/network filesystems where inotify is unreliable. + +The broadcast is the populator's half of the contract: it publishes +``reload_tool_source_cache`` so peer processes can drop their stale index +view. The control-task handler that consumes the message lands with the +follow-up toolbox. + +Design Notes +------------ + +**Why a separate index instead of always querying the store?** A consumer +needs O(N) access to N entries; doing that against the backing store on every +request is a latency hit. Keeping the index in-process and only paying for +invalidation on reload is the better tradeoff. + +**Why an out-of-process populator?** Parsing tools and computing macro +expansions is expensive and shouldn't block worker startup. Keeping the +populator separate also lets it run on a single host while many web workers +share the resulting store. + +**Why hash-keyed storage?** Content-addressed storage gives us cheap +deduplication across versions and shed installations, and idempotent +incremental updates: re-running the populator over an unchanged tree is +effectively a no-op. + +Testing +------- + +- Store unit tests: ``test/unit/app/tools/source_store/`` exercises each backend + through the ``ToolSourceStore`` interface (``test_stores.py``, + ``test_sqlite_store.py``, ``test_composite_store.py``, + ``test_index_versions.py``). +- Populator/discovery unit tests: ``test/unit/scripts/tool_source/`` + (``test_populate_store.py``, ``test_discover.py``, + ``test_build_index_entry.py``, ``test_whoosh_dir.py``). These use fakes + (not mocks) of ``ToolSourceStore`` so behavior is verified against the real + interface. diff --git a/lib/galaxy/app/__init__.py b/lib/galaxy/app/__init__.py index b571c16cae2..fa592f20172 100644 --- a/lib/galaxy/app/__init__.py +++ b/lib/galaxy/app/__init__.py @@ -376,16 +376,7 @@ class MinimalGalaxyApplication(BasicSharedApp, HaltableContainer, SentryClientMi log.warning("Waiting for toolbox reload timed out after 60 seconds") def _configure_tool_config_files(self): - if self.config.shed_tool_config_file not in self.config.tool_configs: - self.config.tool_configs.append(self.config.shed_tool_config_file) - # The value of migrated_tools_config is the file reserved for containing only those tools that have been - # eliminated from the distribution and moved to the tool shed. If migration checking is disabled, only add it if - # it exists (since this may be an existing deployment where migrations were previously run). - if ( - os.path.exists(self.config.migrated_tools_config) - and self.config.migrated_tools_config not in self.config.tool_configs - ): - self.config.tool_configs.append(self.config.migrated_tools_config) + self.config.tool_configs = self.config.all_tool_config_files() def _configure_toolbox(self): self.citations_manager = self._register_singleton(CitationsManager, CitationsManager(self)) diff --git a/lib/galaxy/app_unittest_utils/galaxy_mock.py b/lib/galaxy/app_unittest_utils/galaxy_mock.py index 89575823f38..3a421fb1a0a 100644 --- a/lib/galaxy/app_unittest_utils/galaxy_mock.py +++ b/lib/galaxy/app_unittest_utils/galaxy_mock.py @@ -290,6 +290,8 @@ class MockAppConfig(GalaxyDataTestConfig, CommonConfigurationMixin): self.track_jobs_in_database = False self.amqp_internal_connection = None self.tool_configs = [] + self.tool_source_database_connection = f"sqlite:///{os.path.join(self.data_dir, 'tool_sources.sqlite')}" + self.tool_source_stores = None self.manage_dependency_relationships = False self.enable_tool_shed_check = False self.monitor_thread_join_timeout = 1 diff --git a/lib/galaxy/config/__init__.py b/lib/galaxy/config/__init__.py index 8d0b78f7628..d7f0213118b 100644 --- a/lib/galaxy/config/__init__.py +++ b/lib/galaxy/config/__init__.py @@ -850,6 +850,8 @@ class GalaxyAppConfiguration(GalaxyAppConfigurationAttributes, BaseAppConfigurat if not self.database_connection: # Provide default if not supplied by user db_path = self._in_data_dir("universe.sqlite") self.database_connection = f"sqlite:///{db_path}?isolation_level=IMMEDIATE" + if not self.tool_source_database_connection: + self.tool_source_database_connection = f"sqlite:///{self._in_data_dir('tool_sources.sqlite')}" self.database_engine_options = get_database_engine_options(kwargs) self.database_encoding = kwargs.get("database_encoding") # Create new databases with this encoding self.thread_local_log = None @@ -1357,6 +1359,7 @@ class GalaxyAppConfiguration(GalaxyAppConfigurationAttributes, BaseAppConfigurat try_parsing(self.database_connection, "database_connection") try_parsing(self.install_database_connection, "install_database_connection") + try_parsing(self.tool_source_database_connection, "tool_source_database_connection") if self.interactivetoolsproxy_map is not None: try_parsing(self.interactivetoolsproxy_map, "interactivetoolsproxy_map") try_parsing(self.amqp_internal_connection, "amqp_internal_connection") @@ -1428,6 +1431,24 @@ class GalaxyAppConfiguration(GalaxyAppConfigurationAttributes, BaseAppConfigurat def ensure_tempdir(self): self._ensure_directory(self.new_file_path) + def all_tool_config_files(self) -> list[str]: + """Every tool config the toolbox loads: ``tool_config_file`` plus the + shed tool conf and, when present on disk, the migrated tools conf. + """ + configs = list(self.tool_configs or []) + if self.shed_tool_config_file and self.shed_tool_config_file not in configs: + configs.append(self.shed_tool_config_file) + # migrated_tools_config is reserved for tools eliminated from the + # distribution; only load it when it exists (an existing deployment + # where migrations were previously run). + if ( + self.migrated_tools_config + and os.path.exists(self.migrated_tools_config) + and self.migrated_tools_config not in configs + ): + configs.append(self.migrated_tools_config) + return configs + def check(self): # Check that required directories exist; attempt to create otherwise paths_to_check = [ diff --git a/lib/galaxy/config/_galaxy_config_schema_attributes.py b/lib/galaxy/config/_galaxy_config_schema_attributes.py index 965af791522..0aa51434645 100644 --- a/lib/galaxy/config/_galaxy_config_schema_attributes.py +++ b/lib/galaxy/config/_galaxy_config_schema_attributes.py @@ -37,6 +37,8 @@ class GalaxyAppConfigurationAttributes: migrated_tools_config: str integrated_tool_panel_config: str tool_path: str + tool_source_database_connection: str | None + tool_source_stores: Any tool_dependency_dir: str | None dependency_resolvers_config_file: str conda_prefix: str | None diff --git a/lib/galaxy/config/sample/galaxy.yml.sample b/lib/galaxy/config/sample/galaxy.yml.sample index d58b231215f..dd241728f7c 100644 --- a/lib/galaxy/config/sample/galaxy.yml.sample +++ b/lib/galaxy/config/sample/galaxy.yml.sample @@ -588,6 +588,31 @@ galaxy: # an attribute in the tag. #tool_path: tools + # SQLAlchemy connection string for the tool source store, a + # rebuildable cache of pre-parsed tool sources kept outside Galaxy's + # main database. Multi-host deployments should point every Galaxy + # process at the same URI, such as a SQLite file on a shared + # filesystem. + # Sample default ``sqlite:////tool_sources.sqlite``. + # Populate the store with: python + # scripts/tool_source/populate_store.py + # For details see + # https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html + #tool_source_database_connection: null + + # Optional named tool source stores referenced from individual + # tool_conf files via a top-level ``store=""`` attribute (XML) + # or ``store: `` key (YAML). When any tool_conf opts in, the + # process composes its named store with the default + # (``tool_source_database_connection``) store at runtime, with reads + # tried in declared order and writes always landing on the default. + # Each entry takes a SQLAlchemy ``url`` and an optional ``read_only: + # true`` flag. For SQLite connection-level read-only, use a SQLite URI + # with ``mode=ro&uri=true``. + # For details see + # https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html + #tool_source_stores: null + # Various dependency resolver configuration parameters will have # defaults set relative to this path, such as the default conda # prefix, default Galaxy packages path, legacy tool shed dependencies diff --git a/lib/galaxy/config/schemas/config_schema.yml b/lib/galaxy/config/schemas/config_schema.yml index 06b1aea32e1..733c94e6299 100644 --- a/lib/galaxy/config/schemas/config_schema.yml +++ b/lib/galaxy/config/schemas/config_schema.yml @@ -322,6 +322,41 @@ mapping: Other tool config files must include the tool_path as an attribute in the tag. + tool_source_database_connection: + type: str + required: false + desc: | + SQLAlchemy connection string for the tool source store, a + rebuildable cache of pre-parsed tool sources kept outside Galaxy's + main database. Multi-host deployments should point every Galaxy + process at the same URI, such as a SQLite file on a shared + filesystem. + + Sample default ``sqlite:////tool_sources.sqlite``. + + Populate the store with: python scripts/tool_source/populate_store.py + + For details see + https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html + + tool_source_stores: + type: map + required: false + desc: | + Optional named tool source stores referenced from individual + tool_conf files via a top-level ``store=""`` attribute (XML) + or ``store: `` key (YAML). When any tool_conf opts in, the + process composes its named store with the default + (``tool_source_database_connection``) store at runtime, with reads + tried in declared order and writes always landing on the default. + + Each entry takes a SQLAlchemy ``url`` and an optional + ``read_only: true`` flag. For SQLite connection-level read-only, + use a SQLite URI with ``mode=ro&uri=true``. + + For details see + https://docs.galaxyproject.org/en/master/admin/tool_source_storage.html + tool_dependency_dir: type: str default: dependencies diff --git a/lib/galaxy/queues/__init__.py b/lib/galaxy/queues/__init__.py index be1c476dc7a..9a5caf0b60e 100644 --- a/lib/galaxy/queues/__init__.py +++ b/lib/galaxy/queues/__init__.py @@ -7,9 +7,6 @@ All message queues used by Galaxy import datetime import logging import socket -from typing import ( - TYPE_CHECKING, -) from kombu import ( Connection, @@ -20,12 +17,11 @@ from sqlalchemy import ( or_, select, ) +from sqlalchemy.orm import Session from galaxy.model import WorkerProcess from galaxy.util import now - -if TYPE_CHECKING: - from galaxy.web_stack import ApplicationStack +from galaxy.web_stack import ApplicationStack log = logging.getLogger(__name__) @@ -41,7 +37,26 @@ WEBAPP_APP_TYPE = "webapp" SSE_MONITOR_APP_TYPE = "sse_monitor" -def all_control_queues_for_declare(application_stack: "ApplicationStack", webapp_only: bool = False) -> list[Queue]: +def control_queues_for_session(session: Session, webapp_only: bool = False) -> list[Queue]: + """Build the per-process control-queue declare list from a model session. + + Split out of :func:`all_control_queues_for_declare` so callers that have a + bare session but no ``ApplicationStack`` — notably the standalone + tool-source populator CLI — can build the same routing table. + """ + stmt = select(WorkerProcess).where( + WorkerProcess.update_time > now() - datetime.timedelta(seconds=DEFAULT_ACTIVE_PROCESS_WINDOW_SECONDS) + ) + if webapp_only: + stmt = stmt.where(WorkerProcess.app_type == WEBAPP_APP_TYPE) + else: + # ``!=`` alone would drop NULL app_type rows (job handlers); keep them. + stmt = stmt.where(or_(WorkerProcess.app_type != SSE_MONITOR_APP_TYPE, WorkerProcess.app_type.is_(None))) + processes = session.scalars(stmt).all() + return [Queue(f"control.{p.server_name}@{p.hostname}", galaxy_exchange, routing_key="control.*") for p in processes] + + +def all_control_queues_for_declare(application_stack: ApplicationStack, webapp_only: bool = False) -> list[Queue]: """ For in-memory routing (used by sqlalchemy-based transports), we need to be able to build the entire routing table in producers. @@ -65,20 +80,11 @@ def all_control_queues_for_declare(application_stack: "ApplicationStack", webapp """ app = application_stack.app try: - stmt = select(WorkerProcess).where( - WorkerProcess.update_time > now() - datetime.timedelta(seconds=DEFAULT_ACTIVE_PROCESS_WINDOW_SECONDS) - ) - if webapp_only: - stmt = stmt.where(WorkerProcess.app_type == WEBAPP_APP_TYPE) - else: - # ``!=`` alone would drop NULL app_type rows (job handlers); keep them. - stmt = stmt.where(or_(WorkerProcess.app_type != SSE_MONITOR_APP_TYPE, WorkerProcess.app_type.is_(None))) with app.model.new_session() as session: - processes = session.scalars(stmt).all() + return control_queues_for_session(session, webapp_only=webapp_only) except Exception: log.debug("Failed to look up active processes for control-queue declare", exc_info=True) return [] - return [Queue(f"control.{p.server_name}@{p.hostname}", galaxy_exchange, routing_key="control.*") for p in processes] def control_queues_from_config(config): diff --git a/lib/galaxy/tool_util/toolbox/base.py b/lib/galaxy/tool_util/toolbox/base.py index 379e665fb00..33e47744b18 100644 --- a/lib/galaxy/tool_util/toolbox/base.py +++ b/lib/galaxy/tool_util/toolbox/base.py @@ -5,6 +5,7 @@ import os import string import time from collections import namedtuple +from collections.abc import Iterator from errno import ENOENT from typing import ( Any, @@ -28,6 +29,7 @@ from galaxy.exceptions import ( RequestParameterInvalidException, ) from galaxy.util import ( + Element, etree, ExecutionTimer, listify, @@ -36,6 +38,7 @@ from galaxy.util import ( unicodify, ) from galaxy.util.bunch import Bunch +from galaxy.util.path import StrPath from .filters import FilterFactory from .integrated_panel import ManagesIntegratedToolPanelMixin from .lineages import LineageMap @@ -75,8 +78,6 @@ if TYPE_CHECKING: from galaxy.model.tool_shed_install import ToolShedRepository from galaxy.tools import Tool from galaxy.tools.cache import ToolCache - from galaxy.util import Element - from galaxy.util.path import StrPath log = logging.getLogger(__name__) @@ -161,6 +162,47 @@ class ToolLoadConfigurationConflict(Exception): pass +def walk_tool_directories(directory: StrPath, recursive: bool) -> Iterator[tuple[str, list[str]]]: + """Yield ``(directory, files)`` for ``directory`` and, when ``recursive``, + each subdirectory - skipping hidden/private (``.``/``_`` prefixed) entries. + + ``files`` are candidate tool file paths in sorted order; filtering them + (e.g. via ``looks_like_a_tool``) is the caller's responsibility. + """ + files = [] + subdirs = [] + for name in sorted(os.listdir(directory)): + if name.startswith((".", "_")): + # Very unlikely that we want to load tools from a hidden or private folder + continue + child = os.path.join(str(directory), name) + if os.path.isdir(child): + subdirs.append(child) + else: + files.append(child) + yield str(directory), files + if recursive: + for subdir in subdirs: + yield from walk_tool_directories(subdir, recursive) + + +def resolve_tool_path(tool_path: str | None, config_filename: str, default_tool_path: "StrPath | None" = None) -> str: + """Resolve a tool conf's ``tool_path`` attribute to the directory its tool + files are relative to. + + Expands the ``${tool_conf_dir}`` template; falls back to + ``default_tool_path`` (the toolbox's ``tool_root_dir``, i.e. + ``config.tool_path``) when the conf doesn't set one. + """ + if not tool_path: + # Default to backward compatible config setting. + return str(default_tool_path) if default_tool_path else "" + # Allow use of ${tool_conf_dir} in toolbox config files. + tool_conf_dir = os.path.dirname(config_filename) + tool_path_vars = {"tool_conf_dir": tool_conf_dir} + return string.Template(tool_path).safe_substitute(tool_path_vars) + + class AbstractToolBox(ManagesIntegratedToolPanelMixin): """ Abstract container for managing a ToolPanel - containing tools and @@ -281,7 +323,7 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): config_value = getattr(config, "default_panel_view", None) return config_value or self.__default_panel_view - def create_tool(self, config_file: "StrPath", **kwds) -> "Tool": + def create_tool(self, config_file: StrPath, **kwds: Any) -> "Tool": raise NotImplementedError() def create_dynamic_tool(self, dynamic_tool: "DynamicTool") -> "Tool": @@ -375,7 +417,7 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): config_elems = [] tool_conf_type = "shed tool" if parsing_shed_tool_conf else "tool" log.debug("Tool path for %s configuration %s is %s", tool_conf_type, config_filename, tool_path) - tool_path = self.__resolve_tool_path(tool_path, config_filename) + tool_path = resolve_tool_path(tool_path, config_filename, self._tool_root_dir) # Only load the panel_dict under certain conditions. load_panel_dict = not self._integrated_tool_panel_config_has_contents for item in tool_conf_source.parse_items(): @@ -550,17 +592,6 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): tool_id = tool.id return self._tool_panel.get_section_for_tool_id(tool_id) - def __resolve_tool_path(self, tool_path, config_filename): - if not tool_path: - # Default to backward compatible config setting. - tool_path = self._tool_root_dir - else: - # Allow use of __tool_conf_dir__ in toolbox config files. - tool_conf_dir = os.path.dirname(config_filename) - tool_path_vars = {"tool_conf_dir": tool_conf_dir} - tool_path = string.Template(tool_path).safe_substitute(tool_path_vars) - return tool_path - def add_tool_to_tool_panel_view(self, tool, view_panel_component): self.__add_tool_to_tool_panel(tool, view_panel_component) @@ -995,7 +1026,7 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): log.exception("Error reading tool from path: %s", path) def get_tool_repository_from_xml_item( - self, elem: "Element", path: str + self, elem: Element, path: str ) -> Union[ToolConfRepository, "ToolShedRepository"]: tool_shed_el = elem.find("tool_shed") assert tool_shed_el is not None @@ -1154,14 +1185,14 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): def __watch_directory( self, - directory: "StrPath", - elems, - integrated_elems, + directory: StrPath, + elems: ToolPanelElements, + integrated_elems: ToolPanelElements, load_panel_dict: bool, recursive: bool, force_watch: bool = False, ) -> None: - def quick_load(tool_file: "StrPath", async_load: bool = True) -> str | None: + def quick_load(tool_file: StrPath, async_load: bool = True) -> str | None: if not self._looks_like_a_tool(str(tool_file)): return None try: @@ -1186,30 +1217,24 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): log.exception("Failed to load potential tool %s.", tool_file) return None - tool_loaded = False if not os.path.isdir(directory): log.error("Failed to read tool directory %s.", directory) return - for name in os.listdir(directory): - if name.startswith((".", "_")): - # Very unlikely that we want to load tools from a hidden or private folder - continue - child_path = os.path.join(directory, name) - if os.path.isdir(child_path) and recursive: - self.__watch_directory(child_path, elems, integrated_elems, load_panel_dict, recursive) - elif self._looks_like_a_tool(child_path): - tool_id = quick_load(child_path, async_load=False) - tool_loaded = bool(tool_id) - if (tool_loaded or force_watch) and self._tool_watcher: - self._tool_watcher.watch_directory(directory, quick_load) + for dirpath, files in walk_tool_directories(directory, recursive): + tool_loaded = False + for child_path in files: + if self._looks_like_a_tool(child_path): + tool_loaded = bool(quick_load(child_path, async_load=False)) or tool_loaded + if (tool_loaded or (force_watch and dirpath == str(directory))) and self._tool_watcher: + self._tool_watcher.watch_directory(dirpath, quick_load) def load_tool( self, - config_file: "StrPath", - guid=None, - tool_shed_repository=None, + config_file: StrPath, + guid: str | None = None, + tool_shed_repository: "ToolConfRepository | ToolShedRepository | None" = None, use_cached: bool = False, - **kwds, + **kwds: Any, ) -> "Tool": """Load a single tool from the file named by `config_file` and return an instance of `Tool`.""" # Parse XML configuration file and get the root element @@ -1244,12 +1269,12 @@ class AbstractToolBox(ManagesIntegratedToolPanelMixin): if self._tool_config_watcher: [self._tool_config_watcher.watch_file(macro_path) for macro_path in tool._macro_paths] - def add_tool_to_cache(self, tool: "Tool", config_file: "StrPath") -> None: + def add_tool_to_cache(self, tool: "Tool", config_file: StrPath) -> None: tool_cache: ToolCache | None = getattr(self.app, "tool_cache", None) if tool_cache: tool_cache.cache_tool(config_file, tool) - def load_tool_from_cache(self, config_file: "StrPath", recover_tool: bool = False) -> Union["Tool", None]: + def load_tool_from_cache(self, config_file: StrPath, recover_tool: bool = False) -> Union["Tool", None]: tool_cache: ToolCache | None = getattr(self.app, "tool_cache", None) tool = None if tool_cache: diff --git a/lib/galaxy/tool_util/toolbox/parser.py b/lib/galaxy/tool_util/toolbox/parser.py index a87b5f82613..0d1c284daf5 100644 --- a/lib/galaxy/tool_util/toolbox/parser.py +++ b/lib/galaxy/tool_util/toolbox/parser.py @@ -39,6 +39,15 @@ class ToolConfSource(metaclass=ABCMeta): """Monitor the toolbox configuration source for changes and reload.""" return DEFAULT_MONITOR + def parse_store_name(self) -> str | None: + """Return the named tool source store this conf routes to, or None. + + Lets a single tool_conf opt into a non-default store (e.g. a + CVMFS-resident sqlite bundle) by setting ``store="..."`` on the + XML root or ``store: ...`` in YAML. + """ + return None + class XmlToolConfSource(ToolConfSource): def __init__(self, config_filename: StrPath): @@ -59,6 +68,9 @@ class XmlToolConfSource(ToolConfSource): def parse_monitor(self): return string_as_bool(self.root.get("monitor", DEFAULT_MONITOR)) + def parse_store_name(self) -> str | None: + return self.root.get("store") or None + class YamlToolConfSource(ToolConfSource): def __init__(self, config_filename: StrPath): @@ -78,6 +90,10 @@ class YamlToolConfSource(ToolConfSource): def is_shed_tool_conf(self): return False + def parse_store_name(self) -> str | None: + store = self.as_dict.get("store") + return store or None + class ToolConfItem: """Abstract description of a tool conf item. diff --git a/lib/galaxy/tools/source_store/__init__.py b/lib/galaxy/tools/source_store/__init__.py new file mode 100644 index 00000000000..a83de8905a0 --- /dev/null +++ b/lib/galaxy/tools/source_store/__init__.py @@ -0,0 +1,37 @@ +""" +Tool Source Store - standalone storage for Galaxy tool sources. + +Tool sources and their derived ``ToolIndex`` live in a standalone SQLAlchemy +database chosen by connection URL (``tool_source_database_connection``; +defaults to a ``sqlite:///`` file, but any SQLAlchemy URL such as +``postgresql://`` works just as well). There is a single store +implementation, ``SqlAlchemyToolSourceStore``; a tool_conf may point at a +named store declared in ``tool_source_stores``, and those are layered over +the default in a ``CompositeToolSourceStore``. +""" + +from .factory import ( + build_named_store, + build_tool_source_store, +) +from .index import ( + ToolIndex, + ToolIndexEntry, +) +from .interface import ( + ConfigurationError, + ReadOnlyStoreError, + StoredToolSource, + ToolSourceStore, +) + +__all__ = [ + "StoredToolSource", + "ToolSourceStore", + "ToolIndex", + "ToolIndexEntry", + "build_tool_source_store", + "build_named_store", + "ConfigurationError", + "ReadOnlyStoreError", +] diff --git a/lib/galaxy/tools/source_store/composite.py b/lib/galaxy/tools/source_store/composite.py new file mode 100644 index 00000000000..cb35a766505 --- /dev/null +++ b/lib/galaxy/tools/source_store/composite.py @@ -0,0 +1,239 @@ +""" +Composite tool source store. + +Lets a single Galaxy process serve tools from multiple per-tool-conf +stores. Reads are tried in declared order (first hit wins), writes go to +a designated *default* store. Used to layer e.g. a CVMFS-resident +read-only sqlite bundle on top of the local writable store. + +The composite is invisible to the rest of Galaxy: it implements the same +:class:`ToolSourceStore` interface, and consumers / the populator +keep working unchanged. +""" + +import logging +from collections.abc import Iterator +from typing import ( + Any, +) + +from .index import ( + ToolIndex, + ToolIndexEntry, +) +from .interface import ( + StoredToolSource, + ToolSourceStore, +) + +log = logging.getLogger(__name__) + + +class CompositeToolSourceStore(ToolSourceStore): + """A read-priority store that fans out across several backends. + + Args: + members: Ordered list of ``(name, store)`` pairs consulted for + reads in order. Earlier entries shadow later ones on id/hash + collisions. + default: The store that receives all writes. Must be present in + ``members`` (its name is the value used for write routing). + Must not be ``read_only``. + """ + + def __init__( + self, + members: list[tuple[str, ToolSourceStore]], + default: str, + ) -> None: + if not members: + raise ValueError("CompositeToolSourceStore requires at least one member") + names = [n for n, _ in members] + if default not in names: + raise ValueError(f"default store {default!r} not in members {names!r}") + self._members: list[tuple[str, ToolSourceStore]] = list(members) + self._default_name = default + self._default_store = dict(members)[default] + if self._default_store.read_only: + raise ValueError(f"default store {default!r} is read-only") + # Composite as a whole is writable iff its default store is writable. + self.read_only = False + + # --- write ops: always default ------------------------------------ + + def store(self, tool_source: StoredToolSource) -> str: + return self._default_store.store(tool_source) + + def delete(self, hash: str) -> bool: + return self._default_store.delete(hash) + + def store_index(self, index: ToolIndex) -> None: + self._default_store.store_index(index) + + def update_index_entry(self, entry: ToolIndexEntry) -> None: + self._default_store.update_index_entry(entry) + + def remove_index_entry(self, tool_id: str) -> None: + self._default_store.remove_index_entry(tool_id) + + # --- read ops: priority order -------------------------------------- + + def get(self, hash: str) -> StoredToolSource | None: + for _name, member in self._members: + found = member.get(hash) + if found is not None: + return found + return None + + def exists(self, hash: str) -> bool: + return any(m.exists(hash) for _, m in self._members) + + def get_by_tool_id(self, tool_id: str, version: str | None = None) -> list[StoredToolSource]: + # Union across members, deduped by hash, preserving member order. + seen: set[str] = set() + out: list[StoredToolSource] = [] + for _name, member in self._members: + for src in member.get_by_tool_id(tool_id, version): + if src.hash in seen: + continue + seen.add(src.hash) + out.append(src) + return out + + def get_by_source_path(self, source_path: str) -> StoredToolSource | None: + for _name, member in self._members: + found = member.get_by_source_path(source_path) + if found is not None: + return found + return None + + def list_all(self) -> Iterator[str]: + seen: set[str] = set() + for _name, member in self._members: + for h in member.list_all(): + if h in seen: + continue + seen.add(h) + yield h + + def list_source_paths(self) -> set[str]: + paths: set[str] = set() + for _name, member in self._members: + paths |= member.list_source_paths() + return paths + + @property + def members(self) -> list[tuple[str, ToolSourceStore]]: + """The ``(name, store)`` pairs, in read-priority order.""" + return list(self._members) + + @property + def read_only_member_names(self) -> set[str]: + """Names of member stores no populator can write to. + + The boot coverage check treats paths routed to these differently: + a miss there can never be healed by running the populator, so it + must not trigger one. + """ + return {name for name, member in self._members if member.read_only} + + def count(self) -> int: + # Distinct hashes across the composite. + return sum(1 for _ in self.list_all()) + + def get_stats(self) -> dict[str, Any]: + return { + "backend": "composite", + "count": self.count(), + "default": self._default_name, + "members": [{"name": name, **member.get_stats()} for name, member in self._members], + } + + # --- index --------------------------------------------------------- + + def load_index(self) -> ToolIndex | None: + merged = ToolIndex() + any_loaded = False + for name, member in self._members: + try: + idx = member.load_index() + except Exception as e: + log.error(f"Failed to load index from store {name!r}: {e}") + continue + if idx is None: + continue + any_loaded = True + for tool_id, entry in idx.entries.items(): + # Earlier members win on collision. + if tool_id in merged.entries: + continue + merged.entries[tool_id] = entry + # Merge the per-version map too — ``ToolIndex.get(tool_id, + # tool_version)`` resolves exact versions through it, so skipping + # it would make every non-newest version of a member-store tool + # unreachable. Same collision rule, applied per (id, version). + for tool_id, versions in idx.entries_by_version.items(): + version_bucket = merged.entries_by_version.setdefault(tool_id, {}) + for version, entry in versions.items(): + version_bucket.setdefault(version, entry) + for section_id, ids in idx.by_section.items(): + bucket = merged.by_section.setdefault(section_id, []) + for tid in ids: + if tid not in bucket: + bucket.append(tid) + # Same collision rule per (tool id, section): earlier members' + # placements win, later members append theirs after. + seen_placements = {(item.tool_id, item.section_id) for item in merged.panel_items} + for item in idx.panel_items: + placement_key = (item.tool_id, item.section_id) + if placement_key in seen_placements: + continue + seen_placements.add(placement_key) + merged.panel_items.append(item) + for view_name, view in idx.panel_views.items(): + merged.panel_views.setdefault(view_name, view) + if idx.built_at and (merged.built_at is None or idx.built_at > merged.built_at): + merged.built_at = idx.built_at + if not any_loaded: + return None + return merged + + def invalidate_index_cache(self) -> None: + for _name, member in self._members: + member.invalidate_index_cache() + + def index_is_fresh(self) -> bool | None: + """Aggregate the member verdicts. + + A stale *writable* member makes the composite stale — the populator + can heal it, so report ``False`` and let boot run it. A read-only + member is trusted whenever its index loads under the current schema + (see the SQLAlchemy backend): ``False`` there means no loadable + index at all, which can't be healed locally — warn and continue, + the publisher owns repopulation. A member without a probe + downgrades an otherwise-fresh verdict to ``None`` so the caller + still runs its coverage scan. + """ + verdict: bool | None = True + for name, member in self._members: + fresh = member.index_is_fresh() + if fresh is False: + if member.read_only: + log.warning( + "Read-only tool source store %r has no loadable index; " + "its tools will parse eagerly until it is repopulated upstream", + name, + ) + else: + return False + elif fresh is None: + verdict = None + return verdict + + def close(self) -> None: + """Propagate close() to every member store.""" + for _name, member in self._members: + try: + member.close() + except Exception as e: + log.error(f"Composite store close failed for member '{_name}': {e}") diff --git a/lib/galaxy/tools/source_store/discover.py b/lib/galaxy/tools/source_store/discover.py new file mode 100644 index 00000000000..8820f794c94 --- /dev/null +++ b/lib/galaxy/tools/source_store/discover.py @@ -0,0 +1,487 @@ +""" +Tool discovery utilities. + +Walks Galaxy's tool configuration files to enumerate every ```` referenced +from any tool_conf without booting a full ``ToolBox``. Used by the populator +(``galaxy.tools.source_store.populator``) and by callers that need to compare +on-disk confs against the indexed tool set (cold-start auto-populate, +``reset_shed_tools``). +""" + +import logging +import os +import string +from collections.abc import ( + Collection, + Iterable, + Iterator, +) +from concurrent.futures import ThreadPoolExecutor +from dataclasses import ( + dataclass, + field, +) +from pathlib import Path +from typing import ( + TYPE_CHECKING, +) + +from galaxy.model import _get_datatypes_registry +from galaxy.tool_util.loader_directory import ( + looks_like_a_tool, + looks_like_a_tool_xml, +) +from galaxy.tool_util.toolbox.base import ( + resolve_tool_path, + walk_tool_directories, +) +from galaxy.tool_util.toolbox.parser import ( + get_toolbox_parser, + ToolConfItem, + ToolConfSection, +) +from galaxy.tools import MODEL_TOOLS_PATH +from galaxy.tools.special_tools import hidden_lib_tool_paths +from galaxy.util import ( + listify, + parse_xml, +) + +if TYPE_CHECKING: + from galaxy.config import GalaxyAppConfiguration + +log = logging.getLogger(__name__) + +# Sentinel ``tool_conf`` for datatype-converter discoveries — they have no +# panel conf; the registry loads them after boot via ``load_tool``. +CONVERTER_TOOL_CONF = "" + + +@dataclass +class DiscoveredTool: + """Information about a discovered tool file.""" + + path: str # Absolute path to tool file + tool_conf: str # Path to the tool_conf file that referenced this tool + tool_path: str | None # The tool_path from the tool_conf + guid: str | None = None # GUID for shed tools + is_shed_tool: bool = False + # Shed conf ```` child elements. The populator keys shed entries + # by guid and stamps these on the index entry, so shed stubs answer + # repository metadata without materialising — mirroring what the eager + # walk reads via ``get_tool_repository_from_xml_item``. + tool_shed: str | None = None + repository_name: str | None = None + repository_owner: str | None = None + installed_changeset_revision: str | None = None + # ```` conf id for tools referenced from data + # manager configs — may differ from the tool XML id, and the registry + # is keyed by it. + data_manager_id: str | None = None + # Conf-level ``hidden="true"`` on the ```` element (NOT the XML + # body's ``