Commit Graph
2532 Commits
Author SHA1 Message Date
mvdbeek fc7aa2cd41 source store: correct storage documentation 2026-07-13 11:22:58 +02:00
mvdbeek 31ee132bfb Clarify tool source storage activation docs 2026-07-13 10:27:00 +02:00
mvdbeek 4151167386 Use SQLAlchemy URI for tool source store config 2026-07-13 10:26:59 +02:00
mvdbeek 22c1f9ec5b Drop the main-DB tool source store backend
The store is a rebuildable, content-addressed cache — it does not need
Galaxy's database, and living there was the source of every subtle
lifecycle mechanism in the subsystem: store.commit() committed whatever
request or job transaction was in flight, reads needed a private-Session
isolation hack, and a sa_session parameter threaded through the factory,
populator, watcher and CLI.

Standalone stores are now the only kind: the default is a SQLite file
under tool_source_disk_path (any SQLAlchemy URL via tool_source_stores
for shared multi-host deployments), committing per write on its own
engine. The tool_source_record/tool_index tables, their migration, the
model classes and the whole session plumbing are gone. The ToolSourceStore
ABC loses commit() (every backend persists per write). The dev/watch-mode
watcher now populates per file without broadcasting and keeps its single
debounced reload notification for the batch.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:26:59 +02:00
mvdbeek 761e43d544 Relocate tool_source_store to galaxy.tools.source_store
The package is app-coupled (galaxy.model, galaxy.config, galaxy.queues,
galaxy.datatypes) and only consumed from app-side code, so it belongs
under galaxy.tools rather than a new top-level package. Riding the
existing packages/app tools symlink also removes the need for a
dedicated package symlink. Also drop an unused back-compat alias and
fix stale doc references.

Claude-Session: https://claude.ai/code/session_018L7ZmCv2ubKA3JNeSL8Pkr
2026-07-13 10:26:59 +02:00
mvdbeek 8162853207 Add pluggable tool source store infrastructure
Introduces galaxy.tool_source_store: a queryable store of parsed,
macro-expanded tool sources plus a pre-computed ToolIndex of the metadata
batch consumers need (panel section, labels, EDAM, requirements, test
counts, shed provenance).

- Backends: database (store-owned tool_source_record + tool_index tables,
  migration f5a73c8b9d12), sqlalchemy (read-only SQLite bundles, e.g.
  CVMFS-shipped), and a composite that layers named stores over the
  default with read fall-through.
- scripts/tool_source/populate_store.py walks tool_conf files and writes
  sources + index + whoosh search index; --watch updates the store on
  file changes and broadcasts reload_tool_source_cache. The populator is
  the single writer; consumers are read-only.
- Per-conf opt-in via store="name" on a toolbox conf root, resolved
  against the tool_source_stores catalog in galaxy.yml.
- The tool_source_record table is deliberately separate from tool_source,
  whose rows belong to the job-request path and carry a raw-string
  payload contract.

Nothing reads from the store yet — an on-demand-loading toolbox consuming
it is follow-up work (#22633). Extracted from that PR.
2026-07-13 10:26:57 +02:00
Nicola Soranzo 3c919caaf3 Merge pull request #23049 from nsoranzo/update_committers
[PROCEDURES] Remove Laila from committers group
2026-07-10 17:45:46 +01:00
Marius van den Beek d37d496f59 Merge pull request #23030 from SergeyYakubov/bump-rucio-40.2.0
Bump Rucio clients to 40.2.0
2026-07-07 17:02:14 +02:00
mvdbeek 6d4eb20906 Merge branch 'release_26.1' into dev 2026-07-06 16:12:10 +02:00
mvdbeek a3750b0326 Merge branch 'release_26.0' into release_26.1 2026-07-06 15:49:42 +02:00
benjamin 1f52730c0a feat: add RISC V support for mulled 2026-07-04 11:45:55 +00:00
paperbenniandBjörn Grüning d7feedfe91 Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-07-04 11:45:55 +00:00
paperbenniandBjörn Grüning 90d48a293b Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-07-04 11:45:55 +00:00
paperbenniandBjörn Grüning 3ce2015029 Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-07-04 11:45:55 +00:00
paperbenniandBjörn Grüning ef9e32d3e1 Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-07-04 11:45:55 +00:00
benjamin abed754256 docs: document previously undocumented mulled features 2026-07-04 11:45:55 +00:00
B0r1sD 0977c04216 Update citation to 2026 paper in `config_schema.yml`
Also:
- Apply config schema changes from
  https://github.com/galaxyproject/galaxy/pull/22910
- Update config and docs with `make config-rebuild` .
2026-07-03 12:30:14 +01:00
Nicola Soranzo cabb9e5e46 Remove Laila from committers group
@ElectronicBlueberry left the Freiburg Galaxy Team for greener pastures
a few months ago, thanks for all your contributions to Galaxy and the
community!
2026-07-01 19:18:07 +01:00
Yakubov, Sergey 6241893238 Bump Rucio clients to 40.2.0
Update Galaxy's Rucio test and optional dependency declarations to require rucio-clients 40.2.0.

Adjust the Rucio object store for client 40 behavior, including checksum import changes, ingest checksum handling, Rucio config cache invalidation, and isolated download directories.

Run Rucio integration tests against the Savannah Rucio 40.2.0 container by default while keeping an environment override for the image.
2026-06-26 12:48:43 -04:00
mvdbeek 41272995ea Merge branch 'release_26.1' into dev 2026-06-20 22:49:25 +02:00
mvdbeek a8468a3745 Emit SSE/queue metrics as gauges, sample SSE counts in web workers
Point-in-time counts (SSE connections, control-queue depth, worker
processes) were emitted via statsd timing(), so telegraf/influxdb stored
them as timing aggregations (mean/median/percentiles) instead of a single
value. Add a gauge() method to the statsd client and emit these as gauges.

The SSEConnectionManager is per-process state registered only on the
webapp, so sampling it from the Celery beat task saw an empty manager and
always reported ~0. Move SSE connection sampling into each web worker via
a per-worker daemon thread (SSEConnectionGaugeEmitter), tagging samples
with server_name so per-worker series don't collide. It is opt-in via the
new enable_sse_connection_metrics config (default off) and only runs when
statsd is actually configured.
2026-06-18 09:37:49 +02:00
paperbenni 5cb3ac4146 Merge branch 'dev' of https://github.com/galaxyproject/galaxy into dev 2026-06-15 17:20:14 +02:00
benjamin d36470c09a docs: fix remaining typos 2026-06-14 13:18:29 +00:00
benjamin a9861d7ddc docs: fix more typos 2026-06-14 13:18:21 +00:00
benjamin 01afe742e3 docs: fix typos 2026-06-14 13:18:21 +00:00
benjamin b7896c13a6 docs: fix typos 2026-06-14 13:18:21 +00:00
paperbenni 9a9f7603f0 Merge branch 'dev' of github.com:paperbenni/galaxy into dev 2026-06-14 11:31:30 +02:00
benjamin ba910e321a feat: add RISC V support for mulled 2026-06-14 09:30:57 +00:00
paperbenniandBjörn Grüning 4655ae7274 Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-06-13 11:29:26 +02:00
paperbenniandBjörn Grüning e752aa292e Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-06-13 11:29:12 +02:00
paperbenniandBjörn Grüning 99b4ee0dfb Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-06-13 11:28:47 +02:00
paperbenniandBjörn Grüning c2ed84e409 Update doc/source/admin/special_topics/mulled_containers.rst
Co-authored-by: Björn Grüning <bjoern@gruenings.eu>
2026-06-13 11:28:29 +02:00
Dannon Baker efa5e9b233 Harden agent retries config resolution
Two follow-ups on the configurable-retries change. A shared `default` block
`retries` was silently overriding custom_tool's producer-0, which would
re-enable pydantic-ai's generic retry loop inside the producer's own reflection
loop. The producer's 0 is a correctness requirement, not a tunable, so it now
honors only an explicit `custom_tool.retries` and ignores the default block.
Separately, a non-numeric or blank `retries` value used to blow up with a bare
TypeError/ValueError at agent construction -- which the manager mistook for an
unknown-agent fallback and double-faulted on -- so it now raises a clear
ConfigurationError.
2026-06-12 12:59:27 -04:00
Dannon Baker deb7ecd706 Make agent retry budget configurable via inference_services
ChatGXY's router was hard-failing with 'Exceeded maximum output retries (1)':
its pydantic-ai Agent used the default retry budget of 1, and the router's
process() only caught (OSError, ValueError), so an UnexpectedModelBehavior sailed
past the graceful _handle_fallback and surfaced to the user as a generic 500.

This adds a per-agent 'retries' knob to inference_services, resolved through the
same _get_agent_config chain as temperature/max_tokens and wired into every
agent's Agent(retries=...). The default goes from pydantic-ai's 1 to 3 so a
transient bout of non-conforming model output recovers instead of failing the
run. custom_tool's producer keeps its retries=0 reflection-loop default but can
now be overridden. The router also catches UnexpectedModelBehavior so a genuinely
exhausted budget degrades to the existing fallback message instead of a 500.

Adds unit tests for the resolution chain, the config->Agent wiring, and the
router's graceful fallback (forced via an empty ModelResponse).
2026-06-12 12:59:27 -04:00
John Davis e869b672df Use default_panel_view in docs 2026-06-11 16:54:27 -04:00
mvdbeek 329e36bf33 Add namespace stripping to CachedExplicitSingularityContainerResolver
Add an optional `namespace` config parameter that strips
`docker://quay.io/{namespace}/` from image identifiers when constructing
the local cache path, producing flat filenames compatible with
CVMFS-hosted biocontainer caches (e.g. /cvmfs/singularity.galaxyproject.org/all/).

Mirrors the same stripping already done in pull_mulled_singularity_command.
2026-06-11 15:57:08 +02:00
benjamin 1783bb1540 docs: document previously undocumented mulled features 2026-06-09 23:59:50 +00:00
Dannon Baker d053bfae0b Use the GalaxyAI name in the GTN/IWC refresh config and comments
The agent assistant was rebranded from ChatGXY to GalaxyAI, but this
branch's new config descriptions and code comments still said ChatGXY.
Swap them over and regenerate the sample yml + options rst from the
schema. The lowercase depot bucket path in gtn_database_url stays as-is
-- that's a real URL, not a brand reference.
2026-06-03 10:26:52 -04:00
Dannon Baker fdfa892459 Add iwc_manifest_refresh_interval so admins can tune (or disable, with 0) how
often Galaxy pre-warms the IWC workflow manifest cache. Default of 3600 matches
the in-process TTL so the cache stays continuously warm rather than expiring
between user-driven hits. The periodic celery task follows in a separate
commit.
2026-06-03 10:26:52 -04:00
Dannon Baker 464a9a0b83 HEAD depot first and only re-download when its Last-Modified is newer than the
local file. After each successful download the local mtime is stamped to
depot's Last-Modified so future comparisons are exact and survive local clock
skew. Raises the default refresh interval to once a day -- the dataset updates
weekly or so, and the HEAD check keeps steady-state cost to a few hundred
bytes per tick regardless.
2026-06-03 10:26:52 -04:00
Dannon Baker 1c0a47a9d2 Add gtn_database_refresh_interval so admins can tune (or disable, with 0) how
often Galaxy re-pulls the GTN search database from depot. The periodic celery
task that actually does the refresh follows in a separate commit.
2026-06-03 10:26:52 -04:00
Marius van den Beek 4feff0458d Merge pull request #22707 from dannon/rebrand-chatgxy
Rebrand ChatGXY to GalaxyAI
2026-05-21 08:27:52 +02:00
davelopez 6c1ba633ca Add config options for bulk storage operation retention 2026-05-20 18:52:36 +02:00
davelopez fe3de345a4 Modularizes storage operation logic and refactors related code
Splits storage operation schema, logic, and execution into a dedicated module for improved maintainability and clarity.
2026-05-20 18:37:23 +02:00
Dannon Baker 17969e2fb5 Update admin docs and 26.0 release notes for the GalaxyAI rebrand
Updates user-facing prose in the AI agents admin guide and the upcoming
26.0 release announcement to refer to GalaxyAI. Historical release
notes for 24.2 and 25.0 are intentionally left as ChatGXY since that's
the name the feature shipped under at the time.
2026-05-20 11:35:01 -04:00
John Chilton 4afd63a9a9 History notebooks.
1. What Are History Pages?

History Pages are markdown documents tied to Galaxy histories. They let users (and AI agents) document, annotate, and share analysis narratives alongside the data that produced them. A history can have multiple pages, and each page supports an AI assistant that can read history contents and propose edits.

History Pages are built on the existing Galaxy **Page** model — they are regular Pages with an optional `history_id` foreign key. This unified model means history-attached and standalone pages share the same editor, revision system, AI chat, and API surface. The difference is contextual: history pages gain access to history-aware AI tools and are accessed through the history panel rather than the pages grid.

Every save creates an immutable revision, edits from humans and agents are tracked separately via `edit_source`, and the revision system supports preview and one-click rollback.

---

2. Standalone Pages vs History Pages

The system supports two page contexts through a single unified editor (`PageEditorView`):

| Aspect | Standalone Pages | History Pages |
|--------|-----------------|---------------|
| **Entry point** | Grid list (`/pages/list`) or direct URL | History panel "Pages" button |
| **Route** | `/pages/editor?id=X` | `/histories/:historyId/pages/:pageId` |
| **`history_id`** | null | Set — scopes page to a history |
| **AI chat tools** | Text editing only (no history tools) | Full history tools: `list_history_datasets`, `get_dataset_info`, `get_dataset_peek`, `get_collection_structure`, `resolve_hid` |
| **Drag-and-drop** | From toolbox directives | Also from history panel (datasets/collections) |
| **Permissions modal** | Yes (`ObjectPermissionsModal`) | No — inherits from history sharing |
| **Save & View** | Yes (slug-based published URL) | No (history context, no slug) |
| **Page list** | Grid (`/pages/list`) | Inline `HistoryPageList` within history panel |
| **Window Manager** | "View in Window" grid action | Click from list opens in WinBox |
| **Auto-create** | No | `resolveCurrentPage()` creates on first visit |

Both modes share: editor UI, revision system, AI chat, dirty tracking, diff views, and the same API endpoints.

---

3. User Stories

Researcher documenting an analysis

> *"I ran a ChIP-seq pipeline and want to write up what I did and what the results mean, with embedded dataset previews and plots, right next to the history that contains the data."*

- Opens history panel -> clicks "Pages" button
- System auto-creates a page titled after the history
- Types markdown prose; uses the toolbox to insert dataset references (`history_dataset_display(...)`)
- Drags a dataset from the history panel into the editor -- directive auto-inserted
- Clicks Preview to see rendered markdown with live dataset embeds
- Saves; revision #1 recorded with `edit_source="user"`

AI-assisted page editing

> *"I have 50 datasets from a variant-calling run. I want the AI to summarize what's in my history and draft a methods section."*

- Opens page -> toggles chat panel (split view: 60% editor / 40% chat)
- Types: "Summarize the datasets in this history and draft a Methods section"
- Agent calls `list_history_datasets` -> `get_dataset_info` -> `resolve_hid` tools
- Agent returns a `section_patch` proposal targeting `## Methods`
- User sees a per-section diff with checkboxes; accepts the Methods section, rejects the Introduction rewrite
- Applied content creates revision with `edit_source="agent"`
- Conversation persists across panel close/reopen and page refresh

Sharing and publishing

> *"My analysis is complete. I want to share the page read-only."*

- Shared histories expose their pages in read-only display mode to other users
- Standalone pages use the Permissions modal to manage sharing, publishing, and slug assignment

Revision history and rollback

> *"The agent's last edit broke my formatting. I want to go back."*

- Opens revision panel (right sidebar, 300px)
- Sees revision list with timestamps and source badges: "Manual", "AI", "Restored"
- Clicks an old revision to preview it read-only
- Clicks "Restore" -> creates a new revision from old content (`edit_source="restore"`)

---

4. Architecture Overview

```
+---------------------------------------------------------------------------+
|                          Frontend (Vue 3)                                  |
|                                                                           |
|  HistoryCounter --> HistoryPageView --> PageEditorView <-- PageEditor      |
|       |                 |    |              |                  |           |
|       |           +-----+    +-----+   MarkdownEditor     (standalone     |
|       |           |                |    TextEditor          entry)         |
|       |     HistoryPageList        |    (drag-drop)                       |
|       |                      EditorSplitView                              |
|       |                            |                                      |
|       |   PageRevisionList    PageChatPanel                               |
|       |   PageRevisionView     |         |                                |
|       |                  ChatMessageCell  ProposalDiffView                 |
|       |                  ChatInput       SectionPatchView                  |
|       |                                                                   |
|  pageEditorStore (Pinia) <---> API Client (api/pages.ts)                  |
+---------------------------------+-----------------------------------------+
                                  | REST
+---------------------------------v-----------------------------------------+
|                       Backend (FastAPI)                                    |
|                                                                           |
|  /api/pages (history_id filter) --> PageManager                           |
|  /api/pages/{id}/revisions      --> PageManager (revisions)               |
|  /api/chat (page_id)            --> ChatManager + AgentService            |
|                                          |                                |
|                                  PageAssistantAgent                       |
|                                    +- list_history_datasets               |
|                                    +- get_dataset_info                    |
|                                    +- get_dataset_peek                    |
|                                    +- get_collection_structure            |
|                                    +- resolve_hid                         |
|                                                                           |
|  Models: Page (+ history_id), PageRevision (+ edit_source), ChatExchange  |
|  markdown_util.py: ready_galaxy_markdown_for_export()                     |
+---------------------------------------------------------------------------+
```

---

5. Data Model

Page (extended)

| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `user_id` | int FK -> galaxy_user | Indexed |
| `history_id` | int FK -> history | **Nullable**, indexed. When set, page is history-attached |
| `title` | text | Not versioned |
| `slug` | text | Indexed. Standalone pages only |
| `latest_revision_id` | int FK -> page_revision | Eager-loaded; circular FK with `use_alter` |
| `source_invocation_id` | int FK -> workflow_invocation | Nullable. Tracks "generated from invocation" |
| `published` / `importable` | bool | Standalone sharing features |
| `deleted` | bool | Soft-delete pattern |
| `create_time` / `update_time` | datetime | |

Relationships: `user`, `history` (optional), `revisions` (cascade delete), `latest_revision` (eager), `source_invocation`, `tags`, `annotations`, `ratings`, `users_shared_with`

PageRevision (extended)

| Column | Type | Notes |
|--------|------|-------|
| `id` | int PK | |
| `page_id` | int FK -> page | Indexed |
| `title` | text | Snapshot of title at revision time |
| `content` | text | Raw markdown with internal IDs |
| `content_format` | varchar(32) | `"markdown"` or `"html"` |
| `edit_source` | varchar(16) | **New.** `"user"`, `"agent"`, or `"restore"` |
| `create_time` / `update_time` | datetime | |

ChatExchange (extended)

| Column | Type | Notes |
|--------|------|-------|
| `page_id` | int FK -> page | Nullable, indexed. Scopes chat to a page |

The original `notebook_id` FK was replaced with `page_id` when the HistoryNotebook model was merged into Page.

6. Content Pipeline

Page content flows through two representations:

```
User edits markdown in MarkdownEditor / TextEditor
        |
        v
  +-------------------+
  |  Raw content       |  Stored in DB as-is
  |  (internal IDs)    |  history_dataset_id=42
  +--------+----------+
           |  rewrite_content_for_export()
           v
  +-----------------------+     +----------------------------+
  |  content_editor        |     |  content                    |
  |  (raw, for editor)     |     |  (encoded IDs + expanded    |
  |  Same as DB content    |     |   directives, for render)   |
  +-----------------------+     +----------------------------+
```

The API returns **both** fields in `PageDetails`:
- `content_editor`: What the text editor displays and saves back
- `content`: What the Markdown renderer uses (with encoded IDs the existing Galaxy markdown components expect)

This dual-field pattern avoids the round-trip problems that would arise from encoding/decoding IDs on every save cycle.

---

7. Agent Architecture

PageAssistantAgent

Registered as `AgentType.PAGE_ASSISTANT` in the Galaxy agent framework. Uses pydantic-ai with structured output.

**Tools (5):**

| Tool | Purpose | Returns |
|------|---------|---------|
| `list_history_datasets` | Paginated history item listing | HID, name, type, state, size, internal ID |
| `get_dataset_info` | Detailed metadata for one HID | Name, format, state, size, tool info, metadata |
| `get_dataset_peek` | Pre-computed content preview | First lines of dataset content |
| `get_collection_structure` | Collection element listing | Element names, types, states |
| `resolve_hid` | HID -> directive argument conversion | `history_dataset_id=N` or `history_dataset_collection_id=N` + `job_id` |

When editing a standalone page (no `history_id`), history tools are unavailable -- the agent can still do full-replacement and section-patch edits on the page content.

**Output types (3, discriminated by `mode` literal):**

| Type | When Used | Content |
|------|-----------|---------|
| `FullReplacementEdit` | Complete document rewrite | Full new markdown document |
| `SectionPatchEdit` | Targeted heading-level edit | Target heading + new section content |
| `str` (plain text) | Conversational response | No edit proposal |

**System prompt** is dynamically assembled:
1. Static instructions from `prompts/page_assistant.md`
2. Auto-generated directive reference table (reads `markdown_parse.VALID_ARGUMENTS` at runtime)
3. Current page content injected as context
4. History name and item count summary (when `history_id` is set)

The agent works in HID-space (matching what users see in the history panel) and uses `resolve_hid` to translate to the `history_dataset_id=N` directive arguments that Galaxy's markdown renderer expects.

Chat Persistence

Conversations are scoped per-page via `ChatExchange.page_id`. The flow:

1. User sends message -> `POST /api/chat` with `page_id` and `agent_type="page_assistant"`
2. API looks up page, extracts `history_id` and current content from the page record
3. Agent processes with history tools (if history-attached) and current document context
4. Response stored as `ChatExchange` + `ChatExchangeMessage` with full `agent_response` JSON
5. Frontend persists `exchange_id` in `userLocalStorage` per-page for session continuity

---

8. Frontend Components

Component Tree

```
History-attached entry:
  HistoryCounter (button in history panel)
    +- HistoryPageView (list + display routing -- 176 lines)
         +- HistoryPageList (page picker -- 89 lines)
         +- Markdown (display-only render)
         +- PageEditorView (edit mode delegation)

Standalone entry:
  PageEditor (thin wrapper -- 13 lines)
    +- PageEditorView

PageEditorView (unified editor -- 364 lines)
  +- ClickToEdit (inline title editing)
  +- MarkdownEditor
  |    +- TextEditor (drag-and-drop for history items)
  +- PageRevisionList (sidebar panel -- 88 lines)
  +- PageRevisionView (read-only revision preview -- 59 lines)
  +- EditorSplitView (resizable 60/40 split -- 111 lines)
  |    +- PageChatPanel (agent chat -- 477 lines)
  |         +- ChatMessageCell (shared from ChatGXY)
  |         +- ChatInput (shared from ChatGXY)
  |         +- ProposalDiffView (full-doc diff -- 123 lines)
  |         +- SectionPatchView (per-section diff -- 207 lines)
  +- ObjectPermissionsModal (standalone only -- 16 lines)
  |    +- ObjectPermissions (344 lines)
```

PageEditorView (unified editor)

The core editor component. Adapts based on context:

| Feature | `historyId` set | standalone |
|---------|----------------|------------|
| Back button target | `/histories/:hid/pages` | `/pages/list` |
| Title display | History name (read-only header) | Inline `ClickToEdit` |
| Revisions button | Always | Always |
| Chat button | When agents configured | When agents configured |
| Permissions button | Hidden | Shown |
| Save & View | Hidden | Shown |
| Preview | Navigates with `displayOnly=true` | Opens in Window Manager or navigates |

**View states** (template branching):
1. Loading spinner (no current page yet)
2. Error alert (dismissible)
3. Display-only mode (read-only Markdown render + toolbar with Edit button)
4. Revision view (full-page revision preview with Restore button)
5. Edit mode (toolbar + editor + optional chat/revision sidepanels)

HistoryPageView (history context router)

Routes between three states for history-attached pages:
1. **List mode** (no `pageId`) -> `HistoryPageList`
2. **Display mode** (`displayOnly=true`) -> Markdown renderer with toolbar
3. **Edit mode** (`pageId` set, no `displayOnly`) -> delegates to `PageEditorView`

Handles Window Manager integration: when WM is active, clicking a page in the list opens it in a WinBox window via `displayOnly=true` with `router.push(url, { title, preventWindowManager: false })`.

Pinia Store (`pageEditorStore` -- 472 lines)

**Mode:** `mode: "history" | "standalone"` -- controls which features are available.

**State management:**
- Page list, current page, editor content (raw), title
- Dirty tracking: `isDirty = currentContent !== originalContent || currentTitle !== originalTitle`
- Revision list and selected revision
- UI toggles: `showRevisions`, `showChatPanel` (mutually exclusive)
- Loading/saving flags

**Cross-session persistence (userLocalStorage):**
- `currentPageIds` -- remembers which page was open per-history
- `currentChatExchangeIds` -- remembers chat exchange per-page
- `dismissedChatProposals` -- remembers dismissed proposals per-page

**Smart defaults:**
- `resolveCurrentPage(historyId)` returns stored page, falls back to most recent by update_time, or auto-creates a new one

**Mode differentiation is minimal** -- mostly a UI/UX signal:
- History mode: guard checks require `historyId` for load operations
- Standalone mode: `savePage()` defaults `edit_source` to `"user"` if not specified
- API calls are identical -- unified `/api/pages` endpoints handle both via optional `history_id`

Diff System (`sectionDiffUtils.ts` -- 218 lines)

Built on [jsdiff](https://github.com/kpdecker/jsdiff) (`diff@^8.0.3`).

| Function | Purpose |
|----------|---------|
| `markdownSections(content)` | Split document by `#{1,6}` headings |
| `computeLineDiff(old, new)` | Line-level unified diff |
| `sectionDiff(old, new)` | Per-section change detection |
| `applySectionPatches(old, new, accepted)` | Merge only accepted section changes |
| `applySectionEdit(content, heading, newContent)` | Replace single section |
| `diffStats(changes)` | Count additions/deletions |

**Stale proposal detection:** Uses DJB2 hash of original content. If page content changes after a proposal was generated, Accept buttons are disabled.

Routes

| Path | Component | Notes |
|------|-----------|-------|
| `/histories/:historyId/pages` | HistoryPageView | List mode |
| `/histories/:historyId/pages/:pageId` | HistoryPageView | Edit mode |
| `/histories/:historyId/pages/:pageId?displayOnly=true` | HistoryPageView | Read-only rendered (WM) |
| `/pages/editor?id=X` | PageEditor | Standalone edit |
| `/pages/editor?id=X&displayOnly=true` | PageEditor | Standalone display |
| `/pages/list` | GridPage | Standalone page grid |
| `/published/page?id=X` | PageView | Published/embed view |

Drag-and-Drop

TextEditor supports drag from the history panel when `mode="page"`:
- Uses Galaxy's `eventStore.getDragItems()` infrastructure
- Datasets -> `history_dataset_display(history_dataset_id=...)` directive
- Collections -> `history_dataset_collection_display(history_dataset_collection_id=...)` directive
- Visual feedback: green dashed border on valid dragover

Window Manager Integration

When Galaxy's Window Manager (WinBox) is active:
- **History pages:** Clicking a page in `HistoryPageList` opens it in a WinBox window via `displayOnly=true`
- **Standalone pages:** "View in Window" grid action calls `Galaxy.frame.add()` with embed URL
- **HistoryCounter:** The page button respects WM state -- opens in frame when active
- `onUnmounted` skips `store.$reset()` in display mode (iframe independence)

---

9. API Surface

All page operations use the unified `/api/pages` endpoints. History-attached pages are just pages with `history_id` set.

Page CRUD

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/pages` | List pages (supports `history_id` filter) |
| `POST` | `/api/pages` | Create page (with optional `history_id`) |
| `GET` | `/api/pages/{id}` | Get page (two content fields: `content` + `content_editor`) |
| `PUT` | `/api/pages/{id}` | Update (creates new revision with `edit_source`) |
| `DELETE` | `/api/pages/{id}` | Soft-delete |
| `PUT` | `/api/pages/{id}/undelete` | Restore |

Revisions

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/pages/{id}/revisions` | List revisions |
| `GET` | `/api/pages/{id}/revisions/{rid}` | Get revision content |
| `POST` | `/api/pages/{id}/revisions/{rid}/revert` | Restore to revision (`edit_source="restore"`) |

Sharing & Publishing (standalone pages)

| Method | Path | Purpose |
|--------|------|---------|
| `GET` | `/api/pages/{id}/sharing` | Current sharing status |
| `PUT` | `/api/pages/{id}/enable_link_access` | Enable link sharing |
| `PUT` | `/api/pages/{id}/publish` | Publish page |
| `PUT` | `/api/pages/{id}/share_with_users` | Share with specific users |
| `PUT` | `/api/pages/{id}/slug` | Set URL slug |

Chat

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/api/chat` | Send message (with `page_id` + `agent_type`) |
| `GET` | `/api/chat/page/{page_id}/history` | Retrieve page chat history |

Index Query Parameters

| Param | Default | Notes |
|-------|---------|-------|
| `history_id` | null | Filter pages by history (the key filter for history-attached pages) |
| `show_own` | true | Show user's own pages |
| `show_published` | true | Show published pages |
| `show_shared` | false | Show pages shared with user |
| `search` | null | Freetext search |
| `sort_by` | -- | `create_time`, `title`, `update_time`, `username` |
| `limit` / `offset` | 100 / 0 | Pagination |

---

10. Test Coverage

Summary

| Layer | Tests | LOC | Coverage |
|-------|-------|-----|----------|
| Selenium E2E | 24 | 489 | Navigation, editing, drag-drop, WM, revisions, rename |
| API integration | in `test_pages_history_attached.py` | 13,561 | CRUD, revisions, permissions |
| Vitest (components) | 9 test files | 2,241 | All PageEditor components, store, diff utils |
| Agent unit | 28 | 708 | Structured output, tools, prompt injection, live LLM |
| History tools | 32 | 511 | All 5 tool functions |
| Chat manager | 8 | 149 | Page-scoped persistence, filtering |

Frontend Test Files

| File | Lines | Focus |
|------|-------|-------|
| `PageEditorView.test.ts` | 645 | Unified editor: standalone + history modes, revisions, WM |
| `HistoryPageView.test.ts` | 367 | List/display/edit routing, lifecycle, WM integration |
| `PageChatPanel.test.ts` | 379 | Chat loading, proposals, feedback, staleness |
| `sectionDiffUtils.test.ts` | 265 | Section parsing, diff computation, patch application |
| `PageRevisionList.test.ts` | 207 | Revision list rendering, source labels, restore |
| `HistoryPageList.test.ts` | 185 | Page list, create/select/view events |
| `ProposalDiffView.test.ts` | 66 | Full-replacement diff rendering |
| `SectionPatchView.test.ts` | 68 | Section-level patch UI |
| `EditorSplitView.test.ts` | 59 | Resizable split layout |
| `pageEditorStore.test.ts` | 952 | Store: CRUD, revisions, persistence, standalone mode |

Test Infrastructure

- **Selenium helpers:** 11 methods on `NavigatesGalaxy` (navigate, create, edit, save, rename, revisions)
- **Navigation YAML:** 25+ selectors under `pages.history` section
- **Vitest:** Pinia testing utilities, MSW for HTTP mocking, Vue shallowMount
- **Agent tests:** Mocked pydantic-ai agent + optional live LLM tests (env-gated)

---

11. ChatGXY Extraction

The existing `ChatGXY.vue` (982 lines) was refactored into shared sub-components before building the page chat panel:

| Component | Lines | Purpose |
|-----------|-------|---------|
| `ChatMessageCell.vue` | 110 | Message rendering with role styling, feedback buttons, action suggestions |
| `ChatInput.vue` | ~40 | Textarea + send button with busy state |
| `ActionCard.vue` | 80 | Action suggestion cards with priority-based styling |
| `agentTypes.ts` | 59 | Agent type registry with icons and labels |
| `chatTypes.ts` | 15 | Shared `ChatMessage` interface |
| `chatUtils.ts` | 12 | `generateId()` and `scrollToBottom()` helpers |

---

12. Design Decisions

Model Merge: HistoryNotebook -> Page

The original implementation created separate `HistoryNotebook` and `HistoryNotebookRevision` tables. After removing HID syntax (which was the only structural difference between notebooks and pages), the models were identical. The merge:

- Added `page.history_id` (nullable FK to history) instead of a separate table
- Added `page_revision.edit_source` to track revision provenance
- Changed `chat_exchange.notebook_id` -> `chat_exchange.page_id`
- Eliminated separate API endpoints (`/api/histories/{id}/notebooks/*`), manager, and schema classes
- All page operations now go through the unified `/api/pages` endpoints with optional `history_id` filter

**Benefit:** One model, one API, one editor, one store. No duplication.

HID Syntax (Decided: Removed from storage layer)

History pages originally introduced `hid=N` syntax in stored markdown -- ~630 lines across 16 files for backend resolution, dual content fields, client-side provide/inject, and store-based HID-to-ID mapping.

**Current approach:** Pages store `history_dataset_id=X` (matching existing Page syntax). The agent uses `resolve_hid` as a tool to bridge between user-visible HIDs and directive IDs. This eliminates the resolution machinery while preserving the agent's ability to work with HIDs naturally.

Trade-off: power users hand-editing markdown see opaque IDs, but the toolbox and drag-and-drop handle insertion -- most users never read raw markdown.

UI Convergence

Two parallel editors existed: legacy `PageEditorMarkdown.vue` (Options API, local state, no revisions/chat) and `HistoryNotebookView.vue` (Composition API, Pinia store, full features). The convergence:

- Created `PageEditorView.vue` as a single editor that adapts via `mode: "history" | "standalone"`
- `HistoryPageView.vue` kept only list + display routing; edit mode delegates to `PageEditorView`
- Legacy `PageEditorMarkdown.vue` and `PageEditor/services.js` deleted
- Single `pageEditorStore` handles both modes with minimal branching

Multiple Pages Per History

No unique constraint on `page.history_id`. A history can have multiple pages for different analysis perspectives, collaborators, or document types.

Title Not Versioned

Title lives on `Page`, not on revisions. Renaming doesn't create a new revision -- it's page identity, not content. (PageRevision does have a `title` field for snapshot purposes.)

Revision = Append-Only

Every edit (user save, agent apply, restore) creates a new `PageRevision`. No in-place updates. `edit_source` tracks provenance.

Section-Level Patching

The agent can propose section-level edits (targeted by heading). The frontend shows per-section diffs with individual checkboxes. Users accept/reject sections independently. This is more practical than all-or-nothing for large documents.

Panel Mutual Exclusion

The revision panel and chat panel are mutually exclusive -- toggling one closes the other. This avoids layout complexity and keeps the editor area usable.
2026-05-20 05:36:28 -04:00
Björn Grüning 0d976f5e5b Merge branch 'dev' into my-tool-panel-step2 2026-05-19 18:29:08 +02:00
Marius van den Beek bf9dc71f9d Merge pull request #22555 from guerler/toolrequests.001
Add legacy job submission fallback
2026-05-19 16:45:13 +02:00
mvdbeek df4b4c8a12 Address gx-arch-review feedback on PR #22212
- Extract FavoritesManager (lib/galaxy/managers/favorites.py): owns
  normalize / resolve / persist / order logic; controllers in users.py
  are now 2 lines each, and trans.sa_session.commit moves out of the
  controller layer. Drops the redundant re-normalize on every write
  and replaces FavoriteObjectType.X.value == .value defensive dispatch
  with enum-identity comparison. Renames the set_favorite_order
  endpoint summary away from the ambiguous "top-level" wording.
- Move curated_tool_tags_by_id off the toolbox into ToolsService;
  the controller now routes through the service instead of poking
  trans.app.toolbox directly. Raw aggregates (curated_tool_tags,
  tool_edam_operations, tool_edam_topics) remain on the toolbox since
  they're derived from per-tool data.
- Rename /api/tools/tags -> /api/tags/tool_tags (operation_id
  tags__tool_tags) to disambiguate from /api/tools/{id}; update the
  client store, tests, and the hand-edited OpenAPI schema entry.
  (Run `make update-client-api-schema` before pushing to regenerate
  the schema in its canonical alphabetical position.)
- Promote inline imports to module-level (ontology_data.py logging,
  app/__init__.py configure_tool_tag_mapping); replace
  getattr(self.config, "tool_tag_mappings_file", None) with direct
  attribute access now that the option is registered in the schema.
- Convert ontology_data's import-time bundled-mapping reads to
  lru_cache so the YAML / TSV files load lazily; fix the
  Tool._setup_id comment to point at the actual lowercasing site
  (Tool.parse around `self_ids = [self.id.lower()]`).
- doc/source/admin/tool_panel.rst: 26.0 -> 26.1.
- scripts/extract_tool_sections_from_api.py: annotate the public
  helpers the test suite imports.
- test_to_dict_cache_drops_when_tool_removed now asserts observable
  to_dict() output before/after remove, not private cache membership.
- Split test_tool_discovery_landing into focused tests; extract a
  helper that opens the discovery view and types the standard filter.
- Move the Playwright drag helper from test_tool_panel_search.py into
  TestWithSeleniumMixin.playwright_drag_item_above so future tests
  reuse it.
- Client tests: drop the full-string Whoosh equality in
  utilities.test.ts; hard-code expected Whoosh strings in
  ToolsList.test.ts (drop createWhooshQuery as both sides of the
  assertion) and remove the redundant injectTestRouter mount option
  (ToolsList.vue only uses the useRouter composable); drop the
  FontAwesome data-icon/iconName assertions in ToolPanel.test.ts in
  favor of icon-existence checks; replace three near-identical
  add/remove favorite tests in ToolsListCard.test.ts with an it.each;
  export quoteToolTagValue from Panels/utilities so
  filterConversion.test.js imports the real helper instead of
  reconstructing it inline; delete the spy-only "loads the curated
  tag mapping" test in ToolBoxSearch.test.ts (the gate is covered by
  ToolPanel.test.ts's negative case and the favorite-tag rendering
  tests cover the positive case end-to-end).
2026-05-19 11:43:34 +02:00
Bjoern Gruening 92b3bfc1a3 cleanups, test cleanups 2026-05-08 00:00:08 +02:00