From 0b9b94ae4e95864159fc0eeb37d4a3f6bb72d209 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 30 Apr 2026 15:41:09 +0200 Subject: [PATCH] feat(site/src): add advisor admin settings UI (#24624) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Add the admin-only Advisor settings form on the Agents Behavior page, wired to the PR 3 HTTP API via TanStack Query. Admins can enable/disable the advisor, tune per-run caps and max output tokens, pick a reasoning effort, and optionally select a dedicated advisor model. ## Motivation PR 3 introduced the configuration storage and API but no surface to change it. This PR puts the knobs in the existing admin Behavior UI so operators can turn the feature on/off and tune it without touching the database. ## Changes - `site/src/pages/AgentsPage/components/AdvisorSettings.tsx`: the admin-only settings panel. - TanStack Query hooks (`chatAdvisorConfig`, `updateChatAdvisorConfig`) with optimistic cache invalidation on save. - Inputs for `Enabled`, `MaxUsesPerRun`, `MaxOutputTokens`, `ReasoningEffort`, `ModelConfigID`. - Model-override dropdown renders "Use chat model" when unset. - `ZERO_UUID` / `isUnsetModelConfigId` helpers normalize the model-override payload in both directions: the backend serializes the unset sentinel as `"00000000-0000-0000-0000-000000000000"`, and the frontend must both recognize that sentinel on read and never send `""` on write (which would trigger a backend `invalid UUID length: 0`). - Behavior page wiring so the panel appears in the existing settings flow alongside the rest of the agent behavior toggles. ## Stack context This is **PR 6 of 6** in the advisor feature stack. It is stacked directly on **PR 3** (the config API) and is a sibling of PRs 4/5, so it can be reviewed independently of the chat-rendering work. ## Scope / non-goals - Does not change advisor runtime behavior; that is PR 4's responsibility. - No new audit entries for this settings change; it rides on the same storage mechanism as the existing site-configs pattern. ## Validation - `pnpm --filter site lint:types` - Manual dogfood against `./scripts/develop.sh`: toggle Enabled on/off, save, confirm `GET /api/experimental/chats/config/advisor` reflects changes, confirm non-admin users cannot reach the form. ---
πŸ“‹ Implementation Plan (shared across the advisor stack) # Plan: Add a Mux-style advisor tool to coder agents/chatd ## Outcome Add a first-class `advisor` tool to agent chats in `coderd/x/chatd` that feels native to Coder: - it is a built-in server-side tool, not an MCP/dynamic-tool workaround; - it performs a nested **tool-less** model call for strategic advice; - it is exposed only when eligible, and the prompt mentions it only when it is actually available; - it is treated as a **planning-only** tool so it does not run alongside action tools in the same batch; - it tracks usage/cost separately enough for operators to reason about it; - it has a minimally polished UI in the Agents page; - and it ships with explicit dogfooding evidence, including screenshots and repro videos. ## Design decisions to lock before coding 1. **Primary architecture:** native built-in tool in `chattool/`, backed by a small `chatadvisor` package. 2. **Nested model execution:** reuse chatd's existing model/provider stack for a one-step, tool-less advisor call rather than inventing a new provider pathway. 3. **Execution policy:** treat `advisor` as an exclusive/planning-only tool; mixed batches must return structured policy errors and force the model to retry cleanly. 4. **Availability:** initial rollout is for root agent chats only; disable for child/sub-agent chats until recursion/cost policy is proven. 5. **Prompt sync:** use one eligibility boolean to drive both tool registration and advisor guidance injection. 6. **Persistence/cost split:** MVP should keep advisor usage visible in result metadata and server metrics; only add DB schema if product/billing explicitly needs queryable advisor-specific cost. 7. **UI scope:** generic tool rendering is an acceptable temporary milestone during backend bring-up, but the release candidate should include a dedicated lightweight advisor renderer. ## Delivery model The work should be executed as coordinated workstreams with one integration owner and parallel contributors for low-conflict areas. The integration owner should own `coderd/x/chatd/chatd.go` because prompt assembly, tool registration, and model resolution all converge there. ## Detailed workstreams ### Repo evidence used for this plan
Mux reference and current chatd seams **Mux reference implementation** - `src/node/services/tools/advisor.ts` β€” native advisor tool implementation. - `src/common/constants/advisor.ts` β€” advisor prompt/constants and truncation policy. - `src/common/utils/tools/tools.ts` β€” conditional tool registration. - `src/node/services/streamContextBuilder.ts` β€” injects advisor guidance only when the tool is available. **Current chatd seams** - `coderd/x/chatd/chatd.go` - `processChat()` β€” tool assembly, prompt assembly, and chatloop invocation. - `resolveChatModel()` β€” current model/provider/key resolution seam. - `type Config struct` β€” server-level chatd configuration surface. - `coderd/x/chatd/chatloop/chatloop.go` - `Run()` β€” main streaming/model loop. - `executeTools()` β€” built-in tool execution/batching seam. - `coderd/x/chatd/chattool/` β€” built-in tool implementations. - `site/src/pages/AgentsPage/components/ChatElements/tools/Tool.tsx` β€” tool renderer dispatch. - `site/src/pages/AgentsPage/components/ChatConversation/messageParsing.ts` and `ConversationTimeline.tsx` β€” tool/result merge and rendering flow.
### Workstream map and ownership | Workstream | Primary owner | Main files | Can run in parallel? | Done when | |---|---|---|---|---| | 0. Integration + gating | Integration lead | `coderd/x/chatd/chatd.go` | No; central merge lane | Tool registration, prompt sync, and model selection are wired together | | 1. Advisor runtime + tool | Backend agent | new `coderd/x/chatd/chatadvisor/`, new `coderd/x/chatd/chattool/advisor.go` | Yes | Tool can perform a tool-less advisor call in memory and return structured results | | 2. Planning-only execution policy | Chatloop agent | `coderd/x/chatd/chatloop/chatloop.go`, related tests | Yes | Mixed `advisor` + action-tool batches are rejected cleanly and deterministically | | 3. Metrics/usage/config | Backend/telemetry agent | `chatd.go`, `chatloop/metrics.go`, optional config plumbing | Partially; coordinate with integration lead | Advisor usage is separately visible in metadata/metrics and limits are enforced | | 4. Frontend rendering | Frontend agent | `site/.../tools/Tool.tsx`, new `AdvisorTool.tsx`, stories | Yes after result schema stabilizes | Advisor renders as a readable card and story tests pass | | 5. Dogfood + QA evidence | QA agent | dev server, Storybook, dogfood output | After backend + UI are usable | Repro videos, screenshots, and a concise QA report exist | ### Parallelization rules - **Do not split `coderd/x/chatd/chatd.go` across multiple execution agents without an integration lead.** That file owns prompt building, tool registration, model resolution, and cost persistence. - Workstreams 1 and 2 can be developed in parallel and then stacked onto the integration branch. - Workstream 4 should begin once the backend result schema is agreed on, even if the backend is still behind a feature flag. - Any agent that needs to re-check Mux behavior should clone `coder/mux` into a temporary directory (for example, `$(mktemp -d)/mux`) and inspect it read-only; do not vendor or copy code from Mux directly. ## Phase 0 β€” Preflight and guardrails ### Goals - Align the team on the smallest shippable architecture. - Prevent scope creep into MCP/dynamic-tool/sub-agent variants. - Decide upfront what is MVP vs. follow-up. ### Tasks 1. **Confirm the MVP boundary.** - Ship a built-in advisor tool first. - Do **not** make MCP, dynamic tools, or sub-agents the primary implementation. - Do **not** add transient streaming phases in the first backend PR unless they fall out almost for free. 2. **Confirm local workflow hygiene before coding.** - Ensure the repo is using the project git hooks from `scripts/githooks`. - Do not bypass hooks with `--no-verify`. - Use `./scripts/develop.sh` for the full dev server rather than manual build/run commands. 3. **Lock the model-selection policy.** - **Recommended MVP:** advisor uses the same resolved provider/model/cost config as the current chat, with advisor-specific max-output and usage caps. - **Follow-up only if required:** add a separate `AdvisorModelConfigID`-style override that resolves through the existing `configCache`/model-config path. Do not invent a new free-form `provider:model` parser if chatd already stores provider/model separately. 4. **Lock the persistence policy.** - **Recommended MVP:** no DB migration. Persist advisor-visible metadata in the tool result and record separate metrics in memory/Prometheus. - **Only if product/billing explicitly asks for queryable advisor cost:** add a later DB migration or usage table, following the normal `queries/*.sql` + `make gen` workflow. 5. **Create an execution ADR note in the work item or tracking doc.** - Capture: built-in tool, tool-less nested call, root-chat-only rollout, exclusive execution policy, MVP no-DB-migration default. ### Quality gate - Everyone on the team can state the same answers to these questions: - Is advisor a built-in tool? **Yes.** - Can advisor run with action tools in the same batch? **No.** - Does advisor get tools of its own? **No.** - Is a DB migration required for MVP? **No, unless billing insists.** ## Phase 1 β€” Build the advisor runtime and tool wrapper ### Goals Create the core advisor implementation in a way that is easy to test and keeps `chattool/` thin. ### Files to add - `coderd/x/chatd/chatadvisor/types.go` - `coderd/x/chatd/chatadvisor/guidance.go` - `coderd/x/chatd/chatadvisor/handoff.go` - `coderd/x/chatd/chatadvisor/runtime.go` - `coderd/x/chatd/chatadvisor/runner.go` - `coderd/x/chatd/chattool/advisor.go` ### Responsibilities by file 1. **`types.go`** - Define the input/result schema used by the tool and UI. - Keep the result shape close to Mux so the UI and model both have predictable cases. - Recommended result variants: - `advice` - `limit_reached` - `error` Recommended shape: ```go type AdvisorArgs struct { Question string `json:"question"` } type AdvisorResult struct { Type string `json:"type"` Advice string `json:"advice,omitempty"` Error string `json:"error,omitempty"` AdvisorModel string `json:"advisor_model,omitempty"` RemainingUses int `json:"remaining_uses,omitempty"` Usage *AdvisorUsageResult `json:"usage,omitempty"` } ``` 2. **`guidance.go`** - Hold two strings: - the nested advisor system prompt; - the parent-agent guidance block to inject into the outer system prompt. - The nested advisor prompt must say, in plain language: - you are advising the parent agent; - you do not address the end user directly; - you do not claim actions happened; - you return concise strategic guidance and tradeoffs. 3. **`runtime.go`** - Define the per-run runtime state. - Recommended fields: - resolved model + model config; - provider keys/options reused from the outer chat; - `MaxUsesPerRun`; - `MaxOutputTokens`; - atomic/current call counter; - callback(s) to obtain the current prompt snapshot and current-step snapshot; - optional metrics/usage hook. - Add fail-fast validation for impossible config: nil model, non-positive limits, empty prompt builders, etc. 4. **`handoff.go`** - Build the advisor handoff message from: - the explicit question; - the exact prompt/messages the parent model just used; - the current step's text/reasoning snapshot, if available; - the most recent relevant tool outputs, if they are already in the prompt snapshot. - **Important:** use the already-prepared outer prompt tail, not a fresh DB reload. That keeps the advisor aligned with compaction and the exact context the outer model saw. - Apply hard truncation budgets with recent-context bias. 5. **`runner.go`** - Execute the nested advisor call. - **Recommended implementation:** call `chatloop.Run()` in an in-memory, one-step mode: - `Tools: nil` - `ProviderTools: nil` - `MaxSteps: 1` - `PersistStep`: capture the assistant output in memory instead of writing DB rows - Reuse the existing provider/model/cost path instead of building a second provider runner. - Assert that no tool definitions are passed to the nested call. 6. **`chattool/advisor.go`** - Keep this file thin and consistent with other built-ins. - Responsibilities: - decode `AdvisorArgs`; - validate `Question` is non-empty and bounded; - call the `chatadvisor` runner; - return a structured tool response. ### Defensive programming requirements - Assert `Question` is non-empty after trimming. - Assert runtime limits are positive. - Assert the nested advisor call runs with zero tools/provider tools. - Assert `AdvisorResult.Type` is one of the known variants before returning. - Assert remaining uses never goes negative. ### Acceptance criteria - A unit test can call the advisor tool with a fake model and receive a stable `advice` result. - The nested advisor call is impossible to run with tools accidentally attached. - The core logic lives in `chatadvisor/`, not embedded inside `chatd.go`. ## Phase 2 β€” Wire advisor into chatd and keep prompt/tool availability in sync ### Goals Register the tool in the right place, expose it only when eligible, and inject system guidance only when the tool is present. ### Files to modify - `coderd/x/chatd/chatd.go` - optionally a small helper file if `chatd.go` becomes too crowded ### Tasks 1. **Compute one eligibility boolean in `processChat()`.** Recommended inputs: - server-level advisor enabled flag; - root chat only (`chat.ParentChatID == uuid.Nil` or equivalent existing root/child check); - a usable resolved model/provider exists; - optional experiment/workspace/org gate if product wants staged rollout. 2. **Create the runtime once per outer chat run.** - Use the model/config/keys resolved by `resolveChatModel()`. - Reuse provider options from the current chat's `ChatModelCallConfig`. - Set `MaxUsesPerRun` and `MaxOutputTokens` from advisor config defaults. 3. **Register the tool in the built-in tool block.** - Insert after the skill tools and before MCP tools in `processChat()`. - Record `builtinToolNames["advisor"] = true` so metrics stay bounded. 4. **Inject advisor guidance into the outer system prompt using the same boolean.** - Use `chatprompt.InsertSystem()` in the same prompt assembly path that already injects user/system instructions. - Place the block near the existing instruction insertion, before plan-path/skill context blocks. - Wrap the guidance in an explicit tag like `` so it is easy to spot in tests and future refactors. 5. **Keep advisor out of child chats for the first release.** - That avoids recursion/cost blowups with `spawn_agent` / `wait_agent` flows. - Document this explicitly in the rollout notes and tests. ### Acceptance criteria - If advisor is disabled, neither the tool nor the prompt guidance appears. - If advisor is enabled, both the tool and the prompt guidance appear. - Root chats can use advisor; child chats cannot. - Built-in tool names include `advisor` so metrics do not collapse it into the generic `mcp` label. ## Phase 3 β€” Enforce planning-only execution policy in `chatloop` ### Goals Prevent the model from calling `advisor` and action tools in the same execution batch. ### Files to modify - `coderd/x/chatd/chatloop/chatloop.go` - related chatloop tests ### Recommended implementation Keep the MVP small; do **not** build a general policy engine yet. 1. Add a minimal field to `chatloop.RunOptions`, for example: ```go ExclusiveToolName *string ``` 2. In `Run()` / `executeTools()`, detect the case where the exclusive tool appears in the same local-tool batch as any other locally executed tool. 3. When that happens, synthesize structured tool-result errors for the affected calls instead of executing anything in the batch. - `advisor` should receive a clear error like: _advisor must be called by itself before action tools_. - The sibling action tools should receive a paired policy error like: _this tool was skipped because advisor must run alone_. 4. Let the outer model see those tool errors and retry cleanly. - This is simpler and safer than partial execution or hidden deferral. - It preserves deterministic transcript history for debugging. 5. Pass the just-finished step snapshot into the tool execution context. - The advisor runtime should be able to see the current step's text/reasoning content, because that is often the best hint about what the outer model is trying to decide. ### Why this is the right fit - It matches the intended semantics: advisor is consulted **before** taking action. - It avoids subtle race conditions caused by concurrent built-in tool execution. - It keeps the behavior easy to test with fake models. ### Acceptance criteria - A model-emitted batch containing only `advisor` succeeds. - A model-emitted batch containing `advisor` plus any other locally executed tool returns deterministic policy errors and executes nothing. - Non-advisor tool execution stays unchanged for normal chats. ## Phase 4 β€” Usage limits, metrics, and configuration ### Goals Make advisor safe to operate without over-designing billing/storage in the first release. ### Files to modify - `coderd/x/chatd/chatd.go` - `coderd/x/chatd/chatloop/metrics.go` as needed - `coderd/x/chatd/chatd.go` `Config` struct and constructor path - optional follow-up config/db files only if a separate advisor model or persistent billing is required ### Tasks 1. **Add explicit server config knobs for MVP.** Recommended fields on `chatd.Config` or a nested advisor config struct: - `AdvisorEnabled bool` - `AdvisorMaxUsesPerRun int` - `AdvisorMaxOutputTokens int64` 2. **Track usage per outer run.** - Reset the counter for each `processChat()` invocation. - Return `remaining_uses` in the tool result. - Return `limit_reached` when the cap is exhausted. 3. **Expose advisor usage metadata in the tool result.** - Include model name and token/cost summary if available. - Use the same `callConfig.Cost` calculation path as the outer chat for MVP if advisor reuses the same model. 4. **Record server-side metrics.** - Count advisor invocations, failures, and latency. - Ensure they show up under the built-in tool label `advisor`. 5. **Optional decision gate: separate advisor model.** - If product insists on a stronger/different advisor model, add a follow-up config hook that resolves another existing chat model config through the same `configCache` path. - Keep that out of the first landing PR unless it is required for acceptance. 6. **Optional decision gate: queryable advisor cost.** - If this becomes required, spin a follow-up DB task: - update `coderd/database/queries/*.sql`; - add migration files; - run `make gen`; - update audit mappings if a new auditable type/field is introduced. ### Acceptance criteria - Advisor calls are capped per outer run. - Limit exhaustion is user-visible in the tool result. - Metrics distinguish advisor calls from other built-in tools. - MVP does not require a schema migration unless explicitly approved. ## Phase 5 β€” Frontend rendering and Storybook coverage ### Goals Make advisor feel intentional in the Agents UI without blocking the backend on fancy streaming UI. ### Files to modify - `site/src/pages/AgentsPage/components/ChatElements/tools/Tool.tsx` - new `site/src/pages/AgentsPage/components/ChatElements/tools/AdvisorTool.tsx` - Storybook story file(s) in the same tools directory ### Delivery strategy 1. **Intermediate milestone during backend bring-up:** rely on the existing generic tool renderer if needed. - This is acceptable only as a short-lived integration checkpoint. 2. **Release milestone:** add a dedicated lightweight `AdvisorTool` renderer. - Reuse existing primitives: - `ToolCollapsible` - `ToolIcon` - `Response` for markdown/prose rendering - `ScrollArea` if the advice can be long - Keep styling light and consistent with the Agents page. - Do not add unnecessary React memoization in `site/src/pages/AgentsPage/`; that area is already React-Compiler aware. 3. **Render the structured result states cleanly.** - `advice` β€” readable prose/markdown with optional metadata footer. - `limit_reached` β€” warning-style message. - `error` β€” error state with visible fallback text. - `running` β€” existing tool loading state/spinner is enough for MVP. 4. **Add Storybook coverage instead of ad-hoc component tests.** Recommended stories: - successful advice; - running/loading; - limit reached; - error. 5. **Keep the UI contract narrow.** - Prefer one text field like `advice` plus small metadata rather than a deeply nested schema. - That keeps the UI resilient to prompt iteration. ### Acceptance criteria - The advisor tool card renders readable content rather than raw quoted JSON in the final release branch. - Running, limit, and error states are visibly distinct. - Storybook stories and play assertions cover the new states. - Existing tool rendering flows remain unchanged. ## Phase 6 β€” Automated tests and validation gates ### Backend tests to add 1. **Advisor runtime/tool tests** - question validation; - tool-less nested execution assertion; - success result shaping; - limit-reached result shaping; - error result shaping. 2. **Prompt/gating tests in chatd** - advisor disabled β‡’ no tool, no guidance; - advisor enabled/root chat β‡’ tool + guidance; - child chat β‡’ advisor absent. 3. **Chatloop policy tests** - advisor alone runs; - advisor + action tool mixed batch returns deterministic policy errors; - non-advisor tools still execute normally. 4. **Usage/metrics tests** - per-run cap resets correctly; - builtin tool labeling includes `advisor`; - returned metadata includes model/usage summary when available. ### Frontend tests to add - Storybook `play()` assertions for the advisor renderer states. - Verify expand/collapse behavior and visible fallback text. - Verify the message timeline still renders adjacent tools correctly. ### Recommended command sequence Run these as the implementation matures, not only at the end: 1. Backend-focused gate after phases 1–4: - `make test RUN=TestAdvisor` - `make test RUN=TestChatloopAdvisor` - `make lint` 2. Frontend-focused gate after phase 5: - `pnpm test:storybook src/pages/AgentsPage/components/ChatElements/tools/AdvisorTool.stories.tsx` - `pnpm lint` - `pnpm format` 3. Final repo gate before handoff: - `make pre-commit` - run any additional targeted `make test RUN=...` selections covering touched chatd paths > Use the exact new test names the implementing agents create; the names above are recommended anchors, not existing tests. ## Dogfooding plan ### Principle Dogfood the change as a real agent feature, not just a unit-tested backend. Per the dogfood and `agent-browser` skills, the reviewer should get **watchable repro videos** plus screenshots that make the behavior obvious without reading logs. ### Required setup 1. Start the full dev environment with: - `./scripts/develop.sh` 2. If the frontend renderer changes, also start Storybook from `site/` with: - `pnpm storybook --no-open` 3. Use `agent-browser` directly β€” **never `npx agent-browser`**. 4. Use named browser sessions and an output folder such as: - `./dogfood-output/advisor/` - with subfolders `screenshots/` and `videos/` ### Evidence protocol For every interactive scenario below: 1. Start video recording **before** the action. 2. Capture step-by-step screenshots at human pace. 3. Capture one annotated screenshot of the final state. 4. Stop the recording. 5. Note the exact pass/fail observation in the QA report. For static UI states (for example Storybook error/limit cards), an annotated screenshot is sufficient; video is optional but still encouraged by this project’s review preference. ### Dogfood scenarios #### Scenario A β€” Happy path in the real Agents UI **Goal:** prove that a root agent chat can invoke advisor and produce a readable recommendation before taking further action. Steps: 1. Open the Agents page with an advisor-enabled root chat. 2. Start a repro video. 3. Send a prompt that should reasonably trigger strategic planning, such as an architecture or multi-tradeoff question. 4. Capture screenshots of: - the prompt before send; - the running advisor state; - the completed advisor card and the assistant’s follow-up response. 5. Stop recording. Pass criteria: - advisor appears in the timeline; - the rendered result is readable; - the assistant can continue after consuming the advisor output. #### Scenario B β€” Advisor unavailable path **Goal:** prove the feature is truly gated. Suggested variants (at least one is required, both are better): - feature flag/config off; - child/sub-agent chat. Evidence: - annotated screenshot of the chat/tool state showing advisor is absent; - short video if toggling the gate live is part of the repro. Pass criteria: - no advisor tool is available; - no advisor-specific prompt behavior leaks through. #### Scenario C β€” UI states in Storybook **Goal:** prove the renderer handles non-happy states cleanly. Required story states: - success/advice; - running; - limit reached; - error. Evidence: - one screenshot per state; - at least one short video showing collapse/expand behavior. Pass criteria: - success renders readable advice; - limit/error have visible fallback text; - the component behaves like the other tool cards. #### Scenario D β€” Regression sweep of nearby tools **Goal:** ensure advisor does not break the surrounding chat timeline. Check at minimum: - another existing built-in tool still renders correctly near advisor; - sub-agent/tool cards still expand/collapse normally; - no obvious console errors appear in the Agents page during the advisor flow. Evidence: - screenshots of adjacent tool cards; - console/error capture if anything suspicious appears. ### `agent-browser` usage notes for the QA agent - Prefer `agent-browser batch` for 2+ sequential commands when no intermediate parsing is needed. - Use `snapshot -i` to discover interactive refs. - Re-snapshot after navigation or major DOM changes. - Avoid `wait --load networkidle` unless the page is known to go idle; prefer explicit element/text waits or short fixed waits. - Record videos at human pace and include pauses that a reviewer can follow. ## Rollout plan ### Initial rollout - Gate behind a server-side advisor-enabled flag. - Enable only for selected internal/root agent chats first. - Watch metrics for: - invocation count; - failure rate; - latency; - obvious retry loops. ### Expansion conditions Expand beyond the initial rollout only after the following are true: - mixed-batch policy behavior is stable; - cost impact is understood; - frontend UX is readable in production-like dogfood; - no recursion surprises have appeared with sub-agent flows. ### Explicit non-goals for the first release - advisor inside child/sub-agent chats; - provider-agnostic streaming phase UI; - MCP-based external advisor implementation; - mandatory DB-backed advisor cost reporting. ## Final acceptance checklist - [ ] `advisor` is a built-in chatd tool, not an MCP/dynamic-tool substitute. - [ ] The nested advisor call is tool-less and bounded to one in-memory step. - [ ] One eligibility boolean controls both tool registration and prompt guidance injection. - [ ] Root chats can use advisor; child chats cannot in the initial rollout. - [ ] Mixed advisor/action batches produce deterministic policy errors instead of partial execution. - [ ] Per-run usage caps and limit-reached behavior work. - [ ] Advisor usage is visible in metadata/metrics without forcing a DB migration for MVP. - [ ] The Agents UI has a readable advisor card and Storybook coverage. - [ ] Dogfooding produced screenshots and repro videos for the required scenarios. - [ ] Validation commands (`make lint`, targeted `make test`, Storybook tests, `make pre-commit`) passed before handoff. ## Suggested PR split 1. **PR 1 β€” Backend foundation** - `chatadvisor/` package - `chattool/advisor.go` - `chatloop` exclusive policy - chatd gating/prompt sync - backend tests 2. **PR 2 β€” Frontend + QA** - advisor renderer - stories/play assertions - dogfood artifacts and QA notes 3. **PR 3 β€” Optional follow-ups only if demanded by stakeholders** - separate advisor model override - persistent advisor billing/queryability - transient phase-stream UX
--- _Generated with [`mux`](https://github.com/coder/mux) β€’ Model: `anthropic:claude-opus-4-7` β€’ Thinking: `max`_ --- site/src/api/api.ts | 15 + site/src/api/queries/chats.test.ts | 55 ++ site/src/api/queries/chats.ts | 17 + .../AgentSettingsExperimentsPage.tsx | 26 + ...entSettingsExperimentsPageView.stories.tsx | 18 + .../AgentSettingsExperimentsPageView.tsx | 47 + .../components/AdvisorSettings.stories.tsx | 828 ++++++++++++++++++ .../AgentsPage/components/AdvisorSettings.tsx | 498 +++++++++++ 8 files changed, 1504 insertions(+) create mode 100644 site/src/pages/AgentsPage/components/AdvisorSettings.stories.tsx create mode 100644 site/src/pages/AgentsPage/components/AdvisorSettings.tsx diff --git a/site/src/api/api.ts b/site/src/api/api.ts index f8f9027303..06d3437edf 100644 --- a/site/src/api/api.ts +++ b/site/src/api/api.ts @@ -29,9 +29,11 @@ import { } from "../utils/OneWayWebSocket"; import { type FieldError, isApiError } from "./errors"; import type { + AdvisorConfig, DeleteExternalAuthByIDResponse, DynamicParametersRequest, PostWorkspaceUsageRequest, + UpdateAdvisorConfigRequest, UsersRequest, } from "./typesGenerated"; import * as TypesGen from "./typesGenerated"; @@ -3346,6 +3348,19 @@ class ExperimentalApiMethods { await this.axios.put("/api/experimental/chats/config/desktop-enabled", req); }; + getChatAdvisorConfig = async (): Promise => { + const response = await this.axios.get( + "/api/experimental/chats/config/advisor", + ); + return response.data; + }; + + updateChatAdvisorConfig = async ( + req: UpdateAdvisorConfigRequest, + ): Promise => { + await this.axios.put("/api/experimental/chats/config/advisor", req); + }; + getChatWorkspaceTTL = async (): Promise => { const response = await this.axios.get( diff --git a/site/src/api/queries/chats.test.ts b/site/src/api/queries/chats.test.ts index 15bf01bd76..eb0fb9c2bb 100644 --- a/site/src/api/queries/chats.test.ts +++ b/site/src/api/queries/chats.test.ts @@ -11,6 +11,8 @@ import { addChildToParentInCache, archiveChat, cancelChatListRefetches, + chatAdvisorConfig, + chatAdvisorConfigKey, chatCostSummary, chatCostSummaryKey, chatDebugRunsKey, @@ -37,6 +39,7 @@ import { TERMINAL_RUN_STATUSES, unarchiveChat, unpinChat, + updateChatAdvisorConfig, updateChatPlanMode, updateChildInParentCache, updateInfiniteChatsCache, @@ -57,6 +60,8 @@ vi.mock("#/api/api", () => ({ promoteChatQueuedMessage: vi.fn(), proposeChatTitle: vi.fn(), regenerateChatTitle: vi.fn(), + getChatAdvisorConfig: vi.fn(), + updateChatAdvisorConfig: vi.fn(), }, }, })); @@ -124,6 +129,56 @@ const createTestQueryClient = (): QueryClient => }, }); +describe("advisor config query factories", () => { + it("builds the advisor config query and delegates to the API", async () => { + const advisorConfig: TypesGen.AdvisorConfig = { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "00000000-0000-0000-0000-000000000000", + }; + vi.mocked(API.experimental.getChatAdvisorConfig).mockResolvedValue( + advisorConfig, + ); + + const query = chatAdvisorConfig(); + + expect(query.queryKey).toEqual(chatAdvisorConfigKey); + await expect(query.queryFn()).resolves.toEqual(advisorConfig); + expect(API.experimental.getChatAdvisorConfig).toHaveBeenCalled(); + }); + + it("sends the update request and invalidates the advisor config cache", async () => { + const queryClient = createTestQueryClient(); + queryClient.setQueryData(chatAdvisorConfigKey, { + enabled: false, + max_uses_per_run: 0, + max_output_tokens: 0, + reasoning_effort: "", + model_config_id: "", + } as TypesGen.AdvisorConfig); + + const req: TypesGen.UpdateAdvisorConfigRequest = { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "00000000-0000-0000-0000-000000000000", + }; + vi.mocked(API.experimental.updateChatAdvisorConfig).mockResolvedValue(); + + const mutation = updateChatAdvisorConfig(queryClient); + await mutation.mutationFn(req); + expect(API.experimental.updateChatAdvisorConfig).toHaveBeenCalledWith(req); + + await mutation.onSuccess?.(); + expect(queryClient.getQueryState(chatAdvisorConfigKey)?.isInvalidated).toBe( + true, + ); + }); +}); + describe("invalidateChatListQueries", () => { it("invalidates flat and infinite chat list queries", async () => { const queryClient = createTestQueryClient(); diff --git a/site/src/api/queries/chats.ts b/site/src/api/queries/chats.ts index c2124175d3..26b0c1b8f2 100644 --- a/site/src/api/queries/chats.ts +++ b/site/src/api/queries/chats.ts @@ -1330,6 +1330,23 @@ export const updateChatDesktopEnabled = (queryClient: QueryClient) => ({ }); export * from "./chatDebugLogging"; +export const chatAdvisorConfigKey = ["chat-advisor-config"] as const; + +export const chatAdvisorConfig = () => ({ + queryKey: chatAdvisorConfigKey, + queryFn: (): Promise => + API.experimental.getChatAdvisorConfig(), +}); + +export const updateChatAdvisorConfig = (queryClient: QueryClient) => ({ + mutationFn: (req: TypesGen.UpdateAdvisorConfigRequest) => + API.experimental.updateChatAdvisorConfig(req), + onSuccess: async () => { + await queryClient.invalidateQueries({ + queryKey: chatAdvisorConfigKey, + }); + }, +}); const chatWorkspaceTTLKey = ["chat-workspace-ttl"] as const; diff --git a/site/src/pages/AgentsPage/AgentSettingsExperimentsPage.tsx b/site/src/pages/AgentsPage/AgentSettingsExperimentsPage.tsx index 8e5c6294c7..e576783815 100644 --- a/site/src/pages/AgentsPage/AgentSettingsExperimentsPage.tsx +++ b/site/src/pages/AgentsPage/AgentSettingsExperimentsPage.tsx @@ -1,8 +1,11 @@ import type { FC } from "react"; import { useMutation, useQuery, useQueryClient } from "react-query"; import { + chatAdvisorConfig, chatDebugLogging, chatDesktopEnabled, + chatModelConfigs, + updateChatAdvisorConfig, updateChatDebugLogging, updateChatDesktopEnabled, } from "#/api/queries/chats"; @@ -21,12 +24,23 @@ const AgentSettingsExperimentsPage: FC = () => { ...chatDebugLogging(), enabled: permissions.editDeploymentConfig, }); + const advisorConfigQuery = useQuery({ + ...chatAdvisorConfig(), + enabled: permissions.editDeploymentConfig, + }); + const modelConfigsQuery = useQuery({ + ...chatModelConfigs(), + enabled: permissions.editDeploymentConfig, + }); const saveDesktopEnabledMutation = useMutation( updateChatDesktopEnabled(queryClient), ); const saveDebugLoggingMutation = useMutation( updateChatDebugLogging(queryClient), ); + const saveAdvisorConfigMutation = useMutation( + updateChatAdvisorConfig(queryClient), + ); return ( @@ -39,6 +53,18 @@ const AgentSettingsExperimentsPage: FC = () => { onSaveDebugLogging={saveDebugLoggingMutation.mutate} isSavingDebugLogging={saveDebugLoggingMutation.isPending} isSaveDebugLoggingError={saveDebugLoggingMutation.isError} + advisorConfigData={advisorConfigQuery.data} + isAdvisorConfigLoading={advisorConfigQuery.isLoading} + isAdvisorConfigFetching={advisorConfigQuery.isFetching} + isAdvisorConfigLoadError={advisorConfigQuery.isError} + modelConfigsData={modelConfigsQuery.data ?? []} + modelConfigsError={modelConfigsQuery.error} + isLoadingModelConfigs={modelConfigsQuery.isLoading} + isFetchingModelConfigs={modelConfigsQuery.isFetching} + onSaveAdvisorConfig={saveAdvisorConfigMutation.mutate} + isSavingAdvisorConfig={saveAdvisorConfigMutation.isPending} + isSaveAdvisorConfigError={saveAdvisorConfigMutation.isError} + saveAdvisorConfigError={saveAdvisorConfigMutation.error} /> ); diff --git a/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.stories.tsx b/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.stories.tsx index 1d9513cde5..444dc6ad2b 100644 --- a/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.stories.tsx +++ b/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.stories.tsx @@ -17,6 +17,24 @@ const baseArgs: AgentSettingsExperimentsPageViewProps = { onSaveDebugLogging: fn(), isSavingDebugLogging: false, isSaveDebugLoggingError: false, + advisorConfigData: { + enabled: false, + max_uses_per_run: 0, + max_output_tokens: 0, + reasoning_effort: "", + model_config_id: "", + }, + isAdvisorConfigLoading: false, + isAdvisorConfigFetching: false, + isAdvisorConfigLoadError: false, + modelConfigsData: [], + modelConfigsError: undefined, + isLoadingModelConfigs: false, + isFetchingModelConfigs: false, + onSaveAdvisorConfig: fn(), + isSavingAdvisorConfig: false, + isSaveAdvisorConfigError: false, + saveAdvisorConfigError: undefined, }; const meta = { diff --git a/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.tsx b/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.tsx index 2dd2284ebd..5671cbade9 100644 --- a/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.tsx +++ b/site/src/pages/AgentsPage/AgentSettingsExperimentsPageView.tsx @@ -2,9 +2,15 @@ import type { FC } from "react"; import type { UseMutateFunction } from "react-query"; import type * as TypesGen from "#/api/typesGenerated"; import { AdminChatDebugLoggingSettings } from "./components/AdminChatDebugLoggingSettings"; +import { AdvisorSettings } from "./components/AdvisorSettings"; import { SectionHeader } from "./components/SectionHeader"; import { VirtualDesktopSettings } from "./components/VirtualDesktopSettings"; +interface MutationCallbacks { + onSuccess?: () => void; + onError?: () => void; +} + export interface AgentSettingsExperimentsPageViewProps { desktopEnabledData: TypesGen.ChatDesktopEnabledResponse | undefined; onSaveDesktopEnabled: UseMutateFunction< @@ -24,6 +30,21 @@ export interface AgentSettingsExperimentsPageViewProps { >; isSavingDebugLogging: boolean; isSaveDebugLoggingError: boolean; + advisorConfigData: TypesGen.AdvisorConfig | undefined; + isAdvisorConfigLoading: boolean; + isAdvisorConfigFetching: boolean; + isAdvisorConfigLoadError: boolean; + modelConfigsData: readonly TypesGen.ChatModelConfig[]; + modelConfigsError: unknown; + isLoadingModelConfigs: boolean; + isFetchingModelConfigs: boolean; + onSaveAdvisorConfig: ( + req: TypesGen.UpdateAdvisorConfigRequest, + options?: MutationCallbacks, + ) => void; + isSavingAdvisorConfig: boolean; + isSaveAdvisorConfigError: boolean; + saveAdvisorConfigError: unknown; } export const AgentSettingsExperimentsPageView: FC< @@ -37,6 +58,18 @@ export const AgentSettingsExperimentsPageView: FC< onSaveDebugLogging, isSavingDebugLogging, isSaveDebugLoggingError, + advisorConfigData, + isAdvisorConfigLoading, + isAdvisorConfigFetching, + isAdvisorConfigLoadError, + modelConfigsData, + modelConfigsError, + isLoadingModelConfigs, + isFetchingModelConfigs, + onSaveAdvisorConfig, + isSavingAdvisorConfig, + isSaveAdvisorConfigError, + saveAdvisorConfigError, }) => { return (
@@ -50,6 +83,20 @@ export const AgentSettingsExperimentsPageView: FC< isSavingDesktopEnabled={isSavingDesktopEnabled} isSaveDesktopEnabledError={isSaveDesktopEnabledError} /> + { + options?.onSuccess?.(); + }), + isSavingAdvisorConfig: false, + isSaveAdvisorConfigError: false, + saveAdvisorConfigError: undefined, + }, + decorators: [ + (Story) => ( +
+ +
+ ), + ], +} satisfies Meta; + +export default meta; +type Story = StoryObj; + +export const Default: Story = { + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = canvas.getByRole("switch", { + name: /Enable advisor/i, + }); + + expect( + canvas.queryByRole("spinbutton", { name: /Max uses per run/i }), + ).not.toBeInTheDocument(); + expect( + canvas.queryByRole("combobox", { name: /Advisor model/i }), + ).not.toBeInTheDocument(); + + await userEvent.click(enableAdvisorSwitch); + + await waitFor(() => { + expect( + canvas.getByRole("spinbutton", { name: /Max uses per run/i }), + ).toBeVisible(); + expect( + canvas.getByRole("combobox", { name: /Advisor model/i }), + ).toBeVisible(); + }); + }, +}; + +export const Enabled: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const body = within(document.body); + const maxUsesInput = await canvas.findByRole("spinbutton", { + name: /Max uses per run/i, + }); + const maxOutputTokensInput = canvas.getByRole("spinbutton", { + name: /Max output tokens/i, + }); + const reasoningEffortSelect = canvas.getByRole("combobox", { + name: /Reasoning effort/i, + }); + const advisorModelSelect = canvas.getByRole("combobox", { + name: /Advisor model/i, + }); + const saveButton = canvas.getByRole("button", { name: /Save/i }); + + expect(saveButton).toBeDisabled(); + + await userEvent.clear(maxUsesInput); + await userEvent.type(maxUsesInput, "5"); + await userEvent.clear(maxOutputTokensInput); + await userEvent.type(maxOutputTokensInput, "2048"); + + await userEvent.click(reasoningEffortSelect); + await userEvent.click(await body.findByRole("option", { name: /^High$/i })); + + await userEvent.click(advisorModelSelect); + expect( + body.queryByRole("option", { name: /GPT-3.5 \(Disabled\)/i }), + ).not.toBeInTheDocument(); + await userEvent.click( + await body.findByRole("option", { name: /Claude Sonnet 4/i }), + ); + + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + const [request, options] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "model-2", + }); + expect(typeof options?.onSuccess).toBe("function"); + + await waitFor(() => { + expect(saveButton).toBeDisabled(); + }); + }, +}; + +export const SaveWithUseChatModel: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const maxUsesInput = await canvas.findByRole("spinbutton", { + name: /Max uses per run/i, + }); + + expect( + canvas.getByRole("combobox", { name: /Advisor model/i }), + ).toHaveTextContent(/Use chat model/i); + + await userEvent.clear(maxUsesInput); + await userEvent.type(maxUsesInput, "3"); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request.model_config_id).toBe(nilUUID); + }, +}; + +export const NilUUIDInitialRoundTrip: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + model_config_id: nilUUID, + }, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const advisorModelSelect = await canvas.findByRole("combobox", { + name: /Advisor model/i, + }); + + expect(advisorModelSelect).toHaveTextContent(/Use chat model/i); + }, +}; + +export const CustomConfig: Story = { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 7, + max_output_tokens: 8192, + reasoning_effort: "medium", + model_config_id: "model-2", + }, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const maxUsesInput = await canvas.findByRole("spinbutton", { + name: /Max uses per run/i, + }); + const maxOutputTokensInput = canvas.getByRole("spinbutton", { + name: /Max output tokens/i, + }); + + expect(maxUsesInput).toHaveValue(7); + expect(maxOutputTokensInput).toHaveValue(8192); + expect( + canvas.getByRole("combobox", { name: /Reasoning effort/i }), + ).toHaveTextContent(/Medium/i); + expect( + canvas.getByRole("combobox", { name: /Advisor model/i }), + ).toHaveTextContent(/Claude Sonnet 4/i); + }, +}; + +export const UnavailableSelectedModel: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + model_config_id: "22222222-2222-2222-2222-222222222222", + }, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const advisorModelSelect = await canvas.findByRole("combobox", { + name: /Advisor model/i, + }); + + expect(advisorModelSelect).toHaveTextContent( + /Unavailable model \(22222222-2222-2222-2222-222222222222\)/i, + ); + }, +}; + +export const Loading: Story = { + args: { + advisorConfigData: undefined, + isAdvisorConfigLoading: true, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByRole("switch", { name: /Enable advisor/i }), + ).toBeDisabled(); + expect(canvas.getByRole("button", { name: /Save/i })).toBeDisabled(); + }, +}; + +export const Refetching: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + isAdvisorConfigFetching: true, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByRole("switch", { name: /Enable advisor/i }), + ).toBeDisabled(); + expect( + canvas.getByRole("spinbutton", { name: /Max uses per run/i }), + ).toBeDisabled(); + expect( + canvas.getByRole("combobox", { name: /Reasoning effort/i }), + ).toBeDisabled(); + expect(canvas.getByRole("button", { name: /Save/i })).toBeDisabled(); + }, +}; + +export const LoadingModelConfigs: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + model_config_id: "model-2", + }, + modelConfigs: [], + isLoadingModelConfigs: true, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const advisorModelSelect = await canvas.findByRole("combobox", { + name: /Advisor model/i, + }); + + expect(advisorModelSelect).toBeDisabled(); + expect(advisorModelSelect).toHaveTextContent(/Loading/i); + expect( + canvas.getByText(/Loading chat model overrides\./i), + ).toBeInTheDocument(); + }, +}; + +export const ModelConfigsError: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + model_config_id: "model-2", + }, + modelConfigsError: new Error("fail"), + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const advisorModelSelect = await canvas.findByRole("combobox", { + name: /Advisor model/i, + }); + + expect(advisorModelSelect).toBeDisabled(); + expect( + canvas.getByText( + /Model overrides are unavailable\. The current selection will be sent unchanged\./i, + ), + ).toBeInTheDocument(); + }, +}; + +export const ModelConfigsErrorWithUnsetSelection: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + modelConfigsError: new Error("fail"), + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByText( + /Model overrides are unavailable\. Saving will keep using the chat model\./i, + ), + ).toBeInTheDocument(); + }, +}; + +export const LoadError: Story = { + args: { + advisorConfigData: undefined, + isAdvisorConfigLoadError: true, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByText(/Failed to load advisor settings\./i), + ).toBeInTheDocument(); + }, +}; + +export const Saving: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + isSavingAdvisorConfig: true, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByRole("switch", { name: /Enable advisor/i }), + ).toBeDisabled(); + expect( + canvas.getByRole("spinbutton", { name: /Max uses per run/i }), + ).toBeDisabled(); + expect( + canvas.getByRole("combobox", { name: /Reasoning effort/i }), + ).toBeDisabled(); + expect(canvas.getByRole("button", { name: /Save/i })).toBeDisabled(); + }, +}; + +export const SaveError: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + isSaveAdvisorConfigError: true, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByText(/Failed to save advisor settings\./i), + ).toBeInTheDocument(); + }, +}; + +export const SaveErrorWithDetail: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + }, + isSaveAdvisorConfigError: true, + saveAdvisorConfigError: new Error( + "reasoning_effort must be one of: low, medium, high.", + ), + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + + expect( + canvas.getByText(/reasoning_effort must be one of: low, medium, high\./i), + ).toBeInTheDocument(); + }, +}; + +export const DeselectModelBackToUseChatModel: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + model_config_id: "model-2", + }, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const body = within(document.body); + const advisorModelSelect = await canvas.findByRole("combobox", { + name: /Advisor model/i, + }); + + expect(advisorModelSelect).toHaveTextContent(/Claude Sonnet 4/i); + + await userEvent.click(advisorModelSelect); + await userEvent.click( + await body.findByRole("option", { name: /^Use chat model$/i }), + ); + + await waitFor(() => { + expect(advisorModelSelect).toHaveTextContent(/Use chat model/i); + }); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request.model_config_id).toBe(nilUUID); + }, +}; + +export const DisableAdvisorWithDeletedModel: Story = { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = await canvas.findByRole("switch", { + name: /Enable advisor/i, + }); + + expect( + canvas.getByRole("combobox", { name: /Advisor model/i }), + ).toHaveTextContent( + /Unavailable model \(22222222-2222-2222-2222-222222222222\)/i, + ); + + await userEvent.click(enableAdvisorSwitch); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + // The backend rejects unknown non-nil model IDs, so disabling must + // scrub the stale override rather than forwarding it and failing + // the save with a 400. + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: false, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: nilUUID, + }); + }, +}; + +export const DisableAdvisorWhileModelConfigsLoadingPreservesOverride: Story = { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }, + modelConfigs: [], + isLoadingModelConfigs: true, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = await canvas.findByRole("switch", { + name: /Enable advisor/i, + }); + + await userEvent.click(enableAdvisorSwitch); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + // While the model-configs query is in flight we cannot distinguish a + // genuinely deleted override from one we simply have not fetched yet, + // so the override must be forwarded unchanged. The backend will + // surface a specific 400 if the ID really has been deleted. + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: false, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }); + }, +}; + +export const DisableAdvisorWithModelConfigsErrorPreservesOverride: Story = { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }, + modelConfigs: [], + modelConfigsError: new Error("fail"), + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = await canvas.findByRole("switch", { + name: /Enable advisor/i, + }); + + await userEvent.click(enableAdvisorSwitch); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + // When the model-configs query has failed we cannot verify whether + // the override still exists, so the override must be forwarded + // unchanged rather than silently dropped. The backend surfaces a + // specific 400 if the ID really has been deleted. + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: false, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }); + }, +}; + +export const DisableAdvisorWhileModelConfigsRefetchingPreservesOverride: Story = + { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }, + // Simulate a background refetch with stale cached data: react-query + // keeps `isLoading` false once cached data exists, so only + // `isFetching` flags the in-flight refetch. The cached list does not + // contain the override ID. + modelConfigs: mockModelConfigs, + isLoadingModelConfigs: false, + isFetchingModelConfigs: true, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = await canvas.findByRole("switch", { + name: /Enable advisor/i, + }); + + await userEvent.click(enableAdvisorSwitch); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + // While a background refetch is in flight the cached model list may + // be stale, so the scrub guard must treat the absence of the + // override as indeterminate and forward the ID unchanged. Otherwise + // a still-valid override could be silently dropped just because the + // cache lags behind the server. + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: false, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }); + }, + }; + +export const DisableAdvisorWithDeletedModelAndEmptyModelConfigs: Story = { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "22222222-2222-2222-2222-222222222222", + }, + modelConfigs: [], + modelConfigsError: undefined, + isLoadingModelConfigs: false, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = await canvas.findByRole("switch", { + name: /Enable advisor/i, + }); + + await userEvent.click(enableAdvisorSwitch); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + // An empty model-configs list after a successful load is a definitive + // answer that the override no longer exists, so disabling must scrub + // the stale ID rather than forwarding it and failing the save with a + // 400. This covers the recovery case where every model config has + // been deleted. + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: false, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: nilUUID, + }); + }, +}; + +export const ValidationBlocksSave: Story = { + args: { + advisorConfigData: { + ...defaultAdvisorConfig, + enabled: true, + max_uses_per_run: 5, + }, + }, + play: async ({ canvasElement }) => { + const canvas = within(canvasElement); + const maxUsesInput = await canvas.findByRole("spinbutton", { + name: /Max uses per run/i, + }); + const saveButton = canvas.getByRole("button", { name: /Save/i }); + + expect(saveButton).toBeDisabled(); + + // Dirty-but-invalid state: the field is blank, so client validation + // must keep Save disabled even though the form is now dirty. + await userEvent.clear(maxUsesInput); + await waitFor(() => { + expect(saveButton).toBeDisabled(); + }); + + await userEvent.type(maxUsesInput, "3"); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + }, +}; + +export const DisableThenSave: Story = { + args: { + advisorConfigData: { + enabled: true, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "model-2", + }, + }, + play: async ({ canvasElement, args }) => { + const canvas = within(canvasElement); + const enableAdvisorSwitch = await canvas.findByRole("switch", { + name: /Enable advisor/i, + }); + + expect( + canvas.getByRole("spinbutton", { name: /Max uses per run/i }), + ).toBeVisible(); + + // Clear a numeric field to an invalid value before disabling. After + // disabling the field is hidden, and saving must not silently overwrite + // the stored limit with a coerced value. + const maxUsesInput = canvas.getByRole("spinbutton", { + name: /Max uses per run/i, + }); + await userEvent.clear(maxUsesInput); + + await userEvent.click(enableAdvisorSwitch); + + await waitFor(() => { + expect( + canvas.queryByRole("spinbutton", { name: /Max uses per run/i }), + ).not.toBeInTheDocument(); + }); + + const saveButton = canvas.getByRole("button", { name: /Save/i }); + await waitFor(() => { + expect(saveButton).toBeEnabled(); + }); + + await userEvent.click(saveButton); + + await waitFor(() => { + expect(args.onSaveAdvisorConfig).toHaveBeenCalled(); + }); + + const [request] = args.onSaveAdvisorConfig.mock.calls[0]; + expect(request).toEqual({ + enabled: false, + max_uses_per_run: 5, + max_output_tokens: 2048, + reasoning_effort: "high", + model_config_id: "model-2", + }); + }, +}; diff --git a/site/src/pages/AgentsPage/components/AdvisorSettings.tsx b/site/src/pages/AgentsPage/components/AdvisorSettings.tsx new file mode 100644 index 0000000000..2ea74aa5b1 --- /dev/null +++ b/site/src/pages/AgentsPage/components/AdvisorSettings.tsx @@ -0,0 +1,498 @@ +import { useFormik } from "formik"; +import { TriangleAlertIcon } from "lucide-react"; +import { type FC, useEffect, useId, useRef } from "react"; +import { getErrorMessage } from "#/api/errors"; +import type { + AdvisorConfig, + ChatModelConfig, + UpdateAdvisorConfigRequest, +} from "#/api/typesGenerated"; +import { Badge } from "#/components/Badge/Badge"; +import { Button } from "#/components/Button/Button"; +import { Input } from "#/components/Input/Input"; +import { Label } from "#/components/Label/Label"; +import { + Select, + SelectContent, + SelectItem, + SelectTrigger, + SelectValue, +} from "#/components/Select/Select"; +import { Switch } from "#/components/Switch/Switch"; + +const nilUUID = "00000000-0000-0000-0000-000000000000"; +const advisorReasoningEfforts = ["", "low", "medium", "high"] as const; +type AdvisorReasoningEffort = (typeof advisorReasoningEfforts)[number]; +const chatModelFallbackValue = "__use-chat-model__"; +const unavailableModelValue = "__unavailable-model__"; +const chatReasoningFallbackValue = "__use-chat-reasoning__"; + +interface MutationCallbacks { + onSuccess?: () => void; + onError?: () => void; +} + +interface AdvisorSettingsProps { + advisorConfigData: AdvisorConfig | undefined; + isAdvisorConfigLoading: boolean; + isAdvisorConfigFetching: boolean; + isAdvisorConfigLoadError: boolean; + modelConfigs: readonly ChatModelConfig[]; + modelConfigsError: unknown; + isLoadingModelConfigs: boolean; + isFetchingModelConfigs: boolean; + onSaveAdvisorConfig: ( + req: UpdateAdvisorConfigRequest, + options?: MutationCallbacks, + ) => void; + isSavingAdvisorConfig: boolean; + isSaveAdvisorConfigError: boolean; + saveAdvisorConfigError: unknown; +} + +type AdvisorSettingsFormValues = { + enabled: boolean; + max_uses_per_run: string; + max_output_tokens: string; + reasoning_effort: AdvisorReasoningEffort; + model_config_id: string; +}; + +const isUnsetModelConfigId = (id: string): boolean => + id === "" || id === nilUUID; + +const isAdvisorReasoningEffort = ( + value: string, +): value is AdvisorReasoningEffort => { + return advisorReasoningEfforts.includes(value as AdvisorReasoningEffort); +}; + +const normalizeNonNegativeInteger = ( + value: number | string | undefined, +): number => { + const parsed = typeof value === "number" ? value : Number(value); + if (!Number.isFinite(parsed) || parsed < 0) { + return 0; + } + return Math.trunc(parsed); +}; + +const normalizeAdvisorConfig = ( + config: AdvisorConfig | undefined, +): AdvisorSettingsFormValues => { + const reasoningEffort = config?.reasoning_effort ?? ""; + return { + enabled: config?.enabled ?? false, + max_uses_per_run: String( + normalizeNonNegativeInteger(config?.max_uses_per_run), + ), + max_output_tokens: String( + normalizeNonNegativeInteger(config?.max_output_tokens), + ), + reasoning_effort: isAdvisorReasoningEffort(reasoningEffort) + ? reasoningEffort + : "", + model_config_id: + typeof config?.model_config_id === "string" && + !isUnsetModelConfigId(config.model_config_id) + ? config.model_config_id + : "", + }; +}; + +const toAdvisorConfigRequest = ( + values: AdvisorSettingsFormValues, +): UpdateAdvisorConfigRequest => ({ + enabled: values.enabled, + max_uses_per_run: normalizeNonNegativeInteger(values.max_uses_per_run), + max_output_tokens: normalizeNonNegativeInteger(values.max_output_tokens), + reasoning_effort: values.reasoning_effort, + model_config_id: isUnsetModelConfigId(values.model_config_id) + ? nilUUID + : values.model_config_id, +}); + +const isNonNegativeIntegerString = (value: string): boolean => { + if (value.trim() === "") { + return false; + } + const parsed = Number(value); + return Number.isFinite(parsed) && parsed >= 0 && Number.isInteger(parsed); +}; + +const validateAdvisorConfig = (values: AdvisorSettingsFormValues) => { + const errors: Partial> = {}; + + // Skip validation of the advisor-only fields when the feature is disabled. + // Those inputs are hidden, so an admin disabling the advisor should not be + // blocked by stale invalid values left in hidden fields. + if (!values.enabled) { + return errors; + } + + if (!isNonNegativeIntegerString(values.max_uses_per_run)) { + errors.max_uses_per_run = + "Max uses per run must be a non-negative integer."; + } + + if (!isNonNegativeIntegerString(values.max_output_tokens)) { + errors.max_output_tokens = + "Max output tokens must be a non-negative integer."; + } + + if (!isAdvisorReasoningEffort(values.reasoning_effort)) { + errors.reasoning_effort = "Select a valid reasoning effort."; + } + + return errors; +}; + +const getModelDisplayName = (config: ChatModelConfig): string => + config.display_name.trim() || config.model; + +const getReasoningEffortLabel = (value: AdvisorReasoningEffort): string => { + switch (value) { + case "low": + return "Low"; + case "medium": + return "Medium"; + case "high": + return "High"; + default: + return "Use chat model default"; + } +}; + +export const AdvisorSettings: FC = ({ + advisorConfigData, + isAdvisorConfigLoading, + isAdvisorConfigFetching, + isAdvisorConfigLoadError, + modelConfigs, + modelConfigsError, + isLoadingModelConfigs, + isFetchingModelConfigs, + onSaveAdvisorConfig, + isSavingAdvisorConfig, + isSaveAdvisorConfigError, + saveAdvisorConfigError, +}) => { + const maxUsesId = useId(); + const maxOutputTokensId = useId(); + const hasLoadedAdvisorConfig = advisorConfigData !== undefined; + const enabledModelConfigs = modelConfigs.filter((config) => config.enabled); + + // Track the most recent committed advisor values (the server's view or the + // last successful save). Reading `advisorConfigData` directly in `onSubmit` + // can yield a stale snapshot when a refetch is in flight or has failed, + // which would silently roll back recently saved limits if the user then + // disables the advisor before the query settles. + const committedValuesRef = useRef( + normalizeAdvisorConfig(advisorConfigData), + ); + useEffect(() => { + committedValuesRef.current = normalizeAdvisorConfig(advisorConfigData); + }, [advisorConfigData]); + + const form = useFormik({ + enableReinitialize: true, + validateOnMount: true, + initialValues: normalizeAdvisorConfig(advisorConfigData), + validate: validateAdvisorConfig, + onSubmit: (values, { resetForm }) => { + // When disabling, preserve the last committed values for the hidden + // fields so potentially invalid in-flight edits (empty strings, + // fractional numbers) cannot silently overwrite previously + // configured limits, and so a pending or failed refetch of the + // advisor config cannot revert recently saved values. + let source: AdvisorSettingsFormValues = values.enabled + ? values + : { ...committedValuesRef.current, enabled: false }; + // If the last committed model override references a model config + // that no longer exists, the backend rejects the stale ID with a + // 400. When disabling, clear the override so a simple disable + // stays reliable in that edge case; the override is unusable + // anyway and the admin will reselect one on re-enable. Only scrub + // when model configs have loaded successfully and no refetch is in + // flight: during an initial load, a background refetch, or on + // error we cannot distinguish "truly missing" from "not loaded + // yet", and deciding from stale cache could either preserve a + // now-deleted ID (causing a 400 on disable/save) or silently drop + // an override that is actually still valid but missing from a + // stale cache. `isLoading` alone is insufficient because + // react-query keeps it false during background refetches when + // cached data already exists, so `isFetching` covers that gap. An + // empty list after a successful load is a definitive answer, so + // the scrub still fires (covers the recovery case where every + // model config has been deleted). + if ( + !source.enabled && + !isUnsetModelConfigId(source.model_config_id) && + !isLoadingModelConfigs && + !isFetchingModelConfigs && + !modelConfigsError && + !modelConfigs.some((config) => config.id === source.model_config_id) + ) { + source = { ...source, model_config_id: "" }; + } + const request = toAdvisorConfigRequest(source); + onSaveAdvisorConfig(request, { + onSuccess: () => { + const nextValues = normalizeAdvisorConfig(request); + committedValuesRef.current = nextValues; + resetForm({ values: nextValues }); + }, + }); + }, + }); + + const isFormDisabled = + isSavingAdvisorConfig || + isAdvisorConfigLoading || + isAdvisorConfigFetching || + !hasLoadedAdvisorConfig; + const isModelSelectDisabled = + isFormDisabled || isLoadingModelConfigs || Boolean(modelConfigsError); + const hasUnavailableSelectedModel = + !isLoadingModelConfigs && + !isUnsetModelConfigId(form.values.model_config_id) && + !enabledModelConfigs.some( + (config) => config.id === form.values.model_config_id, + ); + const selectedModelConfig = modelConfigs.find( + (config) => config.id === form.values.model_config_id, + ); + const selectedModelLabel = isUnsetModelConfigId(form.values.model_config_id) + ? "Use chat model" + : isLoadingModelConfigs + ? "Loading..." + : selectedModelConfig + ? getModelDisplayName(selectedModelConfig) + : `Unavailable model (${form.values.model_config_id})`; + const selectedModelValue = isUnsetModelConfigId(form.values.model_config_id) + ? chatModelFallbackValue + : hasUnavailableSelectedModel + ? unavailableModelValue + : form.values.model_config_id; + const modelHelperText = isLoadingModelConfigs + ? "Loading chat model overrides." + : modelConfigsError + ? isUnsetModelConfigId(form.values.model_config_id) + ? "Model overrides are unavailable. Saving will keep using the chat model." + : "Model overrides are unavailable. The current selection will be sent unchanged." + : "Choose a dedicated advisor model, or leave this unset to reuse the chat model."; + + return ( +
+
+

+ Advisor +

+ + + Experimental feature + +
+
+
+

+ Allow root agent chats to call the advisor tool for strategic + guidance. +

+

+ When enabled, you can cap advisor usage per run and optionally use + an override model. +

+
+ + void form.setFieldValue("enabled", checked) + } + aria-label="Enable advisor" + disabled={isFormDisabled} + /> +
+ + {form.values.enabled && ( +
+
+ + + void form.setFieldValue( + "max_uses_per_run", + event.currentTarget.value, + ) + } + onBlur={form.handleBlur} + aria-invalid={Boolean(form.errors.max_uses_per_run)} + disabled={isFormDisabled} + className="h-9 bg-surface-primary text-[13px]" + /> +

+ Set to 0 to leave the per-run call count unlimited. +

+
+ +
+ + + void form.setFieldValue( + "max_output_tokens", + event.currentTarget.value, + ) + } + onBlur={form.handleBlur} + aria-invalid={Boolean(form.errors.max_output_tokens)} + disabled={isFormDisabled} + className="h-9 bg-surface-primary text-[13px]" + /> +

+ Set to 0 to use the server default output limit. +

+
+ +
+ + +

+ Controls how hard the advisor model reasons before responding. + Leave unset to use the model's default. +

+
+ +
+ + +

+ {modelHelperText} +

+
+
+ )} + +
+ +
+ + {isSaveAdvisorConfigError && ( +

+ {getErrorMessage( + saveAdvisorConfigError, + "Failed to save advisor settings.", + )} +

+ )} + {isAdvisorConfigLoadError && ( +

+ Failed to load advisor settings. +

+ )} +
+ ); +};