mirror of
https://github.com/n8n-io/n8n.git
synced 2026-08-28 17:22:01 +08:00
feat(ai-builder): Push a seeded case, and hand the agent its workflow (no-changelog) (#35304)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
committed by
GitHub
parent
a74c6fd11f
commit
166771c77d
@@ -29,11 +29,12 @@ exhaustive field reference; this skill is the opinionated *how*.
|
||||
> `--source langtracer`. You still write the JSON file — it's just the input to
|
||||
> the push, not a committed artifact.
|
||||
>
|
||||
> **Exception — seeded cases.** The case-write API has no `seed` field, so seeded
|
||||
> cases are never pushed. A `seed.mode: "replay"` case is a local throwaway (don't
|
||||
> commit it either — it dies when its trace is pruned); a `seed.mode: "inline"`
|
||||
> case isn't transient and has no suite home, so it's the one sanctioned
|
||||
> exception — it lives as committed JSON. See [`case-shapes.md`](case-shapes.md).
|
||||
> **Seeded cases.** An `inline` seed pushes with the case — the case-write API
|
||||
> stores it verbatim, so the suite is its home like any other case. Only a
|
||||
> `seed.mode: "replay"` case is refused (listed under `skipped:`): it's
|
||||
> reconstructed from a LangSmith trace at run time, so it dies when that trace is
|
||||
> pruned and has no durable home. Don't commit a replay case either — derive a
|
||||
> synthetic case from it. See [`case-shapes.md`](case-shapes.md).
|
||||
|
||||
## Set the autonomy level first
|
||||
|
||||
@@ -206,9 +207,9 @@ calibration you hand the driver the thread link + login to review the real build
|
||||
[Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)); the suite is the
|
||||
case's home, not the repo. Leave the `data/workflows/*.json` file uncommitted
|
||||
(or delete it once it's in the suite). Committing new case JSONs into the repo
|
||||
is no longer the approach. (Exception: seeded cases can't be pushed — an
|
||||
`inline`-seeded case stays committed JSON, a `replay` case is
|
||||
a local throwaway; see [`case-shapes.md`](case-shapes.md).) For a sourced case,
|
||||
is no longer the approach. (An `inline` seed pushes with the case; only a
|
||||
`replay` case is refused — it's a local throwaway; see
|
||||
[`case-shapes.md`](case-shapes.md).) For a sourced case,
|
||||
finish by **linking it to its source thread/finding** over the MCP — see
|
||||
[Link the pushed case to its source](#link-the-pushed-case-to-its-source-provenance-step--always-do-this).
|
||||
|
||||
@@ -667,12 +668,12 @@ npx dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --cha
|
||||
lang-tracer #48), so scenario edits re-push like any other field. A lang-tracer
|
||||
deployment predating that change silently ignores the key; if a pushed scenario
|
||||
edit doesn't land, update the scenario in the lang-tracer UI.
|
||||
- **Seeded cases can't be pushed:** the case-write API has no `seed` field, so the
|
||||
push lists any seeded case under `skipped:` and it never reaches the suite. A
|
||||
`replay` case shouldn't be committed either — it dies when its trace is pruned
|
||||
or deleted — so derive a durable synthetic case as the artifact instead. An
|
||||
`inline`-seeded case isn't transient and has no suite home, so it's the one
|
||||
exception to "don't commit the JSON" — it lives as a committed artifact.
|
||||
- **An `inline` seed pushes with the case:** the case-write API stores it
|
||||
verbatim, so a seeded case lives in a suite like any other. Only a `replay`
|
||||
case is refused — the push lists it under `skipped:`, because it's
|
||||
reconstructed from a LangSmith trace at run time and dies when that trace is
|
||||
pruned or deleted. Don't commit a replay case either; derive a durable
|
||||
synthetic case as the artifact instead.
|
||||
|
||||
### Link the pushed case to its source (provenance step — always do this)
|
||||
|
||||
|
||||
@@ -443,13 +443,12 @@ which user turn goes live.
|
||||
seeds with **no workflow to inspect**. Reproduce the target workflow yourself
|
||||
(a synthetic case whose `executionScenarios` precondition builds the stand-in),
|
||||
or grade the live turn with `processExpectations` only.
|
||||
- **Can't be pushed to a lang-tracer suite either.** The case-write API has no
|
||||
`seed` field, so `eval:langtracer-push` silently lists any seeded case under
|
||||
`skipped:`. Combined with the don't-commit rule above, a `replay` case has **no
|
||||
durable home by design** — the durable artifact is always the synthetic case you
|
||||
derive from it. (An `inline` seed carries no thread dependency and can't be
|
||||
pushed either, so — unlike a normal case — it's the one exception to the skill's
|
||||
"push, don't commit the JSON" rule: it lives as a committed artifact.)
|
||||
- **Can't be pushed to a lang-tracer suite either.** `eval:langtracer-push`
|
||||
refuses a `replay` case and lists it under `skipped:` — a suite is a durable
|
||||
home and this seed isn't. Combined with the don't-commit rule above, a `replay`
|
||||
case has **no durable home by design** — the durable artifact is always the
|
||||
synthetic case you derive from it. (An `inline` seed has no such problem: it
|
||||
pushes with the case and lives in the suite like any other.)
|
||||
|
||||
### `mode: "inline"` — durable synthetic fixture
|
||||
|
||||
@@ -487,10 +486,73 @@ write a message as `{role, text}` and the schema expands it to a full envelope:
|
||||
`text` also takes an array of lines (newline-joined), like a `conversation` turn.
|
||||
The expansion stamps `createdAt` itself — ascending, in the past — so a shorthand
|
||||
message can't order *after* the live turn. Shorthand and full envelopes mix freely
|
||||
in one array; a full envelope keeps its authored `createdAt` — **unless any message
|
||||
in the array is in the FUTURE**, in which case the whole sequence is restamped onto
|
||||
ascending pre-live slots (a future stamp would sort a seeded turn after the live
|
||||
turn, and moving only that one entry would reorder it against the array the
|
||||
transcript is graded from). A near-miss (say
|
||||
in one array; a full envelope keeps its authored `createdAt` — **unless the authored
|
||||
stamps don't already ascend and sit in the past**, in which case the whole sequence
|
||||
is restamped onto ascending pre-live slots. A future stamp would sort a seeded turn
|
||||
after the live turn, and a non-ascending sequence (a shorthand turn appended after
|
||||
later-stamped envelopes, say) would present the history in an order the graded
|
||||
transcript never had; restamping only the offending entry would reorder it against
|
||||
the array the transcript is graded from. A near-miss (say
|
||||
`text: 123`) is deliberately **not** expanded — it fails at load instead of
|
||||
becoming a message the transcript builder would silently drop.
|
||||
|
||||
#### Which opening shape — the agent is handed the workflow, or it has to find it
|
||||
|
||||
Two real conversations look the same in a case file but test different things, and
|
||||
picking wrong makes the case harder than reality.
|
||||
|
||||
**Handed it.** The user is looking at a workflow and opens the assistant: "why is
|
||||
this failing?", "add error handling". They never name it — the editor sends the
|
||||
workflow along as a resource reference and the agent resolves it by **id**. Declare
|
||||
that with `attach` on the opening turn:
|
||||
|
||||
```json
|
||||
"conversation": [
|
||||
{ "role": "user", "text": "why is this failing?", "attach": { "workflow": "wKk3RmT9xQ2bVn7L" } }
|
||||
],
|
||||
"seed": {
|
||||
"mode": "inline",
|
||||
"messages": [ … ],
|
||||
"workflows": [ { "id": "wKk3RmT9xQ2bVn7L", "name": "Batch loop", … } ]
|
||||
}
|
||||
```
|
||||
|
||||
The id is the one the **seed declares**; the harness swaps in the per-run id, so you
|
||||
track nothing. Only the opening turn may carry `attach` (an attachment is a hand-off,
|
||||
not something a user re-sends), and it must name a workflow the inline seed declares —
|
||||
both are refused at load rather than ignored.
|
||||
|
||||
**The opening often has no text at all** — the user opens the assistant on a workflow
|
||||
and waits for it to speak first. Keep `"text": ""` when that's what happened; it's the
|
||||
faithful shape, and openings with no user text jumped from 1% to 31% of the corpus when
|
||||
the editor hand-off shipped, so it is not an edge case. Note that an empty text is
|
||||
valid *only* alongside `attach`: the chat API rejects a message that is empty with
|
||||
nothing attached ("Either message or attachments must be provided"), so the two stand
|
||||
or fall together.
|
||||
|
||||
**Has to find it.** The user refers to the workflow in words: by name ("the Wait node
|
||||
in *Generate leads* failed") or loosely ("the batch image workflow"). No `attach` —
|
||||
finding it *is* part of what the case tests. This also works when the seeded history
|
||||
already shows the agent building it, since the id is in its own record.
|
||||
|
||||
Get this wrong in the "handed it" direction — omit `attach` on a conversation that
|
||||
really had one — and the agent has to guess from prose that deliberately names
|
||||
nothing. It will list workflows and pick, or ask which one, and you will score a
|
||||
clarification failure the real user never hit.
|
||||
|
||||
#### Before you ship a seeded case — three checks
|
||||
|
||||
1. **The defect still bites.** A seed whose workflow isn't broken any more makes the
|
||||
case a silent no-op that passes forever.
|
||||
2. **Run it once with the seed removed. It must fail.** Copy the case, delete `seed`,
|
||||
run both. If the no-seed copy also passes, the seed isn't carrying the test. Name
|
||||
the copy so it doesn't share a `--filter` substring with the real case
|
||||
(`control-noseed-<slug>` works; `<slug>-noseed` would match both).
|
||||
3. **The workflow's skeleton is untouched** if you scrubbed it from a real one: node
|
||||
types, versions and connection topology byte-identical, only string leaves moved.
|
||||
|
||||
Don't grade a seeded case on cost, turn count, `messageBudget`, or "fixed it in one
|
||||
build". A seeded thread starts with an empty sandbox, so the agent re-reads the
|
||||
workflow from the database and re-derives SDK source a real resumed session would
|
||||
still have on disk. The bias is *harder* than reality, so those numbers read worse
|
||||
for a reason that has nothing to do with the builder.
|
||||
|
||||
@@ -69,7 +69,7 @@ cleaned up by the next run rather than contaminating it.
|
||||
|
||||
| Source | When to use it |
|
||||
|---|---|
|
||||
| **`disk`** (default) | **Preferred for local development** — authoring and calibrating the case in front of you: drop the JSON into `data/workflows/`, `--filter` it, iterate. Also the only home of the `agents` tier and the seeded carve-out cases; since the corpus migration the directory holds only those, not the full suite. |
|
||||
| **`disk`** (default) | **Preferred for local development** — authoring and calibrating the case in front of you: drop the JSON into `data/workflows/`, `--filter` it, iterate. Also the only home of the `agents` tier and of a `replay`-seeded case (reconstructed from a trace at run time, so no suite can hold it); since the corpus migration the directory holds only those, not the full suite. |
|
||||
| **`langtracer`** (`--source langtracer --suite baseline`) | Bigger runs (the full corpus or a whole tier), re-running specific cases that already live in the suite, and CI — which always runs this way. Needs `LANGTRACER_URL`/`LANGTRACER_API_KEY` in your env. |
|
||||
|
||||
## Configuration & secrets
|
||||
|
||||
@@ -62,12 +62,38 @@ connected.)
|
||||
voice.
|
||||
5. **Push it to a curated suite** (don't commit the JSON) with
|
||||
`eval:langtracer-push` — see
|
||||
[Push to a lang-tracer suite](SKILL.md#push-to-a-lang-tracer-suite). Exception:
|
||||
seeded cases (any `seed` mode) can't be pushed — the case-write API has no
|
||||
`seed` field, so the push lists them under
|
||||
`skipped:`. And a `replay` case shouldn't be committed either — it dies when
|
||||
its trace is pruned or deleted — so it has no durable home; that's exactly why
|
||||
step 4 turns the confirmed failure into a durable synthetic case.
|
||||
[Push to a lang-tracer suite](SKILL.md#push-to-a-lang-tracer-suite). An `inline`
|
||||
seed rides along with the case. Exception: a `replay` case is refused and listed
|
||||
under `skipped:` — it's reconstructed from a trace at run time, so it dies when
|
||||
that trace is pruned and has no durable home; that's exactly why step 4 turns the
|
||||
confirmed failure into a durable synthetic case.
|
||||
|
||||
## Was the workflow handed over? (sourcing an `attach` opening)
|
||||
|
||||
Whether the opening turn carries `attach` is a **fact about the thread**, not a
|
||||
judgement call — and guessing wrong makes the case harder than reality. Read it two
|
||||
ways, depending on when the thread was imported:
|
||||
|
||||
- **Recorded.** Turn 0 of `get_conversation` carries
|
||||
`resourceAttachments: [{ "type": "workflow", "id": "…" }]`. That's the editor
|
||||
hand-off verbatim. Only kind and ids are stored, never the workflow's name.
|
||||
- **Inferred**, for threads imported before n8n traced it. The tell is an opening turn
|
||||
whose `userMessage` is **empty** — the editor's context block is stripped before the
|
||||
trace, so a hand-off where the user typed nothing leaves a blank record. Corroborate
|
||||
with an early `workflows[get]` on a workflow the user never named.
|
||||
|
||||
Then use the id to recover the workflow itself: find the tool call whose *input*
|
||||
carries that `workflowId` (usually `workflows[get]`, `full: true`) — its **output** is
|
||||
the workflow, nodes and connections and parameters, and that is what you scrub into
|
||||
`seed.workflows`. The id alone is only a join key; it addresses the user's own
|
||||
instance, so it is worthless by itself.
|
||||
|
||||
Two things to expect:
|
||||
|
||||
- **The agent may never have read it.** Then you have an id and no content, and the
|
||||
workflow is yours to write — keep the topology plausible for the complaint.
|
||||
- **The opening often has no text at all.** Keep it that way; see the empty-opening
|
||||
note in [`case-shapes.md`](case-shapes.md).
|
||||
|
||||
## Scrubbing a real workflow into a synthetic seed
|
||||
|
||||
|
||||
@@ -181,7 +181,7 @@ A case can belong to multiple groupings — e.g. PR-tier cases declare `"dataset
|
||||
|
||||
**LangTracer is the source of truth for the workflow-eval corpus** — the `baseline` suite holds the cases, and CI pulls it on every run (see `.github/workflows/test-evals-instance-ai.yml`). The two `--source` modes split the work:
|
||||
|
||||
- **`disk` (the default) — the preferred mode for local development.** Reads `data/workflows/` and `data/agents/`. Use it while authoring and calibrating a case: drop the JSON in, `--filter` it, iterate. It is also the only home of the seeded carve-out cases (the case-write API can't represent them yet). Everything else — including the agents-team cases (suite `agents`: agent-artifact + intent-resolution) — lives in LangTracer, so disk mode is about the case in front of you, not the full suite.
|
||||
- **`disk` (the default) — the preferred mode for local development.** Reads `data/workflows/` and `data/agents/`. Use it while authoring and calibrating a case: drop the JSON in, `--filter` it, iterate. It is also the only home of a `replay`-seeded case — that seed is reconstructed from a LangSmith trace at run time, so no suite can be its durable home. Everything else — including the agents-team cases (suite `agents`: agent-artifact + intent-resolution) — lives in LangTracer, so disk mode is about the case in front of you, not the full suite.
|
||||
- **`langtracer` — for bigger runs, already-pushed cases, and CI.** Pulls a suite from [LangTracer](https://github.com/n8n-io/lang-tracer)'s REST API (`GET /api/v1/suites/:id/export`), validated through the same `EvalTestCaseSchema`. Reach for it locally when you want the real corpus (a full or tier run) or to re-run a specific case that already lives in the suite; CI always runs this way.
|
||||
|
||||
Set these in `.env.local`:
|
||||
@@ -414,7 +414,7 @@ dotenvx run -f ../../../.env.local -- pnpm eval:instance-ai \
|
||||
|
||||
For runs that need to leave the n8n repo (for example, driving the build from a separate Claude project where you have skills configured), three flags decouple the script from its default assumptions:
|
||||
|
||||
- `--workflow-dir <path>` — read test-case JSONs from a different directory (the default `evaluations/data/workflows/` now holds only the seeded carve-out cases; use `--source langtracer --suite <slug>` for the real corpus). When set, the script no longer needs `git rev-parse` to find the repo.
|
||||
- `--workflow-dir <path>` — read test-case JSONs from a different directory (the default `evaluations/data/workflows/` is an authoring dir, not the corpus; use `--source langtracer --suite <slug>` for that). When set, the script no longer needs `git rev-parse` to find the repo.
|
||||
- `--build-cwd <path>` — set the working directory the `claude` subprocess spawns from. Affects which `~/.claude.json` `projects` entry (and which skills) Claude loads.
|
||||
- `--project-id <id>` — instructs the model to pass `projectId` to `create_workflow_from_code` so workflows land in a specific n8n project instead of the user's personal one.
|
||||
|
||||
@@ -673,7 +673,7 @@ To record an isolated cohort without touching the shared dataset or baseline —
|
||||
|
||||
## Adding test cases
|
||||
|
||||
The corpus lives in **LangTracer** — suite `baseline` is what CI runs. Author a case as a local JSON file in `evaluations/data/workflows/` (disk mode picks it up, no registration step), calibrate it against a real build, then push it to the suite with `pnpm eval:langtracer-push --suite baseline <slug>` and delete the local file rather than committing it. Seeded cases (the `seed` slot) are the exception — the case-write API can't represent them yet, so they stay as committed JSON. Every case is validated against `harness/schema.ts`.
|
||||
The corpus lives in **LangTracer** — suite `baseline` is what CI runs. Author a case as a local JSON file in `evaluations/data/workflows/` (disk mode picks it up, no registration step), calibrate it against a real build, then push it to the suite with `pnpm eval:langtracer-push --suite baseline <slug>` and delete the local file rather than committing it. An `inline` seed pushes with the case; a `replay`-seeded case is refused (it's reconstructed from a LangSmith trace at run time, so a suite can't be its home). Every case is validated against `harness/schema.ts`.
|
||||
|
||||
> The essentials are below. For the full authoring guide — picking a case archetype, sizing assertions so a wrong build fails, multi-turn director scripts, seeding vs synthetic, and calibrating against a real build — follow the [`create-instance-ai-eval` skill](../../../../.agents/skills/create-instance-ai-eval/SKILL.md) (with [`case-shapes.md`](../../../../.agents/skills/create-instance-ai-eval/case-shapes.md) and [`running-evals.md`](../../../../.agents/skills/create-instance-ai-eval/running-evals.md)).
|
||||
|
||||
@@ -837,16 +837,52 @@ message as `{role, text}` and the schema expands it into a full envelope for you
|
||||
The expansion stamps `createdAt` itself — ascending and in the past — so a shorthand
|
||||
message can't accidentally order *after* the live turn. Shorthand and full envelopes
|
||||
can be mixed in one `messages` array; a full envelope keeps its authored `createdAt`
|
||||
— **unless any message in the array is stamped in the FUTURE**, in which case the
|
||||
whole sequence is restamped onto the same ascending pre-live slots. A future stamp
|
||||
would sort a seeded turn after the live turn, and clamping only the offending entry
|
||||
would reorder it relative to the array the transcript is graded from.
|
||||
— **unless the authored stamps don't already ascend and sit in the past**, in which
|
||||
case the whole sequence is restamped onto the same ascending pre-live slots. A future
|
||||
stamp would sort a seeded turn after the live turn, and a non-ascending sequence (a
|
||||
shorthand turn appended after later-stamped envelopes, say) would present the history
|
||||
in an order the graded transcript never had; restamping only the offending entry would
|
||||
reorder it relative to the array the transcript is graded from. The slots are fixed,
|
||||
never derived from the current time, so re-pushing an unchanged case is a no-op.
|
||||
A near-miss (say `text: 123`) is deliberately **not** expanded: it falls through to
|
||||
the envelope rules above and fails at load, rather than becoming a message the
|
||||
transcript builder would silently drop.
|
||||
|
||||
The seed lives **in the case body** rather than in a sibling file, so it travels with the case whatever the source — a JSON on disk, a suite pulled with `--source langtracer`, or a case body handed to a dispatcher. (There used to be a `seedFile` path pointing at a sibling JSON. Only the disk loader could resolve it, so a case delivered any other way lost its seed; the key is gone and a case still carrying it fails at load.)
|
||||
|
||||
#### Handing the agent the workflow (`attach`)
|
||||
|
||||
When a real user opens the assistant with a workflow in front of them — "why is this
|
||||
failing?" — the editor sends that workflow as a resource reference and the agent
|
||||
resolves it by **id**, never by name. Declare it on the opening turn:
|
||||
|
||||
```json
|
||||
"conversation": [
|
||||
{ "role": "user", "text": "why is this failing?", "attach": { "workflow": "wKk3RmT9xQ2bVn7L" } }
|
||||
]
|
||||
```
|
||||
|
||||
The id is the one the seed declares; the harness substitutes the per-run remapped id,
|
||||
so the attachment always points at the workflow that actually exists. Only the opening
|
||||
turn may carry it, and it must name a workflow the inline seed declares — both are
|
||||
refused at case load rather than silently ignored.
|
||||
|
||||
Omit it when the user refers to the workflow in words instead ("the batch image
|
||||
workflow") — finding it is then part of what the case tests. Getting this backwards
|
||||
makes a case harder than reality: the agent has to guess from prose that deliberately
|
||||
names nothing, and a clarification the real user never saw scores as a failure.
|
||||
|
||||
> **Pushing an `attach` case needs lang-tracer [#119](https://github.com/n8n-io/lang-tracer/pull/119) deployed.**
|
||||
> Carrying `attach` through import and case-write is that PR's job; a deployment
|
||||
> predating it stores the turn without the key, so the case would come back from
|
||||
> the suite as a hand-off with neither text nor attachment — a quietly different
|
||||
> test. **You don't have to remember this:** the push re-reads the suite export
|
||||
> after every write and fails if the server didn't store what it was sent, naming
|
||||
> the fields it dropped. The same check covers a `seed` against a deployment
|
||||
> predating [#113](https://github.com/n8n-io/lang-tracer/pull/113). Until the
|
||||
> server is upgraded, keep such a case on disk (`--source disk`).
|
||||
> Round-trip coverage: `langtracer-to-exported.test.ts`.
|
||||
|
||||
#### How restore works (all paths)
|
||||
|
||||
At build time the seed is restored right after the credential pin: seeded workflows are recreated under **fresh ids and a per-restore unique name** (`… [seed <8 hex>]`) with node credentials stripped, and the message log is written verbatim. Both are remapped through the history, so parallel iterations never share a workflow row *or* a name — a seeded case's live turn names its workflow the way a user would, so a same-named copy is one the agent can ground on instead, and the judge would then grade a different workflow than the agent edited. Any leftover carrying the seed suffix with the same base name is deleted before the restore; workflows without the suffix (real ones, and anything the agent built) are never touched. Restore failures fail the build — a seeded case cannot meaningfully run unseeded. Seeded turns join the transcript marked as *seeded prior context*, visible to the expectations judge and prompt-aware checks but distinguishable from live behaviour.
|
||||
@@ -963,7 +999,7 @@ evaluations/
|
||||
├── checklist/ # LLM verification with retry
|
||||
├── credentials/ # Test credential seeding
|
||||
├── data/agents/ # authoring dir for intent-resolution cases (the corpus lives in LangTracer suite `agents`)
|
||||
├── data/workflows/ # seeded carve-out case JSONs (the corpus lives in LangTracer)
|
||||
├── data/workflows/ # authoring dir for case JSONs (the corpus lives in LangTracer)
|
||||
├── data/subagent/ # workflow-build compatibility fixture JSON files
|
||||
├── data/pairwise/ # Local pairwise fixture (small smoke set)
|
||||
├── harness/ # Runners: buildWorkflow + executeScenario (e2e), in-memory event bus (discovery)
|
||||
|
||||
+149
-1
@@ -2,6 +2,7 @@ import { vi } from 'vitest';
|
||||
|
||||
import type { N8nClient } from '../clients/n8n-client';
|
||||
import { buildWorkflow } from '../harness/build-workflow';
|
||||
import { recordUserTurn } from '../harness/chat-loop';
|
||||
import type { ConversationSeed } from '../harness/conversation-seed';
|
||||
import type { EvalLogger } from '../harness/logger';
|
||||
|
||||
@@ -14,6 +15,25 @@ vi.mock('../harness/chat-loop', () => ({
|
||||
recordUserTurn: vi.fn(),
|
||||
}));
|
||||
|
||||
// The proxy's real constructor builds an LLM agent; only the script it is handed
|
||||
// matters here, so capture that and stub the rest (runMultiTurnConversation is
|
||||
// mocked above, so no other method is reached).
|
||||
const { proxyScripts } = vi.hoisted(() => ({
|
||||
proxyScripts: [] as Array<Array<{ text: string }>>,
|
||||
}));
|
||||
|
||||
vi.mock('../utils/user-proxy', () => ({
|
||||
UserProxyLlm: class {
|
||||
constructor(config: { conversation: Array<{ text: string }> }) {
|
||||
proxyScripts.push(config.conversation);
|
||||
}
|
||||
respondToConfirmation = vi.fn().mockResolvedValue({ approve: true });
|
||||
ingestEvents = vi.fn();
|
||||
decideFollowUp = vi.fn().mockResolvedValue({ kind: 'done' });
|
||||
getDecisionStats = vi.fn().mockReturnValue({});
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../outcome/workflow-discovery', () => ({
|
||||
buildAgentOutcome: vi.fn().mockResolvedValue({
|
||||
workflowsCreated: [{ id: 'built-wf-1', name: 'Built', nodeCount: 3, active: false }],
|
||||
@@ -251,7 +271,6 @@ describe('buildWorkflow with an inline seed', () => {
|
||||
preRunWorkflowIds: new Set(['leftover-1', 'leftover-2', 'leftover-3']),
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
expect(deleteWorkflow.mock.calls.map((call) => String(call[0]))).toEqual([
|
||||
'leftover-1',
|
||||
'leftover-2',
|
||||
@@ -260,6 +279,135 @@ describe('buildWorkflow with an inline seed', () => {
|
||||
expect(build.success).toBe(true);
|
||||
});
|
||||
|
||||
// The shape a real user creates by opening the assistant with a workflow in front
|
||||
// of them: the product hands the agent a resource reference, so it resolves by id
|
||||
// and never hunts by name.
|
||||
it('sends the attached seed workflow with the opening message, using the REMAPPED id', async () => {
|
||||
const sendMessage = vi.fn().mockResolvedValue({ runId: 'run-1' });
|
||||
const restoreThread = vi
|
||||
.fn()
|
||||
.mockResolvedValue({ restored: 1, workflowIds: ['restored-wf-1'], dataTableIds: [] });
|
||||
await buildWorkflow({
|
||||
client: makeClient(restoreThread, { sendMessage }),
|
||||
...baseConfig,
|
||||
conversation: [
|
||||
{ role: 'user' as const, text: 'why is this failing?', attach: { workflow: SEED_WF_ID } },
|
||||
],
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
const [, , attachments] = sendMessage.mock.calls[0] as [
|
||||
string,
|
||||
string,
|
||||
Array<{ type: string; id: string; name: string }> | undefined,
|
||||
];
|
||||
expect(attachments).toHaveLength(1);
|
||||
expect(attachments?.[0].type).toBe('workflow');
|
||||
// The authored id is rewritten per run, so sending it verbatim would point the
|
||||
// agent at a workflow that doesn't exist on this instance.
|
||||
expect(attachments?.[0].id).not.toBe(SEED_WF_ID);
|
||||
const [, , workflows] = restoreThread.mock.calls[0] as [
|
||||
string,
|
||||
unknown,
|
||||
Array<{ id: string; name: string }>,
|
||||
];
|
||||
expect(attachments?.[0].id).toBe(workflows[0].id);
|
||||
expect(attachments?.[0].name).toBe(workflows[0].name);
|
||||
});
|
||||
|
||||
// The API carries the attachment out of band, so the graded transcript would show a
|
||||
// faithful hand-off (`text: ''` + attach) as a bare empty message: an anomaly to the
|
||||
// judge, and an EMPTY prompt for the prompt-aware checks (userTurnsAsText drops
|
||||
// empty strings). The recorded turn names it instead.
|
||||
it('names the attached workflow in the RECORDED turn, so judges can see the hand-off', async () => {
|
||||
const sendMessage = vi.fn().mockResolvedValue({ runId: 'run-1' });
|
||||
const restoreThread = vi
|
||||
.fn()
|
||||
.mockResolvedValue({ restored: 1, workflowIds: ['restored-wf-1'], dataTableIds: [] });
|
||||
vi.mocked(recordUserTurn).mockClear();
|
||||
|
||||
await buildWorkflow({
|
||||
client: makeClient(restoreThread, { sendMessage }),
|
||||
...baseConfig,
|
||||
// No typed text — exactly the shape the docs promote.
|
||||
conversation: [{ role: 'user' as const, text: '', attach: { workflow: SEED_WF_ID } }],
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
const restoredWorkflows = (
|
||||
restoreThread.mock.calls[0] as [string, unknown, Array<{ id: string; name: string }>]
|
||||
)[2];
|
||||
const [, recordedText] = vi.mocked(recordUserTurn).mock.calls[0];
|
||||
const [, sentText] = sendMessage.mock.calls[0] as [string, string, unknown];
|
||||
|
||||
// The RESTORED (per-run) name, not the authored one — that's what exists on the
|
||||
// instance and what the judge will see referenced.
|
||||
expect(recordedText).toBe(`[attached workflow: ${restoredWorkflows[0].name}]`);
|
||||
// The agent still gets the user's real (empty) text plus the attachment itself.
|
||||
expect(sentText).toBe('');
|
||||
});
|
||||
|
||||
// The proxy renders its script and running transcript from `text` alone, so a
|
||||
// hand-off case with follow-ups would otherwise audit plans and decide follow-ups
|
||||
// against a blank opening turn that never mentions the workflow.
|
||||
it('names the attached workflow in the script the user proxy reads', async () => {
|
||||
const restoreThread = vi
|
||||
.fn()
|
||||
.mockResolvedValue({ restored: 1, workflowIds: ['restored-wf-1'], dataTableIds: [] });
|
||||
proxyScripts.length = 0;
|
||||
|
||||
await buildWorkflow({
|
||||
client: makeClient(restoreThread),
|
||||
...baseConfig,
|
||||
conversation: [
|
||||
{ role: 'user' as const, text: '', attach: { workflow: SEED_WF_ID } },
|
||||
{ role: 'user' as const, text: 'now add error handling' },
|
||||
],
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
const restoredWorkflows = (
|
||||
restoreThread.mock.calls[0] as [string, unknown, Array<{ id: string; name: string }>]
|
||||
)[2];
|
||||
expect(proxyScripts[0][0].text).toBe(`[attached workflow: ${restoredWorkflows[0].name}]`);
|
||||
// Later turns are the author's own text, untouched.
|
||||
expect(proxyScripts[0][1].text).toBe('now add error handling');
|
||||
});
|
||||
|
||||
it('fails loudly when the attached seed workflow is missing from the restore', async () => {
|
||||
// The schema refuses an `attach` the seed does not declare, so a miss here means
|
||||
// the restore/remap lost it. Running on would silently downgrade the case to a
|
||||
// find-it test and grade the wrong thing.
|
||||
const restoreThread = vi
|
||||
.fn()
|
||||
.mockResolvedValue({ restored: 1, workflowIds: ['restored-wf-1'], dataTableIds: [] });
|
||||
|
||||
const result = await buildWorkflow({
|
||||
client: makeClient(restoreThread, { sendMessage: vi.fn().mockResolvedValue({ runId: 'r' }) }),
|
||||
...baseConfig,
|
||||
conversation: [
|
||||
{ role: 'user' as const, text: 'why?', attach: { workflow: 'never-declared' } },
|
||||
],
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
expect(result.error).toMatch(/attaches seeded workflow "never-declared"/);
|
||||
});
|
||||
|
||||
it('sends no attachments when the opening turn declares none', async () => {
|
||||
const sendMessage = vi.fn().mockResolvedValue({ runId: 'run-1' });
|
||||
await buildWorkflow({
|
||||
client: makeClient(
|
||||
vi.fn().mockResolvedValue({ restored: 1, workflowIds: [], dataTableIds: [] }),
|
||||
{ sendMessage },
|
||||
),
|
||||
...baseConfig,
|
||||
seed: { mode: 'inline' as const, ...inlineSeed() },
|
||||
});
|
||||
expect(sendMessage.mock.calls[0][2]).toBeUndefined();
|
||||
});
|
||||
|
||||
it('does not restore anything for a case with no seed', async () => {
|
||||
const restoreThread = vi
|
||||
.fn()
|
||||
|
||||
@@ -213,6 +213,32 @@ describe('remapSeedWorkflowIds', () => {
|
||||
expect(serialized).toContain(`/workflow/${newId}`);
|
||||
});
|
||||
|
||||
it('gives each workflow its own fresh id when one id is a prefix of another', () => {
|
||||
// Rewriting the short id first would eat the long one's prefix, leaving it with
|
||||
// a derived id no later pass matches — so both must come back as clean nanoids.
|
||||
const seed = makeSeed();
|
||||
seed.workflows = [
|
||||
{ id: 'abcdefgh', name: 'Short', nodes: [], connections: {} },
|
||||
{ id: 'abcdefgh12', name: 'Long', nodes: [], connections: {} },
|
||||
];
|
||||
seed.messages = [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2026-01-01T00:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'compare /workflow/abcdefgh and /workflow/abcdefgh12' }],
|
||||
},
|
||||
];
|
||||
|
||||
const remapped = remapSeedWorkflowIds(seed);
|
||||
const [short, long] = remapped.workflows.map((workflow) => workflow.id);
|
||||
|
||||
expect(short).toMatch(/^[0-9A-Za-z]{16}$/);
|
||||
expect(long).toMatch(/^[0-9A-Za-z]{16}$/);
|
||||
expect(JSON.stringify(remapped.messages)).toContain(`/workflow/${short} and /workflow/${long}`);
|
||||
});
|
||||
|
||||
it('returns the seed untouched when there are no workflows', () => {
|
||||
const seed: ConversationSeed = {
|
||||
messages: [
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
import type { CaseSeed } from '../harness/schema';
|
||||
import type { ConversationTurn, TranscriptTurn } from '../types';
|
||||
import {
|
||||
agentTurnsAsText,
|
||||
caseDisplayPrompt,
|
||||
conversationUserTurnsAsText,
|
||||
lastAgentText,
|
||||
perTurnToolCallCounts,
|
||||
@@ -88,6 +90,73 @@ describe('conversationUserTurnsAsText', () => {
|
||||
const conversation: ConversationTurn[] = [{ role: 'assistant', text: 'hello' }];
|
||||
expect(conversationUserTurnsAsText(conversation)).toBe('');
|
||||
});
|
||||
|
||||
// The editor hands the agent a resource reference, not text, so the faithful
|
||||
// hand-off is `text: '' + attach`. Filtered as empty, it would hand the
|
||||
// prompt-aware checks (fulfills-user-request) an EMPTY prompt.
|
||||
it('names an attached workflow so a text-less hand-off is not dropped', () => {
|
||||
const conversation: ConversationTurn[] = [
|
||||
{ role: 'user', text: '', attach: { workflow: 'Batch loop' } },
|
||||
];
|
||||
expect(conversationUserTurnsAsText(conversation)).toBe('[attached workflow: Batch loop]');
|
||||
});
|
||||
|
||||
it('keeps both the attachment and the text when the user typed something', () => {
|
||||
const conversation: ConversationTurn[] = [
|
||||
{ role: 'user', text: 'why is this failing?', attach: { workflow: 'Batch loop' } },
|
||||
];
|
||||
expect(conversationUserTurnsAsText(conversation)).toBe(
|
||||
'[attached workflow: Batch loop] why is this failing?',
|
||||
);
|
||||
});
|
||||
|
||||
// `attach.workflow` is an id; the id means nothing to a prompt-aware check, so
|
||||
// the note carries the name the seed declares for it (what the live path shows).
|
||||
it('names the attachment by the seed workflow name, not its id', () => {
|
||||
const conversation: ConversationTurn[] = [
|
||||
{ role: 'user', text: '', attach: { workflow: 'wKk3RmT9xQ2bVn7L' } },
|
||||
];
|
||||
expect(conversationUserTurnsAsText(conversation, seedDeclaring('Batch loop'))).toBe(
|
||||
'[attached workflow: Batch loop]',
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
/** An inline seed declaring one workflow under the id the tests attach. */
|
||||
function seedDeclaring(name: string): CaseSeed {
|
||||
return {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2020-01-01T00:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'earlier' }],
|
||||
},
|
||||
],
|
||||
workflows: [{ id: 'wKk3RmT9xQ2bVn7L', name, nodes: [], connections: {} }],
|
||||
dataTables: [],
|
||||
};
|
||||
}
|
||||
|
||||
describe('caseDisplayPrompt', () => {
|
||||
it('uses the first authored turn', () => {
|
||||
expect(caseDisplayPrompt({ conversation: [{ role: 'user', text: 'build a webhook' }] })).toBe(
|
||||
'build a webhook',
|
||||
);
|
||||
});
|
||||
|
||||
// Without this the report labels, the comparison table and `Running case: ""`
|
||||
// all come out empty for the faithful hand-off shape.
|
||||
it('names the attachment when the opening turn carries no text', () => {
|
||||
expect(
|
||||
caseDisplayPrompt({
|
||||
conversation: [{ role: 'user', text: '', attach: { workflow: 'wKk3RmT9xQ2bVn7L' } }],
|
||||
seed: seedDeclaring('Batch loop'),
|
||||
}),
|
||||
).toBe('[attached workflow: Batch loop]');
|
||||
});
|
||||
});
|
||||
|
||||
describe('transcriptAsText', () => {
|
||||
|
||||
@@ -215,6 +215,56 @@ describe('EvalTestCaseSchema', () => {
|
||||
expect(inlineSeedOf(parsed).messages[0].createdAt).toBe(authored);
|
||||
});
|
||||
|
||||
// The restamp lands on fixed slots, never `Date.now()`-derived: `createdAt` is
|
||||
// part of the projection `planPush` compares, so a now-based rewrite would read
|
||||
// as an edit on every parse and re-PATCH the case forever.
|
||||
it('restamps a future createdAt to the same value on every parse', () => {
|
||||
const caseJson = {
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2099-01-01T00:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
const first = inlineSeedOf(EvalTestCaseSchema.parse(caseJson)).messages[0].createdAt;
|
||||
const second = inlineSeedOf(EvalTestCaseSchema.parse(caseJson)).messages[0].createdAt;
|
||||
expect(first).toBe(second);
|
||||
expect(Date.parse(String(first))).toBeLessThan(Date.now());
|
||||
});
|
||||
|
||||
// A shorthand turn appended after full envelopes stamps at the fixed epoch,
|
||||
// which is BEFORE their authored stamps — the store would present it first while
|
||||
// `transcriptPrefixFromSeed` still grades array order.
|
||||
it('restamps a mixed seed whose authored timestamps do not ascend', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'user',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'build it' }],
|
||||
},
|
||||
{ role: 'assistant', text: 'Done.' },
|
||||
],
|
||||
},
|
||||
});
|
||||
const at = inlineSeedOf(parsed).messages.map((m) => Date.parse(String(m.createdAt)));
|
||||
expect(at[0]).toBeLessThan(at[1]);
|
||||
expect(at[1]).toBeLessThan(Date.now());
|
||||
});
|
||||
|
||||
// Both arms are strict, so a seed mixing them fails instead of having the
|
||||
// wrong-arm field stripped — which would run the case unseeded and grade it
|
||||
// as a build from scratch.
|
||||
@@ -325,6 +375,163 @@ describe('EvalTestCaseSchema', () => {
|
||||
).toThrow(/full envelope[\s\S]*shorthand/);
|
||||
});
|
||||
|
||||
it('accepts an attach on the opening turn naming a seeded workflow', () => {
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [
|
||||
{ role: 'user', text: 'why is this failing?', attach: { workflow: 'wf12345678' } },
|
||||
],
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
workflows: [{ id: 'wf12345678', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
},
|
||||
});
|
||||
expect(parsed.conversation?.[0].attach).toEqual({ workflow: 'wf12345678' });
|
||||
});
|
||||
|
||||
// An attachment models the user opening the assistant with a workflow already in
|
||||
// front of them, so the turn it rides has to BE the user's. An assistant-first
|
||||
// opener carrying one would be graded against a transcript that never happened.
|
||||
it('rejects an attach on an assistant opening turn', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [
|
||||
{ role: 'assistant', text: 'here is what I built', attach: { workflow: 'wf12345678' } },
|
||||
{ role: 'user', text: 'why is this failing?' },
|
||||
],
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
workflows: [{ id: 'wf12345678', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
},
|
||||
}),
|
||||
).toThrow(/attach.*user|user.*attach/i);
|
||||
});
|
||||
|
||||
it('rejects an attach on a later turn — an attachment is a hand-off', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [
|
||||
{ role: 'user', text: 'why is this failing?' },
|
||||
{ role: 'user', text: 'and now this', attach: { workflow: 'wf12345678' } },
|
||||
],
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
workflows: [{ id: 'wf12345678', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
},
|
||||
}),
|
||||
).toThrow(/only the first conversation turn may carry .attach./);
|
||||
});
|
||||
|
||||
it('rejects an attach naming a workflow the seed does not declare', () => {
|
||||
// A dangling reference hands the agent nothing and reads as a builder failure.
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [{ role: 'user', text: 'why?', attach: { workflow: 'not-in-the-seed' } }],
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
workflows: [{ id: 'wf12345678', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
},
|
||||
}),
|
||||
).toThrow(/must be the id of a workflow the inline seed declares/);
|
||||
});
|
||||
|
||||
it('rejects an empty opening turn that carries no attach', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [{ role: 'user', text: '' }],
|
||||
}),
|
||||
).toThrow(/opening turn with empty text must carry .attach./);
|
||||
});
|
||||
|
||||
it('rejects an empty LATER turn too — the chat API 400s on it just the same', () => {
|
||||
// The guard used to cover only turn 0, so this reached the API mid-run and
|
||||
// surfaced as what reads like an infrastructure fault.
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [
|
||||
{ role: 'user', text: 'build a thing' },
|
||||
{ role: 'user', text: ' ' },
|
||||
],
|
||||
}),
|
||||
).toThrow(/a conversation turn needs text/);
|
||||
});
|
||||
|
||||
it('leaves an empty ASSISTANT turn alone — script data, never posted to chat', () => {
|
||||
// The guard exists for the chat API's 400. Assistant turns are the proxy's
|
||||
// script, so an empty one has nothing to do with that rule.
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [
|
||||
{ role: 'user', text: 'build a thing' },
|
||||
{ role: 'assistant', text: '' },
|
||||
],
|
||||
});
|
||||
expect(parsed.conversation?.[1].text).toBe('');
|
||||
});
|
||||
|
||||
it('does not tell a replay author to add an attach they cannot use', () => {
|
||||
// On a replay case conversation[0] CONTINUES the trace's live turn, and `attach`
|
||||
// needs an inline seed to point at — so the opening-turn advice is a dead end.
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [{ role: 'user', text: '' }],
|
||||
seed: { mode: 'replay', threadId: 'thread-1' },
|
||||
}),
|
||||
).toThrow(/a conversation turn needs text/);
|
||||
});
|
||||
|
||||
it('rejects a seed declaring two workflows with the same id', () => {
|
||||
// Restore index-aligns authored ids with remapped ones and rewrites references
|
||||
// by id, so a duplicate would silently resolve to the wrong workflow.
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
workflows: [
|
||||
{ id: 'wf12345678', name: 'First', nodes: [], connections: {} },
|
||||
{ id: 'wf12345678', name: 'Second', nodes: [], connections: {} },
|
||||
],
|
||||
},
|
||||
}),
|
||||
).toThrow(/seed workflow ids must be unique/);
|
||||
});
|
||||
|
||||
it('accepts an empty opening turn when a seeded workflow is attached', () => {
|
||||
// The faithful hand-off shape: opened on a workflow, nothing typed.
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [{ role: 'user', text: '', attach: { workflow: 'wf12345678' } }],
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [{ role: 'user', text: 'build it' }],
|
||||
workflows: [{ id: 'wf12345678', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
},
|
||||
}),
|
||||
).not.toThrow();
|
||||
});
|
||||
|
||||
it('rejects an attach on a case with no inline seed at all', () => {
|
||||
expect(() =>
|
||||
EvalTestCaseSchema.parse({
|
||||
...validFixture(),
|
||||
conversation: [{ role: 'user', text: 'why?', attach: { workflow: 'wf12345678' } }],
|
||||
}),
|
||||
).toThrow(/must be the id of a workflow the inline seed declares/);
|
||||
});
|
||||
|
||||
it('accepts a replay seed with no conversation (live turn from the trace)', () => {
|
||||
const { conversation: _omit, ...rest } = validFixture();
|
||||
const parsed = EvalTestCaseSchema.parse({
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { WorkflowTestCaseWithFile } from '../data/workflows';
|
||||
import { planPush, toUpdatePatch } from '../langtracer/push';
|
||||
import { comparableDiff, planPush, toUpdatePatch } from '../langtracer/push';
|
||||
import type { LangTracerCreateCaseBody } from '../langtracer/to-exported';
|
||||
|
||||
function item(fileSlug: string, overrides: Record<string, unknown> = {}): WorkflowTestCaseWithFile {
|
||||
@@ -15,6 +15,25 @@ function item(fileSlug: string, overrides: Record<string, unknown> = {}): Workfl
|
||||
} as WorkflowTestCaseWithFile;
|
||||
}
|
||||
|
||||
/** The authored durable seed, in the shape both the disk case and the export carry. */
|
||||
function inlineSeed(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
mode: 'inline' as const,
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'assistant' as const,
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
workflows: [{ id: 'wKk3RmT9xQ2bVn7L', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
dataTables: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
/** A disk-shape exported body (what `GET /suites/:id/export` returns per case). */
|
||||
function body(overrides: Record<string, unknown> = {}): Record<string, unknown> {
|
||||
return {
|
||||
@@ -54,14 +73,165 @@ describe('planPush', () => {
|
||||
expect(plan.unchanged).toEqual([]);
|
||||
});
|
||||
|
||||
it('skips a case that uses an unsupported seeding mode', () => {
|
||||
it('skips a replay-seeded case — its trace expires, so it has no suite home', () => {
|
||||
const plan = planPush([item('c', { seed: { mode: 'replay', threadId: 't' } })], {}, {});
|
||||
expect(plan.skipped).toHaveLength(1);
|
||||
expect(plan.skipped[0].fileSlug).toBe('c');
|
||||
expect(plan.skipped[0].reason).toMatch(/seed \(mode: replay\)/);
|
||||
expect(plan.skipped[0].reason).toMatch(/replay seed/);
|
||||
expect(plan.toCreate).toEqual([]);
|
||||
});
|
||||
|
||||
it('PUSHES an inline-seeded case — the durable kind is exactly what suites are for', () => {
|
||||
const plan = planPush([item('repair-it', { seed: inlineSeed() })], {}, {});
|
||||
expect(plan.skipped).toEqual([]);
|
||||
expect(plan.toCreate).toHaveLength(1);
|
||||
});
|
||||
|
||||
// `seed` joined COMPARED_KEYS for these three: while it was excluded, every
|
||||
// seed-only edit was classified `unchanged` and silently never pushed.
|
||||
it('treats a seed-only addition to a hosted case as an update', () => {
|
||||
const plan = planPush([item('c', { seed: inlineSeed() })], { 'c.json': body() }, { c: 5 });
|
||||
expect(plan.toUpdate.map((u) => u.id)).toEqual([5]);
|
||||
expect(plan.unchanged).toEqual([]);
|
||||
});
|
||||
|
||||
it('treats a seed EDIT on a hosted case as an update', () => {
|
||||
const plan = planPush(
|
||||
[item('c', { seed: inlineSeed({ dataTables: [{ name: 'Orders', columns: [] }] }) })],
|
||||
{ 'c.json': body({ seed: inlineSeed() }) },
|
||||
{ c: 5 },
|
||||
);
|
||||
expect(plan.toUpdate.map((u) => u.id)).toEqual([5]);
|
||||
expect(plan.unchanged).toEqual([]);
|
||||
});
|
||||
|
||||
it('treats a seed REMOVAL as an update, so the patch can clear the stored one', () => {
|
||||
const plan = planPush([item('c')], { 'c.json': body({ seed: inlineSeed() }) }, { c: 5 });
|
||||
expect(plan.toUpdate.map((u) => u.id)).toEqual([5]);
|
||||
expect(plan.unchanged).toEqual([]);
|
||||
});
|
||||
|
||||
// Shorthand is the shape case-shapes.md promotes, and expansion mints a fresh `id`
|
||||
// on every parse. Comparing ids would classify the case `toUpdate` on EVERY push
|
||||
// forever, leaving `--dry-run` permanently dirty. The identical-envelope test
|
||||
// can't catch it, because there both sides share one envelope.
|
||||
it('converges a shorthand-authored seed against its expanded stored export', () => {
|
||||
// Shorthand mints a fresh `id` per parse; its `createdAt` is deterministic, so
|
||||
// the two sides differ only in the id.
|
||||
const disk = inlineSeed({
|
||||
messages: [
|
||||
{
|
||||
id: 'freshly-minted-uuid',
|
||||
createdAt: '2020-01-01T00:00:00.000Z',
|
||||
role: 'assistant',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
const stored = inlineSeed({
|
||||
messages: [
|
||||
{
|
||||
id: 'stored-uuid',
|
||||
createdAt: '2020-01-01T00:00:00.000Z',
|
||||
role: 'assistant',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const plan = planPush(
|
||||
[item('c', { seed: disk })],
|
||||
{ 'c.json': body({ seed: stored }) },
|
||||
{ c: 5 },
|
||||
);
|
||||
|
||||
expect(plan.unchanged.map((c) => c.fileSlug)).toEqual(['c']);
|
||||
expect(plan.toUpdate).toEqual([]);
|
||||
});
|
||||
|
||||
it('still detects a real seed edit under a differing id', () => {
|
||||
const disk = inlineSeed({
|
||||
messages: [
|
||||
{
|
||||
id: 'a',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
role: 'assistant',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'DIFFERENT text' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
const stored = inlineSeed({
|
||||
messages: [
|
||||
{
|
||||
id: 'b',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
role: 'assistant',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const plan = planPush(
|
||||
[item('c', { seed: disk })],
|
||||
{ 'c.json': body({ seed: stored }) },
|
||||
{ c: 5 },
|
||||
);
|
||||
|
||||
expect(plan.toUpdate.map((u) => u.id)).toEqual([5]);
|
||||
expect(plan.unchanged).toEqual([]);
|
||||
});
|
||||
|
||||
// cubic's P2: `createdAt` drives restore ordering, so an authored envelope's
|
||||
// timestamp edit changes what the agent sees and MUST reach the suite. Dropping
|
||||
// it from the comparison alongside `id` would have hidden that.
|
||||
it('detects an authored createdAt edit, which reorders the restored history', () => {
|
||||
const disk = inlineSeed({
|
||||
messages: [
|
||||
{
|
||||
id: 'same-id',
|
||||
createdAt: '2026-07-01T12:00:00.000Z',
|
||||
role: 'assistant',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
const stored = inlineSeed({
|
||||
messages: [
|
||||
{
|
||||
id: 'same-id',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
role: 'assistant',
|
||||
type: 'llm',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const plan = planPush(
|
||||
[item('c', { seed: disk })],
|
||||
{ 'c.json': body({ seed: stored }) },
|
||||
{ c: 5 },
|
||||
);
|
||||
|
||||
expect(plan.toUpdate.map((u) => u.id)).toEqual([5]);
|
||||
expect(plan.unchanged).toEqual([]);
|
||||
});
|
||||
|
||||
it('still reports an identically-seeded case as unchanged', () => {
|
||||
const plan = planPush(
|
||||
[item('c', { seed: inlineSeed() })],
|
||||
{ 'c.json': body({ seed: inlineSeed() }) },
|
||||
{ c: 5 },
|
||||
);
|
||||
expect(plan.unchanged.map((c) => c.fileSlug)).toEqual(['c']);
|
||||
expect(plan.toUpdate).toEqual([]);
|
||||
});
|
||||
|
||||
it('treats a scenario-only difference as an update (PATCH reconciles scenarios by name)', () => {
|
||||
const plan = planPush(
|
||||
[
|
||||
@@ -227,4 +397,56 @@ describe('toUpdatePatch', () => {
|
||||
const patch = toUpdatePatch(createBody({ scenarios }));
|
||||
expect(patch.scenarios).toEqual(scenarios);
|
||||
});
|
||||
|
||||
it('sends an explicit null seed when the case has none, so a PATCH clears a stored one', () => {
|
||||
// lang-tracer treats an omitted `seed` as a no-op, so dropping the seed from a
|
||||
// disk case could never un-seed the hosted case without this.
|
||||
const patch = toUpdatePatch(createBody());
|
||||
expect(patch.seed).toBeNull();
|
||||
});
|
||||
|
||||
it('keeps the seed when the case still has one', () => {
|
||||
const seed = inlineSeed();
|
||||
const patch = toUpdatePatch(createBody({ seed }));
|
||||
expect(patch.seed).toEqual(seed);
|
||||
});
|
||||
});
|
||||
|
||||
// The post-write check the push runs after every create/update. A lang-tracer
|
||||
// deployment predating a field's support ignores the key and still answers 200
|
||||
// (`seed` needs #113, `attach` needs #119), so only re-reading the export can tell
|
||||
// you the suite holds what you authored.
|
||||
describe('comparableDiff (post-write verification)', () => {
|
||||
it('reports nothing when the server stored everything', () => {
|
||||
expect(
|
||||
comparableDiff(body({ seed: inlineSeed() }), item('c', { seed: inlineSeed() }).testCase),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it('names `seed` when a pre-#113 server dropped it', () => {
|
||||
const written = item('c', { seed: inlineSeed() }).testCase;
|
||||
expect(comparableDiff(body(), written)).toEqual(['seed']);
|
||||
});
|
||||
|
||||
it('names `conversation` when a pre-#119 server dropped `attach`', () => {
|
||||
const handoff = [{ role: 'user', text: '', attach: { workflow: 'wKk3RmT9xQ2bVn7L' } }];
|
||||
const written = item('c', { conversation: handoff, seed: inlineSeed() }).testCase;
|
||||
// What such a server gives back: the turn, minus the attachment.
|
||||
const stored = body({ conversation: [{ role: 'user', text: '' }], seed: inlineSeed() });
|
||||
|
||||
expect(comparableDiff(stored, written)).toEqual(['conversation']);
|
||||
});
|
||||
|
||||
it('names every dropped field, not just the first', () => {
|
||||
const handoff = [{ role: 'user', text: '', attach: { workflow: 'wKk3RmT9xQ2bVn7L' } }];
|
||||
const written = item('c', { conversation: handoff, seed: inlineSeed() }).testCase;
|
||||
const stored = body({ conversation: [{ role: 'user', text: '' }] });
|
||||
|
||||
expect(comparableDiff(stored, written).sort()).toEqual(['conversation', 'seed']);
|
||||
});
|
||||
|
||||
it('treats a case missing from the export as everything dropped', () => {
|
||||
const written = item('c', { seed: inlineSeed() }).testCase;
|
||||
expect(comparableDiff(undefined, written)).toContain('seed');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { EvalTestCaseInput } from '../harness/schema';
|
||||
import { EvalTestCaseSchema, type EvalTestCaseInput } from '../harness/schema';
|
||||
import { normalizeExportedCase } from '../langtracer/normalize';
|
||||
import { diskCaseToLangTracerCreate, unsupportedPushReason } from '../langtracer/to-exported';
|
||||
|
||||
/** A minimal schema-parsed disk case (conversation text already collapsed to a string). */
|
||||
@@ -110,15 +111,12 @@ describe('unsupportedPushReason', () => {
|
||||
expect(unsupportedPushReason(diskCase())).toBeNull();
|
||||
});
|
||||
|
||||
// Keyed off the discriminant, so a new mode is flagged without editing the
|
||||
// mapper — the point of the one-slot union.
|
||||
it('flags a replay seed as unsupported, naming the mode', () => {
|
||||
it('flags a replay seed as unsupported — its trace expires, so it has no suite home', () => {
|
||||
const reason = unsupportedPushReason(diskCase({ seed: { mode: 'replay', threadId: 't' } }));
|
||||
expect(reason).toMatch(/seed/);
|
||||
expect(reason).toMatch(/replay/);
|
||||
expect(reason).toMatch(/replay seed/);
|
||||
});
|
||||
|
||||
it('flags an inline seed as unsupported, naming the mode', () => {
|
||||
it('ALLOWS an inline seed — a durable fixture the write API stores verbatim', () => {
|
||||
const reason = unsupportedPushReason(
|
||||
diskCase({
|
||||
seed: {
|
||||
@@ -137,7 +135,117 @@ describe('unsupportedPushReason', () => {
|
||||
},
|
||||
}),
|
||||
);
|
||||
expect(reason).toMatch(/seed/);
|
||||
expect(reason).toMatch(/inline/);
|
||||
expect(reason).toBeNull();
|
||||
});
|
||||
|
||||
it('carries the inline seed into the create body verbatim', () => {
|
||||
const seed = {
|
||||
mode: 'inline' as const,
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'assistant' as const,
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
workflows: [{ id: 'wKk3RmT9xQ2bVn7L', name: 'Batch loop', nodes: [], connections: {} }],
|
||||
dataTables: [],
|
||||
};
|
||||
const body = diskCaseToLangTracerCreate(diskCase({ seed }), 'repair-it', {
|
||||
suiteId: 1,
|
||||
setKind: 'regression',
|
||||
synthetic: true,
|
||||
});
|
||||
expect(body.seed).toEqual(seed);
|
||||
});
|
||||
|
||||
it('omits the seed key entirely for an unseeded case', () => {
|
||||
const body = diskCaseToLangTracerCreate(diskCase(), 'plain', {
|
||||
suiteId: 1,
|
||||
setKind: 'regression',
|
||||
synthetic: true,
|
||||
});
|
||||
expect('seed' in body).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// A suite is only a safe home for a hand-off case if `attach` survives the whole
|
||||
// loop — push, store, export, reparse. It needs lang-tracer #119 deployed to carry
|
||||
// the key; the last case here is what a pre-#119 deployment gives back, and it must
|
||||
// fail loudly rather than run as a quietly different (find-it) test.
|
||||
describe('attach round-trip: write → export → reparse', () => {
|
||||
const WORKFLOW_ID = 'wKk3RmT9xQ2bVn7L';
|
||||
|
||||
function handoffCase(): EvalTestCaseInput {
|
||||
return diskCase({
|
||||
// The faithful editor hand-off: no typed text, a workflow attached.
|
||||
conversation: [{ role: 'user', text: '', attach: { workflow: WORKFLOW_ID } }],
|
||||
seed: {
|
||||
mode: 'inline',
|
||||
messages: [
|
||||
{
|
||||
id: 'm1',
|
||||
type: 'llm',
|
||||
role: 'assistant',
|
||||
createdAt: '2026-06-29T09:00:00.000Z',
|
||||
content: [{ type: 'text', text: 'built it' }],
|
||||
},
|
||||
],
|
||||
workflows: [{ id: WORKFLOW_ID, name: 'Batch loop', nodes: [], connections: {} }],
|
||||
dataTables: [],
|
||||
},
|
||||
} as Partial<EvalTestCaseInput>);
|
||||
}
|
||||
|
||||
/** What `GET /suites/:id/export` returns: disk shape plus export-only keys. */
|
||||
function exportedFrom(conversation: unknown, seed: unknown) {
|
||||
return {
|
||||
id: 42,
|
||||
name: 'handoff',
|
||||
suiteId: 1,
|
||||
createdAt: '2026-08-04T00:00:00.000Z',
|
||||
conversation,
|
||||
seed,
|
||||
complexity: 'simple',
|
||||
tags: ['build'],
|
||||
datasets: ['full'],
|
||||
processExpectations: ['acknowledges the workflow it was handed'],
|
||||
};
|
||||
}
|
||||
|
||||
it('carries attach into the create body', () => {
|
||||
const body = diskCaseToLangTracerCreate(handoffCase(), 'handoff', {
|
||||
suiteId: 1,
|
||||
setKind: 'regression',
|
||||
synthetic: true,
|
||||
});
|
||||
|
||||
expect(body.conversation).toEqual([
|
||||
{ role: 'user', text: '', attach: { workflow: WORKFLOW_ID } },
|
||||
]);
|
||||
});
|
||||
|
||||
it('reparses from the export with the attachment intact', () => {
|
||||
const body = diskCaseToLangTracerCreate(handoffCase(), 'handoff', {
|
||||
suiteId: 1,
|
||||
setKind: 'regression',
|
||||
synthetic: true,
|
||||
});
|
||||
|
||||
const parsed = EvalTestCaseSchema.parse(
|
||||
normalizeExportedCase(exportedFrom(body.conversation, body.seed)),
|
||||
);
|
||||
|
||||
expect(parsed.conversation?.[0].attach).toEqual({ workflow: WORKFLOW_ID });
|
||||
});
|
||||
|
||||
it('fails at load when the deployment stripped attach, instead of running as a find-it case', () => {
|
||||
const stripped = exportedFrom([{ role: 'user', text: '' }], handoffCase().seed);
|
||||
|
||||
expect(() => EvalTestCaseSchema.parse(normalizeExportedCase(stripped))).toThrow(
|
||||
/opening turn with empty text must carry/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -9,7 +9,10 @@ const NO_AGENT_OUTPUT_REASON =
|
||||
'not judged — the build produced no agent output, so there was nothing to grade';
|
||||
|
||||
export interface SelectAuthorExpectationsArgs {
|
||||
testCase: Pick<WorkflowTestCase, 'processExpectations' | 'outcomeExpectations' | 'conversation'>;
|
||||
testCase: Pick<
|
||||
WorkflowTestCase,
|
||||
'processExpectations' | 'outcomeExpectations' | 'conversation' | 'seed'
|
||||
>;
|
||||
/** Captured build transcript, if any. Empty/absent for prebuilt/MCP builds. */
|
||||
transcript: TranscriptTurn[] | undefined;
|
||||
buildSucceeded: boolean;
|
||||
@@ -64,7 +67,12 @@ export function selectAuthorExpectations(args: SelectAuthorExpectationsArgs): {
|
||||
|
||||
const transcript: TranscriptTurn[] = hasAgentOutput
|
||||
? args.transcript!
|
||||
: [{ userMessage: conversationUserTurnsAsText(testCase.conversation), steps: [] }];
|
||||
: [
|
||||
{
|
||||
userMessage: conversationUserTurnsAsText(testCase.conversation, testCase.seed),
|
||||
steps: [],
|
||||
},
|
||||
];
|
||||
|
||||
// A failed build that produced nothing at all: record every expectation as
|
||||
// ungraded rather than handing the judge an empty conversation to describe.
|
||||
|
||||
@@ -11,10 +11,10 @@ import { execFileSync } from 'node:child_process';
|
||||
import { basename } from 'node:path';
|
||||
|
||||
import { loadAgentEvalTestCasesWithFiles } from '../data/agents';
|
||||
import { loadWorkflowTestCasesWithFiles } from '../data/workflows';
|
||||
import { loadWorkflowTestCasesWithFiles, type WorkflowTestCaseWithFile } from '../data/workflows';
|
||||
import { LangTracerClient } from '../langtracer/client';
|
||||
import { resolveLangTracerConfig } from '../langtracer/config';
|
||||
import { planPush, toUpdatePatch } from '../langtracer/push';
|
||||
import { comparableDiff, planPush, toUpdatePatch } from '../langtracer/push';
|
||||
import { diskCaseToLangTracerCreate } from '../langtracer/to-exported';
|
||||
|
||||
interface CliArgs {
|
||||
@@ -252,11 +252,52 @@ async function main() {
|
||||
console.log(` ~ updated ${item.fileSlug} (#${String(id)}, rev ${String(res.revision)})`);
|
||||
}
|
||||
|
||||
await verifyWrites(client, suite.id, [...plan.toCreate, ...plan.toUpdate.map((u) => u.item)]);
|
||||
|
||||
console.log(
|
||||
`\nDone: ${String(plan.toCreate.length)} created, ${String(plan.toUpdate.length)} updated, ${String(plan.unchanged.length)} unchanged, ${String(plan.skipped.length)} skipped.`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-read the suite and confirm the server stored what we sent.
|
||||
*
|
||||
* A lang-tracer deployment predating a field's support ignores that key and still
|
||||
* answers 200 — `seed` before #113, `attach` before #119. Without this the push
|
||||
* reports success while the suite holds a quietly different case: a seeded case
|
||||
* that will run unseeded, or a hand-off that became a find-it test. Both are
|
||||
* deploy-ordering hazards no local check can catch, so ask the server.
|
||||
*/
|
||||
async function verifyWrites(
|
||||
client: LangTracerClient,
|
||||
suiteId: number,
|
||||
written: WorkflowTestCaseWithFile[],
|
||||
): Promise<void> {
|
||||
if (written.length === 0) return;
|
||||
|
||||
const after = await client.exportSuite(suiteId);
|
||||
const dropped = written
|
||||
.map((item) => ({
|
||||
fileSlug: item.fileSlug,
|
||||
keys: comparableDiff(after.files[`${item.fileSlug}.json`], item.testCase),
|
||||
}))
|
||||
.filter((result) => result.keys.length > 0);
|
||||
|
||||
if (dropped.length === 0) {
|
||||
console.log(` verified ${String(written.length)} case(s) round-trip intact`);
|
||||
return;
|
||||
}
|
||||
|
||||
for (const result of dropped) {
|
||||
console.error(` ! ${result.fileSlug}: server did not store ${result.keys.join(', ')}`);
|
||||
}
|
||||
throw new Error(
|
||||
`${String(dropped.length)} case(s) did not round-trip: the fields above were sent but are absent from the suite export. ` +
|
||||
'A lang-tracer deployment can silently ignore a key it predates (`seed` needs #113, `attach` needs #119) — ' +
|
||||
'upgrade it, then re-push. The cases in the suite are NOT what you authored until you do.',
|
||||
);
|
||||
}
|
||||
|
||||
function printBucket(label: string, slugs: string[]): void {
|
||||
const pad = label.padEnd(9);
|
||||
console.log(` ${pad} ${String(slugs.length)}${slugs.length ? ` (${slugs.join(', ')})` : ''}`);
|
||||
|
||||
@@ -16,6 +16,7 @@ import type {
|
||||
InstanceAiThreadStatusResponse,
|
||||
InstanceAiEvalSeedDataTable,
|
||||
InstanceAiEvalSeedWorkflow,
|
||||
InstanceAiWorkflowAttachment,
|
||||
AgentJsonConfig,
|
||||
AgentSkill,
|
||||
EvaluationConfigDto,
|
||||
@@ -212,12 +213,20 @@ export class N8nClient {
|
||||
|
||||
/**
|
||||
* Send a chat message to the instance-ai agent.
|
||||
* POST /rest/instance-ai/chat/:threadId body: { message }
|
||||
* POST /rest/instance-ai/chat/:threadId body: { message, attachments? }
|
||||
*
|
||||
* `attachments` are resource references the agent resolves with its tools — the
|
||||
* same channel the editor uses when a user opens the assistant with a workflow
|
||||
* in front of them, so the agent is handed it by id instead of hunting by name.
|
||||
*/
|
||||
async sendMessage(threadId: string, message: string): Promise<{ runId: string }> {
|
||||
async sendMessage(
|
||||
threadId: string,
|
||||
message: string,
|
||||
attachments?: InstanceAiWorkflowAttachment[],
|
||||
): Promise<{ runId: string }> {
|
||||
const result = await this.fetch(`/rest/instance-ai/chat/${threadId}`, {
|
||||
method: 'POST',
|
||||
body: { message },
|
||||
body: attachments && attachments.length > 0 ? { message, attachments } : { message },
|
||||
});
|
||||
return result as { runId: string };
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
// execution and cleanup.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
import type { InstanceAiConfirmRequest } from '@n8n/api-types';
|
||||
import type { InstanceAiConfirmRequest, InstanceAiWorkflowAttachment } from '@n8n/api-types';
|
||||
import crypto from 'node:crypto';
|
||||
import { setTimeout as delay } from 'node:timers/promises';
|
||||
|
||||
@@ -59,6 +59,7 @@ import type {
|
||||
} from '../types';
|
||||
import {
|
||||
agentTurnsAsText,
|
||||
attachedWorkflowNote,
|
||||
failedBuildsPerTurn,
|
||||
lastAgentText,
|
||||
userTurnsAsText,
|
||||
@@ -98,6 +99,10 @@ interface MultiTurnDriverConfig {
|
||||
/** Appended to the FIRST sent message only (pre-seeded-table hint); the
|
||||
* recorded turn and the proxy's conversation keep the clean prompt. */
|
||||
openingMessageSuffix?: string;
|
||||
/** What the RECORDED opening turn says, when it must differ from what's sent —
|
||||
* an out-of-band attachment has to be named in the transcript the judge reads.
|
||||
* Defaults to the sent text. */
|
||||
recordedOpeningMessage?: string;
|
||||
/** Ids already allowlisted for this thread (from pre-run `createDeclaredCredentials`
|
||||
* seeding) — wires `UserProxyLlm.credentialCreation` so `manual` can create a
|
||||
* real credential when a setup card shows zero existing candidates. Omitted
|
||||
@@ -108,15 +113,25 @@ interface MultiTurnDriverConfig {
|
||||
/** Shared with `createDeclaredCredentials`'s pre-run seeding — see
|
||||
* `CredentialCreationConfig.nameCounts`. */
|
||||
credentialNameCounts?: Map<string, number>;
|
||||
/** Resource references sent with the FIRST message only — an attachment is a
|
||||
* hand-off, not something a user re-sends every turn. */
|
||||
openingAttachments?: InstanceAiWorkflowAttachment[];
|
||||
}
|
||||
|
||||
async function driveMultiTurnConversation(
|
||||
config: MultiTurnDriverConfig,
|
||||
): Promise<ProxyDecisionStats> {
|
||||
const openingMessage = config.conversation[0]?.text ?? '';
|
||||
const recordedOpeningMessage = config.recordedOpeningMessage ?? openingMessage;
|
||||
// The proxy renders both its script and its running transcript from `text` alone,
|
||||
// so it needs the recorded opening too — otherwise it audits every plan and
|
||||
// follow-up against a blank turn that never mentions the workflow.
|
||||
const proxyConversation = config.conversation.map((turn, index) =>
|
||||
index === 0 ? { ...turn, text: recordedOpeningMessage } : turn,
|
||||
);
|
||||
|
||||
const proxy = new UserProxyLlm({
|
||||
conversation: config.conversation,
|
||||
conversation: proxyConversation,
|
||||
messageBudget: config.messageBudget,
|
||||
logger: config.logger,
|
||||
...(config.allowlistedCredentialIds !== undefined
|
||||
@@ -143,10 +158,11 @@ async function driveMultiTurnConversation(
|
||||
return decision;
|
||||
};
|
||||
|
||||
recordUserTurn(config.events, openingMessage);
|
||||
recordUserTurn(config.events, recordedOpeningMessage);
|
||||
await config.client.sendMessage(
|
||||
config.threadId,
|
||||
openingMessage + (config.openingMessageSuffix ?? ''),
|
||||
config.openingAttachments,
|
||||
);
|
||||
|
||||
await runMultiTurnConversation({
|
||||
@@ -318,6 +334,9 @@ export async function buildWorkflow(config: BuildWorkflowConfig): Promise<BuildR
|
||||
let builtDataTableIds: string[] = [];
|
||||
let seededTranscript: TranscriptTurn[] = [];
|
||||
let seedingFailed = false;
|
||||
// Seed-declared workflow id -> the workflow as actually restored (fresh id and
|
||||
// name). Lets an authored `attach` reference survive the per-run remap.
|
||||
let seedWorkflowsBySeedId = new Map<string, { id: string; name: string }>();
|
||||
|
||||
try {
|
||||
const buildStart = Date.now();
|
||||
@@ -416,6 +435,12 @@ export async function buildWorkflow(config: BuildWorkflowConfig): Promise<BuildR
|
||||
if (seed) {
|
||||
try {
|
||||
const remapped = remapSeedWorkflowIds(seed);
|
||||
// The remap preserves order, so index-align the authored ids with the per-run
|
||||
// ones. An author writes the id the seed declares; an attachment has to carry
|
||||
// the id that actually exists on the instance.
|
||||
seedWorkflowsBySeedId = new Map(
|
||||
seed.workflows.map((workflow, index) => [workflow.id, remapped.workflows[index]]),
|
||||
);
|
||||
await evictLeftoverSeedWorkflows(
|
||||
client,
|
||||
remapped,
|
||||
@@ -504,6 +529,34 @@ export async function buildWorkflow(config: BuildWorkflowConfig): Promise<BuildR
|
||||
|
||||
await delay(SSE_SETTLE_DELAY_MS);
|
||||
|
||||
// The opening turn may hand the agent a seeded workflow, the way the editor does.
|
||||
// Resolved AFTER the restore so it carries the id that exists on the instance;
|
||||
// the case schema already refused an `attach` no seeded workflow declares.
|
||||
const attachedSeedWorkflow = conversation[0]?.attach?.workflow;
|
||||
const restoredForAttach =
|
||||
attachedSeedWorkflow === undefined
|
||||
? undefined
|
||||
: seedWorkflowsBySeedId.get(attachedSeedWorkflow);
|
||||
// The schema already refused an `attach` no seeded workflow declares, so a miss
|
||||
// here means the restore/remap dropped it. Fail loudly: sending no attachment
|
||||
// would silently downgrade a hand-off case to a find-it one.
|
||||
if (attachedSeedWorkflow !== undefined && restoredForAttach === undefined) {
|
||||
seedingFailed = true;
|
||||
throw new Error(
|
||||
`The opening turn attaches seeded workflow "${attachedSeedWorkflow}", but the restore produced no workflow for that id — refusing to run the case unattached (it would silently become a find-it test).`,
|
||||
);
|
||||
}
|
||||
const openingAttachments: InstanceAiWorkflowAttachment[] | undefined = restoredForAttach
|
||||
? [{ type: 'workflow', id: restoredForAttach.id, name: restoredForAttach.name }]
|
||||
: undefined;
|
||||
// Name the out-of-band attachment in the RECORDED turn, or the judge and the
|
||||
// prompt-aware checks read a text-less hand-off as a bare empty message — see
|
||||
// `attachedWorkflowNote`. Mirrors `openingMessageSuffix`, which diverges
|
||||
// sent-vs-recorded the other way.
|
||||
const recordedOpeningMessage = [attachedWorkflowNote(restoredForAttach?.name), openingMessage]
|
||||
.filter(Boolean)
|
||||
.join(' ');
|
||||
|
||||
let proxyDecisionStats: ProxyDecisionStats | undefined;
|
||||
if (isMultiTurn) {
|
||||
proxyDecisionStats = await driveMultiTurnConversation({
|
||||
@@ -531,10 +584,16 @@ export async function buildWorkflow(config: BuildWorkflowConfig): Promise<BuildR
|
||||
// The pre-seeded-table note goes to the agent, but the recorded turn
|
||||
// (and the graded transcript) keeps the clean user prompt.
|
||||
openingMessageSuffix: scenarioSeedTablesNote,
|
||||
openingAttachments,
|
||||
recordedOpeningMessage,
|
||||
});
|
||||
} else {
|
||||
recordUserTurn(events, openingMessage);
|
||||
await client.sendMessage(threadId, openingMessage + scenarioSeedTablesNote);
|
||||
recordUserTurn(events, recordedOpeningMessage);
|
||||
await client.sendMessage(
|
||||
threadId,
|
||||
openingMessage + scenarioSeedTablesNote,
|
||||
openingAttachments,
|
||||
);
|
||||
await waitForAllActivity({
|
||||
client,
|
||||
threadId,
|
||||
|
||||
@@ -123,8 +123,18 @@ export const ConversationSeedSchema = z.object({
|
||||
source: z.record(z.unknown()).optional(),
|
||||
/** Native agent message log (user/assistant turns with resolved tool-call blocks). */
|
||||
messages: z.array(SeedMessageSchema).min(1),
|
||||
/** Workflows the history references, recreated on restore. */
|
||||
workflows: z.array(SeedWorkflowSchema).default([]),
|
||||
/** Workflows the history references, recreated on restore. Ids must be distinct:
|
||||
* the restore index-aligns authored ids with their per-run remapped ones, and
|
||||
* `remapSeedWorkflowIds` rewrites references by sequential `replaceAll` — a
|
||||
* duplicate would collapse both to one entry and one fresh id, so an `attach`
|
||||
* or a message reference would point at the wrong workflow. */
|
||||
workflows: z
|
||||
.array(SeedWorkflowSchema)
|
||||
.default([])
|
||||
.refine(
|
||||
(workflows) => new Set(workflows.map((workflow) => workflow.id)).size === workflows.length,
|
||||
{ message: 'seed workflow ids must be unique — references resolve by id' },
|
||||
),
|
||||
/** Data tables the history references, recreated (and id-rewritten) on restore. */
|
||||
dataTables: z.array(SeedDataTableSchema).default([]),
|
||||
});
|
||||
@@ -153,17 +163,27 @@ function isShorthandTurn(
|
||||
);
|
||||
}
|
||||
|
||||
/** Base for every timestamp this module stamps. A CONSTANT, not `Date.now()`: the
|
||||
* same case must yield the same messages on every parse, or the push diff (which
|
||||
* compares `seed`) reads a fresh timestamp as an edit and re-PATCHes the case on
|
||||
* every run forever. Only ordering depends on these values, and fixed past slots
|
||||
* order exactly as well. */
|
||||
const SEED_EPOCH_MS = Date.parse('2020-01-01T00:00:00.000Z');
|
||||
|
||||
/** The slot for the message at `index` — ascending, and in the past, so array
|
||||
* order survives a store that presents messages by `createdAt`. */
|
||||
const seedStampAt = (index: number) => new Date(SEED_EPOCH_MS + index * 1000).toISOString();
|
||||
|
||||
/**
|
||||
* Expand `{role, text}` shorthand messages into native llm envelopes; anything
|
||||
* else passes through for the envelope schema to validate.
|
||||
*
|
||||
* Array-level rather than per-message because the stamped timestamps ascend by
|
||||
* position — slightly in the past, so seeded history always orders before the
|
||||
* live turn and a shorthand author cannot get the ordering wrong. A full
|
||||
* envelope keeps its own authored `createdAt`.
|
||||
* position, so seeded history always orders before the live turn and a shorthand
|
||||
* author cannot get the ordering wrong. A full envelope keeps its own authored
|
||||
* `createdAt` — see `normalizeSeedTimestamps` for when that is overridden.
|
||||
*/
|
||||
export function expandSeedMessageShorthand(messages: unknown[]): unknown[] {
|
||||
const base = Date.now() - (messages.length + 1) * 1000;
|
||||
return messages.map((message, index) => {
|
||||
if (!isShorthandTurn(message)) return message;
|
||||
return {
|
||||
@@ -178,48 +198,47 @@ export function expandSeedMessageShorthand(messages: unknown[]): unknown[] {
|
||||
text: Array.isArray(message.text) ? message.text.join('\n') : message.text,
|
||||
},
|
||||
],
|
||||
createdAt: new Date(base + index * 1000).toISOString(),
|
||||
createdAt: seedStampAt(index),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull an inline seed's timestamps back into the past when the author put any of
|
||||
* them in the future.
|
||||
* Restamp an inline seed's timestamps when the authored ones don't present the
|
||||
* messages the way the array orders them: ascending, and entirely before the live
|
||||
* turn. Only a full envelope can get this wrong (the shorthand stamps its own
|
||||
* slots), and either failure makes the agent see its own history out of order.
|
||||
*
|
||||
* The shorthand stamps its own ascending pre-live timestamps, so it can't get
|
||||
* this wrong; a full envelope keeps what it was authored with, and a future
|
||||
* stamp sorts the seeded turn AFTER the live turn — the agent then sees its own
|
||||
* history out of order, and the judge grades a transcript that never happened.
|
||||
* Restamps the WHOLE sequence, not just the offending entry: a per-message fix
|
||||
* reorders relative to the array (`[future A, past B]` moves only A, leaving the
|
||||
* store presenting B then A while the transcript still grades array order).
|
||||
* Authored timestamps therefore survive only when already ascending and past.
|
||||
*
|
||||
* Restamps the WHOLE sequence, not just the offending entry. A per-message clamp
|
||||
* reorders relative to the array: `[future A, past B]` leaves B alone and moves A
|
||||
* to ~now, so the store presents B then A while `transcriptPrefixFromSeed` still
|
||||
* grades array order. Array order is the authority, so the rewrite reproduces it
|
||||
* on the same ascending slots the shorthand uses. Authored timestamps are
|
||||
* therefore only preserved when every one of them is already in the past.
|
||||
*
|
||||
* Inline (hand-authored) seeds only — a `replay` seed is reconstructed from a
|
||||
* real trace and never reaches this schema, so no real timestamp can be moved.
|
||||
* Inline seeds only — a `replay` seed is reconstructed from a real trace and never
|
||||
* reaches this schema, so no real timestamp can be moved.
|
||||
*/
|
||||
export function clampFutureSeedTimestamps(messages: unknown[]): unknown[] {
|
||||
const now = Date.now();
|
||||
export function normalizeSeedTimestamps(messages: unknown[]): unknown[] {
|
||||
const stampOf = (message: unknown): number | undefined => {
|
||||
if (!isRecord(message) || typeof message.createdAt !== 'string') return undefined;
|
||||
const at = Date.parse(message.createdAt);
|
||||
// Unparseable is the envelope schema's error to report, not ours to paper over.
|
||||
return Number.isNaN(at) ? undefined : at;
|
||||
};
|
||||
const anyFuture = messages.some((message) => (stampOf(message) ?? -Infinity) >= now);
|
||||
if (!anyFuture) return messages;
|
||||
|
||||
const base = now - (messages.length + 1) * 1000;
|
||||
const now = Date.now();
|
||||
let previous = -Infinity;
|
||||
const presentsInArrayOrder = messages.every((message) => {
|
||||
const at = stampOf(message);
|
||||
if (at === undefined) return true;
|
||||
if (at >= now || at <= previous) return false;
|
||||
previous = at;
|
||||
return true;
|
||||
});
|
||||
if (presentsInArrayOrder) return messages;
|
||||
|
||||
return messages.map((message, index) => {
|
||||
if (stampOf(message) === undefined) return message;
|
||||
return {
|
||||
...(message as Record<string, unknown>),
|
||||
createdAt: new Date(base + index * 1000).toISOString(),
|
||||
};
|
||||
if (!isRecord(message) || stampOf(message) === undefined) return message;
|
||||
return { ...message, createdAt: seedStampAt(index) };
|
||||
});
|
||||
}
|
||||
|
||||
@@ -305,7 +324,11 @@ export function remapSeedWorkflowIds(seed: ConversationSeed): ConversationSeed {
|
||||
|
||||
const originalIds = new Set(seed.workflows.map((workflow) => workflow.id));
|
||||
let serialized = JSON.stringify({ messages: seed.messages, workflows: seed.workflows });
|
||||
for (const workflow of seed.workflows) {
|
||||
// Longest id first, for the same reason the name pass below sorts: if one id were a
|
||||
// prefix of another ("abcdefgh" / "abcdefgh12"), rewriting the short one first would
|
||||
// eat the long one's prefix and leave it with a derived id no later pass matches.
|
||||
const byLongestId = [...seed.workflows].sort((a, b) => b.id.length - a.id.length);
|
||||
for (const workflow of byLongestId) {
|
||||
// Workflow ids are long random tokens; a short id would risk rewriting
|
||||
// unrelated substrings, so refuse instead of corrupting the seed.
|
||||
if (workflow.id.length < 8) {
|
||||
|
||||
@@ -4,8 +4,8 @@ import { z } from 'zod';
|
||||
import {
|
||||
ConversationSeedSchema,
|
||||
SeedMessageSchema,
|
||||
clampFutureSeedTimestamps,
|
||||
expandSeedMessageShorthand,
|
||||
normalizeSeedTimestamps,
|
||||
} from './conversation-seed';
|
||||
import { SUPPORTED_CREDENTIAL_TYPES } from '../credentials/seeder';
|
||||
|
||||
@@ -33,6 +33,17 @@ export const conversationTurnTextSchema = z
|
||||
export const ConversationTurnSchema = z.object({
|
||||
role: z.enum(['user', 'assistant']),
|
||||
text: conversationTurnTextSchema,
|
||||
/** Hand the agent a seeded workflow with this turn, the way the editor does when
|
||||
* a user opens the assistant with a workflow in front of them — without it the
|
||||
* eval agent has to guess which workflow prose like "why is this failing?"
|
||||
* means, and we score a clarification failure the real user never hit.
|
||||
*
|
||||
* `workflow` is the id as the seed declares it; the harness swaps in the
|
||||
* per-run remapped id. Opening turn only (refined below). */
|
||||
attach: z
|
||||
.object({ workflow: z.string().min(1) })
|
||||
.strict()
|
||||
.optional(),
|
||||
});
|
||||
|
||||
const ExecutionScenarioSchema = z.object({
|
||||
@@ -52,10 +63,10 @@ const ExecutionScenarioSchema = z.object({
|
||||
/** Prior messages for an inline seed. Accepts a full envelope or the `{role, text}`
|
||||
* shorthand, expanded BEFORE validation so the envelope rules apply to the
|
||||
* expansion and error paths stay per-message (`seed.messages.2.createdAt`).
|
||||
* Future `createdAt` values are pulled back after expansion, so seeded history
|
||||
* can never sort after the live turn. */
|
||||
* Timestamps are normalized after expansion, so seeded history always presents
|
||||
* in array order and never sorts after the live turn. */
|
||||
const inlineSeedMessagesSchema = z.preprocess(
|
||||
(raw) => (Array.isArray(raw) ? clampFutureSeedTimestamps(expandSeedMessageShorthand(raw)) : raw),
|
||||
(raw) => (Array.isArray(raw) ? normalizeSeedTimestamps(expandSeedMessageShorthand(raw)) : raw),
|
||||
z.array(SeedMessageSchema).min(1),
|
||||
);
|
||||
|
||||
@@ -188,6 +199,51 @@ export const EvalTestCaseSchema = evalTestCaseObjectSchema
|
||||
message:
|
||||
'a case needs a conversation, or a seed with mode: replay (which supplies the live turn from the trace)',
|
||||
})
|
||||
// Rejected rather than ignored on a later turn, so a misplaced one can't silently
|
||||
// do nothing.
|
||||
.refine((c) => (c.conversation ?? []).slice(1).every((turn) => turn.attach === undefined), {
|
||||
message: 'only the first conversation turn may carry `attach` — an attachment is a hand-off',
|
||||
})
|
||||
// Grading an assistant turn that carries one would score a transcript that could
|
||||
// not have happened.
|
||||
.refine((c) => c.conversation?.[0]?.attach === undefined || c.conversation[0].role === 'user', {
|
||||
message:
|
||||
'only a `user` turn may carry `attach` — the attachment is the hand-off that opens the conversation',
|
||||
})
|
||||
// A dangling attachment would hand the agent a reference to nothing, which reads
|
||||
// as a builder failure. Only an inline seed declares workflows to point at.
|
||||
.refine(
|
||||
(c) => {
|
||||
const attached = c.conversation?.[0]?.attach?.workflow;
|
||||
if (attached === undefined) return true;
|
||||
const declared = c.seed?.mode === 'inline' ? c.seed.workflows : [];
|
||||
return declared.some((workflow) => workflow.id === attached);
|
||||
},
|
||||
{
|
||||
message:
|
||||
'`attach.workflow` must be the id of a workflow the inline seed declares — otherwise the attachment points at nothing',
|
||||
},
|
||||
)
|
||||
// The chat API refuses a message that is empty with nothing attached, so catch it
|
||||
// at load rather than mid-run as a 400 that reads like an infrastructure fault.
|
||||
// Every user turn, not just the opening; assistant turns are proxy script data
|
||||
// and never posted. Only a non-replay opening may substitute `attach` for text —
|
||||
// a replay case has no inline seed for it to point at.
|
||||
.superRefine((c, ctx) => {
|
||||
const isReplay = c.seed?.mode === 'replay';
|
||||
(c.conversation ?? []).forEach((turn, index) => {
|
||||
if (turn.role === 'assistant' || turn.text.trim().length > 0) return;
|
||||
const openingMayAttach = index === 0 && !isReplay;
|
||||
if (openingMayAttach && turn.attach !== undefined) return;
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ['conversation', index, 'text'],
|
||||
message: openingMayAttach
|
||||
? 'an opening turn with empty text must carry `attach` — the chat API rejects a message that is empty with nothing attached'
|
||||
: 'a conversation turn needs text — the chat API rejects an empty message, and only a non-replay opening turn may substitute `attach`',
|
||||
});
|
||||
});
|
||||
})
|
||||
.superRefine((c, ctx) => {
|
||||
// Note: this message avoids double quotes — ZodError.message is a JSON.stringify of
|
||||
// the issue list, which would otherwise backslash-escape them and break substring/regex
|
||||
|
||||
@@ -37,8 +37,12 @@ export type LangTracerCaseRef = z.infer<typeof caseRefSchema>;
|
||||
* by name on PATCH (upsert + delete missing); a server predating lang-tracer #48
|
||||
* strips the key silently, leaving the old scenarios in place. */
|
||||
export type LangTracerUpdateCaseBody = Partial<
|
||||
Omit<LangTracerCreateCaseBody, 'suiteId' | 'synthetic'>
|
||||
>;
|
||||
Omit<LangTracerCreateCaseBody, 'suiteId' | 'synthetic' | 'seed'>
|
||||
> & {
|
||||
/** Explicit `null` CLEARS a stored seed. An omitted key is a server-side no-op,
|
||||
* so a disk case that drops its seed needs the null to take effect. */
|
||||
seed?: LangTracerCreateCaseBody['seed'] | null;
|
||||
};
|
||||
|
||||
export class LangTracerClient {
|
||||
constructor(private readonly config: LangTracerConfig) {}
|
||||
|
||||
@@ -35,19 +35,24 @@ const COMPARED_KEYS = [
|
||||
// Round-trips faithfully: PATCH /cases/:id reconciles scenario rows by name
|
||||
// (lang-tracer #48) and the export emits them back in disk shape.
|
||||
'executionScenarios',
|
||||
// Stored at `metadata.seed` and emitted back by the export (lang-tracer #113).
|
||||
// Compared so a seed-only edit isn't misread as unchanged and left unpushed.
|
||||
'seed',
|
||||
] as const;
|
||||
|
||||
/** Drop the create-only fields, leaving the patchable set (`scenarios` included —
|
||||
* `PATCH /cases/:id` reconciles them by name since lang-tracer #48). An absent
|
||||
* `scenarios` is sent as an explicit `[]`: a partial PATCH leaves missing keys
|
||||
* untouched, so omitting it would keep the server's old scenario rows alive
|
||||
* forever after a disk case drops its `executionScenarios`. */
|
||||
* `scenarios` is sent as an explicit `[]` and an absent `seed` as an explicit
|
||||
* `null`: a partial PATCH leaves missing keys untouched, so omitting them would
|
||||
* keep the server's old scenario rows / stored seed alive forever after a disk
|
||||
* case drops its `executionScenarios` / `seed`. Both defaults sit before the
|
||||
* spread, so a case that still has them overrides. */
|
||||
export function toUpdatePatch({
|
||||
suiteId,
|
||||
synthetic,
|
||||
...patch
|
||||
}: LangTracerCreateCaseBody): LangTracerUpdateCaseBody {
|
||||
return { scenarios: [], ...patch };
|
||||
return { scenarios: [], seed: null, ...patch };
|
||||
}
|
||||
|
||||
/** `existingBodies`: `<name>.json` → exported (disk-shape) body from `GET /suites/:id/export`.
|
||||
@@ -86,9 +91,16 @@ export function planPush(
|
||||
/** Compare only the PATCH-able disk fields of an exported body against a disk case,
|
||||
* after folding lang-tracer's export-only keys / legacy `buildExpectations`. */
|
||||
function sameComparableFields(existingRaw: unknown, diskTestCase: unknown): boolean {
|
||||
return comparableDiff(existingRaw, diskTestCase).length === 0;
|
||||
}
|
||||
|
||||
/** Which comparable fields differ between an exported body and a disk case. Same
|
||||
* rules as the create/update/unchanged split, so a post-write check can reuse them
|
||||
* to name exactly what a server failed to store. */
|
||||
export function comparableDiff(existingRaw: unknown, diskTestCase: unknown): string[] {
|
||||
const existing = projectComparable(normalizeExportedCase(existingRaw));
|
||||
const disk = projectComparable(diskTestCase);
|
||||
return canonicalize(existing) === canonicalize(disk);
|
||||
return COMPARED_KEYS.filter((key) => canonicalize(existing[key]) !== canonicalize(disk[key]));
|
||||
}
|
||||
|
||||
function projectComparable(src: unknown): Record<string, unknown> {
|
||||
@@ -116,11 +128,38 @@ function projectComparable(src: unknown): Record<string, unknown> {
|
||||
out[key] = datasets;
|
||||
continue;
|
||||
}
|
||||
// Shorthand expansion stamps a fresh `randomUUID()` per parse, so comparing
|
||||
// message ids can never converge — the case would re-PATCH on every push
|
||||
// forever. Ids carry no meaning to a case, so they're dropped here.
|
||||
// `createdAt` is NOT dropped: restore ordering depends on it, so an authored
|
||||
// envelope's timestamp edit must still register. Shorthand's own timestamps
|
||||
// are deterministic (see SHORTHAND_SEED_EPOCH_MS), so they converge anyway.
|
||||
if (key === 'seed') {
|
||||
out[key] = seedWithoutMessageIds(value);
|
||||
continue;
|
||||
}
|
||||
out[key] = value;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Drop message `id`s from a seed before comparing: shorthand expansion mints a
|
||||
* new one per parse, so keeping them would make a shorthand-authored case differ
|
||||
* from its stored export forever. Everything else the author wrote — role,
|
||||
* content, `createdAt`, workflows, data tables — still compares. */
|
||||
function seedWithoutMessageIds(value: unknown): unknown {
|
||||
if (value === null || typeof value !== 'object') return value;
|
||||
const seed: Record<string, unknown> = { ...(value as Record<string, unknown>) };
|
||||
const messages: unknown = seed.messages;
|
||||
if (!Array.isArray(messages)) return seed;
|
||||
seed.messages = (messages as unknown[]).map((message) => {
|
||||
if (message === null || typeof message !== 'object') return message;
|
||||
const { id, ...rest } = message as Record<string, unknown>;
|
||||
return rest;
|
||||
});
|
||||
return seed;
|
||||
}
|
||||
|
||||
/** Stable JSON with sorted object keys, so field/scenario ordering never affects equality. */
|
||||
function canonicalize(value: unknown): string {
|
||||
return JSON.stringify(sortKeysDeep(value));
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
// lang-tracer REST `POST /api/v1/cases` (create_test_case) expects. Split from the
|
||||
// network call so the disk→API key-renaming contract is unit-testable without a server.
|
||||
|
||||
import type { EvalTestCaseInput } from '../harness/schema';
|
||||
import type { CaseSeed, EvalTestCaseInput } from '../harness/schema';
|
||||
|
||||
/** One scenario in the create-case payload (`executionScenarios` renamed to `scenarios`). */
|
||||
export interface LangTracerScenario {
|
||||
@@ -21,7 +21,14 @@ export interface LangTracerCreateCaseBody {
|
||||
synthetic: boolean;
|
||||
suiteId: number;
|
||||
description?: string;
|
||||
conversation?: Array<{ role: 'user' | 'assistant'; text: string }>;
|
||||
/** `attach` is declared, not just tolerated: the turn shape is the push contract,
|
||||
* and leaving it off let a hand-off case type-check while losing its attachment.
|
||||
* Carrying it end-to-end needs lang-tracer #119 deployed. */
|
||||
conversation?: Array<{
|
||||
role: 'user' | 'assistant';
|
||||
text: string;
|
||||
attach?: { workflow: string };
|
||||
}>;
|
||||
evalComplexity: 'simple' | 'medium' | 'complex';
|
||||
evalTags: string[];
|
||||
evalTriggerType?: string;
|
||||
@@ -31,6 +38,10 @@ export interface LangTracerCreateCaseBody {
|
||||
datasets?: string[];
|
||||
messageBudget?: number;
|
||||
credentials?: Array<{ type: string; name?: string }>;
|
||||
/** Inline seed, forwarded verbatim — lang-tracer stores it at `metadata.seed`.
|
||||
* Only the authored arm: a replay seed is derived from a source thread by
|
||||
* promote/scrub over there, so pushing one would fabricate provenance. */
|
||||
seed?: Extract<CaseSeed, { mode: 'inline' }>;
|
||||
}
|
||||
|
||||
export interface ToLangTracerOptions {
|
||||
@@ -39,12 +50,30 @@ export interface ToLangTracerOptions {
|
||||
synthetic: boolean;
|
||||
}
|
||||
|
||||
/** The case-write API has no `seed` field, so no seeded case can be pushed yet —
|
||||
* keyed off the discriminant so a new mode needs no edit here. Returns a
|
||||
/** Seeding modes the case-write API can't take. An INLINE seed is pushable — it's a
|
||||
* durable fixture, and the API stores it verbatim. A REPLAY seed isn't: it points at
|
||||
* a LangSmith trace that expires, lang-tracer derives it from a source thread it
|
||||
* already holds, and such a case is barred from suites anyway. Returns a
|
||||
* human-readable reason, else null. */
|
||||
export function unsupportedPushReason(testCase: EvalTestCaseInput): string | null {
|
||||
if (!testCase.seed) return null;
|
||||
return `uses seed (mode: ${testCase.seed.mode}) — not supported by the case-write API`;
|
||||
const seed = testCase.seed;
|
||||
switch (seed?.mode) {
|
||||
case undefined:
|
||||
case 'inline':
|
||||
return null;
|
||||
case 'replay':
|
||||
return (
|
||||
'uses a replay seed — reconstructed from a LangSmith trace at run time, so it has no ' +
|
||||
'durable home in a suite. Derive a synthetic case from it instead.'
|
||||
);
|
||||
default: {
|
||||
// A new arm must decide its own push-ability here. Approving by default
|
||||
// would push the case while `diskCaseToLangTracerCreate` forwards only
|
||||
// `inline` — landing it in the suite stripped of its seed.
|
||||
const unhandled: never = seed;
|
||||
throw new Error(`Unhandled seed mode: ${JSON.stringify(unhandled)}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Map a schema-parsed disk case to a create-case body. `conversation.text` is already
|
||||
@@ -78,6 +107,8 @@ export function diskCaseToLangTracerCreate(
|
||||
if (testCase.datasets !== undefined) body.datasets = testCase.datasets;
|
||||
if (testCase.messageBudget !== undefined) body.messageBudget = testCase.messageBudget;
|
||||
if (testCase.credentials !== undefined) body.credentials = testCase.credentials;
|
||||
// Replay never reaches here — `unsupportedPushReason` skips those cases upstream.
|
||||
if (testCase.seed?.mode === 'inline') body.seed = testCase.seed;
|
||||
|
||||
return body;
|
||||
}
|
||||
|
||||
@@ -438,7 +438,7 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
|
||||
if (build.success && !build.workflowChecks) {
|
||||
build.workflowChecks = await runWorkflowChecks({
|
||||
workflow: build.workflowJsons[0],
|
||||
prompt: conversationUserTurnsAsText(entry.conversation ?? []),
|
||||
prompt: conversationUserTurnsAsText(entry.conversation ?? [], entry.seed),
|
||||
agentText: undefined,
|
||||
logger,
|
||||
});
|
||||
@@ -467,10 +467,13 @@ export function createBuildOrchestrator(deps: BuildOrchestratorDeps): BuildOrche
|
||||
// No transcript in prebuilt mode, but the authored conversation still
|
||||
// carries the user's request — feed it so prompt-aware checks (e.g.
|
||||
// fulfills_user_request) grade against real intent instead of "".
|
||||
const conversation = testCaseByFileSlug.get(fileSlug)?.conversation ?? [];
|
||||
const prebuiltCase = testCaseByFileSlug.get(fileSlug);
|
||||
build.workflowChecks = await runWorkflowChecks({
|
||||
workflow: build.workflowJsons[0],
|
||||
prompt: conversationUserTurnsAsText(conversation),
|
||||
prompt: conversationUserTurnsAsText(
|
||||
prebuiltCase?.conversation ?? [],
|
||||
prebuiltCase?.seed,
|
||||
),
|
||||
agentText: undefined,
|
||||
logger,
|
||||
});
|
||||
|
||||
@@ -198,6 +198,11 @@ export interface ExecutionScenario {
|
||||
export interface ConversationTurn {
|
||||
role: 'user' | 'assistant';
|
||||
text: string;
|
||||
/** Hand the agent a seeded workflow with this turn (opening turn only), the way
|
||||
* the editor does when a user opens the assistant with a workflow in front of
|
||||
* them. `workflow` is the id as the seed declares it; the harness swaps in the
|
||||
* per-run remapped id. See `ConversationTurnSchema`. */
|
||||
attach?: { workflow: string };
|
||||
}
|
||||
|
||||
export interface TestCaseCredential {
|
||||
|
||||
@@ -3,10 +3,39 @@ import { isRecord } from '@n8n/utils/is-record';
|
||||
import type { CaseSeed } from '../harness/schema';
|
||||
import type { ConversationTurn, ToolInteraction, TranscriptStep, TranscriptTurn } from '../types';
|
||||
|
||||
/** Render a turn's out-of-band workflow attachment for a transcript/prompt, e.g.
|
||||
* `[attached workflow: Batch loop]`, or '' when it has none. The editor hands the
|
||||
* agent a resource reference rather than text, so without this the faithful
|
||||
* hand-off shape (`text: ""` + `attach`) reaches judges and prompt-aware checks
|
||||
* as an empty message.
|
||||
*
|
||||
* `label` is the workflow's NAME — the restored one where the harness knows it
|
||||
* (the live path), else the name the seed declares for that id
|
||||
* (`attachedWorkflowLabel`). An id would mean nothing to a prompt-aware check or
|
||||
* to a human reading the report. */
|
||||
export function attachedWorkflowNote(label: string | undefined): string {
|
||||
return label ? `[attached workflow: ${label}]` : '';
|
||||
}
|
||||
|
||||
/** The name a seed declares for an attached workflow id. The authored-conversation
|
||||
* path has only the id, and the seed is where that id gets its name; falls back
|
||||
* to the id when the seed can't resolve it, so the hand-off stays visible. */
|
||||
function attachedWorkflowLabel(
|
||||
turn: ConversationTurn | undefined,
|
||||
seed: CaseSeed | undefined,
|
||||
): string | undefined {
|
||||
const id = turn?.attach?.workflow;
|
||||
if (id === undefined) return undefined;
|
||||
const declared = seed?.mode === 'inline' ? seed.workflows.find((w) => w.id === id) : undefined;
|
||||
return declared?.name ?? id;
|
||||
}
|
||||
|
||||
/**
|
||||
* Human-readable prompt label for a test case. Authored cases use their first
|
||||
* turn; a `replay` seed carries no authored conversation, so fall back to the
|
||||
* live (non-seeded) user turn captured in the transcript, then to the thread id.
|
||||
* A text-less hand-off (`text: ""` + `attach`) has no prompt text at all, so it
|
||||
* falls back to naming the attachment — otherwise it labels as '' in every report.
|
||||
*/
|
||||
export function caseDisplayPrompt(
|
||||
testCase: { conversation?: ConversationTurn[]; seed?: CaseSeed },
|
||||
@@ -17,7 +46,8 @@ export function caseDisplayPrompt(
|
||||
const liveTurn = transcript?.find((t) => !t.seeded && t.userMessage)?.userMessage;
|
||||
if (liveTurn) return liveTurn;
|
||||
const { seed } = testCase;
|
||||
return seed?.mode === 'replay' ? `[seeded] thread ${seed.threadId.slice(0, 8)}` : '';
|
||||
if (seed?.mode === 'replay') return `[seeded] thread ${seed.threadId.slice(0, 8)}`;
|
||||
return attachedWorkflowNote(attachedWorkflowLabel(testCase.conversation?.[0], seed));
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -42,12 +72,20 @@ export function userTurnsAsText(transcript: TranscriptTurn[]): string {
|
||||
*
|
||||
* Accepts `undefined` because `testCase.conversation` is optional (a `replay`-seeded
|
||||
* case carries none) and callers pass it straight through — no conversation → ''.
|
||||
* `seed` resolves an attachment's id to its declared name.
|
||||
*/
|
||||
export function conversationUserTurnsAsText(conversation: ConversationTurn[] | undefined): string {
|
||||
export function conversationUserTurnsAsText(
|
||||
conversation: ConversationTurn[] | undefined,
|
||||
seed?: CaseSeed,
|
||||
): string {
|
||||
if (!conversation) return '';
|
||||
const turns = conversation
|
||||
.filter((t) => t.role === 'user')
|
||||
.map((t) => t.text)
|
||||
// Name an attachment, so a text-less hand-off isn't filtered out below and
|
||||
// handed to the prompt-aware checks as an empty prompt.
|
||||
.map((t) =>
|
||||
[attachedWorkflowNote(attachedWorkflowLabel(t, seed)), t.text].filter(Boolean).join(' '),
|
||||
)
|
||||
.filter((text) => text.length > 0);
|
||||
|
||||
if (turns.length === 0) return '';
|
||||
|
||||
Reference in New Issue
Block a user