test: Add Instance AI flight-status-change workflow eval (#33228)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-28 17:22:01 +08:00 · 2026-06-30 08:55:40 +02:00
parent a068f3b57b
commit b67ef426e8
6 changed files with 300 additions and 0 deletions
@@ -0,0 +1,154 @@
+---
+name: n8n:create-instance-ai-eval
+description: >-
+  Authors a new Instance AI workflow eval case, writing intent-driven outcome
+  expectations and execution scenarios and calibrating them against a real build
+  run. Use when adding a test case to packages/@n8n/instance-ai/evaluations/data/workflows,
+  or when the user asks to create/add an Instance AI workflow eval.
+---
+
+# Create an Instance AI workflow eval
+
+Each eval is **one JSON file** in
+`packages/@n8n/instance-ai/evaluations/data/workflows/`. The loader
+auto-discovers `*.json` and validates against
+[`schema.ts`](../../../packages/@n8n/instance-ai/evaluations/data/workflows/schema.ts)
+(`.strict()` — unknown keys fail at load). No registration step.
+
+The core principle: **write expectations from intent, then calibrate against a
+real build.** Decide up front what makes *any* correct solution correct (the
+must-haves implied by the prompt), then build the workflow once for real to
+calibrate granularity — loosen what's over-specified, confirm the must-haves are
+achievable, and catch requirements the agent legitimately satisfies a different
+way. Do **not** transcribe one observed build into assertions: that overfits the
+eval into "did the agent reproduce that run" instead of "did it solve the
+problem." See the eval
+[README](../../../packages/@n8n/instance-ai/evaluations/README.md) for the full
+field reference.
+
+## Workflow
+
+1. **State the must-haves first.** From the prompt alone, list what every
+   correct workflow must do (trigger type, the essential operations, the gating
+   condition). These become draft `outcomeExpectations`. Required case fields:
+   `conversation` (≥1 turn, first `user`), `executionScenarios` (≥1),
+   `complexity`, `tags`.
+2. **Draft the case** from the template below; validate it loads (see
+   "Validate") before running.
+3. **Run it once** (see "Run locally") with `--keep-workflows` so the built
+   workflow stays on the instance for inspection.
+4. **Inspect the built workflow** — fetch it via `GET /rest/workflows/<id>` (the
+   run prints `BUILT (<id>)`) and read nodes, parameters, and connections.
+5. **Calibrate** the expectations against what you saw: relax any assertion the
+   build satisfied a valid-but-different way, tighten any that a wrong build
+   would have slipped past, and phrase `executionScenarios`
+   (`dataSetup` → `successCriteria`) to match how the workflow runs on mocked
+   data. Keep the must-haves; adjust only their granularity.
+6. **Re-run to confirm** the calibrated expectations pass. Optionally
+   `--iterations 5` to measure flakiness before adding the case to a tighter
+   tier (`datasets: ["pr", ...]`).
+
+## Template
+
+```json
+{
+  "description": "What this case tests.",
+  "conversation": [
+    { "role": "user", "text": "<the build prompt>" }
+  ],
+  "complexity": "medium",
+  "tags": ["build", "<nodes>", "<concepts>"],
+  "triggerType": "schedule",
+  "datasets": ["full"],
+  "outcomeExpectations": [
+    "<a must-have condition any correct workflow satisfies>"
+  ],
+  "executionScenarios": [
+    {
+      "name": "happy-path",
+      "description": "<what this run exercises>",
+      "dataSetup": "<input + what mocked services return>",
+      "successCriteria": "<observable proof the run succeeded>"
+    }
+  ]
+}
+```
+
+For a clarifying-question / multi-turn case, add `assistant` reference turns and
+a `user` stage direction in `[square brackets]` (proxy behaviour, never sent to
+the builder), e.g. `[Don't mention the channel unless asked; then say 'Slack
+#growth.']`. `processExpectations` judge the conversation; `outcomeExpectations`
+judge the workflow. Both count as pass-rate units.
+
+## Sizing each assertion
+
+An assertion is right-sized when **every correct build passes it and a wrong or
+lazy build fails it**. Quick check — the *substitution test*: would a reasonable
+*alternative* implementation still pass? If no, it's too tight; if a
+non-solution would also pass, it's too loose. Examples for the flight-status
+case:
+
+| Verdict | Assertion | Why |
+|---|---|---|
+| ❌ too tight | "Has an HTTP Request node calling `flightaware.com`" | Overfits to one run; a valid build using AeroDataBox fails. Vendor was never the requirement. |
+| ❌ too tight | "Sends the alert via a Gmail node" | The channel was unspecified; Slack/email/etc. are all correct. |
+| ❌ too loose | "Fetches flight data from somewhere" | A workflow that fetches but never compares would pass — doesn't prove change-detection. |
+| ✅ right | "Fetches current status from an external source via an HTTP Request node" | Any correct build passes; one that hard-codes a status fails. |
+| ✅ right | "Persists the previously-seen status and compares it to the freshly-fetched one" | The defining behaviour; substitution-proof across vendors and storage choices. |
+| ✅ right | "Alert is sent only on the change-detected branch, gated by a conditional" | Proves the gate without pinning the node or channel. |
+
+Put vendor-/channel-specific *intent* in `processExpectations` (judged from the
+conversation), not in `outcomeExpectations`.
+
+## Robust assertions vs harness flakiness
+
+Two different things — keep them apart:
+
+- **Robust assertion design (always do this).** The agent's choices vary run to
+  run — whether it asks a clarifying question, which vendor/source it picks.
+  Source-agnostic `outcomeExpectations` aren't a concession to flakiness; they
+  are the *correct* assertion, because the vendor was never a requirement. Use
+  the substitution test above.
+- **Harness flakiness (a defect — surface and mitigate, don't accept).** The
+  mock layer doesn't always honor `dataSetup` for state-bearing reads (e.g. a
+  Data Table "previous value"), so change-detection scenarios can fail with
+  `[mock_issue]`. A randomly-passing eval is worthless. When you hit this:
+  narrow `dataSetup` to steer the mock, move the fragile intent to a
+  `processExpectation`, or keep the scenario out of tight tiers (`["full"]`
+  only) — and note it in the case `description`. Don't ship a scenario whose
+  pass/fail is noise.
+
+## Negative execution scenarios
+
+Don't stop at the happy path. A workflow that only works when everything goes
+right is under-tested. Add scenarios for the unhappy paths the trigger/source
+implies, and assert the *graceful* behaviour:
+
+- **No data / not-found** — source returns empty or 404 → workflow completes
+  without sending a false alert.
+- **Source error / timeout** — source returns 5xx or times out → workflow does
+  not crash and does not emit a spurious "changed" alert.
+- **Malformed response** — unexpected shape → handled without throwing.
+
+Phrase `successCriteria` as the *absence* of the wrong action ("no alert is
+sent", "run completes without error") as much as the presence of the right one.
+If the mock layer can't reliably produce the failing input, that's harness
+flakiness — see above.
+
+## Validate
+
+```bash
+cd packages/@n8n/instance-ai
+npx tsx -e "import {loadWorkflowTestCasesWithFiles} from './evaluations/data/workflows/index.ts'; console.log(loadWorkflowTestCasesWithFiles('<slug>')[0].fileSlug)"
+```
+
+## Run locally
+
+Needs a live n8n with Instance AI enabled (not a default instance): a model key,
+and a working sandbox. See
+[setup notes](reference.md) for the exact env combo (the README's quick-start
+omits the sandbox-auth pieces). Then, from `packages/@n8n/instance-ai`:
+
+```bash
+pnpm eval:instance-ai --filter <slug> --keep-workflows --verbose
+```
@@ -0,0 +1,53 @@
+# Running the eval locally — setup notes
+
+The eval harness builds the workflow on a **live** n8n instance with Instance AI
+enabled, then mocks execution and verifies. Getting a local build to actually
+run needs more than the README quick-start states. Three blockers, in order:
+
+1. **Sandbox auth.** If the instance runs with
+   `N8N_INSTANCE_AI_SANDBOX_ENABLED=true` + `SANDBOX_PROVIDER=daytona`, every
+   build crashes with `DaytonaAuthManager requires exactly one of staticApiKey
+   or getAuthToken` unless Daytona auth is supplied. Direct mode needs
+   `DAYTONA_API_KEY` + `DAYTONA_API_URL`. (Proxy-vended Daytona tokens via the
+   AI-assistant staging proxy may not engage even with a valid license — prefer
+   direct mode.)
+2. **Force direct sandbox mode.** Direct Daytona mode is only chosen when
+   `isProxyEnabled()` is false (`= isAiAssistantEnabled() && aiAssistant.baseUrl`).
+   Boot with `N8N_AI_ASSISTANT_BASE_URL=` (empty) to force it.
+3. **Orchestrator LLM key.** With the proxy off, the build LLM uses
+   `@ai-sdk/anthropic`, which reads `ANTHROPIC_API_KEY` from env —
+   `N8N_AI_ANTHROPIC_KEY` alone is **not** wired into the non-proxy model client.
+   Set `ANTHROPIC_API_KEY` to the same value.
+
+## Working recipe
+
+Keep machine-specific secrets (Daytona + Anthropic keys, sandbox flags) in a
+local env file, e.g. `.env.eval` at the repo root — **gitignore it**.
+
+```bash
+# 1. boot the instance (E2E_TESTS exposes /rest/e2e/reset for owner seeding)
+E2E_TESTS=true N8N_AI_ASSISTANT_BASE_URL= ANTHROPIC_API_KEY="$ANTH" \
+  npx dotenvx run -f .env.local -f .env.eval -- pnpm dev:ai
+
+# 2. seed the owner (the e2e reset route registers a few seconds after
+#    /healthz returns — retry; default owner nathan@n8n.io / PlaywrightTest123)
+curl -sf -X POST http://localhost:5678/rest/e2e/reset -H "Content-Type: application/json" -d '{"owner":{...}}'
+
+# 3. run the eval
+cd packages/@n8n/instance-ai
+N8N_INSTANCE_AI_MODEL=anthropic/claude-sonnet-4-6 \
+N8N_EVAL_EMAIL=nathan@n8n.io N8N_EVAL_PASSWORD=PlaywrightTest123 \
+  npx dotenvx run -f ../../../.env.local -f ../../../.env.eval -- \
+  pnpm eval:instance-ai --filter <slug> --keep-workflows --verbose
+```
+
+Gotchas:
+
+- `dotenvx` is only on PATH via `npx dotenvx`; the bare `dotenvx` command fails.
+- The eval helper (mock-gen, verifier, user-proxy, expectation judge) also needs
+  the Anthropic key; it reads `N8N_AI_ANTHROPIC_KEY` or `ANTHROPIC_API_KEY` and
+  defaults the model to `anthropic/claude-sonnet-4-6`.
+- Inspect a built workflow with `GET /rest/workflows/<id>` (authenticate via the
+  `N8nClient` in `evaluations/clients/n8n-client.ts`, or any logged-in session).
+- A full build is ~60–180s; the sandbox image is built on first run, so the
+  first build is slower.
@@ -0,0 +1 @@
+../../../../.agents/skills/create-instance-ai-eval
@@ -0,0 +1,36 @@
+# Example env for running Instance AI workflow evals locally.
+#
+# Copy to `.env.eval` (gitignored) and fill in real values:
+#   cp .env.eval.example .env.eval
+#
+# Load it alongside .env.local when booting the instance and running the eval:
+#   E2E_TESTS=true N8N_AI_ASSISTANT_BASE_URL= ANTHROPIC_API_KEY="$ANTHROPIC_KEY" \
+#     npx dotenvx run -f .env.local -f .env.eval -- pnpm dev:ai
+#   cd packages/@n8n/instance-ai && \
+#     npx dotenvx run -f ../../../.env.local -f ../../../.env.eval -- \
+#     pnpm eval:instance-ai --filter <slug> --keep-workflows --verbose
+#
+# See .agents/skills/create-instance-ai-eval/reference.md for the full recipe
+# and packages/@n8n/instance-ai/evaluations/README.md for the harness docs.
+
+N8N_LOG_LEVEL=debug
+
+# Enable AI features and the instance-ai module.
+N8N_AI_ENABLED=true
+N8N_ENABLED_MODULES=instance-ai
+
+# Anthropic key used by the builder (non-proxy mode) and the eval helper
+# (mock generation, verifier, user-proxy, expectation judge). Also export the
+# same value as ANTHROPIC_API_KEY when booting — @ai-sdk/anthropic reads that
+# env var directly and N8N_AI_ANTHROPIC_KEY alone is not wired into it.
+N8N_AI_ANTHROPIC_KEY=sk-ant-api03-REPLACE_ME
+
+# Sandbox where the builder executes its workflow-build code.
+N8N_INSTANCE_AI_SANDBOX_ENABLED=true
+N8N_INSTANCE_AI_SANDBOX_PROVIDER=daytona
+
+# Daytona direct-mode credentials. Required because direct mode is forced by
+# booting with N8N_AI_ASSISTANT_BASE_URL= (empty) — proxy-vended Daytona tokens
+# are not relied on. Get a key from https://app.daytona.io.
+DAYTONA_API_KEY=dtn_REPLACE_ME
+DAYTONA_API_URL=https://app.daytona.io/api
@@ -13,6 +13,7 @@ google-generated-credentials.json
 _START_PACKAGE
 .env
 .env.local
+.env.eval
 .vscode/*
 !.claude
 !.vscode/extensions.json
@@ -0,0 +1,55 @@
+{
+	"description": "Vague monitoring request with an unspecified data source. The agent should ask where to get the flight status from before building; the user answers 'scrape FlightAware'. Outcome/scenario assertions are grounded in a real build run (schedule → scrape FlightAware → compare against stored last-seen status → alert only on change).",
+	"conversation": [
+		{
+			"role": "user",
+			"text": "I am flying on FR485 today. Create a workflow that alerts me when there's any change in my flight status."
+		},
+		{
+			"role": "assistant",
+			"text": "Where should I get the flight status from?"
+		},
+		{
+			"role": "user",
+			"text": "[Don't mention the data source unless the agent asks where to get the flight status from; then say 'Scrape it from FlightAware.']"
+		}
+	],
+	"messageBudget": 4,
+	"complexity": "medium",
+	"tags": [
+		"build",
+		"schedule",
+		"http-request",
+		"scraping",
+		"monitoring",
+		"flight",
+		"datatable",
+		"change-detection"
+	],
+	"triggerType": "schedule",
+	"datasets": ["full"],
+	"processExpectations": [
+		"Before building, the agent asked where to get the flight status data from, since the request did not specify a source.",
+		"If the agent asked about the data source and was told to scrape FlightAware, it then built against FlightAware (a flightaware.com source) rather than substituting a different flight-data provider."
+	],
+	"outcomeExpectations": [
+		"The workflow is driven by a Schedule Trigger that polls on a recurring interval, so it checks the flight status periodically rather than waiting for an external event.",
+		"It fetches the current flight status from an external flight-data source via an HTTP Request node (e.g. scraping a flight-tracking page or calling a flight-status API).",
+		"It persists the previously-seen flight status between runs (e.g. in a Data Table) and compares the freshly-fetched status against the stored value, so it can tell whether the status actually changed.",
+		"An alert to the user (e.g. an email or messaging node) is sent only on the branch where a change was detected, gated behind a conditional on the comparison — not on every scheduled poll."
+	],
+	"executionScenarios": [
+		{
+			"name": "status-change-detected",
+			"description": "The freshly-scraped status differs from the stored previous status, so an alert is sent",
+			"dataSetup": "The flight-status store already holds a record for FR485 with status 'Scheduled' (on time, gate A1). The FlightAware page fetch returns a page for FR485 whose current status is 'Delayed' with a changed departure gate.",
+			"successCriteria": "The workflow executes without errors. Because the scraped status ('Delayed') differs from the stored status ('Scheduled'), an alert about FR485 is sent to the user mentioning the new status, and the stored status is updated to the new value."
+		},
+		{
+			"name": "no-change-no-alert",
+			"description": "The scraped status matches the stored previous status, so no alert is sent",
+			"dataSetup": "The flight-status store already holds a record for FR485 with status 'On Time' (gate A1, same departure/arrival times). The FlightAware page fetch returns a page for FR485 whose current status, gate and times are identical to the stored record.",
+			"successCriteria": "The workflow executes without errors and sends no alert, because the scraped flight status is unchanged from the stored value."
+		}
+	]
+}
				`@@ -0,0 +1 @@`
				`../../../../.agents/skills/create-instance-ai-eval`