opensource/n8n
3
0
Fork 0
mirror of https://github.com/n8n-io/n8n.git synced 2026-09-19 09:51:59 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(core): Improve Instance AI model experiments (no-changelog) (#35034)

Browse Source 
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Riqwan Thamir
2026-08-13 10:15:03 +00:00
committed by GitHub
co-authored by Cursor
parent 408845c4f6
commit 91ebeecb4e
55 changed files with 1942 additions and 235 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/WORKFLOWS.md
+2 -2
View File
@@ -209,7 +209,7 @@ the lab bench.** The gate deliberately exposes only PR re-runs. Anything that
isn't PR gating — baselines, model experiments, arbitrary branch runs — goes
through `test-evals-instance-ai.yml`'s own dispatch form ("Instance AI
Evals: Experiments"): full knob set (branch, filter, tier, suite,
iterations, experiment-name, model), no per-PR cancellation (dispatches run in parallel, e.g. concurrent
iterations, experiment-name, model, model-url, model-key, reasoning-effort, supports-structured-outputs), no per-PR cancellation (dispatches run in parallel, e.g. concurrent
model-comparison arms), and SHA-keyed docker cache hits on master. Evals never
run on fork PRs: the event trigger gates on `head.repo.fork`, and the `pr`
re-run path refuses fork PRs in `resolve` (dispatched runs carry secrets).
@@ -763,7 +763,7 @@ Adding a new channel requires inviting the bot first; the first run otherwise fa
| Cloud/CDN | `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID` |
| GitHub Automation | `N8N_ASSISTANT_APP_ID`, `N8N_ASSISTANT_PRIVATE_KEY` |
| Benchmarking | `BENCHMARK_ARM_*`, `N8N_BENCHMARK_LICENSE_CERT` |
| AI/Evals | `ANTHROPIC_API_KEY`, `EVALS_LANGSMITH_*` |
| AI/Evals | `EVALS_ANTHROPIC_KEY`, `EVALS_OPENAI_KEY`, `EVALS_OPENROUTER_KEY`, `EVALS_XAI_KEY`, `EVALS_BASETEN_KEY`, `EVALS_FIREWORKS_KEY`, `EVALS_TOGETHER_KEY`, `EVALS_DATABRICKS_KEY`, `EVALS_MODAL_KEY`, `EVALS_LYCEUM_KEY`, `EVALS_AZURE_FOUNDRY_KEY`, `EVALS_VERTEX_KEY`, `EVALS_VERTEX_PROJECT_ID`, `EVALS_VERTEX_LOCATION`, `EVALS_LANGSMITH_*` |
### Scoping
.github/workflows/test-evals-instance-ai.yml
+222 -23
View File
@@ -60,7 +60,37 @@ on:
type: string
default: ''
model:
description: 'Model override for all Instance AI agents in the lane backends, provider/model format. Empty = backend default.'
description: 'Model override (provider/model). Empty = anthropic/claude-opus-4-8. Vertex Claude: google-vertex-anthropic/<model>. OpenAI-compat experiments use custom/<model> + model-url + model-key. See evaluations/README.md.'
required: false
type: string
default: ''
model-url:
description: 'Optional N8N_INSTANCE_AI_MODEL_URL (required for custom/* OpenAI-compatible /v1; also anthropic/* Azure Foundry). See evaluations/README.md.'
required: false
type: string
default: ''
model-key:
description: 'Which EVALS_* secret to use for custom/* (or anthropic/* + model-url). One of: baseten, fireworks, together, modal, databricks, azure, lyceum. Empty = keyless custom, or native openai/openrouter/xai/anthropic prefix routing.'
required: false
type: string
default: ''
reasoning-effort:
description: 'Optional custom/* reasoning effort override (none|minimal|low|medium|high|xhigh|max). Empty = runtime known-model map; still empty = omit.'
required: false
type: string
default: ''
supports-structured-outputs:
description: 'Optional custom/* supportsStructuredOutputs override (true|false). Empty = runtime known-model map; still empty = omit.'
required: false
type: string
default: ''
lanes:
description: 'Parallel n8n lane containers (1-11). Empty = 10, or 1 when model-key=baseten (fits Baseten Basic verified ~500k TPM).'
required: false
type: string
default: ''
eval-concurrency:
description: 'Concurrent scenario executions. Empty = 32, or 2 when model-key=baseten (~0.5M TPM / ~12 RPM — fits Baseten Basic verified).'
required: false
type: string
default: ''
@@ -95,7 +125,31 @@ on:
required: false
default: ''
model:
description: 'Model for all Instance AI agents (provider/model). Empty = backend default (anthropic/claude-opus-4-8). For experiments: anthropic/claude-sonnet-4-6, anthropic/claude-sonnet-5.'
description: 'Model for all Instance AI agents (provider/model). Empty = backend default (anthropic/claude-opus-4-8). Vertex Claude: google-vertex-anthropic/<model>. OpenAI-compat experiments: custom/<model> + model-url + model-key.'
required: false
default: ''
model-url:
description: 'Base URL (N8N_INSTANCE_AI_MODEL_URL). Required for custom/*; optional for anthropic/* Azure Foundry.'
required: false
default: ''
model-key:
description: 'EVALS_* secret alias for custom/*: baseten | fireworks | together | modal | databricks | azure | lyceum. Empty = keyless custom or native provider prefix.'
required: false
default: ''
reasoning-effort:
description: 'Optional custom/* reasoning effort override (none|minimal|low|medium|high|xhigh|max). Empty = runtime known-model map; still empty = omit.'
required: false
default: ''
supports-structured-outputs:
description: 'Optional custom/* supportsStructuredOutputs override (true|false). Empty = runtime known-model map; still empty = omit.'
required: false
default: ''
lanes:
description: 'Parallel n8n lane containers (1-11). Empty = 10, or 1 when model-key=baseten (fits Baseten Basic verified ~500k TPM).'
required: false
default: ''
eval-concurrency:
description: 'Concurrent scenarios. Empty = 32, or 2 when model-key=baseten (~0.5M TPM / ~12 RPM — fits Baseten Basic verified).'
required: false
default: ''
@@ -105,16 +159,19 @@ jobs:
# 8vcpu/32GB: full-suite N=10 baselines need ~150 min of eval time and
# did not fit the 4vcpu tier at any timeout.
runs-on: blacksmith-8vcpu-ubuntu-2204
# Long ceiling only for high-N baseline captures; regular runs keep a
# tight guard so a wedged run fails fast. Plain comparison (not fromJSON):
# it coerces the string input to a number, and a malformed dispatch value
# (NaN) falls back to the 90-minute guard instead of erroring the job.
timeout-minutes: ${{ (inputs.iterations || '3') >= 5 && 240 || 90 }}
# Long ceiling for high-N baselines (240) and for Baseten (600: auto
# 1 lane / c=2 is ~16× slower than the Anthropic default, so a ~18 min
# full suite becomes ~5 h; +2 h buffer for provider backoff). Plain
# comparison (not fromJSON): coerces the string input to a number;
# malformed NaN falls back to the 90-minute guard instead of erroring.
timeout-minutes: ${{ (inputs['model-key'] || '') == 'baseten' && 600 || ((inputs.iterations || '3') >= 5 && 240 || 90) }}
env:
# Each port hosts an independent n8n container. The eval CLI's
# work-stealing allocator dispatches builds across them, capped per-lane.
# 10 lanes x 2.5 GB caps = 25 GB, leaving ~7 GB for sandbox + CLI + OS.
LANE_PORTS: '5678,5679,5680,5681,5682,5683,5684,5685,5686,5687'
# Contiguous ports from 5678; 5678..5688 avoids Node fetch()'s blocked-
# port list (max 11 lanes). Defaults: 10 lanes / c=32 for Anthropic;
# model-key=baseten auto-throttles to 1 / 2 to stay under Baseten Basic
# verified (~500k TPM).
LANES: ${{ inputs.lanes != '' && inputs.lanes || ((inputs['model-key'] || '') == 'baseten' && '1' || '10') }}
EVAL_CONCURRENCY: ${{ inputs.eval-concurrency != '' && inputs.eval-concurrency || ((inputs['model-key'] || '') == 'baseten' && '2' || '32') }}
permissions:
contents: read
pull-requests: write
@@ -174,7 +231,20 @@ jobs:
- name: Start n8n containers
env:
EVALS_ANTHROPIC_KEY: ${{ secrets.EVALS_ANTHROPIC_KEY }}
EVALS_OPENAI_KEY: ${{ secrets.EVALS_OPENAI_KEY }}
INSTANCE_AI_BRAVE_SEARCH_API_KEY: ${{ secrets.INSTANCE_AI_BRAVE_SEARCH_API_KEY }}
EVALS_OPENROUTER_KEY: ${{ secrets.EVALS_OPENROUTER_KEY }}
EVALS_XAI_KEY: ${{ secrets.EVALS_XAI_KEY }}
EVALS_BASETEN_KEY: ${{ secrets.EVALS_BASETEN_KEY }}
EVALS_FIREWORKS_KEY: ${{ secrets.EVALS_FIREWORKS_KEY }}
EVALS_TOGETHER_KEY: ${{ secrets.EVALS_TOGETHER_KEY }}
EVALS_DATABRICKS_KEY: ${{ secrets.EVALS_DATABRICKS_KEY }}
EVALS_MODAL_KEY: ${{ secrets.EVALS_MODAL_KEY }}
EVALS_LYCEUM_KEY: ${{ secrets.EVALS_LYCEUM_KEY }}
EVALS_AZURE_FOUNDRY_KEY: ${{ secrets.EVALS_AZURE_FOUNDRY_KEY }}
EVALS_VERTEX_KEY: ${{ secrets.EVALS_VERTEX_KEY }}
EVALS_VERTEX_PROJECT_ID: ${{ secrets.EVALS_VERTEX_PROJECT_ID }}
EVALS_VERTEX_LOCATION: ${{ secrets.EVALS_VERTEX_LOCATION }}
N8N_LICENSE_ACTIVATION_KEY: ${{ secrets.N8N_LICENSE_ACTIVATION_KEY }}
N8N_LICENSE_CERT: ${{ secrets.N8N_LICENSE_CERT }}
N8N_ENCRYPTION_KEY: ${{ secrets.N8N_ENCRYPTION_KEY }}
@@ -187,6 +257,10 @@ jobs:
SANDBOX_NAME_PREFIX: evals-ci-${{ github.head_ref || inputs.branch || github.ref_name }}
SANDBOX_PROVIDER: ${{ inputs.sandbox-provider || 'n8n-sandbox' }}
INSTANCE_AI_MODEL: ${{ inputs.model }}
INSTANCE_AI_MODEL_URL: ${{ inputs.model-url }}
INSTANCE_AI_MODEL_KEY: ${{ inputs.model-key }}
INSTANCE_AI_REASONING_EFFORT: ${{ inputs.reasoning-effort }}
INSTANCE_AI_SUPPORTS_STRUCTURED_OUTPUTS: ${{ inputs.supports-structured-outputs }}
run: |
# Build provider-specific env args
SANDBOX_ARGS=()
@@ -216,8 +290,112 @@ jobs:
if [ -n "$INSTANCE_AI_MODEL" ]; then
MODEL_ARGS+=(-e N8N_INSTANCE_AI_MODEL="$INSTANCE_AI_MODEL")
fi
# Anthropic-compatible custom base (Azure Foundry Claude). Drop /v1/messages
# if pasted from the Foundry Details "Endpoint" field — the Anthropic SDK
# appends /v1/messages itself.
if [ -n "$INSTANCE_AI_MODEL_URL" ]; then
MODEL_ARGS+=(-e N8N_INSTANCE_AI_MODEL_URL="$INSTANCE_AI_MODEL_URL")
fi
IFS=',' read -ra PORTS <<< "$LANE_PORTS"
# custom/* experiment knobs: pass through workflow inputs only.
# When unset, the n8n runtime looks up custom-model-defaults.ts and
# omits the field if still unresolved — do not re-implement that map here.
if [ -n "$INSTANCE_AI_REASONING_EFFORT" ]; then
MODEL_ARGS+=(-e N8N_INSTANCE_AI_REASONING_EFFORT="$INSTANCE_AI_REASONING_EFFORT")
echo "custom reasoning effort override: $INSTANCE_AI_REASONING_EFFORT"
fi
if [ -n "$INSTANCE_AI_SUPPORTS_STRUCTURED_OUTPUTS" ]; then
MODEL_ARGS+=(-e N8N_INSTANCE_AI_SUPPORTS_STRUCTURED_OUTPUTS="$INSTANCE_AI_SUPPORTS_STRUCTURED_OUTPUTS")
echo "custom supportsStructuredOutputs override: $INSTANCE_AI_SUPPORTS_STRUCTURED_OUTPUTS"
fi
# EvalMock / pin-data / in-product eval LLMs always use Anthropic Sonnet,
# independent of the builder experiment model (custom/Kimi, openai/*, …).
# ANTHROPIC_API_KEY is the provider-native key resolveEvalModelConfig prefers
# when N8N_INSTANCE_AI_EVAL_MODEL differs from N8N_INSTANCE_AI_MODEL.
MODEL_ARGS+=(
-e N8N_INSTANCE_AI_EVAL_MODEL=anthropic/claude-sonnet-4-6
-e ANTHROPIC_API_KEY="$EVALS_ANTHROPIC_KEY"
)
# Lane builds use the provider matching N8N_INSTANCE_AI_MODEL.
# The eval CLI step keeps EVALS_ANTHROPIC_KEY for Sonnet verifier/mocks.
MODEL_API_KEY="$EVALS_ANTHROPIC_KEY"
MODEL_PROVIDER="${INSTANCE_AI_MODEL%%/*}"
require_secret() {
local name="$1" value="$2"
if [ -z "$value" ]; then
echo "::error::${name} secret is empty (model=${INSTANCE_AI_MODEL:-default} model-key=${INSTANCE_AI_MODEL_KEY:-})"
exit 1
fi
}
resolve_model_key() {
case "$INSTANCE_AI_MODEL_KEY" in
baseten) require_secret EVALS_BASETEN_KEY "$EVALS_BASETEN_KEY"; MODEL_API_KEY="$EVALS_BASETEN_KEY" ;;
fireworks) require_secret EVALS_FIREWORKS_KEY "$EVALS_FIREWORKS_KEY"; MODEL_API_KEY="$EVALS_FIREWORKS_KEY" ;;
together) require_secret EVALS_TOGETHER_KEY "$EVALS_TOGETHER_KEY"; MODEL_API_KEY="$EVALS_TOGETHER_KEY" ;;
modal) require_secret EVALS_MODAL_KEY "$EVALS_MODAL_KEY"; MODEL_API_KEY="$EVALS_MODAL_KEY" ;;
databricks) require_secret EVALS_DATABRICKS_KEY "$EVALS_DATABRICKS_KEY"; MODEL_API_KEY="$EVALS_DATABRICKS_KEY" ;;
azure) require_secret EVALS_AZURE_FOUNDRY_KEY "$EVALS_AZURE_FOUNDRY_KEY"; MODEL_API_KEY="$EVALS_AZURE_FOUNDRY_KEY" ;;
lyceum) require_secret EVALS_LYCEUM_KEY "$EVALS_LYCEUM_KEY"; MODEL_API_KEY="$EVALS_LYCEUM_KEY" ;;
'') ;;
*)
echo "::error::unknown model-key '${INSTANCE_AI_MODEL_KEY}' (expected baseten|fireworks|together|modal|databricks|azure|lyceum)"
exit 1
;;
esac
}
if [[ "$INSTANCE_AI_MODEL" == custom/* ]]; then
if [ -z "$INSTANCE_AI_MODEL_URL" ]; then
echo "::error::model custom/* requires model-url (OpenAI-compatible base URL ending in /v1)"
exit 1
fi
# model-key selects EVALS_*; empty model-key = keyless custom router.
if [ -n "$INSTANCE_AI_MODEL_KEY" ]; then
resolve_model_key
else
MODEL_API_KEY=""
fi
elif [ "$MODEL_PROVIDER" = "google-vertex-anthropic" ]; then
if [ -n "$INSTANCE_AI_MODEL_URL" ]; then
echo "::error::google-vertex-anthropic/* does not use model-url (leave it empty)"
exit 1
fi
require_secret EVALS_VERTEX_KEY "$EVALS_VERTEX_KEY"
require_secret EVALS_VERTEX_PROJECT_ID "$EVALS_VERTEX_PROJECT_ID"
MODEL_API_KEY=""
MODEL_ARGS+=(
-e N8N_INSTANCE_AI_VERTEX_PROJECT_ID="$EVALS_VERTEX_PROJECT_ID"
-e N8N_INSTANCE_AI_VERTEX_LOCATION="${EVALS_VERTEX_LOCATION:-global}"
-e N8N_INSTANCE_AI_VERTEX_SERVICE_ACCOUNT_JSON="$EVALS_VERTEX_KEY"
)
elif [ -n "$INSTANCE_AI_MODEL_URL" ]; then
# anthropic/* + model-url = Azure Foundry Claude (or model-key=azure).
if [ -n "$INSTANCE_AI_MODEL_KEY" ]; then
resolve_model_key
else
require_secret EVALS_AZURE_FOUNDRY_KEY "$EVALS_AZURE_FOUNDRY_KEY"
MODEL_API_KEY="$EVALS_AZURE_FOUNDRY_KEY"
fi
else
case "$MODEL_PROVIDER" in
openai) require_secret EVALS_OPENAI_KEY "$EVALS_OPENAI_KEY"; MODEL_API_KEY="$EVALS_OPENAI_KEY" ;;
openrouter) require_secret EVALS_OPENROUTER_KEY "$EVALS_OPENROUTER_KEY"; MODEL_API_KEY="$EVALS_OPENROUTER_KEY" ;;
xai) require_secret EVALS_XAI_KEY "$EVALS_XAI_KEY"; MODEL_API_KEY="$EVALS_XAI_KEY" ;;
esac
fi
if [ "$LANES" -lt 1 ] || [ "$LANES" -gt 11 ]; then
echo "::error::lanes must be 1-11 (got $LANES)"
exit 1
fi
PORTS=()
for i in $(seq 0 $((LANES - 1))); do
PORTS+=($((5678 + i)))
done
# Reuse across later steps (Create test users / Assert / Run Evals).
echo "LANE_PORTS=${PORTS[*]}" >> "$GITHUB_ENV"
echo "Starting $LANES lane(s) on ports ${PORTS[*]} (eval concurrency $EVAL_CONCURRENCY)"
for i in "${!PORTS[@]}"; do
port="${PORTS[$i]}"
# Bounded and self-healing: a lane that exhausts its capped heap is
@@ -242,8 +420,8 @@ jobs:
-e E2E_TESTS=true \
-e N8N_ENABLED_MODULES=instance-ai \
-e N8N_AI_ENABLED=true \
-e N8N_INSTANCE_AI_MODEL_API_KEY="$EVALS_ANTHROPIC_KEY" \
-e INSTANCE_AI_BRAVE_SEARCH_API_KEY="$INSTANCE_AI_BRAVE_SEARCH_API_KEY" \
-e N8N_INSTANCE_AI_MODEL_API_KEY="$MODEL_API_KEY" \
-e N8N_AI_ASSISTANT_BASE_URL="" \
-e N8N_INSTANCE_AI_SANDBOX_ENABLED=true \
"${SANDBOX_ARGS[@]}" \
@@ -283,7 +461,7 @@ jobs:
- name: Create test users
run: |
IFS=',' read -ra PORTS <<< "$LANE_PORTS"
read -ra PORTS <<< "$LANE_PORTS"
for port in "${PORTS[@]}"; do
curl -sf -X POST "http://localhost:$port/rest/e2e/reset" \
-H "Content-Type: application/json" \
@@ -304,7 +482,7 @@ jobs:
SANDBOX_PROVIDER: ${{ inputs.sandbox-provider || 'n8n-sandbox' }}
INSTANCE_AI_MODEL: ${{ inputs.model }}
run: |
IFS=',' read -ra PORTS <<< "$LANE_PORTS"
read -ra PORTS <<< "$LANE_PORTS"
bad=0
for i in "${!PORTS[@]}"; do
port="${PORTS[$i]}"
@@ -323,13 +501,15 @@ jobs:
echo " lane $lane: sandboxEnabled=true sandboxProvider=$SANDBOX_PROVIDER ok"
fi
# /preferences returns the effective model name (user pref || config),
# i.e. the name part of provider/model.
# i.e. everything after the first provider/ segment (openrouter/moonshotai/kimi-k3
# → moonshotai/kimi-k3). Use #*/ not ##*/ so nested OpenRouter ids match.
if [ -n "$INSTANCE_AI_MODEL" ]; then
expected_model="${INSTANCE_AI_MODEL#*/}"
effective=$(curl -sf -b "/tmp/cookies-$port.txt" \
"http://localhost:$port/rest/instance-ai/preferences" \
| jq -r '.data.modelName')
if [ "$effective" != "${INSTANCE_AI_MODEL##*/}" ]; then
echo "::error::lane $lane (port $port): expected model '${INSTANCE_AI_MODEL##*/}', got '$effective'"
if [ "$effective" != "$expected_model" ]; then
echo "::error::lane $lane (port $port): expected model '$expected_model', got '$effective'"
bad=$((bad+1))
else
echo " lane $lane: model=$effective ok"
@@ -364,7 +544,10 @@ jobs:
continue-on-error: true
working-directory: packages/@n8n/instance-ai
env:
# Host-side verifier/judges always use Anthropic — never the builder experiment model.
N8N_INSTANCE_AI_EVAL_MODEL: anthropic/claude-sonnet-4-6
N8N_INSTANCE_AI_MODEL_API_KEY: ${{ secrets.EVALS_ANTHROPIC_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.EVALS_ANTHROPIC_KEY }}
LANGSMITH_TRACING: 'true'
LANGSMITH_ENDPOINT: ${{ secrets.EVALS_LANGSMITH_ENDPOINT }}
LANGSMITH_API_KEY: ${{ secrets.EVALS_LANGSMITH_API_KEY }}
@@ -383,13 +566,14 @@ jobs:
EXPERIMENT_NAME: ${{ inputs.experiment-name }}
EVAL_PR_NUMBER: ${{ inputs.pr-number || github.event.pull_request.number }}
run: |
IFS=',' read -ra PORTS <<< "$LANE_PORTS"
read -ra PORTS <<< "$LANE_PORTS"
URLS=()
for port in "${PORTS[@]}"; do
URLS+=("http://localhost:$port")
done
BASE_URLS=$(IFS=,; printf '%s' "${URLS[*]}")
ARGS=(--base-url "$BASE_URLS" --concurrency 32 --verbose --iterations "${ITERATIONS:-3}")
echo "Lanes: $LANES | eval concurrency: $EVAL_CONCURRENCY"
ARGS=(--base-url "$BASE_URLS" --concurrency "$EVAL_CONCURRENCY" --verbose --iterations "${ITERATIONS:-3}")
# LangTracer is the only CI case source (no disk fallback by design).
ARGS+=(--source langtracer --suite "${SUITE:-baseline}")
# Pin the LangSmith cohort: langtracer mode otherwise derives a
@@ -420,6 +604,17 @@ jobs:
if: ${{ always() }}
env:
EVALS_ANTHROPIC_KEY: ${{ secrets.EVALS_ANTHROPIC_KEY }}
EVALS_OPENAI_KEY: ${{ secrets.EVALS_OPENAI_KEY }}
EVALS_OPENROUTER_KEY: ${{ secrets.EVALS_OPENROUTER_KEY }}
EVALS_XAI_KEY: ${{ secrets.EVALS_XAI_KEY }}
EVALS_BASETEN_KEY: ${{ secrets.EVALS_BASETEN_KEY }}
EVALS_FIREWORKS_KEY: ${{ secrets.EVALS_FIREWORKS_KEY }}
EVALS_TOGETHER_KEY: ${{ secrets.EVALS_TOGETHER_KEY }}
EVALS_DATABRICKS_KEY: ${{ secrets.EVALS_DATABRICKS_KEY }}
EVALS_MODAL_KEY: ${{ secrets.EVALS_MODAL_KEY }}
EVALS_LYCEUM_KEY: ${{ secrets.EVALS_LYCEUM_KEY }}
EVALS_AZURE_FOUNDRY_KEY: ${{ secrets.EVALS_AZURE_FOUNDRY_KEY }}
EVALS_VERTEX_KEY: ${{ secrets.EVALS_VERTEX_KEY }}
DAYTONA_API_KEY: ${{ secrets.DAYTONA_API_KEY }}
N8N_LICENSE_ACTIVATION_KEY: ${{ secrets.N8N_LICENSE_ACTIVATION_KEY }}
N8N_LICENSE_CERT: ${{ secrets.N8N_LICENSE_CERT }}
@@ -430,9 +625,13 @@ jobs:
# ::add-mask:: is a single-line workflow command. Multi-line secrets
# (e.g. N8N_LICENSE_CERT is PEM-encoded) must be masked one line at
# a time, otherwise only the first line is registered.
for v in "$EVALS_ANTHROPIC_KEY" "$DAYTONA_API_KEY" \
"$N8N_LICENSE_ACTIVATION_KEY" "$N8N_LICENSE_CERT" \
"$N8N_ENCRYPTION_KEY" "$EVALS_LANGSMITH_API_KEY"; do
for v in "$EVALS_ANTHROPIC_KEY" "$EVALS_OPENAI_KEY" "$EVALS_OPENROUTER_KEY" \
"$EVALS_XAI_KEY" "$EVALS_BASETEN_KEY" "$EVALS_FIREWORKS_KEY" \
"$EVALS_TOGETHER_KEY" "$EVALS_DATABRICKS_KEY" "$EVALS_MODAL_KEY" \
"$EVALS_LYCEUM_KEY" "$EVALS_AZURE_FOUNDRY_KEY" "$EVALS_VERTEX_KEY" \
"$DAYTONA_API_KEY" \
"$N8N_LICENSE_ACTIVATION_KEY" "$N8N_LICENSE_CERT" "$N8N_ENCRYPTION_KEY" \
"$EVALS_LANGSMITH_API_KEY"; do
[ -z "$v" ] && continue
while IFS= read -r line; do
[ -n "$line" ] && echo "::add-mask::$line"