fix(sanitization): secret exposure in function and agent trace spans (#6000)

* fix trace span secret sanitization

* sanitize workflow output logs

* preserve streaming usage estimates

* Fix logging session test after staging merge

* secrets sanitization correctness

* fix(execution): address review regressions

* fix(execution): harden secret trace provenance

* fix(execution): preserve functional state during trace projection

---------

Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
This commit is contained in:
Bill Leoutsakos
2026-08-01 16:37:08 -07:00
committed by GitHub
co-authored by Bill Leoutsakos Vikhyath Mondreti
parent 3740c62112
commit 18214158b9
155 changed files with 13595 additions and 1018 deletions
@@ -156,7 +156,7 @@ From **Settings → Custom Tools** you can:
<FAQ items={[
{ question: "Can I use custom tools in standalone blocks (not agents)?", answer: "No. Custom tools are designed for use within Agent blocks, where the AI model decides when to call them. For deterministic tool execution, use the Function block instead." },
{ question: "How do I pass API keys to my custom tool code?", answer: "Use environment variables with double curly brace syntax: {{MY_API_KEY}}. Create the environment variable in Settings → Secrets, and it will be injected at execution time without appearing in logs." },
{ question: "How do I pass API keys to my custom tool code?", answer: "Use double curly brace syntax such as {{MY_API_KEY}} for a value saved under Settings → Secrets. The real value is injected at execution time, while exact occurrences are masked in the trace copy. If the tool returns the secret, the raw result still reaches the Agent. See Execution log protection under Secrets for details." },
{ question: "Can I use external npm packages?", answer: "No. Custom tool code runs in a sandboxed environment with access to built-in Node.js modules and fetch(), but not external packages. For complex dependencies, consider calling an external API that wraps the functionality you need." },
{ question: "What's the difference between custom tools and the Function block?", answer: "Custom tools are called by AI agents when they decide the tool is relevant — the agent chooses when to use it. Function blocks run deterministically at a fixed point in the workflow. Use custom tools for agent-driven actions and Function blocks for predictable data transformations." },
{ question: "Are custom tools shared across the workspace?", answer: "Yes. Custom tools are workspace-scoped, so all workspace members can use them in their workflows." },
+3 -1
View File
@@ -72,6 +72,8 @@ Authorization: Bearer {{MCP_API_TOKEN}}
When you type `{{` in the URL or header fields, a dropdown appears showing available workspace environment variables.
When a saved secret is successfully substituted this way, exact occurrences of its value are masked in stored MCP tool-call traces. The real URL or header value still reaches the MCP server unchanged. See [Execution log protection](/platform/credentials#execution-log-protection) for the exact scope and limitations.
### Testing and Validation
Click **Test Connection** before saving to verify the server is reachable and discover available tools. The test response shows the number of tools found and the protocol version.
@@ -169,4 +171,4 @@ import { FAQ } from '@/components/ui/faq'
{ question: "How do I update MCP tool schemas after a server changes its available tools?", answer: "Click the Refresh button on the MCP server in your workspace settings. This fetches the latest tool schemas from the server and automatically updates any agent blocks that use those tools with the new parameter definitions." },
{ question: "Can permission groups restrict access to MCP tools?", answer: "Yes. On Enterprise-entitled workspaces, any workspace admin can create a permission group that disables MCP tools for its members using the disableMcpTools option. When this is enabled, affected users will not be able to add or use MCP tools in workflows that belong to that workspace." },
{ question: "What happens if an MCP server goes offline during workflow execution?", answer: "If the MCP server is unreachable during execution, the tool call will fail and return an error. In an Agent block, the AI may attempt to handle the failure gracefully. In a standalone MCP Tool block, the workflow step will fail. Check MCP server logs and verify the server is running and accessible to troubleshoot connectivity issues." },
]} />
]} />
@@ -43,7 +43,7 @@ This is the level you debug at, because a run fails when one of its blocks fails
### Input and output
Each block in the sidebar has two tabs. The **Input** tab shows the resolved values the block actually ran with: the literal values you typed, and the earlier outputs it read by reference (with API keys redacted). The **Output** tab shows what the block produced, formatted as an object, with markdown rendered for agent-generated text.
Each block in the sidebar has two tabs. The **Input** tab shows the resolved values the block actually ran with: the literal values you typed, and the earlier outputs it read by reference. Exact values successfully substituted from Secrets through `{{KEY}}` are masked in this trace copy; see [Execution log protection](/platform/credentials#execution-log-protection). The **Output** tab shows what the block produced, formatted as an object, with markdown rendered for agent-generated text.
The input tab is the important one. A block reads earlier outputs by name, written `<blockName.field>`, and the input tab shows what those references resolved to at run time. If a reference pointed at a value that was not there, you see it here as missing or wrong, not as the tag you wrote. See [how blocks pass data](/workflows/data-flow) for how those references resolve.
@@ -57,7 +57,7 @@ Click any entry to open its sidebar: the run's timeline (start/end, total durati
The block's result — JSON-formatted structured data, markdown rendering for AI content, and a copy button.
</Tab>
<Tab>
What the block received — resolved variable values, referenced outputs, and environment variables. API keys are automatically redacted.
What the block received — resolved variable values, referenced outputs, and environment variables. Exact secret values activated by a successful `{{KEY}}` substitution are masked in this trace view. See [Execution log protection](/platform/credentials#execution-log-protection).
</Tab>
</Tabs>
@@ -95,7 +95,7 @@ import { FAQ } from '@/components/ui/faq'
<FAQ items={[
{ question: "How long are run logs retained?", answer: "Free plans retain logs for 7 days — after that, logs are archived to cloud storage and deleted from the database. Pro, Team, and Enterprise plans retain logs indefinitely with no automatic cleanup." },
{ question: "What data is captured in each run log?", answer: "Each log entry includes the run ID, workflow ID, trigger type, start and end timestamps, total duration in milliseconds, cost breakdown (total cost, token counts, and per-model breakdowns), run data with trace spans, final output, and any associated files. The log details sidebar lets you inspect block-level inputs and outputs." },
{ question: "Are API keys visible in the logs?", answer: "No. API keys and credentials are automatically redacted in the log input tab for security. You can safely inspect block inputs without exposing sensitive values." },
{ question: "Are saved secrets visible in logs?", answer: "When a value saved under Secrets is successfully substituted through {{KEY}}, exact, case-sensitive occurrences are masked throughout the log-facing copy, including the live block-log display, Logs Overview input and output, Trace, log-read APIs, and the Logs block's Get Run Details output. This is not a general redactor: hardcoded or directly read values do not activate masking by themselves, and encoded, hashed, or transformed values are not matched. Functional execution responses, streams, and callbacks remain unchanged. See Execution log protection under Secrets for details." },
{ question: "What is a workflow snapshot?", answer: "A frozen copy of the workflow's structure (blocks, connections, and configuration) captured at run time, so you can see the exact state behind a particular run — useful for debugging workflows that have been modified since." },
{ question: "Can I access logs programmatically?", answer: "Yes. The External API provides endpoints to query logs with filtering by workflow, time range, trigger type, duration, cost, and model. You can also set up webhook, email, or Slack notifications for real-time alerts when runs complete." },
{ question: "What does Live mode do on the Logs page?", answer: "It refreshes the Logs page in real time so new entries appear as they are recorded — useful during deployments or when monitoring active workflows." },
@@ -65,8 +65,14 @@ Select the secret you want to use. The reference appears highlighted in blue and
height={200}
/>
### Execution log protection
When a saved secret is successfully substituted through a `{{KEY}}` reference, Sim masks exact, case-sensitive occurrences of its resolved value in log-facing content. This includes the editor's live block-log display, Logs Overview input and output, stored execution traces, log-read API responses, and the Logs block's **Get Run Details** output. Function and Agent span inputs, outputs, and errors, Agent thinking, and Agent tool-call arguments, results, and errors are protected. The replacement is normally shown as `{{KEY}}`.
This is an observability projection only. Secret resolution and workflow behavior are unchanged: blocks, tools, models, and downstream steps receive the real runtime value. Stored functional execution data, workflow execution responses, streams, callbacks, block state, and snapshots are not rewritten. Log-facing views and read APIs receive a separate protected copy, so the Logs Overview **Workflow Input** and **Workflow Output** are masked without changing the underlying workflow result.
<Callout type="warn">
Secret values are never exposed in the workflow editor or execution logs — they are only resolved during execution.
Masking is activated only when Sim successfully resolves a value from **Settings → Secrets** through `{{KEY}}`. A hardcoded literal, direct `environmentVariables['KEY']` read, or shell `$KEY` read does not activate it by itself. Once activated, every exact occurrence of that value in the run's log-facing content is masked. Encoded, hashed, or otherwise transformed versions are not matched. Do not deliberately return or print secrets.
</Callout>
## Secret Details
@@ -115,7 +121,8 @@ When a workflow runs, secrets resolve in this order:
- **Never hardcode secrets** in workflow input fields — always use `{{KEY}}` references
<FAQ items={[
{ question: "Are my secrets encrypted at rest?", answer: "Yes. Secret values are encrypted before being stored in the database using server-side encryption, so raw values are never persisted in plaintext. They are also never exposed in the workflow editor, logs, or API responses." },
{ question: "Are my secrets encrypted at rest?", answer: "Yes. Values saved under Secrets are encrypted before being stored in the database." },
{ question: "Can a saved secret still appear in a workflow result?", answer: "Yes. Functional data is not rewritten, so the raw value can still reach downstream blocks, tools, and models and can appear in workflow execution responses, streams, or callbacks if your workflow deliberately returns or prints it. Log-facing views and read APIs, including Workflow Output on the Logs Overview and the Logs block's Get Run Details output, receive a protected copy after a successful {{KEY}} substitution." },
{ question: "What happens if both a workspace secret and a personal secret have the same key name?", answer: "The workspace secret takes precedence. During execution, the resolver checks workspace secrets first and uses personal secrets only as a fallback. This ensures production workflows use the shared, team-managed value." },
{ question: "Who determines which personal secret is used for automated runs?", answer: "For manual runs, the personal secrets of the user who clicked Run are used as fallback. For automated runs triggered by API, webhook, or schedule, the personal secrets of the workflow owner are used instead." },
{ question: "Can I import secrets from a .env file?", answer: "Yes. Paste .env-style content (KEY=VALUE format) into any key or value field and the secrets will be auto-populated. The parser supports export KEY=VALUE, quoted values, and inline comments." },
@@ -138,14 +138,19 @@ number reported. A sandbox may declare up to 50 packages.
When a Function block is used as an Agent tool, its code can read every workspace
secret by default — both `{{MY_SECRET}}` and `environmentVariables['MY_SECRET']`.
Use `{{MY_SECRET}}` when the value may appear in execution logs: a successful
double-brace substitution activates [execution-trace masking](/platform/credentials#execution-log-protection),
while direct `environmentVariables['MY_SECRET']` access alone does not activate
it by itself.
To narrow that, set **Secret access** to *Selected secrets* in the block's
tool configuration and pick the names the code may read. Two things change:
- Only those secrets are injected. `{{OTHER_SECRET}}` no longer resolves either.
- The selected **names** are added to the tool's description, so the model knows
what it can reference. Values are never sent to the model — they are injected
server-side at execution.
what it can reference. Values are injected server-side when the Function runs.
If the Function returns a secret, the raw result is still sent to the Agent;
masking affects only the trace and display copy.
Leaving the default (*All secrets*) resolves the list at run time, so a secret
added next month is included automatically.
@@ -46,7 +46,10 @@ This pairs naturally with the [Sim trigger](/workflows/triggers/sim): the trigge
<Callout type="info">
The block always operates on the current workspace. Costs are denominated in credits, both for
the cost filter and the cost output.
the cost filter and the cost output. **Get Run Details** is a log-read path, so both `traceSpans`
and `finalOutput` come from the protected log-facing projection described under
[Execution log protection](/platform/credentials#execution-log-protection). The underlying runtime
result remains unchanged.
</Callout>
<FAQ items={[
@@ -74,9 +74,9 @@ Each run starts fresh from the values defined in the panel. A change made during
## Environment variables
Environment variables store sensitive values like API keys, tokens, and configuration that should never appear in logs or on the canvas. Create them under **Settings → Secrets** by adding a key-value pair.
Environment variables let you keep sensitive values like API keys and tokens out of your saved workflow configuration. Create them under **Settings → Secrets** by adding a key-value pair.
Reference them with double curly braces in any block field, including Agent system prompts and Function block code. The value is substituted before the block runs, and API keys are redacted in logs.
Reference them with double curly braces in any block field, including Agent system prompts and Function block code. The value is substituted before the block runs. When a saved secret is successfully substituted through `{{KEY}}`, exact occurrences of that value are masked in the live block-log display and stored execution trace without changing the runtime value. See [Execution log protection](/platform/credentials#execution-log-protection) for the exact scope and limitations.
```
{{API_KEY}}
+104 -5
View File
@@ -796,16 +796,115 @@ describe('Function Execute API Route', () => {
describe('Template Variable Resolution', () => {
it.concurrent('should resolve environment variables with {{var_name}} syntax', async () => {
const req = createMockRequest('POST', {
code: 'return {{API_KEY}}',
envVars: {
API_KEY: 'secret-key-123',
const req = createMockRequest(
'POST',
{
code: 'return {{API_KEY}}',
envVars: {
API_KEY: 'secret-key-123',
},
},
})
{
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
}
)
const response = await POST(req)
const data = await response.json()
expect(response.status).toBe(200)
expect(data.__resolvedSecretNames).toEqual(['API_KEY'])
})
it('reports only successful references sourced from scoped environment variables', async () => {
const envResponse = await POST(
createMockRequest(
'POST',
{
code: 'return {{SHARED}} + {{ENV_ONLY}} + {{MISSING}}',
params: { SHARED: 'param-value', MISSING: 'ordinary-param' },
envVars: { SHARED: 'secret-value', ENV_ONLY: 'other-secret' },
},
{
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
}
)
)
const envData = await envResponse.json()
const directResponse = await POST(
createMockRequest(
'POST',
{
code: 'return environmentVariables.API_KEY + params.API_KEY',
params: { API_KEY: 'ordinary-param' },
envVars: { API_KEY: 'secret-value' },
},
{
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
}
)
)
const directData = await directResponse.json()
expect(envData.__resolvedSecretNames).toEqual(['ENV_ONLY', 'SHARED'])
expect(directData.__resolvedSecretNames).toEqual([])
})
it('reports shell {{NAME}} substitutions but not direct shell environment access', async () => {
envFlagsMock.isRemoteSandboxEnabled = true
const referencedResponse = await POST(
createMockRequest(
'POST',
{
code: 'printf "%s" "{{API_KEY}}"',
language: 'shell',
envVars: { API_KEY: 'secret-value' },
},
{
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
}
)
)
const referencedData = await referencedResponse.json()
const directResponse = await POST(
createMockRequest(
'POST',
{
code: 'printf "%s" "$API_KEY"',
language: 'shell',
envVars: { API_KEY: 'secret-value' },
},
{
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
}
)
)
const directData = await directResponse.json()
expect(referencedData.__resolvedSecretNames).toEqual(['API_KEY'])
expect(directData.__resolvedSecretNames).toEqual([])
})
it('reports only substitutions allowed by the Function secret scope', async () => {
const response = await POST(
createMockRequest(
'POST',
{
code: 'return {{ALLOWED}} + {{BLOCKED}}',
envVars: { ALLOWED: 'allowed-secret', BLOCKED: 'blocked-secret' },
secretScope: 'selected',
mountedSecrets: ['ALLOWED'],
},
{
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
}
)
)
expect((await response.json()).__resolvedSecretNames).toEqual(['ALLOWED'])
})
it.concurrent('should resolve tag variables with <tag_name> syntax', async () => {
+132 -28
View File
@@ -35,6 +35,12 @@ import {
} from '@/lib/execution/payloads/materialization.server'
import { compactExecutionPayload } from '@/lib/execution/payloads/serializer'
import { materializeLargeValueRef } from '@/lib/execution/payloads/store'
import {
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
RESOLVED_SECRET_NAMES_FIELD,
RESOLVED_SECRET_NAMES_METADATA_V1,
requestsPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
import {
executeInSandbox,
executeShellInSandbox,
@@ -65,6 +71,8 @@ const E2B_JS_WRAPPER_LINES = 3
const E2B_PYTHON_WRAPPER_LINES = 1
const MAX_SANDBOX_OUTPUT_FILES = 20
const MAX_SANDBOX_OUTPUT_BYTES = 50 * 1024 * 1024
const MAX_PRIVATE_RESOLVED_SECRET_NAMES = 10_000
const MAX_PRIVATE_RESOLVED_SECRET_NAMES_BYTES = 1024 * 1024
/** Matches valid JS identifier names (letters, digits, underscore; no leading digit). */
const SAFE_IDENTIFIER = /^[a-zA-Z_][a-zA-Z0-9_]*$/
@@ -586,7 +594,8 @@ function resolveEnvironmentVariables(
code: string,
params: Record<string, any>,
envVars: Record<string, string>,
contextVariables: Record<string, any>
contextVariables: Record<string, any>,
onResolvedSecret?: (name: string) => void
): string {
let resolvedCode = code
@@ -627,6 +636,9 @@ function resolveEnvironmentVariables(
const safeVarName = `__var_${varName.replace(/[^a-zA-Z0-9_]/g, '_')}`
contextVariables[safeVarName] = varValue
if (Object.hasOwn(envVars, varName) && envVars[varName] === varValue) {
onResolvedSecret?.(varName)
}
resolvedCode =
resolvedCode.slice(0, index) + safeVarName + resolvedCode.slice(index + matchStr.length)
}
@@ -704,13 +716,20 @@ function resolveCodeVariables(
blockNameMapping: Record<string, string> = {},
blockOutputSchemas: Record<string, OutputSchema> = {},
workflowVariables: Record<string, unknown> = {},
language = 'javascript'
language = 'javascript',
onResolvedSecret?: (name: string) => void
): { resolvedCode: string; contextVariables: Record<string, unknown> } {
let resolvedCode = code
const contextVariables: Record<string, unknown> = {}
resolvedCode = resolveWorkflowVariables(resolvedCode, workflowVariables, contextVariables)
resolvedCode = resolveEnvironmentVariables(resolvedCode, params, envVars, contextVariables)
resolvedCode = resolveEnvironmentVariables(
resolvedCode,
params,
envVars,
contextVariables,
onResolvedSecret
)
resolvedCode = resolveTagVariables(
resolvedCode,
blockData,
@@ -796,6 +815,8 @@ interface FunctionRouteExecutionContext {
allowLargeValueWorkflowScope?: boolean
userId?: string
requestId: string
resolvedSecretNames: Set<string>
includePrivateResolvedSecretNames: boolean
}
function asRecord(value: unknown): Record<string, unknown> {
@@ -931,7 +952,7 @@ async function functionJsonResponse<T>(
context: FunctionRouteExecutionContext,
init?: ResponseInit
) {
return NextResponse.json(
const response = NextResponse.json(
await compactFunctionRouteBody(
{
...body,
@@ -942,6 +963,52 @@ async function functionJsonResponse<T>(
),
init
)
return appendResolvedSecretNames(response, context)
}
function getPrivateResolvedSecretNames(context: FunctionRouteExecutionContext): string[] | null {
if (context.resolvedSecretNames.size > MAX_PRIVATE_RESOLVED_SECRET_NAMES) return null
const names = Array.from(context.resolvedSecretNames).sort()
let bytes = 0
for (const name of names) {
bytes += Buffer.byteLength(name, 'utf8')
if (bytes > MAX_PRIVATE_RESOLVED_SECRET_NAMES_BYTES) return null
}
return names
}
async function appendResolvedSecretNames(
response: NextResponse,
context: FunctionRouteExecutionContext
): Promise<NextResponse> {
const names = context.includePrivateResolvedSecretNames
? getPrivateResolvedSecretNames(context)
: null
return appendPrivateResolvedSecretNames(response, names)
}
async function appendPrivateResolvedSecretNames(
response: NextResponse,
names: string[] | null
): Promise<NextResponse> {
if (!names) return response
try {
const body = (await response.clone().json()) as Record<string, unknown>
const headers = new Headers(response.headers)
headers.delete('content-length')
headers.set(PRIVATE_TOOL_METADATA_RESPONSE_HEADER, RESOLVED_SECRET_NAMES_METADATA_V1)
return NextResponse.json(
{
...body,
[RESOLVED_SECRET_NAMES_FIELD]: names,
},
{ status: response.status, statusText: response.statusText, headers }
)
} catch {
return response
}
}
/**
@@ -1411,6 +1478,7 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
let resolvedCode = '' // Store resolved code for error reporting
let sourceCodeForErrors: string | undefined
let routeContext: FunctionRouteExecutionContext | undefined
let includePrivateResolvedSecretNames = false
try {
const auth = await checkInternalAuth(req)
@@ -1419,8 +1487,18 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
return NextResponse.json({ error: auth.error || 'Unauthorized' }, { status: 401 })
}
includePrivateResolvedSecretNames = requestsPrivateToolMetadata(
req.headers,
RESOLVED_SECRET_NAMES_METADATA_V1
)
const parsed = await parseRequest(functionExecuteContract, req, {})
if (!parsed.success) return parsed.response
if (!parsed.success) {
return appendPrivateResolvedSecretNames(
parsed.response,
includePrivateResolvedSecretNames ? [] : null
)
}
const { body } = parsed.data
const { DEFAULT_EXECUTION_TIMEOUT_MS } = await import('@/lib/execution/constants')
@@ -1473,12 +1551,15 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
.map((file) => file.sandboxPath)
.filter((path): path is string => Boolean(path))
if (outputSandboxPaths.length > MAX_SANDBOX_OUTPUT_FILES) {
return NextResponse.json(
{
success: false,
error: `Too many sandbox output files requested (${outputSandboxPaths.length}). Maximum is ${MAX_SANDBOX_OUTPUT_FILES}.`,
},
{ status: 400 }
return appendPrivateResolvedSecretNames(
NextResponse.json(
{
success: false,
error: `Too many sandbox output files requested (${outputSandboxPaths.length}). Maximum is ${MAX_SANDBOX_OUTPUT_FILES}.`,
},
{ status: 400 }
),
includePrivateResolvedSecretNames ? [] : null
)
}
@@ -1504,6 +1585,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
allowLargeValueWorkflowScope,
userId: auth.userId,
requestId,
resolvedSecretNames: new Set<string>(),
includePrivateResolvedSecretNames,
}
const lang = isValidCodeLanguage(language) ? language : DEFAULT_CODE_LANGUAGE
@@ -1512,7 +1595,12 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
if (lang === CodeLanguage.Shell) {
// For shell, env vars are injected as OS env vars via shellEnvs.
// Replace {{VAR}} placeholders with $VAR so the shell can access them natively.
resolvedCode = code.replace(/\{\{([A-Za-z_][A-Za-z0-9_]*)\}\}/g, '$$$1')
resolvedCode = code.replace(/\{\{([A-Za-z_][A-Za-z0-9_]*)\}\}/g, (_match, name) => {
if (Object.hasOwn(envVars, name)) {
routeContext?.resolvedSecretNames.add(name)
}
return `$${name}`
})
// Carry pre-resolved block output variables (e.g. __blockRef_N) so they can be
// injected as shell env vars below. The executor replaces block references in the
// code with these names, so the values must be present at runtime.
@@ -1526,7 +1614,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
blockNameMapping,
blockOutputSchemas,
workflowVariables,
lang
lang,
(name) => routeContext?.resolvedSecretNames.add(name)
)
resolvedCode = codeResolution.resolvedCode
// Merge pre-resolved block output variables from the executor. These take precedence
@@ -1625,7 +1714,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
stdout: shellStdout,
executionTime,
})
if (fileExportResponse) return fileExportResponse
if (fileExportResponse) {
return appendResolvedSecretNames(fileExportResponse, routeContext)
}
}
return functionJsonResponse(
@@ -1794,7 +1885,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
stdout,
executionTime,
})
if (fileExportResponse) return fileExportResponse
if (fileExportResponse) {
return appendResolvedSecretNames(fileExportResponse, routeContext)
}
}
return functionJsonResponse(
@@ -1884,7 +1977,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
stdout,
executionTime,
})
if (fileExportResponse) return fileExportResponse
if (fileExportResponse) {
return appendResolvedSecretNames(fileExportResponse, routeContext)
}
}
return functionJsonResponse(
@@ -2043,17 +2138,20 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
{ status: error.statusCode }
)
}
return NextResponse.json(
{
success: false,
error: error.message,
output: {
result: null,
stdout: cleanStdout(stdout),
executionTime,
return appendPrivateResolvedSecretNames(
NextResponse.json(
{
success: false,
error: error.message,
output: {
result: null,
stdout: cleanStdout(stdout),
executionTime,
},
},
},
{ status: error.statusCode }
{ status: error.statusCode }
),
includePrivateResolvedSecretNames ? [] : null
)
}
@@ -2072,7 +2170,10 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
}
return routeContext
? functionJsonResponse(killResponse, routeContext, { status: 500 })
: NextResponse.json(killResponse, { status: 500 })
: appendPrivateResolvedSecretNames(
NextResponse.json(killResponse, { status: 500 }),
includePrivateResolvedSecretNames ? [] : null
)
}
logger.error(`[${requestId}] Function execution failed`, {
@@ -2120,6 +2221,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
return functionJsonResponse(errorResponse, routeContext, { status: 500 })
}
return NextResponse.json(errorResponse, { status: 500 })
return appendPrivateResolvedSecretNames(
NextResponse.json(errorResponse, { status: 500 }),
includePrivateResolvedSecretNames ? [] : null
)
}
})
+10 -6
View File
@@ -8,7 +8,7 @@ import { getSession } from '@/lib/auth'
import { MATERIALIZE_CONCURRENCY, mapWithConcurrency } from '@/lib/core/utils/concurrency'
import { neutralizeCsvFormula } from '@/lib/core/utils/csv'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
import { buildFilterConditions, LogFilterParamsSchema } from '@/lib/logs/filters'
import { expandFolderIdsWithDescendants } from '@/lib/logs/folder-expansion'
import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils'
@@ -112,11 +112,15 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
rows as any[],
MATERIALIZE_CONCURRENCY,
(r) =>
materializeExecutionData(r.executionData as Record<string, unknown> | null, {
workspaceId: params.workspaceId,
workflowId: r.workflowId,
executionId: r.executionId,
})
materializeExecutionDataForDisplay(
r.executionData as Record<string, unknown> | null,
{
workspaceId: params.workspaceId,
workflowId: r.workflowId,
executionId: r.executionId,
userId: session.user.id,
}
)
)
for (let j = 0; j < rows.length; j++) {
@@ -0,0 +1,199 @@
/**
* @vitest-environment node
*/
import { NextRequest } from 'next/server'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockDiscoverServerTools, mockExecuteTool, mockReadResponseToBufferWithLimit } = vi.hoisted(
() => ({
mockDiscoverServerTools: vi.fn(),
mockExecuteTool: vi.fn(),
mockReadResponseToBufferWithLimit: vi.fn(),
})
)
vi.mock('@/lib/core/utils/stream-limits', () => ({
readResponseToBufferWithLimit: mockReadResponseToBufferWithLimit,
}))
vi.mock('@/lib/mcp/middleware', () => ({
withMcpAuth:
() =>
(
handler: (
request: NextRequest,
context: {
userId: string
workspaceId: string
requestId: string
authType: 'internal_jwt' | 'session'
},
routeContext: { params: Promise<Record<string, string>> }
) => Promise<Response>
) =>
(request: NextRequest) =>
handler(
request,
{
userId: 'user-1',
workspaceId: 'workspace-1',
requestId: 'request-1',
authType: request.headers.has('x-test-session') ? 'session' : 'internal_jwt',
},
{ params: Promise.resolve({}) }
),
readMcpJsonBodyWithLimit: (request: NextRequest) => request.json(),
mcpBodyReadErrorResponse: () => null,
}))
vi.mock('@/lib/mcp/service', () => ({
mcpService: {
discoverServerTools: mockDiscoverServerTools,
executeTool: mockExecuteTool,
},
}))
vi.mock('@/lib/billing/core/billing-attribution', () => ({
requireBillingAttributionHeader: () => ({ payerSubscription: { plan: 'pro' } }),
resolveBillingAttribution: async () => ({ payerSubscription: { plan: 'pro' } }),
}))
vi.mock('@/lib/core/execution-limits', () => ({
DEFAULT_EXECUTION_TIMEOUT_MS: 30_000,
getExecutionTimeout: () => 0,
}))
vi.mock('@/ee/access-control/utils/permission-check', () => ({
assertPermissionsAllowed: async () => {},
McpToolsNotAllowedError: class McpToolsNotAllowedError extends Error {},
}))
vi.mock('@/lib/core/telemetry', () => ({
PlatformEvents: { mcpToolExecuted: vi.fn() },
}))
import { POST } from '@/app/api/mcp/tools/execute/route'
const URL = 'http://localhost/api/mcp/tools/execute'
const REQUEST_BODY = {
workspaceId: 'workspace-1',
serverId: 'server-1',
toolName: 'example_tool',
arguments: {},
}
function createRequest(headers: Record<string, string> = {}): NextRequest {
return new NextRequest(URL, {
method: 'POST',
headers: { 'content-type': 'application/json', ...headers },
body: JSON.stringify(REQUEST_BODY),
})
}
describe('MCP tool execution private secret provenance', () => {
beforeEach(() => {
vi.clearAllMocks()
mockDiscoverServerTools.mockResolvedValue([{ name: 'example_tool', inputSchema: {} }])
mockExecuteTool.mockResolvedValue({ content: [{ type: 'text', text: 'ok' }] })
mockReadResponseToBufferWithLimit.mockImplementation(async (response: Response) =>
Buffer.from(await response.arrayBuffer())
)
})
it('returns fail-closed scoped provenance only to an authenticated internal caller', async () => {
mockDiscoverServerTools.mockImplementationOnce(
async (
_userId: string,
_serverId: string,
_workspaceId: string,
_forceRefresh: boolean,
report: (value: unknown) => void
) => {
report({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
return [{ name: 'example_tool', inputSchema: {} }]
}
)
mockExecuteTool.mockImplementationOnce(
async (
_userId: string,
_serverId: string,
_toolCall: unknown,
_workspaceId: string,
_headers: unknown,
report: (value: unknown) => void
) => {
report({
version: 1,
complete: true,
entries: [{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
return { content: [{ type: 'text', text: 'ok' }] }
}
)
const request = createRequest({
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
})
const response = await POST(request, {})
const body = (await response.json()) as Record<string, unknown>
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
'resolved-secret-provenance-v1'
)
expect(body.__resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('does not expose private provenance metadata to a session caller', async () => {
const request = createRequest({
'x-test-session': 'true',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
})
const response = await POST(request, {})
const body = (await response.json()) as Record<string, unknown>
expect(response.headers.has('x-sim-private-tool-metadata')).toBe(false)
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
expect(mockDiscoverServerTools.mock.calls[0]?.[4]).toBeUndefined()
expect(mockExecuteTool.mock.calls[0]?.[5]).toBeUndefined()
})
it('preserves the functional response when private provenance cannot be attached', async () => {
mockReadResponseToBufferWithLimit.mockRejectedValueOnce(new Error('Response exceeds limit'))
mockExecuteTool.mockResolvedValueOnce({
content: [{ type: 'text', text: 'unchanged' }],
})
const request = createRequest({
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
})
const response = await POST(request, {})
const body = (await response.json()) as Record<string, unknown>
expect(response.status).toBe(200)
expect(response.ok).toBe(true)
expect(response.headers.has('x-sim-private-tool-metadata')).toBe(false)
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
expect(body).toMatchObject({
success: true,
data: {
success: true,
output: { content: [{ type: 'text' }] },
},
})
expect(
(body.data as { output: { content: Array<{ text?: unknown }> } }).output.content[0]?.text
).toBe('unchanged')
})
})
+276 -206
View File
@@ -11,8 +11,15 @@ import {
} from '@/lib/billing/core/billing-attribution'
import { getExecutionTimeout } from '@/lib/core/execution-limits'
import type { SubscriptionPlan } from '@/lib/core/rate-limiter/types'
import { readResponseToBufferWithLimit } from '@/lib/core/utils/stream-limits'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { SIM_VIA_HEADER } from '@/lib/execution/call-chain'
import {
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
RESOLVED_SECRET_PROVENANCE_FIELD,
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
requestsPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
import {
mcpBodyReadErrorResponse,
readMcpJsonBodyWithLimit,
@@ -31,8 +38,13 @@ import {
assertPermissionsAllowed,
McpToolsNotAllowedError,
} from '@/ee/access-control/utils/permission-check'
import {
ResolvedSecretTraceProvenanceAccumulator,
type ResolvedSecretTraceProvenanceV1,
} from '@/executor/utils/resolved-secret-trace-registry'
const logger = createLogger('McpToolExecutionAPI')
const MAX_PRIVATE_MCP_RESPONSE_BYTES = 10 * 1024 * 1024
export const dynamic = 'force-dynamic'
@@ -55,6 +67,35 @@ function hasType(prop: unknown): prop is SchemaProperty {
return typeof prop === 'object' && prop !== null && 'type' in prop
}
async function attachPrivateProvenance(
response: NextResponse,
provenance: ResolvedSecretTraceProvenanceAccumulator
): Promise<NextResponse> {
let payload: Record<string, unknown>
try {
const body = await readResponseToBufferWithLimit(response.clone(), {
maxBytes: MAX_PRIVATE_MCP_RESPONSE_BYTES,
label: 'MCP private metadata response',
allowNoBodyFallback: true,
})
const parsed: unknown = JSON.parse(body.toString('utf8'))
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error('MCP response is not a JSON object')
}
payload = parsed as Record<string, unknown>
} catch {
return response
}
const headers = new Headers(response.headers)
headers.delete('content-length')
headers.set(PRIVATE_TOOL_METADATA_RESPONSE_HEADER, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
return NextResponse.json(
{ ...payload, [RESOLVED_SECRET_PROVENANCE_FIELD]: provenance.exportProvenance() },
{ status: response.status, headers }
)
}
/**
* POST - Execute a tool on an MCP server
*/
@@ -62,239 +103,268 @@ export const POST = withRouteHandler(
withMcpAuth('read')(
async (request: NextRequest, { userId, workspaceId, requestId, authType }) => {
let serverId: string | undefined
try {
const rawBody = await readMcpJsonBodyWithLimit(request)
const parsedBody = mcpToolExecutionBodySchema.safeParse(rawBody)
if (!parsedBody.success) {
return createMcpErrorResponse(parsedBody.error, 'Invalid request format', 400)
}
const body = parsedBody.data
logger.info(`[${requestId}] MCP tool execution request received`, {
hasAuthHeader: !!request.headers.get('authorization'),
bodyKeys: Object.keys(body),
serverId: body.serverId,
toolName: body.toolName,
hasWorkflowId: !!body.workflowId,
workflowId: body.workflowId,
userId: userId,
})
const { toolName, arguments: rawArgs } = body
serverId = body.serverId
const args = rawArgs || {}
const includePrivateProvenance =
authType === AuthType.INTERNAL_JWT &&
requestsPrivateToolMetadata(request.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
const resolvedSecretTraceProvenance = includePrivateProvenance
? new ResolvedSecretTraceProvenanceAccumulator({ userId, workspaceId })
: undefined
const recordProvenance = resolvedSecretTraceProvenance
? (provenance: ResolvedSecretTraceProvenanceV1): void => {
resolvedSecretTraceProvenance.record(provenance)
}
: undefined
const response = await (async (): Promise<NextResponse> => {
try {
await assertPermissionsAllowed({
userId,
workspaceId,
toolKind: 'mcp',
const rawBody = await readMcpJsonBodyWithLimit(request)
const parsedBody = mcpToolExecutionBodySchema.safeParse(rawBody)
if (!parsedBody.success) {
return createMcpErrorResponse(parsedBody.error, 'Invalid request format', 400)
}
const body = parsedBody.data
logger.info(`[${requestId}] MCP tool execution request received`, {
hasAuthHeader: !!request.headers.get('authorization'),
bodyKeys: Object.keys(body),
serverId: body.serverId,
toolName: body.toolName,
hasWorkflowId: !!body.workflowId,
workflowId: body.workflowId,
userId: userId,
})
} catch (err) {
if (err instanceof McpToolsNotAllowedError) {
return createMcpErrorResponse(err, err.message, 403)
}
throw err
}
logger.info(
`[${requestId}] Executing tool ${toolName} on server ${serverId} for user ${userId} in workspace ${workspaceId}`
)
const { toolName, arguments: rawArgs } = body
serverId = body.serverId
const args = rawArgs || {}
let tool: McpTool | null = null
try {
const tools = await mcpService.discoverServerTools(userId, serverId, workspaceId)
tool = tools.find((t) => t.name === toolName) ?? null
if (!tool) {
logger.warn(`[${requestId}] Tool ${toolName} not found on server ${serverId}`, {
availableTools: tools.map((t) => t.name),
try {
await assertPermissionsAllowed({
userId,
workspaceId,
toolKind: 'mcp',
})
return createMcpErrorResponse(
new Error('Tool not found'),
'Tool not found on the specified server',
404
)
} catch (err) {
if (err instanceof McpToolsNotAllowedError) {
return createMcpErrorResponse(err, err.message, 403)
}
throw err
}
if (tool.inputSchema?.properties) {
for (const [paramName, paramSchema] of Object.entries(tool.inputSchema.properties)) {
const schema = hasType(paramSchema) ? paramSchema : null
if (!schema) continue
const value = args[paramName]
logger.info(
`[${requestId}] Executing tool ${toolName} on server ${serverId} for user ${userId} in workspace ${workspaceId}`
)
if (value === undefined || value === null) {
continue
}
let tool: McpTool | null = null
try {
const tools = await mcpService.discoverServerTools(
userId,
serverId,
workspaceId,
false,
recordProvenance
)
tool = tools.find((t) => t.name === toolName) ?? null
if (
(schema.type === 'number' || schema.type === 'integer') &&
typeof value === 'string'
) {
const numValue =
schema.type === 'integer' ? Number.parseInt(value) : Number.parseFloat(value)
if (!Number.isNaN(numValue)) {
args[paramName] = numValue
if (!tool) {
logger.warn(`[${requestId}] Tool ${toolName} not found on server ${serverId}`, {
availableTools: tools.map((t) => t.name),
})
return createMcpErrorResponse(
new Error('Tool not found'),
'Tool not found on the specified server',
404
)
}
if (tool.inputSchema?.properties) {
for (const [paramName, paramSchema] of Object.entries(tool.inputSchema.properties)) {
const schema = hasType(paramSchema) ? paramSchema : null
if (!schema) continue
const value = args[paramName]
if (value === undefined || value === null) {
continue
}
} else if (schema.type === 'boolean' && typeof value === 'string') {
if (value.toLowerCase() === 'true') {
args[paramName] = true
} else if (value.toLowerCase() === 'false') {
args[paramName] = false
}
} else if (schema.type === 'array' && typeof value === 'string') {
const stringValue = value.trim()
if (stringValue) {
try {
const parsed = JSON.parse(stringValue)
if (Array.isArray(parsed)) {
args[paramName] = parsed
} else {
args[paramName] = [parsed]
if (
(schema.type === 'number' || schema.type === 'integer') &&
typeof value === 'string'
) {
const numValue =
schema.type === 'integer' ? Number.parseInt(value) : Number.parseFloat(value)
if (!Number.isNaN(numValue)) {
args[paramName] = numValue
}
} else if (schema.type === 'boolean' && typeof value === 'string') {
if (value.toLowerCase() === 'true') {
args[paramName] = true
} else if (value.toLowerCase() === 'false') {
args[paramName] = false
}
} else if (schema.type === 'array' && typeof value === 'string') {
const stringValue = value.trim()
if (stringValue) {
try {
const parsed = JSON.parse(stringValue)
if (Array.isArray(parsed)) {
args[paramName] = parsed
} else {
args[paramName] = [parsed]
}
} catch {
if (stringValue.includes(',')) {
args[paramName] = stringValue
.split(',')
.map((item) => item.trim())
.filter((item) => item)
} else {
args[paramName] = [stringValue]
}
}
} catch {
if (stringValue.includes(',')) {
args[paramName] = stringValue
.split(',')
.map((item) => item.trim())
.filter((item) => item)
} else {
args[paramName] = [stringValue]
}
} else {
args[paramName] = []
}
} else {
args[paramName] = []
}
}
}
} catch (error) {
logger.warn(
`[${requestId}] Failed to discover tools for validation, proceeding without schema`,
error
)
}
} catch (error) {
logger.warn(
`[${requestId}] Failed to discover tools for validation, proceeding without schema`,
error
)
}
if (tool) {
const validationError = validateToolArguments(tool, args)
if (validationError) {
logger.warn(`[${requestId}] Tool validation failed: ${validationError}`)
if (tool) {
const validationError = validateToolArguments(tool, args)
if (validationError) {
logger.warn(`[${requestId}] Tool validation failed: ${validationError}`)
return createMcpErrorResponse(
new Error(`Invalid arguments for tool ${toolName}: ${validationError}`),
'Invalid tool arguments',
400
)
}
}
const toolCall: McpToolCall = {
name: toolName,
arguments: args,
}
const billingAttribution =
authType === AuthType.INTERNAL_JWT
? requireBillingAttributionHeader(request.headers, {
actorUserId: userId,
workspaceId,
})
: await resolveBillingAttribution({ actorUserId: userId, workspaceId })
const executionTimeout = getExecutionTimeout(
billingAttribution.payerSubscription?.plan as SubscriptionPlan | undefined,
'sync'
)
const simViaHeader = request.headers.get(SIM_VIA_HEADER)
const extraHeaders: Record<string, string> = {}
if (simViaHeader) {
extraHeaders[SIM_VIA_HEADER] = simViaHeader
}
let timeoutHandle: ReturnType<typeof setTimeout> | undefined
const executePromise = mcpService.executeTool(
userId,
serverId,
toolCall,
workspaceId,
extraHeaders,
recordProvenance
)
// A zero timeout means "no timeout" (billing-disabled deployments).
const result = await (executionTimeout > 0
? Promise.race([
executePromise,
new Promise<never>((_, reject) => {
timeoutHandle = setTimeout(
() => reject(new Error('Tool execution timeout')),
executionTimeout
)
}),
])
: executePromise
).finally(() => {
if (timeoutHandle !== undefined) clearTimeout(timeoutHandle)
})
const transformedResult = transformToolResult(result)
if (result.isError) {
logger.warn(
`[${requestId}] Tool execution returned error for ${toolName} on ${serverId}`
)
return createMcpErrorResponse(
new Error(`Invalid arguments for tool ${toolName}: ${validationError}`),
'Invalid tool arguments',
transformedResult,
transformedResult.error || 'Tool execution failed',
400
)
}
}
logger.info(`[${requestId}] Successfully executed tool ${toolName} on server ${serverId}`)
const toolCall: McpToolCall = {
name: toolName,
arguments: args,
}
try {
const { PlatformEvents } = await import('@/lib/core/telemetry')
PlatformEvents.mcpToolExecuted({
serverId,
toolName,
status: 'success',
workspaceId,
})
} catch (error) {
logger.warn('Failed to record MCP tool execution telemetry', {
error: getErrorMessage(error),
serverId,
toolName,
workspaceId,
})
}
const billingAttribution =
authType === AuthType.INTERNAL_JWT
? requireBillingAttributionHeader(request.headers, {
actorUserId: userId,
workspaceId,
})
: await resolveBillingAttribution({ actorUserId: userId, workspaceId })
const executionTimeout = getExecutionTimeout(
billingAttribution.payerSubscription?.plan as SubscriptionPlan | undefined,
'sync'
)
const simViaHeader = request.headers.get(SIM_VIA_HEADER)
const extraHeaders: Record<string, string> = {}
if (simViaHeader) {
extraHeaders[SIM_VIA_HEADER] = simViaHeader
}
let timeoutHandle: ReturnType<typeof setTimeout> | undefined
const executePromise = mcpService.executeTool(
userId,
serverId,
toolCall,
workspaceId,
extraHeaders
)
// A zero timeout means "no timeout" (billing-disabled deployments).
const result = await (executionTimeout > 0
? Promise.race([
executePromise,
new Promise<never>((_, reject) => {
timeoutHandle = setTimeout(
() => reject(new Error('Tool execution timeout')),
executionTimeout
)
}),
])
: executePromise
).finally(() => {
if (timeoutHandle !== undefined) clearTimeout(timeoutHandle)
})
const transformedResult = transformToolResult(result)
if (result.isError) {
logger.warn(`[${requestId}] Tool execution returned error for ${toolName} on ${serverId}`)
return createMcpErrorResponse(
transformedResult,
transformedResult.error || 'Tool execution failed',
400
)
}
logger.info(`[${requestId}] Successfully executed tool ${toolName} on server ${serverId}`)
try {
const { PlatformEvents } = await import('@/lib/core/telemetry')
PlatformEvents.mcpToolExecuted({
serverId,
toolName,
status: 'success',
workspaceId,
})
return createMcpSuccessResponse(transformedResult)
} catch (error) {
logger.warn('Failed to record MCP tool execution telemetry', {
error: getErrorMessage(error),
serverId,
toolName,
workspaceId,
})
}
return createMcpSuccessResponse(transformedResult)
} catch (error) {
const bodyErrorResponse = mcpBodyReadErrorResponse(error, request)
if (bodyErrorResponse) return bodyErrorResponse
if (
error instanceof McpOauthAuthorizationRequiredError ||
error instanceof McpOauthRedirectRequired ||
error instanceof UnauthorizedError
) {
const errorServerId =
error instanceof McpOauthAuthorizationRequiredError ? error.serverId : serverId
logger.warn(`[${requestId}] OAuth re-authorization required for MCP tool execution`, {
serverId: errorServerId,
})
return NextResponse.json(
{
success: false,
error: 'OAuth re-authorization required',
code: 'reauth_required',
if (getErrorMessage(error) === 'Tool execution timeout') {
resolvedSecretTraceProvenance?.markIncomplete()
}
const bodyErrorResponse = mcpBodyReadErrorResponse(error, request)
if (bodyErrorResponse) return bodyErrorResponse
if (
error instanceof McpOauthAuthorizationRequiredError ||
error instanceof McpOauthRedirectRequired ||
error instanceof UnauthorizedError
) {
const errorServerId =
error instanceof McpOauthAuthorizationRequiredError ? error.serverId : serverId
logger.warn(`[${requestId}] OAuth re-authorization required for MCP tool execution`, {
serverId: errorServerId,
},
{ status: 401 }
)
})
return NextResponse.json(
{
success: false,
error: 'OAuth re-authorization required',
code: 'reauth_required',
serverId: errorServerId,
},
{ status: 401 }
)
}
logger.error(`[${requestId}] Error executing MCP tool:`, error)
const { message, status } = categorizeError(error)
return createMcpErrorResponse(new Error(message), message, status)
}
})()
logger.error(`[${requestId}] Error executing MCP tool:`, error)
const { message, status } = categorizeError(error)
return createMcpErrorResponse(new Error(message), message, status)
}
return resolvedSecretTraceProvenance
? attachPrivateProvenance(response, resolvedSecretTraceProvenance)
: response
}
)
)
@@ -1,7 +1,98 @@
import { describe, expect, it } from 'vitest'
/**
* @vitest-environment node
*/
import { createMockRequest } from '@sim/testing'
import { beforeEach, describe, expect, it, vi } from 'vitest'
const {
mockAssertActiveWorkspaceAccess,
mockBuildIntegrationToolSchemas,
mockBuildSelectedMcpToolSchemas,
mockBuildTaggedMcpToolSchemas,
mockCheckInternalAuth,
mockComputeWorkspaceEntitlements,
mockDecryptSecret,
mockGenerateWorkspaceContext,
mockGetPersonalAndWorkspaceEnv,
mockProcessContextsServer,
mockRequestExplicitStreamAbort,
mockRequireBillingAttributionHeader,
mockRunHeadlessCopilotLifecycle,
} = vi.hoisted(() => ({
mockAssertActiveWorkspaceAccess: vi.fn(),
mockBuildIntegrationToolSchemas: vi.fn(),
mockBuildSelectedMcpToolSchemas: vi.fn(),
mockBuildTaggedMcpToolSchemas: vi.fn(),
mockCheckInternalAuth: vi.fn(),
mockComputeWorkspaceEntitlements: vi.fn(),
mockDecryptSecret: vi.fn(),
mockGenerateWorkspaceContext: vi.fn(),
mockGetPersonalAndWorkspaceEnv: vi.fn(),
mockProcessContextsServer: vi.fn(),
mockRequestExplicitStreamAbort: vi.fn(),
mockRequireBillingAttributionHeader: vi.fn(),
mockRunHeadlessCopilotLifecycle: vi.fn(),
}))
vi.mock('@/lib/core/security/encryption', () => ({
decryptSecret: mockDecryptSecret,
}))
vi.mock('@/lib/auth/hybrid', () => ({
checkInternalAuth: mockCheckInternalAuth,
}))
vi.mock('@/lib/billing/core/billing-attribution', () => ({
requireBillingAttributionHeader: mockRequireBillingAttributionHeader,
}))
vi.mock('@/lib/copilot/chat/payload', () => ({
buildIntegrationToolSchemas: mockBuildIntegrationToolSchemas,
}))
vi.mock('@/lib/copilot/chat/process-contents', () => ({
processContextsServer: mockProcessContextsServer,
}))
vi.mock('@/lib/copilot/chat/workspace-context', () => ({
generateWorkspaceContext: mockGenerateWorkspaceContext,
}))
vi.mock('@/lib/copilot/entitlements', () => ({
computeWorkspaceEntitlements: mockComputeWorkspaceEntitlements,
}))
vi.mock('@/lib/copilot/mcp-tools', () => ({
buildSelectedMcpToolSchemas: mockBuildSelectedMcpToolSchemas,
buildTaggedMcpToolSchemas: mockBuildTaggedMcpToolSchemas,
}))
vi.mock('@/lib/copilot/request/lifecycle/headless', () => ({
runHeadlessCopilotLifecycle: mockRunHeadlessCopilotLifecycle,
}))
vi.mock('@/lib/copilot/request/session/explicit-abort', () => ({
requestExplicitStreamAbort: mockRequestExplicitStreamAbort,
}))
vi.mock('@/lib/core/config/env-flags', () => ({
isDocSandboxEnabled: false,
}))
vi.mock('@/lib/environment/utils', () => ({
getPersonalAndWorkspaceEnv: mockGetPersonalAndWorkspaceEnv,
}))
vi.mock('@/lib/workspaces/permissions/utils', () => ({
assertActiveWorkspaceAccess: mockAssertActiveWorkspaceAccess,
isWorkspaceAccessDeniedError: vi.fn(() => false),
}))
import type { CopilotLifecycleOptions } from '@/lib/copilot/request/lifecycle/run'
import {
buildExecuteResponsePayload,
CALLER_VISIBLE_SERVER_TOOLS,
POST,
} from '@/app/api/mothership/execute/route'
type Payload = Parameters<typeof buildExecuteResponsePayload>[0]
@@ -48,3 +139,362 @@ describe('buildExecuteResponsePayload', () => {
expect(CALLER_VISIBLE_SERVER_TOOLS.has('complete_scheduled_task')).toBe(true)
})
})
describe('mothership private trace provenance transport', () => {
const requestBody = {
messages: [{ role: 'user', content: 'hello' }],
workspaceId: 'workspace-1',
userId: 'user-1',
chatId: 'chat-1',
messageId: 'message-1',
requestId: 'request-1',
}
beforeEach(() => {
vi.clearAllMocks()
mockCheckInternalAuth.mockResolvedValue({
success: true,
userId: 'user-1',
authType: 'internal_jwt',
})
mockAssertActiveWorkspaceAccess.mockResolvedValue({ permission: 'write' })
mockRequireBillingAttributionHeader.mockReturnValue({
actorUserId: 'user-1',
workspaceId: 'workspace-1',
})
mockGetPersonalAndWorkspaceEnv.mockResolvedValue({
personalEncrypted: { API_KEY: 'encrypted-secret' },
workspaceEncrypted: {},
personalDecrypted: { API_KEY: 'secret-value' },
workspaceDecrypted: {},
decryptionFailures: [],
})
mockGenerateWorkspaceContext.mockResolvedValue({})
mockBuildIntegrationToolSchemas.mockResolvedValue([])
mockBuildSelectedMcpToolSchemas.mockResolvedValue([])
mockBuildTaggedMcpToolSchemas.mockResolvedValue([])
mockComputeWorkspaceEntitlements.mockResolvedValue([])
mockDecryptSecret.mockResolvedValue({ decrypted: 'secret-value' })
mockProcessContextsServer.mockResolvedValue([])
mockRequestExplicitStreamAbort.mockResolvedValue(undefined)
})
function successResult() {
return {
success: true,
content: 'secret-value',
contentBlocks: [],
toolCalls: [],
chatId: 'chat-1',
}
}
function activateSecret(options: CopilotLifecycleOptions): void {
options.resolvedSecretTraceRegistry?.recordResolved('API_KEY', 'secret-value')
}
it('does not expose private provenance unless the internal caller requests it', async () => {
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
activateSecret(options)
return successResult()
}
)
const response = await POST(
createMockRequest(
'POST',
requestBody,
{ Authorization: 'Bearer internal', 'x-sim-billing-attribution': 'billing' },
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect(response.status).toBe(200)
expect(response.headers.get('x-sim-private-tool-metadata')).toBeNull()
expect(body.content).toBe('secret-value')
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
expect(mockGetPersonalAndWorkspaceEnv).not.toHaveBeenCalled()
expect(mockRunHeadlessCopilotLifecycle).toHaveBeenCalledWith(
expect.any(Object),
expect.objectContaining({ resolvedSecretTraceRegistry: undefined })
)
})
it('keeps execution functional and fails trace provenance closed when catalog setup fails', async () => {
mockGetPersonalAndWorkspaceEnv.mockRejectedValueOnce(new Error('catalog unavailable'))
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
expect(options.resolvedSecretTraceRegistry?.isComplete()).toBe(false)
return successResult()
}
)
const response = await POST(
createMockRequest(
'POST',
requestBody,
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
},
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect(response.status).toBe(200)
expect(body.content).toBe('secret-value')
expect(body.__resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('fails provenance closed without changing a runtime value that rotated after catalog load', async () => {
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
expect(
options.resolvedSecretTraceRegistry?.recordResolved('API_KEY', 'rotated-secret-value')
).toBe(false)
return { ...successResult(), content: 'rotated-secret-value' }
}
)
const response = await POST(
createMockRequest(
'POST',
requestBody,
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
},
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect(response.status).toBe(200)
expect(body.content).toBe('rotated-secret-value')
expect(body.__resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('returns encrypted provenance on a marker-gated successful request', async () => {
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
activateSecret(options)
return successResult()
}
)
const response = await POST(
createMockRequest(
'POST',
requestBody,
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
},
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
'resolved-secret-provenance-v1'
)
expect(body.content).toBe('secret-value')
expect(body.__resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: true,
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
expect(JSON.stringify(body.__resolvedSecretTraceProvenance)).not.toContain('secret-value')
})
it('imports MCP schema-discovery provenance before starting the lifecycle', async () => {
const provenance = {
version: 1,
complete: true,
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
}
mockBuildTaggedMcpToolSchemas.mockImplementationOnce(
async (
_userId: string,
_workspaceId: string,
_serverIds: string[],
report: (value: unknown) => void
) => {
report(provenance)
return []
}
)
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
expect(options.resolvedSecretTraceRegistry?.exportProvenance()).toEqual(provenance)
expect(JSON.stringify(payload)).not.toContain('encrypted-secret')
expect(JSON.stringify(payload)).not.toContain('__resolvedSecretTraceProvenance')
return successResult()
}
)
const response = await POST(
createMockRequest(
'POST',
{
...requestBody,
contexts: [{ kind: 'mcp', label: 'Docs', serverId: 'server-1' }],
},
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
},
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect({ status: response.status, provenance: body.__resolvedSecretTraceProvenance }).toEqual({
status: 200,
provenance,
})
})
it('marks the lifecycle registry incomplete for malformed MCP discovery provenance', async () => {
mockBuildTaggedMcpToolSchemas.mockImplementationOnce(
async (
_userId: string,
_workspaceId: string,
_serverIds: string[],
report: (value: unknown) => void
) => {
report({ version: 1, complete: true, entries: 'invalid' })
return []
}
)
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
expect(options.resolvedSecretTraceRegistry?.isComplete()).toBe(false)
return successResult()
}
)
const response = await POST(
createMockRequest(
'POST',
{
...requestBody,
contexts: [{ kind: 'mcp', label: 'Docs', serverId: 'server-1' }],
},
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
},
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect(response.status).toBe(200)
expect(body.__resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('returns encrypted provenance with marker-gated failures', async () => {
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
activateSecret(options)
return {
...successResult(),
success: false,
error: 'failed with secret-value',
content: 'secret-value',
}
}
)
const response = await POST(
createMockRequest(
'POST',
requestBody,
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
},
'http://localhost:3000/api/mothership/execute'
)
)
const body = await response.json()
expect(response.status).toBe(500)
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
'resolved-secret-provenance-v1'
)
expect(body.content).toBe('secret-value')
expect(body.__resolvedSecretTraceProvenance.entries).toEqual([
{ name: 'API_KEY', encryptedValue: 'encrypted-secret' },
])
})
it('places encrypted provenance only on the terminal streamed event', async () => {
mockRunHeadlessCopilotLifecycle.mockImplementation(
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
activateSecret(options)
return successResult()
}
)
const response = await POST(
createMockRequest(
'POST',
requestBody,
{
Authorization: 'Bearer internal',
'x-sim-billing-attribution': 'billing',
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
'x-mothership-execute-stream': 'ndjson',
},
'http://localhost:3000/api/mothership/execute'
)
)
const events = (await response.text())
.trim()
.split('\n')
.map((line) => JSON.parse(line) as Record<string, unknown>)
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
'resolved-secret-provenance-v1'
)
expect(events[0]).toMatchObject({ type: 'heartbeat' })
expect(events[0]).not.toHaveProperty('__resolvedSecretTraceProvenance')
expect(events.at(-1)).toMatchObject({
type: 'final',
data: {
content: 'secret-value',
__resolvedSecretTraceProvenance: {
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
},
},
})
})
})
+193 -28
View File
@@ -20,10 +20,23 @@ import { requestExplicitStreamAbort } from '@/lib/copilot/request/session/explic
import type { StreamEvent } from '@/lib/copilot/request/types'
import { isDocSandboxEnabled } from '@/lib/core/config/env-flags'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { getPersonalAndWorkspaceEnv } from '@/lib/environment/utils'
import {
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
RESOLVED_SECRET_PROVENANCE_FIELD,
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
requestsPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
import {
assertActiveWorkspaceAccess,
isWorkspaceAccessDeniedError,
} from '@/lib/workspaces/permissions/utils'
import {
createIncompleteResolvedSecretTraceRegistry,
createResolvedSecretTraceRegistry,
ResolvedSecretTraceProvenanceAccumulator,
type ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
import type { ChatContext } from '@/stores/panel'
export const maxDuration = 3600
@@ -35,6 +48,28 @@ const MOTHERSHIP_EXECUTE_STREAM_CONTENT_TYPE = 'application/x-ndjson'
const MOTHERSHIP_EXECUTE_HEARTBEAT_INTERVAL_MS = 15_000
const ndjsonEncoder = new TextEncoder()
function withPrivateProvenance<T extends Record<string, unknown>>(
payload: T,
registry: ResolvedSecretTraceRegistry | undefined,
include: boolean
): T & Partial<Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>> {
return {
...payload,
...(include && registry
? { [RESOLVED_SECRET_PROVENANCE_FIELD]: registry.exportProvenance() }
: {}),
}
}
function privateResponseHeaders(
registry: ResolvedSecretTraceRegistry | undefined,
include: boolean
): Record<string, string> {
return include && registry
? { [PRIVATE_TOOL_METADATA_RESPONSE_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1 }
: {}
}
function isAbortError(error: unknown): boolean {
return error instanceof Error && error.name === 'AbortError'
}
@@ -99,6 +134,11 @@ export function buildExecuteResponsePayload(
export const POST = withRouteHandler(async (req: NextRequest) => {
let messageId: string | undefined
let requestId: string | undefined
let resolvedSecretTraceRegistry: ResolvedSecretTraceRegistry | undefined
const includePrivateProvenance = requestsPrivateToolMetadata(
req.headers,
RESOLVED_SECRET_PROVENANCE_METADATA_V1
)
try {
const auth = await checkInternalAuth(req, { requireWorkflowId: false })
@@ -146,6 +186,39 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
actorUserId: userId,
workspaceId,
})
if (includePrivateProvenance) {
const scope = { userId, workspaceId }
try {
const environment = await getPersonalAndWorkspaceEnv(userId, workspaceId, {
workspaceAccess,
})
resolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
personalEncrypted: environment.personalEncrypted,
workspaceEncrypted: environment.workspaceEncrypted,
personalDecrypted: environment.personalDecrypted,
workspaceDecrypted: environment.workspaceDecrypted,
decryptionFailures: environment.decryptionFailures,
scope,
})
} catch (error) {
logger.warn('Failed to build Mothership trace secret catalog', {
error: getErrorMessage(error),
userId,
workspaceId,
})
resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(scope)
}
}
const activeResolvedSecretTraceRegistry = resolvedSecretTraceRegistry
const mcpDiscoveryProvenance = new ResolvedSecretTraceProvenanceAccumulator({
userId,
workspaceId,
})
const recordMcpDiscoveryProvenance = includePrivateProvenance
? (provenance: unknown): void => {
mcpDiscoveryProvenance.record(provenance)
}
: undefined
const effectiveChatId = chatId || generateId()
messageId = providedMessageId || generateId()
@@ -164,17 +237,38 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
)
const nonMcpAgentMentions = agentMentions?.filter((context) => context.kind !== 'mcp')
const userPermission = workspaceAccess.permission
const mothershipToolsPromise = Promise.allSettled([
buildSelectedMcpToolSchemas(
userId,
workspaceId,
mcpTools ?? [],
recordMcpDiscoveryProvenance
),
buildTaggedMcpToolSchemas(
userId,
workspaceId,
taggedMcpServerIds,
recordMcpDiscoveryProvenance
),
]).then(async (results) => {
if (activeResolvedSecretTraceRegistry) {
await activeResolvedSecretTraceRegistry.importProvenance(
mcpDiscoveryProvenance.exportProvenance(),
{ trusted: true }
)
}
const groups = results.map((result) => {
if (result.status === 'rejected') throw result.reason
return result.value
})
const byName = new Map(groups.flat().map((tool) => [tool.name, tool]))
return [...byName.values()]
})
const [workspaceContext, integrationTools, mothershipTools, entitlements, agentContexts] =
await Promise.all([
generateWorkspaceContext(workspaceId, userId, { workspaceAccess }),
buildIntegrationToolSchemas(userId, messageId, undefined, workspaceId),
Promise.all([
buildSelectedMcpToolSchemas(userId, workspaceId, mcpTools ?? []),
buildTaggedMcpToolSchemas(userId, workspaceId, taggedMcpServerIds),
]).then((groups) => {
const byName = new Map(groups.flat().map((tool) => [tool.name, tool]))
return [...byName.values()]
}),
mothershipToolsPromise,
computeWorkspaceEntitlements(workspaceId, userId),
processContextsServer(
nonMcpAgentMentions,
@@ -282,6 +376,7 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
interactive: false,
abortSignal: lifecycleAbortController.signal,
billingAttribution,
resolvedSecretTraceRegistry,
onEvent,
})
@@ -326,7 +421,13 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
allowExplicitAbort = false
if (lifecycleAbortController.signal.aborted) {
send({ type: 'error', error: 'Sim execution aborted' })
send(
withPrivateProvenance(
{ type: 'error', error: 'Sim execution aborted' },
resolvedSecretTraceRegistry,
includePrivateProvenance
)
)
return
}
@@ -343,17 +444,27 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
errors: result.errors,
}
)
send({
type: 'error',
error: result.error || 'Sim execution failed',
content: result.content || '',
})
send(
withPrivateProvenance(
{
type: 'error',
error: result.error || 'Sim execution failed',
content: result.content || '',
},
resolvedSecretTraceRegistry,
includePrivateProvenance
)
)
return
}
send({
type: 'final',
data: buildExecuteResponsePayload(result, effectiveChatId, integrationTools),
data: withPrivateProvenance(
buildExecuteResponsePayload(result, effectiveChatId, integrationTools),
resolvedSecretTraceRegistry,
includePrivateProvenance
),
})
} catch (error) {
if (
@@ -367,7 +478,13 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
: 'Mothership execute aborted',
{ requestId }
)
send({ type: 'error', error: 'Sim execution aborted' })
send(
withPrivateProvenance(
{ type: 'error', error: 'Sim execution aborted' },
resolvedSecretTraceRegistry,
includePrivateProvenance
)
)
return
}
@@ -380,10 +497,16 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
error: getErrorMessage(error, 'Unknown error'),
}
)
send({
type: 'error',
error: getErrorMessage(error, 'Internal server error'),
})
send(
withPrivateProvenance(
{
type: 'error',
error: getErrorMessage(error, 'Internal server error'),
},
resolvedSecretTraceRegistry,
includePrivateProvenance
)
)
} finally {
allowExplicitAbort = false
if (heartbeatId) {
@@ -410,6 +533,7 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
headers: {
'Content-Type': `${MOTHERSHIP_EXECUTE_STREAM_CONTENT_TYPE}; charset=utf-8`,
'Cache-Control': 'no-cache, no-transform',
...privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
},
})
}
@@ -421,7 +545,17 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
if (lifecycleAbortController.signal.aborted || req.signal.aborted) {
reqLogger.info('Mothership execute aborted after lifecycle completion')
return NextResponse.json({ error: 'Sim execution aborted' }, { status: 499 })
return NextResponse.json(
withPrivateProvenance(
{ error: 'Sim execution aborted' },
resolvedSecretTraceRegistry,
includePrivateProvenance
),
{
status: 499,
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
}
)
}
if (!result.success) {
@@ -438,16 +572,30 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
}
)
return NextResponse.json(
withPrivateProvenance(
{
error: result.error || 'Sim execution failed',
content: result.content || '',
},
resolvedSecretTraceRegistry,
includePrivateProvenance
),
{
error: result.error || 'Sim execution failed',
content: result.content || '',
},
{ status: 500 }
status: 500,
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
}
)
}
return NextResponse.json(
buildExecuteResponsePayload(result, effectiveChatId, integrationTools)
withPrivateProvenance(
buildExecuteResponsePayload(result, effectiveChatId, integrationTools),
resolvedSecretTraceRegistry,
includePrivateProvenance
),
{
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
}
)
} finally {
allowExplicitAbort = false
@@ -465,7 +613,17 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
}
)
return NextResponse.json({ error: 'Sim execution aborted' }, { status: 499 })
return NextResponse.json(
withPrivateProvenance(
{ error: 'Sim execution aborted' },
resolvedSecretTraceRegistry,
includePrivateProvenance
),
{
status: 499,
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
}
)
}
if (isWorkspaceAccessDeniedError(error)) {
@@ -481,8 +639,15 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
)
return NextResponse.json(
{ error: getErrorMessage(error, 'Internal server error') },
{ status: 500 }
withPrivateProvenance(
{ error: getErrorMessage(error, 'Internal server error') },
resolvedSecretTraceRegistry,
includePrivateProvenance
),
{
status: 500,
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
}
)
}
})
+3 -2
View File
@@ -7,7 +7,7 @@ import { type NextRequest, NextResponse } from 'next/server'
import { v1GetLogContract } from '@/lib/api/contracts/v1/logs'
import { parseRequest } from '@/lib/api/server'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
import { createApiResponse, getUserLimits } from '@/app/api/v1/logs/meta'
import {
checkRateLimit,
@@ -100,12 +100,13 @@ export const GET = withRouteHandler(
totalDurationMs: log.totalDurationMs,
files: log.files || undefined,
workflow: workflowSummary,
executionData: (await materializeExecutionData(
executionData: (await materializeExecutionDataForDisplay(
log.executionData as Record<string, unknown> | null,
{
workspaceId: log.workspaceId,
workflowId: log.workflowId,
executionId: log.executionId,
userId,
}
)) as any,
cost: log.costTotal != null ? { total: Number(log.costTotal) } : null,
+3 -2
View File
@@ -8,7 +8,7 @@ import { v1ListLogsContract } from '@/lib/api/contracts/v1/logs'
import { parseRequest } from '@/lib/api/server'
import { MATERIALIZE_CONCURRENCY, mapWithConcurrency } from '@/lib/core/utils/concurrency'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
import { buildLogFilters, getOrderBy } from '@/app/api/v1/logs/filters'
import { createApiResponse, getUserLimits } from '@/app/api/v1/logs/meta'
import {
@@ -167,12 +167,13 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
? await mapWithConcurrency(data, MATERIALIZE_CONCURRENCY, async (log) => {
const result = buildBase(log)
if (log.executionData) {
const execData = (await materializeExecutionData(
const execData = (await materializeExecutionDataForDisplay(
log.executionData as Record<string, unknown> | null,
{
workspaceId: log.workspaceId,
workflowId: log.workflowId,
executionId: log.executionId,
userId,
}
)) as any
if (params.includeFinalOutput && execData.finalOutput) {
@@ -9,9 +9,11 @@ import {
executionPreprocessingMockFns,
hybridAuthMockFns,
loggingSessionMock,
queueTableRows,
requestUtilsMockFns,
resetDbChainMock,
resetEnvMock,
schemaMock,
setEnv,
workflowAuthzMockFns,
workflowsPersistenceUtilsMock,
@@ -339,6 +341,125 @@ describe('workflow execute async route', () => {
mockHandlePostExecutionPauseState.mockResolvedValue(undefined)
})
it('reuses raw workflow input by execution ID without returning it to the client', async () => {
const sourceInput = { token: 'raw-secret-1234', nested: { value: 42 } }
queueTableRows(schemaMock.workflowExecutionLogs, [
{
executionId: 'source-execution',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
executionData: { workflowInput: sourceInput },
},
])
const request = createMockRequest(
'POST',
{ inputFromExecutionId: 'source-execution' },
{
'Content-Type': 'application/json',
'X-Execution-Mode': 'async',
Cookie: 'session=value',
}
)
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
const responseBody = await response.json()
expect(response.status).toBe(202)
expect(responseBody).not.toHaveProperty('input')
expect(JSON.stringify(responseBody)).not.toContain('raw-secret-1234')
expect(mockEnqueue).toHaveBeenCalledWith(
'workflow-execution',
expect.objectContaining({ input: sourceInput }),
expect.any(Object)
)
})
it('recovers legacy starter input by execution ID without returning it to the client', async () => {
const sourceInput = { token: 'legacy-retry-input', nested: { value: 42 } }
queueTableRows(schemaMock.workflowExecutionLogs, [
{
executionId: 'source-execution',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
executionData: {
executionState: {
blockStates: {
start: {
output: sourceInput,
executed: false,
executionTime: 0,
},
},
},
},
},
])
const request = createMockRequest(
'POST',
{ inputFromExecutionId: 'source-execution' },
{
'Content-Type': 'application/json',
'X-Execution-Mode': 'async',
Cookie: 'session=value',
}
)
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
const responseBody = await response.json()
expect(response.status).toBe(202)
expect(responseBody).not.toHaveProperty('input')
expect(JSON.stringify(responseBody)).not.toContain('legacy-retry-input')
expect(mockEnqueue).toHaveBeenCalledWith(
'workflow-execution',
expect.objectContaining({ input: sourceInput }),
expect.any(Object)
)
})
it('rejects client input alongside a stored execution input reference', async () => {
const response = await POST(
createMockRequest(
'POST',
{
input: { replacement: true },
inputFromExecutionId: 'source-execution',
},
{ 'Content-Type': 'application/json', Cookie: 'session=value' }
),
{ params: Promise.resolve({ id: 'workflow-1' }) }
)
expect(response.status).toBe(400)
await expect(response.json()).resolves.toEqual({
error: 'Provide either input or inputFromExecutionId, not both',
})
expect(mockEnqueue).not.toHaveBeenCalled()
})
it('rejects stored execution input references from external callers', async () => {
mockCheckHybridAuth.mockResolvedValue({
success: true,
userId: 'personal-key-user-1',
authType: 'api_key',
apiKeyType: 'personal',
})
const response = await POST(
createMockRequest(
'POST',
{ inputFromExecutionId: 'source-execution' },
{ 'Content-Type': 'application/json', 'X-API-Key': 'personal-key' }
),
{ params: Promise.resolve({ id: 'workflow-1' }) }
)
expect(response.status).toBe(403)
await expect(response.json()).resolves.toEqual({
error: 'Stored execution input can only be reused by an authenticated session',
})
expect(mockEnqueue).not.toHaveBeenCalled()
})
it('queues async execution with matching correlation metadata', async () => {
const req = createMockRequest(
'POST',
@@ -750,6 +871,173 @@ describe('workflow execute async route', () => {
expect(mockReleaseExecutionIdClaim).not.toHaveBeenCalled()
})
it('loads trusted run-from-block state by execution ID and preserves its source identity', async () => {
const sourceState = {
blockStates: { previous: { output: { value: 'cached' } } },
executedBlocks: ['previous'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: [],
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: 'owner-1', workspaceId: 'workspace-1' },
},
}
queueTableRows(schemaMock.workflowExecutionLogs, [
{
executionId: 'source-execution',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
executionData: { executionState: sourceState },
},
])
const request = createMockRequest(
'POST',
{
input: { hello: 'world' },
runFromBlock: {
startBlockId: 'start-block',
executionId: 'source-execution',
},
},
{
'Content-Type': 'application/json',
Cookie: 'session=value',
}
)
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
expect(response.status).toBe(200)
expect(mockExecuteWorkflowCore).toHaveBeenCalledWith(
expect.objectContaining({
runFromBlock: {
startBlockId: 'start-block',
sourceSnapshot: sourceState,
sourceExecutionId: 'source-execution',
},
})
)
})
it('falls back to an untrusted client snapshot while stored run-from-block state is pending', async () => {
const sourceSnapshot = {
blockStates: { previous: { output: { value: 'cached' } } },
executedBlocks: ['previous'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: [],
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'untrusted-ciphertext' }],
},
}
queueTableRows(schemaMock.workflowExecutionLogs, [])
const request = createMockRequest(
'POST',
{
input: { hello: 'world' },
runFromBlock: {
startBlockId: 'start-block',
executionId: 'source-execution',
sourceSnapshot,
},
},
{
'Content-Type': 'application/json',
Cookie: 'session=value',
}
)
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
expect(response.status).toBe(200)
expect(mockExecuteWorkflowCore).toHaveBeenCalledWith(
expect.objectContaining({
runFromBlock: {
startBlockId: 'start-block',
sourceSnapshot: expect.objectContaining({
blockStates: sourceSnapshot.blockStates,
executedBlocks: sourceSnapshot.executedBlocks,
}),
},
})
)
const runFromBlock = mockExecuteWorkflowCore.mock.calls[0]?.[0]?.runFromBlock
expect(runFromBlock).not.toHaveProperty('sourceExecutionId')
expect(runFromBlock?.sourceSnapshot).not.toHaveProperty('resolvedSecretTraceProvenance')
})
it('returns encrypted resolution provenance only to an authenticated internal tool caller', async () => {
const caller = EXECUTION_CALLERS[4]
configureExecutionCaller(caller)
const provenance = {
version: 1,
complete: true,
entries: [{ name: 'CHILD_SECRET', encryptedValue: 'encrypted-child-secret' }],
}
mockExecuteWorkflowCore.mockResolvedValueOnce({
success: true,
status: 'completed',
output: { ok: true },
executionState: { resolvedSecretTraceProvenance: provenance },
metadata: {
duration: 100,
startTime: '2026-01-01T00:00:00Z',
endTime: '2026-01-01T00:00:01Z',
},
})
const request = createCallerExecutionRequest(caller, undefined, 'sync')
request.headers.set('x-sim-request-private-tool-metadata', 'resolved-secret-provenance-v1')
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
expect(response.status).toBe(200)
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
'resolved-secret-provenance-v1'
)
await expect(response.json()).resolves.toMatchObject({
output: { ok: true },
__resolvedSecretTraceProvenance: provenance,
})
})
it('does not expose private provenance metadata to non-internal callers', async () => {
const caller = EXECUTION_CALLERS[1]
configureExecutionCaller(caller)
mockExecuteWorkflowCore.mockResolvedValueOnce({
success: true,
status: 'completed',
output: { ok: true },
executionState: {
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [{ name: 'SECRET', encryptedValue: 'encrypted-secret' }],
},
},
metadata: {
duration: 100,
startTime: '2026-01-01T00:00:00Z',
endTime: '2026-01-01T00:00:01Z',
},
})
const request = createCallerExecutionRequest(caller, undefined, 'sync')
request.headers.set('x-sim-request-private-tool-metadata', 'resolved-secret-provenance-v1')
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
const body = await response.json()
expect(response.status).toBe(200)
expect(response.headers.get('x-sim-private-tool-metadata')).toBeNull()
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
})
it('releases the admission reservation when enqueue proves non-acceptance', async () => {
mockEnqueue.mockRejectedValueOnce(
new AsyncJobEnqueueError('queue rejected the job', {
+164 -37
View File
@@ -58,6 +58,12 @@ import {
import { containsLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
import { compactBlockLogs, compactExecutionPayload } from '@/lib/execution/payloads/serializer'
import { type PreprocessExecutionSuccess, preprocessExecution } from '@/lib/execution/preprocessing'
import {
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
RESOLVED_SECRET_PROVENANCE_FIELD,
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
requestsPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
import { LoggingSession } from '@/lib/logs/execution/logging-session'
import {
MAX_MCP_WORKFLOW_RESPONSE_BYTES,
@@ -114,12 +120,18 @@ import {
import { normalizeName } from '@/executor/constants'
import { ExecutionSnapshot } from '@/executor/execution/snapshot'
import type {
BlockCompletionCallbackData,
ChildWorkflowContext,
ExecutionMetadata,
IterationContext,
SerializableExecutionState,
} from '@/executor/execution/types'
import type { BlockLog, NormalizedBlockOutput, StreamingExecution } from '@/executor/types'
import type {
BlockLog,
ExecutionResult,
NormalizedBlockOutput,
StreamingExecution,
} from '@/executor/types'
import { getExecutionErrorStatus, hasExecutionResult } from '@/executor/utils/errors'
import { Serializer } from '@/serializer'
import { CORE_TRIGGER_TYPES, type CoreTriggerType } from '@/stores/logs/filters/types'
@@ -133,6 +145,31 @@ const WORKFLOW_EXECUTION_JOB_ID_PREFIX = 'workflow-execution:'
export const runtime = 'nodejs'
export const dynamic = 'force-dynamic'
function createExecutionJsonResponse(
body: Record<string, unknown>,
init: ResponseInit | undefined,
includePrivateProvenance: boolean,
result?: ExecutionResult
): NextResponse {
if (!includePrivateProvenance) {
return NextResponse.json(body, init)
}
const headers = new Headers(init?.headers)
headers.set(PRIVATE_TOOL_METADATA_RESPONSE_HEADER, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
return NextResponse.json(
{
...body,
[RESOLVED_SECRET_PROVENANCE_FIELD]: result?.executionState?.resolvedSecretTraceProvenance ?? {
version: 1,
complete: false,
entries: [],
},
},
{ ...init, headers }
)
}
async function compactRoutePayload<T>(
value: T,
context: {
@@ -573,6 +610,10 @@ async function handleExecutePost(
const isMcpBridgeRequest =
auth.authType === AuthType.INTERNAL_JWT && req.headers.get(MCP_TOOL_BRIDGE_HEADER) === 'true'
const includePrivateTraceProvenance =
auth.success &&
auth.authType === AuthType.INTERNAL_JWT &&
requestsPrivateToolMetadata(req.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
const useMcpBridgeAuthenticatedUserAsActor =
isMcpBridgeRequest && req.headers.get(MCP_TOOL_BRIDGE_ACTOR_HEADER) === 'authenticated-user'
@@ -682,6 +723,7 @@ async function handleExecutePost(
includeToolCalls: requestedIncludeToolCalls,
useDraftState,
input: validatedInput,
inputFromExecutionId,
isClientSession = false,
includeFileBase64,
base64MaxBytes,
@@ -730,6 +772,20 @@ async function handleExecutePost(
)
}
if (inputFromExecutionId && (isPublicApiAccess || auth.authType !== AuthType.SESSION)) {
return NextResponse.json(
{ error: 'Stored execution input can only be reused by an authenticated session' },
{ status: 403 }
)
}
if (inputFromExecutionId && validatedInput !== undefined) {
return NextResponse.json(
{ error: 'Provide either input or inputFromExecutionId, not both' },
{ status: 400 }
)
}
if (auth.authType === 'api_key') {
if (isClientSession) {
return NextResponse.json(
@@ -776,14 +832,7 @@ async function handleExecutePost(
)
}
if (rawRunFromBlock.sourceSnapshot && !isPublicApiAccess) {
// Public API callers cannot inject arbitrary block state via sourceSnapshot.
// They must use executionId to resume from a server-stored execution state.
resolvedRunFromBlock = {
startBlockId: rawRunFromBlock.startBlockId,
sourceSnapshot: rawRunFromBlock.sourceSnapshot as SerializableExecutionState,
}
} else if (rawRunFromBlock.executionId) {
if (rawRunFromBlock.executionId) {
const { getExecutionStateForWorkflow, getLatestExecutionStateWithExecutionId } =
await import('@/lib/workflows/executor/execution-state')
const sourceExecution =
@@ -795,17 +844,32 @@ async function handleExecutePost(
}
const snapshot = sourceExecution?.state
if (!snapshot) {
return NextResponse.json(
{
error: `No execution state found for ${rawRunFromBlock.executionId === 'latest' ? 'workflow' : `execution ${rawRunFromBlock.executionId}`}. Run the full workflow first.`,
},
{ status: 400 }
)
if (rawRunFromBlock.sourceSnapshot && !isPublicApiAccess) {
resolvedRunFromBlock = {
startBlockId: rawRunFromBlock.startBlockId,
sourceSnapshot: rawRunFromBlock.sourceSnapshot as SerializableExecutionState,
}
} else {
return NextResponse.json(
{
error: `No execution state found for ${rawRunFromBlock.executionId === 'latest' ? 'workflow' : `execution ${rawRunFromBlock.executionId}`}. Run the full workflow first.`,
},
{ status: 400 }
)
}
} else {
resolvedRunFromBlock = {
startBlockId: rawRunFromBlock.startBlockId,
sourceSnapshot: snapshot,
sourceExecutionId: sourceExecution.executionId,
}
}
} else if (rawRunFromBlock.sourceSnapshot && !isPublicApiAccess) {
// Public API callers cannot inject arbitrary block state via sourceSnapshot.
// They must use executionId to resume from a server-stored execution state.
resolvedRunFromBlock = {
startBlockId: rawRunFromBlock.startBlockId,
sourceSnapshot: snapshot,
sourceExecutionId: sourceExecution.executionId,
sourceSnapshot: rawRunFromBlock.sourceSnapshot as SerializableExecutionState,
}
} else {
return NextResponse.json(
@@ -817,7 +881,7 @@ async function handleExecutePost(
// For API key and internal JWT auth, the entire body is the input (except for our control fields)
// For session auth, the input is explicitly provided in the input field
const input = isMcpBridgeRequest
let input = isMcpBridgeRequest
? validatedInput
: isPublicApiAccess ||
auth.authType === AuthType.API_KEY ||
@@ -828,6 +892,7 @@ async function handleExecutePost(
triggerType,
stream,
useDraftState,
inputFromExecutionId: _inputFromExecutionId,
includeFileBase64,
base64MaxBytes,
workflowStateOverride,
@@ -962,6 +1027,20 @@ async function handleExecutePost(
)
}
if (inputFromExecutionId) {
const { getExecutionInputForWorkflow } = await import(
'@/lib/workflows/executor/execution-state'
)
const sourceExecution = await getExecutionInputForWorkflow(inputFromExecutionId, workflowId)
if (!sourceExecution.found) {
return NextResponse.json(
{ error: 'Source workflow execution was not found' },
{ status: 404 }
)
}
input = sourceExecution.input
}
const upstreamBillingAttribution =
auth.authType === AuthType.INTERNAL_JWT && workflowAuthorization.workflow?.workspaceId
? requireBillingAttributionHeader(req.headers, {
@@ -1287,7 +1366,7 @@ async function handleExecutePost(
workflowResponseCompaction
)
return NextResponse.json(
return createExecutionJsonResponse(
{
success: false,
output: compactResultOutput,
@@ -1300,7 +1379,9 @@ async function handleExecutePost(
}
: undefined,
},
{ status: 408 }
{ status: 408 },
includePrivateTraceProvenance,
result
)
}
@@ -1367,7 +1448,12 @@ async function handleExecutePost(
: undefined,
}
return NextResponse.json(filteredResult)
return createExecutionJsonResponse(
filteredResult,
undefined,
includePrivateTraceProvenance,
result
)
} catch (error: unknown) {
const errorMessage = getErrorMessage(error, 'Unknown error')
@@ -1405,7 +1491,7 @@ async function handleExecutePost(
throw compactError
}
}
return NextResponse.json(
return createExecutionJsonResponse(
{
success: false,
output: compactErrorOutput,
@@ -1418,7 +1504,9 @@ async function handleExecutePost(
}
: undefined,
},
{ status }
{ status },
includePrivateTraceProvenance,
executionResult
)
} finally {
requestAbort.cleanup()
@@ -1663,7 +1751,7 @@ async function handleExecutePost(
blockId: string,
blockName: string,
blockType: string,
callbackData: any,
callbackData: BlockCompletionCallbackData,
iterationContext?: IterationContext,
childWorkflowContext?: ChildWorkflowContext
) => {
@@ -1686,7 +1774,13 @@ async function handleExecutePost(
preserveRoot: true,
}),
}
const hasError = compactCallbackData.output?.error
const callbackError = compactCallbackData.output?.error
const hasError = typeof callbackError === 'string' && callbackError.length > 0
const display = await loggingSession.projectDisplayContent({
input: compactCallbackData.input,
output: compactCallbackData.output,
...(hasError ? { error: callbackError } : {}),
})
const childWorkflowData = childWorkflowContext
? {
childWorkflowBlockId: childWorkflowContext.parentBlockId,
@@ -1703,7 +1797,7 @@ async function handleExecutePost(
blockId,
blockName,
blockType,
error: compactCallbackData.output.error,
error: display.error,
})
await sendEvent({
type: 'block:error',
@@ -1715,7 +1809,12 @@ async function handleExecutePost(
blockName,
blockType,
input: compactCallbackData.input,
error: compactCallbackData.output.error,
error: callbackError,
display: {
...(Object.hasOwn(display, 'input') ? { input: display.input } : {}),
...(display.error !== undefined ? { error: display.error } : {}),
...(display.clearLiveDisplay ? { clearLiveDisplay: true as const } : {}),
},
durationMs: compactCallbackData.executionTime || 0,
startedAt: compactCallbackData.startedAt,
executionOrder: compactCallbackData.executionOrder,
@@ -1750,6 +1849,11 @@ async function handleExecutePost(
blockType,
input: compactCallbackData.input,
output: compactCallbackData.output,
display: {
...(Object.hasOwn(display, 'input') ? { input: display.input } : {}),
...(Object.hasOwn(display, 'output') ? { output: display.output } : {}),
...(display.clearLiveDisplay ? { clearLiveDisplay: true as const } : {}),
},
durationMs: compactCallbackData.executionTime || 0,
startedAt: compactCallbackData.startedAt,
executionOrder: compactCallbackData.executionOrder,
@@ -1786,6 +1890,7 @@ async function handleExecutePost(
workflowId,
sendEvent,
forwardAnswerText: answerTextFromSink,
projectDisplay: (field, value) => loggingSession.projectLiveDisplayText(field, value),
})
const reader = streamingExec.stream.getReader()
@@ -1807,12 +1912,13 @@ async function handleExecutePost(
if (answerTextFromSink) continue
const chunk = decoder.decode(value, { stream: true })
const display = await loggingSession.projectLiveDisplayText('chunk', chunk)
await sendEvent({
type: 'stream:chunk',
timestamp: new Date().toISOString(),
executionId,
workflowId,
data: { blockId, chunk },
data: { blockId, chunk, display },
})
}
@@ -1931,7 +2037,10 @@ async function handleExecutePost(
* object storage, so doing it twice would double the latency and storage
* load on the happy path.
*/
const compactedBlockLogs = await compactBlockLogs(result.logs, {
const displayBlockLogs = await loggingSession.projectBlockLogsForDisplay(
result.logs ?? []
)
const compactedBlockLogs = await compactBlockLogs(displayBlockLogs, {
workspaceId,
workflowId,
executionId,
@@ -1947,6 +2056,9 @@ async function handleExecutePost(
})
await loggingSession.markAsFailed(timeoutErrorMessage)
const timeoutDisplay = await loggingSession.projectDisplayContent({
error: timeoutErrorMessage,
})
finalMetaStatus = 'error'
await sendEvent(
@@ -1957,6 +2069,11 @@ async function handleExecutePost(
workflowId,
data: {
error: timeoutErrorMessage,
display: {
...(timeoutDisplay.error !== undefined
? { error: timeoutDisplay.error }
: {}),
},
duration: result.metadata?.duration || 0,
finalBlockLogs: compactedBlockLogs,
},
@@ -2061,13 +2178,16 @@ async function handleExecutePost(
let compactErrorLogs: BlockLog[] | undefined
try {
compactErrorLogs = executionResult?.logs
? await compactBlockLogs(executionResult.logs, {
workspaceId,
workflowId,
executionId,
userId: actorUserId,
requireDurable: true,
})
? await compactBlockLogs(
await loggingSession.projectBlockLogsForDisplay(executionResult.logs),
{
workspaceId,
workflowId,
executionId,
userId: actorUserId,
requireDurable: true,
}
)
: undefined
} catch (compactionError) {
reqLogger.warn('Failed to compact SSE error logs, omitting oversized error details', {
@@ -2076,6 +2196,10 @@ async function handleExecutePost(
}
finalMetaStatus = 'error'
const terminalError = executionResult?.error || errorMessage
const terminalDisplay = await loggingSession.projectDisplayContent({
error: terminalError,
})
await sendEvent(
{
type: 'execution:error',
@@ -2083,7 +2207,10 @@ async function handleExecutePost(
executionId,
workflowId,
data: {
error: executionResult?.error || errorMessage,
error: terminalError,
display: {
...(terminalDisplay.error !== undefined ? { error: terminalDisplay.error } : {}),
},
duration: executionResult?.metadata?.duration || 0,
finalBlockLogs: compactErrorLogs,
},
@@ -9,6 +9,12 @@ import {
} from '@/lib/api/contracts/workflows'
import { parseRequest } from '@/lib/api/server'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import {
collectFunctionalBlockOutputs,
FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE,
type FunctionalExecutionDataSource,
FunctionalOutputsUnavailableError,
} from '@/lib/logs/execution/functional-outputs'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { getAutomaticResumeWaitingMetadata } from '@/lib/workflows/executor/paused-execution-metadata'
import { validateWorkflowAccess } from '@/app/api/workflows/middleware'
@@ -18,32 +24,10 @@ const logger = createLogger('WorkflowExecutionStatusAPI')
type LogStatus = 'pending' | 'running' | 'completed' | 'failed' | 'cancelled'
interface TraceSpanShape {
blockId?: string
output?: Record<string, unknown>
children?: TraceSpanShape[]
}
interface ExecutionDataShape {
interface ExecutionDataShape extends FunctionalExecutionDataSource {
finalOutput?: { error?: string } & Record<string, unknown>
error?: { message?: string } | string
completionFailure?: string
traceSpans?: TraceSpanShape[]
}
function collectBlockOutputs(spans: TraceSpanShape[] | undefined): Map<string, unknown> {
const map = new Map<string, unknown>()
const visit = (list?: TraceSpanShape[]): void => {
if (!list) return
for (const span of list) {
if (span.blockId && span.output !== undefined && !map.has(span.blockId)) {
map.set(span.blockId, span.output)
}
if (span.children) visit(span.children)
}
}
visit(spans)
return map
}
function resolvePath(value: unknown, path: string[]): unknown {
@@ -204,10 +188,23 @@ export const GET = withRouteHandler(
? (executionData.finalOutput ?? null)
: null
const blockOutputs =
selectedOutputs.length > 0
? pickSelectedOutputs(selectedOutputs, collectBlockOutputs(executionData?.traceSpans))
: null
let blockOutputs: Record<string, unknown> | null = null
if (selectedOutputs.length > 0) {
try {
blockOutputs = pickSelectedOutputs(
selectedOutputs,
collectFunctionalBlockOutputs(executionData)
)
} catch (error) {
if (error instanceof FunctionalOutputsUnavailableError) {
return NextResponse.json(
{ error: FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE },
{ status: 409 }
)
}
throw error
}
}
const response: WorkflowExecutionStatusResponse = {
executionId: logRow.executionId,
@@ -13,6 +13,7 @@ const {
mockResolveBillingAttribution,
mockAssertBillingAttributionSnapshot,
mockStart,
mockSetResolvedSecretTraceRegistry,
mockSafeComplete,
mockSafeCompleteWithError,
} = vi.hoisted(() => ({
@@ -21,6 +22,7 @@ const {
mockResolveBillingAttribution: vi.fn(),
mockAssertBillingAttributionSnapshot: vi.fn((value) => value),
mockStart: vi.fn().mockResolvedValue(undefined),
mockSetResolvedSecretTraceRegistry: vi.fn(),
mockSafeComplete: vi.fn().mockResolvedValue(undefined),
mockSafeCompleteWithError: vi.fn().mockResolvedValue(undefined),
}))
@@ -42,6 +44,7 @@ vi.mock('@/lib/logs/execution/logging-session', () => ({
LoggingSession: vi.fn(function LoggingSession() {
return {
start: mockStart,
setResolvedSecretTraceRegistry: mockSetResolvedSecretTraceRegistry,
markAsFailed: vi.fn().mockResolvedValue(undefined),
safeCompleteWithError: mockSafeCompleteWithError,
safeComplete: mockSafeComplete,
@@ -307,4 +310,104 @@ describe('POST /api/workflows/[id]/log completion attribution', () => {
)
expect(mockSafeComplete).toHaveBeenCalledOnce()
})
it('restores trusted Secrets provenance before projecting legacy completion traces', async () => {
const trustedExecutionState = {
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
resolvedSecretTraceProvenance: { version: 1, complete: true, entries: [] },
}
dbChainMockFns.limit.mockResolvedValueOnce([
{
workflowId: OWNER_WORKFLOW_ID,
workspaceId: 'workspace-1',
executionData: {
billingAttribution: storedBillingAttribution,
executionState: trustedExecutionState,
},
},
])
const res = await POST(
makeRequest(OWNER_WORKFLOW_ID, {
executionId: 'trusted-provenance-execution-id',
result: validResult,
}),
{ params: Promise.resolve({ id: OWNER_WORKFLOW_ID }) }
)
expect(res.status).toBe(200)
const registry = mockSetResolvedSecretTraceRegistry.mock.calls[0]?.[0]
expect(registry?.isComplete()).toBe(true)
expect(registry?.exportProvenance()).toEqual({
version: 1,
complete: true,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
expect(mockSafeComplete).toHaveBeenCalledWith(
expect.objectContaining({ executionState: trustedExecutionState })
)
})
it('forwards trusted execution state through legacy error completion', async () => {
const trustedExecutionState = {
blockStates: { 'function-1': { output: { error: 'raw-secret-value' } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
resolvedSecretTraceProvenance: { version: 1, complete: true, entries: [] },
}
dbChainMockFns.limit.mockResolvedValueOnce([
{
workflowId: OWNER_WORKFLOW_ID,
workspaceId: 'workspace-1',
executionData: {
billingAttribution: storedBillingAttribution,
executionState: trustedExecutionState,
},
},
])
const res = await POST(
makeRequest(OWNER_WORKFLOW_ID, {
executionId: 'trusted-error-execution-id',
result: { success: false, error: 'failed' },
}),
{ params: Promise.resolve({ id: OWNER_WORKFLOW_ID }) }
)
expect(res.status).toBe(200)
expect(mockSafeCompleteWithError).toHaveBeenCalledWith(
expect.objectContaining({ executionState: trustedExecutionState })
)
})
it('forces structural-only traces when trusted stored provenance is unavailable', async () => {
dbChainMockFns.limit.mockResolvedValueOnce([
{
workflowId: OWNER_WORKFLOW_ID,
workspaceId: 'workspace-1',
executionData: { billingAttribution: storedBillingAttribution },
},
])
const res = await POST(
makeRequest(OWNER_WORKFLOW_ID, {
executionId: 'legacy-no-provenance-execution-id',
result: validResult,
}),
{ params: Promise.resolve({ id: OWNER_WORKFLOW_ID }) }
)
expect(res.status).toBe(200)
const registry = mockSetResolvedSecretTraceRegistry.mock.calls[0]?.[0]
expect(registry?.isComplete()).toBe(false)
})
})
@@ -13,9 +13,12 @@ import { generateRequestId } from '@/lib/core/utils/request'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { LoggingSession } from '@/lib/logs/execution/logging-session'
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { validateWorkflowAccess } from '@/app/api/workflows/middleware'
import { createErrorResponse, createSuccessResponse } from '@/app/api/workflows/utils'
import type { SerializableExecutionState } from '@/executor/execution/types'
import type { ExecutionResult } from '@/executor/types'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
const logger = createLogger('WorkflowLogAPI')
@@ -119,6 +122,31 @@ export const POST = withRouteHandler(
const isChatExecution = result.metadata?.source === 'chat'
const triggerType = isChatExecution ? 'chat' : 'manual'
const loggingSession = new LoggingSession(id, executionId, triggerType, requestId)
const resolvedSecretTraceRegistry = new ResolvedSecretTraceRegistry([], {
userId: actorUserId,
workspaceId: existingLog.workspaceId,
})
const trustedExecutionData = await materializeExecutionData(
existingLog.executionData as Record<string, unknown>,
{
workspaceId: existingLog.workspaceId,
workflowId: existingLog.workflowId,
executionId,
}
)
const trustedExecutionState =
trustedExecutionData.executionState &&
typeof trustedExecutionData.executionState === 'object' &&
!Array.isArray(trustedExecutionData.executionState)
? (trustedExecutionData.executionState as SerializableExecutionState)
: undefined
const trustedProvenance = trustedExecutionState?.resolvedSecretTraceProvenance
if (trustedProvenance === undefined) {
resolvedSecretTraceRegistry.markIncomplete()
} else {
await resolvedSecretTraceRegistry.importProvenance(trustedProvenance, { trusted: true })
}
loggingSession.setResolvedSecretTraceRegistry(resolvedSecretTraceRegistry)
await loggingSession.start({
userId: actorUserId,
@@ -143,6 +171,7 @@ export const POST = withRouteHandler(
totalDurationMs: totalDuration || result.metadata?.duration || 0,
error: { message },
traceSpans,
executionState: trustedExecutionState,
})
} else {
await loggingSession.safeComplete({
@@ -150,6 +179,7 @@ export const POST = withRouteHandler(
totalDurationMs: totalDuration || result.metadata?.duration || 0,
finalOutput: result.output || {},
traceSpans,
executionState: trustedExecutionState,
})
}
@@ -72,8 +72,6 @@ import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/provide
import { getBlock } from '@/blocks/registry'
import { useFolderMap, useFolders } from '@/hooks/queries/folders'
import {
fetchLogDetail,
logKeys,
prefetchLogDetail,
useCancelExecution,
useDashboardStats,
@@ -90,7 +88,6 @@ import { CORE_TRIGGER_TYPES } from '@/stores/logs/filters/types'
import { Dashboard, ExecutionSnapshot, LogDetails, LogRowContextMenu } from './components'
import {
DELETED_WORKFLOW_LABEL,
extractRetryInput,
formatDate,
getDisplayStatus,
type LogStatus,
@@ -573,17 +570,11 @@ export default function Logs() {
const retryLog = useCallback(
async (log: WorkflowLogRow | null) => {
const workflowId = log?.workflow?.id || log?.workflowId
const logId = log?.id
if (!workflowId || !logId) return
const executionId = log?.executionId
if (!workflowId || !executionId) return
try {
const detailLog = await queryClient.fetchQuery({
queryKey: logKeys.detail(workspaceId, logId),
queryFn: ({ signal }) => fetchLogDetail(logId, workspaceId, signal),
staleTime: 30 * 1000,
})
const input = extractRetryInput(detailLog)
await retryExecution.mutateAsync({ workflowId, input })
await retryExecution.mutateAsync({ workflowId, executionId })
toast.success('Retry started')
} catch {
toast.error('Failed to retry execution')
@@ -2,7 +2,6 @@ import React from 'react'
import { Badge } from '@sim/emcn'
import { formatDuration, formatRelativeTime } from '@sim/utils/formatting'
import { format } from 'date-fns'
import type { WorkflowLogDetail } from '@/lib/api/contracts/logs'
import { getIntegrationMetadata } from '@/lib/logs/get-trigger-options'
import { getBlock } from '@/blocks/registry'
import { CORE_TRIGGER_TYPES } from '@/stores/logs/filters/types'
@@ -213,37 +212,3 @@ export const formatDate = (dateString: string) => {
relative: formatRelativeTime(dateString),
}
}
/**
* Extracts the original workflow input from a log entry for retry.
* Prefers the persisted `workflowInput` field (new logs), falls back to
* reconstructing from `executionState.blockStates` (old logs).
*/
export function extractRetryInput(log: WorkflowLogDetail): unknown | undefined {
const execData = log.executionData
if (!execData) return undefined
if (execData.workflowInput !== undefined) {
return execData.workflowInput
}
const executionState = (execData as Record<string, unknown>).executionState as
| {
blockStates?: Record<
string,
{ output?: unknown; executed?: boolean; executionTime?: number }
>
}
| undefined
if (!executionState?.blockStates) return undefined
// Starter/trigger blocks are pre-populated with executed: false and
// executionTime: 0, which distinguishes them from blocks that actually ran.
for (const state of Object.values(executionState.blockStates)) {
if (state.executed === false && state.executionTime === 0 && state.output != null) {
return state.output
}
}
return undefined
}
@@ -10,8 +10,11 @@ const {
DirectUploadErrorMock,
executionStoreState,
mockExecute,
mockExecuteFromBlock,
mockFetch,
mockResolveStartCandidates,
mockRunUploadStrategy,
mockSelectBestTrigger,
terminalStoreState,
workflowBlocks,
workflowStoreState,
@@ -50,7 +53,7 @@ const {
const executionStoreState = {
workflowExecutions: new Map([['workflow-1', idleExecution]]),
getWorkflowExecution: vi.fn(() => idleExecution),
getCurrentExecutionId: vi.fn(() => null),
getCurrentExecutionId: vi.fn<() => string | null>(() => null),
getLastExecutionSnapshot: vi.fn(() => null),
setCurrentExecutionId: vi.fn(),
setIsExecuting: vi.fn(),
@@ -88,8 +91,11 @@ const {
DirectUploadErrorMock,
executionStoreState,
mockExecute: vi.fn(),
mockExecuteFromBlock: vi.fn(),
mockFetch: vi.fn(),
mockResolveStartCandidates: vi.fn(),
mockRunUploadStrategy: vi.fn(),
mockSelectBestTrigger: vi.fn(),
terminalStoreState,
workflowBlocks,
workflowStoreState,
@@ -133,12 +139,12 @@ vi.mock('@/lib/workflows/input-format', () => ({
vi.mock('@/lib/workflows/triggers/trigger-utils', () => ({
extractTriggerMockPayload: () => ({}),
selectBestTrigger: () => [],
selectBestTrigger: mockSelectBestTrigger,
triggerNeedsMockPayload: () => false,
}))
vi.mock('@/lib/workflows/triggers/triggers', () => ({
resolveStartCandidates: () => [],
resolveStartCandidates: mockResolveStartCandidates,
StartBlockPath: {
SPLIT_API: 'split-api',
SPLIT_INPUT: 'split-input',
@@ -206,7 +212,7 @@ vi.mock('@/hooks/use-execution-stream', () => {
SSEStreamInterruptedError,
useExecutionStream: () => ({
execute: mockExecute,
executeFromBlock: vi.fn(),
executeFromBlock: mockExecuteFromBlock,
reconnect: vi.fn(),
cancel: vi.fn(),
cancelExecute: vi.fn(),
@@ -344,6 +350,9 @@ async function drainStream(value: unknown): Promise<void> {
describe('useWorkflowExecution attachment uploads', () => {
beforeEach(() => {
vi.clearAllMocks()
executionStoreState.getCurrentExecutionId.mockReturnValue(null)
mockResolveStartCandidates.mockReturnValue([])
mockSelectBestTrigger.mockReturnValue([])
vi.stubGlobal('fetch', mockFetch)
mockRunUploadStrategy.mockRejectedValue(
new DirectUploadErrorMock('Server signaled fallback to API upload', 'FALLBACK_REQUIRED')
@@ -355,6 +364,8 @@ describe('useWorkflowExecution attachment uploads', () => {
})
)
mockExecute.mockResolvedValue(undefined)
mockExecuteFromBlock.mockResolvedValue(undefined)
workflowStoreState.edges.length = 0
})
afterEach(() => {
@@ -470,4 +481,174 @@ describe('useWorkflowExecution attachment uploads', () => {
unmount()
})
it('uses only projected live thinking without changing normal settle behavior', async () => {
mockExecute.mockImplementationOnce(async (options) => {
options.onExecutionId?.('execution-1')
await options.callbacks?.onStreamThinking?.({
blockId: 'agent-1',
text: 'sk-resolved-secret',
display: { text: '{{OPENAI_API_KEY}}' },
})
await options.callbacks?.onStreamDone?.({ blockId: 'agent-1' })
await options.callbacks?.onBlockCompleted?.({
blockId: 'agent-1',
blockName: 'Agent 1',
blockType: 'agent',
executionOrder: 1,
output: { content: 'sk-resolved-secret' },
display: { output: { content: '{{OPENAI_API_KEY}}' } },
durationMs: 10,
startedAt: '2026-07-31T00:00:00.000Z',
endedAt: '2026-07-31T00:00:00.010Z',
})
})
const { result, unmount } = renderWorkflowExecutionHook()
await act(async () => {
const runResult = await result().handleRunWorkflow({ input: 'chat input' })
await drainStream(runResult)
})
expect(terminalStoreState.updateConsole).toHaveBeenCalledWith(
'agent-1',
expect.objectContaining({ agentStreamThinking: '{{OPENAI_API_KEY}}' }),
'execution-1'
)
expect(terminalStoreState.updateConsole).not.toHaveBeenCalledWith(
'agent-1',
expect.objectContaining({ clearAgentStreamThinking: true }),
'execution-1'
)
expect(JSON.stringify(terminalStoreState.updateConsole.mock.calls)).not.toContain(
'sk-resolved-secret'
)
unmount()
})
it('preserves legacy live thinking when no display projection field is sent', async () => {
mockExecute.mockImplementationOnce(async (options) => {
options.onExecutionId?.('execution-1')
await options.callbacks?.onStreamThinking?.({
blockId: 'agent-1',
text: 'sk-resolved-secret',
})
await options.callbacks?.onStreamDone?.({ blockId: 'agent-1' })
})
const { result, unmount } = renderWorkflowExecutionHook()
await act(async () => {
const runResult = await result().handleRunWorkflow({ input: 'chat input' })
await drainStream(runResult)
})
expect(terminalStoreState.updateConsole).toHaveBeenCalledWith(
'agent-1',
expect.objectContaining({ agentStreamThinking: 'sk-resolved-secret' }),
'execution-1'
)
unmount()
})
it('clears live thinking when the server sends an empty display projection', async () => {
mockExecute.mockImplementationOnce(async (options) => {
options.onExecutionId?.('execution-1')
await options.callbacks?.onStreamThinking?.({
blockId: 'agent-1',
text: 'sk-resolved-secret',
display: {},
})
})
const { result, unmount } = renderWorkflowExecutionHook()
await act(async () => {
const runResult = await result().handleRunWorkflow({ input: 'chat input' })
await drainStream(runResult)
})
expect(terminalStoreState.updateConsole).toHaveBeenCalledWith(
'agent-1',
{ clearAgentStreamThinking: true },
'execution-1'
)
expect(JSON.stringify(terminalStoreState.updateConsole.mock.calls)).not.toContain(
'sk-resolved-secret'
)
unmount()
})
it('keeps the trusted execution ID when storing a run-until-block snapshot', async () => {
const startCandidate = {
blockId: 'start',
block: workflowBlocks.start,
path: 'legacy-starter',
}
mockResolveStartCandidates.mockReturnValue([startCandidate])
mockSelectBestTrigger.mockReturnValue([startCandidate])
mockExecute.mockImplementationOnce(async (options) => {
const executionId = options.executionId as string
executionStoreState.getCurrentExecutionId.mockReturnValue(executionId)
options.onExecutionId?.(executionId)
await options.callbacks?.onExecutionCompleted?.({
success: true,
output: {},
duration: 10,
startTime: '2026-07-31T00:00:00.000Z',
endTime: '2026-07-31T00:00:00.010Z',
finalBlockLogs: [],
})
})
const { result, unmount } = renderWorkflowExecutionHook()
await act(async () => {
await result().handleRunUntilBlock('function-1', 'workflow-1')
})
const executionId = mockExecute.mock.calls[0]?.[0]?.executionId
expect(executionId).toEqual(expect.any(String))
expect(executionStoreState.setLastExecutionSnapshot).toHaveBeenCalledWith(
'workflow-1',
expect.objectContaining({ sourceExecutionId: executionId })
)
unmount()
})
it('sends the snapshot as a fallback with a trusted run-from-block execution ID', async () => {
const sourceSnapshot = {
blockStates: { start: { output: { value: 'ready' } } },
executedBlocks: ['start'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['start'],
sourceExecutionId: 'source-execution-1',
}
executionStoreState.getLastExecutionSnapshot.mockReturnValueOnce(sourceSnapshot)
workflowStoreState.edges.push({ source: 'start', target: 'function-1' } as never)
const { result, unmount } = renderWorkflowExecutionHook()
await act(async () => {
await result().handleRunFromBlock('function-1', 'workflow-1')
})
expect(mockExecuteFromBlock).toHaveBeenCalledWith(
expect.objectContaining({
workflowId: 'workflow-1',
startBlockId: 'function-1',
sourceExecutionId: 'source-execution-1',
sourceSnapshot,
})
)
unmount()
})
})
@@ -16,6 +16,7 @@ import {
} from '@/components/agent-stream/tool-call-lifecycle'
import { requestJson } from '@/lib/api/client/request'
import { cancelWorkflowExecutionContract, workflowLogContract } from '@/lib/api/contracts/workflows'
import type { SecretSafeBlockLog } from '@/lib/logs/execution/display-types'
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
import { processStreamingBlockLogs } from '@/lib/tokenization'
import type {
@@ -103,6 +104,20 @@ interface DebugValidationResult {
const WORKFLOW_EXECUTION_FAILURE_MESSAGE = 'Workflow execution failed'
function getExecutionDisplayError(data: unknown): {
displayError?: string
hasDisplayProjection: boolean
} {
if (!data || typeof data !== 'object' || !Object.hasOwn(data, 'display')) {
return { hasDisplayProjection: false }
}
const display = (data as { display?: { error?: unknown } }).display
return {
hasDisplayProjection: true,
...(typeof display?.error === 'string' ? { displayError: display.error } : {}),
}
}
async function persistExecutionPointerProgress(
workflowId: string,
executionId: string,
@@ -261,6 +276,16 @@ function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamC
)
}
const clearThinking = (blockId: string) => {
const timer = thinkingFlushTimers.get(blockId)
if (timer !== undefined) {
clearTimeout(timer)
thinkingFlushTimers.delete(blockId)
}
thinkingByBlock.delete(blockId)
updateConsole(blockId, { clearAgentStreamThinking: true }, executionIdRef.current)
}
const settleBlock = (blockId: string, status: 'success' | 'error' | 'cancelled') => {
flushThinking(blockId)
const map = toolCallsByBlock.get(blockId)
@@ -288,8 +313,21 @@ function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamC
}
const onStreamThinking = (data: StreamThinkingData) => {
const display = (
data as StreamThinkingData & {
display?: { text?: string; clearLiveDisplay?: true }
}
).display
const hasDisplayProjection = Object.hasOwn(data, 'display')
const text = hasDisplayProjection ? display?.text : data.text
if (display?.clearLiveDisplay || (hasDisplayProjection && typeof text !== 'string')) {
clearThinking(data.blockId)
return
}
if (!text) return
const prev = thinkingByBlock.get(data.blockId) ?? ''
thinkingByBlock.set(data.blockId, prev + data.text)
thinkingByBlock.set(data.blockId, prev + text)
if (!thinkingFlushTimers.has(data.blockId)) {
thinkingFlushTimers.set(
data.blockId,
@@ -334,7 +372,14 @@ function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamC
settleBlock(data.blockId, 'success')
}
return { flushThinking, settleBlock, settleAll, onStreamThinking, onStreamTool, onStreamDone }
return {
flushThinking,
settleBlock,
settleAll,
onStreamThinking,
onStreamTool,
onStreamDone,
}
}
export function useWorkflowExecution() {
@@ -464,10 +509,12 @@ export function useWorkflowExecution() {
workflowId?: string
executionId?: string
error?: string
displayError?: string
hasDisplayProjection?: boolean
durationMs?: number
blockLogs: BlockLog[]
isPreExecutionError?: boolean
finalBlockLogs?: BlockLog[]
finalBlockLogs?: SecretSafeBlockLog[]
}) => {
if (!params.workflowId) return
sharedHandleExecutionErrorConsole(
@@ -483,7 +530,7 @@ export function useWorkflowExecution() {
workflowId?: string
executionId?: string
durationMs?: number
finalBlockLogs?: BlockLog[]
finalBlockLogs?: SecretSafeBlockLog[]
}) => {
if (!params.workflowId) return
sharedHandleExecutionCancelledConsole(
@@ -862,27 +909,7 @@ export function useWorkflowExecution() {
streamedContent.set(id, chunks.join(''))
}
// Update streamed content and apply tokenization
if (result.logs) {
result.logs.forEach((log: BlockLog) => {
if (streamedContent.has(log.blockId)) {
// For console display, show the actual structured block output instead of formatted streaming content
// This ensures console logs match the block state structure
// Use replaceOutput to completely replace the output instead of merging
// Use the executionId from this execution context
useTerminalConsoleStore.getState().updateConsole(
log.blockId,
{
executionOrder: log.executionOrder,
replaceOutput: log.output,
success: true,
},
executionId
)
}
})
// Process all logs for streaming tokenization
const processedCount = processStreamingBlockLogs(result.logs, streamedContent)
logger.info(`Processed ${processedCount} blocks for streaming tokenization`)
}
@@ -1208,7 +1235,6 @@ export function useWorkflowExecution() {
const activeBlockRefCounts = new Map<string, number>()
const streamedChunks = new Map<string, string[]>()
const agentStreamChrome = createAgentStreamChrome({ executionIdRef, updateConsole })
const settleAgentStreamChrome = agentStreamChrome.settleBlock
const settleAllAgentStreamChrome = agentStreamChrome.settleAll
const accumulatedBlockLogs: BlockLog[] = []
const accumulatedBlockStates = new Map<string, BlockState>()
@@ -1282,8 +1308,7 @@ export function useWorkflowExecution() {
onBlockStarted: blockHandlers.onBlockStarted,
onBlockCompleted: blockHandlers.onBlockCompleted,
onBlockError: (data) => {
// Failures often skip stream:done — settle thinking/tool chrome here.
settleAgentStreamChrome(data.blockId, 'error')
agentStreamChrome.settleBlock(data.blockId, 'error')
blockHandlers.onBlockError(data)
},
onBlockChildWorkflowStarted: blockHandlers.onBlockChildWorkflowStarted,
@@ -1388,6 +1413,7 @@ export function useWorkflowExecution() {
decisions: existingSnapshot?.decisions || { router: {}, condition: {} },
completedLoops: existingSnapshot?.completedLoops || [],
activeExecutionPath: Array.from(mergedExecutedBlocks),
sourceExecutionId: executionIdRef.current,
}
setLastExecutionSnapshot(activeWorkflowId, snapshot)
logger.info('Merged execution snapshot after run-until-block', {
@@ -1403,6 +1429,7 @@ export function useWorkflowExecution() {
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: Array.from(executedBlockIds),
sourceExecutionId: executionIdRef.current,
}
setLastExecutionSnapshot(activeWorkflowId, snapshot)
logger.info('Stored execution snapshot for run-from-block', {
@@ -1508,6 +1535,7 @@ export function useWorkflowExecution() {
workflowId: activeWorkflowId,
executionId: executionIdRef.current,
error: data.error,
...getExecutionDisplayError(data),
durationMs: data.duration,
blockLogs: accumulatedBlockLogs,
isPreExecutionError,
@@ -1927,6 +1955,7 @@ export function useWorkflowExecution() {
const effectiveSnapshot: SerializableExecutionState = isTriggerBlock
? emptySnapshot
: snapshot || emptySnapshot
const sourceExecutionId = isTriggerBlock ? undefined : effectiveSnapshot.sourceExecutionId
// Extract mock payload for trigger blocks
let workflowInput: any
@@ -2009,6 +2038,7 @@ export function useWorkflowExecution() {
workflowId,
startBlockId: blockId,
sourceSnapshot: effectiveSnapshot,
...(sourceExecutionId ? { sourceExecutionId } : {}),
input: workflowInput,
onExecutionId: (id) => {
if (runFromBlockOwnerRef.current !== runOwnerId) return
@@ -2031,7 +2061,6 @@ export function useWorkflowExecution() {
onBlockStarted: blockHandlers.onBlockStarted,
onBlockCompleted: blockHandlers.onBlockCompleted,
onBlockError: (data) => {
// Failures often skip stream:done — settle thinking/tool chrome here.
agentStreamChrome.settleBlock(data.blockId, 'error')
blockHandlers.onBlockError(data)
},
@@ -2065,6 +2094,7 @@ export function useWorkflowExecution() {
const updatedSnapshot: SerializableExecutionState = {
...effectiveSnapshot,
sourceExecutionId: executionId,
blockStates: mergedBlockStates,
executedBlocks: Array.from(mergedExecutedBlocks),
blockLogs: [...effectiveSnapshot.blockLogs, ...accumulatedBlockLogs],
@@ -2116,6 +2146,7 @@ export function useWorkflowExecution() {
workflowId,
executionId,
error: data.error,
...getExecutionDisplayError(data),
durationMs: data.duration,
blockLogs: accumulatedBlockLogs,
finalBlockLogs: data.finalBlockLogs,
@@ -2473,6 +2504,7 @@ export function useWorkflowExecution() {
workflowId: reconnectWorkflowId,
executionId: capturedExecutionId,
error: data.error,
...getExecutionDisplayError(data),
blockLogs: accumulatedBlockLogs,
finalBlockLogs: data.finalBlockLogs,
})
@@ -30,6 +30,49 @@ describe('reconcileFinalBlockLogs (real store)', () => {
} as any)
})
it('replaces completed terminal content with the final server projection', () => {
const store = useTerminalConsoleStore.getState()
store.addConsole({
workflowId: 'wf-1',
blockId: 'function-1',
blockName: 'Function 1',
blockType: 'function',
executionId: 'exec-1',
executionOrder: 1,
isRunning: false,
success: false,
input: { code: 'return resolved-secret-value-123' },
output: { error: 'resolved-secret-value-123' },
error: 'SyntaxError: resolved-secret-value-123',
agentStreamThinking: 'resolved-secret-value-123',
})
const startedAt = new Date().toISOString()
reconcileFinalBlockLogs(store.updateConsole, 'wf-1', 'exec-1', [
{
blockId: 'function-1',
blockName: 'Function 1',
blockType: 'function',
executionOrder: 1,
success: false,
input: { code: 'return {{SECRET_NAME}}' },
output: { error: '{{SECRET_NAME}}' },
error: 'SyntaxError: {{SECRET_NAME}}',
clearLiveDisplay: true,
startedAt,
endedAt: startedAt,
durationMs: 1,
} as any,
])
const entry = useTerminalConsoleStore.getState().getWorkflowEntries('wf-1')[0]
expect(entry.input).toEqual({ code: 'return {{SECRET_NAME}}' })
expect(entry.output).toEqual({ error: '{{SECRET_NAME}}' })
expect(entry.error).toBe('SyntaxError: {{SECRET_NAME}}')
expect(entry.agentStreamThinking).toBeUndefined()
expect(JSON.stringify(entry)).not.toContain('resolved-secret-value-123')
})
it('actually flips a child-workflow inner block from running to success', () => {
const store = useTerminalConsoleStore.getState()
store.addConsole({
@@ -5,6 +5,7 @@ import { resetTerminalConsoleMock, terminalConsoleMockFns } from '@sim/testing'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import {
addExecutionErrorConsoleEntry,
addHttpErrorConsoleEntry,
createBlockEventHandlers,
handleExecutionErrorConsole,
reconcileFinalBlockLogs,
@@ -195,6 +196,171 @@ describe('workflow-execution-utils', () => {
'exec-1',
])
})
it('keeps raw completion data functional while writing only the display projection', async () => {
const accumulatedBlockLogs: BlockLog[] = []
const accumulatedBlockStates = new Map()
const updateConsole = vi.fn()
const onBlockCompleteCallback = vi.fn().mockResolvedValue(undefined)
const handlers = createBlockEventHandlers(
{
workflowId: 'wf-1',
executionIdRef: { current: 'exec-1' },
workflowEdges: [],
activeBlocksSet: new Set<string>(),
activeBlockRefCounts: new Map<string, number>(),
accumulatedBlockLogs,
accumulatedBlockStates,
executedBlockIds: new Set<string>(),
includeStartConsoleEntry: true,
onBlockCompleteCallback,
},
{
addConsole: vi.fn(),
updateConsole,
setActiveBlocks: vi.fn(),
setBlockRunStatus: vi.fn(),
setEdgeRunStatus: vi.fn(),
}
)
handlers.onBlockCompleted({
blockId: 'fn-1',
blockName: 'Function 1',
blockType: 'function',
executionOrder: 1,
input: { code: 'return sk-resolved-secret' },
output: { result: 'sk-resolved-secret' },
display: {
input: { code: 'return {{OPENAI_API_KEY}}' },
output: { result: '{{OPENAI_API_KEY}}' },
},
durationMs: 10,
startedAt: '2026-07-31T00:00:00.000Z',
endedAt: '2026-07-31T00:00:00.010Z',
} as any)
expect(accumulatedBlockLogs[0]).toMatchObject({
input: { code: 'return sk-resolved-secret' },
output: { result: 'sk-resolved-secret' },
})
expect(accumulatedBlockStates.get('fn-1')?.output).toEqual({
result: 'sk-resolved-secret',
})
expect(onBlockCompleteCallback).toHaveBeenCalledWith('fn-1', {
result: 'sk-resolved-secret',
})
expect(updateConsole).toHaveBeenCalledWith(
'fn-1',
expect.objectContaining({
input: { code: 'return {{OPENAI_API_KEY}}' },
replaceOutput: { result: '{{OPENAI_API_KEY}}' },
}),
'exec-1'
)
expect(updateConsole.mock.calls[0][1]).not.toHaveProperty('clearAgentStreamThinking')
expect(JSON.stringify(updateConsole.mock.calls)).not.toContain('sk-resolved-secret')
})
it('does not fall back to a raw block error when the display projection is empty', () => {
const accumulatedBlockLogs: BlockLog[] = []
const accumulatedBlockStates = new Map()
const updateConsole = vi.fn()
const handlers = createBlockEventHandlers(
{
workflowId: 'wf-1',
executionIdRef: { current: 'exec-1' },
workflowEdges: [],
activeBlocksSet: new Set<string>(),
activeBlockRefCounts: new Map<string, number>(),
accumulatedBlockLogs,
accumulatedBlockStates,
executedBlockIds: new Set<string>(),
includeStartConsoleEntry: true,
},
{
addConsole: vi.fn(),
updateConsole,
setActiveBlocks: vi.fn(),
setBlockRunStatus: vi.fn(),
setEdgeRunStatus: vi.fn(),
}
)
handlers.onBlockError({
blockId: 'fn-1',
blockName: 'Function 1',
blockType: 'function',
executionOrder: 1,
input: { code: 'return sk-resolved-secret' },
error: 'SyntaxError: sk-resolved-secret',
display: {},
durationMs: 10,
startedAt: '2026-07-31T00:00:00.000Z',
endedAt: '2026-07-31T00:00:00.010Z',
})
expect(accumulatedBlockLogs[0]?.error).toBe('SyntaxError: sk-resolved-secret')
expect(accumulatedBlockStates.get('fn-1')?.output).toEqual({
error: 'SyntaxError: sk-resolved-secret',
})
expect(updateConsole).toHaveBeenCalledWith(
'fn-1',
expect.objectContaining({
input: {},
replaceOutput: {},
error: 'Block failed',
clearAgentStreamThinking: true,
}),
'exec-1'
)
expect(JSON.stringify(updateConsole.mock.calls)).not.toContain('sk-resolved-secret')
})
it('preserves legacy block error display when the server sends no projection', () => {
const updateConsole = vi.fn()
const handlers = createBlockEventHandlers(
{
workflowId: 'wf-1',
executionIdRef: { current: 'exec-1' },
workflowEdges: [],
activeBlocksSet: new Set<string>(),
activeBlockRefCounts: new Map<string, number>(),
accumulatedBlockLogs: [],
accumulatedBlockStates: new Map(),
executedBlockIds: new Set<string>(),
includeStartConsoleEntry: true,
},
{
addConsole: vi.fn(),
updateConsole,
setActiveBlocks: vi.fn(),
setBlockRunStatus: vi.fn(),
setEdgeRunStatus: vi.fn(),
}
)
handlers.onBlockError({
blockId: 'fn-1',
blockName: 'Function 1',
blockType: 'function',
executionOrder: 1,
input: { code: 'return ordinary-value' },
error: 'SyntaxError: ordinary-value',
durationMs: 10,
startedAt: '2026-07-31T00:00:00.000Z',
endedAt: '2026-07-31T00:00:00.010Z',
})
expect(updateConsole).toHaveBeenCalledWith(
'fn-1',
expect.objectContaining({
input: { code: 'return ordinary-value' },
error: 'SyntaxError: ordinary-value',
}),
'exec-1'
)
})
})
describe('addExecutionErrorConsoleEntry', () => {
@@ -204,6 +370,7 @@ describe('workflow-execution-utils', () => {
workflowId: 'wf-1',
executionId: 'exec-1',
error: 'Run failed',
displayError: 'Safe run failure',
durationMs: 1234,
blockLogs: [],
})
@@ -212,7 +379,45 @@ describe('workflow-execution-utils', () => {
const entry = addConsole.mock.calls[0][0]
expect(entry.blockName).toBe('Run Error')
expect(entry.blockType).toBe('error')
expect(entry.error).toBe('Run failed')
expect(entry.error).toBe('Safe run failure')
})
it('does not use the raw execution error when the server projection is empty', () => {
const addConsole = vi.fn()
addExecutionErrorConsoleEntry(addConsole, {
workflowId: 'wf-1',
executionId: 'exec-1',
error: 'SyntaxError: sk-resolved-secret',
hasDisplayProjection: true,
blockLogs: [],
})
expect(addConsole.mock.calls[0][0].error).toBe('Run failed')
expect(JSON.stringify(addConsole.mock.calls)).not.toContain('sk-resolved-secret')
})
it('preserves legacy execution errors when the server sends no projection', () => {
const addConsole = vi.fn()
addExecutionErrorConsoleEntry(addConsole, {
workflowId: 'wf-1',
executionId: 'exec-1',
error: 'Legacy run failure',
blockLogs: [],
})
expect(addConsole.mock.calls[0][0].error).toBe('Legacy run failure')
})
it('preserves HTTP error detail before SSE projection is available', () => {
const addConsole = vi.fn()
addHttpErrorConsoleEntry(addConsole, {
workflowId: 'wf-1',
executionId: 'exec-1',
error: 'Workflow is archived',
httpStatus: 409,
})
expect(addConsole.mock.calls[0][0].error).toBe('Workflow is archived')
})
it('skips when blockLogs already contain a block-level error', () => {
@@ -407,6 +612,74 @@ describe('workflow-execution-utils', () => {
expect(updateConsole).not.toHaveBeenCalled()
})
it('reprojects completed content without deep-comparing authoritative finalBlockLogs', () => {
terminalConsoleMockFns.mockAddConsole({
workflowId: 'wf-1',
blockId: 'fn-1',
blockName: 'Function',
blockType: 'function',
executionId: 'exec-1',
executionOrder: 1,
isRunning: false,
success: false,
input: { code: 'return sk-resolved-secret' },
output: { error: 'sk-resolved-secret' },
error: 'SyntaxError: sk-resolved-secret',
agentStreamThinking: 'sk-resolved-secret',
})
const updateConsole = vi.fn()
reconcileFinalBlockLogs(updateConsole, 'wf-1', 'exec-1', [
makeLog({
blockId: 'fn-1',
input: { code: 'return {{OPENAI_API_KEY}}' },
output: { error: '{{OPENAI_API_KEY}}' },
error: 'SyntaxError: {{OPENAI_API_KEY}}',
success: false,
}),
])
expect(updateConsole).toHaveBeenCalledWith(
'fn-1',
expect.objectContaining({
input: { code: 'return {{OPENAI_API_KEY}}' },
replaceOutput: { error: '{{OPENAI_API_KEY}}' },
error: 'SyntaxError: {{OPENAI_API_KEY}}',
}),
'exec-1'
)
expect(updateConsole.mock.calls[0][1]).not.toHaveProperty('clearAgentStreamThinking')
expect(JSON.stringify(updateConsole.mock.calls)).not.toContain('sk-resolved-secret')
})
it('clears live content when the final projection is structural-only', () => {
terminalConsoleMockFns.mockAddConsole({
workflowId: 'wf-1',
blockId: 'fn-1',
blockName: 'Function',
blockType: 'function',
executionId: 'exec-1',
executionOrder: 1,
isRunning: false,
success: false,
input: { code: 'return sk-resolved-secret' },
output: { error: 'sk-resolved-secret' },
error: 'SyntaxError: sk-resolved-secret',
})
const updateConsole = vi.fn()
reconcileFinalBlockLogs(updateConsole, 'wf-1', 'exec-1', [
makeLog({ blockId: 'fn-1', success: false }),
])
expect(updateConsole.mock.calls[0][1]).toMatchObject({
input: {},
replaceOutput: {},
error: null,
clearAgentStreamThinking: true,
})
})
it('reconciles child workflow spans before running entries are swept to canceled', () => {
terminalConsoleMockFns.mockAddConsole({
workflowId: 'wf-1',
@@ -1,6 +1,7 @@
import { createLogger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { generateId } from '@sim/utils/id'
import type { SecretSafeBlockLog } from '@/lib/logs/execution/display-types'
import type { TraceSpan } from '@/lib/logs/types'
import type {
BlockChildWorkflowStartedData,
@@ -15,9 +16,6 @@ import {
SSEEventHandlerError,
SSEStreamInterruptedError,
} from '@/hooks/use-execution-stream'
const logger = createLogger('workflow-execution-utils')
import { useExecutionStore } from '@/stores/execution'
import type { ConsoleEntry, ConsoleUpdate } from '@/stores/terminal'
import {
@@ -29,6 +27,11 @@ import {
import { useWorkflowRegistry } from '@/stores/workflows/registry/store'
import { useWorkflowStore } from '@/stores/workflows/workflow/store'
const logger = createLogger('workflow-execution-utils')
const BLOCK_FAILURE_DISPLAY_MESSAGE = 'Block failed'
const RUN_FAILURE_DISPLAY_MESSAGE = 'Run failed'
const VALIDATION_FAILURE_DISPLAY_MESSAGE = 'Workflow validation failed'
/**
* Updates the active blocks set and ref counts for a single block.
* Ref counting ensures a block stays active until all parallel branches for it complete.
@@ -132,6 +135,46 @@ interface BlockEventHandlerDeps {
type BlockChildWorkflowStartedUpdate = BlockChildWorkflowStartedData
interface BlockCompletedDisplayProjection {
input?: unknown
output?: unknown
clearLiveDisplay?: true
}
interface BlockErrorDisplayProjection {
input?: unknown
error?: string
clearLiveDisplay?: true
}
interface ExecutionErrorDisplayProjection {
present: boolean
error?: string
}
function getBlockCompletedDisplay(data: BlockCompletedData): BlockCompletedDisplayProjection {
const event = data as BlockCompletedData & { display?: BlockCompletedDisplayProjection }
if (!Object.hasOwn(event, 'display')) {
return { input: data.input, output: data.output }
}
return event.display ?? {}
}
function getBlockErrorDisplay(data: BlockErrorData): BlockErrorDisplayProjection {
const event = data as BlockErrorData & { display?: BlockErrorDisplayProjection }
if (!Object.hasOwn(event, 'display')) {
return { input: data.input, error: data.error }
}
return event.display ?? {}
}
function getExecutionErrorDisplay(data: { error: string }): ExecutionErrorDisplayProjection {
const event = data as typeof data & { display?: Omit<ExecutionErrorDisplayProjection, 'present'> }
if (!Object.hasOwn(event, 'display')) return { present: false }
const error = event.display?.error
return { present: true, ...(typeof error === 'string' ? { error } : {}) }
}
/**
* Creates block event handlers for SSE execution events.
* Shared by the workflow execution hook and standalone execution utilities.
@@ -299,19 +342,26 @@ export function createBlockEventHandlers(
})
const updateConsoleEntry = (data: BlockCompletedData) => {
const display = getBlockCompletedDisplay(data)
const projectionOmittedContent =
display.clearLiveDisplay === true ||
(Object.hasOwn(data, 'display') &&
((!Object.hasOwn(display, 'input') && data.input !== undefined) ||
(!Object.hasOwn(display, 'output') && data.output !== undefined)))
updateConsole(
data.blockId,
{
blockName: data.blockName,
blockType: data.blockType,
executionOrder: data.executionOrder,
input: data.input || {},
replaceOutput: data.output,
input: display.input ?? {},
replaceOutput: (display.output ?? {}) as Record<string, unknown>,
success: true,
durationMs: data.durationMs,
startedAt: data.startedAt,
endedAt: data.endedAt,
isRunning: false,
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
...extractIterationFields(data),
},
executionIdRef.current
@@ -319,20 +369,27 @@ export function createBlockEventHandlers(
}
const updateConsoleErrorEntry = (data: BlockErrorData) => {
const display = getBlockErrorDisplay(data)
const projectionOmittedContent =
display.clearLiveDisplay === true ||
(Object.hasOwn(data, 'display') &&
((!Object.hasOwn(display, 'input') && data.input !== undefined) ||
(!Object.hasOwn(display, 'error') && data.error !== undefined)))
updateConsole(
data.blockId,
{
blockName: data.blockName,
blockType: data.blockType,
executionOrder: data.executionOrder,
input: data.input || {},
input: display.input ?? {},
replaceOutput: {},
success: false,
error: data.error,
error: display.error || BLOCK_FAILURE_DISPLAY_MESSAGE,
durationMs: data.durationMs,
startedAt: data.startedAt,
endedAt: data.endedAt,
isRunning: false,
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
...extractIterationFields(data),
},
executionIdRef.current
@@ -472,16 +529,16 @@ interface ExecutionConsoleDeps {
}
/**
* Reconciles still-running console entries with the server's authoritative
* `finalBlockLogs` so that any block whose terminal `block:completed`/`block:error`
* SSE event was lost gets the correct success/error state instead of being
* swept to "canceled".
* Reconciles console entries with the server's authoritative, secret-safe
* `finalBlockLogs`. Reapplying running or content-bearing rows both recovers
* dropped terminal events and replaces an earlier live projection after late
* secret activation.
*/
export function reconcileFinalBlockLogs(
updateConsole: UpdateConsoleFn,
workflowId: string,
executionId: string | undefined,
finalBlockLogs: BlockLog[] | undefined
finalBlockLogs: SecretSafeBlockLog[] | undefined
): void {
if (!finalBlockLogs?.length || !executionId) return
for (const log of finalBlockLogs) {
@@ -491,8 +548,17 @@ export function reconcileFinalBlockLogs(
entry.executionId === executionId &&
entry.executionOrder === log.executionOrder
const matchingEntry = entries.find(matchesFinalLog)
const runningEntry = entries.find((entry) => matchesFinalLog(entry) && entry.isRunning)
if (runningEntry) {
const hasExistingContent =
matchingEntry?.input !== undefined ||
matchingEntry?.output !== undefined ||
matchingEntry?.error !== undefined ||
matchingEntry?.agentStreamThinking !== undefined
if (matchingEntry && (matchingEntry.isRunning || hasExistingContent)) {
const projectionOmittedContent =
log.clearLiveDisplay === true ||
(log.input === undefined && matchingEntry.input !== undefined) ||
(log.output === undefined && matchingEntry.output !== undefined) ||
(log.error === undefined && matchingEntry.error !== undefined)
updateConsole(
log.blockId,
{
@@ -500,14 +566,15 @@ export function reconcileFinalBlockLogs(
blockName: log.blockName,
blockType: log.blockType,
replaceOutput: (log.output ?? {}) as Record<string, unknown>,
...(log.input ? { input: log.input } : {}),
input: log.input ?? {},
success: log.success,
...(log.error ? { error: log.error } : {}),
error: log.error ?? null,
durationMs: log.durationMs,
startedAt: log.startedAt,
endedAt: log.endedAt,
isRunning: false,
isCanceled: false,
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
},
executionId
)
@@ -549,19 +616,26 @@ function reconcileChildTraceSpans(
? findConsoleEntryForSpan(workflowId, executionId, childWorkflowInstanceId, span)
: undefined
if (span.blockId) {
const errorMessage = normalizeSpanError(span.output?.error)
const errorMessage = normalizeSpanError(span.errorMessage ?? span.output?.error)
const projectionOmittedContent = matchingEntry
? (span.input === undefined && matchingEntry.input !== undefined) ||
(span.output === undefined && matchingEntry.output !== undefined) ||
(errorMessage === undefined && matchingEntry.error !== undefined)
: false
updateConsole(
span.blockId,
{
...spanConsoleIdentity(span, childWorkflowInstanceId),
input: span.input ?? {},
replaceOutput: (span.output ?? {}) as Record<string, unknown>,
success: span.status !== 'error',
...(errorMessage !== undefined ? { error: errorMessage } : {}),
error: errorMessage ?? null,
durationMs: span.duration,
startedAt: span.startTime,
endedAt: span.endTime,
isRunning: false,
isCanceled: false,
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
},
executionId
)
@@ -670,12 +744,17 @@ export function buildExecutionTiming(durationMs?: number): ExecutionTimingFields
interface ExecutionErrorConsoleParams {
workflowId: string
executionId?: string
/** Raw runtime error used only for functional classification and result propagation. */
error?: string
/** Server-projected error text safe for terminal display. */
displayError?: string
/** Distinguishes an intentionally empty projection from a legacy event with no projection. */
hasDisplayProjection?: boolean
durationMs?: number
blockLogs: BlockLog[]
isPreExecutionError?: boolean
/** Server's authoritative per-block terminal states, used to reconcile lost SSE events. */
finalBlockLogs?: BlockLog[]
finalBlockLogs?: SecretSafeBlockLog[]
}
/**
@@ -702,8 +781,14 @@ export function addExecutionErrorConsoleEntry(
const isPreExecutionError = params.isPreExecutionError ?? false
if (!isPreExecutionError && hasBlockError) return
const errorMessage = params.error || 'Run failed'
const isTimeout = errorMessage.toLowerCase().includes('timed out')
const hasDisplayProjection = params.hasDisplayProjection ?? params.displayError !== undefined
const fallbackMessage = isPreExecutionError
? VALIDATION_FAILURE_DISPLAY_MESSAGE
: RUN_FAILURE_DISPLAY_MESSAGE
const errorMessage = hasDisplayProjection
? params.displayError || fallbackMessage
: params.error || fallbackMessage
const isTimeout = (params.error ?? params.displayError ?? '').toLowerCase().includes('timed out')
const timing = buildExecutionTiming(params.durationMs)
addConsole({
@@ -784,7 +869,7 @@ interface CancelledConsoleParams {
executionId?: string
durationMs?: number
/** Server's authoritative per-block terminal states, used to reconcile lost SSE events. */
finalBlockLogs?: BlockLog[]
finalBlockLogs?: SecretSafeBlockLog[]
}
/**
@@ -1060,6 +1145,7 @@ export async function executeWorkflowWithFullLogging(
if (!isCurrentExecution()) return
executionFinished = true
const errorMessage = data.error || 'Run failed'
const display = getExecutionErrorDisplay(data)
executionResult = {
success: false,
output: {},
@@ -1074,6 +1160,8 @@ export async function executeWorkflowWithFullLogging(
workflowId: wfId,
executionId: executionIdRef.current,
error: errorMessage,
displayError: display.error,
hasDisplayProjection: display.present,
durationMs: data.duration || 0,
blockLogs: accumulatedBlockLogs,
isPreExecutionError: accumulatedBlockLogs.length === 0,
@@ -9,6 +9,7 @@ import {
executionPreprocessingMockFns,
LoggingSessionMock,
loggingSessionMock,
loggingSessionMockFns,
resetDbChainMock,
workflowsPersistenceUtilsMock,
workflowsPersistenceUtilsMockFns,
@@ -16,13 +17,21 @@ import {
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { ADMISSION_ERROR_CODE } from '@/lib/core/admission/transient-failure'
const { mockTask, mockExecuteWorkflowCore, mockGetScheduleTimeValues, mockGetSubBlockValue } =
vi.hoisted(() => ({
mockTask: vi.fn((config) => config),
mockExecuteWorkflowCore: vi.fn(),
mockGetScheduleTimeValues: vi.fn(),
mockGetSubBlockValue: vi.fn(),
}))
const {
mockTask,
mockExecuteWorkflowCore,
mockWasExecutionFinalizedByCore,
mockHasExecutionResult,
mockGetScheduleTimeValues,
mockGetSubBlockValue,
} = vi.hoisted(() => ({
mockTask: vi.fn((config) => config),
mockExecuteWorkflowCore: vi.fn(),
mockWasExecutionFinalizedByCore: vi.fn(),
mockHasExecutionResult: vi.fn(),
mockGetScheduleTimeValues: vi.fn(),
mockGetSubBlockValue: vi.fn(),
}))
const mockPreprocessExecution = executionPreprocessingMockFns.mockPreprocessExecution
const mockLoadDeployedWorkflowState = workflowsPersistenceUtilsMockFns.mockLoadDeployedWorkflowState
@@ -55,7 +64,7 @@ vi.mock('@/lib/logs/execution/trace-spans/trace-spans', () => ({
vi.mock('@/lib/workflows/executor/execution-core', () => ({
executeWorkflowCore: mockExecuteWorkflowCore,
wasExecutionFinalizedByCore: vi.fn().mockReturnValue(false),
wasExecutionFinalizedByCore: mockWasExecutionFinalizedByCore,
}))
vi.mock('@/lib/workflows/executor/human-in-the-loop-manager', () => ({
@@ -78,7 +87,7 @@ vi.mock('@/executor/execution/snapshot', () => ({
}))
vi.mock('@/executor/utils/errors', () => ({
hasExecutionResult: vi.fn().mockReturnValue(false),
hasExecutionResult: mockHasExecutionResult,
}))
import { executeScheduleJob } from './schedule-execution'
@@ -100,6 +109,8 @@ const billingAttribution = {
describe('async preprocessing correlation threading', () => {
beforeEach(() => {
vi.clearAllMocks()
mockWasExecutionFinalizedByCore.mockReturnValue(false)
mockHasExecutionResult.mockReturnValue(false)
resetDbChainMock()
dbChainMockFns.limit.mockResolvedValue([
{
@@ -166,6 +177,84 @@ describe('async preprocessing correlation threading', () => {
)
})
it('preserves a core-finalized execution error for task failure semantics', async () => {
const rawError = Object.assign(new Error('Function 1 failed with activated-secret-value'), {
executionResult: {
success: false,
output: { error: 'Function failed' },
logs: [],
},
})
mockPreprocessExecution.mockResolvedValueOnce({
success: true,
actorUserId: 'actor-1',
workflowRecord: {
id: 'workflow-1',
userId: 'owner-1',
workspaceId: 'workspace-1',
variables: {},
},
billingAttribution,
executionTimeout: {},
})
mockExecuteWorkflowCore.mockRejectedValueOnce(rawError)
mockHasExecutionResult.mockImplementation((error) => error === rawError)
mockWasExecutionFinalizedByCore.mockReturnValue(true)
await expect(
executeWorkflowJob({
workflowId: 'workflow-1',
userId: 'actor-1',
workspaceId: 'workspace-1',
billingAttribution,
triggerType: 'api',
executionId: 'execution-finalized',
requestId: 'request-finalized',
})
).rejects.toBe(rawError)
expect(loggingSessionMockFns.mockWaitForPostExecution).not.toHaveBeenCalled()
expect(mockWasExecutionFinalizedByCore).toHaveBeenCalledWith(rawError, 'execution-finalized')
expect(loggingSessionMockFns.mockSafeCompleteWithError).not.toHaveBeenCalled()
})
it('persists and rethrows the original unfinalized execution error', async () => {
const rawError = new Error('Function 1 failed with activated-secret-value')
mockPreprocessExecution.mockResolvedValueOnce({
success: true,
actorUserId: 'actor-1',
workflowRecord: {
id: 'workflow-1',
userId: 'owner-1',
workspaceId: 'workspace-1',
variables: {},
},
billingAttribution,
executionTimeout: {},
})
mockExecuteWorkflowCore.mockRejectedValueOnce(rawError)
await expect(
executeWorkflowJob({
workflowId: 'workflow-1',
userId: 'actor-1',
workspaceId: 'workspace-1',
billingAttribution,
triggerType: 'api',
executionId: 'execution-fault',
requestId: 'request-fault',
})
).rejects.toBe(rawError)
expect(loggingSessionMockFns.mockSafeCompleteWithError).toHaveBeenCalledWith(
expect.objectContaining({
error: expect.objectContaining({
message: 'Function 1 failed with activated-secret-value',
}),
})
)
})
it('does not pre-start schedule logging before core execution', async () => {
mockPreprocessExecution.mockResolvedValueOnce({
success: true,
@@ -0,0 +1,94 @@
/**
* @vitest-environment node
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
const {
mockTask,
mockGetPausedExecutionById,
mockStartResumeExecution,
mockFindCellContextByExecutionId,
mockSnapshotFromJson,
} = vi.hoisted(() => ({
mockTask: vi.fn((config) => config),
mockGetPausedExecutionById: vi.fn(),
mockStartResumeExecution: vi.fn(),
mockFindCellContextByExecutionId: vi.fn(),
mockSnapshotFromJson: vi.fn(),
}))
vi.mock('@trigger.dev/sdk', () => ({ task: mockTask }))
vi.mock('@/lib/billing/core/billing-attribution', () => ({
assertBillingAttributionSnapshot: vi.fn((value) => value),
}))
vi.mock('@/lib/table/cascade-lock', () => ({ withCascadeLock: vi.fn() }))
vi.mock('@/lib/table/deps', () => ({ isExecCancelled: vi.fn(() => false) }))
vi.mock('@/lib/table/workflow-columns', () => ({
findCellContextByExecutionId: mockFindCellContextByExecutionId,
}))
vi.mock('@/lib/workflows/executor/human-in-the-loop-manager', () => ({
PauseResumeManager: {
getPausedExecutionById: mockGetPausedExecutionById,
startResumeExecution: mockStartResumeExecution,
},
}))
vi.mock('@/executor/execution/snapshot', () => ({
ExecutionSnapshot: { fromJSON: mockSnapshotFromJson },
}))
import { executeResumeJob, type ResumeExecutionPayload } from '@/background/resume-execution'
const payload: ResumeExecutionPayload = {
resumeEntryId: 'resume-entry-1',
resumeExecutionId: 'resume-execution-1',
pausedExecutionId: 'paused-execution-1',
contextId: 'context-1',
resumeInput: {},
userId: 'user-1',
workflowId: 'workflow-1',
parentExecutionId: 'parent-execution-1',
}
describe('executeResumeJob terminal errors', () => {
beforeEach(() => {
vi.clearAllMocks()
mockGetPausedExecutionById.mockResolvedValue({
executionSnapshot: { snapshot: {} },
})
mockSnapshotFromJson.mockReturnValue({
metadata: {
billingAttribution: {
actorUserId: 'user-1',
workspaceId: 'workspace-1',
},
},
})
mockFindCellContextByExecutionId.mockResolvedValue(null)
})
it('rethrows the original core-finalized resume error', async () => {
const rawError = Object.assign(new Error('Agent tool exposed activated-secret-value'), {
executionResult: {
success: false,
output: { error: 'Agent tool failed' },
logs: [],
},
})
mockStartResumeExecution.mockRejectedValue(rawError)
await expect(executeResumeJob(payload)).rejects.toBe(rawError)
})
it('rethrows the original genuine resume fault', async () => {
const rawError = new Error('MCP setup exposed activated-secret-value')
mockStartResumeExecution.mockRejectedValue(rawError)
await expect(executeResumeJob(payload)).rejects.toBe(rawError)
})
})
+75 -1
View File
@@ -21,7 +21,13 @@ vi.mock('@/lib/workflows/schedules/disable-notifications', () => ({
// does not re-export. Widen it rather than rewriting the source's imports.
vi.mock('@sim/db', () => ({ ...databaseMock, ...schemaMock }))
import { applyScheduleFailureUpdate, releaseScheduleLock } from '@/background/schedule-execution'
import {
applyScheduleFailureUpdate,
readScheduledMothershipErrorResponse,
readScheduledMothershipJsonResponse,
releaseScheduleLock,
} from '@/background/schedule-execution'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
const BASE = {
scheduleId: 'schedule-1',
@@ -99,3 +105,71 @@ describe('releaseScheduleLock', () => {
expect(notifyMock).not.toHaveBeenCalled()
})
})
describe('scheduled Mothership response handling', () => {
it('parses JSON responses', async () => {
const response = new Response(JSON.stringify({ content: 'ok' }), {
headers: { 'content-type': 'application/json' },
})
await expect(readScheduledMothershipJsonResponse(response)).resolves.toEqual({ content: 'ok' })
})
it('preserves staging acceptance for responses above the generic tool cap', async () => {
const response = new Response(JSON.stringify({ content: 'unchanged' }), {
headers: {
'content-length': String(10 * 1024 * 1024 + 1),
'content-type': 'application/json',
},
})
await expect(readScheduledMothershipJsonResponse(response)).resolves.toEqual({
content: 'unchanged',
})
})
it('does not include malformed response content in parse failures', async () => {
const response = new Response('secret-bearing-non-json')
const error = await readScheduledMothershipJsonResponse(response).catch((caught) => caught)
expect(error).toBeInstanceOf(Error)
expect(error.message).toBe('Sim execution returned an invalid response')
expect(error.message).not.toContain('secret-bearing-non-json')
})
it('preserves the functional error body when no private metadata is present', async () => {
const registry = {
markIncomplete: vi.fn(),
importProvenance: vi.fn(),
} as unknown as ResolvedSecretTraceRegistry
const response = new Response('secret-bearing-error-body', { status: 500 })
const message = await readScheduledMothershipErrorResponse(response, registry)
expect(message).toBe('secret-bearing-error-body')
expect(registry.markIncomplete).toHaveBeenCalledTimes(1)
})
it('strips private provenance metadata without replacing the provider error', async () => {
const registry = {
markIncomplete: vi.fn(),
importProvenance: vi.fn().mockReturnValue(true),
} as unknown as ResolvedSecretTraceRegistry
const response = new Response(
JSON.stringify({
error: 'provider error detail',
__resolvedSecretTraceProvenance: { version: 1, complete: true, entries: [] },
}),
{
status: 500,
headers: { 'x-sim-private-tool-metadata': 'resolved-secret-provenance-v1' },
}
)
const message = await readScheduledMothershipErrorResponse(response, registry)
expect(JSON.parse(message)).toEqual({ error: 'provider error detail' })
expect(message).not.toContain('__resolvedSecretTraceProvenance')
expect(registry.importProvenance).toHaveBeenCalledOnce()
})
})
+130 -5
View File
@@ -9,6 +9,7 @@ import {
import { createLogger, runWithRequestContext } from '@sim/logger'
import { describeError, toError } from '@sim/utils/errors'
import { generateId } from '@sim/utils/id'
import { isPlainRecord } from '@sim/utils/object'
import { task } from '@trigger.dev/sdk'
import { Cron } from 'croner'
import { and, eq, isNull, ne, type SQL, sql } from 'drizzle-orm'
@@ -32,8 +33,16 @@ import {
} from '@/lib/core/execution-limits'
import type { DbOrTx } from '@/lib/db/types'
import { preprocessExecution } from '@/lib/execution/preprocessing'
import {
PRIVATE_TOOL_METADATA_REQUEST_HEADER,
RESOLVED_SECRET_PROVENANCE_FIELD,
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
responseHasPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
import { LoggingSession } from '@/lib/logs/execution/logging-session'
import { projectTraceSpansForSecrets } from '@/lib/logs/execution/trace-secret-projection'
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
import type { TraceSpan } from '@/lib/logs/types'
import { cleanupExecutionBase64Cache } from '@/lib/uploads/utils/user-file-base64.server'
import {
executeWorkflowCore,
@@ -61,6 +70,7 @@ import { ExecutionSnapshot } from '@/executor/execution/snapshot'
import type { ExecutionMetadata } from '@/executor/execution/types'
import { hasExecutionResult } from '@/executor/utils/errors'
import { buildAPIUrl, buildAuthHeaders } from '@/executor/utils/http'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import { MAX_CONSECUTIVE_FAILURES } from '@/triggers/constants'
const logger = createLogger('ScheduleExecution')
@@ -660,6 +670,7 @@ async function runWorkflowExecution({
stackTrace: error instanceof Error ? error.stack : undefined,
},
traceSpans,
executionState: executionResult?.executionState,
})
throw error
@@ -1207,14 +1218,94 @@ function buildJobPrompt(jobRecord: {
return parts.join('\n')
}
async function consumeJobTraceProvenance(
response: Response,
payload: Record<string, unknown>,
registry: ResolvedSecretTraceRegistry
): Promise<boolean> {
const hasProvenance = Object.hasOwn(payload, RESOLVED_SECRET_PROVENANCE_FIELD)
const provenance = payload[RESOLVED_SECRET_PROVENANCE_FIELD]
delete payload[RESOLVED_SECRET_PROVENANCE_FIELD]
if (
!responseHasPrivateToolMetadata(response.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1) ||
!hasProvenance
) {
registry.markIncomplete()
return false
}
return registry.importProvenance(provenance, { trusted: true })
}
/** Reads and validates a scheduled Mothership JSON response without changing its size contract. */
export async function readScheduledMothershipJsonResponse(
response: Response
): Promise<Record<string, unknown>> {
let payload: unknown
try {
payload = await response.json()
} catch {
throw new Error('Sim execution returned an invalid response')
}
if (!isPlainRecord(payload)) {
throw new Error('Sim execution returned an invalid response')
}
return payload
}
/** Reads the functional error body and strips private provenance metadata when present. */
export async function readScheduledMothershipErrorResponse(
response: Response,
registry: ResolvedSecretTraceRegistry
): Promise<string> {
const responseText = await response.text()
const hasPrivateMarker = responseHasPrivateToolMetadata(
response.headers,
RESOLVED_SECRET_PROVENANCE_METADATA_V1
)
const mayContainPrivateProvenance = responseText.includes(`"${RESOLVED_SECRET_PROVENANCE_FIELD}"`)
if (!hasPrivateMarker && !mayContainPrivateProvenance) {
registry.markIncomplete()
return responseText
}
let payload: unknown
try {
payload = JSON.parse(responseText)
} catch {
registry.markIncomplete()
return responseText
}
if (!isPlainRecord(payload)) {
registry.markIncomplete()
return responseText
}
const hadPrivateProvenance = Object.hasOwn(payload, RESOLVED_SECRET_PROVENANCE_FIELD)
try {
await consumeJobTraceProvenance(response, payload, registry)
} catch {
registry.markIncomplete()
}
return hadPrivateProvenance ? JSON.stringify(payload) : responseText
}
async function createJobLogEntry(params: {
scheduleId: string
workspaceId: string
userId: string
jobTitle: string | null
startTime: Date
endTime: Date
durationMs: number
success: boolean
resolvedSecretTraceRegistry: ResolvedSecretTraceRegistry
responseBody?: Record<string, any>
errorMessage?: string
}): Promise<void> {
@@ -1222,14 +1313,17 @@ async function createJobLogEntry(params: {
const {
scheduleId,
workspaceId,
userId,
jobTitle,
startTime,
endTime,
durationMs,
success,
resolvedSecretTraceRegistry,
responseBody,
} = params
const name = jobTitle || 'Sim Job'
const executionId = generateId()
const toolCallsList = (responseBody?.toolCalls || []).map((tc: Record<string, unknown>) => ({
name: tc.name,
@@ -1246,7 +1340,7 @@ async function createJobLogEntry(params: {
status: tc.error ? 'error' : 'success',
}))
const traceSpan = {
const traceSpan: TraceSpan = {
id: generateId(),
name,
type: 'mothership',
@@ -1263,12 +1357,20 @@ async function createJobLogEntry(params: {
cost: responseBody?.cost || undefined,
tokens: responseBody?.tokens || undefined,
}
const [projectedTraceSpan] = await projectTraceSpansForSecrets([traceSpan], {
registry: resolvedSecretTraceRegistry,
store: {
workspaceId,
executionId,
userId,
},
})
await db.insert(jobExecutionLogs).values({
id: generateId(),
scheduleId,
workspaceId,
executionId: generateId(),
executionId,
level: success ? 'info' : 'error',
status: success ? 'completed' : 'failed',
trigger: 'mothership',
@@ -1277,8 +1379,11 @@ async function createJobLogEntry(params: {
totalDurationMs: durationMs,
executionData: {
enhanced: true,
traceSpans: [traceSpan],
traceSpans: [projectedTraceSpan],
finalOutput: responseBody?.content ? { content: responseBody.content } : undefined,
executionState: {
resolvedSecretTraceProvenance: resolvedSecretTraceRegistry.exportProvenance(),
},
trigger: {
type: 'mothership',
source: name,
@@ -1369,6 +1474,10 @@ export async function executeJobInline(payload: JobExecutionPayload) {
}
const promptText = buildJobPrompt(jobRecord)
const resolvedSecretTraceRegistry = new ResolvedSecretTraceRegistry([], {
userId: jobRecord.sourceUserId,
workspaceId: jobRecord.sourceWorkspaceId,
})
try {
const billingAttribution = await resolveBillingAttribution({
@@ -1382,6 +1491,7 @@ export async function executeJobInline(payload: JobExecutionPayload) {
const url = buildAPIUrl('/api/mothership/execute')
const headers = await buildAuthHeaders(jobRecord.sourceUserId)
headers[BILLING_ATTRIBUTION_HEADER] = serializeBillingAttributionHeader(billingAttribution)
headers[PRIVATE_TOOL_METADATA_REQUEST_HEADER] = RESOLVED_SECRET_PROVENANCE_METADATA_V1
const body = {
messages: [{ role: 'user', content: promptText }],
@@ -1404,7 +1514,11 @@ export async function executeJobInline(payload: JobExecutionPayload) {
})
if (!response.ok) {
const errorText = await response.text().catch(() => {
const errorText = await readScheduledMothershipErrorResponse(
response,
resolvedSecretTraceRegistry
).catch(() => {
resolvedSecretTraceRegistry.markIncomplete()
if (timeoutController.isTimedOut()) {
throw new Error(getTimeoutErrorMessage(null, timeoutController.timeoutMs))
}
@@ -1416,11 +1530,13 @@ export async function executeJobInline(payload: JobExecutionPayload) {
await createJobLogEntry({
scheduleId: payload.scheduleId,
workspaceId: jobRecord.sourceWorkspaceId,
userId: jobRecord.sourceUserId,
jobTitle: jobRecord.jobTitle,
startTime,
endTime,
durationMs,
success: false,
resolvedSecretTraceRegistry,
errorMessage: errorText,
})
@@ -1430,10 +1546,17 @@ export async function executeJobInline(payload: JobExecutionPayload) {
let responseBody: Record<string, any> = {}
let wasCompletedByTool = false
try {
responseBody = await response.json()
const payload = await readScheduledMothershipJsonResponse(response)
responseBody = payload
try {
await consumeJobTraceProvenance(response, payload, resolvedSecretTraceRegistry)
} catch {
resolvedSecretTraceRegistry.markIncomplete()
}
const toolCalls = responseBody?.toolCalls as Array<{ name?: string }> | undefined
wasCompletedByTool = toolCalls?.some((tc) => tc.name === 'complete_scheduled_task') ?? false
} catch {
resolvedSecretTraceRegistry.markIncomplete()
if (timeoutController.isTimedOut()) {
throw new Error(getTimeoutErrorMessage(null, timeoutController.timeoutMs))
}
@@ -1444,11 +1567,13 @@ export async function executeJobInline(payload: JobExecutionPayload) {
await createJobLogEntry({
scheduleId: payload.scheduleId,
workspaceId: jobRecord.sourceWorkspaceId,
userId: jobRecord.sourceUserId,
jobTitle: jobRecord.jobTitle,
startTime,
endTime,
durationMs,
success: true,
resolvedSecretTraceRegistry,
responseBody,
})
+121 -18
View File
@@ -4,30 +4,33 @@
import {
dbChainMockFns,
environmentUtilsMockFns,
executionPreprocessingMock,
executionPreprocessingMockFns,
LoggingSessionMock,
loggingSessionMock,
loggingSessionMockFns,
resetEnvironmentUtilsMock,
} from '@sim/testing'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
const {
mockResolveWebhookRecordProviderConfig,
mockExecuteWorkflowCore,
mockWasExecutionFinalizedByCore,
mockRecordException,
mockGetActiveSpan,
mockExecuteWithIdempotency,
mockReleaseExecutionSlot,
mockLoadDeploymentVersionState,
mockGetProviderHandler,
mockSetResolvedSecretTraceRegistry,
} = vi.hoisted(() => ({
mockResolveWebhookRecordProviderConfig: vi.fn(),
mockExecuteWorkflowCore: vi.fn(),
mockWasExecutionFinalizedByCore: vi.fn(),
mockRecordException: vi.fn(),
mockGetActiveSpan: vi.fn(),
mockExecuteWithIdempotency: vi.fn(),
mockReleaseExecutionSlot: vi.fn(),
mockGetProviderHandler: vi.fn(() => ({})),
mockSetResolvedSecretTraceRegistry: vi.fn(),
mockLoadDeploymentVersionState: vi.fn(
async (_workflowId: string, deploymentVersionId: string) => ({
blocks: {},
@@ -39,9 +42,10 @@ const {
),
}))
vi.mock('@opentelemetry/api', () => ({
trace: { getActiveSpan: mockGetActiveSpan },
}))
const mockGetEffectiveEnvironmentSnapshot =
environmentUtilsMockFns.mockGetEffectiveEnvironmentSnapshot
afterAll(resetEnvironmentUtilsMock)
vi.mock('@/lib/execution/preprocessing', () => executionPreprocessingMock)
vi.mock('@/lib/logs/execution/logging-session', () => loggingSessionMock)
@@ -77,9 +81,7 @@ vi.mock('@/lib/workflows/persistence/utils', () => ({
loadWorkflowDeploymentVersionState: mockLoadDeploymentVersionState,
}))
vi.mock('@/lib/webhooks/providers', () => ({
getProviderHandler: vi.fn(() => ({})),
}))
vi.mock('@/lib/webhooks/providers', () => ({ getProviderHandler: mockGetProviderHandler }))
vi.mock('@/lib/logs/execution/trace-spans/trace-spans', () => ({
buildTraceSpans: vi.fn(() => ({ traceSpans: [] })),
@@ -208,6 +210,17 @@ describe('executeWebhookJob fault vs error handling', () => {
beforeEach(() => {
vi.clearAllMocks()
LoggingSessionMock.mockImplementation(function LoggingSession() {
return {
safeStart: loggingSessionMockFns.mockSafeStart,
safeComplete: loggingSessionMockFns.mockSafeComplete,
safeCompleteWithError: loggingSessionMockFns.mockSafeCompleteWithError,
waitForPostExecution: loggingSessionMockFns.mockWaitForPostExecution,
markAsFailed: loggingSessionMockFns.mockMarkAsFailed,
setResolvedSecretTraceRegistry: mockSetResolvedSecretTraceRegistry,
}
})
mockGetProviderHandler.mockReturnValue({})
mockExecuteWithIdempotency.mockImplementation(
(_provider: string, _key: string, operation: () => Promise<unknown>) => operation()
)
@@ -225,8 +238,15 @@ describe('executeWebhookJob fault vs error handling', () => {
executionTimeout: { async: 120_000 },
})
mockResolveWebhookRecordProviderConfig.mockImplementation(async (record) => record)
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
personalEncrypted: {},
workspaceEncrypted: {},
personalDecrypted: {},
workspaceDecrypted: {},
conflicts: [],
decryptionFailures: [],
})
dbChainMockFns.limit.mockResolvedValue([{ id: 'webhook-1' }])
mockGetActiveSpan.mockReturnValue({ recordException: mockRecordException })
})
it('completes the run (does not throw) when the failure was finalized by core', async () => {
@@ -246,17 +266,14 @@ describe('executeWebhookJob fault vs error handling', () => {
expect(loggingSessionMockFns.mockWaitForPostExecution).toHaveBeenCalled()
// User/workflow errors are already recorded by core — the catch must not re-log them.
expect(loggingSessionMockFns.mockSafeCompleteWithError).not.toHaveBeenCalled()
// The error is still recorded on the run span so it stays visible in traces.
expect(mockRecordException).toHaveBeenCalledWith(
expect.objectContaining({ message: 'Gmail 2 is missing required fields: Label' })
)
})
it('faults the run (re-throws) when the failure was not finalized by core', async () => {
mockExecuteWorkflowCore.mockRejectedValue(new Error('Workflow state not found'))
const rawError = new Error('Workflow state not found')
mockExecuteWorkflowCore.mockRejectedValue(rawError)
mockWasExecutionFinalizedByCore.mockReturnValue(false)
await expect(executeWebhookJob(payload)).rejects.toThrow('Workflow state not found')
await expect(executeWebhookJob(payload)).rejects.toBe(rawError)
// waitForPostExecution must run on every path so the finalized-by-core signal is always reliable.
expect(loggingSessionMockFns.mockWaitForPostExecution).toHaveBeenCalled()
// Pipeline/infra errors are recorded here before re-throwing to fault the trigger.dev run.
@@ -291,6 +308,92 @@ describe('executeWebhookJob fault vs error handling', () => {
)
})
it('passes encrypted webhook resolution provenance into workflow execution', async () => {
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
personalEncrypted: { WEBHOOK_SECRET: 'personal-ciphertext' },
workspaceEncrypted: { WEBHOOK_SECRET: 'workspace-ciphertext' },
personalDecrypted: { WEBHOOK_SECRET: 'personal-value' },
workspaceDecrypted: { WEBHOOK_SECRET: 'workspace-value' },
conflicts: ['WEBHOOK_SECRET'],
decryptionFailures: [],
})
mockResolveWebhookRecordProviderConfig.mockImplementation(
async (record, _userId, _workspaceId, options) => {
options.onResolved('WEBHOOK_SECRET', options.envVars.WEBHOOK_SECRET)
return record
}
)
mockExecuteWorkflowCore.mockResolvedValue({
success: true,
status: 'completed',
output: {},
logs: [],
executionState: {
blockStates: {},
executedBlocks: [],
blockLogs: [],
decisions: {},
completedLoops: [],
activeExecutionPath: [],
},
})
await executeWebhookJob(payload)
expect(mockResolveWebhookRecordProviderConfig).toHaveBeenCalledWith(
{ id: 'webhook-1' },
'user-1',
'workspace-1',
expect.objectContaining({
envVars: { WEBHOOK_SECRET: 'workspace-value' },
onResolved: expect.any(Function),
})
)
expect(mockExecuteWorkflowCore).toHaveBeenCalledWith(
expect.objectContaining({
trustedInitialResolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [{ name: 'WEBHOOK_SECRET', encryptedValue: 'workspace-ciphertext' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
},
})
)
expect(mockSetResolvedSecretTraceRegistry).toHaveBeenCalledOnce()
})
it('installs provenance before a post-resolution webhook setup failure', async () => {
const rawMessage = 'Webhook handler exposed activated-secret-value'
const rawError = new Error(rawMessage)
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
personalEncrypted: {},
workspaceEncrypted: { WEBHOOK_SECRET: 'workspace-ciphertext' },
personalDecrypted: {},
workspaceDecrypted: { WEBHOOK_SECRET: 'activated-secret-value' },
conflicts: [],
decryptionFailures: [],
})
mockResolveWebhookRecordProviderConfig.mockImplementation(
async (record, _userId, _workspaceId, options) => {
options.onResolved('WEBHOOK_SECRET', options.envVars.WEBHOOK_SECRET)
return record
}
)
mockGetProviderHandler.mockReturnValue({
formatInput: vi.fn().mockRejectedValue(rawError),
})
await expect(executeWebhookJob(payload)).rejects.toBe(rawError)
expect(mockSetResolvedSecretTraceRegistry).toHaveBeenCalledOnce()
expect(loggingSessionMockFns.mockSafeCompleteWithError).toHaveBeenCalledWith(
expect.objectContaining({
error: expect.objectContaining({ message: rawMessage }),
})
)
expect(mockExecuteWorkflowCore).not.toHaveBeenCalled()
})
it('acknowledges and skips queued webhook work after the workflow is undeployed', async () => {
executionPreprocessingMockFns.mockPreprocessExecution.mockResolvedValueOnce({
success: true,
+64 -9
View File
@@ -1,4 +1,3 @@
import { trace } from '@opentelemetry/api'
import { db } from '@sim/db'
import { account, webhook } from '@sim/db/schema'
import { createLogger, runWithRequestContext } from '@sim/logger'
@@ -15,6 +14,10 @@ import {
import type { AsyncExecutionCorrelation } from '@/lib/core/async-jobs/types'
import { createTimeoutAbortController, getTimeoutErrorMessage } from '@/lib/core/execution-limits'
import { IdempotencyService, webhookIdempotency } from '@/lib/core/idempotency'
import {
type EnvironmentResolutionSnapshot,
getEffectiveEnvironmentSnapshot,
} from '@/lib/environment/utils'
import { preprocessExecution } from '@/lib/execution/preprocessing'
import { LoggingSession } from '@/lib/logs/execution/logging-session'
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
@@ -22,7 +25,10 @@ import {
type WebhookAttachment,
WebhookAttachmentProcessor,
} from '@/lib/webhooks/attachment-processor'
import { resolveWebhookRecordProviderConfig } from '@/lib/webhooks/env-resolver'
import {
resolveWebhookRecordProviderConfig,
type WebhookEnvResolutionOptions,
} from '@/lib/webhooks/env-resolver'
import { getProviderHandler } from '@/lib/webhooks/providers'
import {
executeWorkflowCore,
@@ -40,6 +46,10 @@ import { ExecutionSnapshot } from '@/executor/execution/snapshot'
import type { ExecutionMetadata } from '@/executor/execution/types'
import type { ExecutionResult } from '@/executor/types'
import { hasExecutionResult } from '@/executor/utils/errors'
import {
createIncompleteResolvedSecretTraceRegistry,
createResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
import { safeAssign } from '@/tools/safe-assign'
import { getTrigger, isTriggerValid } from '@/triggers'
@@ -316,10 +326,32 @@ export async function resolveWebhookExecutionProviderConfig<
webhookRecord: T,
provider: string,
userId: string,
workspaceId?: string
workspaceId?: string,
options?: WebhookEnvResolutionOptions & {
onEnvironmentSnapshot?: (snapshot: EnvironmentResolutionSnapshot) => void | Promise<void>
}
): Promise<T & { providerConfig: Record<string, unknown> }> {
try {
return await resolveWebhookRecordProviderConfig(webhookRecord, userId, workspaceId)
if (!options) {
return await resolveWebhookRecordProviderConfig(webhookRecord, userId, workspaceId)
}
const { onEnvironmentSnapshot, ...resolutionOptions } = options
if (onEnvironmentSnapshot && resolutionOptions.envVars === undefined) {
const snapshot = await getEffectiveEnvironmentSnapshot(userId, workspaceId)
await onEnvironmentSnapshot(snapshot)
resolutionOptions.envVars = {
...snapshot.personalDecrypted,
...snapshot.workspaceDecrypted,
}
}
return await resolveWebhookRecordProviderConfig(
webhookRecord,
userId,
workspaceId,
resolutionOptions
)
} catch (error) {
const errorMessage = toError(error).message
throw new Error(
@@ -489,11 +521,36 @@ async function executeWebhookJobInternal(
throw new Error(`Webhook record not found: ${payload.webhookId}`)
}
const secretScope = { userId: workflowRecord.userId, workspaceId }
let resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(secretScope)
const resolvedWebhookRecord = await resolveWebhookExecutionProviderConfig(
webhookRecord,
payload.provider,
workflowRecord.userId,
workspaceId
workspaceId,
{
onEnvironmentSnapshot: async (secretEnvironment) => {
try {
resolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
personalEncrypted: secretEnvironment.personalEncrypted,
workspaceEncrypted: secretEnvironment.workspaceEncrypted,
personalDecrypted: secretEnvironment.personalDecrypted,
workspaceDecrypted: secretEnvironment.workspaceDecrypted,
decryptionFailures: secretEnvironment.decryptionFailures,
scope: secretScope,
})
} catch (error) {
logger.warn(`[${requestId}] Failed to build webhook trace secret catalog`, {
error: toError(error).message,
})
resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(secretScope)
}
loggingSession.setResolvedSecretTraceRegistry(resolvedSecretTraceRegistry)
},
onResolved: (name, value) => {
resolvedSecretTraceRegistry.recordResolved(name, value)
},
}
)
if (handler.formatInput) {
@@ -665,6 +722,7 @@ async function executeWebhookJobInternal(
snapshot,
callbacks: {},
loggingSession,
trustedInitialResolvedSecretTraceProvenance: resolvedSecretTraceRegistry.exportProvenance(),
includeFileBase64: false,
base64MaxBytes: undefined,
abortSignal: timeoutController.signal,
@@ -712,10 +770,6 @@ async function executeWebhookJobInternal(
// not a trigger.dev job fault — complete the run normally so we don't fire a false alert. Errors
// that were not finalized came from the webhook pipeline itself, so we re-throw to fault below.
if (wasExecutionFinalizedByCore(error, executionId)) {
// Record the exception on the run span so it stays visible in traces without
// marking the span as ERROR — that status is what faults the trigger.dev run.
trace.getActiveSpan()?.recordException(toError(error))
return {
success: false,
workflowId: payload.workflowId,
@@ -757,6 +811,7 @@ async function executeWebhookJobInternal(
stackTrace: errorStack,
},
traceSpans,
executionState: executionResult.executionState,
})
} catch (loggingError) {
logger.error(`[${requestId}] Failed to complete logging session`, loggingError)
@@ -219,6 +219,7 @@ export async function executeWorkflowJob(payload: WorkflowExecutionPayload) {
stackTrace: error instanceof Error ? error.stack : undefined,
},
traceSpans,
executionState: executionResult?.executionState,
})
throw error
@@ -4,11 +4,14 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { clearLargeValueCacheForTests } from '@/lib/execution/payloads/cache'
import { isLargeArrayManifest } from '@/lib/execution/payloads/large-array-manifest-metadata'
import { projectTraceSpansForSecrets } from '@/lib/logs/execution/trace-secret-projection'
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
import { BlockType } from '@/executor/constants'
import type { DAGNode } from '@/executor/dag/builder'
import { BlockExecutor } from '@/executor/execution/block-executor'
import { ExecutionState } from '@/executor/execution/state'
import type { BlockHandler, ExecutionContext } from '@/executor/types'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import { VariableResolver } from '@/executor/variables/resolver'
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
@@ -436,6 +439,108 @@ describe('BlockExecutor', () => {
expect(output?.error).toBeTruthy()
expect(output).not.toEqual({ content: '' })
})
it('projects a resolved secret out of Function syntax-error TraceSpans only', async () => {
const secret = 'function-secret-literal-7f3a91'
const block = createBlock()
block.metadata.name = 'Function 1'
block.config.params = {
code: 'return {{OPENAI_API_KEY}}',
language: 'javascript',
}
const workflow: SerializedWorkflow = {
version: '1',
blocks: [block],
connections: [],
loops: {},
parallels: {},
}
const state = new ExecutionState()
const registry = new ResolvedSecretTraceRegistry([
{
name: 'OPENAI_API_KEY',
plaintext: secret,
encryptedValue: 'encrypted-openai-api-key',
},
])
const resolver = new VariableResolver(workflow, {}, state)
const syntaxError = `Syntax Error: Line 1: \`return ${secret}\` - Invalid or unexpected token`
const handler: BlockHandler = {
canHandle: () => true,
execute: async (_ctx, _block, inputs) => {
expect(inputs.code).toBe(`return ${secret}`)
throw new Error(syntaxError)
},
}
const executor = new BlockExecutor(
[handler],
resolver,
{
workspaceId: 'workspace-1',
executionId: 'execution-1',
userId: 'user-1',
metadata: {
requestId: 'request-1',
executionId: 'execution-1',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
userId: 'user-1',
triggerType: 'manual',
useDraftState: false,
startTime: new Date().toISOString(),
},
},
state
)
const ctx = createContext(state)
ctx.environmentVariables = { OPENAI_API_KEY: secret }
ctx.resolvedSecretTraceRegistry = registry
await expect(executor.execute(ctx, createNode(block), block)).rejects.toThrow(
`Function 1: ${syntaxError}`
)
expect(registry.getActiveMatches()).toEqual([
{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' },
])
expect(state.getBlockOutput(block.id)).toEqual({ error: syntaxError })
expect(ctx.blockLogs[0]).toMatchObject({
input: { code: `return ${secret}` },
output: { error: syntaxError },
error: syntaxError,
})
const rawLogs = structuredClone(ctx.blockLogs)
const { traceSpans: rawTraceSpans } = buildTraceSpans({
success: false,
output: { error: syntaxError },
error: `Function 1: ${syntaxError}`,
logs: ctx.blockLogs,
})
const rawTraceSnapshot = structuredClone(rawTraceSpans)
const projectedTraceSpans = await projectTraceSpansForSecrets(rawTraceSpans, {
registry,
store: {
workspaceId: 'workspace-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
userId: 'user-1',
},
})
expect(ctx.blockLogs).toEqual(rawLogs)
expect(rawTraceSpans).toEqual(rawTraceSnapshot)
expect(projectedTraceSpans).toEqual([
expect.objectContaining({
name: 'Function 1',
input: expect.objectContaining({ code: 'return {{OPENAI_API_KEY}}' }),
output: {
error: 'Syntax Error: Line 1: `return {{OPENAI_API_KEY}}` - Invalid or unexpected token',
},
}),
])
expect(JSON.stringify(projectedTraceSpans)).not.toContain(secret)
})
})
describe('BlockExecutor streaming pump', () => {
+1
View File
@@ -180,6 +180,7 @@ export class ExecutionEngine {
output: this.finalOutput,
error: errorMessage,
logs: this.context.blockLogs,
executionState: this.getSerializableExecutionState(),
metadata: this.context.metadata,
}
+1
View File
@@ -438,6 +438,7 @@ export class DAGExecutor {
},
startRunMetadata: this.contextExtensions.startRunMetadata,
environmentVariables: this.environmentVariables,
resolvedSecretTraceRegistry: this.contextExtensions.resolvedSecretTraceRegistry,
workflowVariables: this.workflowVariables,
decisions: {
router: snapshotState?.decisions?.router
@@ -6,6 +6,7 @@ import type { DAG, DAGNode } from '@/executor/dag/builder'
import { EdgeManager } from '@/executor/execution/edge-manager'
import { serializePauseSnapshot } from '@/executor/execution/snapshot-serializer'
import type { ExecutionContext } from '@/executor/types'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
function createContext(overrides: Partial<ExecutionContext> = {}): ExecutionContext {
return {
@@ -38,6 +39,45 @@ function createContext(overrides: Partial<ExecutionContext> = {}): ExecutionCont
}
describe('serializePauseSnapshot', () => {
it('persists encrypted resolved-secret provenance and the source execution id', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'raw-secret', encryptedValue: 'ciphertext' },
])
registry.recordResolved('TOKEN', 'raw-secret')
const context = createContext({ resolvedSecretTraceRegistry: registry })
const snapshot = serializePauseSnapshot(context, ['next-block'])
const serialized = JSON.parse(snapshot.snapshot)
expect(serialized.state.sourceExecutionId).toBe('execution-1')
expect(serialized.state.resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
})
expect(snapshot.snapshot).not.toContain('raw-secret')
})
it('persists a complete zero-entry provenance state for a fresh execution', () => {
const registry = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
const snapshot = serializePauseSnapshot(
createContext({ resolvedSecretTraceRegistry: registry }),
['next-block']
)
const serialized = JSON.parse(snapshot.snapshot)
expect(serialized.state.resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: true,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('serializes batched parallel accumulated outputs for cross-process resume', () => {
const context = createContext({
parallelExecutions: new Map([
@@ -224,6 +224,19 @@ export function serializePauseSnapshot(
dagIncomingEdges,
deactivatedEdges: edgeManager?.getDeactivatedEdges(),
nodesWithActivatedEdge: edgeManager?.getNodesWithActivatedEdge(),
sourceExecutionId: context.executionId,
trustedLargeValueAccess: {
executionIds: Array.from(
new Set(
[context.executionId, ...(context.largeValueExecutionIds ?? [])].filter(
(id): id is string => Boolean(id)
)
)
),
largeValueKeys: Array.from(new Set(context.largeValueKeys ?? [])),
fileKeys: Array.from(new Set(context.fileKeys ?? [])),
},
resolvedSecretTraceProvenance: context.resolvedSecretTraceRegistry?.exportProvenance(),
}
assertSnapshotValueIsCompact(context.workflowVariables, 'workflow variables')
+28 -11
View File
@@ -10,6 +10,10 @@ import type {
StartBlockRunMetadata,
StreamingExecution,
} from '@/executor/types'
import type {
ResolvedSecretTraceProvenanceV1,
ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
import type { RunFromBlockContext } from '@/executor/utils/run-from-block'
import type { SubflowType } from '@/stores/workflows/workflow/types'
@@ -88,6 +92,16 @@ export interface SerializableExecutionState {
deactivatedEdges?: string[]
nodesWithActivatedEdge?: string[]
completedPauseContexts?: string[]
/** Server execution that produced this state; callers must still verify it against storage. */
sourceExecutionId?: string
/** Server-only closure authorizing offloaded values carried by trusted restored state. */
trustedLargeValueAccess?: {
executionIds: string[]
largeValueKeys: string[]
fileKeys: string[]
}
/** Encrypted-only provenance for Secrets-tab values resolved during this execution. */
resolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceV1
}
/**
@@ -144,6 +158,17 @@ export interface ChildWorkflowContext {
depth: number
}
export interface BlockCompletionCallbackData {
input?: unknown
output: NormalizedBlockOutput
executionTime: number
startedAt: string
executionOrder: number
endedAt: string
/** Per-invocation unique ID linking this workflow block execution to its child block events. */
childWorkflowInstanceId?: string
}
export interface ExecutionCallbacks {
onStream?: (streamingExec: StreamingExecution) => Promise<void>
onBlockStart?: (
@@ -158,7 +183,7 @@ export interface ExecutionCallbacks {
blockId: string,
blockName: string,
blockType: string,
output: any,
output: BlockCompletionCallbackData,
iterationContext?: IterationContext,
childWorkflowContext?: ChildWorkflowContext
) => Promise<void>
@@ -214,6 +239,7 @@ export interface ContextExtensions {
}>
dagIncomingEdges?: Record<string, string[]>
snapshotState?: SerializableExecutionState
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
metadata?: ExecutionMetadata
/**
* Trusted run metadata injected into the Start block output when its
@@ -247,16 +273,7 @@ export interface ContextExtensions {
blockId: string,
blockName: string,
blockType: string,
output: {
input?: any
output: NormalizedBlockOutput
executionTime: number
startedAt: string
executionOrder: number
endedAt: string
/** Per-invocation unique ID linking this workflow block execution to its child block events. */
childWorkflowInstanceId?: string
},
output: BlockCompletionCallbackData,
iterationContext?: IterationContext,
childWorkflowContext?: ChildWorkflowContext
) => Promise<void>
@@ -1182,48 +1182,55 @@ export class AgentBlockHandler implements BlockHandler {
const { blockData, blockNameMapping } = collectBlockData(ctx)
const response = await executeProviderRequest(providerId, {
model,
systemPrompt: 'systemPrompt' in providerRequest ? providerRequest.systemPrompt : undefined,
context: 'context' in providerRequest ? providerRequest.context : undefined,
tools: providerRequest.tools,
temperature: providerRequest.temperature,
maxTokens: providerRequest.maxTokens,
apiKey: finalApiKey,
azureEndpoint: providerRequest.azureEndpoint,
azureApiVersion: providerRequest.azureApiVersion,
vertexProject: providerRequest.vertexProject,
vertexLocation: providerRequest.vertexLocation,
bedrockAccessKeyId: providerRequest.bedrockAccessKeyId,
bedrockSecretKey: providerRequest.bedrockSecretKey,
bedrockRegion: providerRequest.bedrockRegion,
responseFormat: providerRequest.responseFormat,
workflowId: providerRequest.workflowId,
workspaceId: ctx.workspaceId,
userId: ctx.userId,
stream: providerRequest.stream,
messages: 'messages' in providerRequest ? providerRequest.messages : undefined,
environmentVariables: normalizeStringRecord(ctx.environmentVariables),
workflowVariables: normalizeWorkflowVariables(ctx.workflowVariables),
blockData,
blockNameMapping,
isDeployedContext: ctx.isDeployedContext,
callChain: ctx.callChain,
billingAttribution: ctx.metadata.billingAttribution,
// Reaches tool `_context` via `prepareToolExecution`, so a tool that starts
// its own child execution (a custom block) correlates and cancels against
// this real run instead of minting a phantom id.
executionId: ctx.executionId,
reasoningEffort: providerRequest.reasoningEffort,
verbosity: providerRequest.verbosity,
thinkingLevel: providerRequest.thinkingLevel,
promptCaching: providerRequest.promptCaching,
// Stable per-block identity; providers use it to route cache lookups.
blockId: block.id,
previousInteractionId: providerRequest.previousInteractionId,
agentEvents: providerRequest.agentEvents,
abortSignal: ctx.abortSignal,
})
const response = await executeProviderRequest(
providerId,
{
model,
systemPrompt:
'systemPrompt' in providerRequest ? providerRequest.systemPrompt : undefined,
context: 'context' in providerRequest ? providerRequest.context : undefined,
tools: providerRequest.tools,
temperature: providerRequest.temperature,
maxTokens: providerRequest.maxTokens,
apiKey: finalApiKey,
azureEndpoint: providerRequest.azureEndpoint,
azureApiVersion: providerRequest.azureApiVersion,
vertexProject: providerRequest.vertexProject,
vertexLocation: providerRequest.vertexLocation,
bedrockAccessKeyId: providerRequest.bedrockAccessKeyId,
bedrockSecretKey: providerRequest.bedrockSecretKey,
bedrockRegion: providerRequest.bedrockRegion,
responseFormat: providerRequest.responseFormat,
workflowId: providerRequest.workflowId,
workspaceId: ctx.workspaceId,
userId: ctx.userId,
stream: providerRequest.stream,
messages: 'messages' in providerRequest ? providerRequest.messages : undefined,
environmentVariables: normalizeStringRecord(ctx.environmentVariables),
workflowVariables: normalizeWorkflowVariables(ctx.workflowVariables),
blockData,
blockNameMapping,
isDeployedContext: ctx.isDeployedContext,
callChain: ctx.callChain,
billingAttribution: ctx.metadata.billingAttribution,
// Reaches tool `_context` via `prepareToolExecution`, so a tool that starts
// its own child execution (a custom block) correlates and cancels against
// this real run instead of minting a phantom id.
executionId: ctx.executionId,
reasoningEffort: providerRequest.reasoningEffort,
verbosity: providerRequest.verbosity,
thinkingLevel: providerRequest.thinkingLevel,
promptCaching: providerRequest.promptCaching,
// Stable per-block identity; providers use it to route cache lookups.
blockId: block.id,
previousInteractionId: providerRequest.previousInteractionId,
agentEvents: providerRequest.agentEvents,
abortSignal: ctx.abortSignal,
},
{
resolvedSecretTraceRegistry: ctx.resolvedSecretTraceRegistry,
}
)
return this.processProviderResponse(response, block, responseFormat)
} catch (error) {
@@ -5,6 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { BlockType } from '@/executor/constants'
import { MothershipBlockHandler } from '@/executor/handlers/mothership/mothership-handler'
import type { ExecutionContext, StreamingExecution } from '@/executor/types'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import type { SerializedBlock } from '@/serializer/types'
const BILLING_ATTRIBUTION = {
@@ -20,6 +21,13 @@ const BILLING_ATTRIBUTION = {
payerSubscription: null,
} as const
const PRIVATE_PROVENANCE_TYPE = 'resolved-secret-provenance-v1'
const PRIVATE_PROVENANCE = {
version: 1,
complete: true,
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
}
const {
mockBuildAuthHeaders,
mockBuildAPIUrl,
@@ -96,6 +104,19 @@ async function readStreamText(stream: ReadableStream): Promise<string> {
return text
}
function createTraceRegistryMock(): ResolvedSecretTraceRegistry & {
importProvenance: ReturnType<typeof vi.fn>
markIncomplete: ReturnType<typeof vi.fn>
} {
return {
importProvenance: vi.fn().mockResolvedValue(true),
markIncomplete: vi.fn(),
} as unknown as ResolvedSecretTraceRegistry & {
importProvenance: ReturnType<typeof vi.fn>
markIncomplete: ReturnType<typeof vi.fn>
}
}
describe('MothershipBlockHandler', () => {
let handler: MothershipBlockHandler
let block: SerializedBlock
@@ -152,7 +173,7 @@ describe('MothershipBlockHandler', () => {
resetEnvMock()
})
function createNdjsonResponse(events: unknown[]): Response {
function createNdjsonResponse(events: unknown[], headers: Record<string, string> = {}): Response {
const encoder = new TextEncoder()
return new Response(
new ReadableStream({
@@ -165,11 +186,146 @@ describe('MothershipBlockHandler', () => {
}),
{
status: 200,
headers: { 'Content-Type': 'application/x-ndjson; charset=utf-8' },
headers: { 'Content-Type': 'application/x-ndjson; charset=utf-8', ...headers },
}
)
}
it('imports marker-gated JSON provenance without exposing it in block output', async () => {
const registry = createTraceRegistryMock()
context.resolvedSecretTraceRegistry = registry
mockGenerateId
.mockReturnValueOnce('chat-uuid')
.mockReturnValueOnce('message-uuid')
.mockReturnValueOnce('request-uuid')
fetchMock.mockResolvedValue(
new Response(
JSON.stringify({
content: 'raw secret remains functional',
toolCalls: [],
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
}),
{
status: 200,
headers: {
'Content-Type': 'application/json',
'x-sim-private-tool-metadata': PRIVATE_PROVENANCE_TYPE,
},
}
)
)
const result = await handler.execute(context, block, { prompt: 'Hello' })
const [, options] = fetchMock.mock.calls[0] as [string, RequestInit]
expect(options.headers).toMatchObject({
'x-sim-request-private-tool-metadata': PRIVATE_PROVENANCE_TYPE,
})
expect(registry.importProvenance).toHaveBeenCalledWith(PRIVATE_PROVENANCE, {
trusted: true,
})
expect(registry.markIncomplete).not.toHaveBeenCalled()
expect(result).toMatchObject({ content: 'raw secret remains functional' })
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
expect(JSON.stringify(result)).not.toContain('encrypted-secret')
})
it('rejects unmarked provenance while keeping private metadata out of block output', async () => {
const registry = createTraceRegistryMock()
context.resolvedSecretTraceRegistry = registry
mockGenerateId
.mockReturnValueOnce('chat-uuid')
.mockReturnValueOnce('message-uuid')
.mockReturnValueOnce('request-uuid')
fetchMock.mockResolvedValue(
new Response(
JSON.stringify({
content: 'unchanged output',
toolCalls: [],
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
}),
{ status: 200, headers: { 'Content-Type': 'application/json' } }
)
)
const result = await handler.execute(context, block, { prompt: 'Hello' })
expect(registry.importProvenance).not.toHaveBeenCalled()
expect(registry.markIncomplete).toHaveBeenCalledOnce()
expect(result).toMatchObject({ content: 'unchanged output' })
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
expect(JSON.stringify(result)).not.toContain('encrypted-secret')
})
it('imports provenance from a terminal NDJSON error without forcing structural fallback', async () => {
const registry = createTraceRegistryMock()
context.resolvedSecretTraceRegistry = registry
mockGenerateId
.mockReturnValueOnce('chat-uuid')
.mockReturnValueOnce('message-uuid')
.mockReturnValueOnce('request-uuid')
fetchMock.mockResolvedValue(
createNdjsonResponse(
[
{
type: 'error',
error: 'secret-backed failure',
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
},
],
{ 'x-sim-private-tool-metadata': PRIVATE_PROVENANCE_TYPE }
)
)
await expect(handler.execute(context, block, { prompt: 'Hello' })).rejects.toThrow(
'Sim execution failed: secret-backed failure'
)
expect(registry.importProvenance).toHaveBeenCalledWith(PRIVATE_PROVENANCE, {
trusted: true,
})
expect(registry.markIncomplete).not.toHaveBeenCalled()
})
it('imports final provenance for selected-output streaming without adding it to output', async () => {
const registry = createTraceRegistryMock()
context.resolvedSecretTraceRegistry = registry
context.stream = true
context.selectedOutputs = [`${block.id}_content`]
mockGenerateId
.mockReturnValueOnce('chat-uuid')
.mockReturnValueOnce('message-uuid')
.mockReturnValueOnce('request-uuid')
fetchMock.mockResolvedValue(
createNdjsonResponse(
[
{ type: 'chunk', content: 'unchanged' },
{
type: 'final',
data: {
content: 'unchanged',
toolCalls: [],
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
},
},
],
{ 'x-sim-private-tool-metadata': PRIVATE_PROVENANCE_TYPE }
)
)
const result = (await handler.execute(context, block, {
prompt: 'Hello',
})) as StreamingExecution
await expect(readStreamText(result.stream)).resolves.toBe('unchanged')
expect(registry.importProvenance).toHaveBeenCalledWith(PRIVATE_PROVENANCE, {
trusted: true,
})
expect(registry.markIncomplete).not.toHaveBeenCalled()
expect(JSON.stringify(result.execution.output)).not.toContain('__resolvedSecretTraceProvenance')
expect(JSON.stringify(result.execution.output)).not.toContain('encrypted-secret')
})
it('forwards workflow and execution metadata with generated UUID ids', async () => {
mockGenerateId.mockReturnValueOnce('chat-uuid')
mockGenerateId.mockReturnValueOnce('message-uuid')
@@ -8,6 +8,12 @@ import {
import { env } from '@/lib/core/config/env'
import { isExecutionCancelled, isRedisCancellationEnabled } from '@/lib/execution/cancellation'
import { readUserFileContent } from '@/lib/execution/payloads/materialization.server'
import {
PRIVATE_TOOL_METADATA_REQUEST_HEADER,
RESOLVED_SECRET_PROVENANCE_FIELD,
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
responseHasPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
import {
createFileContentFromBase64,
type MessageContent,
@@ -24,6 +30,7 @@ import type {
StreamingExecution,
} from '@/executor/types'
import { buildAPIUrl, buildAuthHeaders, extractAPIErrorMessage } from '@/executor/utils/http'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import type { SerializedBlock } from '@/serializer/types'
const logger = createLogger('MothershipBlockHandler')
@@ -43,13 +50,31 @@ type MothershipExecuteResult = {
tokens?: Record<string, unknown>
toolCalls?: Array<Record<string, unknown>>
cost?: unknown
}
} & Partial<Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>>
type MothershipExecuteStreamEvent =
| { type: 'heartbeat'; timestamp?: string }
| { type: 'chunk'; content?: string }
| { type: 'final'; data: MothershipExecuteResult }
| { type: 'error'; error?: string }
| ({ type: 'error'; error?: string } & Partial<
Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>
>)
async function consumeMothershipProvenance(
payload: Partial<Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>>,
response: Response,
registry?: ResolvedSecretTraceRegistry
): Promise<boolean> {
if (!registry) return true
if (
!responseHasPrivateToolMetadata(response.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1) ||
!Object.hasOwn(payload, RESOLVED_SECRET_PROVENANCE_FIELD)
) {
registry.markIncomplete()
return false
}
return registry.importProvenance(payload[RESOLVED_SECRET_PROVENANCE_FIELD], { trusted: true })
}
function parseMothershipExecuteStreamLine(line: string): MothershipExecuteStreamEvent | undefined {
const trimmed = line.trim()
@@ -100,10 +125,15 @@ function isContentSelectedForStreaming(ctx: ExecutionContext, block: SerializedB
)
}
async function readMothershipExecuteResponse(response: Response): Promise<MothershipExecuteResult> {
async function readMothershipExecuteResponse(
response: Response,
registry?: ResolvedSecretTraceRegistry
): Promise<MothershipExecuteResult> {
const contentType = response.headers.get('content-type') || ''
if (!contentType.includes('application/x-ndjson')) {
return response.json()
const result = (await response.json()) as MothershipExecuteResult
await consumeMothershipProvenance(result, response, registry)
return result
}
if (!response.body) {
@@ -114,8 +144,9 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
const decoder = new TextDecoder()
let buffer = ''
let finalResult: MothershipExecuteResult | undefined
let receivedTerminalProvenance = false
const processLine = (line: string) => {
const processLine = async (line: string): Promise<void> => {
const event = parseMothershipExecuteStreamLine(line)
if (!event) return
@@ -124,10 +155,12 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
}
if (event.type === 'error') {
receivedTerminalProvenance = await consumeMothershipProvenance(event, response, registry)
throw new Error(`Sim execution failed: ${event.error || 'Unknown error'}`)
}
if (event.type === 'final') {
await consumeMothershipProvenance(event.data, response, registry)
finalResult = event.data
return
}
@@ -144,12 +177,12 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
const lines = buffer.split('\n')
buffer = lines.pop() ?? ''
for (const line of lines) {
processLine(line)
await processLine(line)
}
}
buffer += decoder.decode()
processLine(buffer)
await processLine(buffer)
if (!finalResult) {
throw new Error('Sim execution stream ended without a final result')
@@ -157,6 +190,7 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
return finalResult
} finally {
if (!finalResult && !receivedTerminalProvenance) registry?.markIncomplete()
reader.releaseLock()
}
}
@@ -168,6 +202,7 @@ function createMothershipStreamingExecution(
options: {
onCancel?: (reason?: unknown) => void
onDone?: () => void
registry?: ResolvedSecretTraceRegistry
} = {}
): StreamingExecution {
if (!response.body) {
@@ -191,8 +226,9 @@ function createMothershipStreamingExecution(
const encoder = new TextEncoder()
let buffer = ''
let sawFinal = false
let receivedTerminalProvenance = false
const processLine = (line: string) => {
const processLine = async (line: string): Promise<void> => {
const event = parseMothershipExecuteStreamLine(line)
if (!event) return
@@ -208,10 +244,16 @@ function createMothershipStreamingExecution(
}
if (event.type === 'error') {
receivedTerminalProvenance = await consumeMothershipProvenance(
event,
response,
options.registry
)
throw new Error(`Sim execution failed: ${event.error || 'Unknown error'}`)
}
if (event.type === 'final') {
await consumeMothershipProvenance(event.data, response, options.registry)
sawFinal = true
Object.assign(output, formatMothershipBlockOutput(event.data, fallbackChatId))
return
@@ -230,12 +272,12 @@ function createMothershipStreamingExecution(
const lines = buffer.split('\n')
buffer = lines.pop() ?? ''
for (const line of lines) {
processLine(line)
await processLine(line)
}
}
buffer += decoder.decode()
processLine(buffer)
await processLine(buffer)
if (!sawFinal) {
throw new Error('Sim execution stream ended without a final result')
@@ -249,6 +291,7 @@ function createMothershipStreamingExecution(
controller.error(error)
}
} finally {
if (!sawFinal && !receivedTerminalProvenance) options.registry?.markIncomplete()
cleanup()
reader?.releaseLock()
}
@@ -382,6 +425,9 @@ export class MothershipBlockHandler implements BlockHandler {
const headers = await buildAuthHeaders(ctx.userId)
headers.Accept = 'application/x-ndjson'
headers[MOTHERSHIP_EXECUTE_STREAM_HEADER] = MOTHERSHIP_EXECUTE_STREAM_VALUE
if (ctx.resolvedSecretTraceRegistry) {
headers[PRIVATE_TOOL_METADATA_REQUEST_HEADER] = RESOLVED_SECRET_PROVENANCE_METADATA_V1
}
if (!ctx.metadata.billingAttribution) {
throw new Error('Billing attribution is required for Mothership execution')
}
@@ -474,6 +520,14 @@ export class MothershipBlockHandler implements BlockHandler {
})
if (!response.ok) {
if (ctx.resolvedSecretTraceRegistry) {
try {
const payload = (await response.clone().json()) as MothershipExecuteResult
await consumeMothershipProvenance(payload, response, ctx.resolvedSecretTraceRegistry)
} catch {
ctx.resolvedSecretTraceRegistry.markIncomplete()
}
}
const errorMsg = await extractAPIErrorMessage(response)
throw new Error(`Sim execution failed: ${errorMsg}`)
}
@@ -486,12 +540,13 @@ export class MothershipBlockHandler implements BlockHandler {
}
},
onDone: cleanupAbortListeners,
registry: ctx.resolvedSecretTraceRegistry,
})
cleanupImmediately = false
return streamingExecution
}
const result = await readMothershipExecuteResponse(response)
const result = await readMothershipExecuteResponse(response, ctx.resolvedSecretTraceRegistry)
return formatMothershipBlockOutput(result, chatId)
} finally {
if (cleanupImmediately) {
@@ -18,6 +18,7 @@ import {
buildCustomBlockExecutionContext,
runCustomBlockTool,
} from '@/executor/handlers/workflow/custom-block-tool-runner'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
describe('buildCustomBlockExecutionContext', () => {
it('carries consumer identity, inherits the call chain, and is fully scaffolded', () => {
@@ -145,3 +146,16 @@ describe('buildCustomBlockExecutionContext cancellation', () => {
expect(buildCustomBlockExecutionContext({ workspaceId: 'ws-1' }).abortSignal).toBeUndefined()
})
})
describe('buildCustomBlockExecutionContext secret provenance', () => {
it('carries the server-only parent registry without putting it in model parameters', () => {
const registry = new ResolvedSecretTraceRegistry()
const ctx = buildCustomBlockExecutionContext(
{ workspaceId: 'ws-1' },
{ resolvedSecretTraceRegistry: registry }
)
expect(ctx.resolvedSecretTraceRegistry).toBe(registry)
})
})
@@ -4,6 +4,7 @@ import { generateId } from '@sim/utils/id'
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
import { WorkflowBlockHandler } from '@/executor/handlers/workflow/workflow-handler'
import type { ExecutionContext } from '@/executor/types'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import type { SerializedBlock } from '@/serializer/types'
import type { ToolResponse } from '@/tools/types'
@@ -49,7 +50,10 @@ interface CustomBlockToolParams {
*/
export function buildCustomBlockExecutionContext(
context: CustomBlockExecutorContext,
options: { abortSignal?: AbortSignal } = {}
options: {
abortSignal?: AbortSignal
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
} = {}
): ExecutionContext {
// Prefer the invoking agent run's ids so correlation and cancellation both
// point at a real execution; fall back only when a caller could not supply them.
@@ -66,6 +70,7 @@ export function buildCustomBlockExecutionContext(
// Without this the child's cancellation bridge has nothing to abort on:
// the agent tool loop owns the only signal reaching this path.
abortSignal: options.abortSignal,
resolvedSecretTraceRegistry: options.resolvedSecretTraceRegistry,
environmentVariables: {},
blockStates: new Map(),
executedBlocks: new Set(),
@@ -102,7 +107,10 @@ export function buildCustomBlockExecutionContext(
*/
export async function runCustomBlockTool(
params: CustomBlockToolParams,
options: { abortSignal?: AbortSignal } = {}
options: {
abortSignal?: AbortSignal
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
} = {}
): Promise<ToolResponse> {
if (!params.blockType) {
return { success: false, output: {}, error: 'Missing custom block type' }
@@ -110,6 +118,7 @@ export async function runCustomBlockTool(
const ctx = buildCustomBlockExecutionContext(params._context ?? {}, {
abortSignal: options.abortSignal,
resolvedSecretTraceRegistry: options.resolvedSecretTraceRegistry,
})
const block: SerializedBlock = {
id: generateId(),
@@ -1,4 +1,4 @@
import { environmentUtilsMockFns, resetEnvironmentUtilsMock } from '@sim/testing'
import { encryptionMockFns, environmentUtilsMockFns, resetEnvironmentUtilsMock } from '@sim/testing'
import { afterAll, beforeAll, beforeEach, describe, expect, it, type Mock, vi } from 'vitest'
import { getBlock } from '@/blocks/registry'
import { BlockType } from '@/executor/constants'
@@ -9,6 +9,10 @@ import {
WorkflowBlockHandler,
} from '@/executor/handlers/workflow/workflow-handler'
import type { ExecutionContext } from '@/executor/types'
import {
ANONYMOUS_SECRET_TRACE_REPLACEMENT,
ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
import type { SerializedBlock } from '@/serializer/types'
const {
@@ -24,6 +28,8 @@ const {
mockSafeComplete,
mockSafeCompleteWithError,
mockSafeCompleteWithCancellation,
mockSetResolvedSecretTraceRegistry,
mockSetTraceLargeValueAccess,
mockDispose,
executorOptions,
loggingSessionArgs,
@@ -40,6 +46,8 @@ const {
mockSafeComplete: vi.fn(),
mockSafeCompleteWithError: vi.fn(),
mockSafeCompleteWithCancellation: vi.fn(),
mockSetResolvedSecretTraceRegistry: vi.fn(),
mockSetTraceLargeValueAccess: vi.fn(),
mockDispose: vi.fn(),
executorOptions: [] as Array<Record<string, any>>,
loggingSessionArgs: [] as Array<any[]>,
@@ -54,6 +62,8 @@ vi.mock('@/lib/logs/execution/logging-session', () => ({
safeComplete = mockSafeComplete
safeCompleteWithError = mockSafeCompleteWithError
safeCompleteWithCancellation = mockSafeCompleteWithCancellation
setResolvedSecretTraceRegistry = mockSetResolvedSecretTraceRegistry
setTraceLargeValueAccess = mockSetTraceLargeValueAccess
onBlockStart = vi.fn()
onBlockComplete = vi.fn()
},
@@ -63,6 +73,11 @@ vi.mock('@/lib/logs/execution/trace-spans/trace-spans', () => ({
buildTraceSpans: mockBuildTraceSpans,
}))
vi.mock('@/lib/core/security/encryption', () => ({
decryptSecret: encryptionMockFns.mockDecryptSecret,
encryptSecret: encryptionMockFns.mockEncryptSecret,
}))
vi.mock('@/lib/workflows/custom-blocks/child-execution', () => ({
admitCustomBlockChildExecution: mockAdmitCustomBlockChildExecution,
trackChildRun: mockTrackChildRun,
@@ -1147,6 +1162,54 @@ describe('WorkflowBlockHandler', () => {
expect(ctx.largeValueExecutionIds).toContain('grandchild-execution-id')
})
it('imports only publisher secret provenance that crosses the curated output boundary', async () => {
encryptionMockFns.mockDecryptSecret.mockResolvedValueOnce({
decrypted: 'publisher-secret',
})
mockGetPersonalAndWorkspaceEnv.mockResolvedValueOnce({
personalDecrypted: { SECRET: 'publisher-secret', UNUSED: 'unused-secret' },
workspaceDecrypted: {},
personalEncrypted: {
SECRET: 'publisher-ciphertext',
UNUSED: 'unused-ciphertext',
},
workspaceEncrypted: {},
decryptionFailures: [],
})
mockExecutorExecute.mockImplementationOnce(async () => {
const childRegistry = executorOptions.at(-1)?.contextExtensions
.resolvedSecretTraceRegistry as ResolvedSecretTraceRegistry
childRegistry.recordResolved('SECRET', 'publisher-secret')
childRegistry.recordResolved('UNUSED', 'unused-secret')
return {
success: true,
output: {},
logs: [
{
blockId: 'b1',
success: true,
output: { content: 'value=publisher-secret' },
},
],
}
})
const parentRegistry = new ResolvedSecretTraceRegistry()
const result = await handler.execute(
customBlockContext({ resolvedSecretTraceRegistry: parentRegistry }),
customBlock(),
{}
)
expect(result).toMatchObject({ answer: 'value=publisher-secret', success: true })
expect(parentRegistry.getActiveMatches()).toEqual([
{
plaintext: 'publisher-secret',
replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT,
},
])
expect(mockSetResolvedSecretTraceRegistry).toHaveBeenCalledTimes(1)
})
it('does not duplicate ids across repeated invocations', async () => {
const ctx = customBlockContext()
await handler.execute(ctx, customBlock(), {})
@@ -1179,9 +1242,19 @@ describe('WorkflowBlockHandler', () => {
})
it('completes the child session and disposes the cancellation bridge', async () => {
const executionState = {
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
}
mockExecutorExecute.mockResolvedValue({
success: true,
output: { data: 'ok' },
executionState,
})
await handler.execute(customBlockContext(), customBlock(), {})
expect(mockSafeComplete).toHaveBeenCalledTimes(1)
expect(mockSafeComplete).toHaveBeenCalledWith(expect.objectContaining({ executionState }))
expect(mockSafeCompleteWithError).not.toHaveBeenCalled()
expect(mockDispose).toHaveBeenCalledTimes(1)
})
@@ -1189,15 +1262,22 @@ describe('WorkflowBlockHandler', () => {
it('records a cancelled child through the cancellation path', async () => {
// Production shape: the engine reports cancellation as `success: false`
// plus `status: 'cancelled'` on the ExecutionResult (never on metadata).
const executionState = {
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
}
mockExecutorExecute.mockResolvedValue({
success: false,
output: {},
status: 'cancelled',
executionState,
})
await handler.execute(customBlockContext(), customBlock(), {}).catch(() => {})
expect(mockSafeCompleteWithCancellation).toHaveBeenCalledTimes(1)
expect(mockSafeCompleteWithCancellation).toHaveBeenCalledWith(
expect.objectContaining({ executionState })
)
expect(mockSafeComplete).not.toHaveBeenCalled()
// Already finalized as cancelled — must not be re-completed as an error.
expect(mockSafeCompleteWithError).not.toHaveBeenCalled()
@@ -1407,12 +1487,14 @@ describe('WorkflowBlockHandler', () => {
})
it('leaves regular workflow blocks entirely alone', async () => {
const registry = new ResolvedSecretTraceRegistry()
const ctx = {
...mockContext,
workspaceId: 'workspace-1',
executionId: 'parent-execution-id',
onBlockStart: vi.fn(),
onStream: vi.fn(),
resolvedSecretTraceRegistry: registry,
} as unknown as ExecutionContext
mockFetch.mockResolvedValue({
ok: true,
@@ -1431,6 +1513,7 @@ describe('WorkflowBlockHandler', () => {
expect(loggingSessionArgs).toHaveLength(0)
const extensions = executorOptions[0].contextExtensions
expect(extensions.executionId).toBe('parent-execution-id')
expect(extensions.resolvedSecretTraceRegistry).toBe(registry)
expect(extensions.onStream).toBe(ctx.onStream)
expect(extensions.childWorkflowContext).toBeDefined()
})
@@ -45,6 +45,7 @@ import { buildAPIUrl, buildAuthHeaders } from '@/executor/utils/http'
import { getIterationContext } from '@/executor/utils/iteration-context'
import { parseJSON } from '@/executor/utils/json'
import { lazyCleanupInputMapping } from '@/executor/utils/lazy-cleanup'
import { createResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import { isRunMetadataEnabled, resolveExecutorStartBlock } from '@/executor/utils/start-block'
import { Serializer } from '@/serializer'
import type { SerializedBlock } from '@/serializer/types'
@@ -269,6 +270,7 @@ export class WorkflowBlockHandler implements BlockHandler {
/** Set for custom blocks only: the child's own execution id / log row. */
let childExecutionId: string | undefined
let childSession: LoggingSession | undefined
let childResolvedSecretTraceRegistry = ctx.resolvedSecretTraceRegistry
let childSessionStarted = false
/** Set once the child's session reached a terminal state, so the catch doesn't re-complete it. */
let childSessionFinalized = false
@@ -431,6 +433,24 @@ export class WorkflowBlockHandler implements BlockHandler {
...ownerEnv.personalEncrypted,
...ownerEnv.workspaceEncrypted,
}
childResolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
personalEncrypted: ownerEnv.personalEncrypted,
workspaceEncrypted: ownerEnv.workspaceEncrypted,
personalDecrypted: ownerEnv.personalDecrypted,
workspaceDecrypted: ownerEnv.workspaceDecrypted,
decryptionFailures: ownerEnv.decryptionFailures,
scope: { userId: loadUserId, workspaceId: sourceWorkspaceId },
})
if (ctx.resolvedSecretTraceRegistry) {
const crossingProvenance = ctx.resolvedSecretTraceRegistry.exportProvenanceForValue(
childWorkflowInput,
{ anonymous: true }
)
await childResolvedSecretTraceRegistry.importProvenance(crossingProvenance, {
trusted: true,
anonymous: true,
})
}
// Custom-block children authenticate internal tool calls as the source
// owner in the source workspace, so the consumer's snapshot would fail
// the internal routes' actor/workspace scope match. Resolve the
@@ -459,6 +479,7 @@ export class WorkflowBlockHandler implements BlockHandler {
// child is part of that same logical run and must not add a second.
{ baseExecutionCharge: 0 }
)
childSession.setResolvedSecretTraceRegistry(childResolvedSecretTraceRegistry)
const correlation = buildCustomBlockCorrelation({
invokerExecutionId: ctx.executionId,
invokerRequestId: ctx.metadata.requestId,
@@ -509,6 +530,12 @@ export class WorkflowBlockHandler implements BlockHandler {
for (const id of [ctx.executionId, childExecutionId]) {
if (id && !sharedLargeValueIds.includes(id)) sharedLargeValueIds.push(id)
}
childSession.setTraceLargeValueAccess({
largeValueExecutionIds: sharedLargeValueIds,
largeValueKeys: ctx.largeValueKeys,
fileKeys: ctx.fileKeys,
allowLargeValueWorkflowScope: ctx.allowLargeValueWorkflowScope,
})
}
// Trusted run metadata for the child's Start block. Every field describes
@@ -570,6 +597,7 @@ export class WorkflowBlockHandler implements BlockHandler {
// internal tool calls (knowledge, guardrails, MCP, Mothership) can
// attach the required billing attribution header.
billingAttribution: childBillingAttribution,
resolvedSecretTraceRegistry: childResolvedSecretTraceRegistry,
// Fall back to the inherited metadata so a toggle-off intermediate
// child still carries the trusted identity chain to deeper children.
startRunMetadata: childStartRunMetadata ?? inherited,
@@ -672,7 +700,18 @@ export class WorkflowBlockHandler implements BlockHandler {
// failures surface identically; we just reshape the successful output. The
// child's spend is billed by its own session, not rolled onto this block.
if (isCustomBlock) {
return this.projectCustomBlockOutput(executionResult, exposedOutputs)
const exposedOutput = this.projectCustomBlockOutput(executionResult, exposedOutputs)
if (ctx.resolvedSecretTraceRegistry && childResolvedSecretTraceRegistry) {
const crossingProvenance = childResolvedSecretTraceRegistry.exportProvenanceForValue(
exposedOutput,
{ anonymous: true }
)
await ctx.resolvedSecretTraceRegistry.importProvenance(crossingProvenance, {
trusted: true,
anonymous: true,
})
}
return exposedOutput
}
return mappedResult
@@ -766,7 +805,12 @@ export class WorkflowBlockHandler implements BlockHandler {
// Cancellation lives on `ExecutionResult.status` — `ExecutionMetadata.status`
// has no 'cancelled' member, so reading it there never matches.
if (executionResult.status === 'cancelled') {
await session.safeCompleteWithCancellation({ endedAt, totalDurationMs, traceSpans })
await session.safeCompleteWithCancellation({
endedAt,
totalDurationMs,
traceSpans,
executionState: executionResult.executionState,
})
return
}
@@ -776,6 +820,7 @@ export class WorkflowBlockHandler implements BlockHandler {
finalOutput: executionResult.output ?? {},
traceSpans,
workflowInput,
executionState: executionResult.executionState,
})
}
@@ -792,6 +837,7 @@ export class WorkflowBlockHandler implements BlockHandler {
totalDurationMs: totalDuration ?? 0,
error: { message: normalized.message, stackTrace: normalized.stack },
traceSpans,
executionState: executionResult?.executionState,
})
}
+2
View File
@@ -9,6 +9,7 @@ import type {
PiiBlockOutputRedaction,
SerializableExecutionState,
} from '@/executor/execution/types'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import type { RunFromBlockContext } from '@/executor/utils/run-from-block'
import type { AgentStreamSink, UnsubscribeAgentStreamSink } from '@/providers/stream-events'
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
@@ -354,6 +355,7 @@ export interface ExecutionContext {
/** Trusted run metadata for the Start block's "Add run metadata" toggle. */
startRunMetadata?: StartBlockRunMetadata
environmentVariables: Record<string, string>
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
workflowVariables?: Record<string, any>
decisions: {
@@ -0,0 +1,44 @@
import { describe, expect, it, vi } from 'vitest'
import { resolveEnvVarReferences } from '@/executor/utils/reference-validation'
describe('resolveEnvVarReferences provenance', () => {
it('reports each successful exact, embedded, and deep substitution', () => {
const onResolved = vi.fn()
const result = resolveEnvVarReferences(
{
exact: '{{TOKEN}}',
embedded: 'Bearer {{TOKEN}} and {{OTHER}}',
missing: '{{MISSING}}',
},
{ TOKEN: 'secret', OTHER: 'other-secret' },
{ deep: true, onResolved }
)
expect(result).toEqual({
exact: 'secret',
embedded: 'Bearer secret and other-secret',
missing: '{{MISSING}}',
})
expect(onResolved.mock.calls).toEqual([
['TOKEN', 'secret'],
['TOKEN', 'secret'],
['OTHER', 'other-secret'],
])
})
it('does not report disabled embedded or missing references', () => {
const onResolved = vi.fn()
expect(
resolveEnvVarReferences(
'prefix {{TOKEN}} {{MISSING}}',
{ TOKEN: 'secret' },
{
allowEmbedded: false,
onResolved,
}
)
).toBe('prefix {{TOKEN}} {{MISSING}}')
expect(onResolved).not.toHaveBeenCalled()
})
})
@@ -26,6 +26,7 @@ export interface EnvVarResolveOptions {
onMissing?: 'keep' | 'throw' | 'empty'
deep?: boolean
missingKeys?: string[]
onResolved?: (name: string, value: string) => void
}
/**
@@ -37,7 +38,9 @@ export interface EnvVarResolveOptions {
* - `onMissing: 'keep'` - Unknown patterns pass through (e.g., Grafana's `{{instance}}`)
* - `deep: false` - Only processes strings by default; set `true` for nested objects
*/
export const ENV_VAR_RESOLVE_DEFAULTS: Required<Omit<EnvVarResolveOptions, 'missingKeys'>> = {
export const ENV_VAR_RESOLVE_DEFAULTS: Required<
Omit<EnvVarResolveOptions, 'missingKeys' | 'onResolved'>
> = {
resolveExactMatch: true,
allowEmbedded: true,
trimKeys: true,
@@ -70,7 +73,10 @@ export function resolveEnvVarReferences(
if (exactMatch) {
const envKey = trimKeys ? exactMatch[1].trim() : exactMatch[1]
const envValue = envVars[envKey]
if (envValue !== undefined) return envValue
if (envValue !== undefined) {
if (Object.hasOwn(envVars, envKey)) options.onResolved?.(envKey, envValue)
return envValue
}
if (options.missingKeys) options.missingKeys.push(envKey)
if (onMissing === 'throw') {
throw new Error(`Environment variable "${envKey}" was not found`)
@@ -88,7 +94,10 @@ export function resolveEnvVarReferences(
return value.replace(envVarPattern, (match, varName) => {
const envKey = trimKeys ? String(varName).trim() : String(varName)
const envValue = envVars[envKey]
if (envValue !== undefined) return envValue
if (envValue !== undefined) {
if (Object.hasOwn(envVars, envKey)) options.onResolved?.(envKey, envValue)
return envValue
}
if (options.missingKeys) options.missingKeys.push(envKey)
if (onMissing === 'throw') {
throw new Error(`Environment variable "${envKey}" was not found`)
@@ -0,0 +1,655 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockDecryptSecret } = vi.hoisted(() => ({
mockDecryptSecret: vi.fn(),
}))
vi.mock('@/lib/core/security/encryption', () => ({
decryptSecret: mockDecryptSecret,
}))
import {
ANONYMOUS_SECRET_TRACE_REPLACEMENT,
createResolvedSecretTraceRegistry,
isResolvedSecretTraceProvenanceV1,
ResolvedSecretTraceProvenanceAccumulator,
type ResolvedSecretTraceProvenanceV1,
ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
describe('ResolvedSecretTraceProvenanceAccumulator', () => {
const scope = { userId: 'user-1', workspaceId: 'workspace-1' }
it('unions cold, warm, and retry reports while complete', () => {
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
expect(
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' }],
scope,
})
).toBe(true)
expect(
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' }],
scope,
})
).toBe(true)
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' }],
scope,
})
expect(accumulator.exportProvenance()).toEqual({
version: 1,
complete: true,
entries: [
{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' },
{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' },
],
scope,
})
})
it('discards accumulated and future entries once completeness is lost', () => {
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' }],
scope,
})
expect(
accumulator.record({
version: 1,
complete: false,
entries: [],
scope,
})
).toBe(true)
expect(
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' }],
scope,
})
).toBe(true)
expect(accumulator.exportProvenance()).toEqual({
version: 1,
complete: false,
entries: [],
scope,
})
})
it('discards mismatched-scope reports and marks them incomplete', () => {
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
expect(
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-value' }],
scope: { userId: 'user-1', workspaceId: 'workspace-2' },
})
).toBe(true)
expect(accumulator.exportProvenance()).toEqual({
version: 1,
complete: false,
entries: [],
scope,
})
})
it('fails closed for malformed reports and terminal incompleteness', () => {
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-value' }],
scope,
})
expect(accumulator.record({ version: 1 })).toBe(false)
expect(accumulator.exportProvenance()).toEqual({
version: 1,
complete: false,
entries: [],
scope,
})
accumulator.record({
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-value' }],
scope,
})
accumulator.markIncomplete()
expect(accumulator.exportProvenance().entries).toEqual([])
})
})
describe('ResolvedSecretTraceRegistry', () => {
beforeEach(() => {
vi.clearAllMocks()
mockDecryptSecret.mockImplementation(async (encryptedValue: string) => ({
decrypted: `decrypted:${encryptedValue}`,
}))
})
it('starts with an inert catalog and activates only an exact successful resolution', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'API_KEY', plaintext: 'secret-value', encryptedValue: 'encrypted-value' },
])
expect(registry.getActiveMatches()).toEqual([])
expect(registry.recordResolved('API_KEY', 'wrong-value')).toBe(false)
expect(registry.recordResolved('MISSING', 'secret-value')).toBe(false)
expect(registry.getActiveMatches()).toEqual([])
expect(registry.isComplete()).toBe(false)
expect(registry.recordResolved('API_KEY', 'secret-value')).toBe(true)
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'secret-value', replacement: '{{API_KEY}}' },
])
})
it('uses the workspace catalog entry when personal and workspace names conflict', async () => {
const registry = await createResolvedSecretTraceRegistry({
personalEncrypted: { SHARED: 'personal-encrypted' },
workspaceEncrypted: { SHARED: 'workspace-encrypted' },
personalDecrypted: { SHARED: 'personal-secret' },
workspaceDecrypted: { SHARED: 'workspace-secret' },
})
expect(registry.recordResolved('SHARED', 'workspace-secret')).toBe(true)
expect(registry.exportProvenance()).toEqual({
version: 1,
complete: true,
entries: [{ name: 'SHARED', encryptedValue: 'workspace-encrypted' }],
})
})
it('ignores empty decryption failures but fails closed for a resolved value outside the catalog', async () => {
const registry = await createResolvedSecretTraceRegistry({
personalEncrypted: { FAILED: 'failed-ciphertext' },
workspaceEncrypted: {},
personalDecrypted: { FAILED: '', DECRYPTED_ONLY: 'not-catalogued' },
workspaceDecrypted: {},
decryptionFailures: ['FAILED'],
})
expect(registry.isComplete()).toBe(true)
expect(registry.recordResolved('FAILED', '')).toBe(false)
expect(registry.isComplete()).toBe(true)
expect(registry.recordResolved('DECRYPTED_ONLY', 'not-catalogued')).toBe(false)
expect(registry.isComplete()).toBe(false)
})
it('keeps a successful workspace override when the shadowed personal value failed', async () => {
const registry = await createResolvedSecretTraceRegistry({
personalEncrypted: { SHARED: 'broken-personal-ciphertext' },
workspaceEncrypted: { SHARED: 'workspace-ciphertext' },
personalDecrypted: { SHARED: '' },
workspaceDecrypted: { SHARED: 'workspace-secret' },
decryptionFailures: ['SHARED'],
})
expect(registry.recordResolved('SHARED', 'workspace-secret')).toBe(true)
expect(registry.exportProvenance().entries).toEqual([
{ name: 'SHARED', encryptedValue: 'workspace-ciphertext' },
])
})
it('exports encrypted active provenance without plaintext', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'raw-secret', encryptedValue: 'ciphertext' },
])
registry.recordResolved('TOKEN', 'raw-secret')
const serialized = JSON.stringify(registry.exportProvenance())
expect(serialized).toContain('ciphertext')
expect(serialized).not.toContain('raw-secret')
})
it('restores old encrypted values alongside the current catalog after rotation', async () => {
mockDecryptSecret.mockResolvedValueOnce({ decrypted: 'old-secret' })
const oldProvenance: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'old-ciphertext' }],
}
const registry = await createResolvedSecretTraceRegistry({
personalEncrypted: { TOKEN: 'new-ciphertext' },
workspaceEncrypted: {},
personalDecrypted: { TOKEN: 'new-secret' },
workspaceDecrypted: {},
restoredProvenance: oldProvenance,
restoreTrusted: true,
requireRestoredProvenance: true,
})
registry.recordResolved('TOKEN', 'new-secret')
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'new-secret', replacement: '{{TOKEN}}' },
{ plaintext: 'old-secret', replacement: '{{TOKEN}}' },
])
expect(registry.exportProvenance().entries).toEqual([
{ name: 'TOKEN', encryptedValue: 'new-ciphertext' },
{ name: 'TOKEN', encryptedValue: 'old-ciphertext' },
])
})
it('marks untrusted, missing, malformed, and undecryptable restoration incomplete', async () => {
const provenance: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
}
const untrusted = new ResolvedSecretTraceRegistry()
expect(await untrusted.importProvenance(provenance, { trusted: false })).toBe(false)
expect(untrusted.isComplete()).toBe(false)
expect(mockDecryptSecret).not.toHaveBeenCalled()
const missing = await createResolvedSecretTraceRegistry({
personalEncrypted: {},
workspaceEncrypted: {},
personalDecrypted: {},
workspaceDecrypted: {},
requireRestoredProvenance: true,
restoreTrusted: true,
})
expect(missing.isComplete()).toBe(false)
const malformed = new ResolvedSecretTraceRegistry()
expect(await malformed.importProvenance({ version: 1 }, { trusted: true })).toBe(false)
expect(malformed.isComplete()).toBe(false)
mockDecryptSecret.mockRejectedValueOnce(new Error('cannot decrypt'))
const undecryptable = new ResolvedSecretTraceRegistry()
expect(await undecryptable.importProvenance(provenance, { trusted: true })).toBe(false)
expect(undecryptable.isComplete()).toBe(false)
})
it('uses anonymous replacements for cross-scope provenance', async () => {
mockDecryptSecret.mockResolvedValueOnce({ decrypted: 'publisher-secret' })
const registry = new ResolvedSecretTraceRegistry()
await registry.importProvenance(
{
version: 1,
complete: true,
entries: [{ name: 'PUBLISHER_TOKEN', encryptedValue: 'publisher-ciphertext' }],
},
{ trusted: true, anonymous: true }
)
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'publisher-secret', replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT },
])
expect(registry.exportProvenance().entries).toEqual([
{ encryptedValue: 'publisher-ciphertext' },
])
})
it('preserves labels only when imported provenance has the same complete scope', async () => {
const provenance: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
}
const sameScope = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
const mismatchedScope = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-2',
})
const missingReceiverScope = new ResolvedSecretTraceRegistry()
const missingSourceScope = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
expect(await sameScope.importProvenance(provenance, { trusted: true })).toBe(true)
expect(await mismatchedScope.importProvenance(provenance, { trusted: true })).toBe(true)
expect(await missingReceiverScope.importProvenance(provenance, { trusted: true })).toBe(true)
expect(
await missingSourceScope.importProvenance(
{ version: 1, complete: true, entries: provenance.entries },
{ trusted: true }
)
).toBe(true)
expect(sameScope.getActiveMatches()).toEqual([
{ plaintext: 'decrypted:ciphertext', replacement: '{{TOKEN}}' },
])
for (const registry of [mismatchedScope, missingReceiverScope, missingSourceScope]) {
expect(registry.getActiveMatches()).toEqual([
{
plaintext: 'decrypted:ciphertext',
replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT,
},
])
}
})
it('imports all same-scope provenance across an in-process boundary', async () => {
const registry = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
const provenance: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [
{ name: 'PRESENT', encryptedValue: 'present-ciphertext' },
{ name: 'DORMANT_IN_OUTPUT', encryptedValue: 'other-ciphertext' },
],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
}
expect(
await registry.importCrossingProvenance(
provenance,
{ output: 'decrypted:present-ciphertext' },
{ trusted: true }
)
).toBe(true)
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'decrypted:present-ciphertext', replacement: '{{PRESENT}}' },
{ plaintext: 'decrypted:other-ciphertext', replacement: '{{DORMANT_IN_OUTPUT}}' },
])
})
it('filters and anonymizes provenance crossing from another scope', async () => {
const registry = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
const provenance: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [
{ name: 'PRESENT', encryptedValue: 'present-ciphertext' },
{ name: 'ABSENT', encryptedValue: 'absent-ciphertext' },
],
scope: { userId: 'user-1', workspaceId: 'workspace-2' },
}
expect(
await registry.importCrossingProvenance(
provenance,
{ output: 'decrypted:present-ciphertext' },
{ trusted: true }
)
).toBe(true)
expect(registry.getActiveMatches()).toEqual([
{
plaintext: 'decrypted:present-ciphertext',
replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT,
},
])
})
it('exports only active secrets whose exact literals cross a value boundary', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'PRESENT', plaintext: 'present-secret', encryptedValue: 'present-ciphertext' },
{ name: 'ABSENT', plaintext: 'absent-secret', encryptedValue: 'absent-ciphertext' },
{ name: 'UNUSED', plaintext: 'unused-secret', encryptedValue: 'unused-ciphertext' },
])
registry.recordResolved('PRESENT', 'present-secret')
registry.recordResolved('ABSENT', 'absent-secret')
const provenance = registry.exportProvenanceForValue(
{ nested: [{ 'key-present-secret': new Error('failed with present-secret') }] },
{ anonymous: true }
)
expect(provenance).toEqual({
version: 1,
complete: true,
entries: [{ encryptedValue: 'present-ciphertext' }],
})
})
it('exports active numeric, boolean, and null literals crossing a value boundary', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'NUMBER', plaintext: '1234', encryptedValue: 'number-ciphertext' },
{ name: 'BOOLEAN', plaintext: 'false', encryptedValue: 'boolean-ciphertext' },
{ name: 'NULL', plaintext: 'null', encryptedValue: 'null-ciphertext' },
{ name: 'ABSENT', plaintext: '5678', encryptedValue: 'absent-ciphertext' },
])
registry.recordResolved('NUMBER', '1234')
registry.recordResolved('BOOLEAN', 'false')
registry.recordResolved('NULL', 'null')
registry.recordResolved('ABSENT', '5678')
expect(
registry.exportProvenanceForValue(
{ number: 1234, boolean: false, nullable: null },
{ anonymous: true }
)
).toEqual({
version: 1,
complete: true,
entries: [
{ encryptedValue: 'boolean-ciphertext' },
{ encryptedValue: 'null-ciphertext' },
{ encryptedValue: 'number-ciphertext' },
],
})
})
it('marks a bounded cross-boundary scan incomplete when an enumerable accessor is opaque', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'secret', encryptedValue: 'ciphertext' },
])
registry.recordResolved('TOKEN', 'secret')
const value = {}
Object.defineProperty(value, 'opaque', {
enumerable: true,
get: () => 'secret',
})
expect(registry.exportProvenanceForValue(value, { anonymous: true })).toEqual({
version: 1,
complete: false,
entries: [],
})
})
it('does not claim a complete cross-boundary scan for opaque large-value refs', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'secret', encryptedValue: 'ciphertext' },
])
registry.recordResolved('TOKEN', 'secret')
expect(
registry.exportProvenanceForValue(
{
__simLargeValueRef: true,
version: 1,
id: 'lv_ABCDEFGHIJKL',
kind: 'object',
size: 1024,
},
{ anonymous: true }
)
).toEqual({ version: 1, complete: false, entries: [] })
})
it('does not snapshot or enqueue an entire wide crossing object', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'needle-value', encryptedValue: 'ciphertext' },
])
registry.recordResolved('TOKEN', 'needle-value')
const wideValue: Record<string, string> = {}
for (let index = 0; index < 30_000; index++) {
wideValue[`field_${index}`] = 'safe-value'
}
const descriptorSnapshotSpy = vi.spyOn(Object, 'getOwnPropertyDescriptors')
const provenance = registry.exportProvenanceForValue(wideValue, { anonymous: true })
const descriptorSnapshotCalls = descriptorSnapshotSpy.mock.calls.length
descriptorSnapshotSpy.mockRestore()
expect(provenance).toEqual({
version: 1,
complete: false,
entries: [],
})
expect(descriptorSnapshotCalls).toBe(0)
})
it('handles duplicate and empty values deterministically', () => {
const registry = new ResolvedSecretTraceRegistry([
{ name: 'Z_TOKEN', plaintext: 'same', encryptedValue: 'z-ciphertext' },
{ name: 'A_TOKEN', plaintext: 'same', encryptedValue: 'a-ciphertext' },
{ name: 'EMPTY', plaintext: '', encryptedValue: 'empty-ciphertext' },
{ name: 'A', plaintext: 'A', encryptedValue: 'short-ciphertext' },
])
registry.recordResolved('Z_TOKEN', 'same')
registry.recordResolved('A_TOKEN', 'same')
registry.recordResolved('EMPTY', '')
expect(registry.getActiveMatches()).toEqual([{ plaintext: 'same', replacement: '{{A_TOKEN}}' }])
registry.recordResolved('A', 'A')
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'same', replacement: '{{A_TOKEN}}' },
{ plaintext: 'A', replacement: '{{A}}' },
])
})
it('marks the registry incomplete when active provenance exceeds its hard cap', () => {
const entries = Array.from({ length: 10_001 }, (_, index) => ({
name: `SECRET_${index}`,
plaintext: `value-${index}`,
encryptedValue: `ciphertext-${index}`,
}))
const registry = new ResolvedSecretTraceRegistry(entries)
for (const entry of entries) {
registry.recordResolved(entry.name, entry.plaintext)
}
expect(registry.isComplete()).toBe(false)
expect(registry.exportProvenance().entries).toEqual([])
})
it('bounds provenance by serialized JSON bytes including control-character escapes', () => {
const encryptedValue = '\u0000'.repeat(1_400_000)
const provenance: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [{ name: 'TOKEN', encryptedValue }],
}
expect(Buffer.byteLength(encryptedValue, 'utf8')).toBeLessThan(8 * 1024 * 1024)
expect(Buffer.byteLength(JSON.stringify(provenance), 'utf8')).toBeGreaterThan(8 * 1024 * 1024)
expect(isResolvedSecretTraceProvenanceV1(provenance)).toBe(false)
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'secret', encryptedValue },
])
expect(registry.recordResolved('TOKEN', 'secret')).toBe(true)
expect(registry.isComplete()).toBe(false)
expect(registry.exportProvenance().entries).toEqual([])
})
it('rejects incomplete provenance that still carries entries', () => {
expect(
isResolvedSecretTraceProvenanceV1({
version: 1,
complete: false,
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
})
).toBe(false)
})
it('rejects non-canonical provenance fields before applying the serialized-size bound', () => {
const entry = { name: 'TOKEN', encryptedValue: 'ciphertext' }
const scope = { userId: 'user-1', workspaceId: 'workspace-1' }
expect(
isResolvedSecretTraceProvenanceV1({
version: 1,
complete: true,
entries: [entry],
scope,
extra: 'not-transported',
})
).toBe(false)
expect(
isResolvedSecretTraceProvenanceV1({
version: 1,
complete: true,
entries: [{ ...entry, extra: 'not-transported' }],
scope,
})
).toBe(false)
expect(
isResolvedSecretTraceProvenanceV1({
version: 1,
complete: true,
entries: [entry],
scope: { ...scope, extra: 'not-transported' },
})
).toBe(false)
const entries = [entry]
Object.assign(entries, { extra: 'not-transported' })
expect(isResolvedSecretTraceProvenanceV1({ version: 1, complete: true, entries, scope })).toBe(
false
)
})
it('stops consuming a dormant catalog when its entry cap is exceeded', () => {
let yieldedEntries = 0
function* catalogEntries() {
for (let index = 0; index < 20_000; index++) {
yieldedEntries++
yield {
name: `SECRET_${index}`,
plaintext: `value-${index}`,
encryptedValue: `ciphertext-${index}`,
}
}
}
const registry = new ResolvedSecretTraceRegistry(catalogEntries())
expect(yieldedEntries).toBe(10_001)
expect(registry.isComplete()).toBe(false)
expect(registry.exportProvenance().entries).toEqual([])
})
it('marks an oversized dormant catalog value incomplete without retaining it', () => {
const oversizedPlaintext = 'x'.repeat(8 * 1024 * 1024)
const registry = new ResolvedSecretTraceRegistry([
{
name: 'OVERSIZED',
plaintext: oversizedPlaintext,
encryptedValue: 'ciphertext',
},
])
expect(registry.isComplete()).toBe(false)
expect(registry.recordResolved('OVERSIZED', oversizedPlaintext)).toBe(false)
expect(registry.getActiveMatches()).toEqual([])
})
})
@@ -0,0 +1,784 @@
import { decryptSecret } from '@/lib/core/security/encryption'
import { isLargeArrayManifest } from '@/lib/execution/payloads/large-array-manifest-metadata'
import { isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
export const ANONYMOUS_SECRET_TRACE_REPLACEMENT = '[REDACTED_SECRET]'
const MAX_PROVENANCE_ENTRIES = 10_000
const MAX_SERIALIZED_PROVENANCE_BYTES = 8 * 1024 * 1024
const MAX_TRACE_CATALOG_ENTRIES = MAX_PROVENANCE_ENTRIES
const MAX_TRACE_CATALOG_BYTES = 8 * 1024 * 1024
const MAX_PROVENANCE_FILTER_NODES = 50_000
const MAX_PROVENANCE_FILTER_CHARACTERS = 16 * 1024 * 1024
const MAX_PROVENANCE_FILTER_COMPARISONS = 1_000_000
const ERROR_CONTENT_PROPERTY_NAMES = ['name', 'message', 'stack', 'cause', 'errors'] as const
const PROVENANCE_PROPERTY_NAMES = new Set(['version', 'complete', 'entries', 'scope'])
const PROVENANCE_ENTRY_PROPERTY_NAMES = new Set(['encryptedValue', 'name'])
const PROVENANCE_SCOPE_PROPERTY_NAMES = new Set(['userId', 'workspaceId'])
export interface ResolvedSecretTraceCatalogEntry {
name: string
plaintext: string
encryptedValue: string
}
export interface ResolvedSecretTraceMatch {
plaintext: string
replacement: string
}
export interface ResolvedSecretTraceProvenanceEntryV1 {
encryptedValue: string
name?: string
}
export interface ResolvedSecretTraceScopeV1 {
userId: string
workspaceId?: string
}
export interface ResolvedSecretTraceProvenanceV1 {
version: 1
complete: boolean
entries: ResolvedSecretTraceProvenanceEntryV1[]
scope?: ResolvedSecretTraceScopeV1
}
interface ActiveSecretEntry extends ResolvedSecretTraceCatalogEntry {
anonymous: boolean
}
export interface ImportResolvedSecretTraceProvenanceOptions {
trusted: boolean
anonymous?: boolean
}
export interface ExportResolvedSecretTraceProvenanceForValueOptions {
anonymous?: boolean
}
export interface CreateResolvedSecretTraceRegistryOptions {
personalEncrypted: Record<string, string>
workspaceEncrypted: Record<string, string>
personalDecrypted: Record<string, string>
workspaceDecrypted: Record<string, string>
decryptionFailures?: readonly string[]
restoredProvenance?: unknown
restoreTrusted?: boolean
requireRestoredProvenance?: boolean
scope?: ResolvedSecretTraceScopeV1
}
function compareStrings(left: string, right: string): number {
if (left < right) return -1
if (left > right) return 1
return 0
}
function cloneProvenanceScope(scope: ResolvedSecretTraceScopeV1): ResolvedSecretTraceScopeV1 {
return {
userId: scope.userId,
...(scope.workspaceId ? { workspaceId: scope.workspaceId } : {}),
}
}
function serializedJsonStringByteSize(value: string): number {
let byteSize = 2
for (let index = 0; index < value.length; index++) {
const codeUnit = value.charCodeAt(index)
if (codeUnit === 0x22 || codeUnit === 0x5c) {
byteSize += 2
} else if (
codeUnit === 0x08 ||
codeUnit === 0x09 ||
codeUnit === 0x0a ||
codeUnit === 0x0c ||
codeUnit === 0x0d
) {
byteSize += 2
} else if (codeUnit <= 0x1f) {
byteSize += 6
} else if (codeUnit >= 0xd800 && codeUnit <= 0xdbff) {
const nextCodeUnit = value.charCodeAt(index + 1)
if (nextCodeUnit >= 0xdc00 && nextCodeUnit <= 0xdfff) {
byteSize += 4
index++
} else {
byteSize += 6
}
} else if (codeUnit >= 0xdc00 && codeUnit <= 0xdfff) {
byteSize += 6
} else if (codeUnit <= 0x7f) {
byteSize++
} else if (codeUnit <= 0x7ff) {
byteSize += 2
} else {
byteSize += 3
}
}
return byteSize
}
function serializedProvenanceEntryByteSize(entry: ResolvedSecretTraceProvenanceEntryV1): number {
let byteSize =
Buffer.byteLength('{"encryptedValue":', 'utf8') +
serializedJsonStringByteSize(entry.encryptedValue)
if (entry.name !== undefined) {
byteSize += Buffer.byteLength(',"name":', 'utf8') + serializedJsonStringByteSize(entry.name)
}
return byteSize + 1
}
function serializedProvenanceEnvelopeByteSize(
complete: boolean,
scope: ResolvedSecretTraceScopeV1 | undefined
): number {
let byteSize = Buffer.byteLength(
`{"version":1,"complete":${complete ? 'true' : 'false'},"entries":[]`,
'utf8'
)
if (scope) {
byteSize +=
Buffer.byteLength(',"scope":{"userId":', 'utf8') + serializedJsonStringByteSize(scope.userId)
if (scope.workspaceId !== undefined) {
byteSize +=
Buffer.byteLength(',"workspaceId":', 'utf8') +
serializedJsonStringByteSize(scope.workspaceId)
}
byteSize++
}
return byteSize + 1
}
function isSerializedProvenanceWithinLimit(
complete: boolean,
entries: readonly ResolvedSecretTraceProvenanceEntryV1[],
scope: ResolvedSecretTraceScopeV1 | undefined
): boolean {
let byteSize = serializedProvenanceEnvelopeByteSize(complete, scope)
for (let index = 0; index < entries.length; index++) {
byteSize += serializedProvenanceEntryByteSize(entries[index]) + (index === 0 ? 0 : 1)
if (byteSize > MAX_SERIALIZED_PROVENANCE_BYTES) return false
}
return byteSize <= MAX_SERIALIZED_PROVENANCE_BYTES
}
function toProvenanceEntry(entry: ActiveSecretEntry): ResolvedSecretTraceProvenanceEntryV1 {
return {
encryptedValue: entry.encryptedValue,
...(!entry.anonymous && entry.name ? { name: entry.name } : {}),
}
}
function hasOwn(record: Record<string, string>, name: string): boolean {
return Object.hasOwn(record, name)
}
function isExactPlainDataRecord(
value: unknown,
allowedProperties: ReadonlySet<string>,
requiredProperties: readonly string[]
): value is Record<string, unknown> {
if (value === null || typeof value !== 'object' || Array.isArray(value)) return false
try {
const prototype = Object.getPrototypeOf(value)
if (prototype !== Object.prototype && prototype !== null) return false
const properties = Reflect.ownKeys(value)
for (const property of properties) {
if (typeof property !== 'string' || !allowedProperties.has(property)) return false
const descriptor = Object.getOwnPropertyDescriptor(value, property)
if (!descriptor?.enumerable || !('value' in descriptor)) return false
}
return requiredProperties.every((property) => Object.hasOwn(value, property))
} catch {
return false
}
}
function isExactProvenanceEntriesArray(value: unknown): value is unknown[] {
if (!Array.isArray(value)) return false
try {
if (Object.getPrototypeOf(value) !== Array.prototype) return false
const properties = Reflect.ownKeys(value)
if (properties.length !== value.length + 1 || !properties.includes('length')) return false
for (let index = 0; index < value.length; index++) {
const descriptor = Object.getOwnPropertyDescriptor(value, String(index))
if (!descriptor?.enumerable || !('value' in descriptor)) return false
}
return true
} catch {
return false
}
}
function catalogEntryByteSize(entry: ResolvedSecretTraceCatalogEntry): number {
return (
Buffer.byteLength(entry.name, 'utf8') +
Buffer.byteLength(entry.plaintext, 'utf8') +
Buffer.byteLength(entry.encryptedValue, 'utf8')
)
}
function buildEffectiveCatalogEntry(
options: CreateResolvedSecretTraceRegistryOptions,
failedNames: ReadonlySet<string>,
name: string,
encryptedValue: string
): ResolvedSecretTraceCatalogEntry | undefined {
const plaintext = hasOwn(options.workspaceDecrypted, name)
? options.workspaceDecrypted[name]
: options.personalDecrypted[name]
if (plaintext === undefined || (plaintext.length === 0 && failedNames.has(name))) {
return undefined
}
return { name, plaintext, encryptedValue }
}
function* iterateEffectiveCatalogEntries(
options: CreateResolvedSecretTraceRegistryOptions,
failedNames: ReadonlySet<string>
): Generator<ResolvedSecretTraceCatalogEntry> {
for (const name in options.personalEncrypted) {
if (!hasOwn(options.personalEncrypted, name)) continue
const encryptedValue = hasOwn(options.workspaceEncrypted, name)
? options.workspaceEncrypted[name]
: options.personalEncrypted[name]
const entry = buildEffectiveCatalogEntry(options, failedNames, name, encryptedValue)
if (entry) yield entry
}
for (const name in options.workspaceEncrypted) {
if (!hasOwn(options.workspaceEncrypted, name) || hasOwn(options.personalEncrypted, name)) {
continue
}
const entry = buildEffectiveCatalogEntry(
options,
failedNames,
name,
options.workspaceEncrypted[name]
)
if (entry) yield entry
}
}
function isProvenanceEntry(value: unknown): value is ResolvedSecretTraceProvenanceEntryV1 {
if (!isExactPlainDataRecord(value, PROVENANCE_ENTRY_PROPERTY_NAMES, ['encryptedValue'])) {
return false
}
const entry = value
return (
typeof entry.encryptedValue === 'string' &&
entry.encryptedValue.length > 0 &&
(entry.name === undefined || (typeof entry.name === 'string' && entry.name.length > 0))
)
}
function isProvenanceScope(value: unknown): value is ResolvedSecretTraceScopeV1 {
if (!isExactPlainDataRecord(value, PROVENANCE_SCOPE_PROPERTY_NAMES, ['userId'])) return false
const scope = value
return (
typeof scope.userId === 'string' &&
scope.userId.length > 0 &&
(scope.workspaceId === undefined ||
(typeof scope.workspaceId === 'string' && scope.workspaceId.length > 0))
)
}
function scopesMatch(
left: ResolvedSecretTraceScopeV1 | undefined,
right: ResolvedSecretTraceScopeV1 | undefined
): boolean {
return (
(left === undefined && right === undefined) ||
(left !== undefined &&
right !== undefined &&
left.userId === right.userId &&
left.workspaceId === right.workspaceId)
)
}
export function isResolvedSecretTraceProvenanceV1(
value: unknown
): value is ResolvedSecretTraceProvenanceV1 {
if (
!isExactPlainDataRecord(value, PROVENANCE_PROPERTY_NAMES, ['version', 'complete', 'entries'])
) {
return false
}
const provenance = value
if (
provenance.version !== 1 ||
typeof provenance.complete !== 'boolean' ||
!isExactProvenanceEntriesArray(provenance.entries) ||
provenance.entries.length > MAX_PROVENANCE_ENTRIES ||
!provenance.entries.every(isProvenanceEntry) ||
(!provenance.complete && provenance.entries.length > 0) ||
(provenance.scope !== undefined && !isProvenanceScope(provenance.scope))
) {
return false
}
return isSerializedProvenanceWithinLimit(
provenance.complete,
provenance.entries,
provenance.scope
)
}
/**
* Unions encrypted provenance reports emitted during one internal transport invocation.
* It never decrypts values or projects trace content; the execution registry remains the
* only owner of plaintext matches and replacement policy.
*/
export class ResolvedSecretTraceProvenanceAccumulator {
private readonly scope?: ResolvedSecretTraceScopeV1
private provenance: ResolvedSecretTraceProvenanceV1
constructor(scope?: ResolvedSecretTraceScopeV1) {
this.scope = scope ? cloneProvenanceScope(scope) : undefined
this.provenance = this.emptyProvenance(true)
}
/** Adds one cold, warm-pool, or retry report without decrypting its entries. */
record(provenance: unknown): boolean {
if (!isResolvedSecretTraceProvenanceV1(provenance)) {
this.provenance = this.emptyProvenance(false)
return false
}
const sameScope = scopesMatch(provenance.scope, this.scope)
const complete = this.provenance.complete && provenance.complete && sameScope
if (!complete) {
this.provenance = this.emptyProvenance(false)
return true
}
const entries = new Map(
this.provenance.entries.map((entry) => [
`${entry.name ?? ''}\u0000${entry.encryptedValue}`,
entry,
])
)
for (const entry of provenance.entries) {
const scopedEntry: ResolvedSecretTraceProvenanceEntryV1 = {
encryptedValue: entry.encryptedValue,
...(entry.name ? { name: entry.name } : {}),
}
entries.set(`${scopedEntry.name ?? ''}\u0000${scopedEntry.encryptedValue}`, scopedEntry)
}
const merged: ResolvedSecretTraceProvenanceV1 = {
version: 1,
complete: true,
entries: [...entries.values()],
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
}
if (!isResolvedSecretTraceProvenanceV1(merged)) {
this.provenance = this.emptyProvenance(false)
return false
}
this.provenance = merged
return true
}
/** Marks the invocation incomplete and discards entries that can no longer be trusted. */
markIncomplete(): void {
this.provenance = this.emptyProvenance(false)
}
exportProvenance(): ResolvedSecretTraceProvenanceV1 {
return structuredClone(this.provenance)
}
private emptyProvenance(complete: boolean): ResolvedSecretTraceProvenanceV1 {
return {
version: 1,
complete,
entries: [],
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
}
}
}
/**
* Tracks Secrets-tab values that were actually substituted through `{{NAME}}` during one run.
* The current catalog is inert until a successful resolution activates an entry.
*/
export class ResolvedSecretTraceRegistry {
private readonly catalog = new Map<string, ResolvedSecretTraceCatalogEntry>()
private catalogBytes = 0
private readonly activeEntries = new Map<string, ActiveSecretEntry>()
private activeProvenanceEntryBytes = 0
private complete = true
private readonly scope?: ResolvedSecretTraceScopeV1
private readonly completeProvenanceEnvelopeBytes: number
constructor(
catalogEntries: Iterable<ResolvedSecretTraceCatalogEntry> = [],
scope?: ResolvedSecretTraceScopeV1
) {
this.scope = scope ? cloneProvenanceScope(scope) : undefined
this.completeProvenanceEnvelopeBytes = serializedProvenanceEnvelopeByteSize(true, this.scope)
if (this.completeProvenanceEnvelopeBytes > MAX_SERIALIZED_PROVENANCE_BYTES) {
this.markIncomplete()
}
let catalogEntriesSeen = 0
for (const entry of catalogEntries) {
catalogEntriesSeen++
if (catalogEntriesSeen > MAX_TRACE_CATALOG_ENTRIES) {
this.markIncomplete()
break
}
if (!this.addCatalogEntry(entry)) break
}
}
/** Activates a configured secret only when the resolved runtime value matches its catalog value. */
recordResolved(name: string, resolvedValue: string): boolean {
if (resolvedValue.length === 0) return false
const catalogEntry = this.catalog.get(name)
if (!catalogEntry || catalogEntry.plaintext !== resolvedValue) {
this.markIncomplete()
return false
}
this.addActiveEntry({ ...catalogEntry, anonymous: false })
return true
}
/** Imports encrypted provenance only from a boundary that has already established trust. */
async importProvenance(
provenance: unknown,
options: ImportResolvedSecretTraceProvenanceOptions
): Promise<boolean> {
if (!options.trusted || !isResolvedSecretTraceProvenanceV1(provenance)) {
this.markIncomplete()
return false
}
if (!provenance.complete) {
this.markIncomplete()
}
const sameScope = scopesMatch(provenance.scope, this.scope)
let importedAll = true
for (const entry of provenance.entries) {
try {
const { decrypted } = await decryptSecret(entry.encryptedValue)
this.addActiveEntry({
name: entry.name ?? '',
plaintext: decrypted,
encryptedValue: entry.encryptedValue,
anonymous: options.anonymous === true || !sameScope || entry.name === undefined,
})
} catch {
importedAll = false
this.markIncomplete()
}
}
return importedAll
}
/** Imports all same-scope provenance, or only anonymous literals crossing a trust boundary. */
async importCrossingProvenance(
provenance: unknown,
crossingValue: unknown,
options: { trusted: boolean }
): Promise<boolean> {
if (!options.trusted || !isResolvedSecretTraceProvenanceV1(provenance)) {
return this.importProvenance(provenance, { trusted: options.trusted })
}
if (scopesMatch(provenance.scope, this.scope)) {
return this.importProvenance(provenance, { trusted: true })
}
const sourceRegistry = new ResolvedSecretTraceRegistry([], provenance.scope)
const sourceImported = await sourceRegistry.importProvenance(provenance, { trusted: true })
const crossingProvenance = sourceRegistry.exportProvenanceForValue(crossingValue, {
anonymous: true,
})
const crossingImported = await this.importProvenance(crossingProvenance, { trusted: true })
return sourceImported && crossingImported
}
/** Returns deterministic literal replacements for the terminal TraceSpan projection. */
getActiveMatches(): readonly ResolvedSecretTraceMatch[] {
const candidatesByPlaintext = new Map<string, ActiveSecretEntry[]>()
for (const entry of this.activeEntries.values()) {
if (entry.plaintext.length === 0) continue
const candidates = candidatesByPlaintext.get(entry.plaintext) ?? []
candidates.push(entry)
candidatesByPlaintext.set(entry.plaintext, candidates)
}
const matches = [...candidatesByPlaintext.keys()].map((plaintext) => {
const candidates = candidatesByPlaintext.get(plaintext) ?? []
const anonymous = candidates.some((candidate) => candidate.anonymous)
const firstNamed = candidates
.filter((candidate) => !candidate.anonymous)
.sort((left, right) => compareStrings(left.name, right.name))[0]
const replacement = anonymous
? ANONYMOUS_SECRET_TRACE_REPLACEMENT
: `{{${firstNamed?.name ?? ''}}}`
return { plaintext, replacement }
})
return matches.sort(
(left, right) =>
right.plaintext.length - left.plaintext.length ||
compareStrings(left.plaintext, right.plaintext) ||
compareStrings(left.replacement, right.replacement)
)
}
isComplete(): boolean {
return this.complete
}
markIncomplete(): void {
this.complete = false
}
/** Serializes only encrypted active values; plaintext never enters execution state. */
exportProvenance(): ResolvedSecretTraceProvenanceV1 {
const entries = this.complete
? this.buildProvenanceEntries([...this.activeEntries.values()])
: []
return {
version: 1,
complete: this.complete,
entries,
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
}
}
/**
* Exports only active provenance whose exact plaintext occurs in a cross-boundary value.
* Bounded traversal returns incomplete provenance instead of broadening to unrelated secrets.
*/
exportProvenanceForValue(
value: unknown,
options: ExportResolvedSecretTraceProvenanceForValueOptions = {}
): ResolvedSecretTraceProvenanceV1 {
if (!this.complete) return { version: 1, complete: false, entries: [] }
const candidatesByPlaintext = new Map<string, ActiveSecretEntry>()
const sortedActiveEntries = [...this.activeEntries.values()].sort(
(left, right) =>
compareStrings(left.name, right.name) ||
compareStrings(left.encryptedValue, right.encryptedValue)
)
for (const entry of sortedActiveEntries) {
if (entry.plaintext.length > 0 && !candidatesByPlaintext.has(entry.plaintext)) {
candidatesByPlaintext.set(entry.plaintext, entry)
}
}
const matchedEntries = new Map<string, ActiveSecretEntry>()
const pendingValues: unknown[] = [value]
const visited = new WeakSet<object>()
let scannedNodes = 0
let scannedCharacters = 0
let comparisons = 0
let enumeratedProperties = 0
let scanComplete = true
const scanString = (candidate: string): boolean => {
scannedCharacters += candidate.length
if (scannedCharacters > MAX_PROVENANCE_FILTER_CHARACTERS) return false
for (const [plaintext, entry] of candidatesByPlaintext) {
if (matchedEntries.has(plaintext)) continue
comparisons++
if (comparisons > MAX_PROVENANCE_FILTER_COMPARISONS) return false
if (candidate.includes(plaintext)) {
matchedEntries.set(plaintext, entry)
}
}
return true
}
const scanProperty = (key: string, descriptor: PropertyDescriptor): boolean => {
if (scannedNodes + pendingValues.length >= MAX_PROVENANCE_FILTER_NODES) return false
scannedNodes++
if (!scanString(key)) return false
if (matchedEntries.size >= candidatesByPlaintext.size) return true
if ('value' in descriptor) {
if (scannedNodes + pendingValues.length >= MAX_PROVENANCE_FILTER_NODES) return false
pendingValues.push(descriptor.value)
} else if (descriptor.enumerable) {
return false
}
return true
}
while (pendingValues.length > 0 && matchedEntries.size < candidatesByPlaintext.size) {
const current = pendingValues.pop()
scannedNodes++
if (scannedNodes > MAX_PROVENANCE_FILTER_NODES) {
scanComplete = false
break
}
if (
typeof current === 'string' ||
typeof current === 'number' ||
typeof current === 'boolean' ||
current === null
) {
if (!scanString(String(current))) scanComplete = false
if (!scanComplete) break
continue
}
if (typeof current !== 'object' || visited.has(current)) {
continue
}
visited.add(current)
if (isLargeValueRef(current) || isLargeArrayManifest(current)) {
scanComplete = false
break
}
try {
for (const key of ERROR_CONTENT_PROPERTY_NAMES) {
const descriptor = Object.getOwnPropertyDescriptor(current, key)
if (descriptor && !descriptor.enumerable && !scanProperty(key, descriptor)) {
scanComplete = false
break
}
if (matchedEntries.size >= candidatesByPlaintext.size) break
}
if (!scanComplete || matchedEntries.size >= candidatesByPlaintext.size) break
for (const key in current as Record<string, unknown>) {
enumeratedProperties++
if (enumeratedProperties > MAX_PROVENANCE_FILTER_NODES) {
scanComplete = false
break
}
const descriptor = Object.getOwnPropertyDescriptor(current, key)
if (!descriptor) continue
if (!scanProperty(key, descriptor)) {
scanComplete = false
break
}
if (matchedEntries.size >= candidatesByPlaintext.size) break
}
if (!scanComplete) break
} catch {
scanComplete = false
break
}
}
const complete = this.complete && scanComplete
const entries = complete
? this.buildProvenanceEntries([...matchedEntries.values()], options.anonymous)
: []
return {
version: 1,
complete,
entries,
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
}
}
private addActiveEntry(entry: ActiveSecretEntry): void {
const key = `${entry.anonymous ? 'anonymous' : entry.name}\u0000${entry.encryptedValue}`
if (this.activeEntries.has(key)) {
this.activeEntries.set(key, entry)
return
}
const entryBytes = serializedProvenanceEntryByteSize(toProvenanceEntry(entry))
const separatorBytes = this.activeEntries.size === 0 ? 0 : 1
if (
this.activeEntries.size >= MAX_PROVENANCE_ENTRIES ||
this.completeProvenanceEnvelopeBytes +
this.activeProvenanceEntryBytes +
separatorBytes +
entryBytes >
MAX_SERIALIZED_PROVENANCE_BYTES
) {
this.markIncomplete()
return
}
this.activeEntries.set(key, entry)
this.activeProvenanceEntryBytes += separatorBytes + entryBytes
}
private addCatalogEntry(entry: ResolvedSecretTraceCatalogEntry): boolean {
const existing = this.catalog.get(entry.name)
const nextCatalogBytes =
this.catalogBytes -
(existing ? catalogEntryByteSize(existing) : 0) +
catalogEntryByteSize(entry)
const nextCatalogSize = this.catalog.size + (existing ? 0 : 1)
if (nextCatalogSize > MAX_TRACE_CATALOG_ENTRIES || nextCatalogBytes > MAX_TRACE_CATALOG_BYTES) {
this.markIncomplete()
return false
}
this.catalog.set(entry.name, { ...entry })
this.catalogBytes = nextCatalogBytes
return true
}
private buildProvenanceEntries(
activeEntries: ActiveSecretEntry[],
forceAnonymous = false
): ResolvedSecretTraceProvenanceEntryV1[] {
return activeEntries
.sort(
(left, right) =>
compareStrings(left.name, right.name) ||
compareStrings(left.encryptedValue, right.encryptedValue)
)
.map((entry) => ({
...toProvenanceEntry({ ...entry, anonymous: forceAnonymous || entry.anonymous }),
}))
}
}
/** Builds the effective workspace-over-personal catalog and restores trusted active provenance. */
export async function createResolvedSecretTraceRegistry(
options: CreateResolvedSecretTraceRegistryOptions
): Promise<ResolvedSecretTraceRegistry> {
const failedNames = new Set(options.decryptionFailures ?? [])
const registry = new ResolvedSecretTraceRegistry(
iterateEffectiveCatalogEntries(options, failedNames),
options.scope
)
if (options.restoredProvenance !== undefined) {
await registry.importProvenance(options.restoredProvenance, {
trusted: options.restoreTrusted === true,
})
} else if (options.requireRestoredProvenance) {
registry.markIncomplete()
}
return registry
}
/** Creates a scoped fail-closed registry when trusted catalog provenance is unavailable. */
export function createIncompleteResolvedSecretTraceRegistry(
scope?: ResolvedSecretTraceScopeV1
): ResolvedSecretTraceRegistry {
const registry = new ResolvedSecretTraceRegistry([], scope)
registry.markIncomplete()
return registry
}
@@ -9,6 +9,7 @@ import {
import { BlockType } from '@/executor/constants'
import { ExecutionState } from '@/executor/execution/state'
import type { ExecutionContext } from '@/executor/types'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import { VariableResolver } from '@/executor/variables/resolver'
import { navigatePathAsync } from '@/executor/variables/resolvers/reference-async.server'
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
@@ -114,6 +115,27 @@ function createResolver(
}
describe('VariableResolver function block inputs', () => {
it('records a secret reached through workflow-variable indirection', async () => {
const { ctx, resolver } = createResolver()
const registry = new ResolvedSecretTraceRegistry([
{ name: 'TOKEN', plaintext: 'resolved-secret', encryptedValue: 'ciphertext' },
])
ctx.workflowVariables = {
'var-1': { id: 'var-1', name: 'indirect', type: 'string', value: '{{TOKEN}}' },
}
ctx.environmentVariables = { TOKEN: 'resolved-secret' }
ctx.resolvedSecretTraceRegistry = registry
const result = await resolver.resolveInputs(ctx, 'function', {
value: '<variable.indirect>',
})
expect(result.value).toBe('resolved-secret')
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'resolved-secret', replacement: '{{TOKEN}}' },
])
})
it('returns empty inputs when params are missing', async () => {
const { block, ctx, resolver } = createResolver()
@@ -1,4 +1,5 @@
import { describe, expect, it } from 'vitest'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
import { EnvResolver } from './env'
import type { ResolutionContext } from './reference'
@@ -48,6 +49,23 @@ describe('EnvResolver', () => {
})
describe('resolve', () => {
it('records only successful Secrets-tab substitutions', () => {
const resolver = new EnvResolver()
const registry = new ResolvedSecretTraceRegistry([
{ name: 'API_KEY', plaintext: 'secret-api-key', encryptedValue: 'ciphertext' },
])
const ctx = createTestContext({ API_KEY: 'secret-api-key' })
ctx.executionContext.resolvedSecretTraceRegistry = registry
expect(resolver.resolve('{{MISSING}}', ctx)).toBe('{{MISSING}}')
expect(registry.getActiveMatches()).toEqual([])
expect(resolver.resolve('{{API_KEY}}', ctx)).toBe('secret-api-key')
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'secret-api-key', replacement: '{{API_KEY}}' },
])
})
it.concurrent('should resolve existing environment variable', () => {
const resolver = new EnvResolver()
const ctx = createTestContext({ API_KEY: 'secret-api-key' })
@@ -16,6 +16,9 @@ export class EnvResolver implements Resolver {
if (value === undefined) {
return reference
}
if (Object.hasOwn(context.executionContext.environmentVariables, varName)) {
context.executionContext.resolvedSecretTraceRegistry?.recordResolved(varName, value)
}
return value
}
}
+12 -2
View File
@@ -375,12 +375,22 @@ export function useCancelExecution(workspaceId: string) {
export function useRetryExecution() {
const queryClient = useQueryClient()
return useMutation({
mutationFn: async ({ workflowId, input }: { workflowId: string; input?: unknown }) => {
mutationFn: async ({
workflowId,
executionId,
}: {
workflowId: string
executionId: string
}) => {
// boundary-raw-fetch: stream response, body is a ReadableStream consumed one chunk at a time
const res = await fetch(`/api/workflows/${workflowId}/execute`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ input, triggerType: 'manual', stream: true }),
body: JSON.stringify({
inputFromExecutionId: executionId,
triggerType: 'manual',
stream: true,
}),
})
if (!res.ok) {
const data = await res.json().catch(() => ({}))
+8 -2
View File
@@ -210,7 +210,8 @@ export interface ExecuteStreamOptions {
export interface ExecuteFromBlockOptions {
workflowId: string
startBlockId: string
sourceSnapshot: SerializableExecutionState
sourceSnapshot?: SerializableExecutionState
sourceExecutionId?: string
input?: any
onExecutionId?: (executionId: string) => void
callbacks?: ExecutionStreamCallbacks
@@ -348,6 +349,7 @@ export function useExecutionStream() {
workflowId,
startBlockId,
sourceSnapshot,
sourceExecutionId,
input,
onExecutionId,
callbacks = {},
@@ -370,7 +372,11 @@ export function useExecutionStream() {
body: JSON.stringify({
stream: true,
input,
runFromBlock: { startBlockId, sourceSnapshot },
runFromBlock: {
startBlockId,
...(sourceExecutionId ? { executionId: sourceExecutionId } : {}),
...(sourceSnapshot ? { sourceSnapshot } : {}),
},
}),
signal: abortController.signal,
})
@@ -28,6 +28,12 @@ describe('workflow contracts', () => {
})
})
it('accepts a trusted prior-execution input reference', () => {
expect(
executeWorkflowBodySchema.parse({ inputFromExecutionId: 'execution-123' })
).toMatchObject({ inputFromExecutionId: 'execution-123' })
})
it('normalizes null React Flow edge handles in execution overrides', () => {
const parsed = executeWorkflowBodySchema.parse({
workflowStateOverride: {
+2
View File
@@ -360,6 +360,8 @@ export const executeWorkflowBodySchema = z.object({
includeToolCalls: z.boolean().optional().default(false),
useDraftState: z.boolean().optional(),
input: z.any().optional(),
/** Trusted server-side reuse of a prior execution's raw workflow input. */
inputFromExecutionId: executionIdSchema.optional(),
isClientSession: z.boolean().optional(),
includeFileBase64: z.boolean().optional().default(true),
base64MaxBytes: z.number().int().positive().optional(),
+119 -8
View File
@@ -11,7 +11,7 @@ const { discoverServerTools, validateMcpToolsAllowed } = vi.hoisted(() => ({
vi.mock('@/lib/mcp/service', () => ({ mcpService: { discoverServerTools } }))
vi.mock('@/ee/access-control/utils/permission-check', () => ({ validateMcpToolsAllowed }))
import { buildSelectedMcpToolSchemas, buildTaggedMcpToolSchemas } from './mcp-tools'
import { buildSelectedMcpToolSchemas, buildTaggedMcpToolSchemas } from '@/lib/copilot/mcp-tools'
describe('mothership MCP tool schemas', () => {
beforeEach(() => {
@@ -50,19 +50,130 @@ describe('mothership MCP tool schemas', () => {
])
})
it('reports tagged-server discovery provenance without placing it in tool schemas', async () => {
const provenance = {
version: 1,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: 'user-1', workspaceId: 'ws-1' },
}
discoverServerTools.mockImplementationOnce(
async (
_userId: string,
_serverId: string,
_workspaceId: string,
_forceRefresh: boolean,
report: (value: unknown) => void
) => {
report(provenance)
return [
{
serverId: 'mcp-server-1',
name: 'search',
description: 'Search docs',
inputSchema: { type: 'object' },
},
]
}
)
const recordProvenance = vi.fn()
const tools = await buildTaggedMcpToolSchemas(
'user-1',
'ws-1',
['mcp-server-1'],
recordProvenance
)
expect(discoverServerTools).toHaveBeenCalledWith(
'user-1',
'mcp-server-1',
'ws-1',
false,
expect.any(Function)
)
expect(recordProvenance).toHaveBeenCalledWith(provenance)
expect(JSON.stringify(tools)).not.toContain('encrypted-token')
expect(JSON.stringify(tools)).not.toContain('resolvedSecretTraceProvenance')
})
it('reports incomplete provenance when tagged-server discovery returns no report', async () => {
discoverServerTools.mockResolvedValue([])
const recordProvenance = vi.fn()
await buildTaggedMcpToolSchemas('user-1', 'ws-1', ['mcp-server-1'], recordProvenance)
expect(recordProvenance).toHaveBeenCalledWith({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'ws-1' },
})
})
it('uses a selected block tool cached schema without discovering the server', async () => {
const tools = await buildSelectedMcpToolSchemas('user-1', 'ws-1', [
{
type: 'mcp',
params: { serverId: 'mcp-server-1', toolName: 'search', serverName: 'Docs' },
schema: { type: 'object', properties: { query: { type: 'string' } } },
},
])
const recordProvenance = vi.fn()
const tools = await buildSelectedMcpToolSchemas(
'user-1',
'ws-1',
[
{
type: 'mcp',
params: { serverId: 'mcp-server-1', toolName: 'search', serverName: 'Docs' },
schema: { type: 'object', properties: { query: { type: 'string' } } },
},
],
recordProvenance
)
expect(discoverServerTools).not.toHaveBeenCalled()
expect(recordProvenance).not.toHaveBeenCalled()
expect(tools[0]).toMatchObject({
name: 'mcp-server-1-search',
input_schema: { type: 'object', properties: { query: { type: 'string' } } },
})
})
it('reports provenance from selected tools that require server discovery', async () => {
const provenance = {
version: 1,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: 'user-1', workspaceId: 'ws-1' },
}
discoverServerTools.mockImplementationOnce(
async (
_userId: string,
_serverId: string,
_workspaceId: string,
_forceRefresh: boolean,
report: (value: unknown) => void
) => {
report(provenance)
return [
{
serverId: 'mcp-server-1',
name: 'search',
inputSchema: { type: 'object' },
},
]
}
)
const recordProvenance = vi.fn()
const tools = await buildSelectedMcpToolSchemas(
'user-1',
'ws-1',
[
{
type: 'mcp',
params: { serverId: 'mcp-server-1', toolName: 'search' },
},
],
recordProvenance
)
expect(recordProvenance).toHaveBeenCalledWith(provenance)
expect(tools[0]).toMatchObject({ name: 'mcp-server-1-search' })
})
})
+41 -6
View File
@@ -8,6 +8,8 @@ import type { ToolInput } from '@/executor/handlers/agent/types'
const logger = createLogger('CopilotMcpTools')
type ResolvedSecretTraceProvenanceCallback = (provenance: unknown) => void
function toMothershipMcpTool(tool: {
serverId: string
serverName?: string
@@ -51,11 +53,26 @@ function dedupeMcpTools(tools: ToolSchema[]): ToolSchema[] {
async function discoverServerTools(
userId: string,
workspaceId: string,
serverId: string
serverId: string,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<McpTool[]> {
let provenanceReported = false
try {
const { mcpService } = await import('@/lib/mcp/service')
return await mcpService.discoverServerTools(userId, serverId, workspaceId)
if (!onResolvedSecretTraceProvenance) {
return await mcpService.discoverServerTools(userId, serverId, workspaceId)
}
return await mcpService.discoverServerTools(
userId,
serverId,
workspaceId,
false,
(provenance) => {
provenanceReported = true
onResolvedSecretTraceProvenance(provenance)
}
)
} catch (error) {
logger.warn('Failed to resolve tagged MCP server tools', {
serverId,
@@ -63,6 +80,15 @@ async function discoverServerTools(
error: toError(error).message,
})
return []
} finally {
if (onResolvedSecretTraceProvenance && !provenanceReported) {
onResolvedSecretTraceProvenance({
version: 1,
complete: false,
entries: [],
scope: { userId, workspaceId },
})
}
}
}
@@ -73,14 +99,17 @@ async function discoverServerTools(
export async function buildTaggedMcpToolSchemas(
userId: string,
workspaceId: string,
serverIds: string[]
serverIds: string[],
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<ToolSchema[]> {
const uniqueServerIds = [...new Set(serverIds.filter(Boolean))]
if (uniqueServerIds.length === 0) return []
await validateMcpToolsAllowed(userId, workspaceId)
const discovered = await Promise.all(
uniqueServerIds.map((serverId) => discoverServerTools(userId, workspaceId, serverId))
uniqueServerIds.map((serverId) =>
discoverServerTools(userId, workspaceId, serverId, onResolvedSecretTraceProvenance)
)
)
return dedupeMcpTools(discovered.flat().map(toMothershipMcpTool))
}
@@ -93,7 +122,8 @@ export async function buildTaggedMcpToolSchemas(
export async function buildSelectedMcpToolSchemas(
userId: string,
workspaceId: string,
selections: ToolInput[]
selections: ToolInput[],
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<ToolSchema[]> {
const selected = selections.filter(
(tool) =>
@@ -130,7 +160,12 @@ export async function buildSelectedMcpToolSchemas(
let discovery = discoveredByServer.get(serverId)
if (!discovery) {
discovery = discoverServerTools(userId, workspaceId, serverId)
discovery = discoverServerTools(
userId,
workspaceId,
serverId,
onResolvedSecretTraceProvenance
)
discoveredByServer.set(serverId, discovery)
}
const match = (await discovery).find((tool) => tool.name === toolName)
@@ -10,6 +10,7 @@ import {
} from '@sim/testing'
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
import type { ExecutionContext, StreamingContext } from '@/lib/copilot/request/types'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
const mockGetEffectiveDecryptedEnv = environmentUtilsMockFns.mockGetEffectiveDecryptedEnv
@@ -148,6 +149,44 @@ describe('runCopilotLifecycle', () => {
mockGetMothershipSourceEnvHeaders.mockReturnValue({})
})
it('threads trace provenance through server execution context only', async () => {
const registry = {} as ResolvedSecretTraceRegistry
const executionContext: ExecutionContext = {
userId: 'user-1',
workflowId: '',
workspaceId: 'ws-1',
decryptedEnvVars: {},
}
let capturedExecutionContext: ExecutionContext | undefined
let capturedRequestBody = ''
mockRunStreamLoop.mockImplementationOnce(
async (
_url: string,
request: RequestInit,
_streamingContext: StreamingContext,
context: ExecutionContext
) => {
capturedExecutionContext = context
capturedRequestBody = String(request.body)
}
)
await runCopilotLifecycle(
{ message: 'hello', messageId: 'stream-private-context' },
{
userId: 'user-1',
workspaceId: 'ws-1',
executionContext,
resolvedSecretTraceRegistry: registry,
}
)
expect(capturedExecutionContext?.resolvedSecretTraceRegistry).toBe(registry)
expect(capturedRequestBody).not.toContain('resolvedSecretTraceRegistry')
expect(capturedRequestBody).not.toContain('resolved-secret-provenance')
expect(executionContext).not.toHaveProperty('resolvedSecretTraceRegistry')
})
describe('tool permission feature flag', () => {
const runMothershipTurn = () =>
runCopilotLifecycle(
@@ -62,6 +62,7 @@ import {
isHosted,
} from '@/lib/core/config/env-flags'
import { getEffectiveDecryptedEnv } from '@/lib/environment/utils'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
const logger = createLogger('CopilotLifecycle')
@@ -95,6 +96,7 @@ export interface CopilotLifecycleOptions extends OrchestratorOptions {
onGoTraceId?: (goTraceId: string) => void
executionContext?: ExecutionContext
billingAttribution?: BillingAttributionSnapshot
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
}
/**
@@ -161,6 +163,9 @@ export async function runCopilotLifecycle(
abortSignal: options.abortSignal,
billingAttribution:
options.billingAttribution ?? options.executionContext.billingAttribution,
...(options.resolvedSecretTraceRegistry
? { resolvedSecretTraceRegistry: options.resolvedSecretTraceRegistry }
: {}),
},
}
: {}),
@@ -177,6 +182,7 @@ export async function runCopilotLifecycle(
runId: resolvedRunId,
abortSignal: lifecycleOptions.abortSignal,
billingAttribution: lifecycleOptions.billingAttribution,
resolvedSecretTraceRegistry: lifecycleOptions.resolvedSecretTraceRegistry,
}))
const shouldUseHostedBillingProtocol = isHosted && isCopilotBillingAttributionV1Enabled
if (
@@ -993,6 +999,7 @@ async function buildExecutionContext(
runId?: string
abortSignal?: AbortSignal
billingAttribution?: BillingAttributionSnapshot
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
}
): Promise<ExecutionContext> {
const {
@@ -1004,6 +1011,7 @@ async function buildExecutionContext(
runId,
abortSignal,
billingAttribution,
resolvedSecretTraceRegistry,
} = params
const userTimezone =
typeof requestPayload?.userTimezone === 'string' ? requestPayload.userTimezone : undefined
@@ -1039,6 +1047,9 @@ async function buildExecutionContext(
execContext.runId = runId
execContext.abortSignal = abortSignal
if (billingAttribution) execContext.billingAttribution = billingAttribution
if (resolvedSecretTraceRegistry) {
execContext.resolvedSecretTraceRegistry = resolvedSecretTraceRegistry
}
return execContext
}
@@ -4,6 +4,7 @@
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { DEFAULT_EXECUTION_TIMEOUT_MS } from '@/lib/execution/constants'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
const { isKnownTool, isSimExecuted, isClientExecuted } = vi.hoisted(() => ({
isKnownTool: vi.fn(),
@@ -111,6 +112,35 @@ describe('copilot tool executor fallback', () => {
expect(appParams._context).not.toHaveProperty('billingAttribution')
})
it('passes trace provenance out-of-band without exposing it in app tool parameters', async () => {
isKnownTool.mockReturnValue(false)
isSimExecuted.mockReturnValue(false)
executeAppTool.mockResolvedValue({ success: true, output: { result: 'unchanged' } })
const registry = {} as ResolvedSecretTraceRegistry
const result = await executeTool(
'gmail_read',
{ query: 'hello' },
{
userId: 'user-1',
workflowId: 'workflow-1',
resolvedSecretTraceRegistry: registry,
}
)
expect(executeAppTool).toHaveBeenCalledWith(
'gmail_read',
expect.objectContaining({
query: 'hello',
_context: expect.not.objectContaining({ resolvedSecretTraceRegistry: expect.anything() }),
}),
{ resolvedSecretTraceRegistry: registry }
)
const appParams = executeAppTool.mock.calls[0]?.[1]
expect(JSON.stringify(appParams)).not.toContain('resolvedSecretTraceRegistry')
expect(result).toEqual({ success: true, output: { result: 'unchanged' } })
})
it('uses the registered handler for client-routed tools when running headless (Mothership block)', async () => {
isKnownTool.mockReturnValue(true)
isSimExecuted.mockReturnValue(false)
@@ -55,7 +55,11 @@ export async function executeTool(
(isSimExecuted(toolId) || (isClientExecuted(toolId) && hasHandler(toolId)))
if (!canUseRegisteredHandler) {
const appParams = buildAppToolParams(normalizedParams, context)
return executeAppTool(toolId, appParams)
return context.resolvedSecretTraceRegistry
? executeAppTool(toolId, appParams, {
resolvedSecretTraceRegistry: context.resolvedSecretTraceRegistry,
})
: executeAppTool(toolId, appParams)
}
if (context.abortSignal?.aborted) {
@@ -1,5 +1,6 @@
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
import type { MothershipResource } from '@/lib/copilot/resources/types'
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
export interface ToolExecutionContext {
userId: string
@@ -24,6 +25,7 @@ export interface ToolExecutionContext {
userTimezone?: string
userPermission?: string
decryptedEnvVars?: Record<string, string>
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
}
export interface ToolExecutionResult {
@@ -89,6 +89,32 @@ function mountedFiles() {
const snapshotCacheOn = (flag: string) => Promise.resolve(flag === 'table-snapshot-cache')
describe('executeFunctionExecute trace-secret provenance', () => {
beforeEach(() => {
vi.clearAllMocks()
mockExecuteTool.mockResolvedValue({ success: true })
})
it('forwards the registry only through server execution options', async () => {
const resolvedSecretTraceRegistry = { recordResolved: vi.fn() }
await executeFunctionExecute({ code: 'return {{API_KEY}}' }, {
userId: 'u1',
workspaceId: 'ws_1',
resolvedSecretTraceRegistry,
} as never)
expect(mockExecuteTool).toHaveBeenCalledWith(
'function_execute',
expect.not.objectContaining({ resolvedSecretTraceRegistry: expect.anything() }),
{ resolvedSecretTraceRegistry }
)
const appParams = mockExecuteTool.mock.calls[0]?.[1] as Record<string, unknown>
expect(appParams._context).not.toHaveProperty('resolvedSecretTraceRegistry')
expect(JSON.stringify(appParams)).not.toContain('resolvedSecretTraceRegistry')
})
})
describe('executeFunctionExecute table mounts', () => {
beforeEach(() => {
vi.clearAllMocks()
@@ -509,5 +509,9 @@ export async function executeFunctionExecute(
enforceCredentialAccess: true,
}
return executeAppTool('function_execute', enrichedParams)
return context.resolvedSecretTraceRegistry
? executeAppTool('function_execute', enrichedParams, {
resolvedSecretTraceRegistry: context.resolvedSecretTraceRegistry,
})
: executeAppTool('function_execute', enrichedParams)
}
@@ -23,6 +23,10 @@ afterAll(() => {
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
import type { ExecutionContext } from '@/lib/copilot/request/types'
import {
ANONYMOUS_SECRET_TRACE_REPLACEMENT,
ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
const {
ensureWorkflowAccessMock,
@@ -39,6 +43,7 @@ const {
checkAttributedUsageLimitsMock,
reserveExecutionSlotMock,
releaseExecutionSlotMock,
decryptSecretMock,
} = vi.hoisted(() => ({
ensureWorkflowAccessMock: vi.fn(),
ensureWorkspaceAccessMock: vi.fn(),
@@ -54,6 +59,7 @@ const {
checkAttributedUsageLimitsMock: vi.fn(),
reserveExecutionSlotMock: vi.fn(),
releaseExecutionSlotMock: vi.fn(),
decryptSecretMock: vi.fn(),
}))
vi.mock('@sim/audit', () => ({
@@ -79,6 +85,11 @@ vi.mock('@/lib/billing/calculations/usage-reservation', () => ({
UsageReservationUnavailableError: class UsageReservationUnavailableError extends Error {},
}))
vi.mock('@/lib/core/security/encryption', () => ({
decryptSecret: decryptSecretMock,
encryptSecret: vi.fn(),
}))
vi.mock('@/lib/workflows/executor/execute-workflow', () => ({
executeWorkflow: executeWorkflowMock,
}))
@@ -297,6 +308,7 @@ describe('executeCreateWorkflow billing attribution', () => {
payerUsage: { currentUsage: 1, limit: 10 },
})
reserveExecutionSlotMock.mockResolvedValue({ reserved: true, created: true })
decryptSecretMock.mockResolvedValue({ decrypted: 'secret-value' })
listFoldersMock.mockResolvedValue([])
})
@@ -580,6 +592,7 @@ describe('Copilot workflow execution billing attribution', () => {
payerUsage: { currentUsage: 1, limit: 10 },
})
reserveExecutionSlotMock.mockResolvedValue({ reserved: true, created: true })
decryptSecretMock.mockResolvedValue({ decrypted: 'secret-value' })
})
async function expectBillingAttributionForwarded(
@@ -602,6 +615,178 @@ describe('Copilot workflow execution billing attribution', () => {
)
})
it('passes only input-crossing parent provenance to the child execution', async () => {
const registry = new ResolvedSecretTraceRegistry(
[
{
name: 'INPUT_SECRET',
plaintext: 'input-secret',
encryptedValue: 'input-ciphertext',
},
{
name: 'UNRELATED_SECRET',
plaintext: 'unrelated-secret',
encryptedValue: 'unrelated-ciphertext',
},
],
{ userId: 'user-1', workspaceId: 'workspace-1' }
)
registry.recordResolved('INPUT_SECRET', 'input-secret')
registry.recordResolved('UNRELATED_SECRET', 'unrelated-secret')
resolveTriggerRunOptionsMock.mockReturnValueOnce([
{
triggerBlockId: 'trigger-1',
blockName: 'Start',
mockPayload: { value: 'input-secret' },
},
])
executeWorkflowMock.mockResolvedValueOnce({
success: true,
output: { ok: true },
logs: [],
metadata: { executionId: 'new-execution-1' },
executionState: {
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
},
},
})
const result = await executeRunWorkflow(
{ workflowId: 'workflow-1', useMockPayload: true },
{ ...executionContext, resolvedSecretTraceRegistry: registry }
)
expect(result.success).toBe(true)
expect(executeWorkflowMock.mock.calls[0]?.[2]).toEqual({ value: 'input-secret' })
expect(executeWorkflowMock.mock.calls[0]?.[4]).toEqual(
expect.objectContaining({
trustedInitialResolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [{ name: 'INPUT_SECRET', encryptedValue: 'input-ciphertext' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
},
})
)
expect(JSON.stringify(result)).not.toContain('input-ciphertext')
expect(JSON.stringify(result)).not.toContain('unrelated-ciphertext')
})
it('imports child provenance without returning private metadata to the model', async () => {
const registry = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
const context: ExecutionContext = {
...executionContext,
resolvedSecretTraceRegistry: registry,
}
executeWorkflowMock.mockResolvedValueOnce({
success: true,
output: { value: 'secret-value' },
logs: [],
metadata: { executionId: 'new-execution-1' },
executionState: {
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
},
},
})
const result = await executeRunWorkflow(
{ workflowId: 'workflow-1', useMockPayload: true },
context
)
expect(result).toMatchObject({
success: true,
output: { output: { value: 'secret-value' } },
})
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'secret-value', replacement: '{{API_KEY}}' },
])
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
expect(JSON.stringify(result)).not.toContain('encrypted-secret')
})
it('marks provenance incomplete when child execution returns no trusted state', async () => {
const registry = new ResolvedSecretTraceRegistry()
const context: ExecutionContext = {
...executionContext,
resolvedSecretTraceRegistry: registry,
}
const result = await executeRunWorkflow(
{ workflowId: 'workflow-1', useMockPayload: true },
context
)
expect(result.success).toBe(true)
expect(registry.isComplete()).toBe(false)
})
it('filters and anonymizes cross-workspace child provenance to values that cross back', async () => {
const registry = new ResolvedSecretTraceRegistry([], {
userId: 'user-1',
workspaceId: 'workspace-1',
})
const context: ExecutionContext = {
...executionContext,
resolvedSecretTraceRegistry: registry,
}
ensureWorkflowAccessMock.mockResolvedValueOnce({
workflow: {
id: 'workflow-2',
userId: 'owner-2',
workspaceId: 'workspace-2',
variables: {},
},
})
resolveBillingAttributionMock.mockResolvedValueOnce(childBillingAttribution)
decryptSecretMock.mockImplementation(async (encryptedValue: string) => ({
decrypted: encryptedValue === 'used-ciphertext' ? 'used-secret' : 'workspace-only-secret',
}))
executeWorkflowMock.mockResolvedValueOnce({
success: true,
output: { value: 'used-secret' },
logs: [],
metadata: { executionId: 'child-execution' },
executionState: {
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [
{ name: 'USED', encryptedValue: 'used-ciphertext' },
{ name: 'WORKSPACE_ONLY', encryptedValue: 'workspace-only-ciphertext' },
],
scope: { userId: 'user-1', workspaceId: 'workspace-2' },
},
},
})
const result = await executeRunWorkflow(
{ workflowId: 'workflow-2', useMockPayload: true },
context
)
expect(result).toMatchObject({
success: true,
output: { output: { value: 'used-secret' } },
})
expect(registry.getActiveMatches()).toEqual([
{ plaintext: 'used-secret', replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT },
])
expect(JSON.stringify(result)).not.toContain('workspace-only-ciphertext')
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
})
it('passes immutable attribution when running until a block', async () => {
await expectBillingAttributionForwarded(() =>
executeRunWorkflowUntilBlock(
@@ -97,7 +97,9 @@ async function executeCopilotWorkflowTarget(params: {
)
try {
return await executeWorkflow(
const trustedInitialResolvedSecretTraceProvenance =
params.context.resolvedSecretTraceRegistry?.exportProvenanceForValue(params.input)
const result = await executeWorkflow(
params.workflow,
generateRequestId(),
params.input,
@@ -105,10 +107,34 @@ async function executeCopilotWorkflowTarget(params: {
{
...params.options,
billingAttribution: admission.billingAttribution,
...(trustedInitialResolvedSecretTraceProvenance
? { trustedInitialResolvedSecretTraceProvenance }
: {}),
},
childExecutionId
)
if (params.context.resolvedSecretTraceRegistry) {
await params.context.resolvedSecretTraceRegistry.importCrossingProvenance(
result.executionState?.resolvedSecretTraceProvenance,
{ output: result.output, logs: result.logs, error: result.error },
{ trusted: true }
)
}
return result
} catch (error) {
if (params.context.resolvedSecretTraceRegistry) {
const executionResult = hasExecutionResult(error) ? error.executionResult : undefined
await params.context.resolvedSecretTraceRegistry.importCrossingProvenance(
executionResult?.executionState?.resolvedSecretTraceProvenance,
{
output: executionResult?.output,
logs: executionResult?.logs,
error: executionResult?.error,
thrownMessage: toError(error).message,
},
{ trusted: true }
)
}
if (admission.targetReservation) {
await releaseExecutionSlot(childExecutionId)
}
@@ -0,0 +1,131 @@
/**
* @vitest-environment node
*/
import { dbChainMockFns, resetDbChainMock } from '@sim/testing'
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
const { checkWorkspaceAccessMock, materializeExecutionDataForDisplayMock } = vi.hoisted(() => ({
checkWorkspaceAccessMock: vi.fn(),
materializeExecutionDataForDisplayMock: vi.fn(),
}))
vi.mock('@/lib/workspaces/permissions/utils', () => ({
checkWorkspaceAccess: checkWorkspaceAccessMock,
}))
vi.mock('@/lib/logs/execution/trace-store', () => ({
materializeExecutionDataForDisplay: materializeExecutionDataForDisplayMock,
}))
import { getJobLogsServerTool } from './get-job-logs'
const RAW_SECRET = 'sk-job-log-secret'
const MASKED_SECRET = '{{OPENAI_API_KEY}}'
const CONTEXT = { userId: 'user-1', workspaceId: 'workspace-1' }
function jobLogRow(overrides: Record<string, unknown> = {}) {
return {
id: 'log-1',
executionId: 'execution-1',
status: 'success',
level: 'info',
trigger: 'schedule',
startedAt: new Date('2026-07-31T00:00:00.000Z'),
endedAt: new Date('2026-07-31T00:00:01.000Z'),
totalDurationMs: 1000,
executionData: {
finalOutput: { result: RAW_SECRET },
traceSpans: [{ id: 'span-1', output: { result: RAW_SECRET } }],
},
cost: null,
...overrides,
}
}
describe('getJobLogsServerTool', () => {
afterAll(resetDbChainMock)
beforeEach(() => {
vi.clearAllMocks()
resetDbChainMock()
checkWorkspaceAccessMock.mockResolvedValue({ hasAccess: true })
})
it('returns the secret-safe log projection instead of raw successful output', async () => {
const row = jobLogRow()
dbChainMockFns.limit.mockResolvedValueOnce([row])
materializeExecutionDataForDisplayMock.mockResolvedValueOnce({
finalOutput: { result: MASKED_SECRET },
traceSpans: [
{
id: 'span-1',
name: 'Function 1',
type: 'function',
status: 'success',
duration: 1000,
startTime: '2026-07-31T00:00:00.000Z',
endTime: '2026-07-31T00:00:01.000Z',
output: { result: MASKED_SECRET },
},
],
})
const result = await getJobLogsServerTool.execute(
{ jobId: 'schedule-1', includeDetails: true },
CONTEXT
)
expect(result).toEqual([
expect.objectContaining({
executionId: 'execution-1',
output: { result: MASKED_SECRET },
}),
])
expect(JSON.stringify(result)).not.toContain(RAW_SECRET)
expect(materializeExecutionDataForDisplayMock).toHaveBeenCalledWith(row.executionData, {
workspaceId: 'workspace-1',
workflowId: null,
executionId: 'execution-1',
userId: 'user-1',
})
})
it('does not fall back to raw output or errors when provenance is incomplete', async () => {
dbChainMockFns.limit.mockResolvedValueOnce([
jobLogRow({
status: 'error',
executionData: {
finalOutput: { error: RAW_SECRET },
error: RAW_SECRET,
traceSpans: [{ id: 'span-1', output: { error: RAW_SECRET } }],
},
}),
])
materializeExecutionDataForDisplayMock.mockResolvedValueOnce({
traceSpans: [
{
id: 'span-1',
name: 'Function 1',
type: 'function',
status: 'error',
duration: 1000,
startTime: '2026-07-31T00:00:00.000Z',
endTime: '2026-07-31T00:00:01.000Z',
},
],
})
const result = await getJobLogsServerTool.execute(
{ jobId: 'schedule-1', includeDetails: true },
CONTEXT
)
expect(result).toEqual([
expect.not.objectContaining({
output: expect.anything(),
error: expect.anything(),
}),
])
expect(JSON.stringify(result)).not.toContain(RAW_SECRET)
})
})
@@ -4,6 +4,7 @@ import { createLogger } from '@sim/logger'
import { and, desc, eq } from 'drizzle-orm'
import { GetScheduledTaskLogs } from '@/lib/copilot/generated/tool-catalog-v1'
import type { BaseServerTool, ServerToolContext } from '@/lib/copilot/tools/server/base-tool'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
import type { TraceSpan } from '@/lib/logs/types'
import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils'
@@ -188,34 +189,47 @@ export const getJobLogsServerTool: BaseServerTool<GetJobLogsArgs, JobLogEntry[]>
.orderBy(desc(jobExecutionLogs.startedAt))
.limit(executionId ? 1 : clampedLimit)
const entries: JobLogEntry[] = rows.map((row) => {
const executionData = row.executionData as any
const details = includeDetails ? extractOutputAndError(executionData) : null
const entries: JobLogEntry[] = await Promise.all(
rows.map(async (row) => {
const executionData = await materializeExecutionDataForDisplay(
row.executionData as Record<string, unknown> | null,
{
workspaceId: wsId,
workflowId: null,
executionId: row.executionId,
userId: context.userId,
}
)
const details = includeDetails ? extractOutputAndError(executionData) : null
const entry: JobLogEntry = {
executionId: row.executionId,
status: row.status,
trigger: row.trigger,
startedAt: row.startedAt.toISOString(),
endedAt: row.endedAt ? row.endedAt.toISOString() : null,
durationMs: row.totalDurationMs ?? null,
}
if (details) {
if (details.error) entry.error = details.error
if (details.toolCalls.length > 0) entry.toolCalls = details.toolCalls
if (details.output) entry.output = details.output
if (details.cost) entry.cost = details.cost
if (details.tokens) entry.tokens = details.tokens
} else {
const errorMsg = executionData?.error || executionData?.traceSpans?.[0]?.output?.error
if (row.status === 'error' && errorMsg) {
entry.error = typeof errorMsg === 'string' ? errorMsg : JSON.stringify(errorMsg)
const entry: JobLogEntry = {
executionId: row.executionId,
status: row.status,
trigger: row.trigger,
startedAt: row.startedAt.toISOString(),
endedAt: row.endedAt ? row.endedAt.toISOString() : null,
durationMs: row.totalDurationMs ?? null,
}
}
return entry
})
if (details) {
if (details.error) entry.error = details.error
if (details.toolCalls.length > 0) entry.toolCalls = details.toolCalls
if (details.output) entry.output = details.output
if (details.cost) entry.cost = details.cost
if (details.tokens) entry.tokens = details.tokens
} else {
const traceSpans = Array.isArray(executionData.traceSpans)
? (executionData.traceSpans as TraceSpan[])
: []
const errorMsg = executionData.error || traceSpans[0]?.output?.error
if (row.status === 'error' && errorMsg) {
entry.error = typeof errorMsg === 'string' ? errorMsg : JSON.stringify(errorMsg)
}
}
return entry
})
)
logger.info('Job logs prepared', {
jobId,
+1 -54
View File
@@ -18,7 +18,6 @@
import { context, type Span, SpanStatusCode, trace } from '@opentelemetry/api'
import { createLogger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import { TraceAttr } from '@/lib/copilot/generated/trace-attributes-v1'
import type { TraceSpan } from '@/lib/logs/types'
import { hostedKeyMetrics } from '@/lib/monitoring/metrics'
@@ -335,7 +334,6 @@ export function createOTelSpansForWorkflowExecution(params: {
endTime: string
totalDurationMs: number
status: 'success' | 'error'
error?: string
}): void {
try {
const tracer = getTracer()
@@ -358,11 +356,8 @@ export function createOTelSpansForWorkflowExecution(params: {
if (params.status === 'error') {
rootSpan.setStatus({
code: SpanStatusCode.ERROR,
message: params.error || 'Workflow execution failed',
message: 'Workflow execution failed',
})
if (params.error) {
rootSpan.recordException(new Error(params.error))
}
} else {
rootSpan.setStatus({ code: SpanStatusCode.OK })
}
@@ -387,52 +382,6 @@ export function createOTelSpansForWorkflowExecution(params: {
}
}
/**
* Create a real-time OpenTelemetry span for a block execution
* Can be called from block handlers during execution for real-time tracing
*/
export async function traceBlockExecution<T>(
blockType: string,
blockId: string,
blockName: string,
fn: (span: Span) => Promise<T>
): Promise<T> {
const tracer = getTracer()
const blockMapping = BLOCK_TYPE_MAPPING[blockType] || {
spanName: `block.${blockType}`,
spanKind: 'internal',
getAttributes: () => ({}),
}
return tracer.startActiveSpan(
blockMapping.spanName,
{
attributes: {
[TraceAttr.BlockType]: blockType,
[TraceAttr.BlockId]: blockId,
[TraceAttr.BlockName]: blockName,
},
},
async (span) => {
try {
const result = await fn(span)
span.setStatus({ code: SpanStatusCode.OK })
return result
} catch (error) {
span.setStatus({
code: SpanStatusCode.ERROR,
message: getErrorMessage(error, 'Block execution failed'),
})
span.recordException(toError(error))
throw error
} finally {
span.end()
}
}
)
}
/**
* Track platform events (workflow creation, knowledge base operations, etc.)
*/
@@ -672,7 +621,6 @@ export const PlatformEvents = {
blocksExecuted: number
hasErrors: boolean
totalCost?: number
errorMessage?: string
}) => {
trackPlatformEvent('platform.workflow.executed', {
'workflow.id': attrs.workflowId,
@@ -682,7 +630,6 @@ export const PlatformEvents = {
'execution.blocks_executed': attrs.blocksExecuted,
'execution.has_errors': attrs.hasErrors,
...(attrs.totalCost !== undefined && { 'execution.total_cost': attrs.totalCost }),
...(attrs.errorMessage && { 'execution.error_message': attrs.errorMessage }),
})
},
@@ -1,6 +1,7 @@
import { dbReplica } from '@sim/db'
import { jobExecutionLogs } from '@sim/db/schema'
import { and, inArray, isNotNull } from 'drizzle-orm'
import { MATERIALIZE_CONCURRENCY, mapWithConcurrency } from '@/lib/core/utils/concurrency'
import {
decodeTimeCursor,
encodeTimeCursor,
@@ -10,6 +11,7 @@ import {
} from '@/lib/data-drains/sources/cursor'
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
type JobLogRow = typeof jobExecutionLogs.$inferSelect
@@ -40,6 +42,16 @@ async function* pages(input: SourcePageInput): AsyncIterable<JobLogRow[]> {
.limit(input.chunkSize)
if (rows.length === 0) return
const displayExecutionData = await mapWithConcurrency(rows, MATERIALIZE_CONCURRENCY, (row) =>
materializeExecutionDataForDisplay(row.executionData as Record<string, unknown> | null, {
workspaceId: row.workspaceId,
workflowId: null,
executionId: row.executionId,
})
)
for (let index = 0; index < rows.length; index += 1) {
rows[index].executionData = displayExecutionData[index] as JobLogRow['executionData']
}
yield rows
const last = rows[rows.length - 1]
cursor = { ts: last.endedAt!.toISOString(), id: last.id }
@@ -11,7 +11,7 @@ import {
} from '@/lib/data-drains/sources/cursor'
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
type WorkflowLogRow = typeof workflowExecutionLogs.$inferSelect
@@ -55,7 +55,7 @@ async function* pages(input: SourcePageInput): AsyncIterable<WorkflowLogRow[]> {
// Use the order-preserving returned array (the util's documented contract)
// and write back, rather than mutating rows inside the mapper.
const materialized = await mapWithConcurrency(rows, MATERIALIZE_CONCURRENCY, (row) =>
materializeExecutionData(row.executionData as Record<string, unknown> | null, {
materializeExecutionDataForDisplay(row.executionData as Record<string, unknown> | null, {
workspaceId: row.workspaceId,
workflowId: row.workflowId,
executionId: row.executionId,
+83
View File
@@ -0,0 +1,83 @@
/**
* @vitest-environment node
*/
import { dbChainMockFns, encryptionMock, encryptionMockFns, resetDbChainMock } from '@sim/testing'
import { beforeEach, describe, expect, it, vi } from 'vitest'
vi.unmock('@/lib/environment/utils')
vi.mock('@/lib/core/security/encryption', () => encryptionMock)
import {
getEffectiveDecryptedEnv,
getEffectiveEnvironmentSnapshot,
invalidateEffectiveDecryptedEnvCache,
} from '@/lib/environment/utils'
describe('effective environment resolution cache', () => {
beforeEach(() => {
vi.clearAllMocks()
resetDbChainMock()
encryptionMockFns.mockDecryptSecret.mockReset()
encryptionMockFns.mockEncryptSecret.mockReset()
invalidateEffectiveDecryptedEnvCache({ userId: 'user-1' })
dbChainMockFns.limit.mockResolvedValue([{ variables: { API_KEY: 'encrypted-value' } }])
encryptionMockFns.mockDecryptSecret.mockResolvedValue({ decrypted: 'runtime-value' })
})
it('shares one atomic snapshot and returns defensive clones', async () => {
const [decrypted, snapshot] = await Promise.all([
getEffectiveDecryptedEnv('user-1'),
getEffectiveEnvironmentSnapshot('user-1'),
])
expect(decrypted).toEqual({ API_KEY: 'runtime-value' })
expect(snapshot).toMatchObject({
personalEncrypted: { API_KEY: 'encrypted-value' },
personalDecrypted: { API_KEY: 'runtime-value' },
})
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledOnce()
decrypted.API_KEY = 'mutated-runtime'
snapshot.personalEncrypted.API_KEY = 'mutated-ciphertext'
snapshot.personalDecrypted.API_KEY = 'mutated-snapshot'
snapshot.conflicts.push('MUTATED')
await expect(getEffectiveDecryptedEnv('user-1')).resolves.toEqual({
API_KEY: 'runtime-value',
})
await expect(getEffectiveEnvironmentSnapshot('user-1')).resolves.toMatchObject({
personalEncrypted: { API_KEY: 'encrypted-value' },
personalDecrypted: { API_KEY: 'runtime-value' },
conflicts: [],
})
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledOnce()
})
it('evicts rejected loads and retries the canonical lookup', async () => {
dbChainMockFns.limit.mockRejectedValueOnce(new Error('database unavailable'))
await expect(getEffectiveEnvironmentSnapshot('user-1')).rejects.toThrow('database unavailable')
dbChainMockFns.limit.mockResolvedValue([{ variables: { API_KEY: 'encrypted-value' } }])
await expect(getEffectiveDecryptedEnv('user-1')).resolves.toEqual({
API_KEY: 'runtime-value',
})
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledOnce()
})
it('reloads the full snapshot after invalidation', async () => {
await expect(getEffectiveDecryptedEnv('user-1')).resolves.toEqual({
API_KEY: 'runtime-value',
})
invalidateEffectiveDecryptedEnvCache({ userId: 'user-1' })
dbChainMockFns.limit.mockResolvedValue([{ variables: { API_KEY: 'rotated-ciphertext' } }])
encryptionMockFns.mockDecryptSecret.mockResolvedValue({ decrypted: 'rotated-runtime' })
await expect(getEffectiveEnvironmentSnapshot('user-1')).resolves.toMatchObject({
personalEncrypted: { API_KEY: 'rotated-ciphertext' },
personalDecrypted: { API_KEY: 'rotated-runtime' },
})
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledTimes(2)
})
})
+74 -45
View File
@@ -14,26 +14,44 @@ import {
import { checkWorkspaceAccess, type WorkspaceAccess } from '@/lib/workspaces/permissions/utils'
const logger = createLogger('EnvironmentUtils')
const EFFECTIVE_DECRYPTED_ENV_CACHE_TTL_MS = 2_000
const EFFECTIVE_DECRYPTED_ENV_CACHE_MAX_ENTRIES = 1_000
const EFFECTIVE_ENVIRONMENT_CACHE_TTL_MS = 2_000
const EFFECTIVE_ENVIRONMENT_CACHE_MAX_ENTRIES = 1_000
interface EffectiveDecryptedEnvCacheEntry {
userId: string
workspaceId?: string
promise: Promise<Record<string, string>>
export interface EnvironmentResolutionSnapshot {
personalEncrypted: Record<string, string>
workspaceEncrypted: Record<string, string>
personalDecrypted: Record<string, string>
workspaceDecrypted: Record<string, string>
conflicts: string[]
decryptionFailures: string[]
}
const effectiveDecryptedEnvCache = new LRUCache<string, EffectiveDecryptedEnvCacheEntry>({
max: EFFECTIVE_DECRYPTED_ENV_CACHE_MAX_ENTRIES,
ttl: EFFECTIVE_DECRYPTED_ENV_CACHE_TTL_MS,
interface EffectiveEnvironmentCacheEntry {
userId: string
workspaceId?: string
promise: Promise<EnvironmentResolutionSnapshot>
}
const effectiveEnvironmentCache = new LRUCache<string, EffectiveEnvironmentCacheEntry>({
max: EFFECTIVE_ENVIRONMENT_CACHE_MAX_ENTRIES,
ttl: EFFECTIVE_ENVIRONMENT_CACHE_TTL_MS,
})
function getEffectiveDecryptedEnvCacheKey(userId: string, workspaceId?: string): string {
function getEffectiveEnvironmentCacheKey(userId: string, workspaceId?: string): string {
return JSON.stringify([userId, workspaceId ?? null])
}
function cloneEnvVars(envVars: Record<string, string>): Record<string, string> {
return { ...envVars }
function cloneEnvironmentResolutionSnapshot(
snapshot: EnvironmentResolutionSnapshot
): EnvironmentResolutionSnapshot {
return {
personalEncrypted: { ...snapshot.personalEncrypted },
workspaceEncrypted: { ...snapshot.workspaceEncrypted },
personalDecrypted: { ...snapshot.personalDecrypted },
workspaceDecrypted: { ...snapshot.workspaceDecrypted },
conflicts: [...snapshot.conflicts],
decryptionFailures: [...snapshot.decryptionFailures],
}
}
export function invalidateEffectiveDecryptedEnvCache(input: {
@@ -43,13 +61,13 @@ export function invalidateEffectiveDecryptedEnvCache(input: {
const { userId, workspaceId } = input
if (!userId && !workspaceId) return
effectiveDecryptedEnvCache.forEach((entry, cacheKey) => {
effectiveEnvironmentCache.forEach((entry, cacheKey) => {
if (userId && entry.userId === userId) {
effectiveDecryptedEnvCache.delete(cacheKey)
effectiveEnvironmentCache.delete(cacheKey)
return
}
if (workspaceId && entry.workspaceId === workspaceId) {
effectiveDecryptedEnvCache.delete(cacheKey)
effectiveEnvironmentCache.delete(cacheKey)
}
})
}
@@ -94,14 +112,7 @@ export async function getPersonalAndWorkspaceEnv(
userId: string,
workspaceId?: string,
options?: { workspaceAccess?: WorkspaceAccess }
): Promise<{
personalEncrypted: Record<string, string>
workspaceEncrypted: Record<string, string>
personalDecrypted: Record<string, string>
workspaceDecrypted: Record<string, string>
conflicts: string[]
decryptionFailures: string[]
}> {
): Promise<EnvironmentResolutionSnapshot> {
let workspaceCanAdmin = false
if (workspaceId) {
const access = options?.workspaceAccess ?? (await checkWorkspaceAccess(workspaceId, userId))
@@ -357,6 +368,42 @@ export async function upsertWorkspaceEnvVars(
return updatedKeys
}
async function getCachedEnvironmentResolutionSnapshot(
userId: string,
workspaceId?: string
): Promise<EnvironmentResolutionSnapshot> {
const cacheKey = getEffectiveEnvironmentCacheKey(userId, workspaceId)
const cached = effectiveEnvironmentCache.get(cacheKey)
if (cached) {
return cached.promise
}
const promise = getPersonalAndWorkspaceEnv(userId, workspaceId).catch((error) => {
effectiveEnvironmentCache.delete(cacheKey)
throw error
})
effectiveEnvironmentCache.set(cacheKey, {
userId,
workspaceId,
promise,
})
return promise
}
/**
* Returns a defensive clone of the cached environment snapshot used for runtime resolution.
*/
export async function getEffectiveEnvironmentSnapshot(
userId: string,
workspaceId?: string
): Promise<EnvironmentResolutionSnapshot> {
return cloneEnvironmentResolutionSnapshot(
await getCachedEnvironmentResolutionSnapshot(userId, workspaceId)
)
}
/**
* Returns a merged decrypted env map for webhook/copilot/MCP config resolution.
*/
@@ -364,27 +411,9 @@ export async function getEffectiveDecryptedEnv(
userId: string,
workspaceId?: string
): Promise<Record<string, string>> {
const cacheKey = getEffectiveDecryptedEnvCacheKey(userId, workspaceId)
const cached = effectiveDecryptedEnvCache.get(cacheKey)
if (cached) {
return cloneEnvVars(await cached.promise)
}
const promise = getPersonalAndWorkspaceEnv(userId, workspaceId)
.then(({ personalDecrypted, workspaceDecrypted }) => ({
...personalDecrypted,
...workspaceDecrypted,
}))
.catch((error) => {
effectiveDecryptedEnvCache.delete(cacheKey)
throw error
})
effectiveDecryptedEnvCache.set(cacheKey, {
const { personalDecrypted, workspaceDecrypted } = await getCachedEnvironmentResolutionSnapshot(
userId,
workspaceId,
promise,
})
return cloneEnvVars(await promise)
workspaceId
)
return { ...personalDecrypted, ...workspaceDecrypted }
}
+18 -4
View File
@@ -23,6 +23,8 @@ interface CompactState {
seen: WeakSet<object>
}
const BLOCK_LOG_COMPACTION_CONCURRENCY = 4
function getJsonAndSize(value: unknown): { json: string; size: number } | null {
try {
const json = JSON.stringify(value)
@@ -260,8 +262,15 @@ export async function compactBlockLogs(
return logs
}
return Promise.all(
logs.map(async (log) => {
const compactedLogs = new Array<BlockLog>(logs.length)
let cursor = 0
const worker = async (): Promise<void> => {
while (true) {
const index = cursor
cursor += 1
if (index >= logs.length) return
const log = logs[index]
const compactedLog = { ...log }
if ('input' in compactedLog) {
compactedLog.input = await compactExecutionPayload(compactedLog.input, options)
@@ -275,7 +284,12 @@ export async function compactBlockLogs(
options
)
}
return compactedLog
})
compactedLogs[index] = compactedLog
}
}
await Promise.all(
Array.from({ length: Math.min(BLOCK_LOG_COMPACTION_CONCURRENCY, logs.length) }, worker)
)
return compactedLogs
}
@@ -0,0 +1,62 @@
/**
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import {
getPrivateToolMetadataField,
isPrivateToolMetadataType,
PRIVATE_TOOL_METADATA_REQUEST_HEADER,
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
RESOLVED_SECRET_NAMES_FIELD,
RESOLVED_SECRET_NAMES_METADATA_V1,
RESOLVED_SECRET_PROVENANCE_FIELD,
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
requestsPrivateToolMetadata,
responseHasPrivateToolMetadata,
} from '@/lib/execution/private-tool-metadata'
describe('private tool metadata protocol', () => {
it('keeps the versioned wire markers stable', () => {
expect(PRIVATE_TOOL_METADATA_REQUEST_HEADER).toBe('x-sim-request-private-tool-metadata')
expect(PRIVATE_TOOL_METADATA_RESPONSE_HEADER).toBe('x-sim-private-tool-metadata')
expect(RESOLVED_SECRET_NAMES_METADATA_V1).toBe('resolved-secret-names-v1')
expect(RESOLVED_SECRET_PROVENANCE_METADATA_V1).toBe('resolved-secret-provenance-v1')
expect(RESOLVED_SECRET_NAMES_FIELD).toBe('__resolvedSecretNames')
expect(RESOLVED_SECRET_PROVENANCE_FIELD).toBe('__resolvedSecretTraceProvenance')
})
it('accepts only exact request and response markers', () => {
const requestHeaders = new Headers({
[PRIVATE_TOOL_METADATA_REQUEST_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1,
})
const responseHeaders = new Headers({
[PRIVATE_TOOL_METADATA_RESPONSE_HEADER]: RESOLVED_SECRET_NAMES_METADATA_V1,
})
expect(
requestsPrivateToolMetadata(requestHeaders, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
).toBe(true)
expect(requestsPrivateToolMetadata(requestHeaders, RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(
false
)
expect(responseHasPrivateToolMetadata(responseHeaders, RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(
true
)
expect(
responseHasPrivateToolMetadata(responseHeaders, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
).toBe(false)
})
it('maps each marker to its private payload field', () => {
expect(isPrivateToolMetadataType(RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(true)
expect(isPrivateToolMetadataType(RESOLVED_SECRET_PROVENANCE_METADATA_V1)).toBe(true)
expect(isPrivateToolMetadataType('resolved-secret-provenance-v2')).toBe(false)
expect(isPrivateToolMetadataType(null)).toBe(false)
expect(getPrivateToolMetadataField(RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(
RESOLVED_SECRET_NAMES_FIELD
)
expect(getPrivateToolMetadataField(RESOLVED_SECRET_PROVENANCE_METADATA_V1)).toBe(
RESOLVED_SECRET_PROVENANCE_FIELD
)
})
})
@@ -0,0 +1,44 @@
export const PRIVATE_TOOL_METADATA_REQUEST_HEADER = 'x-sim-request-private-tool-metadata'
export const PRIVATE_TOOL_METADATA_RESPONSE_HEADER = 'x-sim-private-tool-metadata'
export const RESOLVED_SECRET_NAMES_METADATA_V1 = 'resolved-secret-names-v1'
export const RESOLVED_SECRET_PROVENANCE_METADATA_V1 = 'resolved-secret-provenance-v1'
export const RESOLVED_SECRET_NAMES_FIELD = '__resolvedSecretNames'
export const RESOLVED_SECRET_PROVENANCE_FIELD = '__resolvedSecretTraceProvenance'
export type PrivateToolMetadataType =
| typeof RESOLVED_SECRET_NAMES_METADATA_V1
| typeof RESOLVED_SECRET_PROVENANCE_METADATA_V1
interface HeaderReader {
get(name: string): string | null
}
export function isPrivateToolMetadataType(value: string | null): value is PrivateToolMetadataType {
return (
value === RESOLVED_SECRET_NAMES_METADATA_V1 || value === RESOLVED_SECRET_PROVENANCE_METADATA_V1
)
}
export function requestsPrivateToolMetadata(
headers: HeaderReader,
expectedType: PrivateToolMetadataType
): boolean {
return headers.get(PRIVATE_TOOL_METADATA_REQUEST_HEADER) === expectedType
}
export function responseHasPrivateToolMetadata(
headers: HeaderReader,
expectedType: PrivateToolMetadataType
): boolean {
return headers.get(PRIVATE_TOOL_METADATA_RESPONSE_HEADER) === expectedType
}
export function getPrivateToolMetadataField(
type: PrivateToolMetadataType
): typeof RESOLVED_SECRET_NAMES_FIELD | typeof RESOLVED_SECRET_PROVENANCE_FIELD {
return type === RESOLVED_SECRET_NAMES_METADATA_V1
? RESOLVED_SECRET_NAMES_FIELD
: RESOLVED_SECRET_PROVENANCE_FIELD
}
@@ -0,0 +1,6 @@
import type { BlockLog } from '@/executor/types'
/** A server-projected BlockLog copy carrying presentation-only reconciliation hints. */
export interface SecretSafeBlockLog extends BlockLog {
clearLiveDisplay?: true
}
@@ -0,0 +1,66 @@
/**
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import {
collectFunctionalBlockOutputs,
FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE,
FunctionalOutputsUnavailableError,
getFunctionalBlockOutput,
} from '@/lib/logs/execution/functional-outputs'
describe('functional execution outputs', () => {
it('prefers raw execution state over projected TraceSpan output', () => {
const data = {
executionState: {
blockStates: {
'function-1': { output: { result: 1234 } },
},
},
traceSpans: [
{
blockId: 'function-1',
output: { result: '{{OPENAI_API_KEY}}' },
},
],
}
expect(getFunctionalBlockOutput(data, 'function-1')).toEqual({ result: 1234 })
})
it('uses nested TraceSpan output for legacy rows without execution state', () => {
const outputs = collectFunctionalBlockOutputs({
traceSpans: [
{
children: [{ blockId: 'function-1', output: { result: 'legacy' } }],
},
],
})
expect(outputs.get('function-1')).toEqual({ result: 'legacy' })
})
it('does not mix projected TraceSpan output into an available raw state', () => {
const outputs = collectFunctionalBlockOutputs({
executionState: {
blockStates: {
'function-1': { output: { result: 1234 } },
},
},
traceSpans: [{ blockId: 'function-2', output: { result: '{{OPENAI_API_KEY}}' } }],
})
expect(outputs.get('function-1')).toEqual({ result: 1234 })
expect(outputs.has('function-2')).toBe(false)
})
it('fails instead of returning projected traces when raw state was truncated', () => {
expect(() =>
collectFunctionalBlockOutputs({
executionDataTruncated: true,
traceSpans: [{ blockId: 'function-1', output: { result: '{{OPENAI_API_KEY}}' } }],
})
).toThrowError(new FunctionalOutputsUnavailableError())
expect(FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE).toContain('could not be retained in full')
})
})
@@ -0,0 +1,62 @@
export interface FunctionalTraceSpanSource {
blockId?: string
output?: unknown
children?: FunctionalTraceSpanSource[]
}
export interface FunctionalExecutionDataSource {
traceSpans?: FunctionalTraceSpanSource[]
executionState?: {
blockStates?: Record<string, { output?: unknown }>
}
executionDataTruncated?: boolean
}
export const FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE =
'Raw block outputs are unavailable because the execution data could not be retained in full.'
export class FunctionalOutputsUnavailableError extends Error {
constructor() {
super(FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE)
this.name = 'FunctionalOutputsUnavailableError'
}
}
function collectTraceBlockOutputs(
spans: FunctionalTraceSpanSource[] | undefined,
outputs: Map<string, unknown>
): void {
if (!spans) return
for (const span of spans) {
if (span.blockId && span.output !== undefined && !outputs.has(span.blockId)) {
outputs.set(span.blockId, span.output)
}
collectTraceBlockOutputs(span.children, outputs)
}
}
/** Returns functional block outputs, preferring raw execution state over display TraceSpans. */
export function collectFunctionalBlockOutputs(
data: FunctionalExecutionDataSource | undefined
): Map<string, unknown> {
const outputs = new Map<string, unknown>()
const blockStates = data?.executionState?.blockStates
if (blockStates) {
for (const [blockId, blockState] of Object.entries(blockStates)) {
if (blockState?.output !== undefined) outputs.set(blockId, blockState.output)
}
return outputs
}
if (data?.executionDataTruncated) throw new FunctionalOutputsUnavailableError()
collectTraceBlockOutputs(data?.traceSpans, outputs)
return outputs
}
export function getFunctionalBlockOutput(
data: FunctionalExecutionDataSource | undefined,
blockId: string
): unknown {
return collectFunctionalBlockOutputs(data).get(blockId)
}
+116 -5
View File
@@ -3,6 +3,8 @@ import { dbChainMockFns, queueTableRows, resetDbChainMock } from '@sim/testing'
import { afterAll, beforeEach, describe, expect, test, vi } from 'vitest'
import { recordUsage } from '@/lib/billing/core/usage-log'
import { ExecutionLogger } from '@/lib/logs/execution/logger'
import type { WorkflowExecutionLog } from '@/lib/logs/types'
import type { SerializableExecutionState } from '@/executor/execution/types'
afterAll(resetDbChainMock)
@@ -242,6 +244,56 @@ describe('ExecutionLogger', () => {
expect(completedData.billingAttribution).toEqual(billingAttribution)
})
test('preserves server-only lifecycle metadata after execution-state PII masking', () => {
const loggerInstance = new ExecutionLogger() as unknown as {
preservePrivateExecutionStateMetadata(
redactedState: SerializableExecutionState | undefined,
originalState: SerializableExecutionState | undefined
): SerializableExecutionState | undefined
}
const provenance = {
version: 1 as const,
complete: true,
entries: [{ name: 'API_SECRET', encryptedValue: 'enc:original-ciphertext' }],
}
const trustedLargeValueAccess = {
executionIds: ['execution-1'],
largeValueKeys: ['execution/workspace-1/workflow-1/execution-1/value.json'],
fileKeys: ['workspace-1/file-1'],
}
const originalState: SerializableExecutionState = {
blockStates: {},
executedBlocks: [],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: [],
resolvedSecretTraceProvenance: provenance,
trustedLargeValueAccess,
}
const redactedState: SerializableExecutionState = {
...originalState,
resolvedSecretTraceProvenance: {
...provenance,
entries: [{ name: 'API_SECRET', encryptedValue: '[MASKED]' }],
},
trustedLargeValueAccess: {
executionIds: [],
largeValueKeys: [],
fileKeys: [],
},
}
const preserved = loggerInstance.preservePrivateExecutionStateMetadata(
redactedState,
originalState
)
expect(preserved?.resolvedSecretTraceProvenance).toBe(provenance)
expect(preserved?.trustedLargeValueAccess).toBe(trustedLargeValueAccess)
expect(preserved?.blockStates).toBe(redactedState.blockStates)
})
test('summarizes oversized execution data before storage', () => {
const loggerInstance = new ExecutionLogger() as any
const largePayload = 'x'.repeat(1_100_000)
@@ -325,12 +377,71 @@ describe('ExecutionLogger', () => {
activeExecutionPathLength: 0,
pendingQueueLength: 0,
})
expect(compacted.traceSpans?.[0]?.children?.[0]?.input).toEqual({
_truncated: true,
reason: 'execution_data_size_limit',
originalBytes: expect.any(Number),
summary: 'object with 2 keys',
expect(compacted.traceSpans?.[0]?.children?.[0]).not.toHaveProperty('input')
})
test('retains tool-call structure when aggregate trace content exceeds the compaction cap', () => {
const loggerInstance = new ExecutionLogger() as unknown as {
compactExecutionDataForStorage(
executionData: WorkflowExecutionLog['executionData'],
executionId: string
): WorkflowExecutionLog['executionData']
}
const oversizedContent = 'x'.repeat(9_000)
const modelToolCalls = Array.from({ length: 200 }, (_, index) => ({
id: `model-call-${index}-${'m'.repeat(40)}`,
name: 'lookup',
arguments: oversizedContent,
}))
const toolCalls = Array.from({ length: 200 }, (_, index) => ({
id: `legacy-call-${index}-${'l'.repeat(40)}`,
name: 'legacy_lookup',
duration: 1,
startTime: '2025-01-01T00:00:00.000Z',
endTime: '2025-01-01T00:00:00.001Z',
status: 'success' as const,
input: oversizedContent,
output: oversizedContent,
error: oversizedContent,
}))
const compacted = loggerInstance.compactExecutionDataForStorage(
{
traceSpans: [
{
id: 'span-1',
name: 'Agent',
type: 'agent',
duration: 1,
startTime: '2025-01-01T00:00:00.000Z',
endTime: '2025-01-01T00:00:00.001Z',
status: 'success',
modelToolCalls,
toolCalls,
},
],
finalOutput: { data: 'y'.repeat(1_100_000) },
},
'execution-tool-structure'
)
expect(compacted.executionDataTruncated).toBe(true)
expect(compacted.traceSpans?.[0]?.modelToolCalls).toHaveLength(modelToolCalls.length)
expect(compacted.traceSpans?.[0]?.modelToolCalls?.[0]).toEqual({
id: modelToolCalls[0].id,
name: 'lookup',
})
expect(compacted.traceSpans?.[0]?.toolCalls).toHaveLength(toolCalls.length)
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).toEqual(
expect.objectContaining({
id: toolCalls[0].id,
name: 'legacy_lookup',
status: 'success',
})
)
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).not.toHaveProperty('input')
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).not.toHaveProperty('output')
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).not.toHaveProperty('error')
})
})
+192 -42
View File
@@ -51,8 +51,9 @@ import {
import { snapshotService } from '@/lib/logs/execution/snapshot/service'
import { traceSpansIndicateFailure } from '@/lib/logs/execution/trace-spans/trace-spans'
import {
copyTraceSpansWithoutCosts,
externalizeExecutionData,
stripSpanCosts,
materializeExecutionData,
TRACE_STORE_REF_KEY,
} from '@/lib/logs/execution/trace-store'
import type {
@@ -186,13 +187,87 @@ function summarizeValueForExecutionData(value: unknown, maxBytes: number): unkno
}
}
function summarizeTextForExecutionData(value: string | undefined): string | undefined {
if (!value) return value
const size = getJsonByteSize(value, MAX_TRACE_IO_BYTES)
if (size === undefined || size <= MAX_TRACE_IO_BYTES) {
return value
function retainBoundedTraceContent<T>(value: T, maxBytes = MAX_TRACE_IO_BYTES): T | undefined {
const size = getJsonByteSize(value, maxBytes)
return size !== undefined && size <= maxBytes ? value : undefined
}
function stripModelToolCallArguments(
calls: NonNullable<TraceSpan['modelToolCalls']>
): NonNullable<TraceSpan['modelToolCalls']> {
return calls.map(({ arguments: _arguments, ...call }) => call as (typeof calls)[number])
}
function compactModelToolCalls(
calls: NonNullable<TraceSpan['modelToolCalls']>
): NonNullable<TraceSpan['modelToolCalls']> | undefined {
const compacted = calls.map(({ arguments: callArguments, ...call }) => {
const retainedArguments = retainBoundedTraceContent(callArguments)
return {
...call,
...(retainedArguments !== undefined ? { arguments: retainedArguments } : {}),
} as (typeof calls)[number]
})
return retainBoundedTraceContent(compacted)
}
function compactLegacyToolCalls(
calls: NonNullable<TraceSpan['toolCalls']>
): NonNullable<TraceSpan['toolCalls']> | undefined {
const compacted = calls.map(({ input, output, error, ...call }) => ({
...call,
...(retainBoundedTraceContent(input) !== undefined ? { input } : {}),
...(retainBoundedTraceContent(output) !== undefined ? { output } : {}),
...(retainBoundedTraceContent(error) !== undefined ? { error } : {}),
}))
return retainBoundedTraceContent(compacted)
}
function stripLegacyToolCallContent(
calls: NonNullable<TraceSpan['toolCalls']>
): NonNullable<TraceSpan['toolCalls']> {
return calls.map(({ input: _input, output: _output, error: _error, ...call }) => call)
}
function compactProviderTiming(
providerTiming: NonNullable<TraceSpan['providerTiming']>
): NonNullable<TraceSpan['providerTiming']> {
return {
...providerTiming,
segments: providerTiming.segments.map(
({ assistantContent, thinkingContent, errorMessage, toolCalls, ...segment }) => ({
...segment,
...(retainBoundedTraceContent(assistantContent) !== undefined ? { assistantContent } : {}),
...(retainBoundedTraceContent(thinkingContent) !== undefined ? { thinkingContent } : {}),
...(retainBoundedTraceContent(errorMessage) !== undefined ? { errorMessage } : {}),
...(toolCalls
? {
toolCalls: compactModelToolCalls(toolCalls) ?? stripModelToolCallArguments(toolCalls),
}
: {}),
})
),
}
}
function stripProviderTimingContent(
providerTiming: NonNullable<TraceSpan['providerTiming']>
): NonNullable<TraceSpan['providerTiming']> {
return {
...providerTiming,
segments: providerTiming.segments.map(
({
assistantContent: _assistantContent,
thinkingContent: _thinkingContent,
errorMessage: _errorMessage,
toolCalls,
...segment
}) => ({
...segment,
...(toolCalls ? { toolCalls: stripModelToolCallArguments(toolCalls) } : {}),
})
),
}
return `[Truncated ${size} byte text value due to execution log size limit]`
}
function summarizeTraceSpansForExecutionData(traceSpans?: TraceSpan[]): TraceSpan[] | undefined {
@@ -201,33 +276,39 @@ function summarizeTraceSpansForExecutionData(traceSpans?: TraceSpan[]): TraceSpa
}
return traceSpans.map((span) => {
const { input, output, children, thinking, modelToolCalls, ...rest } = span
const {
input,
output,
children,
thinking,
errorMessage,
modelToolCalls,
toolCalls,
providerTiming,
...rest
} = span
const summarized: TraceSpan = { ...rest }
if (input !== undefined) {
summarized.input = summarizeValueForExecutionData(input, MAX_TRACE_IO_BYTES) as Record<
string,
unknown
>
}
if (output !== undefined) {
summarized.output = summarizeValueForExecutionData(output, MAX_TRACE_IO_BYTES) as Record<
string,
unknown
>
}
const retainedInput = retainBoundedTraceContent(input)
if (retainedInput !== undefined) summarized.input = retainedInput
const retainedOutput = retainBoundedTraceContent(output)
if (retainedOutput !== undefined) summarized.output = retainedOutput
if (children?.length) {
summarized.children = summarizeTraceSpansForExecutionData(children)
}
if (thinking !== undefined) {
summarized.thinking = summarizeTextForExecutionData(thinking)
const retainedThinking = retainBoundedTraceContent(thinking)
if (retainedThinking !== undefined) summarized.thinking = retainedThinking
const retainedError = retainBoundedTraceContent(errorMessage)
if (retainedError !== undefined) summarized.errorMessage = retainedError
if (modelToolCalls) {
summarized.modelToolCalls =
compactModelToolCalls(modelToolCalls) ?? stripModelToolCallArguments(modelToolCalls)
}
if (
modelToolCalls !== undefined &&
(getJsonByteSize(modelToolCalls, MAX_TRACE_IO_BYTES) ?? 0) <= MAX_TRACE_IO_BYTES
) {
summarized.modelToolCalls = modelToolCalls
if (toolCalls) {
summarized.toolCalls =
compactLegacyToolCalls(toolCalls) ?? stripLegacyToolCallContent(toolCalls)
}
if (providerTiming) summarized.providerTiming = compactProviderTiming(providerTiming)
return summarized
})
@@ -244,11 +325,17 @@ function summarizeTraceSpansWithoutIo(traceSpans?: TraceSpan[]): TraceSpan[] | u
output: _output,
children,
thinking: _thinking,
modelToolCalls: _modelToolCalls,
errorMessage: _errorMessage,
modelToolCalls,
toolCalls,
providerTiming,
...rest
} = span
return {
...rest,
...(modelToolCalls ? { modelToolCalls: stripModelToolCallArguments(modelToolCalls) } : {}),
...(toolCalls ? { toolCalls: stripLegacyToolCallContent(toolCalls) } : {}),
...(providerTiming ? { providerTiming: stripProviderTimingContent(providerTiming) } : {}),
...(children?.length ? { children: summarizeTraceSpansWithoutIo(children) } : {}),
}
})
@@ -698,6 +785,73 @@ export class ExecutionLogger implements IExecutionLoggerService {
})
}
/** Restores server-only lifecycle metadata after broad execution-state PII masking. */
private preservePrivateExecutionStateMetadata(
redactedState: SerializableExecutionState | undefined,
originalState: SerializableExecutionState | undefined
): SerializableExecutionState | undefined {
const provenance = originalState?.resolvedSecretTraceProvenance
const trustedLargeValueAccess = originalState?.trustedLargeValueAccess
return redactedState
? {
...redactedState,
...(provenance !== undefined ? { resolvedSecretTraceProvenance: provenance } : {}),
...(trustedLargeValueAccess !== undefined ? { trustedLargeValueAccess } : {}),
}
: redactedState
}
async loadTraceSpansForProjection(params: {
executionId: string
workflowId: string
workspaceId: string | null
traceSpans: TraceSpan[]
isResume?: boolean
}): Promise<TraceSpan[]> {
let sourceSpans = params.traceSpans
if (params.isResume && sourceSpans.length === 0) {
const [existingLog] = await execDb
.select({ executionData: workflowExecutionLogs.executionData })
.from(workflowExecutionLogs)
.where(eq(workflowExecutionLogs.executionId, params.executionId))
.limit(1)
const executionData = await materializeExecutionData(
existingLog?.executionData as Record<string, unknown> | null,
{
workspaceId: params.workspaceId,
workflowId: params.workflowId,
executionId: params.executionId,
}
)
if (Array.isArray(executionData.traceSpans)) {
sourceSpans = executionData.traceSpans as TraceSpan[]
}
}
return sourceSpans
}
async prepareTraceSpansForProjection(params: {
workflowId: string
executionId: string
workspaceId: string | null
userId?: string | null
traceSpans: TraceSpan[]
}): Promise<TraceSpan[]> {
const filtered = filterForDisplay(params.traceSpans)
const redacted = redactApiKeys(filtered)
const pii = await this.applyPiiRedaction(
params.workspaceId,
{ traceSpans: redacted },
{
workflowId: params.workflowId,
executionId: params.executionId,
userId: params.userId ?? undefined,
}
)
return pii.traceSpans as TraceSpan[]
}
async completeWorkflowExecution(params: {
executionId: string
endedAt: string
@@ -785,11 +939,7 @@ export class ExecutionLogger implements IExecutionLoggerService {
const status = statusOverride ?? (hasErrors ? 'failed' : 'completed')
// For resume executions, rebuild trace spans from the aggregated logs
const mergedTraceSpans = isResume
? traceSpans && traceSpans.length > 0
? traceSpans
: existingExecutionData?.traceSpans || []
: traceSpans
const mergedTraceSpans = traceSpans
const executionCost = {
total: costSummary.totalCost,
@@ -826,13 +976,12 @@ export class ExecutionLogger implements IExecutionLoggerService {
builtExecutionData.workflowInput
)
const filteredTraceSpans = filterForDisplay(builtExecutionData.traceSpans)
const preparedTraceSpans = builtExecutionData.traceSpans
const filteredFinalOutput = filterForDisplay(builtExecutionData.finalOutput)
const filteredWorkflowInput =
builtExecutionData.workflowInput !== undefined
? filterForDisplay(builtExecutionData.workflowInput)
: undefined
const redactedTraceSpans = redactApiKeys(filteredTraceSpans)
const redactedFinalOutput = redactApiKeys(filteredFinalOutput)
const redactedWorkflowInput =
filteredWorkflowInput !== undefined ? redactApiKeys(filteredWorkflowInput) : undefined
@@ -840,7 +989,7 @@ export class ExecutionLogger implements IExecutionLoggerService {
const pii = await this.applyPiiRedaction(
existingLog?.workspaceId ?? null,
{
traceSpans: redactedTraceSpans,
traceSpans: [],
finalOutput: redactedFinalOutput,
...(redactedWorkflowInput !== undefined ? { workflowInput: redactedWorkflowInput } : {}),
...(builtExecutionData.error !== undefined ? { error: builtExecutionData.error } : {}),
@@ -899,9 +1048,14 @@ export class ExecutionLogger implements IExecutionLoggerService {
? Math.max(0, Math.round(rawDurationMs))
: 0
const safeExecutionState = this.preservePrivateExecutionStateMetadata(
pii.executionState as SerializableExecutionState | undefined,
builtExecutionData.executionState
)
const cleanExecutionData: ExecutionData = {
...builtExecutionData,
traceSpans: pii.traceSpans as TraceSpan[],
traceSpans: copyTraceSpansWithoutCosts(preparedTraceSpans),
finalOutput: pii.finalOutput as BlockOutputData,
...(pii.workflowInput !== undefined ? { workflowInput: pii.workflowInput } : {}),
...(pii.error !== undefined ? { error: pii.error as string } : {}),
@@ -909,9 +1063,7 @@ export class ExecutionLogger implements IExecutionLoggerService {
? { completionFailure: pii.completionFailure as string }
: {}),
...(pii.trigger !== undefined ? { trigger: pii.trigger as ExecutionTrigger } : {}),
...(pii.executionState !== undefined
? { executionState: pii.executionState as SerializableExecutionState }
: {}),
...(safeExecutionState !== undefined ? { executionState: safeExecutionState } : {}),
...(pii.environment !== undefined
? { environment: pii.environment as ExecutionEnvironment }
: {}),
@@ -920,8 +1072,6 @@ export class ExecutionLogger implements IExecutionLoggerService {
: {}),
}
stripSpanCosts((cleanExecutionData as Record<string, unknown>).traceSpans)
// Bounded in-memory form. Returned to callers (notification delivery/events)
// and reused as the inline-storage fallback below. This is a no-op for
// payloads already within MAX_EXECUTION_DATA_BYTES.
@@ -9,14 +9,27 @@ const dbMocks = vi.hoisted(() => ({
const {
completeWorkflowExecutionMock,
loadTraceSpansForProjectionMock,
prepareTraceSpansForProjectionMock,
startWorkflowExecutionMock,
loadWorkflowStateForExecutionMock,
releaseExecutionSlotMock,
createOTelSpansMock,
workflowExecutedMock,
} = vi.hoisted(() => ({
completeWorkflowExecutionMock: vi.fn(),
loadTraceSpansForProjectionMock: vi.fn(),
prepareTraceSpansForProjectionMock: vi.fn(),
startWorkflowExecutionMock: vi.fn(),
loadWorkflowStateForExecutionMock: vi.fn(),
releaseExecutionSlotMock: vi.fn(),
createOTelSpansMock: vi.fn(),
workflowExecutedMock: vi.fn(),
}))
const { materializeLargeValueRefMock, storeLargeValueMock } = vi.hoisted(() => ({
materializeLargeValueRefMock: vi.fn(),
storeLargeValueMock: vi.fn(),
}))
vi.mock('drizzle-orm', () => ({
@@ -29,6 +42,8 @@ vi.mock('@/lib/logs/execution/logger', () => ({
executionLogger: {
startWorkflowExecution: startWorkflowExecutionMock,
completeWorkflowExecution: completeWorkflowExecutionMock,
loadTraceSpansForProjection: loadTraceSpansForProjectionMock,
prepareTraceSpansForProjection: prepareTraceSpansForProjectionMock,
},
}))
@@ -36,6 +51,16 @@ vi.mock('@/lib/billing/calculations/usage-reservation', () => ({
releaseExecutionSlot: releaseExecutionSlotMock,
}))
vi.mock('@/lib/core/telemetry', () => ({
createOTelSpansForWorkflowExecution: createOTelSpansMock,
PlatformEvents: { workflowExecuted: workflowExecutedMock },
}))
vi.mock('@/lib/execution/payloads/store', () => ({
materializeLargeValueRef: materializeLargeValueRefMock,
storeLargeValue: storeLargeValueMock,
}))
const {
setLastStartedBlockMock,
setLastCompletedBlockMock,
@@ -73,10 +98,105 @@ vi.mock('@/lib/logs/execution/logging-factory', () => ({
}))
import { calculateCostSummary } from '@/lib/logs/execution/logging-factory'
import type {
ResolvedSecretTraceMatch,
ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
import { LoggingSession } from './logging-session'
afterAll(resetDbChainMock)
function createSecretRegistry(
matches: ResolvedSecretTraceMatch[],
complete = true
): ResolvedSecretTraceRegistry {
return {
isComplete: () => complete,
getActiveMatches: () => matches,
exportProvenance: () => ({ version: 1, complete, entries: [] }),
} as unknown as ResolvedSecretTraceRegistry
}
describe('LoggingSession terminal provenance', () => {
beforeEach(() => {
vi.clearAllMocks()
resetDbChainMock()
dbChainMockFns.limit.mockResolvedValue([])
completeWorkflowExecutionMock.mockResolvedValue({})
releaseExecutionSlotMock.mockResolvedValue(undefined)
})
it.each([
[
'error',
(session: LoggingSession) => session.completeWithError({ error: { message: 'failed' } }),
],
['cancellation', (session: LoggingSession) => session.completeWithCancellation()],
['pause', (session: LoggingSession) => session.completeWithPause()],
])('persists complete zero-entry provenance on %s finalization', async (_name, finalize) => {
const session = new LoggingSession('workflow-1', `execution-${_name}`, 'manual')
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
await finalize(session)
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
executionState: expect.objectContaining({
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [],
},
}),
})
)
})
it.each(['cancellation', 'pause'] as const)(
'preserves raw execution state on %s finalization',
async (finalization) => {
const session = new LoggingSession('workflow-1', `execution-state-${finalization}`, 'manual')
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
const executionState = {
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
}
if (finalization === 'cancellation') {
await session.completeWithCancellation({ executionState })
} else {
await session.completeWithPause({ executionState })
}
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
executionState: expect.objectContaining({
blockStates: executionState.blockStates,
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [],
},
}),
})
)
}
)
})
beforeEach(() => {
loadTraceSpansForProjectionMock.mockImplementation(
async ({ traceSpans }: { traceSpans: unknown[] }) => traceSpans
)
prepareTraceSpansForProjectionMock.mockImplementation(
async ({ traceSpans }: { traceSpans: unknown[] }) => traceSpans
)
})
describe('LoggingSession start snapshots', () => {
beforeEach(() => {
vi.clearAllMocks()
@@ -224,6 +344,7 @@ describe('LoggingSession completion retries', () => {
it('starts a new error completion attempt after a non-error completion and fallback both fail', async () => {
const session = new LoggingSession('workflow-1', 'execution-3', 'api', 'req-1')
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
completeWorkflowExecutionMock
.mockRejectedValueOnce(new Error('success finalize failed'))
@@ -250,6 +371,7 @@ describe('LoggingSession completion retries', () => {
it('preserves successful final output during fallback completion', async () => {
const session = new LoggingSession('workflow-1', 'execution-5', 'api', 'req-1')
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
completeWorkflowExecutionMock
.mockRejectedValueOnce(new Error('success finalize failed'))
@@ -268,6 +390,441 @@ describe('LoggingSession completion retries', () => {
)
})
it('projects only TraceSpans while preserving functional completion values', async () => {
const session = new LoggingSession('workflow-1', 'execution-safe', 'api', 'req-1')
const secret = 'sk-demo / trace?token=7f3a91'
const rawFinalOutput = {
result: {
resolvedAtRuntime: true,
echoed: `prefix:${secret}:suffix`,
encoded: encodeURIComponent(secret),
ordinary: 'us-east-1',
},
}
const rawTraceSpans = [
{
id: 'span-safe',
name: 'Function',
type: 'function',
duration: 1,
startTime: '2026-07-01T00:00:00.000Z',
endTime: '2026-07-01T00:00:00.001Z',
status: 'success',
output: { echoed: secret, encoded: encodeURIComponent(secret) },
},
]
const rawWorkflowInput = { prompt: `use ${secret}` }
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' }])
)
completeWorkflowExecutionMock.mockResolvedValue({})
await session.safeComplete({
finalOutput: rawFinalOutput,
traceSpans: rawTraceSpans as any,
workflowInput: rawWorkflowInput,
})
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
finalOutput: rawFinalOutput,
workflowInput: rawWorkflowInput,
traceSpans: [
expect.objectContaining({
output: {
echoed: '{{OPENAI_API_KEY}}',
encoded: encodeURIComponent(secret),
},
}),
],
})
)
expect(rawFinalOutput.result.echoed).toBe(`prefix:${secret}:suffix`)
expect(rawTraceSpans[0].output.echoed).toBe(secret)
expect(calculateCostSummary).toHaveBeenCalledWith(rawTraceSpans, undefined)
const persistedSpans = completeWorkflowExecutionMock.mock.calls[0]?.[0].traceSpans
expect(createOTelSpansMock).toHaveBeenCalledWith(
expect.objectContaining({ traceSpans: persistedSpans })
)
expect(createOTelSpansMock.mock.calls[0]?.[0].traceSpans).toBe(persistedSpans)
})
it('projects secrets before display filters can truncate a long active literal', async () => {
const session = new LoggingSession('workflow-1', 'execution-long-secret', 'api', 'req-1')
const secret = `secret-${'x'.repeat(16_000)}`
const sourceTraceSpans = [
{
id: 'span-long-secret',
name: 'Function',
type: 'function',
duration: 1,
startTime: '2026-07-01T00:00:00.000Z',
endTime: '2026-07-01T00:00:00.001Z',
output: { result: secret },
},
]
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: secret, replacement: '{{LONG_SECRET}}' }])
)
prepareTraceSpansForProjectionMock.mockImplementationOnce(
async ({ traceSpans }: { traceSpans: Array<{ output?: { result?: string } }> }) => {
expect(traceSpans[0]?.output?.result).toBe('{{LONG_SECRET}}')
return traceSpans
}
)
await session.safeComplete({ traceSpans: sourceTraceSpans as any })
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
traceSpans: [expect.objectContaining({ output: { result: '{{LONG_SECRET}}' } })],
})
)
expect(sourceTraceSpans[0].output.result).toBe(secret)
})
it('fails closed when generic log transforms reintroduce a secret literal for persistence and OTel', async () => {
const session = new LoggingSession('workflow-1', 'execution-invariant', 'api', 'req-1')
const sourceTraceSpans = [
{
id: 'span-invariant',
name: 'Function',
type: 'function',
duration: 1,
startTime: '2026-07-01T00:00:00.000Z',
endTime: '2026-07-01T00:00:00.001Z',
status: 'success',
output: { apiKey: 'ordinary-value' },
},
]
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: 'E', replacement: '{{X}}' }])
)
prepareTraceSpansForProjectionMock.mockImplementationOnce(
async ({ traceSpans }: { traceSpans: Array<Record<string, unknown>> }) =>
traceSpans.map((span) => ({ ...span, output: { apiKey: '[REDACTED]' } }))
)
await session.safeComplete({ traceSpans: sourceTraceSpans as any })
const persistedSpans = completeWorkflowExecutionMock.mock.calls[0]?.[0].traceSpans
expect(persistedSpans).toEqual([
expect.objectContaining({
id: 'span-invariant',
status: 'success',
}),
])
expect(persistedSpans[0]).not.toHaveProperty('output')
expect(createOTelSpansMock).toHaveBeenCalledWith(
expect.objectContaining({ traceSpans: persistedSpans })
)
expect(createOTelSpansMock.mock.calls[0]?.[0].traceSpans).toBe(persistedSpans)
expect(sourceTraceSpans[0].output).toEqual({ apiKey: 'ordinary-value' })
})
it('hydrates post-transform refs before sharing structural-only spans with persistence and OTel', async () => {
const session = new LoggingSession('workflow-1', 'execution-ref-invariant', 'api', 'req-1')
const sourceTraceSpans = [
{
id: 'span-ref-invariant',
name: 'Function',
type: 'function',
duration: 1,
startTime: '2026-07-01T00:00:00.000Z',
endTime: '2026-07-01T00:00:00.001Z',
status: 'success',
output: { apiKey: 'ordinary-value' },
},
]
const ref = {
__simLargeValueRef: true,
version: 1,
id: 'lv_bbbbbbbbbbbb',
kind: 'string',
size: 32,
preview: '{{X}}',
} as const
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: 'E', replacement: '{{X}}' }])
)
prepareTraceSpansForProjectionMock.mockImplementationOnce(
async ({ traceSpans }: { traceSpans: Array<Record<string, unknown>> }) =>
traceSpans.map((span) => ({ ...span, output: { payload: ref } }))
)
materializeLargeValueRefMock.mockResolvedValue({ value: 'hidden-E' })
await session.safeComplete({ traceSpans: sourceTraceSpans as any })
const persistedSpans = completeWorkflowExecutionMock.mock.calls[0]?.[0].traceSpans
expect(materializeLargeValueRefMock).toHaveBeenCalledWith(
ref,
expect.objectContaining({
executionId: 'execution-ref-invariant',
trackReference: false,
})
)
expect(storeLargeValueMock).not.toHaveBeenCalled()
expect(persistedSpans).toEqual([
expect.objectContaining({
id: 'span-ref-invariant',
status: 'success',
}),
])
expect(persistedSpans[0]).not.toHaveProperty('output')
expect(createOTelSpansMock.mock.calls[0]?.[0].traceSpans).toBe(persistedSpans)
expect(sourceTraceSpans[0].output).toEqual({ apiKey: 'ordinary-value' })
})
it('projects synthetic error spans without copying the raw error into OTel metadata', async () => {
const session = new LoggingSession('workflow-1', 'execution-error-safe', 'api', 'req-1')
const secret = 'sk-demo-error-7f3a91'
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' }])
)
completeWorkflowExecutionMock.mockResolvedValue({})
const rawExecutionState = {
blockStates: { 'function-1': { output: { result: secret } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
}
await session.safeCompleteWithError({
error: { message: `Function failed with ${secret}` },
executionState: rawExecutionState,
})
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
finalOutput: { error: `Function failed with ${secret}` },
traceSpans: [
expect.objectContaining({
output: { error: 'Function failed with {{OPENAI_API_KEY}}' },
}),
],
completionFailure: `Function failed with ${secret}`,
executionState: expect.objectContaining({
blockStates: { 'function-1': { output: { result: secret } } },
}),
})
)
expect(createOTelSpansMock).toHaveBeenCalledWith(
expect.not.objectContaining({ error: expect.anything() })
)
expect(workflowExecutedMock).toHaveBeenCalledWith(
expect.not.objectContaining({ errorMessage: expect.anything() })
)
})
it('keeps fallback functional output unchanged', async () => {
const session = new LoggingSession('workflow-1', 'execution-fallback-safe', 'api', 'req-1')
const secret = 'sk-demo-fallback-7f3a91'
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' }])
)
completeWorkflowExecutionMock
.mockRejectedValueOnce(new Error('primary persistence failed'))
.mockResolvedValueOnce({})
const executionState = {
blockStates: { 'function-1': { output: { result: secret } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
}
await session.safeComplete({
finalOutput: { echoed: secret },
executionState,
})
expect(completeWorkflowExecutionMock).toHaveBeenLastCalledWith(
expect.objectContaining({
finalOutput: { echoed: secret },
finalizationPath: 'fallback_completed',
executionState: expect.objectContaining({ blockStates: executionState.blockStates }),
})
)
})
it('persists structural-only spans when installed provenance is incomplete', async () => {
const session = new LoggingSession('workflow-1', 'execution-incomplete', 'api', 'req-1')
session.setResolvedSecretTraceRegistry(createSecretRegistry([], false))
completeWorkflowExecutionMock.mockResolvedValue({})
await session.safeComplete({
finalOutput: { raw: 'functional-data' },
traceSpans: [
{
id: 'span-1',
name: 'Agent',
type: 'agent',
duration: 1,
startTime: '2026-07-01T00:00:00.000Z',
endTime: '2026-07-01T00:00:00.001Z',
status: 'success',
output: { raw: 'unknown-provenance' },
},
],
})
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
finalOutput: { raw: 'functional-data' },
traceSpans: [
expect.not.objectContaining({
output: expect.anything(),
}),
],
})
)
})
it('fails closed to structural-only spans when provenance was not installed', async () => {
const session = new LoggingSession('workflow-1', 'execution-no-registry', 'api', 'req-1')
completeWorkflowExecutionMock.mockResolvedValue({})
await session.safeComplete({
traceSpans: [
{
id: 'span-1',
name: 'Function',
type: 'function',
duration: 1,
startTime: '2026-07-01T00:00:00.000Z',
endTime: '2026-07-01T00:00:00.001Z',
output: { unknown: 'provenance' },
},
],
})
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({
finalOutput: {},
traceSpans: [expect.not.objectContaining({ output: expect.anything() })],
})
)
})
it('projects live block errors and terminal block logs without mutating raw callback data', async () => {
const session = new LoggingSession('workflow-1', 'execution-display-safe', 'manual', 'req-1')
const secret = '1234'
const rawError = `Reference Error: Line 1: return blah +${secret} - blah is not defined`
const rawLog = {
blockId: 'function-1',
blockName: 'Function 1',
blockType: 'function',
startedAt: '2026-07-01T00:00:00.000Z',
endedAt: '2026-07-01T00:00:00.001Z',
durationMs: 1,
success: false,
executionOrder: 1,
input: { code: `return blah +${secret}` },
output: { error: rawError },
error: rawError,
}
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: secret, replacement: '{{NUMBER_SECRET}}' }])
)
const display = await session.projectDisplayContent({
input: rawLog.input,
output: rawLog.output,
error: rawError,
})
const [displayLog] = await session.projectBlockLogsForDisplay([rawLog])
expect(display).toEqual({
input: { code: 'return blah +{{NUMBER_SECRET}}' },
output: {
error: 'Reference Error: Line 1: return blah +{{NUMBER_SECRET}} - blah is not defined',
},
error: 'Reference Error: Line 1: return blah +{{NUMBER_SECRET}} - blah is not defined',
clearLiveDisplay: true,
})
expect(displayLog.input).toEqual(display.input)
expect(displayLog.output).toEqual(display.output)
expect(displayLog.error).toBe(display.error)
expect(displayLog.clearLiveDisplay).toBe(true)
expect(rawLog.input.code).toBe(`return blah +${secret}`)
expect(rawLog.output.error).toBe(rawError)
expect(rawLog.error).toBe(rawError)
})
it('projects large terminal log sets in bounded batches without dropping rows', async () => {
const session = new LoggingSession('workflow-1', 'execution-display-batches', 'manual', 'req-1')
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: 'raw-secret', replacement: '{{TOKEN}}' }])
)
const rawLogs = Array.from({ length: 129 }, (_, index) => ({
blockId: `function-${index}`,
blockName: `Function ${index}`,
blockType: 'function',
startedAt: '2026-07-01T00:00:00.000Z',
endedAt: '2026-07-01T00:00:00.001Z',
durationMs: 1,
success: true,
executionOrder: index,
output: { value: `row-${index}:raw-secret` },
}))
const displayLogs = await session.projectBlockLogsForDisplay(rawLogs)
expect(displayLogs).toHaveLength(rawLogs.length)
expect(displayLogs[0].output).toEqual({ value: 'row-0:{{TOKEN}}' })
expect(displayLogs[128].output).toEqual({ value: 'row-128:{{TOKEN}}' })
expect(rawLogs[128].output.value).toBe('row-128:raw-secret')
})
it('projects a numeric Function result produced by a resolved numeric secret', async () => {
const session = new LoggingSession('workflow-1', 'execution-numeric-secret', 'manual', 'req-1')
session.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: '1234', replacement: '{{OPENAI_API_KEY}}' }])
)
const rawLog = {
blockId: 'function-1',
blockName: 'Function 1',
blockType: 'function',
startedAt: '2026-07-01T00:00:00.000Z',
endedAt: '2026-07-01T00:00:00.001Z',
durationMs: 1,
success: true,
executionOrder: 1,
input: { code: 'return 1234' },
output: { result: 1234, stdout: '' },
}
const [displayLog] = await session.projectBlockLogsForDisplay([rawLog])
expect(displayLog.input).toEqual({ code: 'return {{OPENAI_API_KEY}}' })
expect(displayLog.output).toEqual({ result: '{{OPENAI_API_KEY}}', stdout: '' })
expect(rawLog.output.result).toBe(1234)
})
it('suppresses live deltas once a resolved secret is active', async () => {
const active = new LoggingSession('workflow-1', 'execution-live-active', 'manual', 'req-1')
active.setResolvedSecretTraceRegistry(
createSecretRegistry([{ plaintext: 'split-secret', replacement: '{{SECRET}}' }])
)
const inactive = new LoggingSession('workflow-1', 'execution-live-inactive', 'manual', 'req-1')
inactive.setResolvedSecretTraceRegistry(createSecretRegistry([]))
await expect(active.projectLiveDisplayText('chunk', 'split-')).resolves.toEqual({
clearLiveDisplay: true,
})
await expect(inactive.projectLiveDisplayText('chunk', 'ordinary text')).resolves.toEqual({
chunk: 'ordinary text',
})
})
it('derives fallback cost from trace spans when the primary completion fails', async () => {
const session = new LoggingSession('workflow-1', 'execution-6', 'api', 'req-1') as any
@@ -328,6 +885,7 @@ describe('LoggingSession completion retries', () => {
it('persists failed error semantics when completeWithError receives non-error trace spans', async () => {
const session = new LoggingSession('workflow-1', 'execution-4', 'api', 'req-1')
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
const traceSpans = [
{
id: 'span-1',
@@ -431,6 +989,14 @@ describe('LoggingSession completion retries', () => {
completeWorkflowExecutionMock
.mockRejectedValueOnce(new Error('pause finalize failed'))
.mockResolvedValueOnce({})
const executionState = {
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
}
await expect(
session.safeCompleteWithPause({
@@ -438,11 +1004,18 @@ describe('LoggingSession completion retries', () => {
totalDurationMs: 10,
traceSpans: [],
workflowInput: { hello: 'world' },
executionState,
})
).resolves.toBeUndefined()
expect(session.hasCompleted()).toBe(true)
expect(completeWorkflowExecutionMock).toHaveBeenCalledTimes(2)
expect(completeWorkflowExecutionMock).toHaveBeenLastCalledWith(
expect.objectContaining({
finalizationPath: 'paused',
executionState: expect.objectContaining({ blockStates: executionState.blockStates }),
})
)
})
it('persists last started block independently from cost accumulation', async () => {
@@ -667,11 +1240,22 @@ describe('completeWithError cancelled-status guard', () => {
dbChainMockFns.limit.mockRejectedValueOnce(new Error('DB connection lost'))
completeWorkflowExecutionMock.mockResolvedValue({})
const session = new LoggingSession('workflow-1', 'execution-1', 'api', 'req-1')
const executionState = {
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
executedBlocks: ['function-1'],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: ['function-1'],
}
await session.safeCompleteWithError({ error: { message: 'block failed' } })
await session.safeCompleteWithError({
error: { message: 'block failed' },
executionState,
})
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
expect.objectContaining({ finalizationPath: 'force_failed' })
expect.objectContaining({ finalizationPath: 'force_failed', executionState })
)
expect(session.hasCompleted()).toBe(true)
})
+285 -25
View File
@@ -6,6 +6,8 @@ import { and, eq, sql } from 'drizzle-orm'
import { releaseExecutionSlot } from '@/lib/billing/calculations/usage-reservation'
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
import { isRetryableInfrastructureError } from '@/lib/core/errors/retryable-infrastructure'
import type { LargeValueStoreContext } from '@/lib/execution/payloads/store'
import type { SecretSafeBlockLog } from '@/lib/logs/execution/display-types'
import { executionLogger } from '@/lib/logs/execution/logger'
import {
type CostSummaryOptions,
@@ -21,6 +23,10 @@ import {
setLastCompletedBlock,
setLastStartedBlock,
} from '@/lib/logs/execution/progress-markers'
import {
enforceTraceSpanSecretInvariant,
projectTraceSpansForSecrets,
} from '@/lib/logs/execution/trace-secret-projection'
import { traceSpansIndicateFailure } from '@/lib/logs/execution/trace-spans/trace-spans'
import type {
ExecutionEnvironment,
@@ -32,6 +38,8 @@ import type {
WorkflowState,
} from '@/lib/logs/types'
import type { SerializableExecutionState } from '@/executor/execution/types'
import type { BlockLog } from '@/executor/types'
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
type TriggerData = Record<string, unknown> & {
correlation?: NonNullable<ExecutionTrigger['data']>['correlation']
@@ -83,11 +91,32 @@ function buildCompletedMarkerPersistenceQuery(params: {
/** Progress-marker and status writes on `workflow_execution_logs` use the exec pool. */
const execDb = dbFor('exec')
const BLOCK_LOG_PROJECTION_BATCH_SIZE = 64
function structuralBlockLog(log: BlockLog): BlockLog {
const {
input: _input,
output: _output,
error: _error,
childTraceSpans: _childTraceSpans,
...structural
} = log
return structural
}
const logger = createLogger('LoggingSession')
type CompletionAttempt = 'complete' | 'error' | 'cancelled' | 'paused'
export interface SecretSafeDisplayContent {
input?: unknown
output?: unknown
error?: string
text?: string
chunk?: string
clearLiveDisplay?: true
}
export interface SessionStartParams {
userId?: string
/** Explicit initiating actor for callers that do not populate `userId`. */
@@ -120,12 +149,14 @@ export interface SessionErrorCompleteParams {
}
traceSpans?: TraceSpan[]
skipCost?: boolean
executionState?: SerializableExecutionState
}
export interface SessionCancelledParams {
endedAt?: string
totalDurationMs?: number
traceSpans?: TraceSpan[]
executionState?: SerializableExecutionState
}
export interface SessionPausedParams {
@@ -133,6 +164,7 @@ export interface SessionPausedParams {
totalDurationMs?: number
traceSpans?: TraceSpan[]
workflowInput?: any
executionState?: SerializableExecutionState
}
export interface LoggingSessionOptions {
@@ -166,6 +198,8 @@ export class LoggingSession {
private costOptions?: CostSummaryOptions
private pendingProgressWrites = new Set<Promise<void>>()
private postExecutionPromise: Promise<void> | null = null
private resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
private traceLargeValueAccess: LargeValueStoreContext = {}
constructor(
workflowId: string,
@@ -186,6 +220,187 @@ export class LoggingSession {
: undefined
}
/** Installs the run-scoped provenance used only at the terminal TraceSpan boundary. */
setResolvedSecretTraceRegistry(registry: ResolvedSecretTraceRegistry): void {
this.resolvedSecretTraceRegistry = registry
}
/** Adds the trusted execution-ref scope needed to rewrite offloaded trace content. */
setTraceLargeValueAccess(context: LargeValueStoreContext): void {
this.traceLargeValueAccess = context
}
private getSecretProjectionStore(): LargeValueStoreContext {
return {
...this.traceLargeValueAccess,
workspaceId: this.environment?.workspaceId,
workflowId: this.workflowId,
executionId: this.executionId,
userId: this.actorUserId ?? this.environment?.userId,
}
}
private async projectRawTraceSpans(traceSpans: TraceSpan[]): Promise<TraceSpan[]> {
return projectTraceSpansForSecrets(traceSpans, {
registry: this.resolvedSecretTraceRegistry,
store: this.getSecretProjectionStore(),
})
}
/**
* Produces a display-only copy of known observability content through the same
* projector used for persisted TraceSpans. Runtime values and callback payloads
* remain untouched, and an unavailable projection yields no content fields.
*/
async projectDisplayContent(
content: SecretSafeDisplayContent
): Promise<SecretSafeDisplayContent> {
try {
const envelope: Record<string, unknown> = {}
for (const key of ['input', 'output', 'error', 'text', 'chunk'] as const) {
if (Object.hasOwn(content, key)) envelope[key] = content[key]
}
const now = new Date().toISOString()
const [projectedSpan] = await this.projectRawTraceSpans([
{
id: 'secret-safe-display-projection',
name: 'Display Projection',
type: 'display',
duration: 0,
startTime: now,
endTime: now,
output: envelope,
},
])
const projected = this.readProjectedDisplayContent(projectedSpan?.output)
return this.shouldClearLiveDisplay() ? { ...projected, clearLiveDisplay: true } : projected
} catch {
logger.warn('Display secret projection failed; omitting display content')
return {}
}
}
private readProjectedDisplayContent(
projectedEnvelope: TraceSpan['output'] | undefined
): SecretSafeDisplayContent {
if (!projectedEnvelope) return {}
const projected: SecretSafeDisplayContent = {}
if (Object.hasOwn(projectedEnvelope, 'input')) projected.input = projectedEnvelope.input
if (Object.hasOwn(projectedEnvelope, 'output')) projected.output = projectedEnvelope.output
if (typeof projectedEnvelope.error === 'string') projected.error = projectedEnvelope.error
if (typeof projectedEnvelope.text === 'string') projected.text = projectedEnvelope.text
if (typeof projectedEnvelope.chunk === 'string') projected.chunk = projectedEnvelope.chunk
return projected
}
/**
* Projects terminal reconciliation logs without changing the executor-owned
* BlockLogs. Child traces use the identical TraceSpan projection boundary.
*/
async projectBlockLogsForDisplay(blockLogs: BlockLog[]): Promise<SecretSafeBlockLog[]> {
const now = new Date().toISOString()
const displayLogs: SecretSafeBlockLog[] = []
const clearLiveDisplay = this.shouldClearLiveDisplay()
for (let offset = 0; offset < blockLogs.length; offset += BLOCK_LOG_PROJECTION_BATCH_SIZE) {
const batch = blockLogs.slice(offset, offset + BLOCK_LOG_PROJECTION_BATCH_SIZE)
let projectedLogs: TraceSpan[]
try {
projectedLogs = await this.projectRawTraceSpans(
batch.map((log, index) => ({
id: `secret-safe-block-log-${offset + index}`,
name: 'Block Log Display Projection',
type: 'display',
duration: 0,
startTime: now,
endTime: now,
output: {
...(log.input !== undefined ? { input: log.input } : {}),
...(log.output !== undefined ? { output: log.output } : {}),
...(log.error !== undefined ? { error: log.error } : {}),
},
...(log.childTraceSpans ? { children: log.childTraceSpans } : {}),
}))
)
} catch {
logger.warn('Block-log secret projection failed; retaining structural logs only')
displayLogs.push(...batch.map(structuralBlockLog))
continue
}
for (let index = 0; index < batch.length; index += 1) {
const log = batch[index]
const display = this.readProjectedDisplayContent(projectedLogs[index]?.output)
displayLogs.push({
...structuralBlockLog(log),
...(clearLiveDisplay ? { clearLiveDisplay: true as const } : {}),
...(Object.hasOwn(display, 'input')
? { input: display.input as Record<string, unknown> }
: {}),
...(Object.hasOwn(display, 'output')
? { output: display.output as BlockLog['output'] }
: {}),
...(display.error !== undefined ? { error: display.error } : {}),
...(projectedLogs[index]?.children
? { childTraceSpans: projectedLogs[index].children }
: {}),
})
}
}
return displayLogs
}
/**
* Live deltas may split one literal across multiple events. Once provenance is
* active (or incomplete), suppress their display copy instead of attempting a
* per-chunk replacement that could miss the split value.
*/
async projectLiveDisplayText(
field: 'text' | 'chunk',
value: string
): Promise<SecretSafeDisplayContent> {
if (
!this.resolvedSecretTraceRegistry?.isComplete() ||
this.resolvedSecretTraceRegistry.getActiveMatches().length > 0
) {
return { clearLiveDisplay: true }
}
return this.projectDisplayContent({ [field]: value })
}
private shouldClearLiveDisplay(): boolean {
return (
!this.resolvedSecretTraceRegistry?.isComplete() ||
this.resolvedSecretTraceRegistry.getActiveMatches().length > 0
)
}
private async projectTraceSpans(traceSpans: TraceSpan[]): Promise<TraceSpan[]> {
const sourceTraceSpans = await executionLogger.loadTraceSpansForProjection({
executionId: this.executionId,
workflowId: this.workflowId,
workspaceId: this.environment?.workspaceId ?? null,
traceSpans,
isResume: this.isResume,
})
const secretSafeTraceSpans = await this.projectRawTraceSpans(sourceTraceSpans)
const preparedTraceSpans = await executionLogger.prepareTraceSpansForProjection({
executionId: this.executionId,
workflowId: this.workflowId,
workspaceId: this.environment?.workspaceId ?? null,
userId: this.actorUserId ?? this.environment?.userId,
traceSpans: secretSafeTraceSpans,
})
const invariantSafeTraceSpans = await enforceTraceSpanSecretInvariant(preparedTraceSpans, {
registry: this.resolvedSecretTraceRegistry,
store: this.getSecretProjectionStore(),
})
return invariantSafeTraceSpans
}
async onBlockStart(
blockId: string,
blockName: string,
@@ -303,6 +518,7 @@ export class LoggingSession {
level?: 'info' | 'error'
status?: 'completed' | 'failed' | 'cancelled' | 'pending'
}): Promise<void> {
const executionState = this.withResolvedSecretTraceProvenance(params.executionState)
await executionLogger.completeWorkflowExecution({
executionId: this.executionId,
endedAt: params.endedAt,
@@ -311,7 +527,7 @@ export class LoggingSession {
finalOutput: params.finalOutput,
traceSpans: params.traceSpans,
workflowInput: params.workflowInput,
executionState: params.executionState,
executionState,
finalizationPath: params.finalizationPath,
completionFailure: params.completionFailure,
isResume: this.isResume,
@@ -337,6 +553,27 @@ export class LoggingSession {
}
}
private withResolvedSecretTraceProvenance(
executionState?: SerializableExecutionState
): SerializableExecutionState | undefined {
if (!this.resolvedSecretTraceRegistry) return executionState
const resolvedSecretTraceProvenance = this.resolvedSecretTraceRegistry.exportProvenance()
if (executionState) {
return { ...executionState, resolvedSecretTraceProvenance }
}
return {
blockStates: {},
executedBlocks: [],
blockLogs: [],
decisions: { router: {}, condition: {} },
completedLoops: [],
activeExecutionPath: [],
resolvedSecretTraceProvenance,
}
}
async onBlockComplete(
blockId: string,
blockName: string,
@@ -371,6 +608,14 @@ export class LoggingSession {
} = params
this.actorUserId = billingAttribution?.actorUserId ?? actorUserId ?? userId ?? null
this.billingAttribution = billingAttribution
if (!this.resolvedSecretTraceRegistry) {
const scopeUserId = userId ?? this.actorUserId
this.resolvedSecretTraceRegistry = new ResolvedSecretTraceRegistry(
[],
scopeUserId ? { userId: scopeUserId, workspaceId } : undefined
)
if (skipLogCreation) this.resolvedSecretTraceRegistry.markIncomplete()
}
try {
this.trigger = createTriggerObject(this.triggerType, triggerData)
@@ -419,35 +664,38 @@ export class LoggingSession {
}
this.completing = true
const { endedAt, totalDurationMs, finalOutput, traceSpans, workflowInput, executionState } =
params
const { endedAt, totalDurationMs, workflowInput, executionState } = params
const finalOutput = params.finalOutput || {}
const rawTraceSpans = params.traceSpans || []
try {
const costSummary = calculateCostSummary(traceSpans || [], this.costOptions)
const costSummary = calculateCostSummary(rawTraceSpans, this.costOptions)
const endTime = endedAt || new Date().toISOString()
const duration = totalDurationMs || 0
const hasErrors = traceSpansIndicateFailure(rawTraceSpans)
const traceSpans = await this.projectTraceSpans(rawTraceSpans)
await this.completeExecutionWithFinalization({
endedAt: endTime,
totalDurationMs: duration,
costSummary,
finalOutput: finalOutput || {},
traceSpans: traceSpans || [],
finalOutput,
traceSpans,
workflowInput,
executionState,
finalizationPath: 'completed',
level: hasErrors ? 'error' : 'info',
status: hasErrors ? 'failed' : 'completed',
})
this.completed = true
if (traceSpans && traceSpans.length > 0) {
if (traceSpans.length > 0) {
try {
const { PlatformEvents, createOTelSpansForWorkflowExecution } = await import(
'@/lib/core/telemetry'
)
const hasErrors = traceSpansIndicateFailure(traceSpans)
PlatformEvents.workflowExecuted({
workflowId: this.workflowId,
durationMs: duration,
@@ -513,13 +761,14 @@ export class LoggingSession {
return
}
const { endedAt, totalDurationMs, error, traceSpans, skipCost } = params
const { endedAt, totalDurationMs, error, skipCost } = params
const rawTraceSpans = params.traceSpans || []
const endTime = endedAt ? new Date(endedAt) : new Date()
const durationMs = typeof totalDurationMs === 'number' ? totalDurationMs : 0
const startTime = new Date(endTime.getTime() - Math.max(1, durationMs))
const hasProvidedSpans = Array.isArray(traceSpans) && traceSpans.length > 0
const hasProvidedSpans = rawTraceSpans.length > 0
// calculateCostSummary([]) / (undefined) already returns the base-charge
// summary, so the no-spans branch needs no separate literal.
@@ -535,7 +784,7 @@ export class LoggingSession {
models: {},
charges: {},
}
: calculateCostSummary(traceSpans, this.costOptions)
: calculateCostSummary(rawTraceSpans, this.costOptions)
const message = error?.message || 'Run failed before starting blocks'
@@ -551,7 +800,7 @@ export class LoggingSession {
output: { error: message },
}
const spans = hasProvidedSpans ? traceSpans : [errorSpan]
const spans = await this.projectTraceSpans(hasProvidedSpans ? rawTraceSpans : [errorSpan])
await this.completeExecutionWithFinalization({
endedAt: endTime.toISOString(),
@@ -559,6 +808,7 @@ export class LoggingSession {
costSummary,
finalOutput: { error: message },
traceSpans: spans,
executionState: params.executionState,
level: 'error',
status: 'failed',
finalizationPath: 'force_failed',
@@ -578,7 +828,6 @@ export class LoggingSession {
trigger: this.triggerType,
blocksExecuted: spans.length,
hasErrors: true,
errorMessage: message,
})
createOTelSpansForWorkflowExecution({
@@ -591,7 +840,6 @@ export class LoggingSession {
endTime: endTime.toISOString(),
totalDurationMs: Math.max(1, durationMs),
status: 'error',
error: message,
})
} catch (_e) {
// Silently fail
@@ -622,7 +870,8 @@ export class LoggingSession {
this.completing = true
try {
const { endedAt, totalDurationMs, traceSpans } = params
const { endedAt, totalDurationMs } = params
const rawTraceSpans = params.traceSpans || []
const endTime = endedAt ? new Date(endedAt) : new Date()
const durationMs = typeof totalDurationMs === 'number' ? totalDurationMs : 0
@@ -646,14 +895,16 @@ export class LoggingSession {
// calculateCostSummary handles empty/undefined spans by returning the
// base-charge summary, so no separate no-spans literal is needed.
const costSummary = calculateCostSummary(traceSpans, this.costOptions)
const costSummary = calculateCostSummary(rawTraceSpans, this.costOptions)
const traceSpans = await this.projectTraceSpans(rawTraceSpans)
await this.completeExecutionWithFinalization({
endedAt: endTime.toISOString(),
totalDurationMs: Math.max(1, durationMs),
costSummary,
finalOutput: { cancelled: true },
traceSpans: traceSpans || [],
traceSpans,
executionState: params.executionState,
finalizationPath: 'cancelled',
status: 'cancelled',
})
@@ -669,11 +920,11 @@ export class LoggingSession {
durationMs: Math.max(1, durationMs),
status: 'cancelled',
trigger: this.triggerType,
blocksExecuted: traceSpans?.length || 0,
blocksExecuted: traceSpans.length,
hasErrors: false,
})
if (traceSpans && traceSpans.length > 0) {
if (traceSpans.length > 0) {
const startTime = new Date(endTime.getTime() - Math.max(1, durationMs))
createOTelSpansForWorkflowExecution({
workflowId: this.workflowId,
@@ -716,7 +967,8 @@ export class LoggingSession {
this.completing = true
try {
const { endedAt, totalDurationMs, traceSpans, workflowInput } = params
const { endedAt, totalDurationMs, workflowInput } = params
const rawTraceSpans = params.traceSpans || []
const endTime = endedAt ? new Date(endedAt) : new Date()
const durationMs = typeof totalDurationMs === 'number' ? totalDurationMs : 0
@@ -740,15 +992,17 @@ export class LoggingSession {
// calculateCostSummary handles empty/undefined spans by returning the
// base-charge summary, so no separate no-spans literal is needed.
const costSummary = calculateCostSummary(traceSpans, this.costOptions)
const costSummary = calculateCostSummary(rawTraceSpans, this.costOptions)
const traceSpans = await this.projectTraceSpans(rawTraceSpans)
await this.completeExecutionWithFinalization({
endedAt: endTime.toISOString(),
totalDurationMs: Math.max(1, durationMs),
costSummary,
finalOutput: { paused: true },
traceSpans: traceSpans || [],
traceSpans,
workflowInput,
executionState: params.executionState,
finalizationPath: 'paused',
status: 'pending',
})
@@ -764,12 +1018,12 @@ export class LoggingSession {
durationMs: Math.max(1, durationMs),
status: 'paused',
trigger: this.triggerType,
blocksExecuted: traceSpans?.length || 0,
blocksExecuted: traceSpans.length,
hasErrors: false,
totalCost: costSummary.totalCost || 0,
})
if (traceSpans && traceSpans.length > 0) {
if (traceSpans.length > 0) {
const startTime = new Date(endTime.getTime() - Math.max(1, durationMs))
createOTelSpansForWorkflowExecution({
workflowId: this.workflowId,
@@ -947,6 +1201,7 @@ export class LoggingSession {
isError: false,
finalizationPath: 'fallback_completed',
finalOutput: params.finalOutput || {},
executionState: params.executionState,
})
}
}
@@ -976,6 +1231,7 @@ export class LoggingSession {
finalOutput: {
error: params?.error?.message || `Execution failed to store trace spans: ${errorMsg}`,
},
executionState: params?.executionState,
status: 'failed',
})
}
@@ -1005,6 +1261,7 @@ export class LoggingSession {
isError: false,
finalizationPath: 'cancelled',
finalOutput: { cancelled: true },
executionState: params?.executionState,
status: 'cancelled',
})
}
@@ -1032,6 +1289,7 @@ export class LoggingSession {
isError: false,
finalizationPath: 'paused',
finalOutput: { paused: true },
executionState: params?.executionState,
status: 'pending',
})
}
@@ -1128,6 +1386,7 @@ export class LoggingSession {
isError: boolean
finalizationPath: ExecutionFinalizationPath
finalOutput?: Record<string, unknown>
executionState?: SerializableExecutionState
status?: 'completed' | 'failed' | 'cancelled' | 'pending'
}): Promise<void> {
if (this.completed || this.completing) {
@@ -1155,6 +1414,7 @@ export class LoggingSession {
costSummary,
finalOutput,
traceSpans: [],
executionState: params.executionState,
finalizationPath: params.finalizationPath,
completionFailure: params.errorMessage,
level: params.isError ? 'error' : 'info',
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,140 @@
/**
* @vitest-environment node
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { decryptSecretMock } = vi.hoisted(() => ({
decryptSecretMock: vi.fn(),
}))
vi.mock('@/lib/core/security/encryption', () => ({
decryptSecret: decryptSecretMock,
}))
import { projectExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
const CONTEXT = {
workspaceId: 'workspace-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
userId: 'user-1',
}
beforeEach(() => {
vi.clearAllMocks()
decryptSecretMock.mockResolvedValue({ decrypted: '1234' })
})
describe('projectExecutionDataForDisplay', () => {
it('projects persisted output, input, errors, and spans from trusted provenance', async () => {
const executionData = {
finalOutput: { result: 1234, derived: 1239 },
workflowInput: { nested: { token: 'prefix-1234-suffix' } },
completionFailure: 'Function failed with 1234',
errorDetails: { blockId: 'function-1', error: 'Invalid token 1234' },
traceSpans: [
{
id: 'span-1',
name: 'Function 1',
type: 'function',
duration: 1,
startTime: '2026-07-31T00:00:00.000Z',
endTime: '2026-07-31T00:00:00.001Z',
output: { result: 1234 },
},
],
executionState: {
blockStates: { 'function-1': { output: { result: 1234 } } },
resolvedSecretTraceProvenance: {
version: 1 as const,
complete: true,
entries: [{ name: 'OPENAI_API_KEY', encryptedValue: 'ciphertext' }],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
},
},
}
const displayData = await projectExecutionDataForDisplay(executionData, CONTEXT)
expect(displayData.finalOutput).toEqual({
result: '{{OPENAI_API_KEY}}',
derived: 1239,
})
expect(displayData.workflowInput).toEqual({
nested: { token: 'prefix-{{OPENAI_API_KEY}}-suffix' },
})
expect(displayData.completionFailure).toBe('Function failed with {{OPENAI_API_KEY}}')
expect(displayData.errorDetails).toEqual({
blockId: 'function-1',
error: 'Invalid token {{OPENAI_API_KEY}}',
})
expect(displayData.traceSpans).toEqual([
expect.objectContaining({ output: { result: '{{OPENAI_API_KEY}}' } }),
])
expect(displayData).not.toHaveProperty('executionState')
expect(executionData.finalOutput).toEqual({ result: 1234, derived: 1239 })
expect(executionData.executionState.resolvedSecretTraceProvenance.entries).toEqual([
{ name: 'OPENAI_API_KEY', encryptedValue: 'ciphertext' },
])
expect(JSON.stringify(displayData)).not.toContain('1234')
})
it('omits log content and keeps only structural traces without trusted provenance', async () => {
const displayData = await projectExecutionDataForDisplay(
{
finalOutput: { result: 'unknown-secret' },
workflowInput: { token: 'unknown-secret' },
completionFailure: 'unknown-secret',
traceSpans: [
{
id: 'span-1',
name: 'Function 1',
type: 'function',
duration: 1,
startTime: '2026-07-31T00:00:00.000Z',
endTime: '2026-07-31T00:00:00.001Z',
output: { result: 'unknown-secret' },
},
],
},
CONTEXT
)
expect(displayData).not.toHaveProperty('finalOutput')
expect(displayData).not.toHaveProperty('workflowInput')
expect(displayData).not.toHaveProperty('completionFailure')
expect(displayData.traceSpans).toEqual([
expect.not.objectContaining({ output: expect.anything() }),
])
})
it('preserves direct literals when trusted provenance has no activated secrets', async () => {
const displayData = await projectExecutionDataForDisplay(
{
finalOutput: { result: 'direct-literal' },
executionState: {
resolvedSecretTraceProvenance: {
version: 1,
complete: true,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
},
},
},
CONTEXT
)
expect(displayData.finalOutput).toEqual({ result: 'direct-literal' })
})
it('omits malformed trace content even when it was present on the stored row', async () => {
const displayData = await projectExecutionDataForDisplay(
{
traceSpans: { output: 'unsafe' },
},
CONTEXT
)
expect(displayData).not.toHaveProperty('traceSpans')
})
})
+119 -1
View File
@@ -1,7 +1,14 @@
import { createLogger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { omit } from '@sim/utils/object'
import { isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
import { materializeLargeValueRef, storeLargeValue } from '@/lib/execution/payloads/store'
import { projectTraceSpansForSecrets } from '@/lib/logs/execution/trace-secret-projection'
import type { TraceSpan } from '@/lib/logs/types'
import {
isResolvedSecretTraceProvenanceV1,
ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
const logger = createLogger('TraceStore')
@@ -26,10 +33,11 @@ const INLINE_MARKER_KEYS = ['hasTraceSpans', 'traceSpanCount'] as const
* Read-path context. Resolves an externalized payload by storage key, authorized
* via the (already-authorized) workspace — no owner needed.
*/
interface TraceStoreReadContext {
export interface TraceStoreReadContext {
workspaceId: string | null
workflowId: string | null
executionId: string
userId?: string
}
/**
@@ -69,6 +77,14 @@ export function stripSpanCosts(spans: unknown): void {
}
}
/** Creates a persistence-owned span tree with per-span cost fields removed. */
export function copyTraceSpansWithoutCosts(spans?: TraceSpan[]): TraceSpan[] | undefined {
return spans?.map(({ cost: _cost, children, ...span }) => ({
...span,
...(children ? { children: copyTraceSpansWithoutCosts(children) } : {}),
}))
}
/**
* Externalizes heavy `execution_data` to object storage as a single large value
* (reusing the execution-context large-value store + its reference/dependency/GC
@@ -177,3 +193,105 @@ export async function materializeExecutionData(
return markers
}
}
const LOG_DISPLAY_CONTENT_KEYS = [
'finalOutput',
'workflowInput',
'blockInput',
'blockExecutions',
'error',
'errorDetails',
'completionFailure',
'message',
] as const
const LOG_DISPLAY_PROJECTION_SPAN_ID = 'secret-safe-log-display-projection'
/**
* Materializes trusted execution data and returns its log-facing projection.
* Functional readers must continue using {@link materializeExecutionData}.
*/
export async function materializeExecutionDataForDisplay(
executionData: Record<string, unknown> | null | undefined,
context: TraceStoreReadContext
): Promise<Record<string, unknown>> {
const materialized = await materializeExecutionData(executionData, context)
return projectExecutionDataForDisplay(materialized, context)
}
/**
* Projects execution-log content with the encrypted provenance saved by the
* trusted executor. Missing or malformed provenance deliberately yields a
* structural-only log instead of returning content that cannot be proven safe.
*/
export async function projectExecutionDataForDisplay(
executionData: Record<string, unknown>,
context: TraceStoreReadContext
): Promise<Record<string, unknown>> {
const executionState =
executionData.executionState &&
typeof executionData.executionState === 'object' &&
!Array.isArray(executionData.executionState)
? (executionData.executionState as Record<string, unknown>)
: undefined
const provenance = executionState?.resolvedSecretTraceProvenance
let registry: ResolvedSecretTraceRegistry | undefined
if (isResolvedSecretTraceProvenanceV1(provenance)) {
registry = new ResolvedSecretTraceRegistry([], provenance.scope)
await registry.importProvenance(provenance, { trusted: true })
}
const envelope: Record<string, unknown> = {}
for (const key of LOG_DISPLAY_CONTENT_KEYS) {
if (Object.hasOwn(executionData, key)) envelope[key] = executionData[key]
}
const now = new Date().toISOString()
const syntheticSpan: TraceSpan = {
id: LOG_DISPLAY_PROJECTION_SPAN_ID,
name: 'Log Display Projection',
type: 'display',
duration: 0,
startTime: now,
endTime: now,
output: envelope,
}
const sourceTraceSpans = Array.isArray(executionData.traceSpans)
? (executionData.traceSpans as TraceSpan[])
: []
const projectedSpans = await projectTraceSpansForSecrets([syntheticSpan, ...sourceTraceSpans], {
registry,
allowLargeValueWrites: false,
store: {
workspaceId: context.workspaceId ?? undefined,
workflowId: context.workflowId ?? undefined,
executionId: context.executionId,
userId: context.userId,
trackReference: false,
},
})
const displayData = omit(executionData, [
...LOG_DISPLAY_CONTENT_KEYS,
'executionState',
'traceSpans',
]) as Record<string, unknown>
const projectedEnvelope = projectedSpans.find(
(span) => span.id === LOG_DISPLAY_PROJECTION_SPAN_ID
)?.output
if (projectedEnvelope) {
for (const key of LOG_DISPLAY_CONTENT_KEYS) {
if (Object.hasOwn(projectedEnvelope, key)) displayData[key] = projectedEnvelope[key]
}
}
if (Array.isArray(executionData.traceSpans)) {
displayData.traceSpans = projectedSpans.filter(
(span) => span.id !== LOG_DISPLAY_PROJECTION_SPAN_ID
)
}
return displayData
}
+17 -4
View File
@@ -15,7 +15,7 @@ import {
pickLatestCompletedMarker,
pickLatestStartedMarker,
} from '@/lib/logs/execution/progress-markers'
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils'
type LookupColumn = 'id' | 'executionId'
@@ -170,9 +170,14 @@ export async function fetchLogDetail({
// Trace spans / heavy execution data may live in object storage; resolve the
// pointer here (no-op for inline / pre-externalization rows).
const executionData = await materializeExecutionData(
const executionData = await materializeExecutionDataForDisplay(
log.executionData as Record<string, unknown> | null,
{ workspaceId, workflowId: log.workflowId, executionId: log.executionId }
{
workspaceId,
workflowId: log.workflowId,
executionId: log.executionId,
userId,
}
)
const liveMarkers =
@@ -248,7 +253,15 @@ export async function fetchLogDetail({
const jobLog = jobRows[0]
if (!jobLog) return null
const execData = (jobLog.executionData as Record<string, unknown> | null) ?? {}
const execData = await materializeExecutionDataForDisplay(
jobLog.executionData as Record<string, unknown> | null,
{
workspaceId,
workflowId: null,
executionId: jobLog.executionId,
userId,
}
)
return {
id: jobLog.id,
workflowId: null,
+16
View File
@@ -433,6 +433,22 @@ export interface SnapshotCreationResult {
}
export interface ExecutionLoggerService {
loadTraceSpansForProjection(params: {
executionId: string
workflowId: string
workspaceId: string | null
traceSpans: TraceSpan[]
isResume?: boolean
}): Promise<TraceSpan[]>
prepareTraceSpansForProjection(params: {
executionId: string
workflowId: string
workspaceId: string | null
userId?: string | null
traceSpans: TraceSpan[]
}): Promise<TraceSpan[]>
startWorkflowExecution(params: {
workflowId: string
workspaceId: string
+8
View File
@@ -74,6 +74,7 @@ export class McpClient {
private authProvider?: McpClientOptions['authProvider']
private isConnected = false
private closeGuardedTransport?: () => Promise<void>
private readonly resolvedSecretTraceProvenance?: McpClientOptions['resolvedSecretTraceProvenance']
constructor(options: McpClientOptions) {
this.config = options.config
@@ -84,6 +85,7 @@ export class McpClient {
}
this.onToolsChanged = options.onToolsChanged
this.authProvider = options.authProvider
this.resolvedSecretTraceProvenance = options.resolvedSecretTraceProvenance
const resolvedIP = options.resolvedIP
this.connectionStatus = { connected: false }
@@ -133,6 +135,12 @@ export class McpClient {
}
}
getResolvedSecretTraceProvenance(): McpClientOptions['resolvedSecretTraceProvenance'] {
return this.resolvedSecretTraceProvenance
? structuredClone(this.resolvedSecretTraceProvenance)
: undefined
}
/**
* Initialize connection to MCP server.
* If an `onToolsChanged` callback was provided, registers a notification handler
+182
View File
@@ -0,0 +1,182 @@
/**
* @vitest-environment node
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
const { mockGetEffectiveEnvironmentSnapshot } = vi.hoisted(() => ({
mockGetEffectiveEnvironmentSnapshot: vi.fn(),
}))
vi.mock('@/lib/environment/utils', () => ({
getEffectiveEnvironmentSnapshot: mockGetEffectiveEnvironmentSnapshot,
}))
import { resolveMcpConfigEnvVars } from '@/lib/mcp/resolve-config'
const BASE_CONFIG = {
id: 'server-1',
name: 'Server',
transport: 'streamable-http' as const,
url: 'https://{{MCP_HOST}}/mcp',
headers: {
Authorization: 'Bearer {{MCP_TOKEN}}',
'X-Direct': 'literal-secret',
},
}
describe('resolveMcpConfigEnvVars secret provenance', () => {
beforeEach(() => {
vi.clearAllMocks()
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
personalEncrypted: {
MCP_HOST: 'personal-host-encrypted',
MCP_TOKEN: 'personal-token-encrypted',
UNUSED: 'unused-encrypted',
},
workspaceEncrypted: { MCP_TOKEN: 'workspace-token-encrypted' },
personalDecrypted: {
MCP_HOST: 'api.example.com',
MCP_TOKEN: 'personal-token',
UNUSED: 'literal-secret',
},
workspaceDecrypted: { MCP_TOKEN: 'workspace-token' },
conflicts: ['MCP_TOKEN'],
decryptionFailures: [],
})
})
it('returns encrypted provenance only for successful {{NAME}} substitutions', async () => {
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
expect(result.config.url).toBe('https://api.example.com/mcp')
expect(result.config.headers).toEqual({
Authorization: 'Bearer workspace-token',
'X-Direct': 'literal-secret',
})
expect(result.resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: true,
entries: [
{ name: 'MCP_HOST', encryptedValue: 'personal-host-encrypted' },
{ name: 'MCP_TOKEN', encryptedValue: 'workspace-token-encrypted' },
],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
expect(JSON.stringify(result.resolvedSecretTraceProvenance)).not.toContain('workspace-token"')
expect(JSON.stringify(result.resolvedSecretTraceProvenance)).not.toContain('literal-secret')
})
it('marks provenance incomplete when a resolved value has no encrypted catalog entry', async () => {
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
personalEncrypted: {},
workspaceEncrypted: {},
personalDecrypted: { MCP_HOST: 'api.example.com', MCP_TOKEN: 'token' },
workspaceDecrypted: {},
conflicts: [],
decryptionFailures: [],
})
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
expect(result.resolvedSecretTraceProvenance).toEqual({
version: 1,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('does not let an unused configured-secret failure affect invocation provenance', async () => {
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
personalEncrypted: {
MCP_HOST: 'host-encrypted',
MCP_TOKEN: 'token-encrypted',
UNUSED: 'broken-encrypted',
},
workspaceEncrypted: {},
personalDecrypted: {
MCP_HOST: 'api.example.com',
MCP_TOKEN: 'token',
UNUSED: '',
},
workspaceDecrypted: {},
conflicts: [],
decryptionFailures: ['UNUSED'],
})
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
expect(result.resolvedSecretTraceProvenance.complete).toBe(true)
expect(result.resolvedSecretTraceProvenance.entries).toHaveLength(2)
})
it('reports successful substitutions before strict missing-reference failure', async () => {
const onProvenance = vi.fn()
const config = {
...BASE_CONFIG,
headers: {
Authorization: 'Bearer {{MCP_TOKEN}}',
'X-Missing': '{{NOT_CONFIGURED}}',
},
}
await expect(
resolveMcpConfigEnvVars(config, 'user-1', 'workspace-1', {
onResolvedSecretTraceProvenance: onProvenance,
})
).rejects.toThrow('NOT_CONFIGURED')
expect(onProvenance).toHaveBeenCalledWith({
version: 1,
complete: true,
entries: [
{ name: 'MCP_HOST', encryptedValue: 'personal-host-encrypted' },
{ name: 'MCP_TOKEN', encryptedValue: 'workspace-token-encrypted' },
],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
})
})
it('reports incomplete provenance when the Secrets catalog cannot be loaded', async () => {
const onProvenance = vi.fn()
mockGetEffectiveEnvironmentSnapshot.mockRejectedValueOnce(new Error('database unavailable'))
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1', {
onResolvedSecretTraceProvenance: onProvenance,
})
const expected = {
version: 1 as const,
complete: false,
entries: [],
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
}
expect(result.resolvedSecretTraceProvenance).toEqual(expected)
expect(onProvenance).toHaveBeenCalledWith(expected)
expect(result.config).toEqual(BASE_CONFIG)
})
it('uses one atomic cached snapshot for runtime values and encrypted provenance', async () => {
mockGetEffectiveEnvironmentSnapshot.mockResolvedValueOnce({
personalEncrypted: { MCP_HOST: 'old-host-ciphertext', MCP_TOKEN: 'old-token-ciphertext' },
workspaceEncrypted: {},
personalDecrypted: { MCP_HOST: 'old.example.com', MCP_TOKEN: 'old-token' },
workspaceDecrypted: {},
conflicts: [],
decryptionFailures: [],
})
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
expect(mockGetEffectiveEnvironmentSnapshot).toHaveBeenCalledOnce()
expect(mockGetEffectiveEnvironmentSnapshot).toHaveBeenCalledWith('user-1', 'workspace-1')
expect(result.config).toMatchObject({
url: 'https://old.example.com/mcp',
headers: expect.objectContaining({ Authorization: 'Bearer old-token' }),
})
expect(result.resolvedSecretTraceProvenance.entries).toEqual([
{ name: 'MCP_HOST', encryptedValue: 'old-host-ciphertext' },
{ name: 'MCP_TOKEN', encryptedValue: 'old-token-ciphertext' },
])
})
})
+57 -7
View File
@@ -5,15 +5,26 @@
*/
import { createLogger } from '@sim/logger'
import { getEffectiveDecryptedEnv } from '@/lib/environment/utils'
import { getErrorMessage } from '@sim/utils/errors'
import {
type EnvironmentResolutionSnapshot,
getEffectiveEnvironmentSnapshot,
} from '@/lib/environment/utils'
import type { McpServerConfig } from '@/lib/mcp/types'
import { resolveEnvVarReferences } from '@/executor/utils/reference-validation'
import {
createIncompleteResolvedSecretTraceRegistry,
createResolvedSecretTraceRegistry,
type ResolvedSecretTraceProvenanceV1,
type ResolvedSecretTraceRegistry,
} from '@/executor/utils/resolved-secret-trace-registry'
const logger = createLogger('McpResolveConfig')
export interface ResolveMcpConfigOptions {
/** If true, throws an error when env vars are missing. Default: true */
strict?: boolean
onResolvedSecretTraceProvenance?: (provenance: ResolvedSecretTraceProvenanceV1) => void
}
/**
@@ -31,22 +42,55 @@ export async function resolveMcpConfigEnvVars(
userId: string,
workspaceId?: string,
options: ResolveMcpConfigOptions = {}
): Promise<{ config: McpServerConfig; missingVars: string[] }> {
): Promise<{
config: McpServerConfig
missingVars: string[]
resolvedSecretTraceProvenance: ResolvedSecretTraceProvenanceV1
}> {
const { strict = true } = options
const allMissingVars: string[] = []
const scope = { userId, ...(workspaceId ? { workspaceId } : {}) }
let envVars: Record<string, string> = {}
let env: EnvironmentResolutionSnapshot
try {
envVars = await getEffectiveDecryptedEnv(userId, workspaceId)
env = await getEffectiveEnvironmentSnapshot(userId, workspaceId)
} catch (error) {
logger.error('Failed to fetch environment variables for MCP config:', error)
return { config, missingVars: [] }
const resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(scope)
const provenance = resolvedSecretTraceRegistry.exportProvenance()
options.onResolvedSecretTraceProvenance?.(provenance)
return {
config,
missingVars: [],
resolvedSecretTraceProvenance: provenance,
}
}
const envVars = { ...env.personalDecrypted, ...env.workspaceDecrypted }
let resolvedSecretTraceRegistry: ResolvedSecretTraceRegistry
try {
resolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
personalEncrypted: env.personalEncrypted,
workspaceEncrypted: env.workspaceEncrypted,
personalDecrypted: env.personalDecrypted,
workspaceDecrypted: env.workspaceDecrypted,
decryptionFailures: env.decryptionFailures,
scope,
})
} catch (error) {
logger.warn('Failed to build MCP trace secret catalog; provenance will be incomplete', {
error: getErrorMessage(error),
})
resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(scope)
}
const resolveValue = (value: string): string => {
const missingVars: string[] = []
const resolved = resolveEnvVarReferences(value, envVars, {
missingKeys: missingVars,
onResolved: (name, resolvedValue) => {
resolvedSecretTraceRegistry.recordResolved(name, resolvedValue)
},
}) as string
allMissingVars.push(...missingVars)
return resolved
@@ -66,7 +110,9 @@ export async function resolveMcpConfigEnvVars(
resolvedConfig.headers = resolvedHeaders
}
// Handle missing vars based on strict mode
const resolvedSecretTraceProvenance = resolvedSecretTraceRegistry.exportProvenance()
options.onResolvedSecretTraceProvenance?.(resolvedSecretTraceProvenance)
if (allMissingVars.length > 0) {
const uniqueMissing = Array.from(new Set(allMissingVars))
@@ -81,5 +127,9 @@ export async function resolveMcpConfigEnvVars(
})
}
return { config: resolvedConfig, missingVars: allMissingVars }
return {
config: resolvedConfig,
missingVars: allMissingVars,
resolvedSecretTraceProvenance,
}
}
+216 -2
View File
@@ -15,6 +15,7 @@ import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
const {
MockMcpClient,
mockCallTool,
mockListTools,
mockConnect,
mockDisconnect,
mockAcquire,
@@ -24,12 +25,24 @@ const {
poolClient,
} = vi.hoisted(() => {
const mockCallTool = vi.fn()
const mockListTools = vi.fn()
const mockConnect = vi.fn()
const mockDisconnect = vi.fn()
const mockRelease = vi.fn(async () => {})
const poolClient = { callTool: mockCallTool, disconnect: vi.fn() }
const poolClient = {
callTool: mockCallTool,
listTools: mockListTools,
disconnect: vi.fn(),
getResolvedSecretTraceProvenance: vi.fn(() => ({
version: 1 as const,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: 'user-1', workspaceId: 'ws-1' },
})),
}
return {
mockCallTool,
mockListTools,
mockConnect,
mockDisconnect,
mockRelease,
@@ -50,9 +63,10 @@ const {
connect: mockConnect,
disconnect: mockDisconnect,
callTool: mockCallTool,
listTools: vi.fn(async () => []),
listTools: mockListTools,
hasListChangedCapability: vi.fn(() => false),
onClose: vi.fn(),
getResolvedSecretTraceProvenance: poolClient.getResolvedSecretTraceProvenance,
})
}
}
@@ -117,6 +131,7 @@ describe('McpService connection reuse wiring', () => {
mockResolveEnvVars.mockImplementation(async (config: unknown) => ({ config }))
mockAcquire.mockResolvedValue({ client: poolClient, release: mockRelease })
mockCallTool.mockResolvedValue({ content: [] })
mockListTools.mockResolvedValue([])
})
afterAll(() => {
@@ -137,6 +152,161 @@ describe('McpService connection reuse wiring', () => {
expect(mockResolveEnvVars).not.toHaveBeenCalled()
})
it('emits the pooled connection provenance on every invocation', async () => {
const onProvenance = vi.fn()
await mcpService.executeTool(
USER_ID,
'server-1',
{ name: 'first', arguments: {} },
WORKSPACE_ID,
undefined,
onProvenance
)
await mcpService.executeTool(
USER_ID,
'server-1',
{ name: 'second', arguments: {} },
WORKSPACE_ID,
undefined,
onProvenance
)
expect(onProvenance).toHaveBeenCalledTimes(2)
expect(onProvenance).toHaveBeenNthCalledWith(1, {
version: 1,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
})
expect(onProvenance).toHaveBeenNthCalledWith(2, {
version: 1,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
})
})
it('emits cold-connection provenance once even though the connected client retains it', async () => {
const onProvenance = vi.fn()
const provenance = {
version: 1 as const,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
}
mockResolveEnvVars.mockImplementationOnce(
async (
config: unknown,
_userId: unknown,
_workspaceId: unknown,
options: {
onResolvedSecretTraceProvenance?: (value: typeof provenance) => void
}
) => {
options.onResolvedSecretTraceProvenance?.(provenance)
return { config, resolvedSecretTraceProvenance: provenance }
}
)
mockAcquire.mockImplementationOnce(
async ({ create }: { create: () => Promise<typeof poolClient> }) => ({
client: await create(),
release: mockRelease,
})
)
await mcpService.executeTool(
USER_ID,
'server-1',
{ name: 'do', arguments: {} },
WORKSPACE_ID,
undefined,
onProvenance
)
expect(mockResolveEnvVars).toHaveBeenCalledTimes(1)
expect(onProvenance).toHaveBeenCalledTimes(1)
expect(onProvenance).toHaveBeenCalledWith(provenance)
})
it('emits cold-connection provenance for tools/list', async () => {
const onProvenance = vi.fn()
const provenance = {
version: 1 as const,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
}
mockResolveEnvVars.mockImplementationOnce(
async (
config: unknown,
_userId: unknown,
_workspaceId: unknown,
options: {
onResolvedSecretTraceProvenance?: (value: typeof provenance) => void
}
) => {
options.onResolvedSecretTraceProvenance?.(provenance)
return { config, resolvedSecretTraceProvenance: provenance }
}
)
mockAcquire.mockImplementationOnce(
async ({ create }: { create: () => Promise<typeof poolClient> }) => ({
client: await create(),
release: mockRelease,
})
)
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
expect(mockResolveEnvVars).toHaveBeenCalledTimes(1)
expect(mockListTools).toHaveBeenCalledTimes(1)
expect(onProvenance).toHaveBeenCalledTimes(1)
expect(onProvenance).toHaveBeenCalledWith(provenance)
})
it('emits retained provenance on every warm-pool tools/list invocation', async () => {
const onProvenance = vi.fn()
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
expect(mockResolveEnvVars).not.toHaveBeenCalled()
expect(mockListTools).toHaveBeenCalledTimes(2)
expect(onProvenance).toHaveBeenCalledTimes(2)
expect(onProvenance).toHaveBeenNthCalledWith(1, {
version: 1,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
})
expect(onProvenance).toHaveBeenNthCalledWith(2, {
version: 1,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
})
})
it('reports incomplete provenance when a pooled tools/list client has no retained report', async () => {
const onProvenance = vi.fn()
const legacyClient = {
...poolClient,
getResolvedSecretTraceProvenance: undefined,
}
mockAcquire.mockResolvedValueOnce({ client: legacyClient, release: mockRelease })
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
expect(mockListTools).toHaveBeenCalledTimes(1)
expect(onProvenance).toHaveBeenCalledWith({
version: 1,
complete: false,
entries: [],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
})
})
it('bypasses the pool for calls carrying per-request headers', async () => {
await mcpService.executeTool(USER_ID, 'server-1', { name: 'do', arguments: {} }, WORKSPACE_ID, {
Authorization: 'Bearer per-call',
@@ -226,4 +396,48 @@ describe('McpService connection reuse wiring', () => {
expect(mockRelease).toHaveBeenCalledWith(true, false)
expect(mockAcquire).toHaveBeenCalledTimes(2)
})
it('reports both retained and rotated provenance when an auth retry rebuilds the client', async () => {
const staleProvenance = {
version: 1 as const,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token-v1' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
}
const rotatedProvenance = {
version: 1 as const,
complete: true,
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token-v2' }],
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
}
const staleClient = {
...poolClient,
getResolvedSecretTraceProvenance: vi.fn(() => staleProvenance),
}
const rotatedClient = {
...poolClient,
getResolvedSecretTraceProvenance: vi.fn(() => rotatedProvenance),
}
mockAcquire
.mockResolvedValueOnce({ client: staleClient, release: mockRelease })
.mockResolvedValueOnce({ client: rotatedClient, release: mockRelease })
mockCallTool
.mockRejectedValueOnce(new UnauthorizedError('stale key'))
.mockResolvedValueOnce({ content: [] })
const onProvenance = vi.fn()
await mcpService.executeTool(
USER_ID,
'server-1',
{ name: 'do', arguments: {} },
WORKSPACE_ID,
undefined,
onProvenance
)
expect(onProvenance.mock.calls.map(([provenance]) => provenance)).toEqual([
staleProvenance,
rotatedProvenance,
])
})
})
+141 -21
View File
@@ -42,6 +42,10 @@ import {
type McpTransport,
} from '@/lib/mcp/types'
import { MCP_CLIENT_CONSTANTS, MCP_CONSTANTS } from '@/lib/mcp/utils'
import {
isResolvedSecretTraceProvenanceV1,
type ResolvedSecretTraceProvenanceV1,
} from '@/executor/utils/resolved-secret-trace-registry'
const logger = createLogger('McpService')
@@ -55,6 +59,59 @@ function failureCacheKey(workspaceId: string, serverId: string): string {
const FAILURE_CACHE_SENTINEL: McpTool[] = []
type ResolvedSecretTraceProvenanceCallback = (provenance: ResolvedSecretTraceProvenanceV1) => void
function reportRetainedClientProvenance(
provenance: unknown,
userId: string,
workspaceId: string,
callback?: ResolvedSecretTraceProvenanceCallback
): void {
if (!callback) return
callback(
isResolvedSecretTraceProvenanceV1(provenance)
? provenance
: {
version: 1,
complete: false,
entries: [],
scope: { userId, workspaceId },
}
)
}
function isSameProvenance(
left: ResolvedSecretTraceProvenanceV1,
right: ResolvedSecretTraceProvenanceV1
): boolean {
if (
left.complete !== right.complete ||
left.scope?.userId !== right.scope?.userId ||
left.scope?.workspaceId !== right.scope?.workspaceId ||
left.entries.length !== right.entries.length
) {
return false
}
return left.entries.every((entry, index) => {
const other = right.entries[index]
return entry.name === other.name && entry.encryptedValue === other.encryptedValue
})
}
function createInvocationProvenanceReporter(
callback?: ResolvedSecretTraceProvenanceCallback
): ResolvedSecretTraceProvenanceCallback | undefined {
if (!callback) return undefined
let lastReported: ResolvedSecretTraceProvenanceV1 | undefined
return (provenance) => {
if (lastReported && isSameProvenance(lastReported, provenance)) return
lastReported = provenance
callback(provenance)
}
}
type DiscoveryOutcome =
| { kind: 'cached'; tools: McpTool[] }
| { kind: 'fetched'; tools: McpTool[] }
@@ -212,14 +269,25 @@ class McpService {
private async resolveConfigEnvVars(
config: McpServerConfig,
userId: string,
workspaceId?: string
): Promise<{ config: McpServerConfig; resolvedIP: string | null }> {
const { config: resolvedConfig } = await resolveMcpConfigEnvVars(config, userId, workspaceId, {
strict: true,
})
workspaceId?: string,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<{
config: McpServerConfig
resolvedIP: string | null
resolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceV1
}> {
const { config: resolvedConfig, resolvedSecretTraceProvenance } = await resolveMcpConfigEnvVars(
config,
userId,
workspaceId,
{
strict: true,
onResolvedSecretTraceProvenance,
}
)
validateMcpDomain(resolvedConfig.url)
const resolvedIP = await validateMcpServerSsrf(resolvedConfig.url)
return { config: resolvedConfig, resolvedIP }
return { config: resolvedConfig, resolvedIP, resolvedSecretTraceProvenance }
}
private async getServerConfig(
@@ -298,7 +366,8 @@ class McpService {
private async createClient(
config: McpServerConfig,
resolvedIP: string | null,
userId?: string
userId?: string,
resolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceV1
): Promise<McpClient> {
const securityPolicy = {
requireConsent: true,
@@ -312,6 +381,7 @@ class McpService {
config,
securityPolicy,
resolvedIP: resolvedIP ?? undefined,
resolvedSecretTraceProvenance,
})
await client.connect()
return client
@@ -343,6 +413,7 @@ class McpService {
securityPolicy,
authProvider,
resolvedIP: resolvedIP ?? undefined,
resolvedSecretTraceProvenance,
})
await client.connect()
return client
@@ -367,18 +438,24 @@ class McpService {
config: McpServerConfig,
userId: string,
workspaceId: string,
extraHeaders?: Record<string, string>
extraHeaders?: Record<string, string>,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): () => Promise<McpClient> {
return async () => {
const { config: resolvedConfig, resolvedIP } = await this.resolveConfigEnvVars(
const {
config: resolvedConfig,
resolvedIP,
resolvedSecretTraceProvenance,
} = await this.resolveConfigEnvVars(
config,
userId,
workspaceId
workspaceId,
onResolvedSecretTraceProvenance
)
if (extraHeaders) {
resolvedConfig.headers = { ...resolvedConfig.headers, ...extraHeaders }
}
return this.createClient(resolvedConfig, resolvedIP, userId)
return this.createClient(resolvedConfig, resolvedIP, userId, resolvedSecretTraceProvenance)
}
}
@@ -392,7 +469,8 @@ class McpService {
private async fetchServerTools(
config: McpServerConfig,
userId: string,
workspaceId: string
workspaceId: string,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<McpTool[]> {
for (let attempt = 0; ; attempt++) {
try {
@@ -402,8 +480,16 @@ class McpService {
serverId: config.id,
allowPool: true,
},
this.buildClient(config, userId, workspaceId),
(client) => client.listTools()
this.buildClient(config, userId, workspaceId, undefined, onResolvedSecretTraceProvenance),
(client) => {
reportRetainedClientProvenance(
client.getResolvedSecretTraceProvenance?.(),
userId,
workspaceId,
onResolvedSecretTraceProvenance
)
return client.listTools()
}
)
} catch (error) {
if (attempt === 0 && isAuthError(error) && config.authType !== 'oauth') continue
@@ -462,10 +548,12 @@ class McpService {
serverId: string,
toolCall: McpToolCall,
workspaceId: string,
extraHeaders?: Record<string, string>
extraHeaders?: Record<string, string>,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<McpToolResult> {
const requestId = generateRequestId()
const maxRetries = 2
const reportProvenance = createInvocationProvenanceReporter(onResolvedSecretTraceProvenance)
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
@@ -485,8 +573,22 @@ class McpService {
serverId,
allowPool: !hasExtraHeaders,
},
this.buildClient(config, userId, workspaceId, hasExtraHeaders ? extraHeaders : undefined),
(client) => client.callTool(toolCall)
this.buildClient(
config,
userId,
workspaceId,
hasExtraHeaders ? extraHeaders : undefined,
reportProvenance
),
(client) => {
reportRetainedClientProvenance(
client.getResolvedSecretTraceProvenance?.(),
userId,
workspaceId,
reportProvenance
)
return client.callTool(toolCall)
}
)
logger.info(`[${requestId}] Successfully executed tool ${toolCall.name}`)
return result
@@ -874,8 +976,19 @@ class McpService {
userId: string,
serverId: string,
workspaceId: string,
forceRefresh = false
forceRefresh = false,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<McpTool[]> {
if (onResolvedSecretTraceProvenance) {
return this.discoverServerToolsImpl(
userId,
serverId,
workspaceId,
forceRefresh,
createInvocationProvenanceReporter(onResolvedSecretTraceProvenance)
)
}
const inflightKey = `${workspaceId}:${serverId}:${userId}:${forceRefresh ? 'force' : 'cache'}`
const existing = this.inflightServerDiscovery.get(inflightKey)
if (existing) return existing
@@ -884,7 +997,8 @@ class McpService {
userId,
serverId,
workspaceId,
forceRefresh
forceRefresh,
undefined
).finally(() => {
this.inflightServerDiscovery.delete(inflightKey)
})
@@ -896,7 +1010,8 @@ class McpService {
userId: string,
serverId: string,
workspaceId: string,
forceRefresh: boolean
forceRefresh: boolean,
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
): Promise<McpTool[]> {
const requestId = generateRequestId()
const discoveryStartedAt = new Date()
@@ -934,7 +1049,12 @@ class McpService {
}
authType = config.authType
const tools = await this.fetchServerTools(config, userId, workspaceId)
const tools = await this.fetchServerTools(
config,
userId,
workspaceId,
onResolvedSecretTraceProvenance
)
logger.info(`[${requestId}] Discovered ${tools.length} tools from server ${config.name}`)
await Promise.allSettled([
this.cacheAdapter

Some files were not shown because too many files have changed in this diff Show More