mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
fix(sanitization): secret exposure in function and agent trace spans (#6000)
* fix trace span secret sanitization * sanitize workflow output logs * preserve streaming usage estimates * Fix logging session test after staging merge * secrets sanitization correctness * fix(execution): address review regressions * fix(execution): harden secret trace provenance * fix(execution): preserve functional state during trace projection --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
This commit is contained in:
co-authored by
Bill Leoutsakos
Vikhyath Mondreti
parent
3740c62112
commit
18214158b9
@@ -156,7 +156,7 @@ From **Settings → Custom Tools** you can:
|
||||
|
||||
<FAQ items={[
|
||||
{ question: "Can I use custom tools in standalone blocks (not agents)?", answer: "No. Custom tools are designed for use within Agent blocks, where the AI model decides when to call them. For deterministic tool execution, use the Function block instead." },
|
||||
{ question: "How do I pass API keys to my custom tool code?", answer: "Use environment variables with double curly brace syntax: {{MY_API_KEY}}. Create the environment variable in Settings → Secrets, and it will be injected at execution time without appearing in logs." },
|
||||
{ question: "How do I pass API keys to my custom tool code?", answer: "Use double curly brace syntax such as {{MY_API_KEY}} for a value saved under Settings → Secrets. The real value is injected at execution time, while exact occurrences are masked in the trace copy. If the tool returns the secret, the raw result still reaches the Agent. See Execution log protection under Secrets for details." },
|
||||
{ question: "Can I use external npm packages?", answer: "No. Custom tool code runs in a sandboxed environment with access to built-in Node.js modules and fetch(), but not external packages. For complex dependencies, consider calling an external API that wraps the functionality you need." },
|
||||
{ question: "What's the difference between custom tools and the Function block?", answer: "Custom tools are called by AI agents when they decide the tool is relevant — the agent chooses when to use it. Function blocks run deterministically at a fixed point in the workflow. Use custom tools for agent-driven actions and Function blocks for predictable data transformations." },
|
||||
{ question: "Are custom tools shared across the workspace?", answer: "Yes. Custom tools are workspace-scoped, so all workspace members can use them in their workflows." },
|
||||
|
||||
@@ -72,6 +72,8 @@ Authorization: Bearer {{MCP_API_TOKEN}}
|
||||
|
||||
When you type `{{` in the URL or header fields, a dropdown appears showing available workspace environment variables.
|
||||
|
||||
When a saved secret is successfully substituted this way, exact occurrences of its value are masked in stored MCP tool-call traces. The real URL or header value still reaches the MCP server unchanged. See [Execution log protection](/platform/credentials#execution-log-protection) for the exact scope and limitations.
|
||||
|
||||
### Testing and Validation
|
||||
|
||||
Click **Test Connection** before saving to verify the server is reachable and discover available tools. The test response shows the number of tools found and the protocol version.
|
||||
@@ -169,4 +171,4 @@ import { FAQ } from '@/components/ui/faq'
|
||||
{ question: "How do I update MCP tool schemas after a server changes its available tools?", answer: "Click the Refresh button on the MCP server in your workspace settings. This fetches the latest tool schemas from the server and automatically updates any agent blocks that use those tools with the new parameter definitions." },
|
||||
{ question: "Can permission groups restrict access to MCP tools?", answer: "Yes. On Enterprise-entitled workspaces, any workspace admin can create a permission group that disables MCP tools for its members using the disableMcpTools option. When this is enabled, affected users will not be able to add or use MCP tools in workflows that belong to that workspace." },
|
||||
{ question: "What happens if an MCP server goes offline during workflow execution?", answer: "If the MCP server is unreachable during execution, the tool call will fail and return an error. In an Agent block, the AI may attempt to handle the failure gracefully. In a standalone MCP Tool block, the workflow step will fail. Check MCP server logs and verify the server is running and accessible to troubleshoot connectivity issues." },
|
||||
]} />
|
||||
]} />
|
||||
|
||||
@@ -43,7 +43,7 @@ This is the level you debug at, because a run fails when one of its blocks fails
|
||||
|
||||
### Input and output
|
||||
|
||||
Each block in the sidebar has two tabs. The **Input** tab shows the resolved values the block actually ran with: the literal values you typed, and the earlier outputs it read by reference (with API keys redacted). The **Output** tab shows what the block produced, formatted as an object, with markdown rendered for agent-generated text.
|
||||
Each block in the sidebar has two tabs. The **Input** tab shows the resolved values the block actually ran with: the literal values you typed, and the earlier outputs it read by reference. Exact values successfully substituted from Secrets through `{{KEY}}` are masked in this trace copy; see [Execution log protection](/platform/credentials#execution-log-protection). The **Output** tab shows what the block produced, formatted as an object, with markdown rendered for agent-generated text.
|
||||
|
||||
The input tab is the important one. A block reads earlier outputs by name, written `<blockName.field>`, and the input tab shows what those references resolved to at run time. If a reference pointed at a value that was not there, you see it here as missing or wrong, not as the tag you wrote. See [how blocks pass data](/workflows/data-flow) for how those references resolve.
|
||||
|
||||
|
||||
@@ -57,7 +57,7 @@ Click any entry to open its sidebar: the run's timeline (start/end, total durati
|
||||
The block's result — JSON-formatted structured data, markdown rendering for AI content, and a copy button.
|
||||
</Tab>
|
||||
<Tab>
|
||||
What the block received — resolved variable values, referenced outputs, and environment variables. API keys are automatically redacted.
|
||||
What the block received — resolved variable values, referenced outputs, and environment variables. Exact secret values activated by a successful `{{KEY}}` substitution are masked in this trace view. See [Execution log protection](/platform/credentials#execution-log-protection).
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
@@ -95,7 +95,7 @@ import { FAQ } from '@/components/ui/faq'
|
||||
<FAQ items={[
|
||||
{ question: "How long are run logs retained?", answer: "Free plans retain logs for 7 days — after that, logs are archived to cloud storage and deleted from the database. Pro, Team, and Enterprise plans retain logs indefinitely with no automatic cleanup." },
|
||||
{ question: "What data is captured in each run log?", answer: "Each log entry includes the run ID, workflow ID, trigger type, start and end timestamps, total duration in milliseconds, cost breakdown (total cost, token counts, and per-model breakdowns), run data with trace spans, final output, and any associated files. The log details sidebar lets you inspect block-level inputs and outputs." },
|
||||
{ question: "Are API keys visible in the logs?", answer: "No. API keys and credentials are automatically redacted in the log input tab for security. You can safely inspect block inputs without exposing sensitive values." },
|
||||
{ question: "Are saved secrets visible in logs?", answer: "When a value saved under Secrets is successfully substituted through {{KEY}}, exact, case-sensitive occurrences are masked throughout the log-facing copy, including the live block-log display, Logs Overview input and output, Trace, log-read APIs, and the Logs block's Get Run Details output. This is not a general redactor: hardcoded or directly read values do not activate masking by themselves, and encoded, hashed, or transformed values are not matched. Functional execution responses, streams, and callbacks remain unchanged. See Execution log protection under Secrets for details." },
|
||||
{ question: "What is a workflow snapshot?", answer: "A frozen copy of the workflow's structure (blocks, connections, and configuration) captured at run time, so you can see the exact state behind a particular run — useful for debugging workflows that have been modified since." },
|
||||
{ question: "Can I access logs programmatically?", answer: "Yes. The External API provides endpoints to query logs with filtering by workflow, time range, trigger type, duration, cost, and model. You can also set up webhook, email, or Slack notifications for real-time alerts when runs complete." },
|
||||
{ question: "What does Live mode do on the Logs page?", answer: "It refreshes the Logs page in real time so new entries appear as they are recorded — useful during deployments or when monitoring active workflows." },
|
||||
|
||||
@@ -65,8 +65,14 @@ Select the secret you want to use. The reference appears highlighted in blue and
|
||||
height={200}
|
||||
/>
|
||||
|
||||
### Execution log protection
|
||||
|
||||
When a saved secret is successfully substituted through a `{{KEY}}` reference, Sim masks exact, case-sensitive occurrences of its resolved value in log-facing content. This includes the editor's live block-log display, Logs Overview input and output, stored execution traces, log-read API responses, and the Logs block's **Get Run Details** output. Function and Agent span inputs, outputs, and errors, Agent thinking, and Agent tool-call arguments, results, and errors are protected. The replacement is normally shown as `{{KEY}}`.
|
||||
|
||||
This is an observability projection only. Secret resolution and workflow behavior are unchanged: blocks, tools, models, and downstream steps receive the real runtime value. Stored functional execution data, workflow execution responses, streams, callbacks, block state, and snapshots are not rewritten. Log-facing views and read APIs receive a separate protected copy, so the Logs Overview **Workflow Input** and **Workflow Output** are masked without changing the underlying workflow result.
|
||||
|
||||
<Callout type="warn">
|
||||
Secret values are never exposed in the workflow editor or execution logs — they are only resolved during execution.
|
||||
Masking is activated only when Sim successfully resolves a value from **Settings → Secrets** through `{{KEY}}`. A hardcoded literal, direct `environmentVariables['KEY']` read, or shell `$KEY` read does not activate it by itself. Once activated, every exact occurrence of that value in the run's log-facing content is masked. Encoded, hashed, or otherwise transformed versions are not matched. Do not deliberately return or print secrets.
|
||||
</Callout>
|
||||
|
||||
## Secret Details
|
||||
@@ -115,7 +121,8 @@ When a workflow runs, secrets resolve in this order:
|
||||
- **Never hardcode secrets** in workflow input fields — always use `{{KEY}}` references
|
||||
|
||||
<FAQ items={[
|
||||
{ question: "Are my secrets encrypted at rest?", answer: "Yes. Secret values are encrypted before being stored in the database using server-side encryption, so raw values are never persisted in plaintext. They are also never exposed in the workflow editor, logs, or API responses." },
|
||||
{ question: "Are my secrets encrypted at rest?", answer: "Yes. Values saved under Secrets are encrypted before being stored in the database." },
|
||||
{ question: "Can a saved secret still appear in a workflow result?", answer: "Yes. Functional data is not rewritten, so the raw value can still reach downstream blocks, tools, and models and can appear in workflow execution responses, streams, or callbacks if your workflow deliberately returns or prints it. Log-facing views and read APIs, including Workflow Output on the Logs Overview and the Logs block's Get Run Details output, receive a protected copy after a successful {{KEY}} substitution." },
|
||||
{ question: "What happens if both a workspace secret and a personal secret have the same key name?", answer: "The workspace secret takes precedence. During execution, the resolver checks workspace secrets first and uses personal secrets only as a fallback. This ensures production workflows use the shared, team-managed value." },
|
||||
{ question: "Who determines which personal secret is used for automated runs?", answer: "For manual runs, the personal secrets of the user who clicked Run are used as fallback. For automated runs triggered by API, webhook, or schedule, the personal secrets of the workflow owner are used instead." },
|
||||
{ question: "Can I import secrets from a .env file?", answer: "Yes. Paste .env-style content (KEY=VALUE format) into any key or value field and the secrets will be auto-populated. The parser supports export KEY=VALUE, quoted values, and inline comments." },
|
||||
|
||||
@@ -138,14 +138,19 @@ number reported. A sandbox may declare up to 50 packages.
|
||||
|
||||
When a Function block is used as an Agent tool, its code can read every workspace
|
||||
secret by default — both `{{MY_SECRET}}` and `environmentVariables['MY_SECRET']`.
|
||||
Use `{{MY_SECRET}}` when the value may appear in execution logs: a successful
|
||||
double-brace substitution activates [execution-trace masking](/platform/credentials#execution-log-protection),
|
||||
while direct `environmentVariables['MY_SECRET']` access alone does not activate
|
||||
it by itself.
|
||||
|
||||
To narrow that, set **Secret access** to *Selected secrets* in the block's
|
||||
tool configuration and pick the names the code may read. Two things change:
|
||||
|
||||
- Only those secrets are injected. `{{OTHER_SECRET}}` no longer resolves either.
|
||||
- The selected **names** are added to the tool's description, so the model knows
|
||||
what it can reference. Values are never sent to the model — they are injected
|
||||
server-side at execution.
|
||||
what it can reference. Values are injected server-side when the Function runs.
|
||||
If the Function returns a secret, the raw result is still sent to the Agent;
|
||||
masking affects only the trace and display copy.
|
||||
|
||||
Leaving the default (*All secrets*) resolves the list at run time, so a secret
|
||||
added next month is included automatically.
|
||||
|
||||
@@ -46,7 +46,10 @@ This pairs naturally with the [Sim trigger](/workflows/triggers/sim): the trigge
|
||||
|
||||
<Callout type="info">
|
||||
The block always operates on the current workspace. Costs are denominated in credits, both for
|
||||
the cost filter and the cost output.
|
||||
the cost filter and the cost output. **Get Run Details** is a log-read path, so both `traceSpans`
|
||||
and `finalOutput` come from the protected log-facing projection described under
|
||||
[Execution log protection](/platform/credentials#execution-log-protection). The underlying runtime
|
||||
result remains unchanged.
|
||||
</Callout>
|
||||
|
||||
<FAQ items={[
|
||||
|
||||
@@ -74,9 +74,9 @@ Each run starts fresh from the values defined in the panel. A change made during
|
||||
|
||||
## Environment variables
|
||||
|
||||
Environment variables store sensitive values like API keys, tokens, and configuration that should never appear in logs or on the canvas. Create them under **Settings → Secrets** by adding a key-value pair.
|
||||
Environment variables let you keep sensitive values like API keys and tokens out of your saved workflow configuration. Create them under **Settings → Secrets** by adding a key-value pair.
|
||||
|
||||
Reference them with double curly braces in any block field, including Agent system prompts and Function block code. The value is substituted before the block runs, and API keys are redacted in logs.
|
||||
Reference them with double curly braces in any block field, including Agent system prompts and Function block code. The value is substituted before the block runs. When a saved secret is successfully substituted through `{{KEY}}`, exact occurrences of that value are masked in the live block-log display and stored execution trace without changing the runtime value. See [Execution log protection](/platform/credentials#execution-log-protection) for the exact scope and limitations.
|
||||
|
||||
```
|
||||
{{API_KEY}}
|
||||
|
||||
@@ -796,16 +796,115 @@ describe('Function Execute API Route', () => {
|
||||
|
||||
describe('Template Variable Resolution', () => {
|
||||
it.concurrent('should resolve environment variables with {{var_name}} syntax', async () => {
|
||||
const req = createMockRequest('POST', {
|
||||
code: 'return {{API_KEY}}',
|
||||
envVars: {
|
||||
API_KEY: 'secret-key-123',
|
||||
const req = createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
code: 'return {{API_KEY}}',
|
||||
envVars: {
|
||||
API_KEY: 'secret-key-123',
|
||||
},
|
||||
},
|
||||
})
|
||||
{
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(req)
|
||||
const data = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(data.__resolvedSecretNames).toEqual(['API_KEY'])
|
||||
})
|
||||
|
||||
it('reports only successful references sourced from scoped environment variables', async () => {
|
||||
const envResponse = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
code: 'return {{SHARED}} + {{ENV_ONLY}} + {{MISSING}}',
|
||||
params: { SHARED: 'param-value', MISSING: 'ordinary-param' },
|
||||
envVars: { SHARED: 'secret-value', ENV_ONLY: 'other-secret' },
|
||||
},
|
||||
{
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
|
||||
}
|
||||
)
|
||||
)
|
||||
const envData = await envResponse.json()
|
||||
|
||||
const directResponse = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
code: 'return environmentVariables.API_KEY + params.API_KEY',
|
||||
params: { API_KEY: 'ordinary-param' },
|
||||
envVars: { API_KEY: 'secret-value' },
|
||||
},
|
||||
{
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
|
||||
}
|
||||
)
|
||||
)
|
||||
const directData = await directResponse.json()
|
||||
|
||||
expect(envData.__resolvedSecretNames).toEqual(['ENV_ONLY', 'SHARED'])
|
||||
expect(directData.__resolvedSecretNames).toEqual([])
|
||||
})
|
||||
|
||||
it('reports shell {{NAME}} substitutions but not direct shell environment access', async () => {
|
||||
envFlagsMock.isRemoteSandboxEnabled = true
|
||||
|
||||
const referencedResponse = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
code: 'printf "%s" "{{API_KEY}}"',
|
||||
language: 'shell',
|
||||
envVars: { API_KEY: 'secret-value' },
|
||||
},
|
||||
{
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
|
||||
}
|
||||
)
|
||||
)
|
||||
const referencedData = await referencedResponse.json()
|
||||
|
||||
const directResponse = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
code: 'printf "%s" "$API_KEY"',
|
||||
language: 'shell',
|
||||
envVars: { API_KEY: 'secret-value' },
|
||||
},
|
||||
{
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
|
||||
}
|
||||
)
|
||||
)
|
||||
const directData = await directResponse.json()
|
||||
|
||||
expect(referencedData.__resolvedSecretNames).toEqual(['API_KEY'])
|
||||
expect(directData.__resolvedSecretNames).toEqual([])
|
||||
})
|
||||
|
||||
it('reports only substitutions allowed by the Function secret scope', async () => {
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
code: 'return {{ALLOWED}} + {{BLOCKED}}',
|
||||
envVars: { ALLOWED: 'allowed-secret', BLOCKED: 'blocked-secret' },
|
||||
secretScope: 'selected',
|
||||
mountedSecrets: ['ALLOWED'],
|
||||
},
|
||||
{
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
expect((await response.json()).__resolvedSecretNames).toEqual(['ALLOWED'])
|
||||
})
|
||||
|
||||
it.concurrent('should resolve tag variables with <tag_name> syntax', async () => {
|
||||
|
||||
@@ -35,6 +35,12 @@ import {
|
||||
} from '@/lib/execution/payloads/materialization.server'
|
||||
import { compactExecutionPayload } from '@/lib/execution/payloads/serializer'
|
||||
import { materializeLargeValueRef } from '@/lib/execution/payloads/store'
|
||||
import {
|
||||
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
|
||||
RESOLVED_SECRET_NAMES_FIELD,
|
||||
RESOLVED_SECRET_NAMES_METADATA_V1,
|
||||
requestsPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
import {
|
||||
executeInSandbox,
|
||||
executeShellInSandbox,
|
||||
@@ -65,6 +71,8 @@ const E2B_JS_WRAPPER_LINES = 3
|
||||
const E2B_PYTHON_WRAPPER_LINES = 1
|
||||
const MAX_SANDBOX_OUTPUT_FILES = 20
|
||||
const MAX_SANDBOX_OUTPUT_BYTES = 50 * 1024 * 1024
|
||||
const MAX_PRIVATE_RESOLVED_SECRET_NAMES = 10_000
|
||||
const MAX_PRIVATE_RESOLVED_SECRET_NAMES_BYTES = 1024 * 1024
|
||||
|
||||
/** Matches valid JS identifier names (letters, digits, underscore; no leading digit). */
|
||||
const SAFE_IDENTIFIER = /^[a-zA-Z_][a-zA-Z0-9_]*$/
|
||||
@@ -586,7 +594,8 @@ function resolveEnvironmentVariables(
|
||||
code: string,
|
||||
params: Record<string, any>,
|
||||
envVars: Record<string, string>,
|
||||
contextVariables: Record<string, any>
|
||||
contextVariables: Record<string, any>,
|
||||
onResolvedSecret?: (name: string) => void
|
||||
): string {
|
||||
let resolvedCode = code
|
||||
|
||||
@@ -627,6 +636,9 @@ function resolveEnvironmentVariables(
|
||||
|
||||
const safeVarName = `__var_${varName.replace(/[^a-zA-Z0-9_]/g, '_')}`
|
||||
contextVariables[safeVarName] = varValue
|
||||
if (Object.hasOwn(envVars, varName) && envVars[varName] === varValue) {
|
||||
onResolvedSecret?.(varName)
|
||||
}
|
||||
resolvedCode =
|
||||
resolvedCode.slice(0, index) + safeVarName + resolvedCode.slice(index + matchStr.length)
|
||||
}
|
||||
@@ -704,13 +716,20 @@ function resolveCodeVariables(
|
||||
blockNameMapping: Record<string, string> = {},
|
||||
blockOutputSchemas: Record<string, OutputSchema> = {},
|
||||
workflowVariables: Record<string, unknown> = {},
|
||||
language = 'javascript'
|
||||
language = 'javascript',
|
||||
onResolvedSecret?: (name: string) => void
|
||||
): { resolvedCode: string; contextVariables: Record<string, unknown> } {
|
||||
let resolvedCode = code
|
||||
const contextVariables: Record<string, unknown> = {}
|
||||
|
||||
resolvedCode = resolveWorkflowVariables(resolvedCode, workflowVariables, contextVariables)
|
||||
resolvedCode = resolveEnvironmentVariables(resolvedCode, params, envVars, contextVariables)
|
||||
resolvedCode = resolveEnvironmentVariables(
|
||||
resolvedCode,
|
||||
params,
|
||||
envVars,
|
||||
contextVariables,
|
||||
onResolvedSecret
|
||||
)
|
||||
resolvedCode = resolveTagVariables(
|
||||
resolvedCode,
|
||||
blockData,
|
||||
@@ -796,6 +815,8 @@ interface FunctionRouteExecutionContext {
|
||||
allowLargeValueWorkflowScope?: boolean
|
||||
userId?: string
|
||||
requestId: string
|
||||
resolvedSecretNames: Set<string>
|
||||
includePrivateResolvedSecretNames: boolean
|
||||
}
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> {
|
||||
@@ -931,7 +952,7 @@ async function functionJsonResponse<T>(
|
||||
context: FunctionRouteExecutionContext,
|
||||
init?: ResponseInit
|
||||
) {
|
||||
return NextResponse.json(
|
||||
const response = NextResponse.json(
|
||||
await compactFunctionRouteBody(
|
||||
{
|
||||
...body,
|
||||
@@ -942,6 +963,52 @@ async function functionJsonResponse<T>(
|
||||
),
|
||||
init
|
||||
)
|
||||
return appendResolvedSecretNames(response, context)
|
||||
}
|
||||
|
||||
function getPrivateResolvedSecretNames(context: FunctionRouteExecutionContext): string[] | null {
|
||||
if (context.resolvedSecretNames.size > MAX_PRIVATE_RESOLVED_SECRET_NAMES) return null
|
||||
|
||||
const names = Array.from(context.resolvedSecretNames).sort()
|
||||
let bytes = 0
|
||||
for (const name of names) {
|
||||
bytes += Buffer.byteLength(name, 'utf8')
|
||||
if (bytes > MAX_PRIVATE_RESOLVED_SECRET_NAMES_BYTES) return null
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
async function appendResolvedSecretNames(
|
||||
response: NextResponse,
|
||||
context: FunctionRouteExecutionContext
|
||||
): Promise<NextResponse> {
|
||||
const names = context.includePrivateResolvedSecretNames
|
||||
? getPrivateResolvedSecretNames(context)
|
||||
: null
|
||||
return appendPrivateResolvedSecretNames(response, names)
|
||||
}
|
||||
|
||||
async function appendPrivateResolvedSecretNames(
|
||||
response: NextResponse,
|
||||
names: string[] | null
|
||||
): Promise<NextResponse> {
|
||||
if (!names) return response
|
||||
|
||||
try {
|
||||
const body = (await response.clone().json()) as Record<string, unknown>
|
||||
const headers = new Headers(response.headers)
|
||||
headers.delete('content-length')
|
||||
headers.set(PRIVATE_TOOL_METADATA_RESPONSE_HEADER, RESOLVED_SECRET_NAMES_METADATA_V1)
|
||||
return NextResponse.json(
|
||||
{
|
||||
...body,
|
||||
[RESOLVED_SECRET_NAMES_FIELD]: names,
|
||||
},
|
||||
{ status: response.status, statusText: response.statusText, headers }
|
||||
)
|
||||
} catch {
|
||||
return response
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1411,6 +1478,7 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
let resolvedCode = '' // Store resolved code for error reporting
|
||||
let sourceCodeForErrors: string | undefined
|
||||
let routeContext: FunctionRouteExecutionContext | undefined
|
||||
let includePrivateResolvedSecretNames = false
|
||||
|
||||
try {
|
||||
const auth = await checkInternalAuth(req)
|
||||
@@ -1419,8 +1487,18 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
return NextResponse.json({ error: auth.error || 'Unauthorized' }, { status: 401 })
|
||||
}
|
||||
|
||||
includePrivateResolvedSecretNames = requestsPrivateToolMetadata(
|
||||
req.headers,
|
||||
RESOLVED_SECRET_NAMES_METADATA_V1
|
||||
)
|
||||
|
||||
const parsed = await parseRequest(functionExecuteContract, req, {})
|
||||
if (!parsed.success) return parsed.response
|
||||
if (!parsed.success) {
|
||||
return appendPrivateResolvedSecretNames(
|
||||
parsed.response,
|
||||
includePrivateResolvedSecretNames ? [] : null
|
||||
)
|
||||
}
|
||||
const { body } = parsed.data
|
||||
|
||||
const { DEFAULT_EXECUTION_TIMEOUT_MS } = await import('@/lib/execution/constants')
|
||||
@@ -1473,12 +1551,15 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
.map((file) => file.sandboxPath)
|
||||
.filter((path): path is string => Boolean(path))
|
||||
if (outputSandboxPaths.length > MAX_SANDBOX_OUTPUT_FILES) {
|
||||
return NextResponse.json(
|
||||
{
|
||||
success: false,
|
||||
error: `Too many sandbox output files requested (${outputSandboxPaths.length}). Maximum is ${MAX_SANDBOX_OUTPUT_FILES}.`,
|
||||
},
|
||||
{ status: 400 }
|
||||
return appendPrivateResolvedSecretNames(
|
||||
NextResponse.json(
|
||||
{
|
||||
success: false,
|
||||
error: `Too many sandbox output files requested (${outputSandboxPaths.length}). Maximum is ${MAX_SANDBOX_OUTPUT_FILES}.`,
|
||||
},
|
||||
{ status: 400 }
|
||||
),
|
||||
includePrivateResolvedSecretNames ? [] : null
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1504,6 +1585,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
allowLargeValueWorkflowScope,
|
||||
userId: auth.userId,
|
||||
requestId,
|
||||
resolvedSecretNames: new Set<string>(),
|
||||
includePrivateResolvedSecretNames,
|
||||
}
|
||||
|
||||
const lang = isValidCodeLanguage(language) ? language : DEFAULT_CODE_LANGUAGE
|
||||
@@ -1512,7 +1595,12 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
if (lang === CodeLanguage.Shell) {
|
||||
// For shell, env vars are injected as OS env vars via shellEnvs.
|
||||
// Replace {{VAR}} placeholders with $VAR so the shell can access them natively.
|
||||
resolvedCode = code.replace(/\{\{([A-Za-z_][A-Za-z0-9_]*)\}\}/g, '$$$1')
|
||||
resolvedCode = code.replace(/\{\{([A-Za-z_][A-Za-z0-9_]*)\}\}/g, (_match, name) => {
|
||||
if (Object.hasOwn(envVars, name)) {
|
||||
routeContext?.resolvedSecretNames.add(name)
|
||||
}
|
||||
return `$${name}`
|
||||
})
|
||||
// Carry pre-resolved block output variables (e.g. __blockRef_N) so they can be
|
||||
// injected as shell env vars below. The executor replaces block references in the
|
||||
// code with these names, so the values must be present at runtime.
|
||||
@@ -1526,7 +1614,8 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
blockNameMapping,
|
||||
blockOutputSchemas,
|
||||
workflowVariables,
|
||||
lang
|
||||
lang,
|
||||
(name) => routeContext?.resolvedSecretNames.add(name)
|
||||
)
|
||||
resolvedCode = codeResolution.resolvedCode
|
||||
// Merge pre-resolved block output variables from the executor. These take precedence
|
||||
@@ -1625,7 +1714,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
stdout: shellStdout,
|
||||
executionTime,
|
||||
})
|
||||
if (fileExportResponse) return fileExportResponse
|
||||
if (fileExportResponse) {
|
||||
return appendResolvedSecretNames(fileExportResponse, routeContext)
|
||||
}
|
||||
}
|
||||
|
||||
return functionJsonResponse(
|
||||
@@ -1794,7 +1885,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
stdout,
|
||||
executionTime,
|
||||
})
|
||||
if (fileExportResponse) return fileExportResponse
|
||||
if (fileExportResponse) {
|
||||
return appendResolvedSecretNames(fileExportResponse, routeContext)
|
||||
}
|
||||
}
|
||||
|
||||
return functionJsonResponse(
|
||||
@@ -1884,7 +1977,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
stdout,
|
||||
executionTime,
|
||||
})
|
||||
if (fileExportResponse) return fileExportResponse
|
||||
if (fileExportResponse) {
|
||||
return appendResolvedSecretNames(fileExportResponse, routeContext)
|
||||
}
|
||||
}
|
||||
|
||||
return functionJsonResponse(
|
||||
@@ -2043,17 +2138,20 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
{ status: error.statusCode }
|
||||
)
|
||||
}
|
||||
return NextResponse.json(
|
||||
{
|
||||
success: false,
|
||||
error: error.message,
|
||||
output: {
|
||||
result: null,
|
||||
stdout: cleanStdout(stdout),
|
||||
executionTime,
|
||||
return appendPrivateResolvedSecretNames(
|
||||
NextResponse.json(
|
||||
{
|
||||
success: false,
|
||||
error: error.message,
|
||||
output: {
|
||||
result: null,
|
||||
stdout: cleanStdout(stdout),
|
||||
executionTime,
|
||||
},
|
||||
},
|
||||
},
|
||||
{ status: error.statusCode }
|
||||
{ status: error.statusCode }
|
||||
),
|
||||
includePrivateResolvedSecretNames ? [] : null
|
||||
)
|
||||
}
|
||||
|
||||
@@ -2072,7 +2170,10 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
}
|
||||
return routeContext
|
||||
? functionJsonResponse(killResponse, routeContext, { status: 500 })
|
||||
: NextResponse.json(killResponse, { status: 500 })
|
||||
: appendPrivateResolvedSecretNames(
|
||||
NextResponse.json(killResponse, { status: 500 }),
|
||||
includePrivateResolvedSecretNames ? [] : null
|
||||
)
|
||||
}
|
||||
|
||||
logger.error(`[${requestId}] Function execution failed`, {
|
||||
@@ -2120,6 +2221,9 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
return functionJsonResponse(errorResponse, routeContext, { status: 500 })
|
||||
}
|
||||
|
||||
return NextResponse.json(errorResponse, { status: 500 })
|
||||
return appendPrivateResolvedSecretNames(
|
||||
NextResponse.json(errorResponse, { status: 500 }),
|
||||
includePrivateResolvedSecretNames ? [] : null
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -8,7 +8,7 @@ import { getSession } from '@/lib/auth'
|
||||
import { MATERIALIZE_CONCURRENCY, mapWithConcurrency } from '@/lib/core/utils/concurrency'
|
||||
import { neutralizeCsvFormula } from '@/lib/core/utils/csv'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
import { buildFilterConditions, LogFilterParamsSchema } from '@/lib/logs/filters'
|
||||
import { expandFolderIdsWithDescendants } from '@/lib/logs/folder-expansion'
|
||||
import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils'
|
||||
@@ -112,11 +112,15 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
|
||||
rows as any[],
|
||||
MATERIALIZE_CONCURRENCY,
|
||||
(r) =>
|
||||
materializeExecutionData(r.executionData as Record<string, unknown> | null, {
|
||||
workspaceId: params.workspaceId,
|
||||
workflowId: r.workflowId,
|
||||
executionId: r.executionId,
|
||||
})
|
||||
materializeExecutionDataForDisplay(
|
||||
r.executionData as Record<string, unknown> | null,
|
||||
{
|
||||
workspaceId: params.workspaceId,
|
||||
workflowId: r.workflowId,
|
||||
executionId: r.executionId,
|
||||
userId: session.user.id,
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
for (let j = 0; j < rows.length; j++) {
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { NextRequest } from 'next/server'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockDiscoverServerTools, mockExecuteTool, mockReadResponseToBufferWithLimit } = vi.hoisted(
|
||||
() => ({
|
||||
mockDiscoverServerTools: vi.fn(),
|
||||
mockExecuteTool: vi.fn(),
|
||||
mockReadResponseToBufferWithLimit: vi.fn(),
|
||||
})
|
||||
)
|
||||
|
||||
vi.mock('@/lib/core/utils/stream-limits', () => ({
|
||||
readResponseToBufferWithLimit: mockReadResponseToBufferWithLimit,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/mcp/middleware', () => ({
|
||||
withMcpAuth:
|
||||
() =>
|
||||
(
|
||||
handler: (
|
||||
request: NextRequest,
|
||||
context: {
|
||||
userId: string
|
||||
workspaceId: string
|
||||
requestId: string
|
||||
authType: 'internal_jwt' | 'session'
|
||||
},
|
||||
routeContext: { params: Promise<Record<string, string>> }
|
||||
) => Promise<Response>
|
||||
) =>
|
||||
(request: NextRequest) =>
|
||||
handler(
|
||||
request,
|
||||
{
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
requestId: 'request-1',
|
||||
authType: request.headers.has('x-test-session') ? 'session' : 'internal_jwt',
|
||||
},
|
||||
{ params: Promise.resolve({}) }
|
||||
),
|
||||
readMcpJsonBodyWithLimit: (request: NextRequest) => request.json(),
|
||||
mcpBodyReadErrorResponse: () => null,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/mcp/service', () => ({
|
||||
mcpService: {
|
||||
discoverServerTools: mockDiscoverServerTools,
|
||||
executeTool: mockExecuteTool,
|
||||
},
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/billing/core/billing-attribution', () => ({
|
||||
requireBillingAttributionHeader: () => ({ payerSubscription: { plan: 'pro' } }),
|
||||
resolveBillingAttribution: async () => ({ payerSubscription: { plan: 'pro' } }),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/execution-limits', () => ({
|
||||
DEFAULT_EXECUTION_TIMEOUT_MS: 30_000,
|
||||
getExecutionTimeout: () => 0,
|
||||
}))
|
||||
|
||||
vi.mock('@/ee/access-control/utils/permission-check', () => ({
|
||||
assertPermissionsAllowed: async () => {},
|
||||
McpToolsNotAllowedError: class McpToolsNotAllowedError extends Error {},
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/telemetry', () => ({
|
||||
PlatformEvents: { mcpToolExecuted: vi.fn() },
|
||||
}))
|
||||
|
||||
import { POST } from '@/app/api/mcp/tools/execute/route'
|
||||
|
||||
const URL = 'http://localhost/api/mcp/tools/execute'
|
||||
const REQUEST_BODY = {
|
||||
workspaceId: 'workspace-1',
|
||||
serverId: 'server-1',
|
||||
toolName: 'example_tool',
|
||||
arguments: {},
|
||||
}
|
||||
|
||||
function createRequest(headers: Record<string, string> = {}): NextRequest {
|
||||
return new NextRequest(URL, {
|
||||
method: 'POST',
|
||||
headers: { 'content-type': 'application/json', ...headers },
|
||||
body: JSON.stringify(REQUEST_BODY),
|
||||
})
|
||||
}
|
||||
|
||||
describe('MCP tool execution private secret provenance', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockDiscoverServerTools.mockResolvedValue([{ name: 'example_tool', inputSchema: {} }])
|
||||
mockExecuteTool.mockResolvedValue({ content: [{ type: 'text', text: 'ok' }] })
|
||||
mockReadResponseToBufferWithLimit.mockImplementation(async (response: Response) =>
|
||||
Buffer.from(await response.arrayBuffer())
|
||||
)
|
||||
})
|
||||
|
||||
it('returns fail-closed scoped provenance only to an authenticated internal caller', async () => {
|
||||
mockDiscoverServerTools.mockImplementationOnce(
|
||||
async (
|
||||
_userId: string,
|
||||
_serverId: string,
|
||||
_workspaceId: string,
|
||||
_forceRefresh: boolean,
|
||||
report: (value: unknown) => void
|
||||
) => {
|
||||
report({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
return [{ name: 'example_tool', inputSchema: {} }]
|
||||
}
|
||||
)
|
||||
mockExecuteTool.mockImplementationOnce(
|
||||
async (
|
||||
_userId: string,
|
||||
_serverId: string,
|
||||
_toolCall: unknown,
|
||||
_workspaceId: string,
|
||||
_headers: unknown,
|
||||
report: (value: unknown) => void
|
||||
) => {
|
||||
report({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
return { content: [{ type: 'text', text: 'ok' }] }
|
||||
}
|
||||
)
|
||||
const request = createRequest({
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
})
|
||||
|
||||
const response = await POST(request, {})
|
||||
const body = (await response.json()) as Record<string, unknown>
|
||||
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
|
||||
'resolved-secret-provenance-v1'
|
||||
)
|
||||
expect(body.__resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('does not expose private provenance metadata to a session caller', async () => {
|
||||
const request = createRequest({
|
||||
'x-test-session': 'true',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
})
|
||||
|
||||
const response = await POST(request, {})
|
||||
const body = (await response.json()) as Record<string, unknown>
|
||||
|
||||
expect(response.headers.has('x-sim-private-tool-metadata')).toBe(false)
|
||||
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
|
||||
expect(mockDiscoverServerTools.mock.calls[0]?.[4]).toBeUndefined()
|
||||
expect(mockExecuteTool.mock.calls[0]?.[5]).toBeUndefined()
|
||||
})
|
||||
|
||||
it('preserves the functional response when private provenance cannot be attached', async () => {
|
||||
mockReadResponseToBufferWithLimit.mockRejectedValueOnce(new Error('Response exceeds limit'))
|
||||
mockExecuteTool.mockResolvedValueOnce({
|
||||
content: [{ type: 'text', text: 'unchanged' }],
|
||||
})
|
||||
const request = createRequest({
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
})
|
||||
|
||||
const response = await POST(request, {})
|
||||
const body = (await response.json()) as Record<string, unknown>
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.ok).toBe(true)
|
||||
expect(response.headers.has('x-sim-private-tool-metadata')).toBe(false)
|
||||
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
|
||||
expect(body).toMatchObject({
|
||||
success: true,
|
||||
data: {
|
||||
success: true,
|
||||
output: { content: [{ type: 'text' }] },
|
||||
},
|
||||
})
|
||||
expect(
|
||||
(body.data as { output: { content: Array<{ text?: unknown }> } }).output.content[0]?.text
|
||||
).toBe('unchanged')
|
||||
})
|
||||
})
|
||||
@@ -11,8 +11,15 @@ import {
|
||||
} from '@/lib/billing/core/billing-attribution'
|
||||
import { getExecutionTimeout } from '@/lib/core/execution-limits'
|
||||
import type { SubscriptionPlan } from '@/lib/core/rate-limiter/types'
|
||||
import { readResponseToBufferWithLimit } from '@/lib/core/utils/stream-limits'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { SIM_VIA_HEADER } from '@/lib/execution/call-chain'
|
||||
import {
|
||||
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
requestsPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
import {
|
||||
mcpBodyReadErrorResponse,
|
||||
readMcpJsonBodyWithLimit,
|
||||
@@ -31,8 +38,13 @@ import {
|
||||
assertPermissionsAllowed,
|
||||
McpToolsNotAllowedError,
|
||||
} from '@/ee/access-control/utils/permission-check'
|
||||
import {
|
||||
ResolvedSecretTraceProvenanceAccumulator,
|
||||
type ResolvedSecretTraceProvenanceV1,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('McpToolExecutionAPI')
|
||||
const MAX_PRIVATE_MCP_RESPONSE_BYTES = 10 * 1024 * 1024
|
||||
|
||||
export const dynamic = 'force-dynamic'
|
||||
|
||||
@@ -55,6 +67,35 @@ function hasType(prop: unknown): prop is SchemaProperty {
|
||||
return typeof prop === 'object' && prop !== null && 'type' in prop
|
||||
}
|
||||
|
||||
async function attachPrivateProvenance(
|
||||
response: NextResponse,
|
||||
provenance: ResolvedSecretTraceProvenanceAccumulator
|
||||
): Promise<NextResponse> {
|
||||
let payload: Record<string, unknown>
|
||||
try {
|
||||
const body = await readResponseToBufferWithLimit(response.clone(), {
|
||||
maxBytes: MAX_PRIVATE_MCP_RESPONSE_BYTES,
|
||||
label: 'MCP private metadata response',
|
||||
allowNoBodyFallback: true,
|
||||
})
|
||||
const parsed: unknown = JSON.parse(body.toString('utf8'))
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new Error('MCP response is not a JSON object')
|
||||
}
|
||||
payload = parsed as Record<string, unknown>
|
||||
} catch {
|
||||
return response
|
||||
}
|
||||
|
||||
const headers = new Headers(response.headers)
|
||||
headers.delete('content-length')
|
||||
headers.set(PRIVATE_TOOL_METADATA_RESPONSE_HEADER, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
|
||||
return NextResponse.json(
|
||||
{ ...payload, [RESOLVED_SECRET_PROVENANCE_FIELD]: provenance.exportProvenance() },
|
||||
{ status: response.status, headers }
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* POST - Execute a tool on an MCP server
|
||||
*/
|
||||
@@ -62,239 +103,268 @@ export const POST = withRouteHandler(
|
||||
withMcpAuth('read')(
|
||||
async (request: NextRequest, { userId, workspaceId, requestId, authType }) => {
|
||||
let serverId: string | undefined
|
||||
try {
|
||||
const rawBody = await readMcpJsonBodyWithLimit(request)
|
||||
const parsedBody = mcpToolExecutionBodySchema.safeParse(rawBody)
|
||||
|
||||
if (!parsedBody.success) {
|
||||
return createMcpErrorResponse(parsedBody.error, 'Invalid request format', 400)
|
||||
}
|
||||
|
||||
const body = parsedBody.data
|
||||
|
||||
logger.info(`[${requestId}] MCP tool execution request received`, {
|
||||
hasAuthHeader: !!request.headers.get('authorization'),
|
||||
bodyKeys: Object.keys(body),
|
||||
serverId: body.serverId,
|
||||
toolName: body.toolName,
|
||||
hasWorkflowId: !!body.workflowId,
|
||||
workflowId: body.workflowId,
|
||||
userId: userId,
|
||||
})
|
||||
|
||||
const { toolName, arguments: rawArgs } = body
|
||||
serverId = body.serverId
|
||||
const args = rawArgs || {}
|
||||
|
||||
const includePrivateProvenance =
|
||||
authType === AuthType.INTERNAL_JWT &&
|
||||
requestsPrivateToolMetadata(request.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
|
||||
const resolvedSecretTraceProvenance = includePrivateProvenance
|
||||
? new ResolvedSecretTraceProvenanceAccumulator({ userId, workspaceId })
|
||||
: undefined
|
||||
const recordProvenance = resolvedSecretTraceProvenance
|
||||
? (provenance: ResolvedSecretTraceProvenanceV1): void => {
|
||||
resolvedSecretTraceProvenance.record(provenance)
|
||||
}
|
||||
: undefined
|
||||
const response = await (async (): Promise<NextResponse> => {
|
||||
try {
|
||||
await assertPermissionsAllowed({
|
||||
userId,
|
||||
workspaceId,
|
||||
toolKind: 'mcp',
|
||||
const rawBody = await readMcpJsonBodyWithLimit(request)
|
||||
const parsedBody = mcpToolExecutionBodySchema.safeParse(rawBody)
|
||||
|
||||
if (!parsedBody.success) {
|
||||
return createMcpErrorResponse(parsedBody.error, 'Invalid request format', 400)
|
||||
}
|
||||
|
||||
const body = parsedBody.data
|
||||
|
||||
logger.info(`[${requestId}] MCP tool execution request received`, {
|
||||
hasAuthHeader: !!request.headers.get('authorization'),
|
||||
bodyKeys: Object.keys(body),
|
||||
serverId: body.serverId,
|
||||
toolName: body.toolName,
|
||||
hasWorkflowId: !!body.workflowId,
|
||||
workflowId: body.workflowId,
|
||||
userId: userId,
|
||||
})
|
||||
} catch (err) {
|
||||
if (err instanceof McpToolsNotAllowedError) {
|
||||
return createMcpErrorResponse(err, err.message, 403)
|
||||
}
|
||||
throw err
|
||||
}
|
||||
|
||||
logger.info(
|
||||
`[${requestId}] Executing tool ${toolName} on server ${serverId} for user ${userId} in workspace ${workspaceId}`
|
||||
)
|
||||
const { toolName, arguments: rawArgs } = body
|
||||
serverId = body.serverId
|
||||
const args = rawArgs || {}
|
||||
|
||||
let tool: McpTool | null = null
|
||||
try {
|
||||
const tools = await mcpService.discoverServerTools(userId, serverId, workspaceId)
|
||||
tool = tools.find((t) => t.name === toolName) ?? null
|
||||
|
||||
if (!tool) {
|
||||
logger.warn(`[${requestId}] Tool ${toolName} not found on server ${serverId}`, {
|
||||
availableTools: tools.map((t) => t.name),
|
||||
try {
|
||||
await assertPermissionsAllowed({
|
||||
userId,
|
||||
workspaceId,
|
||||
toolKind: 'mcp',
|
||||
})
|
||||
return createMcpErrorResponse(
|
||||
new Error('Tool not found'),
|
||||
'Tool not found on the specified server',
|
||||
404
|
||||
)
|
||||
} catch (err) {
|
||||
if (err instanceof McpToolsNotAllowedError) {
|
||||
return createMcpErrorResponse(err, err.message, 403)
|
||||
}
|
||||
throw err
|
||||
}
|
||||
|
||||
if (tool.inputSchema?.properties) {
|
||||
for (const [paramName, paramSchema] of Object.entries(tool.inputSchema.properties)) {
|
||||
const schema = hasType(paramSchema) ? paramSchema : null
|
||||
if (!schema) continue
|
||||
const value = args[paramName]
|
||||
logger.info(
|
||||
`[${requestId}] Executing tool ${toolName} on server ${serverId} for user ${userId} in workspace ${workspaceId}`
|
||||
)
|
||||
|
||||
if (value === undefined || value === null) {
|
||||
continue
|
||||
}
|
||||
let tool: McpTool | null = null
|
||||
try {
|
||||
const tools = await mcpService.discoverServerTools(
|
||||
userId,
|
||||
serverId,
|
||||
workspaceId,
|
||||
false,
|
||||
recordProvenance
|
||||
)
|
||||
tool = tools.find((t) => t.name === toolName) ?? null
|
||||
|
||||
if (
|
||||
(schema.type === 'number' || schema.type === 'integer') &&
|
||||
typeof value === 'string'
|
||||
) {
|
||||
const numValue =
|
||||
schema.type === 'integer' ? Number.parseInt(value) : Number.parseFloat(value)
|
||||
if (!Number.isNaN(numValue)) {
|
||||
args[paramName] = numValue
|
||||
if (!tool) {
|
||||
logger.warn(`[${requestId}] Tool ${toolName} not found on server ${serverId}`, {
|
||||
availableTools: tools.map((t) => t.name),
|
||||
})
|
||||
return createMcpErrorResponse(
|
||||
new Error('Tool not found'),
|
||||
'Tool not found on the specified server',
|
||||
404
|
||||
)
|
||||
}
|
||||
|
||||
if (tool.inputSchema?.properties) {
|
||||
for (const [paramName, paramSchema] of Object.entries(tool.inputSchema.properties)) {
|
||||
const schema = hasType(paramSchema) ? paramSchema : null
|
||||
if (!schema) continue
|
||||
const value = args[paramName]
|
||||
|
||||
if (value === undefined || value === null) {
|
||||
continue
|
||||
}
|
||||
} else if (schema.type === 'boolean' && typeof value === 'string') {
|
||||
if (value.toLowerCase() === 'true') {
|
||||
args[paramName] = true
|
||||
} else if (value.toLowerCase() === 'false') {
|
||||
args[paramName] = false
|
||||
}
|
||||
} else if (schema.type === 'array' && typeof value === 'string') {
|
||||
const stringValue = value.trim()
|
||||
if (stringValue) {
|
||||
try {
|
||||
const parsed = JSON.parse(stringValue)
|
||||
if (Array.isArray(parsed)) {
|
||||
args[paramName] = parsed
|
||||
} else {
|
||||
args[paramName] = [parsed]
|
||||
|
||||
if (
|
||||
(schema.type === 'number' || schema.type === 'integer') &&
|
||||
typeof value === 'string'
|
||||
) {
|
||||
const numValue =
|
||||
schema.type === 'integer' ? Number.parseInt(value) : Number.parseFloat(value)
|
||||
if (!Number.isNaN(numValue)) {
|
||||
args[paramName] = numValue
|
||||
}
|
||||
} else if (schema.type === 'boolean' && typeof value === 'string') {
|
||||
if (value.toLowerCase() === 'true') {
|
||||
args[paramName] = true
|
||||
} else if (value.toLowerCase() === 'false') {
|
||||
args[paramName] = false
|
||||
}
|
||||
} else if (schema.type === 'array' && typeof value === 'string') {
|
||||
const stringValue = value.trim()
|
||||
if (stringValue) {
|
||||
try {
|
||||
const parsed = JSON.parse(stringValue)
|
||||
if (Array.isArray(parsed)) {
|
||||
args[paramName] = parsed
|
||||
} else {
|
||||
args[paramName] = [parsed]
|
||||
}
|
||||
} catch {
|
||||
if (stringValue.includes(',')) {
|
||||
args[paramName] = stringValue
|
||||
.split(',')
|
||||
.map((item) => item.trim())
|
||||
.filter((item) => item)
|
||||
} else {
|
||||
args[paramName] = [stringValue]
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
if (stringValue.includes(',')) {
|
||||
args[paramName] = stringValue
|
||||
.split(',')
|
||||
.map((item) => item.trim())
|
||||
.filter((item) => item)
|
||||
} else {
|
||||
args[paramName] = [stringValue]
|
||||
}
|
||||
} else {
|
||||
args[paramName] = []
|
||||
}
|
||||
} else {
|
||||
args[paramName] = []
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
logger.warn(
|
||||
`[${requestId}] Failed to discover tools for validation, proceeding without schema`,
|
||||
error
|
||||
)
|
||||
}
|
||||
} catch (error) {
|
||||
logger.warn(
|
||||
`[${requestId}] Failed to discover tools for validation, proceeding without schema`,
|
||||
error
|
||||
)
|
||||
}
|
||||
|
||||
if (tool) {
|
||||
const validationError = validateToolArguments(tool, args)
|
||||
if (validationError) {
|
||||
logger.warn(`[${requestId}] Tool validation failed: ${validationError}`)
|
||||
if (tool) {
|
||||
const validationError = validateToolArguments(tool, args)
|
||||
if (validationError) {
|
||||
logger.warn(`[${requestId}] Tool validation failed: ${validationError}`)
|
||||
return createMcpErrorResponse(
|
||||
new Error(`Invalid arguments for tool ${toolName}: ${validationError}`),
|
||||
'Invalid tool arguments',
|
||||
400
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
const toolCall: McpToolCall = {
|
||||
name: toolName,
|
||||
arguments: args,
|
||||
}
|
||||
|
||||
const billingAttribution =
|
||||
authType === AuthType.INTERNAL_JWT
|
||||
? requireBillingAttributionHeader(request.headers, {
|
||||
actorUserId: userId,
|
||||
workspaceId,
|
||||
})
|
||||
: await resolveBillingAttribution({ actorUserId: userId, workspaceId })
|
||||
const executionTimeout = getExecutionTimeout(
|
||||
billingAttribution.payerSubscription?.plan as SubscriptionPlan | undefined,
|
||||
'sync'
|
||||
)
|
||||
|
||||
const simViaHeader = request.headers.get(SIM_VIA_HEADER)
|
||||
const extraHeaders: Record<string, string> = {}
|
||||
if (simViaHeader) {
|
||||
extraHeaders[SIM_VIA_HEADER] = simViaHeader
|
||||
}
|
||||
|
||||
let timeoutHandle: ReturnType<typeof setTimeout> | undefined
|
||||
const executePromise = mcpService.executeTool(
|
||||
userId,
|
||||
serverId,
|
||||
toolCall,
|
||||
workspaceId,
|
||||
extraHeaders,
|
||||
recordProvenance
|
||||
)
|
||||
// A zero timeout means "no timeout" (billing-disabled deployments).
|
||||
const result = await (executionTimeout > 0
|
||||
? Promise.race([
|
||||
executePromise,
|
||||
new Promise<never>((_, reject) => {
|
||||
timeoutHandle = setTimeout(
|
||||
() => reject(new Error('Tool execution timeout')),
|
||||
executionTimeout
|
||||
)
|
||||
}),
|
||||
])
|
||||
: executePromise
|
||||
).finally(() => {
|
||||
if (timeoutHandle !== undefined) clearTimeout(timeoutHandle)
|
||||
})
|
||||
|
||||
const transformedResult = transformToolResult(result)
|
||||
|
||||
if (result.isError) {
|
||||
logger.warn(
|
||||
`[${requestId}] Tool execution returned error for ${toolName} on ${serverId}`
|
||||
)
|
||||
return createMcpErrorResponse(
|
||||
new Error(`Invalid arguments for tool ${toolName}: ${validationError}`),
|
||||
'Invalid tool arguments',
|
||||
transformedResult,
|
||||
transformedResult.error || 'Tool execution failed',
|
||||
400
|
||||
)
|
||||
}
|
||||
}
|
||||
logger.info(`[${requestId}] Successfully executed tool ${toolName} on server ${serverId}`)
|
||||
|
||||
const toolCall: McpToolCall = {
|
||||
name: toolName,
|
||||
arguments: args,
|
||||
}
|
||||
try {
|
||||
const { PlatformEvents } = await import('@/lib/core/telemetry')
|
||||
PlatformEvents.mcpToolExecuted({
|
||||
serverId,
|
||||
toolName,
|
||||
status: 'success',
|
||||
workspaceId,
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn('Failed to record MCP tool execution telemetry', {
|
||||
error: getErrorMessage(error),
|
||||
serverId,
|
||||
toolName,
|
||||
workspaceId,
|
||||
})
|
||||
}
|
||||
|
||||
const billingAttribution =
|
||||
authType === AuthType.INTERNAL_JWT
|
||||
? requireBillingAttributionHeader(request.headers, {
|
||||
actorUserId: userId,
|
||||
workspaceId,
|
||||
})
|
||||
: await resolveBillingAttribution({ actorUserId: userId, workspaceId })
|
||||
const executionTimeout = getExecutionTimeout(
|
||||
billingAttribution.payerSubscription?.plan as SubscriptionPlan | undefined,
|
||||
'sync'
|
||||
)
|
||||
|
||||
const simViaHeader = request.headers.get(SIM_VIA_HEADER)
|
||||
const extraHeaders: Record<string, string> = {}
|
||||
if (simViaHeader) {
|
||||
extraHeaders[SIM_VIA_HEADER] = simViaHeader
|
||||
}
|
||||
|
||||
let timeoutHandle: ReturnType<typeof setTimeout> | undefined
|
||||
const executePromise = mcpService.executeTool(
|
||||
userId,
|
||||
serverId,
|
||||
toolCall,
|
||||
workspaceId,
|
||||
extraHeaders
|
||||
)
|
||||
// A zero timeout means "no timeout" (billing-disabled deployments).
|
||||
const result = await (executionTimeout > 0
|
||||
? Promise.race([
|
||||
executePromise,
|
||||
new Promise<never>((_, reject) => {
|
||||
timeoutHandle = setTimeout(
|
||||
() => reject(new Error('Tool execution timeout')),
|
||||
executionTimeout
|
||||
)
|
||||
}),
|
||||
])
|
||||
: executePromise
|
||||
).finally(() => {
|
||||
if (timeoutHandle !== undefined) clearTimeout(timeoutHandle)
|
||||
})
|
||||
|
||||
const transformedResult = transformToolResult(result)
|
||||
|
||||
if (result.isError) {
|
||||
logger.warn(`[${requestId}] Tool execution returned error for ${toolName} on ${serverId}`)
|
||||
return createMcpErrorResponse(
|
||||
transformedResult,
|
||||
transformedResult.error || 'Tool execution failed',
|
||||
400
|
||||
)
|
||||
}
|
||||
logger.info(`[${requestId}] Successfully executed tool ${toolName} on server ${serverId}`)
|
||||
|
||||
try {
|
||||
const { PlatformEvents } = await import('@/lib/core/telemetry')
|
||||
PlatformEvents.mcpToolExecuted({
|
||||
serverId,
|
||||
toolName,
|
||||
status: 'success',
|
||||
workspaceId,
|
||||
})
|
||||
return createMcpSuccessResponse(transformedResult)
|
||||
} catch (error) {
|
||||
logger.warn('Failed to record MCP tool execution telemetry', {
|
||||
error: getErrorMessage(error),
|
||||
serverId,
|
||||
toolName,
|
||||
workspaceId,
|
||||
})
|
||||
}
|
||||
|
||||
return createMcpSuccessResponse(transformedResult)
|
||||
} catch (error) {
|
||||
const bodyErrorResponse = mcpBodyReadErrorResponse(error, request)
|
||||
if (bodyErrorResponse) return bodyErrorResponse
|
||||
if (
|
||||
error instanceof McpOauthAuthorizationRequiredError ||
|
||||
error instanceof McpOauthRedirectRequired ||
|
||||
error instanceof UnauthorizedError
|
||||
) {
|
||||
const errorServerId =
|
||||
error instanceof McpOauthAuthorizationRequiredError ? error.serverId : serverId
|
||||
logger.warn(`[${requestId}] OAuth re-authorization required for MCP tool execution`, {
|
||||
serverId: errorServerId,
|
||||
})
|
||||
return NextResponse.json(
|
||||
{
|
||||
success: false,
|
||||
error: 'OAuth re-authorization required',
|
||||
code: 'reauth_required',
|
||||
if (getErrorMessage(error) === 'Tool execution timeout') {
|
||||
resolvedSecretTraceProvenance?.markIncomplete()
|
||||
}
|
||||
const bodyErrorResponse = mcpBodyReadErrorResponse(error, request)
|
||||
if (bodyErrorResponse) return bodyErrorResponse
|
||||
if (
|
||||
error instanceof McpOauthAuthorizationRequiredError ||
|
||||
error instanceof McpOauthRedirectRequired ||
|
||||
error instanceof UnauthorizedError
|
||||
) {
|
||||
const errorServerId =
|
||||
error instanceof McpOauthAuthorizationRequiredError ? error.serverId : serverId
|
||||
logger.warn(`[${requestId}] OAuth re-authorization required for MCP tool execution`, {
|
||||
serverId: errorServerId,
|
||||
},
|
||||
{ status: 401 }
|
||||
)
|
||||
})
|
||||
return NextResponse.json(
|
||||
{
|
||||
success: false,
|
||||
error: 'OAuth re-authorization required',
|
||||
code: 'reauth_required',
|
||||
serverId: errorServerId,
|
||||
},
|
||||
{ status: 401 }
|
||||
)
|
||||
}
|
||||
|
||||
logger.error(`[${requestId}] Error executing MCP tool:`, error)
|
||||
|
||||
const { message, status } = categorizeError(error)
|
||||
return createMcpErrorResponse(new Error(message), message, status)
|
||||
}
|
||||
})()
|
||||
|
||||
logger.error(`[${requestId}] Error executing MCP tool:`, error)
|
||||
|
||||
const { message, status } = categorizeError(error)
|
||||
return createMcpErrorResponse(new Error(message), message, status)
|
||||
}
|
||||
return resolvedSecretTraceProvenance
|
||||
? attachPrivateProvenance(response, resolvedSecretTraceProvenance)
|
||||
: response
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
@@ -1,7 +1,98 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { createMockRequest } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const {
|
||||
mockAssertActiveWorkspaceAccess,
|
||||
mockBuildIntegrationToolSchemas,
|
||||
mockBuildSelectedMcpToolSchemas,
|
||||
mockBuildTaggedMcpToolSchemas,
|
||||
mockCheckInternalAuth,
|
||||
mockComputeWorkspaceEntitlements,
|
||||
mockDecryptSecret,
|
||||
mockGenerateWorkspaceContext,
|
||||
mockGetPersonalAndWorkspaceEnv,
|
||||
mockProcessContextsServer,
|
||||
mockRequestExplicitStreamAbort,
|
||||
mockRequireBillingAttributionHeader,
|
||||
mockRunHeadlessCopilotLifecycle,
|
||||
} = vi.hoisted(() => ({
|
||||
mockAssertActiveWorkspaceAccess: vi.fn(),
|
||||
mockBuildIntegrationToolSchemas: vi.fn(),
|
||||
mockBuildSelectedMcpToolSchemas: vi.fn(),
|
||||
mockBuildTaggedMcpToolSchemas: vi.fn(),
|
||||
mockCheckInternalAuth: vi.fn(),
|
||||
mockComputeWorkspaceEntitlements: vi.fn(),
|
||||
mockDecryptSecret: vi.fn(),
|
||||
mockGenerateWorkspaceContext: vi.fn(),
|
||||
mockGetPersonalAndWorkspaceEnv: vi.fn(),
|
||||
mockProcessContextsServer: vi.fn(),
|
||||
mockRequestExplicitStreamAbort: vi.fn(),
|
||||
mockRequireBillingAttributionHeader: vi.fn(),
|
||||
mockRunHeadlessCopilotLifecycle: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/security/encryption', () => ({
|
||||
decryptSecret: mockDecryptSecret,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/auth/hybrid', () => ({
|
||||
checkInternalAuth: mockCheckInternalAuth,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/billing/core/billing-attribution', () => ({
|
||||
requireBillingAttributionHeader: mockRequireBillingAttributionHeader,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/chat/payload', () => ({
|
||||
buildIntegrationToolSchemas: mockBuildIntegrationToolSchemas,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/chat/process-contents', () => ({
|
||||
processContextsServer: mockProcessContextsServer,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/chat/workspace-context', () => ({
|
||||
generateWorkspaceContext: mockGenerateWorkspaceContext,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/entitlements', () => ({
|
||||
computeWorkspaceEntitlements: mockComputeWorkspaceEntitlements,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/mcp-tools', () => ({
|
||||
buildSelectedMcpToolSchemas: mockBuildSelectedMcpToolSchemas,
|
||||
buildTaggedMcpToolSchemas: mockBuildTaggedMcpToolSchemas,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/request/lifecycle/headless', () => ({
|
||||
runHeadlessCopilotLifecycle: mockRunHeadlessCopilotLifecycle,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/copilot/request/session/explicit-abort', () => ({
|
||||
requestExplicitStreamAbort: mockRequestExplicitStreamAbort,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/config/env-flags', () => ({
|
||||
isDocSandboxEnabled: false,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/environment/utils', () => ({
|
||||
getPersonalAndWorkspaceEnv: mockGetPersonalAndWorkspaceEnv,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workspaces/permissions/utils', () => ({
|
||||
assertActiveWorkspaceAccess: mockAssertActiveWorkspaceAccess,
|
||||
isWorkspaceAccessDeniedError: vi.fn(() => false),
|
||||
}))
|
||||
|
||||
import type { CopilotLifecycleOptions } from '@/lib/copilot/request/lifecycle/run'
|
||||
import {
|
||||
buildExecuteResponsePayload,
|
||||
CALLER_VISIBLE_SERVER_TOOLS,
|
||||
POST,
|
||||
} from '@/app/api/mothership/execute/route'
|
||||
|
||||
type Payload = Parameters<typeof buildExecuteResponsePayload>[0]
|
||||
@@ -48,3 +139,362 @@ describe('buildExecuteResponsePayload', () => {
|
||||
expect(CALLER_VISIBLE_SERVER_TOOLS.has('complete_scheduled_task')).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('mothership private trace provenance transport', () => {
|
||||
const requestBody = {
|
||||
messages: [{ role: 'user', content: 'hello' }],
|
||||
workspaceId: 'workspace-1',
|
||||
userId: 'user-1',
|
||||
chatId: 'chat-1',
|
||||
messageId: 'message-1',
|
||||
requestId: 'request-1',
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockCheckInternalAuth.mockResolvedValue({
|
||||
success: true,
|
||||
userId: 'user-1',
|
||||
authType: 'internal_jwt',
|
||||
})
|
||||
mockAssertActiveWorkspaceAccess.mockResolvedValue({ permission: 'write' })
|
||||
mockRequireBillingAttributionHeader.mockReturnValue({
|
||||
actorUserId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
mockGetPersonalAndWorkspaceEnv.mockResolvedValue({
|
||||
personalEncrypted: { API_KEY: 'encrypted-secret' },
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: { API_KEY: 'secret-value' },
|
||||
workspaceDecrypted: {},
|
||||
decryptionFailures: [],
|
||||
})
|
||||
mockGenerateWorkspaceContext.mockResolvedValue({})
|
||||
mockBuildIntegrationToolSchemas.mockResolvedValue([])
|
||||
mockBuildSelectedMcpToolSchemas.mockResolvedValue([])
|
||||
mockBuildTaggedMcpToolSchemas.mockResolvedValue([])
|
||||
mockComputeWorkspaceEntitlements.mockResolvedValue([])
|
||||
mockDecryptSecret.mockResolvedValue({ decrypted: 'secret-value' })
|
||||
mockProcessContextsServer.mockResolvedValue([])
|
||||
mockRequestExplicitStreamAbort.mockResolvedValue(undefined)
|
||||
})
|
||||
|
||||
function successResult() {
|
||||
return {
|
||||
success: true,
|
||||
content: 'secret-value',
|
||||
contentBlocks: [],
|
||||
toolCalls: [],
|
||||
chatId: 'chat-1',
|
||||
}
|
||||
}
|
||||
|
||||
function activateSecret(options: CopilotLifecycleOptions): void {
|
||||
options.resolvedSecretTraceRegistry?.recordResolved('API_KEY', 'secret-value')
|
||||
}
|
||||
|
||||
it('does not expose private provenance unless the internal caller requests it', async () => {
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
activateSecret(options)
|
||||
return successResult()
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
requestBody,
|
||||
{ Authorization: 'Bearer internal', 'x-sim-billing-attribution': 'billing' },
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBeNull()
|
||||
expect(body.content).toBe('secret-value')
|
||||
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
|
||||
expect(mockGetPersonalAndWorkspaceEnv).not.toHaveBeenCalled()
|
||||
expect(mockRunHeadlessCopilotLifecycle).toHaveBeenCalledWith(
|
||||
expect.any(Object),
|
||||
expect.objectContaining({ resolvedSecretTraceRegistry: undefined })
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps execution functional and fails trace provenance closed when catalog setup fails', async () => {
|
||||
mockGetPersonalAndWorkspaceEnv.mockRejectedValueOnce(new Error('catalog unavailable'))
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
expect(options.resolvedSecretTraceRegistry?.isComplete()).toBe(false)
|
||||
return successResult()
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
requestBody,
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(body.content).toBe('secret-value')
|
||||
expect(body.__resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('fails provenance closed without changing a runtime value that rotated after catalog load', async () => {
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
expect(
|
||||
options.resolvedSecretTraceRegistry?.recordResolved('API_KEY', 'rotated-secret-value')
|
||||
).toBe(false)
|
||||
return { ...successResult(), content: 'rotated-secret-value' }
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
requestBody,
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(body.content).toBe('rotated-secret-value')
|
||||
expect(body.__resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('returns encrypted provenance on a marker-gated successful request', async () => {
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
activateSecret(options)
|
||||
return successResult()
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
requestBody,
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
|
||||
'resolved-secret-provenance-v1'
|
||||
)
|
||||
expect(body.content).toBe('secret-value')
|
||||
expect(body.__resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
expect(JSON.stringify(body.__resolvedSecretTraceProvenance)).not.toContain('secret-value')
|
||||
})
|
||||
|
||||
it('imports MCP schema-discovery provenance before starting the lifecycle', async () => {
|
||||
const provenance = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
}
|
||||
mockBuildTaggedMcpToolSchemas.mockImplementationOnce(
|
||||
async (
|
||||
_userId: string,
|
||||
_workspaceId: string,
|
||||
_serverIds: string[],
|
||||
report: (value: unknown) => void
|
||||
) => {
|
||||
report(provenance)
|
||||
return []
|
||||
}
|
||||
)
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
expect(options.resolvedSecretTraceRegistry?.exportProvenance()).toEqual(provenance)
|
||||
expect(JSON.stringify(payload)).not.toContain('encrypted-secret')
|
||||
expect(JSON.stringify(payload)).not.toContain('__resolvedSecretTraceProvenance')
|
||||
return successResult()
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
...requestBody,
|
||||
contexts: [{ kind: 'mcp', label: 'Docs', serverId: 'server-1' }],
|
||||
},
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect({ status: response.status, provenance: body.__resolvedSecretTraceProvenance }).toEqual({
|
||||
status: 200,
|
||||
provenance,
|
||||
})
|
||||
})
|
||||
|
||||
it('marks the lifecycle registry incomplete for malformed MCP discovery provenance', async () => {
|
||||
mockBuildTaggedMcpToolSchemas.mockImplementationOnce(
|
||||
async (
|
||||
_userId: string,
|
||||
_workspaceId: string,
|
||||
_serverIds: string[],
|
||||
report: (value: unknown) => void
|
||||
) => {
|
||||
report({ version: 1, complete: true, entries: 'invalid' })
|
||||
return []
|
||||
}
|
||||
)
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
expect(options.resolvedSecretTraceRegistry?.isComplete()).toBe(false)
|
||||
return successResult()
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
...requestBody,
|
||||
contexts: [{ kind: 'mcp', label: 'Docs', serverId: 'server-1' }],
|
||||
},
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(body.__resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('returns encrypted provenance with marker-gated failures', async () => {
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
activateSecret(options)
|
||||
return {
|
||||
...successResult(),
|
||||
success: false,
|
||||
error: 'failed with secret-value',
|
||||
content: 'secret-value',
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
requestBody,
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(500)
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
|
||||
'resolved-secret-provenance-v1'
|
||||
)
|
||||
expect(body.content).toBe('secret-value')
|
||||
expect(body.__resolvedSecretTraceProvenance.entries).toEqual([
|
||||
{ name: 'API_KEY', encryptedValue: 'encrypted-secret' },
|
||||
])
|
||||
})
|
||||
|
||||
it('places encrypted provenance only on the terminal streamed event', async () => {
|
||||
mockRunHeadlessCopilotLifecycle.mockImplementation(
|
||||
async (_payload: Record<string, unknown>, options: CopilotLifecycleOptions) => {
|
||||
activateSecret(options)
|
||||
return successResult()
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
requestBody,
|
||||
{
|
||||
Authorization: 'Bearer internal',
|
||||
'x-sim-billing-attribution': 'billing',
|
||||
'x-sim-request-private-tool-metadata': 'resolved-secret-provenance-v1',
|
||||
'x-mothership-execute-stream': 'ndjson',
|
||||
},
|
||||
'http://localhost:3000/api/mothership/execute'
|
||||
)
|
||||
)
|
||||
const events = (await response.text())
|
||||
.trim()
|
||||
.split('\n')
|
||||
.map((line) => JSON.parse(line) as Record<string, unknown>)
|
||||
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
|
||||
'resolved-secret-provenance-v1'
|
||||
)
|
||||
expect(events[0]).toMatchObject({ type: 'heartbeat' })
|
||||
expect(events[0]).not.toHaveProperty('__resolvedSecretTraceProvenance')
|
||||
expect(events.at(-1)).toMatchObject({
|
||||
type: 'final',
|
||||
data: {
|
||||
content: 'secret-value',
|
||||
__resolvedSecretTraceProvenance: {
|
||||
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
|
||||
},
|
||||
},
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
@@ -20,10 +20,23 @@ import { requestExplicitStreamAbort } from '@/lib/copilot/request/session/explic
|
||||
import type { StreamEvent } from '@/lib/copilot/request/types'
|
||||
import { isDocSandboxEnabled } from '@/lib/core/config/env-flags'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { getPersonalAndWorkspaceEnv } from '@/lib/environment/utils'
|
||||
import {
|
||||
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
requestsPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
import {
|
||||
assertActiveWorkspaceAccess,
|
||||
isWorkspaceAccessDeniedError,
|
||||
} from '@/lib/workspaces/permissions/utils'
|
||||
import {
|
||||
createIncompleteResolvedSecretTraceRegistry,
|
||||
createResolvedSecretTraceRegistry,
|
||||
ResolvedSecretTraceProvenanceAccumulator,
|
||||
type ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { ChatContext } from '@/stores/panel'
|
||||
|
||||
export const maxDuration = 3600
|
||||
@@ -35,6 +48,28 @@ const MOTHERSHIP_EXECUTE_STREAM_CONTENT_TYPE = 'application/x-ndjson'
|
||||
const MOTHERSHIP_EXECUTE_HEARTBEAT_INTERVAL_MS = 15_000
|
||||
const ndjsonEncoder = new TextEncoder()
|
||||
|
||||
function withPrivateProvenance<T extends Record<string, unknown>>(
|
||||
payload: T,
|
||||
registry: ResolvedSecretTraceRegistry | undefined,
|
||||
include: boolean
|
||||
): T & Partial<Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>> {
|
||||
return {
|
||||
...payload,
|
||||
...(include && registry
|
||||
? { [RESOLVED_SECRET_PROVENANCE_FIELD]: registry.exportProvenance() }
|
||||
: {}),
|
||||
}
|
||||
}
|
||||
|
||||
function privateResponseHeaders(
|
||||
registry: ResolvedSecretTraceRegistry | undefined,
|
||||
include: boolean
|
||||
): Record<string, string> {
|
||||
return include && registry
|
||||
? { [PRIVATE_TOOL_METADATA_RESPONSE_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1 }
|
||||
: {}
|
||||
}
|
||||
|
||||
function isAbortError(error: unknown): boolean {
|
||||
return error instanceof Error && error.name === 'AbortError'
|
||||
}
|
||||
@@ -99,6 +134,11 @@ export function buildExecuteResponsePayload(
|
||||
export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
let messageId: string | undefined
|
||||
let requestId: string | undefined
|
||||
let resolvedSecretTraceRegistry: ResolvedSecretTraceRegistry | undefined
|
||||
const includePrivateProvenance = requestsPrivateToolMetadata(
|
||||
req.headers,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1
|
||||
)
|
||||
|
||||
try {
|
||||
const auth = await checkInternalAuth(req, { requireWorkflowId: false })
|
||||
@@ -146,6 +186,39 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
actorUserId: userId,
|
||||
workspaceId,
|
||||
})
|
||||
if (includePrivateProvenance) {
|
||||
const scope = { userId, workspaceId }
|
||||
try {
|
||||
const environment = await getPersonalAndWorkspaceEnv(userId, workspaceId, {
|
||||
workspaceAccess,
|
||||
})
|
||||
resolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: environment.personalEncrypted,
|
||||
workspaceEncrypted: environment.workspaceEncrypted,
|
||||
personalDecrypted: environment.personalDecrypted,
|
||||
workspaceDecrypted: environment.workspaceDecrypted,
|
||||
decryptionFailures: environment.decryptionFailures,
|
||||
scope,
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn('Failed to build Mothership trace secret catalog', {
|
||||
error: getErrorMessage(error),
|
||||
userId,
|
||||
workspaceId,
|
||||
})
|
||||
resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(scope)
|
||||
}
|
||||
}
|
||||
const activeResolvedSecretTraceRegistry = resolvedSecretTraceRegistry
|
||||
const mcpDiscoveryProvenance = new ResolvedSecretTraceProvenanceAccumulator({
|
||||
userId,
|
||||
workspaceId,
|
||||
})
|
||||
const recordMcpDiscoveryProvenance = includePrivateProvenance
|
||||
? (provenance: unknown): void => {
|
||||
mcpDiscoveryProvenance.record(provenance)
|
||||
}
|
||||
: undefined
|
||||
|
||||
const effectiveChatId = chatId || generateId()
|
||||
messageId = providedMessageId || generateId()
|
||||
@@ -164,17 +237,38 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
)
|
||||
const nonMcpAgentMentions = agentMentions?.filter((context) => context.kind !== 'mcp')
|
||||
const userPermission = workspaceAccess.permission
|
||||
const mothershipToolsPromise = Promise.allSettled([
|
||||
buildSelectedMcpToolSchemas(
|
||||
userId,
|
||||
workspaceId,
|
||||
mcpTools ?? [],
|
||||
recordMcpDiscoveryProvenance
|
||||
),
|
||||
buildTaggedMcpToolSchemas(
|
||||
userId,
|
||||
workspaceId,
|
||||
taggedMcpServerIds,
|
||||
recordMcpDiscoveryProvenance
|
||||
),
|
||||
]).then(async (results) => {
|
||||
if (activeResolvedSecretTraceRegistry) {
|
||||
await activeResolvedSecretTraceRegistry.importProvenance(
|
||||
mcpDiscoveryProvenance.exportProvenance(),
|
||||
{ trusted: true }
|
||||
)
|
||||
}
|
||||
const groups = results.map((result) => {
|
||||
if (result.status === 'rejected') throw result.reason
|
||||
return result.value
|
||||
})
|
||||
const byName = new Map(groups.flat().map((tool) => [tool.name, tool]))
|
||||
return [...byName.values()]
|
||||
})
|
||||
const [workspaceContext, integrationTools, mothershipTools, entitlements, agentContexts] =
|
||||
await Promise.all([
|
||||
generateWorkspaceContext(workspaceId, userId, { workspaceAccess }),
|
||||
buildIntegrationToolSchemas(userId, messageId, undefined, workspaceId),
|
||||
Promise.all([
|
||||
buildSelectedMcpToolSchemas(userId, workspaceId, mcpTools ?? []),
|
||||
buildTaggedMcpToolSchemas(userId, workspaceId, taggedMcpServerIds),
|
||||
]).then((groups) => {
|
||||
const byName = new Map(groups.flat().map((tool) => [tool.name, tool]))
|
||||
return [...byName.values()]
|
||||
}),
|
||||
mothershipToolsPromise,
|
||||
computeWorkspaceEntitlements(workspaceId, userId),
|
||||
processContextsServer(
|
||||
nonMcpAgentMentions,
|
||||
@@ -282,6 +376,7 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
interactive: false,
|
||||
abortSignal: lifecycleAbortController.signal,
|
||||
billingAttribution,
|
||||
resolvedSecretTraceRegistry,
|
||||
onEvent,
|
||||
})
|
||||
|
||||
@@ -326,7 +421,13 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
allowExplicitAbort = false
|
||||
|
||||
if (lifecycleAbortController.signal.aborted) {
|
||||
send({ type: 'error', error: 'Sim execution aborted' })
|
||||
send(
|
||||
withPrivateProvenance(
|
||||
{ type: 'error', error: 'Sim execution aborted' },
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
)
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -343,17 +444,27 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
errors: result.errors,
|
||||
}
|
||||
)
|
||||
send({
|
||||
type: 'error',
|
||||
error: result.error || 'Sim execution failed',
|
||||
content: result.content || '',
|
||||
})
|
||||
send(
|
||||
withPrivateProvenance(
|
||||
{
|
||||
type: 'error',
|
||||
error: result.error || 'Sim execution failed',
|
||||
content: result.content || '',
|
||||
},
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
)
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
send({
|
||||
type: 'final',
|
||||
data: buildExecuteResponsePayload(result, effectiveChatId, integrationTools),
|
||||
data: withPrivateProvenance(
|
||||
buildExecuteResponsePayload(result, effectiveChatId, integrationTools),
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
),
|
||||
})
|
||||
} catch (error) {
|
||||
if (
|
||||
@@ -367,7 +478,13 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
: 'Mothership execute aborted',
|
||||
{ requestId }
|
||||
)
|
||||
send({ type: 'error', error: 'Sim execution aborted' })
|
||||
send(
|
||||
withPrivateProvenance(
|
||||
{ type: 'error', error: 'Sim execution aborted' },
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
)
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -380,10 +497,16 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
error: getErrorMessage(error, 'Unknown error'),
|
||||
}
|
||||
)
|
||||
send({
|
||||
type: 'error',
|
||||
error: getErrorMessage(error, 'Internal server error'),
|
||||
})
|
||||
send(
|
||||
withPrivateProvenance(
|
||||
{
|
||||
type: 'error',
|
||||
error: getErrorMessage(error, 'Internal server error'),
|
||||
},
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
)
|
||||
)
|
||||
} finally {
|
||||
allowExplicitAbort = false
|
||||
if (heartbeatId) {
|
||||
@@ -410,6 +533,7 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
headers: {
|
||||
'Content-Type': `${MOTHERSHIP_EXECUTE_STREAM_CONTENT_TYPE}; charset=utf-8`,
|
||||
'Cache-Control': 'no-cache, no-transform',
|
||||
...privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
|
||||
},
|
||||
})
|
||||
}
|
||||
@@ -421,7 +545,17 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
|
||||
if (lifecycleAbortController.signal.aborted || req.signal.aborted) {
|
||||
reqLogger.info('Mothership execute aborted after lifecycle completion')
|
||||
return NextResponse.json({ error: 'Sim execution aborted' }, { status: 499 })
|
||||
return NextResponse.json(
|
||||
withPrivateProvenance(
|
||||
{ error: 'Sim execution aborted' },
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
),
|
||||
{
|
||||
status: 499,
|
||||
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
if (!result.success) {
|
||||
@@ -438,16 +572,30 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
}
|
||||
)
|
||||
return NextResponse.json(
|
||||
withPrivateProvenance(
|
||||
{
|
||||
error: result.error || 'Sim execution failed',
|
||||
content: result.content || '',
|
||||
},
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
),
|
||||
{
|
||||
error: result.error || 'Sim execution failed',
|
||||
content: result.content || '',
|
||||
},
|
||||
{ status: 500 }
|
||||
status: 500,
|
||||
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
return NextResponse.json(
|
||||
buildExecuteResponsePayload(result, effectiveChatId, integrationTools)
|
||||
withPrivateProvenance(
|
||||
buildExecuteResponsePayload(result, effectiveChatId, integrationTools),
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
),
|
||||
{
|
||||
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
|
||||
}
|
||||
)
|
||||
} finally {
|
||||
allowExplicitAbort = false
|
||||
@@ -465,7 +613,17 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
}
|
||||
)
|
||||
|
||||
return NextResponse.json({ error: 'Sim execution aborted' }, { status: 499 })
|
||||
return NextResponse.json(
|
||||
withPrivateProvenance(
|
||||
{ error: 'Sim execution aborted' },
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
),
|
||||
{
|
||||
status: 499,
|
||||
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
if (isWorkspaceAccessDeniedError(error)) {
|
||||
@@ -481,8 +639,15 @@ export const POST = withRouteHandler(async (req: NextRequest) => {
|
||||
)
|
||||
|
||||
return NextResponse.json(
|
||||
{ error: getErrorMessage(error, 'Internal server error') },
|
||||
{ status: 500 }
|
||||
withPrivateProvenance(
|
||||
{ error: getErrorMessage(error, 'Internal server error') },
|
||||
resolvedSecretTraceRegistry,
|
||||
includePrivateProvenance
|
||||
),
|
||||
{
|
||||
status: 500,
|
||||
headers: privateResponseHeaders(resolvedSecretTraceRegistry, includePrivateProvenance),
|
||||
}
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -7,7 +7,7 @@ import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { v1GetLogContract } from '@/lib/api/contracts/v1/logs'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
import { createApiResponse, getUserLimits } from '@/app/api/v1/logs/meta'
|
||||
import {
|
||||
checkRateLimit,
|
||||
@@ -100,12 +100,13 @@ export const GET = withRouteHandler(
|
||||
totalDurationMs: log.totalDurationMs,
|
||||
files: log.files || undefined,
|
||||
workflow: workflowSummary,
|
||||
executionData: (await materializeExecutionData(
|
||||
executionData: (await materializeExecutionDataForDisplay(
|
||||
log.executionData as Record<string, unknown> | null,
|
||||
{
|
||||
workspaceId: log.workspaceId,
|
||||
workflowId: log.workflowId,
|
||||
executionId: log.executionId,
|
||||
userId,
|
||||
}
|
||||
)) as any,
|
||||
cost: log.costTotal != null ? { total: Number(log.costTotal) } : null,
|
||||
|
||||
@@ -8,7 +8,7 @@ import { v1ListLogsContract } from '@/lib/api/contracts/v1/logs'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { MATERIALIZE_CONCURRENCY, mapWithConcurrency } from '@/lib/core/utils/concurrency'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
import { buildLogFilters, getOrderBy } from '@/app/api/v1/logs/filters'
|
||||
import { createApiResponse, getUserLimits } from '@/app/api/v1/logs/meta'
|
||||
import {
|
||||
@@ -167,12 +167,13 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
|
||||
? await mapWithConcurrency(data, MATERIALIZE_CONCURRENCY, async (log) => {
|
||||
const result = buildBase(log)
|
||||
if (log.executionData) {
|
||||
const execData = (await materializeExecutionData(
|
||||
const execData = (await materializeExecutionDataForDisplay(
|
||||
log.executionData as Record<string, unknown> | null,
|
||||
{
|
||||
workspaceId: log.workspaceId,
|
||||
workflowId: log.workflowId,
|
||||
executionId: log.executionId,
|
||||
userId,
|
||||
}
|
||||
)) as any
|
||||
if (params.includeFinalOutput && execData.finalOutput) {
|
||||
|
||||
@@ -9,9 +9,11 @@ import {
|
||||
executionPreprocessingMockFns,
|
||||
hybridAuthMockFns,
|
||||
loggingSessionMock,
|
||||
queueTableRows,
|
||||
requestUtilsMockFns,
|
||||
resetDbChainMock,
|
||||
resetEnvMock,
|
||||
schemaMock,
|
||||
setEnv,
|
||||
workflowAuthzMockFns,
|
||||
workflowsPersistenceUtilsMock,
|
||||
@@ -339,6 +341,125 @@ describe('workflow execute async route', () => {
|
||||
mockHandlePostExecutionPauseState.mockResolvedValue(undefined)
|
||||
})
|
||||
|
||||
it('reuses raw workflow input by execution ID without returning it to the client', async () => {
|
||||
const sourceInput = { token: 'raw-secret-1234', nested: { value: 42 } }
|
||||
queueTableRows(schemaMock.workflowExecutionLogs, [
|
||||
{
|
||||
executionId: 'source-execution',
|
||||
workflowId: 'workflow-1',
|
||||
workspaceId: 'workspace-1',
|
||||
executionData: { workflowInput: sourceInput },
|
||||
},
|
||||
])
|
||||
const request = createMockRequest(
|
||||
'POST',
|
||||
{ inputFromExecutionId: 'source-execution' },
|
||||
{
|
||||
'Content-Type': 'application/json',
|
||||
'X-Execution-Mode': 'async',
|
||||
Cookie: 'session=value',
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
|
||||
const responseBody = await response.json()
|
||||
|
||||
expect(response.status).toBe(202)
|
||||
expect(responseBody).not.toHaveProperty('input')
|
||||
expect(JSON.stringify(responseBody)).not.toContain('raw-secret-1234')
|
||||
expect(mockEnqueue).toHaveBeenCalledWith(
|
||||
'workflow-execution',
|
||||
expect.objectContaining({ input: sourceInput }),
|
||||
expect.any(Object)
|
||||
)
|
||||
})
|
||||
|
||||
it('recovers legacy starter input by execution ID without returning it to the client', async () => {
|
||||
const sourceInput = { token: 'legacy-retry-input', nested: { value: 42 } }
|
||||
queueTableRows(schemaMock.workflowExecutionLogs, [
|
||||
{
|
||||
executionId: 'source-execution',
|
||||
workflowId: 'workflow-1',
|
||||
workspaceId: 'workspace-1',
|
||||
executionData: {
|
||||
executionState: {
|
||||
blockStates: {
|
||||
start: {
|
||||
output: sourceInput,
|
||||
executed: false,
|
||||
executionTime: 0,
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
])
|
||||
const request = createMockRequest(
|
||||
'POST',
|
||||
{ inputFromExecutionId: 'source-execution' },
|
||||
{
|
||||
'Content-Type': 'application/json',
|
||||
'X-Execution-Mode': 'async',
|
||||
Cookie: 'session=value',
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
|
||||
const responseBody = await response.json()
|
||||
|
||||
expect(response.status).toBe(202)
|
||||
expect(responseBody).not.toHaveProperty('input')
|
||||
expect(JSON.stringify(responseBody)).not.toContain('legacy-retry-input')
|
||||
expect(mockEnqueue).toHaveBeenCalledWith(
|
||||
'workflow-execution',
|
||||
expect.objectContaining({ input: sourceInput }),
|
||||
expect.any(Object)
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects client input alongside a stored execution input reference', async () => {
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
input: { replacement: true },
|
||||
inputFromExecutionId: 'source-execution',
|
||||
},
|
||||
{ 'Content-Type': 'application/json', Cookie: 'session=value' }
|
||||
),
|
||||
{ params: Promise.resolve({ id: 'workflow-1' }) }
|
||||
)
|
||||
|
||||
expect(response.status).toBe(400)
|
||||
await expect(response.json()).resolves.toEqual({
|
||||
error: 'Provide either input or inputFromExecutionId, not both',
|
||||
})
|
||||
expect(mockEnqueue).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects stored execution input references from external callers', async () => {
|
||||
mockCheckHybridAuth.mockResolvedValue({
|
||||
success: true,
|
||||
userId: 'personal-key-user-1',
|
||||
authType: 'api_key',
|
||||
apiKeyType: 'personal',
|
||||
})
|
||||
const response = await POST(
|
||||
createMockRequest(
|
||||
'POST',
|
||||
{ inputFromExecutionId: 'source-execution' },
|
||||
{ 'Content-Type': 'application/json', 'X-API-Key': 'personal-key' }
|
||||
),
|
||||
{ params: Promise.resolve({ id: 'workflow-1' }) }
|
||||
)
|
||||
|
||||
expect(response.status).toBe(403)
|
||||
await expect(response.json()).resolves.toEqual({
|
||||
error: 'Stored execution input can only be reused by an authenticated session',
|
||||
})
|
||||
expect(mockEnqueue).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('queues async execution with matching correlation metadata', async () => {
|
||||
const req = createMockRequest(
|
||||
'POST',
|
||||
@@ -750,6 +871,173 @@ describe('workflow execute async route', () => {
|
||||
expect(mockReleaseExecutionIdClaim).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('loads trusted run-from-block state by execution ID and preserves its source identity', async () => {
|
||||
const sourceState = {
|
||||
blockStates: { previous: { output: { value: 'cached' } } },
|
||||
executedBlocks: ['previous'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: [],
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: 'owner-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
}
|
||||
queueTableRows(schemaMock.workflowExecutionLogs, [
|
||||
{
|
||||
executionId: 'source-execution',
|
||||
workflowId: 'workflow-1',
|
||||
workspaceId: 'workspace-1',
|
||||
executionData: { executionState: sourceState },
|
||||
},
|
||||
])
|
||||
const request = createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
input: { hello: 'world' },
|
||||
runFromBlock: {
|
||||
startBlockId: 'start-block',
|
||||
executionId: 'source-execution',
|
||||
},
|
||||
},
|
||||
{
|
||||
'Content-Type': 'application/json',
|
||||
Cookie: 'session=value',
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(mockExecuteWorkflowCore).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
runFromBlock: {
|
||||
startBlockId: 'start-block',
|
||||
sourceSnapshot: sourceState,
|
||||
sourceExecutionId: 'source-execution',
|
||||
},
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('falls back to an untrusted client snapshot while stored run-from-block state is pending', async () => {
|
||||
const sourceSnapshot = {
|
||||
blockStates: { previous: { output: { value: 'cached' } } },
|
||||
executedBlocks: ['previous'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: [],
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'untrusted-ciphertext' }],
|
||||
},
|
||||
}
|
||||
queueTableRows(schemaMock.workflowExecutionLogs, [])
|
||||
const request = createMockRequest(
|
||||
'POST',
|
||||
{
|
||||
input: { hello: 'world' },
|
||||
runFromBlock: {
|
||||
startBlockId: 'start-block',
|
||||
executionId: 'source-execution',
|
||||
sourceSnapshot,
|
||||
},
|
||||
},
|
||||
{
|
||||
'Content-Type': 'application/json',
|
||||
Cookie: 'session=value',
|
||||
}
|
||||
)
|
||||
|
||||
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(mockExecuteWorkflowCore).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
runFromBlock: {
|
||||
startBlockId: 'start-block',
|
||||
sourceSnapshot: expect.objectContaining({
|
||||
blockStates: sourceSnapshot.blockStates,
|
||||
executedBlocks: sourceSnapshot.executedBlocks,
|
||||
}),
|
||||
},
|
||||
})
|
||||
)
|
||||
const runFromBlock = mockExecuteWorkflowCore.mock.calls[0]?.[0]?.runFromBlock
|
||||
expect(runFromBlock).not.toHaveProperty('sourceExecutionId')
|
||||
expect(runFromBlock?.sourceSnapshot).not.toHaveProperty('resolvedSecretTraceProvenance')
|
||||
})
|
||||
|
||||
it('returns encrypted resolution provenance only to an authenticated internal tool caller', async () => {
|
||||
const caller = EXECUTION_CALLERS[4]
|
||||
configureExecutionCaller(caller)
|
||||
const provenance = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'CHILD_SECRET', encryptedValue: 'encrypted-child-secret' }],
|
||||
}
|
||||
mockExecuteWorkflowCore.mockResolvedValueOnce({
|
||||
success: true,
|
||||
status: 'completed',
|
||||
output: { ok: true },
|
||||
executionState: { resolvedSecretTraceProvenance: provenance },
|
||||
metadata: {
|
||||
duration: 100,
|
||||
startTime: '2026-01-01T00:00:00Z',
|
||||
endTime: '2026-01-01T00:00:01Z',
|
||||
},
|
||||
})
|
||||
const request = createCallerExecutionRequest(caller, undefined, 'sync')
|
||||
request.headers.set('x-sim-request-private-tool-metadata', 'resolved-secret-provenance-v1')
|
||||
|
||||
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBe(
|
||||
'resolved-secret-provenance-v1'
|
||||
)
|
||||
await expect(response.json()).resolves.toMatchObject({
|
||||
output: { ok: true },
|
||||
__resolvedSecretTraceProvenance: provenance,
|
||||
})
|
||||
})
|
||||
|
||||
it('does not expose private provenance metadata to non-internal callers', async () => {
|
||||
const caller = EXECUTION_CALLERS[1]
|
||||
configureExecutionCaller(caller)
|
||||
mockExecuteWorkflowCore.mockResolvedValueOnce({
|
||||
success: true,
|
||||
status: 'completed',
|
||||
output: { ok: true },
|
||||
executionState: {
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'SECRET', encryptedValue: 'encrypted-secret' }],
|
||||
},
|
||||
},
|
||||
metadata: {
|
||||
duration: 100,
|
||||
startTime: '2026-01-01T00:00:00Z',
|
||||
endTime: '2026-01-01T00:00:01Z',
|
||||
},
|
||||
})
|
||||
const request = createCallerExecutionRequest(caller, undefined, 'sync')
|
||||
request.headers.set('x-sim-request-private-tool-metadata', 'resolved-secret-provenance-v1')
|
||||
|
||||
const response = await POST(request, { params: Promise.resolve({ id: 'workflow-1' }) })
|
||||
const body = await response.json()
|
||||
|
||||
expect(response.status).toBe(200)
|
||||
expect(response.headers.get('x-sim-private-tool-metadata')).toBeNull()
|
||||
expect(body).not.toHaveProperty('__resolvedSecretTraceProvenance')
|
||||
})
|
||||
|
||||
it('releases the admission reservation when enqueue proves non-acceptance', async () => {
|
||||
mockEnqueue.mockRejectedValueOnce(
|
||||
new AsyncJobEnqueueError('queue rejected the job', {
|
||||
|
||||
@@ -58,6 +58,12 @@ import {
|
||||
import { containsLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
|
||||
import { compactBlockLogs, compactExecutionPayload } from '@/lib/execution/payloads/serializer'
|
||||
import { type PreprocessExecutionSuccess, preprocessExecution } from '@/lib/execution/preprocessing'
|
||||
import {
|
||||
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
requestsPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
import { LoggingSession } from '@/lib/logs/execution/logging-session'
|
||||
import {
|
||||
MAX_MCP_WORKFLOW_RESPONSE_BYTES,
|
||||
@@ -114,12 +120,18 @@ import {
|
||||
import { normalizeName } from '@/executor/constants'
|
||||
import { ExecutionSnapshot } from '@/executor/execution/snapshot'
|
||||
import type {
|
||||
BlockCompletionCallbackData,
|
||||
ChildWorkflowContext,
|
||||
ExecutionMetadata,
|
||||
IterationContext,
|
||||
SerializableExecutionState,
|
||||
} from '@/executor/execution/types'
|
||||
import type { BlockLog, NormalizedBlockOutput, StreamingExecution } from '@/executor/types'
|
||||
import type {
|
||||
BlockLog,
|
||||
ExecutionResult,
|
||||
NormalizedBlockOutput,
|
||||
StreamingExecution,
|
||||
} from '@/executor/types'
|
||||
import { getExecutionErrorStatus, hasExecutionResult } from '@/executor/utils/errors'
|
||||
import { Serializer } from '@/serializer'
|
||||
import { CORE_TRIGGER_TYPES, type CoreTriggerType } from '@/stores/logs/filters/types'
|
||||
@@ -133,6 +145,31 @@ const WORKFLOW_EXECUTION_JOB_ID_PREFIX = 'workflow-execution:'
|
||||
export const runtime = 'nodejs'
|
||||
export const dynamic = 'force-dynamic'
|
||||
|
||||
function createExecutionJsonResponse(
|
||||
body: Record<string, unknown>,
|
||||
init: ResponseInit | undefined,
|
||||
includePrivateProvenance: boolean,
|
||||
result?: ExecutionResult
|
||||
): NextResponse {
|
||||
if (!includePrivateProvenance) {
|
||||
return NextResponse.json(body, init)
|
||||
}
|
||||
|
||||
const headers = new Headers(init?.headers)
|
||||
headers.set(PRIVATE_TOOL_METADATA_RESPONSE_HEADER, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
|
||||
return NextResponse.json(
|
||||
{
|
||||
...body,
|
||||
[RESOLVED_SECRET_PROVENANCE_FIELD]: result?.executionState?.resolvedSecretTraceProvenance ?? {
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
},
|
||||
},
|
||||
{ ...init, headers }
|
||||
)
|
||||
}
|
||||
|
||||
async function compactRoutePayload<T>(
|
||||
value: T,
|
||||
context: {
|
||||
@@ -573,6 +610,10 @@ async function handleExecutePost(
|
||||
|
||||
const isMcpBridgeRequest =
|
||||
auth.authType === AuthType.INTERNAL_JWT && req.headers.get(MCP_TOOL_BRIDGE_HEADER) === 'true'
|
||||
const includePrivateTraceProvenance =
|
||||
auth.success &&
|
||||
auth.authType === AuthType.INTERNAL_JWT &&
|
||||
requestsPrivateToolMetadata(req.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
|
||||
const useMcpBridgeAuthenticatedUserAsActor =
|
||||
isMcpBridgeRequest && req.headers.get(MCP_TOOL_BRIDGE_ACTOR_HEADER) === 'authenticated-user'
|
||||
|
||||
@@ -682,6 +723,7 @@ async function handleExecutePost(
|
||||
includeToolCalls: requestedIncludeToolCalls,
|
||||
useDraftState,
|
||||
input: validatedInput,
|
||||
inputFromExecutionId,
|
||||
isClientSession = false,
|
||||
includeFileBase64,
|
||||
base64MaxBytes,
|
||||
@@ -730,6 +772,20 @@ async function handleExecutePost(
|
||||
)
|
||||
}
|
||||
|
||||
if (inputFromExecutionId && (isPublicApiAccess || auth.authType !== AuthType.SESSION)) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Stored execution input can only be reused by an authenticated session' },
|
||||
{ status: 403 }
|
||||
)
|
||||
}
|
||||
|
||||
if (inputFromExecutionId && validatedInput !== undefined) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Provide either input or inputFromExecutionId, not both' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
|
||||
if (auth.authType === 'api_key') {
|
||||
if (isClientSession) {
|
||||
return NextResponse.json(
|
||||
@@ -776,14 +832,7 @@ async function handleExecutePost(
|
||||
)
|
||||
}
|
||||
|
||||
if (rawRunFromBlock.sourceSnapshot && !isPublicApiAccess) {
|
||||
// Public API callers cannot inject arbitrary block state via sourceSnapshot.
|
||||
// They must use executionId to resume from a server-stored execution state.
|
||||
resolvedRunFromBlock = {
|
||||
startBlockId: rawRunFromBlock.startBlockId,
|
||||
sourceSnapshot: rawRunFromBlock.sourceSnapshot as SerializableExecutionState,
|
||||
}
|
||||
} else if (rawRunFromBlock.executionId) {
|
||||
if (rawRunFromBlock.executionId) {
|
||||
const { getExecutionStateForWorkflow, getLatestExecutionStateWithExecutionId } =
|
||||
await import('@/lib/workflows/executor/execution-state')
|
||||
const sourceExecution =
|
||||
@@ -795,17 +844,32 @@ async function handleExecutePost(
|
||||
}
|
||||
const snapshot = sourceExecution?.state
|
||||
if (!snapshot) {
|
||||
return NextResponse.json(
|
||||
{
|
||||
error: `No execution state found for ${rawRunFromBlock.executionId === 'latest' ? 'workflow' : `execution ${rawRunFromBlock.executionId}`}. Run the full workflow first.`,
|
||||
},
|
||||
{ status: 400 }
|
||||
)
|
||||
if (rawRunFromBlock.sourceSnapshot && !isPublicApiAccess) {
|
||||
resolvedRunFromBlock = {
|
||||
startBlockId: rawRunFromBlock.startBlockId,
|
||||
sourceSnapshot: rawRunFromBlock.sourceSnapshot as SerializableExecutionState,
|
||||
}
|
||||
} else {
|
||||
return NextResponse.json(
|
||||
{
|
||||
error: `No execution state found for ${rawRunFromBlock.executionId === 'latest' ? 'workflow' : `execution ${rawRunFromBlock.executionId}`}. Run the full workflow first.`,
|
||||
},
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
} else {
|
||||
resolvedRunFromBlock = {
|
||||
startBlockId: rawRunFromBlock.startBlockId,
|
||||
sourceSnapshot: snapshot,
|
||||
sourceExecutionId: sourceExecution.executionId,
|
||||
}
|
||||
}
|
||||
} else if (rawRunFromBlock.sourceSnapshot && !isPublicApiAccess) {
|
||||
// Public API callers cannot inject arbitrary block state via sourceSnapshot.
|
||||
// They must use executionId to resume from a server-stored execution state.
|
||||
resolvedRunFromBlock = {
|
||||
startBlockId: rawRunFromBlock.startBlockId,
|
||||
sourceSnapshot: snapshot,
|
||||
sourceExecutionId: sourceExecution.executionId,
|
||||
sourceSnapshot: rawRunFromBlock.sourceSnapshot as SerializableExecutionState,
|
||||
}
|
||||
} else {
|
||||
return NextResponse.json(
|
||||
@@ -817,7 +881,7 @@ async function handleExecutePost(
|
||||
|
||||
// For API key and internal JWT auth, the entire body is the input (except for our control fields)
|
||||
// For session auth, the input is explicitly provided in the input field
|
||||
const input = isMcpBridgeRequest
|
||||
let input = isMcpBridgeRequest
|
||||
? validatedInput
|
||||
: isPublicApiAccess ||
|
||||
auth.authType === AuthType.API_KEY ||
|
||||
@@ -828,6 +892,7 @@ async function handleExecutePost(
|
||||
triggerType,
|
||||
stream,
|
||||
useDraftState,
|
||||
inputFromExecutionId: _inputFromExecutionId,
|
||||
includeFileBase64,
|
||||
base64MaxBytes,
|
||||
workflowStateOverride,
|
||||
@@ -962,6 +1027,20 @@ async function handleExecutePost(
|
||||
)
|
||||
}
|
||||
|
||||
if (inputFromExecutionId) {
|
||||
const { getExecutionInputForWorkflow } = await import(
|
||||
'@/lib/workflows/executor/execution-state'
|
||||
)
|
||||
const sourceExecution = await getExecutionInputForWorkflow(inputFromExecutionId, workflowId)
|
||||
if (!sourceExecution.found) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Source workflow execution was not found' },
|
||||
{ status: 404 }
|
||||
)
|
||||
}
|
||||
input = sourceExecution.input
|
||||
}
|
||||
|
||||
const upstreamBillingAttribution =
|
||||
auth.authType === AuthType.INTERNAL_JWT && workflowAuthorization.workflow?.workspaceId
|
||||
? requireBillingAttributionHeader(req.headers, {
|
||||
@@ -1287,7 +1366,7 @@ async function handleExecutePost(
|
||||
workflowResponseCompaction
|
||||
)
|
||||
|
||||
return NextResponse.json(
|
||||
return createExecutionJsonResponse(
|
||||
{
|
||||
success: false,
|
||||
output: compactResultOutput,
|
||||
@@ -1300,7 +1379,9 @@ async function handleExecutePost(
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
{ status: 408 }
|
||||
{ status: 408 },
|
||||
includePrivateTraceProvenance,
|
||||
result
|
||||
)
|
||||
}
|
||||
|
||||
@@ -1367,7 +1448,12 @@ async function handleExecutePost(
|
||||
: undefined,
|
||||
}
|
||||
|
||||
return NextResponse.json(filteredResult)
|
||||
return createExecutionJsonResponse(
|
||||
filteredResult,
|
||||
undefined,
|
||||
includePrivateTraceProvenance,
|
||||
result
|
||||
)
|
||||
} catch (error: unknown) {
|
||||
const errorMessage = getErrorMessage(error, 'Unknown error')
|
||||
|
||||
@@ -1405,7 +1491,7 @@ async function handleExecutePost(
|
||||
throw compactError
|
||||
}
|
||||
}
|
||||
return NextResponse.json(
|
||||
return createExecutionJsonResponse(
|
||||
{
|
||||
success: false,
|
||||
output: compactErrorOutput,
|
||||
@@ -1418,7 +1504,9 @@ async function handleExecutePost(
|
||||
}
|
||||
: undefined,
|
||||
},
|
||||
{ status }
|
||||
{ status },
|
||||
includePrivateTraceProvenance,
|
||||
executionResult
|
||||
)
|
||||
} finally {
|
||||
requestAbort.cleanup()
|
||||
@@ -1663,7 +1751,7 @@ async function handleExecutePost(
|
||||
blockId: string,
|
||||
blockName: string,
|
||||
blockType: string,
|
||||
callbackData: any,
|
||||
callbackData: BlockCompletionCallbackData,
|
||||
iterationContext?: IterationContext,
|
||||
childWorkflowContext?: ChildWorkflowContext
|
||||
) => {
|
||||
@@ -1686,7 +1774,13 @@ async function handleExecutePost(
|
||||
preserveRoot: true,
|
||||
}),
|
||||
}
|
||||
const hasError = compactCallbackData.output?.error
|
||||
const callbackError = compactCallbackData.output?.error
|
||||
const hasError = typeof callbackError === 'string' && callbackError.length > 0
|
||||
const display = await loggingSession.projectDisplayContent({
|
||||
input: compactCallbackData.input,
|
||||
output: compactCallbackData.output,
|
||||
...(hasError ? { error: callbackError } : {}),
|
||||
})
|
||||
const childWorkflowData = childWorkflowContext
|
||||
? {
|
||||
childWorkflowBlockId: childWorkflowContext.parentBlockId,
|
||||
@@ -1703,7 +1797,7 @@ async function handleExecutePost(
|
||||
blockId,
|
||||
blockName,
|
||||
blockType,
|
||||
error: compactCallbackData.output.error,
|
||||
error: display.error,
|
||||
})
|
||||
await sendEvent({
|
||||
type: 'block:error',
|
||||
@@ -1715,7 +1809,12 @@ async function handleExecutePost(
|
||||
blockName,
|
||||
blockType,
|
||||
input: compactCallbackData.input,
|
||||
error: compactCallbackData.output.error,
|
||||
error: callbackError,
|
||||
display: {
|
||||
...(Object.hasOwn(display, 'input') ? { input: display.input } : {}),
|
||||
...(display.error !== undefined ? { error: display.error } : {}),
|
||||
...(display.clearLiveDisplay ? { clearLiveDisplay: true as const } : {}),
|
||||
},
|
||||
durationMs: compactCallbackData.executionTime || 0,
|
||||
startedAt: compactCallbackData.startedAt,
|
||||
executionOrder: compactCallbackData.executionOrder,
|
||||
@@ -1750,6 +1849,11 @@ async function handleExecutePost(
|
||||
blockType,
|
||||
input: compactCallbackData.input,
|
||||
output: compactCallbackData.output,
|
||||
display: {
|
||||
...(Object.hasOwn(display, 'input') ? { input: display.input } : {}),
|
||||
...(Object.hasOwn(display, 'output') ? { output: display.output } : {}),
|
||||
...(display.clearLiveDisplay ? { clearLiveDisplay: true as const } : {}),
|
||||
},
|
||||
durationMs: compactCallbackData.executionTime || 0,
|
||||
startedAt: compactCallbackData.startedAt,
|
||||
executionOrder: compactCallbackData.executionOrder,
|
||||
@@ -1786,6 +1890,7 @@ async function handleExecutePost(
|
||||
workflowId,
|
||||
sendEvent,
|
||||
forwardAnswerText: answerTextFromSink,
|
||||
projectDisplay: (field, value) => loggingSession.projectLiveDisplayText(field, value),
|
||||
})
|
||||
|
||||
const reader = streamingExec.stream.getReader()
|
||||
@@ -1807,12 +1912,13 @@ async function handleExecutePost(
|
||||
if (answerTextFromSink) continue
|
||||
|
||||
const chunk = decoder.decode(value, { stream: true })
|
||||
const display = await loggingSession.projectLiveDisplayText('chunk', chunk)
|
||||
await sendEvent({
|
||||
type: 'stream:chunk',
|
||||
timestamp: new Date().toISOString(),
|
||||
executionId,
|
||||
workflowId,
|
||||
data: { blockId, chunk },
|
||||
data: { blockId, chunk, display },
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1931,7 +2037,10 @@ async function handleExecutePost(
|
||||
* object storage, so doing it twice would double the latency and storage
|
||||
* load on the happy path.
|
||||
*/
|
||||
const compactedBlockLogs = await compactBlockLogs(result.logs, {
|
||||
const displayBlockLogs = await loggingSession.projectBlockLogsForDisplay(
|
||||
result.logs ?? []
|
||||
)
|
||||
const compactedBlockLogs = await compactBlockLogs(displayBlockLogs, {
|
||||
workspaceId,
|
||||
workflowId,
|
||||
executionId,
|
||||
@@ -1947,6 +2056,9 @@ async function handleExecutePost(
|
||||
})
|
||||
|
||||
await loggingSession.markAsFailed(timeoutErrorMessage)
|
||||
const timeoutDisplay = await loggingSession.projectDisplayContent({
|
||||
error: timeoutErrorMessage,
|
||||
})
|
||||
|
||||
finalMetaStatus = 'error'
|
||||
await sendEvent(
|
||||
@@ -1957,6 +2069,11 @@ async function handleExecutePost(
|
||||
workflowId,
|
||||
data: {
|
||||
error: timeoutErrorMessage,
|
||||
display: {
|
||||
...(timeoutDisplay.error !== undefined
|
||||
? { error: timeoutDisplay.error }
|
||||
: {}),
|
||||
},
|
||||
duration: result.metadata?.duration || 0,
|
||||
finalBlockLogs: compactedBlockLogs,
|
||||
},
|
||||
@@ -2061,13 +2178,16 @@ async function handleExecutePost(
|
||||
let compactErrorLogs: BlockLog[] | undefined
|
||||
try {
|
||||
compactErrorLogs = executionResult?.logs
|
||||
? await compactBlockLogs(executionResult.logs, {
|
||||
workspaceId,
|
||||
workflowId,
|
||||
executionId,
|
||||
userId: actorUserId,
|
||||
requireDurable: true,
|
||||
})
|
||||
? await compactBlockLogs(
|
||||
await loggingSession.projectBlockLogsForDisplay(executionResult.logs),
|
||||
{
|
||||
workspaceId,
|
||||
workflowId,
|
||||
executionId,
|
||||
userId: actorUserId,
|
||||
requireDurable: true,
|
||||
}
|
||||
)
|
||||
: undefined
|
||||
} catch (compactionError) {
|
||||
reqLogger.warn('Failed to compact SSE error logs, omitting oversized error details', {
|
||||
@@ -2076,6 +2196,10 @@ async function handleExecutePost(
|
||||
}
|
||||
|
||||
finalMetaStatus = 'error'
|
||||
const terminalError = executionResult?.error || errorMessage
|
||||
const terminalDisplay = await loggingSession.projectDisplayContent({
|
||||
error: terminalError,
|
||||
})
|
||||
await sendEvent(
|
||||
{
|
||||
type: 'execution:error',
|
||||
@@ -2083,7 +2207,10 @@ async function handleExecutePost(
|
||||
executionId,
|
||||
workflowId,
|
||||
data: {
|
||||
error: executionResult?.error || errorMessage,
|
||||
error: terminalError,
|
||||
display: {
|
||||
...(terminalDisplay.error !== undefined ? { error: terminalDisplay.error } : {}),
|
||||
},
|
||||
duration: executionResult?.metadata?.duration || 0,
|
||||
finalBlockLogs: compactErrorLogs,
|
||||
},
|
||||
|
||||
@@ -9,6 +9,12 @@ import {
|
||||
} from '@/lib/api/contracts/workflows'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import {
|
||||
collectFunctionalBlockOutputs,
|
||||
FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE,
|
||||
type FunctionalExecutionDataSource,
|
||||
FunctionalOutputsUnavailableError,
|
||||
} from '@/lib/logs/execution/functional-outputs'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { getAutomaticResumeWaitingMetadata } from '@/lib/workflows/executor/paused-execution-metadata'
|
||||
import { validateWorkflowAccess } from '@/app/api/workflows/middleware'
|
||||
@@ -18,32 +24,10 @@ const logger = createLogger('WorkflowExecutionStatusAPI')
|
||||
|
||||
type LogStatus = 'pending' | 'running' | 'completed' | 'failed' | 'cancelled'
|
||||
|
||||
interface TraceSpanShape {
|
||||
blockId?: string
|
||||
output?: Record<string, unknown>
|
||||
children?: TraceSpanShape[]
|
||||
}
|
||||
|
||||
interface ExecutionDataShape {
|
||||
interface ExecutionDataShape extends FunctionalExecutionDataSource {
|
||||
finalOutput?: { error?: string } & Record<string, unknown>
|
||||
error?: { message?: string } | string
|
||||
completionFailure?: string
|
||||
traceSpans?: TraceSpanShape[]
|
||||
}
|
||||
|
||||
function collectBlockOutputs(spans: TraceSpanShape[] | undefined): Map<string, unknown> {
|
||||
const map = new Map<string, unknown>()
|
||||
const visit = (list?: TraceSpanShape[]): void => {
|
||||
if (!list) return
|
||||
for (const span of list) {
|
||||
if (span.blockId && span.output !== undefined && !map.has(span.blockId)) {
|
||||
map.set(span.blockId, span.output)
|
||||
}
|
||||
if (span.children) visit(span.children)
|
||||
}
|
||||
}
|
||||
visit(spans)
|
||||
return map
|
||||
}
|
||||
|
||||
function resolvePath(value: unknown, path: string[]): unknown {
|
||||
@@ -204,10 +188,23 @@ export const GET = withRouteHandler(
|
||||
? (executionData.finalOutput ?? null)
|
||||
: null
|
||||
|
||||
const blockOutputs =
|
||||
selectedOutputs.length > 0
|
||||
? pickSelectedOutputs(selectedOutputs, collectBlockOutputs(executionData?.traceSpans))
|
||||
: null
|
||||
let blockOutputs: Record<string, unknown> | null = null
|
||||
if (selectedOutputs.length > 0) {
|
||||
try {
|
||||
blockOutputs = pickSelectedOutputs(
|
||||
selectedOutputs,
|
||||
collectFunctionalBlockOutputs(executionData)
|
||||
)
|
||||
} catch (error) {
|
||||
if (error instanceof FunctionalOutputsUnavailableError) {
|
||||
return NextResponse.json(
|
||||
{ error: FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE },
|
||||
{ status: 409 }
|
||||
)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
const response: WorkflowExecutionStatusResponse = {
|
||||
executionId: logRow.executionId,
|
||||
|
||||
@@ -13,6 +13,7 @@ const {
|
||||
mockResolveBillingAttribution,
|
||||
mockAssertBillingAttributionSnapshot,
|
||||
mockStart,
|
||||
mockSetResolvedSecretTraceRegistry,
|
||||
mockSafeComplete,
|
||||
mockSafeCompleteWithError,
|
||||
} = vi.hoisted(() => ({
|
||||
@@ -21,6 +22,7 @@ const {
|
||||
mockResolveBillingAttribution: vi.fn(),
|
||||
mockAssertBillingAttributionSnapshot: vi.fn((value) => value),
|
||||
mockStart: vi.fn().mockResolvedValue(undefined),
|
||||
mockSetResolvedSecretTraceRegistry: vi.fn(),
|
||||
mockSafeComplete: vi.fn().mockResolvedValue(undefined),
|
||||
mockSafeCompleteWithError: vi.fn().mockResolvedValue(undefined),
|
||||
}))
|
||||
@@ -42,6 +44,7 @@ vi.mock('@/lib/logs/execution/logging-session', () => ({
|
||||
LoggingSession: vi.fn(function LoggingSession() {
|
||||
return {
|
||||
start: mockStart,
|
||||
setResolvedSecretTraceRegistry: mockSetResolvedSecretTraceRegistry,
|
||||
markAsFailed: vi.fn().mockResolvedValue(undefined),
|
||||
safeCompleteWithError: mockSafeCompleteWithError,
|
||||
safeComplete: mockSafeComplete,
|
||||
@@ -307,4 +310,104 @@ describe('POST /api/workflows/[id]/log completion attribution', () => {
|
||||
)
|
||||
expect(mockSafeComplete).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('restores trusted Secrets provenance before projecting legacy completion traces', async () => {
|
||||
const trustedExecutionState = {
|
||||
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
resolvedSecretTraceProvenance: { version: 1, complete: true, entries: [] },
|
||||
}
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([
|
||||
{
|
||||
workflowId: OWNER_WORKFLOW_ID,
|
||||
workspaceId: 'workspace-1',
|
||||
executionData: {
|
||||
billingAttribution: storedBillingAttribution,
|
||||
executionState: trustedExecutionState,
|
||||
},
|
||||
},
|
||||
])
|
||||
|
||||
const res = await POST(
|
||||
makeRequest(OWNER_WORKFLOW_ID, {
|
||||
executionId: 'trusted-provenance-execution-id',
|
||||
result: validResult,
|
||||
}),
|
||||
{ params: Promise.resolve({ id: OWNER_WORKFLOW_ID }) }
|
||||
)
|
||||
|
||||
expect(res.status).toBe(200)
|
||||
const registry = mockSetResolvedSecretTraceRegistry.mock.calls[0]?.[0]
|
||||
expect(registry?.isComplete()).toBe(true)
|
||||
expect(registry?.exportProvenance()).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
expect(mockSafeComplete).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ executionState: trustedExecutionState })
|
||||
)
|
||||
})
|
||||
|
||||
it('forwards trusted execution state through legacy error completion', async () => {
|
||||
const trustedExecutionState = {
|
||||
blockStates: { 'function-1': { output: { error: 'raw-secret-value' } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
resolvedSecretTraceProvenance: { version: 1, complete: true, entries: [] },
|
||||
}
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([
|
||||
{
|
||||
workflowId: OWNER_WORKFLOW_ID,
|
||||
workspaceId: 'workspace-1',
|
||||
executionData: {
|
||||
billingAttribution: storedBillingAttribution,
|
||||
executionState: trustedExecutionState,
|
||||
},
|
||||
},
|
||||
])
|
||||
|
||||
const res = await POST(
|
||||
makeRequest(OWNER_WORKFLOW_ID, {
|
||||
executionId: 'trusted-error-execution-id',
|
||||
result: { success: false, error: 'failed' },
|
||||
}),
|
||||
{ params: Promise.resolve({ id: OWNER_WORKFLOW_ID }) }
|
||||
)
|
||||
|
||||
expect(res.status).toBe(200)
|
||||
expect(mockSafeCompleteWithError).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ executionState: trustedExecutionState })
|
||||
)
|
||||
})
|
||||
|
||||
it('forces structural-only traces when trusted stored provenance is unavailable', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([
|
||||
{
|
||||
workflowId: OWNER_WORKFLOW_ID,
|
||||
workspaceId: 'workspace-1',
|
||||
executionData: { billingAttribution: storedBillingAttribution },
|
||||
},
|
||||
])
|
||||
|
||||
const res = await POST(
|
||||
makeRequest(OWNER_WORKFLOW_ID, {
|
||||
executionId: 'legacy-no-provenance-execution-id',
|
||||
result: validResult,
|
||||
}),
|
||||
{ params: Promise.resolve({ id: OWNER_WORKFLOW_ID }) }
|
||||
)
|
||||
|
||||
expect(res.status).toBe(200)
|
||||
const registry = mockSetResolvedSecretTraceRegistry.mock.calls[0]?.[0]
|
||||
expect(registry?.isComplete()).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -13,9 +13,12 @@ import { generateRequestId } from '@/lib/core/utils/request'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { LoggingSession } from '@/lib/logs/execution/logging-session'
|
||||
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { validateWorkflowAccess } from '@/app/api/workflows/middleware'
|
||||
import { createErrorResponse, createSuccessResponse } from '@/app/api/workflows/utils'
|
||||
import type { SerializableExecutionState } from '@/executor/execution/types'
|
||||
import type { ExecutionResult } from '@/executor/types'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('WorkflowLogAPI')
|
||||
|
||||
@@ -119,6 +122,31 @@ export const POST = withRouteHandler(
|
||||
const isChatExecution = result.metadata?.source === 'chat'
|
||||
const triggerType = isChatExecution ? 'chat' : 'manual'
|
||||
const loggingSession = new LoggingSession(id, executionId, triggerType, requestId)
|
||||
const resolvedSecretTraceRegistry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: actorUserId,
|
||||
workspaceId: existingLog.workspaceId,
|
||||
})
|
||||
const trustedExecutionData = await materializeExecutionData(
|
||||
existingLog.executionData as Record<string, unknown>,
|
||||
{
|
||||
workspaceId: existingLog.workspaceId,
|
||||
workflowId: existingLog.workflowId,
|
||||
executionId,
|
||||
}
|
||||
)
|
||||
const trustedExecutionState =
|
||||
trustedExecutionData.executionState &&
|
||||
typeof trustedExecutionData.executionState === 'object' &&
|
||||
!Array.isArray(trustedExecutionData.executionState)
|
||||
? (trustedExecutionData.executionState as SerializableExecutionState)
|
||||
: undefined
|
||||
const trustedProvenance = trustedExecutionState?.resolvedSecretTraceProvenance
|
||||
if (trustedProvenance === undefined) {
|
||||
resolvedSecretTraceRegistry.markIncomplete()
|
||||
} else {
|
||||
await resolvedSecretTraceRegistry.importProvenance(trustedProvenance, { trusted: true })
|
||||
}
|
||||
loggingSession.setResolvedSecretTraceRegistry(resolvedSecretTraceRegistry)
|
||||
|
||||
await loggingSession.start({
|
||||
userId: actorUserId,
|
||||
@@ -143,6 +171,7 @@ export const POST = withRouteHandler(
|
||||
totalDurationMs: totalDuration || result.metadata?.duration || 0,
|
||||
error: { message },
|
||||
traceSpans,
|
||||
executionState: trustedExecutionState,
|
||||
})
|
||||
} else {
|
||||
await loggingSession.safeComplete({
|
||||
@@ -150,6 +179,7 @@ export const POST = withRouteHandler(
|
||||
totalDurationMs: totalDuration || result.metadata?.duration || 0,
|
||||
finalOutput: result.output || {},
|
||||
traceSpans,
|
||||
executionState: trustedExecutionState,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -72,8 +72,6 @@ import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/provide
|
||||
import { getBlock } from '@/blocks/registry'
|
||||
import { useFolderMap, useFolders } from '@/hooks/queries/folders'
|
||||
import {
|
||||
fetchLogDetail,
|
||||
logKeys,
|
||||
prefetchLogDetail,
|
||||
useCancelExecution,
|
||||
useDashboardStats,
|
||||
@@ -90,7 +88,6 @@ import { CORE_TRIGGER_TYPES } from '@/stores/logs/filters/types'
|
||||
import { Dashboard, ExecutionSnapshot, LogDetails, LogRowContextMenu } from './components'
|
||||
import {
|
||||
DELETED_WORKFLOW_LABEL,
|
||||
extractRetryInput,
|
||||
formatDate,
|
||||
getDisplayStatus,
|
||||
type LogStatus,
|
||||
@@ -573,17 +570,11 @@ export default function Logs() {
|
||||
const retryLog = useCallback(
|
||||
async (log: WorkflowLogRow | null) => {
|
||||
const workflowId = log?.workflow?.id || log?.workflowId
|
||||
const logId = log?.id
|
||||
if (!workflowId || !logId) return
|
||||
const executionId = log?.executionId
|
||||
if (!workflowId || !executionId) return
|
||||
|
||||
try {
|
||||
const detailLog = await queryClient.fetchQuery({
|
||||
queryKey: logKeys.detail(workspaceId, logId),
|
||||
queryFn: ({ signal }) => fetchLogDetail(logId, workspaceId, signal),
|
||||
staleTime: 30 * 1000,
|
||||
})
|
||||
const input = extractRetryInput(detailLog)
|
||||
await retryExecution.mutateAsync({ workflowId, input })
|
||||
await retryExecution.mutateAsync({ workflowId, executionId })
|
||||
toast.success('Retry started')
|
||||
} catch {
|
||||
toast.error('Failed to retry execution')
|
||||
|
||||
@@ -2,7 +2,6 @@ import React from 'react'
|
||||
import { Badge } from '@sim/emcn'
|
||||
import { formatDuration, formatRelativeTime } from '@sim/utils/formatting'
|
||||
import { format } from 'date-fns'
|
||||
import type { WorkflowLogDetail } from '@/lib/api/contracts/logs'
|
||||
import { getIntegrationMetadata } from '@/lib/logs/get-trigger-options'
|
||||
import { getBlock } from '@/blocks/registry'
|
||||
import { CORE_TRIGGER_TYPES } from '@/stores/logs/filters/types'
|
||||
@@ -213,37 +212,3 @@ export const formatDate = (dateString: string) => {
|
||||
relative: formatRelativeTime(dateString),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts the original workflow input from a log entry for retry.
|
||||
* Prefers the persisted `workflowInput` field (new logs), falls back to
|
||||
* reconstructing from `executionState.blockStates` (old logs).
|
||||
*/
|
||||
export function extractRetryInput(log: WorkflowLogDetail): unknown | undefined {
|
||||
const execData = log.executionData
|
||||
if (!execData) return undefined
|
||||
|
||||
if (execData.workflowInput !== undefined) {
|
||||
return execData.workflowInput
|
||||
}
|
||||
|
||||
const executionState = (execData as Record<string, unknown>).executionState as
|
||||
| {
|
||||
blockStates?: Record<
|
||||
string,
|
||||
{ output?: unknown; executed?: boolean; executionTime?: number }
|
||||
>
|
||||
}
|
||||
| undefined
|
||||
if (!executionState?.blockStates) return undefined
|
||||
|
||||
// Starter/trigger blocks are pre-populated with executed: false and
|
||||
// executionTime: 0, which distinguishes them from blocks that actually ran.
|
||||
for (const state of Object.values(executionState.blockStates)) {
|
||||
if (state.executed === false && state.executionTime === 0 && state.output != null) {
|
||||
return state.output
|
||||
}
|
||||
}
|
||||
|
||||
return undefined
|
||||
}
|
||||
|
||||
+185
-4
@@ -10,8 +10,11 @@ const {
|
||||
DirectUploadErrorMock,
|
||||
executionStoreState,
|
||||
mockExecute,
|
||||
mockExecuteFromBlock,
|
||||
mockFetch,
|
||||
mockResolveStartCandidates,
|
||||
mockRunUploadStrategy,
|
||||
mockSelectBestTrigger,
|
||||
terminalStoreState,
|
||||
workflowBlocks,
|
||||
workflowStoreState,
|
||||
@@ -50,7 +53,7 @@ const {
|
||||
const executionStoreState = {
|
||||
workflowExecutions: new Map([['workflow-1', idleExecution]]),
|
||||
getWorkflowExecution: vi.fn(() => idleExecution),
|
||||
getCurrentExecutionId: vi.fn(() => null),
|
||||
getCurrentExecutionId: vi.fn<() => string | null>(() => null),
|
||||
getLastExecutionSnapshot: vi.fn(() => null),
|
||||
setCurrentExecutionId: vi.fn(),
|
||||
setIsExecuting: vi.fn(),
|
||||
@@ -88,8 +91,11 @@ const {
|
||||
DirectUploadErrorMock,
|
||||
executionStoreState,
|
||||
mockExecute: vi.fn(),
|
||||
mockExecuteFromBlock: vi.fn(),
|
||||
mockFetch: vi.fn(),
|
||||
mockResolveStartCandidates: vi.fn(),
|
||||
mockRunUploadStrategy: vi.fn(),
|
||||
mockSelectBestTrigger: vi.fn(),
|
||||
terminalStoreState,
|
||||
workflowBlocks,
|
||||
workflowStoreState,
|
||||
@@ -133,12 +139,12 @@ vi.mock('@/lib/workflows/input-format', () => ({
|
||||
|
||||
vi.mock('@/lib/workflows/triggers/trigger-utils', () => ({
|
||||
extractTriggerMockPayload: () => ({}),
|
||||
selectBestTrigger: () => [],
|
||||
selectBestTrigger: mockSelectBestTrigger,
|
||||
triggerNeedsMockPayload: () => false,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workflows/triggers/triggers', () => ({
|
||||
resolveStartCandidates: () => [],
|
||||
resolveStartCandidates: mockResolveStartCandidates,
|
||||
StartBlockPath: {
|
||||
SPLIT_API: 'split-api',
|
||||
SPLIT_INPUT: 'split-input',
|
||||
@@ -206,7 +212,7 @@ vi.mock('@/hooks/use-execution-stream', () => {
|
||||
SSEStreamInterruptedError,
|
||||
useExecutionStream: () => ({
|
||||
execute: mockExecute,
|
||||
executeFromBlock: vi.fn(),
|
||||
executeFromBlock: mockExecuteFromBlock,
|
||||
reconnect: vi.fn(),
|
||||
cancel: vi.fn(),
|
||||
cancelExecute: vi.fn(),
|
||||
@@ -344,6 +350,9 @@ async function drainStream(value: unknown): Promise<void> {
|
||||
describe('useWorkflowExecution attachment uploads', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
executionStoreState.getCurrentExecutionId.mockReturnValue(null)
|
||||
mockResolveStartCandidates.mockReturnValue([])
|
||||
mockSelectBestTrigger.mockReturnValue([])
|
||||
vi.stubGlobal('fetch', mockFetch)
|
||||
mockRunUploadStrategy.mockRejectedValue(
|
||||
new DirectUploadErrorMock('Server signaled fallback to API upload', 'FALLBACK_REQUIRED')
|
||||
@@ -355,6 +364,8 @@ describe('useWorkflowExecution attachment uploads', () => {
|
||||
})
|
||||
)
|
||||
mockExecute.mockResolvedValue(undefined)
|
||||
mockExecuteFromBlock.mockResolvedValue(undefined)
|
||||
workflowStoreState.edges.length = 0
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
@@ -470,4 +481,174 @@ describe('useWorkflowExecution attachment uploads', () => {
|
||||
|
||||
unmount()
|
||||
})
|
||||
|
||||
it('uses only projected live thinking without changing normal settle behavior', async () => {
|
||||
mockExecute.mockImplementationOnce(async (options) => {
|
||||
options.onExecutionId?.('execution-1')
|
||||
await options.callbacks?.onStreamThinking?.({
|
||||
blockId: 'agent-1',
|
||||
text: 'sk-resolved-secret',
|
||||
display: { text: '{{OPENAI_API_KEY}}' },
|
||||
})
|
||||
await options.callbacks?.onStreamDone?.({ blockId: 'agent-1' })
|
||||
await options.callbacks?.onBlockCompleted?.({
|
||||
blockId: 'agent-1',
|
||||
blockName: 'Agent 1',
|
||||
blockType: 'agent',
|
||||
executionOrder: 1,
|
||||
output: { content: 'sk-resolved-secret' },
|
||||
display: { output: { content: '{{OPENAI_API_KEY}}' } },
|
||||
durationMs: 10,
|
||||
startedAt: '2026-07-31T00:00:00.000Z',
|
||||
endedAt: '2026-07-31T00:00:00.010Z',
|
||||
})
|
||||
})
|
||||
|
||||
const { result, unmount } = renderWorkflowExecutionHook()
|
||||
|
||||
await act(async () => {
|
||||
const runResult = await result().handleRunWorkflow({ input: 'chat input' })
|
||||
await drainStream(runResult)
|
||||
})
|
||||
|
||||
expect(terminalStoreState.updateConsole).toHaveBeenCalledWith(
|
||||
'agent-1',
|
||||
expect.objectContaining({ agentStreamThinking: '{{OPENAI_API_KEY}}' }),
|
||||
'execution-1'
|
||||
)
|
||||
expect(terminalStoreState.updateConsole).not.toHaveBeenCalledWith(
|
||||
'agent-1',
|
||||
expect.objectContaining({ clearAgentStreamThinking: true }),
|
||||
'execution-1'
|
||||
)
|
||||
expect(JSON.stringify(terminalStoreState.updateConsole.mock.calls)).not.toContain(
|
||||
'sk-resolved-secret'
|
||||
)
|
||||
|
||||
unmount()
|
||||
})
|
||||
|
||||
it('preserves legacy live thinking when no display projection field is sent', async () => {
|
||||
mockExecute.mockImplementationOnce(async (options) => {
|
||||
options.onExecutionId?.('execution-1')
|
||||
await options.callbacks?.onStreamThinking?.({
|
||||
blockId: 'agent-1',
|
||||
text: 'sk-resolved-secret',
|
||||
})
|
||||
await options.callbacks?.onStreamDone?.({ blockId: 'agent-1' })
|
||||
})
|
||||
|
||||
const { result, unmount } = renderWorkflowExecutionHook()
|
||||
|
||||
await act(async () => {
|
||||
const runResult = await result().handleRunWorkflow({ input: 'chat input' })
|
||||
await drainStream(runResult)
|
||||
})
|
||||
|
||||
expect(terminalStoreState.updateConsole).toHaveBeenCalledWith(
|
||||
'agent-1',
|
||||
expect.objectContaining({ agentStreamThinking: 'sk-resolved-secret' }),
|
||||
'execution-1'
|
||||
)
|
||||
|
||||
unmount()
|
||||
})
|
||||
|
||||
it('clears live thinking when the server sends an empty display projection', async () => {
|
||||
mockExecute.mockImplementationOnce(async (options) => {
|
||||
options.onExecutionId?.('execution-1')
|
||||
await options.callbacks?.onStreamThinking?.({
|
||||
blockId: 'agent-1',
|
||||
text: 'sk-resolved-secret',
|
||||
display: {},
|
||||
})
|
||||
})
|
||||
|
||||
const { result, unmount } = renderWorkflowExecutionHook()
|
||||
|
||||
await act(async () => {
|
||||
const runResult = await result().handleRunWorkflow({ input: 'chat input' })
|
||||
await drainStream(runResult)
|
||||
})
|
||||
|
||||
expect(terminalStoreState.updateConsole).toHaveBeenCalledWith(
|
||||
'agent-1',
|
||||
{ clearAgentStreamThinking: true },
|
||||
'execution-1'
|
||||
)
|
||||
expect(JSON.stringify(terminalStoreState.updateConsole.mock.calls)).not.toContain(
|
||||
'sk-resolved-secret'
|
||||
)
|
||||
|
||||
unmount()
|
||||
})
|
||||
|
||||
it('keeps the trusted execution ID when storing a run-until-block snapshot', async () => {
|
||||
const startCandidate = {
|
||||
blockId: 'start',
|
||||
block: workflowBlocks.start,
|
||||
path: 'legacy-starter',
|
||||
}
|
||||
mockResolveStartCandidates.mockReturnValue([startCandidate])
|
||||
mockSelectBestTrigger.mockReturnValue([startCandidate])
|
||||
mockExecute.mockImplementationOnce(async (options) => {
|
||||
const executionId = options.executionId as string
|
||||
executionStoreState.getCurrentExecutionId.mockReturnValue(executionId)
|
||||
options.onExecutionId?.(executionId)
|
||||
await options.callbacks?.onExecutionCompleted?.({
|
||||
success: true,
|
||||
output: {},
|
||||
duration: 10,
|
||||
startTime: '2026-07-31T00:00:00.000Z',
|
||||
endTime: '2026-07-31T00:00:00.010Z',
|
||||
finalBlockLogs: [],
|
||||
})
|
||||
})
|
||||
|
||||
const { result, unmount } = renderWorkflowExecutionHook()
|
||||
|
||||
await act(async () => {
|
||||
await result().handleRunUntilBlock('function-1', 'workflow-1')
|
||||
})
|
||||
|
||||
const executionId = mockExecute.mock.calls[0]?.[0]?.executionId
|
||||
expect(executionId).toEqual(expect.any(String))
|
||||
expect(executionStoreState.setLastExecutionSnapshot).toHaveBeenCalledWith(
|
||||
'workflow-1',
|
||||
expect.objectContaining({ sourceExecutionId: executionId })
|
||||
)
|
||||
|
||||
unmount()
|
||||
})
|
||||
|
||||
it('sends the snapshot as a fallback with a trusted run-from-block execution ID', async () => {
|
||||
const sourceSnapshot = {
|
||||
blockStates: { start: { output: { value: 'ready' } } },
|
||||
executedBlocks: ['start'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['start'],
|
||||
sourceExecutionId: 'source-execution-1',
|
||||
}
|
||||
executionStoreState.getLastExecutionSnapshot.mockReturnValueOnce(sourceSnapshot)
|
||||
workflowStoreState.edges.push({ source: 'start', target: 'function-1' } as never)
|
||||
|
||||
const { result, unmount } = renderWorkflowExecutionHook()
|
||||
|
||||
await act(async () => {
|
||||
await result().handleRunFromBlock('function-1', 'workflow-1')
|
||||
})
|
||||
|
||||
expect(mockExecuteFromBlock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
workflowId: 'workflow-1',
|
||||
startBlockId: 'function-1',
|
||||
sourceExecutionId: 'source-execution-1',
|
||||
sourceSnapshot,
|
||||
})
|
||||
)
|
||||
|
||||
unmount()
|
||||
})
|
||||
})
|
||||
|
||||
+60
-28
@@ -16,6 +16,7 @@ import {
|
||||
} from '@/components/agent-stream/tool-call-lifecycle'
|
||||
import { requestJson } from '@/lib/api/client/request'
|
||||
import { cancelWorkflowExecutionContract, workflowLogContract } from '@/lib/api/contracts/workflows'
|
||||
import type { SecretSafeBlockLog } from '@/lib/logs/execution/display-types'
|
||||
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
import { processStreamingBlockLogs } from '@/lib/tokenization'
|
||||
import type {
|
||||
@@ -103,6 +104,20 @@ interface DebugValidationResult {
|
||||
|
||||
const WORKFLOW_EXECUTION_FAILURE_MESSAGE = 'Workflow execution failed'
|
||||
|
||||
function getExecutionDisplayError(data: unknown): {
|
||||
displayError?: string
|
||||
hasDisplayProjection: boolean
|
||||
} {
|
||||
if (!data || typeof data !== 'object' || !Object.hasOwn(data, 'display')) {
|
||||
return { hasDisplayProjection: false }
|
||||
}
|
||||
const display = (data as { display?: { error?: unknown } }).display
|
||||
return {
|
||||
hasDisplayProjection: true,
|
||||
...(typeof display?.error === 'string' ? { displayError: display.error } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
async function persistExecutionPointerProgress(
|
||||
workflowId: string,
|
||||
executionId: string,
|
||||
@@ -261,6 +276,16 @@ function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamC
|
||||
)
|
||||
}
|
||||
|
||||
const clearThinking = (blockId: string) => {
|
||||
const timer = thinkingFlushTimers.get(blockId)
|
||||
if (timer !== undefined) {
|
||||
clearTimeout(timer)
|
||||
thinkingFlushTimers.delete(blockId)
|
||||
}
|
||||
thinkingByBlock.delete(blockId)
|
||||
updateConsole(blockId, { clearAgentStreamThinking: true }, executionIdRef.current)
|
||||
}
|
||||
|
||||
const settleBlock = (blockId: string, status: 'success' | 'error' | 'cancelled') => {
|
||||
flushThinking(blockId)
|
||||
const map = toolCallsByBlock.get(blockId)
|
||||
@@ -288,8 +313,21 @@ function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamC
|
||||
}
|
||||
|
||||
const onStreamThinking = (data: StreamThinkingData) => {
|
||||
const display = (
|
||||
data as StreamThinkingData & {
|
||||
display?: { text?: string; clearLiveDisplay?: true }
|
||||
}
|
||||
).display
|
||||
const hasDisplayProjection = Object.hasOwn(data, 'display')
|
||||
const text = hasDisplayProjection ? display?.text : data.text
|
||||
if (display?.clearLiveDisplay || (hasDisplayProjection && typeof text !== 'string')) {
|
||||
clearThinking(data.blockId)
|
||||
return
|
||||
}
|
||||
if (!text) return
|
||||
|
||||
const prev = thinkingByBlock.get(data.blockId) ?? ''
|
||||
thinkingByBlock.set(data.blockId, prev + data.text)
|
||||
thinkingByBlock.set(data.blockId, prev + text)
|
||||
if (!thinkingFlushTimers.has(data.blockId)) {
|
||||
thinkingFlushTimers.set(
|
||||
data.blockId,
|
||||
@@ -334,7 +372,14 @@ function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamC
|
||||
settleBlock(data.blockId, 'success')
|
||||
}
|
||||
|
||||
return { flushThinking, settleBlock, settleAll, onStreamThinking, onStreamTool, onStreamDone }
|
||||
return {
|
||||
flushThinking,
|
||||
settleBlock,
|
||||
settleAll,
|
||||
onStreamThinking,
|
||||
onStreamTool,
|
||||
onStreamDone,
|
||||
}
|
||||
}
|
||||
|
||||
export function useWorkflowExecution() {
|
||||
@@ -464,10 +509,12 @@ export function useWorkflowExecution() {
|
||||
workflowId?: string
|
||||
executionId?: string
|
||||
error?: string
|
||||
displayError?: string
|
||||
hasDisplayProjection?: boolean
|
||||
durationMs?: number
|
||||
blockLogs: BlockLog[]
|
||||
isPreExecutionError?: boolean
|
||||
finalBlockLogs?: BlockLog[]
|
||||
finalBlockLogs?: SecretSafeBlockLog[]
|
||||
}) => {
|
||||
if (!params.workflowId) return
|
||||
sharedHandleExecutionErrorConsole(
|
||||
@@ -483,7 +530,7 @@ export function useWorkflowExecution() {
|
||||
workflowId?: string
|
||||
executionId?: string
|
||||
durationMs?: number
|
||||
finalBlockLogs?: BlockLog[]
|
||||
finalBlockLogs?: SecretSafeBlockLog[]
|
||||
}) => {
|
||||
if (!params.workflowId) return
|
||||
sharedHandleExecutionCancelledConsole(
|
||||
@@ -862,27 +909,7 @@ export function useWorkflowExecution() {
|
||||
streamedContent.set(id, chunks.join(''))
|
||||
}
|
||||
|
||||
// Update streamed content and apply tokenization
|
||||
if (result.logs) {
|
||||
result.logs.forEach((log: BlockLog) => {
|
||||
if (streamedContent.has(log.blockId)) {
|
||||
// For console display, show the actual structured block output instead of formatted streaming content
|
||||
// This ensures console logs match the block state structure
|
||||
// Use replaceOutput to completely replace the output instead of merging
|
||||
// Use the executionId from this execution context
|
||||
useTerminalConsoleStore.getState().updateConsole(
|
||||
log.blockId,
|
||||
{
|
||||
executionOrder: log.executionOrder,
|
||||
replaceOutput: log.output,
|
||||
success: true,
|
||||
},
|
||||
executionId
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
// Process all logs for streaming tokenization
|
||||
const processedCount = processStreamingBlockLogs(result.logs, streamedContent)
|
||||
logger.info(`Processed ${processedCount} blocks for streaming tokenization`)
|
||||
}
|
||||
@@ -1208,7 +1235,6 @@ export function useWorkflowExecution() {
|
||||
const activeBlockRefCounts = new Map<string, number>()
|
||||
const streamedChunks = new Map<string, string[]>()
|
||||
const agentStreamChrome = createAgentStreamChrome({ executionIdRef, updateConsole })
|
||||
const settleAgentStreamChrome = agentStreamChrome.settleBlock
|
||||
const settleAllAgentStreamChrome = agentStreamChrome.settleAll
|
||||
const accumulatedBlockLogs: BlockLog[] = []
|
||||
const accumulatedBlockStates = new Map<string, BlockState>()
|
||||
@@ -1282,8 +1308,7 @@ export function useWorkflowExecution() {
|
||||
onBlockStarted: blockHandlers.onBlockStarted,
|
||||
onBlockCompleted: blockHandlers.onBlockCompleted,
|
||||
onBlockError: (data) => {
|
||||
// Failures often skip stream:done — settle thinking/tool chrome here.
|
||||
settleAgentStreamChrome(data.blockId, 'error')
|
||||
agentStreamChrome.settleBlock(data.blockId, 'error')
|
||||
blockHandlers.onBlockError(data)
|
||||
},
|
||||
onBlockChildWorkflowStarted: blockHandlers.onBlockChildWorkflowStarted,
|
||||
@@ -1388,6 +1413,7 @@ export function useWorkflowExecution() {
|
||||
decisions: existingSnapshot?.decisions || { router: {}, condition: {} },
|
||||
completedLoops: existingSnapshot?.completedLoops || [],
|
||||
activeExecutionPath: Array.from(mergedExecutedBlocks),
|
||||
sourceExecutionId: executionIdRef.current,
|
||||
}
|
||||
setLastExecutionSnapshot(activeWorkflowId, snapshot)
|
||||
logger.info('Merged execution snapshot after run-until-block', {
|
||||
@@ -1403,6 +1429,7 @@ export function useWorkflowExecution() {
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: Array.from(executedBlockIds),
|
||||
sourceExecutionId: executionIdRef.current,
|
||||
}
|
||||
setLastExecutionSnapshot(activeWorkflowId, snapshot)
|
||||
logger.info('Stored execution snapshot for run-from-block', {
|
||||
@@ -1508,6 +1535,7 @@ export function useWorkflowExecution() {
|
||||
workflowId: activeWorkflowId,
|
||||
executionId: executionIdRef.current,
|
||||
error: data.error,
|
||||
...getExecutionDisplayError(data),
|
||||
durationMs: data.duration,
|
||||
blockLogs: accumulatedBlockLogs,
|
||||
isPreExecutionError,
|
||||
@@ -1927,6 +1955,7 @@ export function useWorkflowExecution() {
|
||||
const effectiveSnapshot: SerializableExecutionState = isTriggerBlock
|
||||
? emptySnapshot
|
||||
: snapshot || emptySnapshot
|
||||
const sourceExecutionId = isTriggerBlock ? undefined : effectiveSnapshot.sourceExecutionId
|
||||
|
||||
// Extract mock payload for trigger blocks
|
||||
let workflowInput: any
|
||||
@@ -2009,6 +2038,7 @@ export function useWorkflowExecution() {
|
||||
workflowId,
|
||||
startBlockId: blockId,
|
||||
sourceSnapshot: effectiveSnapshot,
|
||||
...(sourceExecutionId ? { sourceExecutionId } : {}),
|
||||
input: workflowInput,
|
||||
onExecutionId: (id) => {
|
||||
if (runFromBlockOwnerRef.current !== runOwnerId) return
|
||||
@@ -2031,7 +2061,6 @@ export function useWorkflowExecution() {
|
||||
onBlockStarted: blockHandlers.onBlockStarted,
|
||||
onBlockCompleted: blockHandlers.onBlockCompleted,
|
||||
onBlockError: (data) => {
|
||||
// Failures often skip stream:done — settle thinking/tool chrome here.
|
||||
agentStreamChrome.settleBlock(data.blockId, 'error')
|
||||
blockHandlers.onBlockError(data)
|
||||
},
|
||||
@@ -2065,6 +2094,7 @@ export function useWorkflowExecution() {
|
||||
|
||||
const updatedSnapshot: SerializableExecutionState = {
|
||||
...effectiveSnapshot,
|
||||
sourceExecutionId: executionId,
|
||||
blockStates: mergedBlockStates,
|
||||
executedBlocks: Array.from(mergedExecutedBlocks),
|
||||
blockLogs: [...effectiveSnapshot.blockLogs, ...accumulatedBlockLogs],
|
||||
@@ -2116,6 +2146,7 @@ export function useWorkflowExecution() {
|
||||
workflowId,
|
||||
executionId,
|
||||
error: data.error,
|
||||
...getExecutionDisplayError(data),
|
||||
durationMs: data.duration,
|
||||
blockLogs: accumulatedBlockLogs,
|
||||
finalBlockLogs: data.finalBlockLogs,
|
||||
@@ -2473,6 +2504,7 @@ export function useWorkflowExecution() {
|
||||
workflowId: reconnectWorkflowId,
|
||||
executionId: capturedExecutionId,
|
||||
error: data.error,
|
||||
...getExecutionDisplayError(data),
|
||||
blockLogs: accumulatedBlockLogs,
|
||||
finalBlockLogs: data.finalBlockLogs,
|
||||
})
|
||||
|
||||
+43
@@ -30,6 +30,49 @@ describe('reconcileFinalBlockLogs (real store)', () => {
|
||||
} as any)
|
||||
})
|
||||
|
||||
it('replaces completed terminal content with the final server projection', () => {
|
||||
const store = useTerminalConsoleStore.getState()
|
||||
store.addConsole({
|
||||
workflowId: 'wf-1',
|
||||
blockId: 'function-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
executionId: 'exec-1',
|
||||
executionOrder: 1,
|
||||
isRunning: false,
|
||||
success: false,
|
||||
input: { code: 'return resolved-secret-value-123' },
|
||||
output: { error: 'resolved-secret-value-123' },
|
||||
error: 'SyntaxError: resolved-secret-value-123',
|
||||
agentStreamThinking: 'resolved-secret-value-123',
|
||||
})
|
||||
|
||||
const startedAt = new Date().toISOString()
|
||||
reconcileFinalBlockLogs(store.updateConsole, 'wf-1', 'exec-1', [
|
||||
{
|
||||
blockId: 'function-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
executionOrder: 1,
|
||||
success: false,
|
||||
input: { code: 'return {{SECRET_NAME}}' },
|
||||
output: { error: '{{SECRET_NAME}}' },
|
||||
error: 'SyntaxError: {{SECRET_NAME}}',
|
||||
clearLiveDisplay: true,
|
||||
startedAt,
|
||||
endedAt: startedAt,
|
||||
durationMs: 1,
|
||||
} as any,
|
||||
])
|
||||
|
||||
const entry = useTerminalConsoleStore.getState().getWorkflowEntries('wf-1')[0]
|
||||
expect(entry.input).toEqual({ code: 'return {{SECRET_NAME}}' })
|
||||
expect(entry.output).toEqual({ error: '{{SECRET_NAME}}' })
|
||||
expect(entry.error).toBe('SyntaxError: {{SECRET_NAME}}')
|
||||
expect(entry.agentStreamThinking).toBeUndefined()
|
||||
expect(JSON.stringify(entry)).not.toContain('resolved-secret-value-123')
|
||||
})
|
||||
|
||||
it('actually flips a child-workflow inner block from running to success', () => {
|
||||
const store = useTerminalConsoleStore.getState()
|
||||
store.addConsole({
|
||||
|
||||
+274
-1
@@ -5,6 +5,7 @@ import { resetTerminalConsoleMock, terminalConsoleMockFns } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
addExecutionErrorConsoleEntry,
|
||||
addHttpErrorConsoleEntry,
|
||||
createBlockEventHandlers,
|
||||
handleExecutionErrorConsole,
|
||||
reconcileFinalBlockLogs,
|
||||
@@ -195,6 +196,171 @@ describe('workflow-execution-utils', () => {
|
||||
'exec-1',
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps raw completion data functional while writing only the display projection', async () => {
|
||||
const accumulatedBlockLogs: BlockLog[] = []
|
||||
const accumulatedBlockStates = new Map()
|
||||
const updateConsole = vi.fn()
|
||||
const onBlockCompleteCallback = vi.fn().mockResolvedValue(undefined)
|
||||
const handlers = createBlockEventHandlers(
|
||||
{
|
||||
workflowId: 'wf-1',
|
||||
executionIdRef: { current: 'exec-1' },
|
||||
workflowEdges: [],
|
||||
activeBlocksSet: new Set<string>(),
|
||||
activeBlockRefCounts: new Map<string, number>(),
|
||||
accumulatedBlockLogs,
|
||||
accumulatedBlockStates,
|
||||
executedBlockIds: new Set<string>(),
|
||||
includeStartConsoleEntry: true,
|
||||
onBlockCompleteCallback,
|
||||
},
|
||||
{
|
||||
addConsole: vi.fn(),
|
||||
updateConsole,
|
||||
setActiveBlocks: vi.fn(),
|
||||
setBlockRunStatus: vi.fn(),
|
||||
setEdgeRunStatus: vi.fn(),
|
||||
}
|
||||
)
|
||||
|
||||
handlers.onBlockCompleted({
|
||||
blockId: 'fn-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
executionOrder: 1,
|
||||
input: { code: 'return sk-resolved-secret' },
|
||||
output: { result: 'sk-resolved-secret' },
|
||||
display: {
|
||||
input: { code: 'return {{OPENAI_API_KEY}}' },
|
||||
output: { result: '{{OPENAI_API_KEY}}' },
|
||||
},
|
||||
durationMs: 10,
|
||||
startedAt: '2026-07-31T00:00:00.000Z',
|
||||
endedAt: '2026-07-31T00:00:00.010Z',
|
||||
} as any)
|
||||
|
||||
expect(accumulatedBlockLogs[0]).toMatchObject({
|
||||
input: { code: 'return sk-resolved-secret' },
|
||||
output: { result: 'sk-resolved-secret' },
|
||||
})
|
||||
expect(accumulatedBlockStates.get('fn-1')?.output).toEqual({
|
||||
result: 'sk-resolved-secret',
|
||||
})
|
||||
expect(onBlockCompleteCallback).toHaveBeenCalledWith('fn-1', {
|
||||
result: 'sk-resolved-secret',
|
||||
})
|
||||
expect(updateConsole).toHaveBeenCalledWith(
|
||||
'fn-1',
|
||||
expect.objectContaining({
|
||||
input: { code: 'return {{OPENAI_API_KEY}}' },
|
||||
replaceOutput: { result: '{{OPENAI_API_KEY}}' },
|
||||
}),
|
||||
'exec-1'
|
||||
)
|
||||
expect(updateConsole.mock.calls[0][1]).not.toHaveProperty('clearAgentStreamThinking')
|
||||
expect(JSON.stringify(updateConsole.mock.calls)).not.toContain('sk-resolved-secret')
|
||||
})
|
||||
|
||||
it('does not fall back to a raw block error when the display projection is empty', () => {
|
||||
const accumulatedBlockLogs: BlockLog[] = []
|
||||
const accumulatedBlockStates = new Map()
|
||||
const updateConsole = vi.fn()
|
||||
const handlers = createBlockEventHandlers(
|
||||
{
|
||||
workflowId: 'wf-1',
|
||||
executionIdRef: { current: 'exec-1' },
|
||||
workflowEdges: [],
|
||||
activeBlocksSet: new Set<string>(),
|
||||
activeBlockRefCounts: new Map<string, number>(),
|
||||
accumulatedBlockLogs,
|
||||
accumulatedBlockStates,
|
||||
executedBlockIds: new Set<string>(),
|
||||
includeStartConsoleEntry: true,
|
||||
},
|
||||
{
|
||||
addConsole: vi.fn(),
|
||||
updateConsole,
|
||||
setActiveBlocks: vi.fn(),
|
||||
setBlockRunStatus: vi.fn(),
|
||||
setEdgeRunStatus: vi.fn(),
|
||||
}
|
||||
)
|
||||
|
||||
handlers.onBlockError({
|
||||
blockId: 'fn-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
executionOrder: 1,
|
||||
input: { code: 'return sk-resolved-secret' },
|
||||
error: 'SyntaxError: sk-resolved-secret',
|
||||
display: {},
|
||||
durationMs: 10,
|
||||
startedAt: '2026-07-31T00:00:00.000Z',
|
||||
endedAt: '2026-07-31T00:00:00.010Z',
|
||||
})
|
||||
|
||||
expect(accumulatedBlockLogs[0]?.error).toBe('SyntaxError: sk-resolved-secret')
|
||||
expect(accumulatedBlockStates.get('fn-1')?.output).toEqual({
|
||||
error: 'SyntaxError: sk-resolved-secret',
|
||||
})
|
||||
expect(updateConsole).toHaveBeenCalledWith(
|
||||
'fn-1',
|
||||
expect.objectContaining({
|
||||
input: {},
|
||||
replaceOutput: {},
|
||||
error: 'Block failed',
|
||||
clearAgentStreamThinking: true,
|
||||
}),
|
||||
'exec-1'
|
||||
)
|
||||
expect(JSON.stringify(updateConsole.mock.calls)).not.toContain('sk-resolved-secret')
|
||||
})
|
||||
|
||||
it('preserves legacy block error display when the server sends no projection', () => {
|
||||
const updateConsole = vi.fn()
|
||||
const handlers = createBlockEventHandlers(
|
||||
{
|
||||
workflowId: 'wf-1',
|
||||
executionIdRef: { current: 'exec-1' },
|
||||
workflowEdges: [],
|
||||
activeBlocksSet: new Set<string>(),
|
||||
activeBlockRefCounts: new Map<string, number>(),
|
||||
accumulatedBlockLogs: [],
|
||||
accumulatedBlockStates: new Map(),
|
||||
executedBlockIds: new Set<string>(),
|
||||
includeStartConsoleEntry: true,
|
||||
},
|
||||
{
|
||||
addConsole: vi.fn(),
|
||||
updateConsole,
|
||||
setActiveBlocks: vi.fn(),
|
||||
setBlockRunStatus: vi.fn(),
|
||||
setEdgeRunStatus: vi.fn(),
|
||||
}
|
||||
)
|
||||
|
||||
handlers.onBlockError({
|
||||
blockId: 'fn-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
executionOrder: 1,
|
||||
input: { code: 'return ordinary-value' },
|
||||
error: 'SyntaxError: ordinary-value',
|
||||
durationMs: 10,
|
||||
startedAt: '2026-07-31T00:00:00.000Z',
|
||||
endedAt: '2026-07-31T00:00:00.010Z',
|
||||
})
|
||||
|
||||
expect(updateConsole).toHaveBeenCalledWith(
|
||||
'fn-1',
|
||||
expect.objectContaining({
|
||||
input: { code: 'return ordinary-value' },
|
||||
error: 'SyntaxError: ordinary-value',
|
||||
}),
|
||||
'exec-1'
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe('addExecutionErrorConsoleEntry', () => {
|
||||
@@ -204,6 +370,7 @@ describe('workflow-execution-utils', () => {
|
||||
workflowId: 'wf-1',
|
||||
executionId: 'exec-1',
|
||||
error: 'Run failed',
|
||||
displayError: 'Safe run failure',
|
||||
durationMs: 1234,
|
||||
blockLogs: [],
|
||||
})
|
||||
@@ -212,7 +379,45 @@ describe('workflow-execution-utils', () => {
|
||||
const entry = addConsole.mock.calls[0][0]
|
||||
expect(entry.blockName).toBe('Run Error')
|
||||
expect(entry.blockType).toBe('error')
|
||||
expect(entry.error).toBe('Run failed')
|
||||
expect(entry.error).toBe('Safe run failure')
|
||||
})
|
||||
|
||||
it('does not use the raw execution error when the server projection is empty', () => {
|
||||
const addConsole = vi.fn()
|
||||
addExecutionErrorConsoleEntry(addConsole, {
|
||||
workflowId: 'wf-1',
|
||||
executionId: 'exec-1',
|
||||
error: 'SyntaxError: sk-resolved-secret',
|
||||
hasDisplayProjection: true,
|
||||
blockLogs: [],
|
||||
})
|
||||
|
||||
expect(addConsole.mock.calls[0][0].error).toBe('Run failed')
|
||||
expect(JSON.stringify(addConsole.mock.calls)).not.toContain('sk-resolved-secret')
|
||||
})
|
||||
|
||||
it('preserves legacy execution errors when the server sends no projection', () => {
|
||||
const addConsole = vi.fn()
|
||||
addExecutionErrorConsoleEntry(addConsole, {
|
||||
workflowId: 'wf-1',
|
||||
executionId: 'exec-1',
|
||||
error: 'Legacy run failure',
|
||||
blockLogs: [],
|
||||
})
|
||||
|
||||
expect(addConsole.mock.calls[0][0].error).toBe('Legacy run failure')
|
||||
})
|
||||
|
||||
it('preserves HTTP error detail before SSE projection is available', () => {
|
||||
const addConsole = vi.fn()
|
||||
addHttpErrorConsoleEntry(addConsole, {
|
||||
workflowId: 'wf-1',
|
||||
executionId: 'exec-1',
|
||||
error: 'Workflow is archived',
|
||||
httpStatus: 409,
|
||||
})
|
||||
|
||||
expect(addConsole.mock.calls[0][0].error).toBe('Workflow is archived')
|
||||
})
|
||||
|
||||
it('skips when blockLogs already contain a block-level error', () => {
|
||||
@@ -407,6 +612,74 @@ describe('workflow-execution-utils', () => {
|
||||
expect(updateConsole).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reprojects completed content without deep-comparing authoritative finalBlockLogs', () => {
|
||||
terminalConsoleMockFns.mockAddConsole({
|
||||
workflowId: 'wf-1',
|
||||
blockId: 'fn-1',
|
||||
blockName: 'Function',
|
||||
blockType: 'function',
|
||||
executionId: 'exec-1',
|
||||
executionOrder: 1,
|
||||
isRunning: false,
|
||||
success: false,
|
||||
input: { code: 'return sk-resolved-secret' },
|
||||
output: { error: 'sk-resolved-secret' },
|
||||
error: 'SyntaxError: sk-resolved-secret',
|
||||
agentStreamThinking: 'sk-resolved-secret',
|
||||
})
|
||||
|
||||
const updateConsole = vi.fn()
|
||||
reconcileFinalBlockLogs(updateConsole, 'wf-1', 'exec-1', [
|
||||
makeLog({
|
||||
blockId: 'fn-1',
|
||||
input: { code: 'return {{OPENAI_API_KEY}}' },
|
||||
output: { error: '{{OPENAI_API_KEY}}' },
|
||||
error: 'SyntaxError: {{OPENAI_API_KEY}}',
|
||||
success: false,
|
||||
}),
|
||||
])
|
||||
|
||||
expect(updateConsole).toHaveBeenCalledWith(
|
||||
'fn-1',
|
||||
expect.objectContaining({
|
||||
input: { code: 'return {{OPENAI_API_KEY}}' },
|
||||
replaceOutput: { error: '{{OPENAI_API_KEY}}' },
|
||||
error: 'SyntaxError: {{OPENAI_API_KEY}}',
|
||||
}),
|
||||
'exec-1'
|
||||
)
|
||||
expect(updateConsole.mock.calls[0][1]).not.toHaveProperty('clearAgentStreamThinking')
|
||||
expect(JSON.stringify(updateConsole.mock.calls)).not.toContain('sk-resolved-secret')
|
||||
})
|
||||
|
||||
it('clears live content when the final projection is structural-only', () => {
|
||||
terminalConsoleMockFns.mockAddConsole({
|
||||
workflowId: 'wf-1',
|
||||
blockId: 'fn-1',
|
||||
blockName: 'Function',
|
||||
blockType: 'function',
|
||||
executionId: 'exec-1',
|
||||
executionOrder: 1,
|
||||
isRunning: false,
|
||||
success: false,
|
||||
input: { code: 'return sk-resolved-secret' },
|
||||
output: { error: 'sk-resolved-secret' },
|
||||
error: 'SyntaxError: sk-resolved-secret',
|
||||
})
|
||||
|
||||
const updateConsole = vi.fn()
|
||||
reconcileFinalBlockLogs(updateConsole, 'wf-1', 'exec-1', [
|
||||
makeLog({ blockId: 'fn-1', success: false }),
|
||||
])
|
||||
|
||||
expect(updateConsole.mock.calls[0][1]).toMatchObject({
|
||||
input: {},
|
||||
replaceOutput: {},
|
||||
error: null,
|
||||
clearAgentStreamThinking: true,
|
||||
})
|
||||
})
|
||||
|
||||
it('reconciles child workflow spans before running entries are swept to canceled', () => {
|
||||
terminalConsoleMockFns.mockAddConsole({
|
||||
workflowId: 'wf-1',
|
||||
|
||||
+110
-22
@@ -1,6 +1,7 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { generateId } from '@sim/utils/id'
|
||||
import type { SecretSafeBlockLog } from '@/lib/logs/execution/display-types'
|
||||
import type { TraceSpan } from '@/lib/logs/types'
|
||||
import type {
|
||||
BlockChildWorkflowStartedData,
|
||||
@@ -15,9 +16,6 @@ import {
|
||||
SSEEventHandlerError,
|
||||
SSEStreamInterruptedError,
|
||||
} from '@/hooks/use-execution-stream'
|
||||
|
||||
const logger = createLogger('workflow-execution-utils')
|
||||
|
||||
import { useExecutionStore } from '@/stores/execution'
|
||||
import type { ConsoleEntry, ConsoleUpdate } from '@/stores/terminal'
|
||||
import {
|
||||
@@ -29,6 +27,11 @@ import {
|
||||
import { useWorkflowRegistry } from '@/stores/workflows/registry/store'
|
||||
import { useWorkflowStore } from '@/stores/workflows/workflow/store'
|
||||
|
||||
const logger = createLogger('workflow-execution-utils')
|
||||
const BLOCK_FAILURE_DISPLAY_MESSAGE = 'Block failed'
|
||||
const RUN_FAILURE_DISPLAY_MESSAGE = 'Run failed'
|
||||
const VALIDATION_FAILURE_DISPLAY_MESSAGE = 'Workflow validation failed'
|
||||
|
||||
/**
|
||||
* Updates the active blocks set and ref counts for a single block.
|
||||
* Ref counting ensures a block stays active until all parallel branches for it complete.
|
||||
@@ -132,6 +135,46 @@ interface BlockEventHandlerDeps {
|
||||
|
||||
type BlockChildWorkflowStartedUpdate = BlockChildWorkflowStartedData
|
||||
|
||||
interface BlockCompletedDisplayProjection {
|
||||
input?: unknown
|
||||
output?: unknown
|
||||
clearLiveDisplay?: true
|
||||
}
|
||||
|
||||
interface BlockErrorDisplayProjection {
|
||||
input?: unknown
|
||||
error?: string
|
||||
clearLiveDisplay?: true
|
||||
}
|
||||
|
||||
interface ExecutionErrorDisplayProjection {
|
||||
present: boolean
|
||||
error?: string
|
||||
}
|
||||
|
||||
function getBlockCompletedDisplay(data: BlockCompletedData): BlockCompletedDisplayProjection {
|
||||
const event = data as BlockCompletedData & { display?: BlockCompletedDisplayProjection }
|
||||
if (!Object.hasOwn(event, 'display')) {
|
||||
return { input: data.input, output: data.output }
|
||||
}
|
||||
return event.display ?? {}
|
||||
}
|
||||
|
||||
function getBlockErrorDisplay(data: BlockErrorData): BlockErrorDisplayProjection {
|
||||
const event = data as BlockErrorData & { display?: BlockErrorDisplayProjection }
|
||||
if (!Object.hasOwn(event, 'display')) {
|
||||
return { input: data.input, error: data.error }
|
||||
}
|
||||
return event.display ?? {}
|
||||
}
|
||||
|
||||
function getExecutionErrorDisplay(data: { error: string }): ExecutionErrorDisplayProjection {
|
||||
const event = data as typeof data & { display?: Omit<ExecutionErrorDisplayProjection, 'present'> }
|
||||
if (!Object.hasOwn(event, 'display')) return { present: false }
|
||||
const error = event.display?.error
|
||||
return { present: true, ...(typeof error === 'string' ? { error } : {}) }
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates block event handlers for SSE execution events.
|
||||
* Shared by the workflow execution hook and standalone execution utilities.
|
||||
@@ -299,19 +342,26 @@ export function createBlockEventHandlers(
|
||||
})
|
||||
|
||||
const updateConsoleEntry = (data: BlockCompletedData) => {
|
||||
const display = getBlockCompletedDisplay(data)
|
||||
const projectionOmittedContent =
|
||||
display.clearLiveDisplay === true ||
|
||||
(Object.hasOwn(data, 'display') &&
|
||||
((!Object.hasOwn(display, 'input') && data.input !== undefined) ||
|
||||
(!Object.hasOwn(display, 'output') && data.output !== undefined)))
|
||||
updateConsole(
|
||||
data.blockId,
|
||||
{
|
||||
blockName: data.blockName,
|
||||
blockType: data.blockType,
|
||||
executionOrder: data.executionOrder,
|
||||
input: data.input || {},
|
||||
replaceOutput: data.output,
|
||||
input: display.input ?? {},
|
||||
replaceOutput: (display.output ?? {}) as Record<string, unknown>,
|
||||
success: true,
|
||||
durationMs: data.durationMs,
|
||||
startedAt: data.startedAt,
|
||||
endedAt: data.endedAt,
|
||||
isRunning: false,
|
||||
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
|
||||
...extractIterationFields(data),
|
||||
},
|
||||
executionIdRef.current
|
||||
@@ -319,20 +369,27 @@ export function createBlockEventHandlers(
|
||||
}
|
||||
|
||||
const updateConsoleErrorEntry = (data: BlockErrorData) => {
|
||||
const display = getBlockErrorDisplay(data)
|
||||
const projectionOmittedContent =
|
||||
display.clearLiveDisplay === true ||
|
||||
(Object.hasOwn(data, 'display') &&
|
||||
((!Object.hasOwn(display, 'input') && data.input !== undefined) ||
|
||||
(!Object.hasOwn(display, 'error') && data.error !== undefined)))
|
||||
updateConsole(
|
||||
data.blockId,
|
||||
{
|
||||
blockName: data.blockName,
|
||||
blockType: data.blockType,
|
||||
executionOrder: data.executionOrder,
|
||||
input: data.input || {},
|
||||
input: display.input ?? {},
|
||||
replaceOutput: {},
|
||||
success: false,
|
||||
error: data.error,
|
||||
error: display.error || BLOCK_FAILURE_DISPLAY_MESSAGE,
|
||||
durationMs: data.durationMs,
|
||||
startedAt: data.startedAt,
|
||||
endedAt: data.endedAt,
|
||||
isRunning: false,
|
||||
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
|
||||
...extractIterationFields(data),
|
||||
},
|
||||
executionIdRef.current
|
||||
@@ -472,16 +529,16 @@ interface ExecutionConsoleDeps {
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconciles still-running console entries with the server's authoritative
|
||||
* `finalBlockLogs` so that any block whose terminal `block:completed`/`block:error`
|
||||
* SSE event was lost gets the correct success/error state instead of being
|
||||
* swept to "canceled".
|
||||
* Reconciles console entries with the server's authoritative, secret-safe
|
||||
* `finalBlockLogs`. Reapplying running or content-bearing rows both recovers
|
||||
* dropped terminal events and replaces an earlier live projection after late
|
||||
* secret activation.
|
||||
*/
|
||||
export function reconcileFinalBlockLogs(
|
||||
updateConsole: UpdateConsoleFn,
|
||||
workflowId: string,
|
||||
executionId: string | undefined,
|
||||
finalBlockLogs: BlockLog[] | undefined
|
||||
finalBlockLogs: SecretSafeBlockLog[] | undefined
|
||||
): void {
|
||||
if (!finalBlockLogs?.length || !executionId) return
|
||||
for (const log of finalBlockLogs) {
|
||||
@@ -491,8 +548,17 @@ export function reconcileFinalBlockLogs(
|
||||
entry.executionId === executionId &&
|
||||
entry.executionOrder === log.executionOrder
|
||||
const matchingEntry = entries.find(matchesFinalLog)
|
||||
const runningEntry = entries.find((entry) => matchesFinalLog(entry) && entry.isRunning)
|
||||
if (runningEntry) {
|
||||
const hasExistingContent =
|
||||
matchingEntry?.input !== undefined ||
|
||||
matchingEntry?.output !== undefined ||
|
||||
matchingEntry?.error !== undefined ||
|
||||
matchingEntry?.agentStreamThinking !== undefined
|
||||
if (matchingEntry && (matchingEntry.isRunning || hasExistingContent)) {
|
||||
const projectionOmittedContent =
|
||||
log.clearLiveDisplay === true ||
|
||||
(log.input === undefined && matchingEntry.input !== undefined) ||
|
||||
(log.output === undefined && matchingEntry.output !== undefined) ||
|
||||
(log.error === undefined && matchingEntry.error !== undefined)
|
||||
updateConsole(
|
||||
log.blockId,
|
||||
{
|
||||
@@ -500,14 +566,15 @@ export function reconcileFinalBlockLogs(
|
||||
blockName: log.blockName,
|
||||
blockType: log.blockType,
|
||||
replaceOutput: (log.output ?? {}) as Record<string, unknown>,
|
||||
...(log.input ? { input: log.input } : {}),
|
||||
input: log.input ?? {},
|
||||
success: log.success,
|
||||
...(log.error ? { error: log.error } : {}),
|
||||
error: log.error ?? null,
|
||||
durationMs: log.durationMs,
|
||||
startedAt: log.startedAt,
|
||||
endedAt: log.endedAt,
|
||||
isRunning: false,
|
||||
isCanceled: false,
|
||||
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
|
||||
},
|
||||
executionId
|
||||
)
|
||||
@@ -549,19 +616,26 @@ function reconcileChildTraceSpans(
|
||||
? findConsoleEntryForSpan(workflowId, executionId, childWorkflowInstanceId, span)
|
||||
: undefined
|
||||
if (span.blockId) {
|
||||
const errorMessage = normalizeSpanError(span.output?.error)
|
||||
const errorMessage = normalizeSpanError(span.errorMessage ?? span.output?.error)
|
||||
const projectionOmittedContent = matchingEntry
|
||||
? (span.input === undefined && matchingEntry.input !== undefined) ||
|
||||
(span.output === undefined && matchingEntry.output !== undefined) ||
|
||||
(errorMessage === undefined && matchingEntry.error !== undefined)
|
||||
: false
|
||||
updateConsole(
|
||||
span.blockId,
|
||||
{
|
||||
...spanConsoleIdentity(span, childWorkflowInstanceId),
|
||||
input: span.input ?? {},
|
||||
replaceOutput: (span.output ?? {}) as Record<string, unknown>,
|
||||
success: span.status !== 'error',
|
||||
...(errorMessage !== undefined ? { error: errorMessage } : {}),
|
||||
error: errorMessage ?? null,
|
||||
durationMs: span.duration,
|
||||
startedAt: span.startTime,
|
||||
endedAt: span.endTime,
|
||||
isRunning: false,
|
||||
isCanceled: false,
|
||||
...(projectionOmittedContent ? { clearAgentStreamThinking: true } : {}),
|
||||
},
|
||||
executionId
|
||||
)
|
||||
@@ -670,12 +744,17 @@ export function buildExecutionTiming(durationMs?: number): ExecutionTimingFields
|
||||
interface ExecutionErrorConsoleParams {
|
||||
workflowId: string
|
||||
executionId?: string
|
||||
/** Raw runtime error used only for functional classification and result propagation. */
|
||||
error?: string
|
||||
/** Server-projected error text safe for terminal display. */
|
||||
displayError?: string
|
||||
/** Distinguishes an intentionally empty projection from a legacy event with no projection. */
|
||||
hasDisplayProjection?: boolean
|
||||
durationMs?: number
|
||||
blockLogs: BlockLog[]
|
||||
isPreExecutionError?: boolean
|
||||
/** Server's authoritative per-block terminal states, used to reconcile lost SSE events. */
|
||||
finalBlockLogs?: BlockLog[]
|
||||
finalBlockLogs?: SecretSafeBlockLog[]
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -702,8 +781,14 @@ export function addExecutionErrorConsoleEntry(
|
||||
const isPreExecutionError = params.isPreExecutionError ?? false
|
||||
if (!isPreExecutionError && hasBlockError) return
|
||||
|
||||
const errorMessage = params.error || 'Run failed'
|
||||
const isTimeout = errorMessage.toLowerCase().includes('timed out')
|
||||
const hasDisplayProjection = params.hasDisplayProjection ?? params.displayError !== undefined
|
||||
const fallbackMessage = isPreExecutionError
|
||||
? VALIDATION_FAILURE_DISPLAY_MESSAGE
|
||||
: RUN_FAILURE_DISPLAY_MESSAGE
|
||||
const errorMessage = hasDisplayProjection
|
||||
? params.displayError || fallbackMessage
|
||||
: params.error || fallbackMessage
|
||||
const isTimeout = (params.error ?? params.displayError ?? '').toLowerCase().includes('timed out')
|
||||
const timing = buildExecutionTiming(params.durationMs)
|
||||
|
||||
addConsole({
|
||||
@@ -784,7 +869,7 @@ interface CancelledConsoleParams {
|
||||
executionId?: string
|
||||
durationMs?: number
|
||||
/** Server's authoritative per-block terminal states, used to reconcile lost SSE events. */
|
||||
finalBlockLogs?: BlockLog[]
|
||||
finalBlockLogs?: SecretSafeBlockLog[]
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -1060,6 +1145,7 @@ export async function executeWorkflowWithFullLogging(
|
||||
if (!isCurrentExecution()) return
|
||||
executionFinished = true
|
||||
const errorMessage = data.error || 'Run failed'
|
||||
const display = getExecutionErrorDisplay(data)
|
||||
executionResult = {
|
||||
success: false,
|
||||
output: {},
|
||||
@@ -1074,6 +1160,8 @@ export async function executeWorkflowWithFullLogging(
|
||||
workflowId: wfId,
|
||||
executionId: executionIdRef.current,
|
||||
error: errorMessage,
|
||||
displayError: display.error,
|
||||
hasDisplayProjection: display.present,
|
||||
durationMs: data.duration || 0,
|
||||
blockLogs: accumulatedBlockLogs,
|
||||
isPreExecutionError: accumulatedBlockLogs.length === 0,
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
executionPreprocessingMockFns,
|
||||
LoggingSessionMock,
|
||||
loggingSessionMock,
|
||||
loggingSessionMockFns,
|
||||
resetDbChainMock,
|
||||
workflowsPersistenceUtilsMock,
|
||||
workflowsPersistenceUtilsMockFns,
|
||||
@@ -16,13 +17,21 @@ import {
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { ADMISSION_ERROR_CODE } from '@/lib/core/admission/transient-failure'
|
||||
|
||||
const { mockTask, mockExecuteWorkflowCore, mockGetScheduleTimeValues, mockGetSubBlockValue } =
|
||||
vi.hoisted(() => ({
|
||||
mockTask: vi.fn((config) => config),
|
||||
mockExecuteWorkflowCore: vi.fn(),
|
||||
mockGetScheduleTimeValues: vi.fn(),
|
||||
mockGetSubBlockValue: vi.fn(),
|
||||
}))
|
||||
const {
|
||||
mockTask,
|
||||
mockExecuteWorkflowCore,
|
||||
mockWasExecutionFinalizedByCore,
|
||||
mockHasExecutionResult,
|
||||
mockGetScheduleTimeValues,
|
||||
mockGetSubBlockValue,
|
||||
} = vi.hoisted(() => ({
|
||||
mockTask: vi.fn((config) => config),
|
||||
mockExecuteWorkflowCore: vi.fn(),
|
||||
mockWasExecutionFinalizedByCore: vi.fn(),
|
||||
mockHasExecutionResult: vi.fn(),
|
||||
mockGetScheduleTimeValues: vi.fn(),
|
||||
mockGetSubBlockValue: vi.fn(),
|
||||
}))
|
||||
|
||||
const mockPreprocessExecution = executionPreprocessingMockFns.mockPreprocessExecution
|
||||
const mockLoadDeployedWorkflowState = workflowsPersistenceUtilsMockFns.mockLoadDeployedWorkflowState
|
||||
@@ -55,7 +64,7 @@ vi.mock('@/lib/logs/execution/trace-spans/trace-spans', () => ({
|
||||
|
||||
vi.mock('@/lib/workflows/executor/execution-core', () => ({
|
||||
executeWorkflowCore: mockExecuteWorkflowCore,
|
||||
wasExecutionFinalizedByCore: vi.fn().mockReturnValue(false),
|
||||
wasExecutionFinalizedByCore: mockWasExecutionFinalizedByCore,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workflows/executor/human-in-the-loop-manager', () => ({
|
||||
@@ -78,7 +87,7 @@ vi.mock('@/executor/execution/snapshot', () => ({
|
||||
}))
|
||||
|
||||
vi.mock('@/executor/utils/errors', () => ({
|
||||
hasExecutionResult: vi.fn().mockReturnValue(false),
|
||||
hasExecutionResult: mockHasExecutionResult,
|
||||
}))
|
||||
|
||||
import { executeScheduleJob } from './schedule-execution'
|
||||
@@ -100,6 +109,8 @@ const billingAttribution = {
|
||||
describe('async preprocessing correlation threading', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockWasExecutionFinalizedByCore.mockReturnValue(false)
|
||||
mockHasExecutionResult.mockReturnValue(false)
|
||||
resetDbChainMock()
|
||||
dbChainMockFns.limit.mockResolvedValue([
|
||||
{
|
||||
@@ -166,6 +177,84 @@ describe('async preprocessing correlation threading', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('preserves a core-finalized execution error for task failure semantics', async () => {
|
||||
const rawError = Object.assign(new Error('Function 1 failed with activated-secret-value'), {
|
||||
executionResult: {
|
||||
success: false,
|
||||
output: { error: 'Function failed' },
|
||||
logs: [],
|
||||
},
|
||||
})
|
||||
mockPreprocessExecution.mockResolvedValueOnce({
|
||||
success: true,
|
||||
actorUserId: 'actor-1',
|
||||
workflowRecord: {
|
||||
id: 'workflow-1',
|
||||
userId: 'owner-1',
|
||||
workspaceId: 'workspace-1',
|
||||
variables: {},
|
||||
},
|
||||
billingAttribution,
|
||||
executionTimeout: {},
|
||||
})
|
||||
mockExecuteWorkflowCore.mockRejectedValueOnce(rawError)
|
||||
mockHasExecutionResult.mockImplementation((error) => error === rawError)
|
||||
mockWasExecutionFinalizedByCore.mockReturnValue(true)
|
||||
|
||||
await expect(
|
||||
executeWorkflowJob({
|
||||
workflowId: 'workflow-1',
|
||||
userId: 'actor-1',
|
||||
workspaceId: 'workspace-1',
|
||||
billingAttribution,
|
||||
triggerType: 'api',
|
||||
executionId: 'execution-finalized',
|
||||
requestId: 'request-finalized',
|
||||
})
|
||||
).rejects.toBe(rawError)
|
||||
|
||||
expect(loggingSessionMockFns.mockWaitForPostExecution).not.toHaveBeenCalled()
|
||||
expect(mockWasExecutionFinalizedByCore).toHaveBeenCalledWith(rawError, 'execution-finalized')
|
||||
expect(loggingSessionMockFns.mockSafeCompleteWithError).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('persists and rethrows the original unfinalized execution error', async () => {
|
||||
const rawError = new Error('Function 1 failed with activated-secret-value')
|
||||
mockPreprocessExecution.mockResolvedValueOnce({
|
||||
success: true,
|
||||
actorUserId: 'actor-1',
|
||||
workflowRecord: {
|
||||
id: 'workflow-1',
|
||||
userId: 'owner-1',
|
||||
workspaceId: 'workspace-1',
|
||||
variables: {},
|
||||
},
|
||||
billingAttribution,
|
||||
executionTimeout: {},
|
||||
})
|
||||
mockExecuteWorkflowCore.mockRejectedValueOnce(rawError)
|
||||
|
||||
await expect(
|
||||
executeWorkflowJob({
|
||||
workflowId: 'workflow-1',
|
||||
userId: 'actor-1',
|
||||
workspaceId: 'workspace-1',
|
||||
billingAttribution,
|
||||
triggerType: 'api',
|
||||
executionId: 'execution-fault',
|
||||
requestId: 'request-fault',
|
||||
})
|
||||
).rejects.toBe(rawError)
|
||||
|
||||
expect(loggingSessionMockFns.mockSafeCompleteWithError).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
error: expect.objectContaining({
|
||||
message: 'Function 1 failed with activated-secret-value',
|
||||
}),
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('does not pre-start schedule logging before core execution', async () => {
|
||||
mockPreprocessExecution.mockResolvedValueOnce({
|
||||
success: true,
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const {
|
||||
mockTask,
|
||||
mockGetPausedExecutionById,
|
||||
mockStartResumeExecution,
|
||||
mockFindCellContextByExecutionId,
|
||||
mockSnapshotFromJson,
|
||||
} = vi.hoisted(() => ({
|
||||
mockTask: vi.fn((config) => config),
|
||||
mockGetPausedExecutionById: vi.fn(),
|
||||
mockStartResumeExecution: vi.fn(),
|
||||
mockFindCellContextByExecutionId: vi.fn(),
|
||||
mockSnapshotFromJson: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@trigger.dev/sdk', () => ({ task: mockTask }))
|
||||
|
||||
vi.mock('@/lib/billing/core/billing-attribution', () => ({
|
||||
assertBillingAttributionSnapshot: vi.fn((value) => value),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/table/cascade-lock', () => ({ withCascadeLock: vi.fn() }))
|
||||
vi.mock('@/lib/table/deps', () => ({ isExecCancelled: vi.fn(() => false) }))
|
||||
|
||||
vi.mock('@/lib/table/workflow-columns', () => ({
|
||||
findCellContextByExecutionId: mockFindCellContextByExecutionId,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workflows/executor/human-in-the-loop-manager', () => ({
|
||||
PauseResumeManager: {
|
||||
getPausedExecutionById: mockGetPausedExecutionById,
|
||||
startResumeExecution: mockStartResumeExecution,
|
||||
},
|
||||
}))
|
||||
|
||||
vi.mock('@/executor/execution/snapshot', () => ({
|
||||
ExecutionSnapshot: { fromJSON: mockSnapshotFromJson },
|
||||
}))
|
||||
|
||||
import { executeResumeJob, type ResumeExecutionPayload } from '@/background/resume-execution'
|
||||
|
||||
const payload: ResumeExecutionPayload = {
|
||||
resumeEntryId: 'resume-entry-1',
|
||||
resumeExecutionId: 'resume-execution-1',
|
||||
pausedExecutionId: 'paused-execution-1',
|
||||
contextId: 'context-1',
|
||||
resumeInput: {},
|
||||
userId: 'user-1',
|
||||
workflowId: 'workflow-1',
|
||||
parentExecutionId: 'parent-execution-1',
|
||||
}
|
||||
|
||||
describe('executeResumeJob terminal errors', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockGetPausedExecutionById.mockResolvedValue({
|
||||
executionSnapshot: { snapshot: {} },
|
||||
})
|
||||
mockSnapshotFromJson.mockReturnValue({
|
||||
metadata: {
|
||||
billingAttribution: {
|
||||
actorUserId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
},
|
||||
},
|
||||
})
|
||||
mockFindCellContextByExecutionId.mockResolvedValue(null)
|
||||
})
|
||||
|
||||
it('rethrows the original core-finalized resume error', async () => {
|
||||
const rawError = Object.assign(new Error('Agent tool exposed activated-secret-value'), {
|
||||
executionResult: {
|
||||
success: false,
|
||||
output: { error: 'Agent tool failed' },
|
||||
logs: [],
|
||||
},
|
||||
})
|
||||
mockStartResumeExecution.mockRejectedValue(rawError)
|
||||
|
||||
await expect(executeResumeJob(payload)).rejects.toBe(rawError)
|
||||
})
|
||||
|
||||
it('rethrows the original genuine resume fault', async () => {
|
||||
const rawError = new Error('MCP setup exposed activated-secret-value')
|
||||
mockStartResumeExecution.mockRejectedValue(rawError)
|
||||
|
||||
await expect(executeResumeJob(payload)).rejects.toBe(rawError)
|
||||
})
|
||||
})
|
||||
@@ -21,7 +21,13 @@ vi.mock('@/lib/workflows/schedules/disable-notifications', () => ({
|
||||
// does not re-export. Widen it rather than rewriting the source's imports.
|
||||
vi.mock('@sim/db', () => ({ ...databaseMock, ...schemaMock }))
|
||||
|
||||
import { applyScheduleFailureUpdate, releaseScheduleLock } from '@/background/schedule-execution'
|
||||
import {
|
||||
applyScheduleFailureUpdate,
|
||||
readScheduledMothershipErrorResponse,
|
||||
readScheduledMothershipJsonResponse,
|
||||
releaseScheduleLock,
|
||||
} from '@/background/schedule-execution'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const BASE = {
|
||||
scheduleId: 'schedule-1',
|
||||
@@ -99,3 +105,71 @@ describe('releaseScheduleLock', () => {
|
||||
expect(notifyMock).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('scheduled Mothership response handling', () => {
|
||||
it('parses JSON responses', async () => {
|
||||
const response = new Response(JSON.stringify({ content: 'ok' }), {
|
||||
headers: { 'content-type': 'application/json' },
|
||||
})
|
||||
|
||||
await expect(readScheduledMothershipJsonResponse(response)).resolves.toEqual({ content: 'ok' })
|
||||
})
|
||||
|
||||
it('preserves staging acceptance for responses above the generic tool cap', async () => {
|
||||
const response = new Response(JSON.stringify({ content: 'unchanged' }), {
|
||||
headers: {
|
||||
'content-length': String(10 * 1024 * 1024 + 1),
|
||||
'content-type': 'application/json',
|
||||
},
|
||||
})
|
||||
|
||||
await expect(readScheduledMothershipJsonResponse(response)).resolves.toEqual({
|
||||
content: 'unchanged',
|
||||
})
|
||||
})
|
||||
|
||||
it('does not include malformed response content in parse failures', async () => {
|
||||
const response = new Response('secret-bearing-non-json')
|
||||
const error = await readScheduledMothershipJsonResponse(response).catch((caught) => caught)
|
||||
|
||||
expect(error).toBeInstanceOf(Error)
|
||||
expect(error.message).toBe('Sim execution returned an invalid response')
|
||||
expect(error.message).not.toContain('secret-bearing-non-json')
|
||||
})
|
||||
|
||||
it('preserves the functional error body when no private metadata is present', async () => {
|
||||
const registry = {
|
||||
markIncomplete: vi.fn(),
|
||||
importProvenance: vi.fn(),
|
||||
} as unknown as ResolvedSecretTraceRegistry
|
||||
const response = new Response('secret-bearing-error-body', { status: 500 })
|
||||
|
||||
const message = await readScheduledMothershipErrorResponse(response, registry)
|
||||
|
||||
expect(message).toBe('secret-bearing-error-body')
|
||||
expect(registry.markIncomplete).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('strips private provenance metadata without replacing the provider error', async () => {
|
||||
const registry = {
|
||||
markIncomplete: vi.fn(),
|
||||
importProvenance: vi.fn().mockReturnValue(true),
|
||||
} as unknown as ResolvedSecretTraceRegistry
|
||||
const response = new Response(
|
||||
JSON.stringify({
|
||||
error: 'provider error detail',
|
||||
__resolvedSecretTraceProvenance: { version: 1, complete: true, entries: [] },
|
||||
}),
|
||||
{
|
||||
status: 500,
|
||||
headers: { 'x-sim-private-tool-metadata': 'resolved-secret-provenance-v1' },
|
||||
}
|
||||
)
|
||||
|
||||
const message = await readScheduledMothershipErrorResponse(response, registry)
|
||||
|
||||
expect(JSON.parse(message)).toEqual({ error: 'provider error detail' })
|
||||
expect(message).not.toContain('__resolvedSecretTraceProvenance')
|
||||
expect(registry.importProvenance).toHaveBeenCalledOnce()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
import { createLogger, runWithRequestContext } from '@sim/logger'
|
||||
import { describeError, toError } from '@sim/utils/errors'
|
||||
import { generateId } from '@sim/utils/id'
|
||||
import { isPlainRecord } from '@sim/utils/object'
|
||||
import { task } from '@trigger.dev/sdk'
|
||||
import { Cron } from 'croner'
|
||||
import { and, eq, isNull, ne, type SQL, sql } from 'drizzle-orm'
|
||||
@@ -32,8 +33,16 @@ import {
|
||||
} from '@/lib/core/execution-limits'
|
||||
import type { DbOrTx } from '@/lib/db/types'
|
||||
import { preprocessExecution } from '@/lib/execution/preprocessing'
|
||||
import {
|
||||
PRIVATE_TOOL_METADATA_REQUEST_HEADER,
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
responseHasPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
import { LoggingSession } from '@/lib/logs/execution/logging-session'
|
||||
import { projectTraceSpansForSecrets } from '@/lib/logs/execution/trace-secret-projection'
|
||||
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
import type { TraceSpan } from '@/lib/logs/types'
|
||||
import { cleanupExecutionBase64Cache } from '@/lib/uploads/utils/user-file-base64.server'
|
||||
import {
|
||||
executeWorkflowCore,
|
||||
@@ -61,6 +70,7 @@ import { ExecutionSnapshot } from '@/executor/execution/snapshot'
|
||||
import type { ExecutionMetadata } from '@/executor/execution/types'
|
||||
import { hasExecutionResult } from '@/executor/utils/errors'
|
||||
import { buildAPIUrl, buildAuthHeaders } from '@/executor/utils/http'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { MAX_CONSECUTIVE_FAILURES } from '@/triggers/constants'
|
||||
|
||||
const logger = createLogger('ScheduleExecution')
|
||||
@@ -660,6 +670,7 @@ async function runWorkflowExecution({
|
||||
stackTrace: error instanceof Error ? error.stack : undefined,
|
||||
},
|
||||
traceSpans,
|
||||
executionState: executionResult?.executionState,
|
||||
})
|
||||
|
||||
throw error
|
||||
@@ -1207,14 +1218,94 @@ function buildJobPrompt(jobRecord: {
|
||||
return parts.join('\n')
|
||||
}
|
||||
|
||||
async function consumeJobTraceProvenance(
|
||||
response: Response,
|
||||
payload: Record<string, unknown>,
|
||||
registry: ResolvedSecretTraceRegistry
|
||||
): Promise<boolean> {
|
||||
const hasProvenance = Object.hasOwn(payload, RESOLVED_SECRET_PROVENANCE_FIELD)
|
||||
const provenance = payload[RESOLVED_SECRET_PROVENANCE_FIELD]
|
||||
delete payload[RESOLVED_SECRET_PROVENANCE_FIELD]
|
||||
|
||||
if (
|
||||
!responseHasPrivateToolMetadata(response.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1) ||
|
||||
!hasProvenance
|
||||
) {
|
||||
registry.markIncomplete()
|
||||
return false
|
||||
}
|
||||
|
||||
return registry.importProvenance(provenance, { trusted: true })
|
||||
}
|
||||
|
||||
/** Reads and validates a scheduled Mothership JSON response without changing its size contract. */
|
||||
export async function readScheduledMothershipJsonResponse(
|
||||
response: Response
|
||||
): Promise<Record<string, unknown>> {
|
||||
let payload: unknown
|
||||
|
||||
try {
|
||||
payload = await response.json()
|
||||
} catch {
|
||||
throw new Error('Sim execution returned an invalid response')
|
||||
}
|
||||
|
||||
if (!isPlainRecord(payload)) {
|
||||
throw new Error('Sim execution returned an invalid response')
|
||||
}
|
||||
return payload
|
||||
}
|
||||
|
||||
/** Reads the functional error body and strips private provenance metadata when present. */
|
||||
export async function readScheduledMothershipErrorResponse(
|
||||
response: Response,
|
||||
registry: ResolvedSecretTraceRegistry
|
||||
): Promise<string> {
|
||||
const responseText = await response.text()
|
||||
const hasPrivateMarker = responseHasPrivateToolMetadata(
|
||||
response.headers,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1
|
||||
)
|
||||
const mayContainPrivateProvenance = responseText.includes(`"${RESOLVED_SECRET_PROVENANCE_FIELD}"`)
|
||||
|
||||
if (!hasPrivateMarker && !mayContainPrivateProvenance) {
|
||||
registry.markIncomplete()
|
||||
return responseText
|
||||
}
|
||||
|
||||
let payload: unknown
|
||||
|
||||
try {
|
||||
payload = JSON.parse(responseText)
|
||||
} catch {
|
||||
registry.markIncomplete()
|
||||
return responseText
|
||||
}
|
||||
|
||||
if (!isPlainRecord(payload)) {
|
||||
registry.markIncomplete()
|
||||
return responseText
|
||||
}
|
||||
|
||||
const hadPrivateProvenance = Object.hasOwn(payload, RESOLVED_SECRET_PROVENANCE_FIELD)
|
||||
try {
|
||||
await consumeJobTraceProvenance(response, payload, registry)
|
||||
} catch {
|
||||
registry.markIncomplete()
|
||||
}
|
||||
return hadPrivateProvenance ? JSON.stringify(payload) : responseText
|
||||
}
|
||||
|
||||
async function createJobLogEntry(params: {
|
||||
scheduleId: string
|
||||
workspaceId: string
|
||||
userId: string
|
||||
jobTitle: string | null
|
||||
startTime: Date
|
||||
endTime: Date
|
||||
durationMs: number
|
||||
success: boolean
|
||||
resolvedSecretTraceRegistry: ResolvedSecretTraceRegistry
|
||||
responseBody?: Record<string, any>
|
||||
errorMessage?: string
|
||||
}): Promise<void> {
|
||||
@@ -1222,14 +1313,17 @@ async function createJobLogEntry(params: {
|
||||
const {
|
||||
scheduleId,
|
||||
workspaceId,
|
||||
userId,
|
||||
jobTitle,
|
||||
startTime,
|
||||
endTime,
|
||||
durationMs,
|
||||
success,
|
||||
resolvedSecretTraceRegistry,
|
||||
responseBody,
|
||||
} = params
|
||||
const name = jobTitle || 'Sim Job'
|
||||
const executionId = generateId()
|
||||
|
||||
const toolCallsList = (responseBody?.toolCalls || []).map((tc: Record<string, unknown>) => ({
|
||||
name: tc.name,
|
||||
@@ -1246,7 +1340,7 @@ async function createJobLogEntry(params: {
|
||||
status: tc.error ? 'error' : 'success',
|
||||
}))
|
||||
|
||||
const traceSpan = {
|
||||
const traceSpan: TraceSpan = {
|
||||
id: generateId(),
|
||||
name,
|
||||
type: 'mothership',
|
||||
@@ -1263,12 +1357,20 @@ async function createJobLogEntry(params: {
|
||||
cost: responseBody?.cost || undefined,
|
||||
tokens: responseBody?.tokens || undefined,
|
||||
}
|
||||
const [projectedTraceSpan] = await projectTraceSpansForSecrets([traceSpan], {
|
||||
registry: resolvedSecretTraceRegistry,
|
||||
store: {
|
||||
workspaceId,
|
||||
executionId,
|
||||
userId,
|
||||
},
|
||||
})
|
||||
|
||||
await db.insert(jobExecutionLogs).values({
|
||||
id: generateId(),
|
||||
scheduleId,
|
||||
workspaceId,
|
||||
executionId: generateId(),
|
||||
executionId,
|
||||
level: success ? 'info' : 'error',
|
||||
status: success ? 'completed' : 'failed',
|
||||
trigger: 'mothership',
|
||||
@@ -1277,8 +1379,11 @@ async function createJobLogEntry(params: {
|
||||
totalDurationMs: durationMs,
|
||||
executionData: {
|
||||
enhanced: true,
|
||||
traceSpans: [traceSpan],
|
||||
traceSpans: [projectedTraceSpan],
|
||||
finalOutput: responseBody?.content ? { content: responseBody.content } : undefined,
|
||||
executionState: {
|
||||
resolvedSecretTraceProvenance: resolvedSecretTraceRegistry.exportProvenance(),
|
||||
},
|
||||
trigger: {
|
||||
type: 'mothership',
|
||||
source: name,
|
||||
@@ -1369,6 +1474,10 @@ export async function executeJobInline(payload: JobExecutionPayload) {
|
||||
}
|
||||
|
||||
const promptText = buildJobPrompt(jobRecord)
|
||||
const resolvedSecretTraceRegistry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: jobRecord.sourceUserId,
|
||||
workspaceId: jobRecord.sourceWorkspaceId,
|
||||
})
|
||||
|
||||
try {
|
||||
const billingAttribution = await resolveBillingAttribution({
|
||||
@@ -1382,6 +1491,7 @@ export async function executeJobInline(payload: JobExecutionPayload) {
|
||||
const url = buildAPIUrl('/api/mothership/execute')
|
||||
const headers = await buildAuthHeaders(jobRecord.sourceUserId)
|
||||
headers[BILLING_ATTRIBUTION_HEADER] = serializeBillingAttributionHeader(billingAttribution)
|
||||
headers[PRIVATE_TOOL_METADATA_REQUEST_HEADER] = RESOLVED_SECRET_PROVENANCE_METADATA_V1
|
||||
|
||||
const body = {
|
||||
messages: [{ role: 'user', content: promptText }],
|
||||
@@ -1404,7 +1514,11 @@ export async function executeJobInline(payload: JobExecutionPayload) {
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
const errorText = await response.text().catch(() => {
|
||||
const errorText = await readScheduledMothershipErrorResponse(
|
||||
response,
|
||||
resolvedSecretTraceRegistry
|
||||
).catch(() => {
|
||||
resolvedSecretTraceRegistry.markIncomplete()
|
||||
if (timeoutController.isTimedOut()) {
|
||||
throw new Error(getTimeoutErrorMessage(null, timeoutController.timeoutMs))
|
||||
}
|
||||
@@ -1416,11 +1530,13 @@ export async function executeJobInline(payload: JobExecutionPayload) {
|
||||
await createJobLogEntry({
|
||||
scheduleId: payload.scheduleId,
|
||||
workspaceId: jobRecord.sourceWorkspaceId,
|
||||
userId: jobRecord.sourceUserId,
|
||||
jobTitle: jobRecord.jobTitle,
|
||||
startTime,
|
||||
endTime,
|
||||
durationMs,
|
||||
success: false,
|
||||
resolvedSecretTraceRegistry,
|
||||
errorMessage: errorText,
|
||||
})
|
||||
|
||||
@@ -1430,10 +1546,17 @@ export async function executeJobInline(payload: JobExecutionPayload) {
|
||||
let responseBody: Record<string, any> = {}
|
||||
let wasCompletedByTool = false
|
||||
try {
|
||||
responseBody = await response.json()
|
||||
const payload = await readScheduledMothershipJsonResponse(response)
|
||||
responseBody = payload
|
||||
try {
|
||||
await consumeJobTraceProvenance(response, payload, resolvedSecretTraceRegistry)
|
||||
} catch {
|
||||
resolvedSecretTraceRegistry.markIncomplete()
|
||||
}
|
||||
const toolCalls = responseBody?.toolCalls as Array<{ name?: string }> | undefined
|
||||
wasCompletedByTool = toolCalls?.some((tc) => tc.name === 'complete_scheduled_task') ?? false
|
||||
} catch {
|
||||
resolvedSecretTraceRegistry.markIncomplete()
|
||||
if (timeoutController.isTimedOut()) {
|
||||
throw new Error(getTimeoutErrorMessage(null, timeoutController.timeoutMs))
|
||||
}
|
||||
@@ -1444,11 +1567,13 @@ export async function executeJobInline(payload: JobExecutionPayload) {
|
||||
await createJobLogEntry({
|
||||
scheduleId: payload.scheduleId,
|
||||
workspaceId: jobRecord.sourceWorkspaceId,
|
||||
userId: jobRecord.sourceUserId,
|
||||
jobTitle: jobRecord.jobTitle,
|
||||
startTime,
|
||||
endTime,
|
||||
durationMs,
|
||||
success: true,
|
||||
resolvedSecretTraceRegistry,
|
||||
responseBody,
|
||||
})
|
||||
|
||||
|
||||
@@ -4,30 +4,33 @@
|
||||
|
||||
import {
|
||||
dbChainMockFns,
|
||||
environmentUtilsMockFns,
|
||||
executionPreprocessingMock,
|
||||
executionPreprocessingMockFns,
|
||||
LoggingSessionMock,
|
||||
loggingSessionMock,
|
||||
loggingSessionMockFns,
|
||||
resetEnvironmentUtilsMock,
|
||||
} from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const {
|
||||
mockResolveWebhookRecordProviderConfig,
|
||||
mockExecuteWorkflowCore,
|
||||
mockWasExecutionFinalizedByCore,
|
||||
mockRecordException,
|
||||
mockGetActiveSpan,
|
||||
mockExecuteWithIdempotency,
|
||||
mockReleaseExecutionSlot,
|
||||
mockLoadDeploymentVersionState,
|
||||
mockGetProviderHandler,
|
||||
mockSetResolvedSecretTraceRegistry,
|
||||
} = vi.hoisted(() => ({
|
||||
mockResolveWebhookRecordProviderConfig: vi.fn(),
|
||||
mockExecuteWorkflowCore: vi.fn(),
|
||||
mockWasExecutionFinalizedByCore: vi.fn(),
|
||||
mockRecordException: vi.fn(),
|
||||
mockGetActiveSpan: vi.fn(),
|
||||
mockExecuteWithIdempotency: vi.fn(),
|
||||
mockReleaseExecutionSlot: vi.fn(),
|
||||
mockGetProviderHandler: vi.fn(() => ({})),
|
||||
mockSetResolvedSecretTraceRegistry: vi.fn(),
|
||||
mockLoadDeploymentVersionState: vi.fn(
|
||||
async (_workflowId: string, deploymentVersionId: string) => ({
|
||||
blocks: {},
|
||||
@@ -39,9 +42,10 @@ const {
|
||||
),
|
||||
}))
|
||||
|
||||
vi.mock('@opentelemetry/api', () => ({
|
||||
trace: { getActiveSpan: mockGetActiveSpan },
|
||||
}))
|
||||
const mockGetEffectiveEnvironmentSnapshot =
|
||||
environmentUtilsMockFns.mockGetEffectiveEnvironmentSnapshot
|
||||
|
||||
afterAll(resetEnvironmentUtilsMock)
|
||||
|
||||
vi.mock('@/lib/execution/preprocessing', () => executionPreprocessingMock)
|
||||
vi.mock('@/lib/logs/execution/logging-session', () => loggingSessionMock)
|
||||
@@ -77,9 +81,7 @@ vi.mock('@/lib/workflows/persistence/utils', () => ({
|
||||
loadWorkflowDeploymentVersionState: mockLoadDeploymentVersionState,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/webhooks/providers', () => ({
|
||||
getProviderHandler: vi.fn(() => ({})),
|
||||
}))
|
||||
vi.mock('@/lib/webhooks/providers', () => ({ getProviderHandler: mockGetProviderHandler }))
|
||||
|
||||
vi.mock('@/lib/logs/execution/trace-spans/trace-spans', () => ({
|
||||
buildTraceSpans: vi.fn(() => ({ traceSpans: [] })),
|
||||
@@ -208,6 +210,17 @@ describe('executeWebhookJob fault vs error handling', () => {
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
LoggingSessionMock.mockImplementation(function LoggingSession() {
|
||||
return {
|
||||
safeStart: loggingSessionMockFns.mockSafeStart,
|
||||
safeComplete: loggingSessionMockFns.mockSafeComplete,
|
||||
safeCompleteWithError: loggingSessionMockFns.mockSafeCompleteWithError,
|
||||
waitForPostExecution: loggingSessionMockFns.mockWaitForPostExecution,
|
||||
markAsFailed: loggingSessionMockFns.mockMarkAsFailed,
|
||||
setResolvedSecretTraceRegistry: mockSetResolvedSecretTraceRegistry,
|
||||
}
|
||||
})
|
||||
mockGetProviderHandler.mockReturnValue({})
|
||||
mockExecuteWithIdempotency.mockImplementation(
|
||||
(_provider: string, _key: string, operation: () => Promise<unknown>) => operation()
|
||||
)
|
||||
@@ -225,8 +238,15 @@ describe('executeWebhookJob fault vs error handling', () => {
|
||||
executionTimeout: { async: 120_000 },
|
||||
})
|
||||
mockResolveWebhookRecordProviderConfig.mockImplementation(async (record) => record)
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
|
||||
personalEncrypted: {},
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: {},
|
||||
workspaceDecrypted: {},
|
||||
conflicts: [],
|
||||
decryptionFailures: [],
|
||||
})
|
||||
dbChainMockFns.limit.mockResolvedValue([{ id: 'webhook-1' }])
|
||||
mockGetActiveSpan.mockReturnValue({ recordException: mockRecordException })
|
||||
})
|
||||
|
||||
it('completes the run (does not throw) when the failure was finalized by core', async () => {
|
||||
@@ -246,17 +266,14 @@ describe('executeWebhookJob fault vs error handling', () => {
|
||||
expect(loggingSessionMockFns.mockWaitForPostExecution).toHaveBeenCalled()
|
||||
// User/workflow errors are already recorded by core — the catch must not re-log them.
|
||||
expect(loggingSessionMockFns.mockSafeCompleteWithError).not.toHaveBeenCalled()
|
||||
// The error is still recorded on the run span so it stays visible in traces.
|
||||
expect(mockRecordException).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ message: 'Gmail 2 is missing required fields: Label' })
|
||||
)
|
||||
})
|
||||
|
||||
it('faults the run (re-throws) when the failure was not finalized by core', async () => {
|
||||
mockExecuteWorkflowCore.mockRejectedValue(new Error('Workflow state not found'))
|
||||
const rawError = new Error('Workflow state not found')
|
||||
mockExecuteWorkflowCore.mockRejectedValue(rawError)
|
||||
mockWasExecutionFinalizedByCore.mockReturnValue(false)
|
||||
|
||||
await expect(executeWebhookJob(payload)).rejects.toThrow('Workflow state not found')
|
||||
await expect(executeWebhookJob(payload)).rejects.toBe(rawError)
|
||||
// waitForPostExecution must run on every path so the finalized-by-core signal is always reliable.
|
||||
expect(loggingSessionMockFns.mockWaitForPostExecution).toHaveBeenCalled()
|
||||
// Pipeline/infra errors are recorded here before re-throwing to fault the trigger.dev run.
|
||||
@@ -291,6 +308,92 @@ describe('executeWebhookJob fault vs error handling', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('passes encrypted webhook resolution provenance into workflow execution', async () => {
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
|
||||
personalEncrypted: { WEBHOOK_SECRET: 'personal-ciphertext' },
|
||||
workspaceEncrypted: { WEBHOOK_SECRET: 'workspace-ciphertext' },
|
||||
personalDecrypted: { WEBHOOK_SECRET: 'personal-value' },
|
||||
workspaceDecrypted: { WEBHOOK_SECRET: 'workspace-value' },
|
||||
conflicts: ['WEBHOOK_SECRET'],
|
||||
decryptionFailures: [],
|
||||
})
|
||||
mockResolveWebhookRecordProviderConfig.mockImplementation(
|
||||
async (record, _userId, _workspaceId, options) => {
|
||||
options.onResolved('WEBHOOK_SECRET', options.envVars.WEBHOOK_SECRET)
|
||||
return record
|
||||
}
|
||||
)
|
||||
mockExecuteWorkflowCore.mockResolvedValue({
|
||||
success: true,
|
||||
status: 'completed',
|
||||
output: {},
|
||||
logs: [],
|
||||
executionState: {
|
||||
blockStates: {},
|
||||
executedBlocks: [],
|
||||
blockLogs: [],
|
||||
decisions: {},
|
||||
completedLoops: [],
|
||||
activeExecutionPath: [],
|
||||
},
|
||||
})
|
||||
|
||||
await executeWebhookJob(payload)
|
||||
|
||||
expect(mockResolveWebhookRecordProviderConfig).toHaveBeenCalledWith(
|
||||
{ id: 'webhook-1' },
|
||||
'user-1',
|
||||
'workspace-1',
|
||||
expect.objectContaining({
|
||||
envVars: { WEBHOOK_SECRET: 'workspace-value' },
|
||||
onResolved: expect.any(Function),
|
||||
})
|
||||
)
|
||||
expect(mockExecuteWorkflowCore).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
trustedInitialResolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'WEBHOOK_SECRET', encryptedValue: 'workspace-ciphertext' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
})
|
||||
)
|
||||
expect(mockSetResolvedSecretTraceRegistry).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('installs provenance before a post-resolution webhook setup failure', async () => {
|
||||
const rawMessage = 'Webhook handler exposed activated-secret-value'
|
||||
const rawError = new Error(rawMessage)
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
|
||||
personalEncrypted: {},
|
||||
workspaceEncrypted: { WEBHOOK_SECRET: 'workspace-ciphertext' },
|
||||
personalDecrypted: {},
|
||||
workspaceDecrypted: { WEBHOOK_SECRET: 'activated-secret-value' },
|
||||
conflicts: [],
|
||||
decryptionFailures: [],
|
||||
})
|
||||
mockResolveWebhookRecordProviderConfig.mockImplementation(
|
||||
async (record, _userId, _workspaceId, options) => {
|
||||
options.onResolved('WEBHOOK_SECRET', options.envVars.WEBHOOK_SECRET)
|
||||
return record
|
||||
}
|
||||
)
|
||||
mockGetProviderHandler.mockReturnValue({
|
||||
formatInput: vi.fn().mockRejectedValue(rawError),
|
||||
})
|
||||
|
||||
await expect(executeWebhookJob(payload)).rejects.toBe(rawError)
|
||||
|
||||
expect(mockSetResolvedSecretTraceRegistry).toHaveBeenCalledOnce()
|
||||
expect(loggingSessionMockFns.mockSafeCompleteWithError).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
error: expect.objectContaining({ message: rawMessage }),
|
||||
})
|
||||
)
|
||||
expect(mockExecuteWorkflowCore).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('acknowledges and skips queued webhook work after the workflow is undeployed', async () => {
|
||||
executionPreprocessingMockFns.mockPreprocessExecution.mockResolvedValueOnce({
|
||||
success: true,
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
import { trace } from '@opentelemetry/api'
|
||||
import { db } from '@sim/db'
|
||||
import { account, webhook } from '@sim/db/schema'
|
||||
import { createLogger, runWithRequestContext } from '@sim/logger'
|
||||
@@ -15,6 +14,10 @@ import {
|
||||
import type { AsyncExecutionCorrelation } from '@/lib/core/async-jobs/types'
|
||||
import { createTimeoutAbortController, getTimeoutErrorMessage } from '@/lib/core/execution-limits'
|
||||
import { IdempotencyService, webhookIdempotency } from '@/lib/core/idempotency'
|
||||
import {
|
||||
type EnvironmentResolutionSnapshot,
|
||||
getEffectiveEnvironmentSnapshot,
|
||||
} from '@/lib/environment/utils'
|
||||
import { preprocessExecution } from '@/lib/execution/preprocessing'
|
||||
import { LoggingSession } from '@/lib/logs/execution/logging-session'
|
||||
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
@@ -22,7 +25,10 @@ import {
|
||||
type WebhookAttachment,
|
||||
WebhookAttachmentProcessor,
|
||||
} from '@/lib/webhooks/attachment-processor'
|
||||
import { resolveWebhookRecordProviderConfig } from '@/lib/webhooks/env-resolver'
|
||||
import {
|
||||
resolveWebhookRecordProviderConfig,
|
||||
type WebhookEnvResolutionOptions,
|
||||
} from '@/lib/webhooks/env-resolver'
|
||||
import { getProviderHandler } from '@/lib/webhooks/providers'
|
||||
import {
|
||||
executeWorkflowCore,
|
||||
@@ -40,6 +46,10 @@ import { ExecutionSnapshot } from '@/executor/execution/snapshot'
|
||||
import type { ExecutionMetadata } from '@/executor/execution/types'
|
||||
import type { ExecutionResult } from '@/executor/types'
|
||||
import { hasExecutionResult } from '@/executor/utils/errors'
|
||||
import {
|
||||
createIncompleteResolvedSecretTraceRegistry,
|
||||
createResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { safeAssign } from '@/tools/safe-assign'
|
||||
import { getTrigger, isTriggerValid } from '@/triggers'
|
||||
|
||||
@@ -316,10 +326,32 @@ export async function resolveWebhookExecutionProviderConfig<
|
||||
webhookRecord: T,
|
||||
provider: string,
|
||||
userId: string,
|
||||
workspaceId?: string
|
||||
workspaceId?: string,
|
||||
options?: WebhookEnvResolutionOptions & {
|
||||
onEnvironmentSnapshot?: (snapshot: EnvironmentResolutionSnapshot) => void | Promise<void>
|
||||
}
|
||||
): Promise<T & { providerConfig: Record<string, unknown> }> {
|
||||
try {
|
||||
return await resolveWebhookRecordProviderConfig(webhookRecord, userId, workspaceId)
|
||||
if (!options) {
|
||||
return await resolveWebhookRecordProviderConfig(webhookRecord, userId, workspaceId)
|
||||
}
|
||||
|
||||
const { onEnvironmentSnapshot, ...resolutionOptions } = options
|
||||
if (onEnvironmentSnapshot && resolutionOptions.envVars === undefined) {
|
||||
const snapshot = await getEffectiveEnvironmentSnapshot(userId, workspaceId)
|
||||
await onEnvironmentSnapshot(snapshot)
|
||||
resolutionOptions.envVars = {
|
||||
...snapshot.personalDecrypted,
|
||||
...snapshot.workspaceDecrypted,
|
||||
}
|
||||
}
|
||||
|
||||
return await resolveWebhookRecordProviderConfig(
|
||||
webhookRecord,
|
||||
userId,
|
||||
workspaceId,
|
||||
resolutionOptions
|
||||
)
|
||||
} catch (error) {
|
||||
const errorMessage = toError(error).message
|
||||
throw new Error(
|
||||
@@ -489,11 +521,36 @@ async function executeWebhookJobInternal(
|
||||
throw new Error(`Webhook record not found: ${payload.webhookId}`)
|
||||
}
|
||||
|
||||
const secretScope = { userId: workflowRecord.userId, workspaceId }
|
||||
let resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(secretScope)
|
||||
const resolvedWebhookRecord = await resolveWebhookExecutionProviderConfig(
|
||||
webhookRecord,
|
||||
payload.provider,
|
||||
workflowRecord.userId,
|
||||
workspaceId
|
||||
workspaceId,
|
||||
{
|
||||
onEnvironmentSnapshot: async (secretEnvironment) => {
|
||||
try {
|
||||
resolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: secretEnvironment.personalEncrypted,
|
||||
workspaceEncrypted: secretEnvironment.workspaceEncrypted,
|
||||
personalDecrypted: secretEnvironment.personalDecrypted,
|
||||
workspaceDecrypted: secretEnvironment.workspaceDecrypted,
|
||||
decryptionFailures: secretEnvironment.decryptionFailures,
|
||||
scope: secretScope,
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn(`[${requestId}] Failed to build webhook trace secret catalog`, {
|
||||
error: toError(error).message,
|
||||
})
|
||||
resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(secretScope)
|
||||
}
|
||||
loggingSession.setResolvedSecretTraceRegistry(resolvedSecretTraceRegistry)
|
||||
},
|
||||
onResolved: (name, value) => {
|
||||
resolvedSecretTraceRegistry.recordResolved(name, value)
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
if (handler.formatInput) {
|
||||
@@ -665,6 +722,7 @@ async function executeWebhookJobInternal(
|
||||
snapshot,
|
||||
callbacks: {},
|
||||
loggingSession,
|
||||
trustedInitialResolvedSecretTraceProvenance: resolvedSecretTraceRegistry.exportProvenance(),
|
||||
includeFileBase64: false,
|
||||
base64MaxBytes: undefined,
|
||||
abortSignal: timeoutController.signal,
|
||||
@@ -712,10 +770,6 @@ async function executeWebhookJobInternal(
|
||||
// not a trigger.dev job fault — complete the run normally so we don't fire a false alert. Errors
|
||||
// that were not finalized came from the webhook pipeline itself, so we re-throw to fault below.
|
||||
if (wasExecutionFinalizedByCore(error, executionId)) {
|
||||
// Record the exception on the run span so it stays visible in traces without
|
||||
// marking the span as ERROR — that status is what faults the trigger.dev run.
|
||||
trace.getActiveSpan()?.recordException(toError(error))
|
||||
|
||||
return {
|
||||
success: false,
|
||||
workflowId: payload.workflowId,
|
||||
@@ -757,6 +811,7 @@ async function executeWebhookJobInternal(
|
||||
stackTrace: errorStack,
|
||||
},
|
||||
traceSpans,
|
||||
executionState: executionResult.executionState,
|
||||
})
|
||||
} catch (loggingError) {
|
||||
logger.error(`[${requestId}] Failed to complete logging session`, loggingError)
|
||||
|
||||
@@ -219,6 +219,7 @@ export async function executeWorkflowJob(payload: WorkflowExecutionPayload) {
|
||||
stackTrace: error instanceof Error ? error.stack : undefined,
|
||||
},
|
||||
traceSpans,
|
||||
executionState: executionResult?.executionState,
|
||||
})
|
||||
|
||||
throw error
|
||||
|
||||
@@ -4,11 +4,14 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { clearLargeValueCacheForTests } from '@/lib/execution/payloads/cache'
|
||||
import { isLargeArrayManifest } from '@/lib/execution/payloads/large-array-manifest-metadata'
|
||||
import { projectTraceSpansForSecrets } from '@/lib/logs/execution/trace-secret-projection'
|
||||
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
import { BlockType } from '@/executor/constants'
|
||||
import type { DAGNode } from '@/executor/dag/builder'
|
||||
import { BlockExecutor } from '@/executor/execution/block-executor'
|
||||
import { ExecutionState } from '@/executor/execution/state'
|
||||
import type { BlockHandler, ExecutionContext } from '@/executor/types'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { VariableResolver } from '@/executor/variables/resolver'
|
||||
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
|
||||
|
||||
@@ -436,6 +439,108 @@ describe('BlockExecutor', () => {
|
||||
expect(output?.error).toBeTruthy()
|
||||
expect(output).not.toEqual({ content: '' })
|
||||
})
|
||||
|
||||
it('projects a resolved secret out of Function syntax-error TraceSpans only', async () => {
|
||||
const secret = 'function-secret-literal-7f3a91'
|
||||
const block = createBlock()
|
||||
block.metadata.name = 'Function 1'
|
||||
block.config.params = {
|
||||
code: 'return {{OPENAI_API_KEY}}',
|
||||
language: 'javascript',
|
||||
}
|
||||
const workflow: SerializedWorkflow = {
|
||||
version: '1',
|
||||
blocks: [block],
|
||||
connections: [],
|
||||
loops: {},
|
||||
parallels: {},
|
||||
}
|
||||
const state = new ExecutionState()
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{
|
||||
name: 'OPENAI_API_KEY',
|
||||
plaintext: secret,
|
||||
encryptedValue: 'encrypted-openai-api-key',
|
||||
},
|
||||
])
|
||||
const resolver = new VariableResolver(workflow, {}, state)
|
||||
const syntaxError = `Syntax Error: Line 1: \`return ${secret}\` - Invalid or unexpected token`
|
||||
const handler: BlockHandler = {
|
||||
canHandle: () => true,
|
||||
execute: async (_ctx, _block, inputs) => {
|
||||
expect(inputs.code).toBe(`return ${secret}`)
|
||||
throw new Error(syntaxError)
|
||||
},
|
||||
}
|
||||
const executor = new BlockExecutor(
|
||||
[handler],
|
||||
resolver,
|
||||
{
|
||||
workspaceId: 'workspace-1',
|
||||
executionId: 'execution-1',
|
||||
userId: 'user-1',
|
||||
metadata: {
|
||||
requestId: 'request-1',
|
||||
executionId: 'execution-1',
|
||||
workflowId: 'workflow-1',
|
||||
workspaceId: 'workspace-1',
|
||||
userId: 'user-1',
|
||||
triggerType: 'manual',
|
||||
useDraftState: false,
|
||||
startTime: new Date().toISOString(),
|
||||
},
|
||||
},
|
||||
state
|
||||
)
|
||||
const ctx = createContext(state)
|
||||
ctx.environmentVariables = { OPENAI_API_KEY: secret }
|
||||
ctx.resolvedSecretTraceRegistry = registry
|
||||
|
||||
await expect(executor.execute(ctx, createNode(block), block)).rejects.toThrow(
|
||||
`Function 1: ${syntaxError}`
|
||||
)
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' },
|
||||
])
|
||||
expect(state.getBlockOutput(block.id)).toEqual({ error: syntaxError })
|
||||
expect(ctx.blockLogs[0]).toMatchObject({
|
||||
input: { code: `return ${secret}` },
|
||||
output: { error: syntaxError },
|
||||
error: syntaxError,
|
||||
})
|
||||
|
||||
const rawLogs = structuredClone(ctx.blockLogs)
|
||||
const { traceSpans: rawTraceSpans } = buildTraceSpans({
|
||||
success: false,
|
||||
output: { error: syntaxError },
|
||||
error: `Function 1: ${syntaxError}`,
|
||||
logs: ctx.blockLogs,
|
||||
})
|
||||
const rawTraceSnapshot = structuredClone(rawTraceSpans)
|
||||
const projectedTraceSpans = await projectTraceSpansForSecrets(rawTraceSpans, {
|
||||
registry,
|
||||
store: {
|
||||
workspaceId: 'workspace-1',
|
||||
workflowId: 'workflow-1',
|
||||
executionId: 'execution-1',
|
||||
userId: 'user-1',
|
||||
},
|
||||
})
|
||||
|
||||
expect(ctx.blockLogs).toEqual(rawLogs)
|
||||
expect(rawTraceSpans).toEqual(rawTraceSnapshot)
|
||||
expect(projectedTraceSpans).toEqual([
|
||||
expect.objectContaining({
|
||||
name: 'Function 1',
|
||||
input: expect.objectContaining({ code: 'return {{OPENAI_API_KEY}}' }),
|
||||
output: {
|
||||
error: 'Syntax Error: Line 1: `return {{OPENAI_API_KEY}}` - Invalid or unexpected token',
|
||||
},
|
||||
}),
|
||||
])
|
||||
expect(JSON.stringify(projectedTraceSpans)).not.toContain(secret)
|
||||
})
|
||||
})
|
||||
|
||||
describe('BlockExecutor streaming pump', () => {
|
||||
|
||||
@@ -180,6 +180,7 @@ export class ExecutionEngine {
|
||||
output: this.finalOutput,
|
||||
error: errorMessage,
|
||||
logs: this.context.blockLogs,
|
||||
executionState: this.getSerializableExecutionState(),
|
||||
metadata: this.context.metadata,
|
||||
}
|
||||
|
||||
|
||||
@@ -438,6 +438,7 @@ export class DAGExecutor {
|
||||
},
|
||||
startRunMetadata: this.contextExtensions.startRunMetadata,
|
||||
environmentVariables: this.environmentVariables,
|
||||
resolvedSecretTraceRegistry: this.contextExtensions.resolvedSecretTraceRegistry,
|
||||
workflowVariables: this.workflowVariables,
|
||||
decisions: {
|
||||
router: snapshotState?.decisions?.router
|
||||
|
||||
@@ -6,6 +6,7 @@ import type { DAG, DAGNode } from '@/executor/dag/builder'
|
||||
import { EdgeManager } from '@/executor/execution/edge-manager'
|
||||
import { serializePauseSnapshot } from '@/executor/execution/snapshot-serializer'
|
||||
import type { ExecutionContext } from '@/executor/types'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
function createContext(overrides: Partial<ExecutionContext> = {}): ExecutionContext {
|
||||
return {
|
||||
@@ -38,6 +39,45 @@ function createContext(overrides: Partial<ExecutionContext> = {}): ExecutionCont
|
||||
}
|
||||
|
||||
describe('serializePauseSnapshot', () => {
|
||||
it('persists encrypted resolved-secret provenance and the source execution id', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'raw-secret', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
registry.recordResolved('TOKEN', 'raw-secret')
|
||||
const context = createContext({ resolvedSecretTraceRegistry: registry })
|
||||
|
||||
const snapshot = serializePauseSnapshot(context, ['next-block'])
|
||||
const serialized = JSON.parse(snapshot.snapshot)
|
||||
|
||||
expect(serialized.state.sourceExecutionId).toBe('execution-1')
|
||||
expect(serialized.state.resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
|
||||
})
|
||||
expect(snapshot.snapshot).not.toContain('raw-secret')
|
||||
})
|
||||
|
||||
it('persists a complete zero-entry provenance state for a fresh execution', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
|
||||
const snapshot = serializePauseSnapshot(
|
||||
createContext({ resolvedSecretTraceRegistry: registry }),
|
||||
['next-block']
|
||||
)
|
||||
const serialized = JSON.parse(snapshot.snapshot)
|
||||
|
||||
expect(serialized.state.resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('serializes batched parallel accumulated outputs for cross-process resume', () => {
|
||||
const context = createContext({
|
||||
parallelExecutions: new Map([
|
||||
|
||||
@@ -224,6 +224,19 @@ export function serializePauseSnapshot(
|
||||
dagIncomingEdges,
|
||||
deactivatedEdges: edgeManager?.getDeactivatedEdges(),
|
||||
nodesWithActivatedEdge: edgeManager?.getNodesWithActivatedEdge(),
|
||||
sourceExecutionId: context.executionId,
|
||||
trustedLargeValueAccess: {
|
||||
executionIds: Array.from(
|
||||
new Set(
|
||||
[context.executionId, ...(context.largeValueExecutionIds ?? [])].filter(
|
||||
(id): id is string => Boolean(id)
|
||||
)
|
||||
)
|
||||
),
|
||||
largeValueKeys: Array.from(new Set(context.largeValueKeys ?? [])),
|
||||
fileKeys: Array.from(new Set(context.fileKeys ?? [])),
|
||||
},
|
||||
resolvedSecretTraceProvenance: context.resolvedSecretTraceRegistry?.exportProvenance(),
|
||||
}
|
||||
|
||||
assertSnapshotValueIsCompact(context.workflowVariables, 'workflow variables')
|
||||
|
||||
@@ -10,6 +10,10 @@ import type {
|
||||
StartBlockRunMetadata,
|
||||
StreamingExecution,
|
||||
} from '@/executor/types'
|
||||
import type {
|
||||
ResolvedSecretTraceProvenanceV1,
|
||||
ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { RunFromBlockContext } from '@/executor/utils/run-from-block'
|
||||
import type { SubflowType } from '@/stores/workflows/workflow/types'
|
||||
|
||||
@@ -88,6 +92,16 @@ export interface SerializableExecutionState {
|
||||
deactivatedEdges?: string[]
|
||||
nodesWithActivatedEdge?: string[]
|
||||
completedPauseContexts?: string[]
|
||||
/** Server execution that produced this state; callers must still verify it against storage. */
|
||||
sourceExecutionId?: string
|
||||
/** Server-only closure authorizing offloaded values carried by trusted restored state. */
|
||||
trustedLargeValueAccess?: {
|
||||
executionIds: string[]
|
||||
largeValueKeys: string[]
|
||||
fileKeys: string[]
|
||||
}
|
||||
/** Encrypted-only provenance for Secrets-tab values resolved during this execution. */
|
||||
resolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceV1
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -144,6 +158,17 @@ export interface ChildWorkflowContext {
|
||||
depth: number
|
||||
}
|
||||
|
||||
export interface BlockCompletionCallbackData {
|
||||
input?: unknown
|
||||
output: NormalizedBlockOutput
|
||||
executionTime: number
|
||||
startedAt: string
|
||||
executionOrder: number
|
||||
endedAt: string
|
||||
/** Per-invocation unique ID linking this workflow block execution to its child block events. */
|
||||
childWorkflowInstanceId?: string
|
||||
}
|
||||
|
||||
export interface ExecutionCallbacks {
|
||||
onStream?: (streamingExec: StreamingExecution) => Promise<void>
|
||||
onBlockStart?: (
|
||||
@@ -158,7 +183,7 @@ export interface ExecutionCallbacks {
|
||||
blockId: string,
|
||||
blockName: string,
|
||||
blockType: string,
|
||||
output: any,
|
||||
output: BlockCompletionCallbackData,
|
||||
iterationContext?: IterationContext,
|
||||
childWorkflowContext?: ChildWorkflowContext
|
||||
) => Promise<void>
|
||||
@@ -214,6 +239,7 @@ export interface ContextExtensions {
|
||||
}>
|
||||
dagIncomingEdges?: Record<string, string[]>
|
||||
snapshotState?: SerializableExecutionState
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
metadata?: ExecutionMetadata
|
||||
/**
|
||||
* Trusted run metadata injected into the Start block output when its
|
||||
@@ -247,16 +273,7 @@ export interface ContextExtensions {
|
||||
blockId: string,
|
||||
blockName: string,
|
||||
blockType: string,
|
||||
output: {
|
||||
input?: any
|
||||
output: NormalizedBlockOutput
|
||||
executionTime: number
|
||||
startedAt: string
|
||||
executionOrder: number
|
||||
endedAt: string
|
||||
/** Per-invocation unique ID linking this workflow block execution to its child block events. */
|
||||
childWorkflowInstanceId?: string
|
||||
},
|
||||
output: BlockCompletionCallbackData,
|
||||
iterationContext?: IterationContext,
|
||||
childWorkflowContext?: ChildWorkflowContext
|
||||
) => Promise<void>
|
||||
|
||||
@@ -1182,48 +1182,55 @@ export class AgentBlockHandler implements BlockHandler {
|
||||
|
||||
const { blockData, blockNameMapping } = collectBlockData(ctx)
|
||||
|
||||
const response = await executeProviderRequest(providerId, {
|
||||
model,
|
||||
systemPrompt: 'systemPrompt' in providerRequest ? providerRequest.systemPrompt : undefined,
|
||||
context: 'context' in providerRequest ? providerRequest.context : undefined,
|
||||
tools: providerRequest.tools,
|
||||
temperature: providerRequest.temperature,
|
||||
maxTokens: providerRequest.maxTokens,
|
||||
apiKey: finalApiKey,
|
||||
azureEndpoint: providerRequest.azureEndpoint,
|
||||
azureApiVersion: providerRequest.azureApiVersion,
|
||||
vertexProject: providerRequest.vertexProject,
|
||||
vertexLocation: providerRequest.vertexLocation,
|
||||
bedrockAccessKeyId: providerRequest.bedrockAccessKeyId,
|
||||
bedrockSecretKey: providerRequest.bedrockSecretKey,
|
||||
bedrockRegion: providerRequest.bedrockRegion,
|
||||
responseFormat: providerRequest.responseFormat,
|
||||
workflowId: providerRequest.workflowId,
|
||||
workspaceId: ctx.workspaceId,
|
||||
userId: ctx.userId,
|
||||
stream: providerRequest.stream,
|
||||
messages: 'messages' in providerRequest ? providerRequest.messages : undefined,
|
||||
environmentVariables: normalizeStringRecord(ctx.environmentVariables),
|
||||
workflowVariables: normalizeWorkflowVariables(ctx.workflowVariables),
|
||||
blockData,
|
||||
blockNameMapping,
|
||||
isDeployedContext: ctx.isDeployedContext,
|
||||
callChain: ctx.callChain,
|
||||
billingAttribution: ctx.metadata.billingAttribution,
|
||||
// Reaches tool `_context` via `prepareToolExecution`, so a tool that starts
|
||||
// its own child execution (a custom block) correlates and cancels against
|
||||
// this real run instead of minting a phantom id.
|
||||
executionId: ctx.executionId,
|
||||
reasoningEffort: providerRequest.reasoningEffort,
|
||||
verbosity: providerRequest.verbosity,
|
||||
thinkingLevel: providerRequest.thinkingLevel,
|
||||
promptCaching: providerRequest.promptCaching,
|
||||
// Stable per-block identity; providers use it to route cache lookups.
|
||||
blockId: block.id,
|
||||
previousInteractionId: providerRequest.previousInteractionId,
|
||||
agentEvents: providerRequest.agentEvents,
|
||||
abortSignal: ctx.abortSignal,
|
||||
})
|
||||
const response = await executeProviderRequest(
|
||||
providerId,
|
||||
{
|
||||
model,
|
||||
systemPrompt:
|
||||
'systemPrompt' in providerRequest ? providerRequest.systemPrompt : undefined,
|
||||
context: 'context' in providerRequest ? providerRequest.context : undefined,
|
||||
tools: providerRequest.tools,
|
||||
temperature: providerRequest.temperature,
|
||||
maxTokens: providerRequest.maxTokens,
|
||||
apiKey: finalApiKey,
|
||||
azureEndpoint: providerRequest.azureEndpoint,
|
||||
azureApiVersion: providerRequest.azureApiVersion,
|
||||
vertexProject: providerRequest.vertexProject,
|
||||
vertexLocation: providerRequest.vertexLocation,
|
||||
bedrockAccessKeyId: providerRequest.bedrockAccessKeyId,
|
||||
bedrockSecretKey: providerRequest.bedrockSecretKey,
|
||||
bedrockRegion: providerRequest.bedrockRegion,
|
||||
responseFormat: providerRequest.responseFormat,
|
||||
workflowId: providerRequest.workflowId,
|
||||
workspaceId: ctx.workspaceId,
|
||||
userId: ctx.userId,
|
||||
stream: providerRequest.stream,
|
||||
messages: 'messages' in providerRequest ? providerRequest.messages : undefined,
|
||||
environmentVariables: normalizeStringRecord(ctx.environmentVariables),
|
||||
workflowVariables: normalizeWorkflowVariables(ctx.workflowVariables),
|
||||
blockData,
|
||||
blockNameMapping,
|
||||
isDeployedContext: ctx.isDeployedContext,
|
||||
callChain: ctx.callChain,
|
||||
billingAttribution: ctx.metadata.billingAttribution,
|
||||
// Reaches tool `_context` via `prepareToolExecution`, so a tool that starts
|
||||
// its own child execution (a custom block) correlates and cancels against
|
||||
// this real run instead of minting a phantom id.
|
||||
executionId: ctx.executionId,
|
||||
reasoningEffort: providerRequest.reasoningEffort,
|
||||
verbosity: providerRequest.verbosity,
|
||||
thinkingLevel: providerRequest.thinkingLevel,
|
||||
promptCaching: providerRequest.promptCaching,
|
||||
// Stable per-block identity; providers use it to route cache lookups.
|
||||
blockId: block.id,
|
||||
previousInteractionId: providerRequest.previousInteractionId,
|
||||
agentEvents: providerRequest.agentEvents,
|
||||
abortSignal: ctx.abortSignal,
|
||||
},
|
||||
{
|
||||
resolvedSecretTraceRegistry: ctx.resolvedSecretTraceRegistry,
|
||||
}
|
||||
)
|
||||
|
||||
return this.processProviderResponse(response, block, responseFormat)
|
||||
} catch (error) {
|
||||
|
||||
@@ -5,6 +5,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { BlockType } from '@/executor/constants'
|
||||
import { MothershipBlockHandler } from '@/executor/handlers/mothership/mothership-handler'
|
||||
import type { ExecutionContext, StreamingExecution } from '@/executor/types'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { SerializedBlock } from '@/serializer/types'
|
||||
|
||||
const BILLING_ATTRIBUTION = {
|
||||
@@ -20,6 +21,13 @@ const BILLING_ATTRIBUTION = {
|
||||
payerSubscription: null,
|
||||
} as const
|
||||
|
||||
const PRIVATE_PROVENANCE_TYPE = 'resolved-secret-provenance-v1'
|
||||
const PRIVATE_PROVENANCE = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
|
||||
}
|
||||
|
||||
const {
|
||||
mockBuildAuthHeaders,
|
||||
mockBuildAPIUrl,
|
||||
@@ -96,6 +104,19 @@ async function readStreamText(stream: ReadableStream): Promise<string> {
|
||||
return text
|
||||
}
|
||||
|
||||
function createTraceRegistryMock(): ResolvedSecretTraceRegistry & {
|
||||
importProvenance: ReturnType<typeof vi.fn>
|
||||
markIncomplete: ReturnType<typeof vi.fn>
|
||||
} {
|
||||
return {
|
||||
importProvenance: vi.fn().mockResolvedValue(true),
|
||||
markIncomplete: vi.fn(),
|
||||
} as unknown as ResolvedSecretTraceRegistry & {
|
||||
importProvenance: ReturnType<typeof vi.fn>
|
||||
markIncomplete: ReturnType<typeof vi.fn>
|
||||
}
|
||||
}
|
||||
|
||||
describe('MothershipBlockHandler', () => {
|
||||
let handler: MothershipBlockHandler
|
||||
let block: SerializedBlock
|
||||
@@ -152,7 +173,7 @@ describe('MothershipBlockHandler', () => {
|
||||
resetEnvMock()
|
||||
})
|
||||
|
||||
function createNdjsonResponse(events: unknown[]): Response {
|
||||
function createNdjsonResponse(events: unknown[], headers: Record<string, string> = {}): Response {
|
||||
const encoder = new TextEncoder()
|
||||
return new Response(
|
||||
new ReadableStream({
|
||||
@@ -165,11 +186,146 @@ describe('MothershipBlockHandler', () => {
|
||||
}),
|
||||
{
|
||||
status: 200,
|
||||
headers: { 'Content-Type': 'application/x-ndjson; charset=utf-8' },
|
||||
headers: { 'Content-Type': 'application/x-ndjson; charset=utf-8', ...headers },
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
it('imports marker-gated JSON provenance without exposing it in block output', async () => {
|
||||
const registry = createTraceRegistryMock()
|
||||
context.resolvedSecretTraceRegistry = registry
|
||||
mockGenerateId
|
||||
.mockReturnValueOnce('chat-uuid')
|
||||
.mockReturnValueOnce('message-uuid')
|
||||
.mockReturnValueOnce('request-uuid')
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
content: 'raw secret remains functional',
|
||||
toolCalls: [],
|
||||
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
|
||||
}),
|
||||
{
|
||||
status: 200,
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-sim-private-tool-metadata': PRIVATE_PROVENANCE_TYPE,
|
||||
},
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
const result = await handler.execute(context, block, { prompt: 'Hello' })
|
||||
|
||||
const [, options] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(options.headers).toMatchObject({
|
||||
'x-sim-request-private-tool-metadata': PRIVATE_PROVENANCE_TYPE,
|
||||
})
|
||||
expect(registry.importProvenance).toHaveBeenCalledWith(PRIVATE_PROVENANCE, {
|
||||
trusted: true,
|
||||
})
|
||||
expect(registry.markIncomplete).not.toHaveBeenCalled()
|
||||
expect(result).toMatchObject({ content: 'raw secret remains functional' })
|
||||
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
|
||||
expect(JSON.stringify(result)).not.toContain('encrypted-secret')
|
||||
})
|
||||
|
||||
it('rejects unmarked provenance while keeping private metadata out of block output', async () => {
|
||||
const registry = createTraceRegistryMock()
|
||||
context.resolvedSecretTraceRegistry = registry
|
||||
mockGenerateId
|
||||
.mockReturnValueOnce('chat-uuid')
|
||||
.mockReturnValueOnce('message-uuid')
|
||||
.mockReturnValueOnce('request-uuid')
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
content: 'unchanged output',
|
||||
toolCalls: [],
|
||||
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
|
||||
}),
|
||||
{ status: 200, headers: { 'Content-Type': 'application/json' } }
|
||||
)
|
||||
)
|
||||
|
||||
const result = await handler.execute(context, block, { prompt: 'Hello' })
|
||||
|
||||
expect(registry.importProvenance).not.toHaveBeenCalled()
|
||||
expect(registry.markIncomplete).toHaveBeenCalledOnce()
|
||||
expect(result).toMatchObject({ content: 'unchanged output' })
|
||||
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
|
||||
expect(JSON.stringify(result)).not.toContain('encrypted-secret')
|
||||
})
|
||||
|
||||
it('imports provenance from a terminal NDJSON error without forcing structural fallback', async () => {
|
||||
const registry = createTraceRegistryMock()
|
||||
context.resolvedSecretTraceRegistry = registry
|
||||
mockGenerateId
|
||||
.mockReturnValueOnce('chat-uuid')
|
||||
.mockReturnValueOnce('message-uuid')
|
||||
.mockReturnValueOnce('request-uuid')
|
||||
fetchMock.mockResolvedValue(
|
||||
createNdjsonResponse(
|
||||
[
|
||||
{
|
||||
type: 'error',
|
||||
error: 'secret-backed failure',
|
||||
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
|
||||
},
|
||||
],
|
||||
{ 'x-sim-private-tool-metadata': PRIVATE_PROVENANCE_TYPE }
|
||||
)
|
||||
)
|
||||
|
||||
await expect(handler.execute(context, block, { prompt: 'Hello' })).rejects.toThrow(
|
||||
'Sim execution failed: secret-backed failure'
|
||||
)
|
||||
|
||||
expect(registry.importProvenance).toHaveBeenCalledWith(PRIVATE_PROVENANCE, {
|
||||
trusted: true,
|
||||
})
|
||||
expect(registry.markIncomplete).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('imports final provenance for selected-output streaming without adding it to output', async () => {
|
||||
const registry = createTraceRegistryMock()
|
||||
context.resolvedSecretTraceRegistry = registry
|
||||
context.stream = true
|
||||
context.selectedOutputs = [`${block.id}_content`]
|
||||
mockGenerateId
|
||||
.mockReturnValueOnce('chat-uuid')
|
||||
.mockReturnValueOnce('message-uuid')
|
||||
.mockReturnValueOnce('request-uuid')
|
||||
fetchMock.mockResolvedValue(
|
||||
createNdjsonResponse(
|
||||
[
|
||||
{ type: 'chunk', content: 'unchanged' },
|
||||
{
|
||||
type: 'final',
|
||||
data: {
|
||||
content: 'unchanged',
|
||||
toolCalls: [],
|
||||
__resolvedSecretTraceProvenance: PRIVATE_PROVENANCE,
|
||||
},
|
||||
},
|
||||
],
|
||||
{ 'x-sim-private-tool-metadata': PRIVATE_PROVENANCE_TYPE }
|
||||
)
|
||||
)
|
||||
|
||||
const result = (await handler.execute(context, block, {
|
||||
prompt: 'Hello',
|
||||
})) as StreamingExecution
|
||||
|
||||
await expect(readStreamText(result.stream)).resolves.toBe('unchanged')
|
||||
expect(registry.importProvenance).toHaveBeenCalledWith(PRIVATE_PROVENANCE, {
|
||||
trusted: true,
|
||||
})
|
||||
expect(registry.markIncomplete).not.toHaveBeenCalled()
|
||||
expect(JSON.stringify(result.execution.output)).not.toContain('__resolvedSecretTraceProvenance')
|
||||
expect(JSON.stringify(result.execution.output)).not.toContain('encrypted-secret')
|
||||
})
|
||||
|
||||
it('forwards workflow and execution metadata with generated UUID ids', async () => {
|
||||
mockGenerateId.mockReturnValueOnce('chat-uuid')
|
||||
mockGenerateId.mockReturnValueOnce('message-uuid')
|
||||
|
||||
@@ -8,6 +8,12 @@ import {
|
||||
import { env } from '@/lib/core/config/env'
|
||||
import { isExecutionCancelled, isRedisCancellationEnabled } from '@/lib/execution/cancellation'
|
||||
import { readUserFileContent } from '@/lib/execution/payloads/materialization.server'
|
||||
import {
|
||||
PRIVATE_TOOL_METADATA_REQUEST_HEADER,
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
responseHasPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
import {
|
||||
createFileContentFromBase64,
|
||||
type MessageContent,
|
||||
@@ -24,6 +30,7 @@ import type {
|
||||
StreamingExecution,
|
||||
} from '@/executor/types'
|
||||
import { buildAPIUrl, buildAuthHeaders, extractAPIErrorMessage } from '@/executor/utils/http'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { SerializedBlock } from '@/serializer/types'
|
||||
|
||||
const logger = createLogger('MothershipBlockHandler')
|
||||
@@ -43,13 +50,31 @@ type MothershipExecuteResult = {
|
||||
tokens?: Record<string, unknown>
|
||||
toolCalls?: Array<Record<string, unknown>>
|
||||
cost?: unknown
|
||||
}
|
||||
} & Partial<Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>>
|
||||
|
||||
type MothershipExecuteStreamEvent =
|
||||
| { type: 'heartbeat'; timestamp?: string }
|
||||
| { type: 'chunk'; content?: string }
|
||||
| { type: 'final'; data: MothershipExecuteResult }
|
||||
| { type: 'error'; error?: string }
|
||||
| ({ type: 'error'; error?: string } & Partial<
|
||||
Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>
|
||||
>)
|
||||
|
||||
async function consumeMothershipProvenance(
|
||||
payload: Partial<Record<typeof RESOLVED_SECRET_PROVENANCE_FIELD, unknown>>,
|
||||
response: Response,
|
||||
registry?: ResolvedSecretTraceRegistry
|
||||
): Promise<boolean> {
|
||||
if (!registry) return true
|
||||
if (
|
||||
!responseHasPrivateToolMetadata(response.headers, RESOLVED_SECRET_PROVENANCE_METADATA_V1) ||
|
||||
!Object.hasOwn(payload, RESOLVED_SECRET_PROVENANCE_FIELD)
|
||||
) {
|
||||
registry.markIncomplete()
|
||||
return false
|
||||
}
|
||||
return registry.importProvenance(payload[RESOLVED_SECRET_PROVENANCE_FIELD], { trusted: true })
|
||||
}
|
||||
|
||||
function parseMothershipExecuteStreamLine(line: string): MothershipExecuteStreamEvent | undefined {
|
||||
const trimmed = line.trim()
|
||||
@@ -100,10 +125,15 @@ function isContentSelectedForStreaming(ctx: ExecutionContext, block: SerializedB
|
||||
)
|
||||
}
|
||||
|
||||
async function readMothershipExecuteResponse(response: Response): Promise<MothershipExecuteResult> {
|
||||
async function readMothershipExecuteResponse(
|
||||
response: Response,
|
||||
registry?: ResolvedSecretTraceRegistry
|
||||
): Promise<MothershipExecuteResult> {
|
||||
const contentType = response.headers.get('content-type') || ''
|
||||
if (!contentType.includes('application/x-ndjson')) {
|
||||
return response.json()
|
||||
const result = (await response.json()) as MothershipExecuteResult
|
||||
await consumeMothershipProvenance(result, response, registry)
|
||||
return result
|
||||
}
|
||||
|
||||
if (!response.body) {
|
||||
@@ -114,8 +144,9 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
|
||||
const decoder = new TextDecoder()
|
||||
let buffer = ''
|
||||
let finalResult: MothershipExecuteResult | undefined
|
||||
let receivedTerminalProvenance = false
|
||||
|
||||
const processLine = (line: string) => {
|
||||
const processLine = async (line: string): Promise<void> => {
|
||||
const event = parseMothershipExecuteStreamLine(line)
|
||||
if (!event) return
|
||||
|
||||
@@ -124,10 +155,12 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
|
||||
}
|
||||
|
||||
if (event.type === 'error') {
|
||||
receivedTerminalProvenance = await consumeMothershipProvenance(event, response, registry)
|
||||
throw new Error(`Sim execution failed: ${event.error || 'Unknown error'}`)
|
||||
}
|
||||
|
||||
if (event.type === 'final') {
|
||||
await consumeMothershipProvenance(event.data, response, registry)
|
||||
finalResult = event.data
|
||||
return
|
||||
}
|
||||
@@ -144,12 +177,12 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
|
||||
const lines = buffer.split('\n')
|
||||
buffer = lines.pop() ?? ''
|
||||
for (const line of lines) {
|
||||
processLine(line)
|
||||
await processLine(line)
|
||||
}
|
||||
}
|
||||
|
||||
buffer += decoder.decode()
|
||||
processLine(buffer)
|
||||
await processLine(buffer)
|
||||
|
||||
if (!finalResult) {
|
||||
throw new Error('Sim execution stream ended without a final result')
|
||||
@@ -157,6 +190,7 @@ async function readMothershipExecuteResponse(response: Response): Promise<Mother
|
||||
|
||||
return finalResult
|
||||
} finally {
|
||||
if (!finalResult && !receivedTerminalProvenance) registry?.markIncomplete()
|
||||
reader.releaseLock()
|
||||
}
|
||||
}
|
||||
@@ -168,6 +202,7 @@ function createMothershipStreamingExecution(
|
||||
options: {
|
||||
onCancel?: (reason?: unknown) => void
|
||||
onDone?: () => void
|
||||
registry?: ResolvedSecretTraceRegistry
|
||||
} = {}
|
||||
): StreamingExecution {
|
||||
if (!response.body) {
|
||||
@@ -191,8 +226,9 @@ function createMothershipStreamingExecution(
|
||||
const encoder = new TextEncoder()
|
||||
let buffer = ''
|
||||
let sawFinal = false
|
||||
let receivedTerminalProvenance = false
|
||||
|
||||
const processLine = (line: string) => {
|
||||
const processLine = async (line: string): Promise<void> => {
|
||||
const event = parseMothershipExecuteStreamLine(line)
|
||||
if (!event) return
|
||||
|
||||
@@ -208,10 +244,16 @@ function createMothershipStreamingExecution(
|
||||
}
|
||||
|
||||
if (event.type === 'error') {
|
||||
receivedTerminalProvenance = await consumeMothershipProvenance(
|
||||
event,
|
||||
response,
|
||||
options.registry
|
||||
)
|
||||
throw new Error(`Sim execution failed: ${event.error || 'Unknown error'}`)
|
||||
}
|
||||
|
||||
if (event.type === 'final') {
|
||||
await consumeMothershipProvenance(event.data, response, options.registry)
|
||||
sawFinal = true
|
||||
Object.assign(output, formatMothershipBlockOutput(event.data, fallbackChatId))
|
||||
return
|
||||
@@ -230,12 +272,12 @@ function createMothershipStreamingExecution(
|
||||
const lines = buffer.split('\n')
|
||||
buffer = lines.pop() ?? ''
|
||||
for (const line of lines) {
|
||||
processLine(line)
|
||||
await processLine(line)
|
||||
}
|
||||
}
|
||||
|
||||
buffer += decoder.decode()
|
||||
processLine(buffer)
|
||||
await processLine(buffer)
|
||||
|
||||
if (!sawFinal) {
|
||||
throw new Error('Sim execution stream ended without a final result')
|
||||
@@ -249,6 +291,7 @@ function createMothershipStreamingExecution(
|
||||
controller.error(error)
|
||||
}
|
||||
} finally {
|
||||
if (!sawFinal && !receivedTerminalProvenance) options.registry?.markIncomplete()
|
||||
cleanup()
|
||||
reader?.releaseLock()
|
||||
}
|
||||
@@ -382,6 +425,9 @@ export class MothershipBlockHandler implements BlockHandler {
|
||||
const headers = await buildAuthHeaders(ctx.userId)
|
||||
headers.Accept = 'application/x-ndjson'
|
||||
headers[MOTHERSHIP_EXECUTE_STREAM_HEADER] = MOTHERSHIP_EXECUTE_STREAM_VALUE
|
||||
if (ctx.resolvedSecretTraceRegistry) {
|
||||
headers[PRIVATE_TOOL_METADATA_REQUEST_HEADER] = RESOLVED_SECRET_PROVENANCE_METADATA_V1
|
||||
}
|
||||
if (!ctx.metadata.billingAttribution) {
|
||||
throw new Error('Billing attribution is required for Mothership execution')
|
||||
}
|
||||
@@ -474,6 +520,14 @@ export class MothershipBlockHandler implements BlockHandler {
|
||||
})
|
||||
|
||||
if (!response.ok) {
|
||||
if (ctx.resolvedSecretTraceRegistry) {
|
||||
try {
|
||||
const payload = (await response.clone().json()) as MothershipExecuteResult
|
||||
await consumeMothershipProvenance(payload, response, ctx.resolvedSecretTraceRegistry)
|
||||
} catch {
|
||||
ctx.resolvedSecretTraceRegistry.markIncomplete()
|
||||
}
|
||||
}
|
||||
const errorMsg = await extractAPIErrorMessage(response)
|
||||
throw new Error(`Sim execution failed: ${errorMsg}`)
|
||||
}
|
||||
@@ -486,12 +540,13 @@ export class MothershipBlockHandler implements BlockHandler {
|
||||
}
|
||||
},
|
||||
onDone: cleanupAbortListeners,
|
||||
registry: ctx.resolvedSecretTraceRegistry,
|
||||
})
|
||||
cleanupImmediately = false
|
||||
return streamingExecution
|
||||
}
|
||||
|
||||
const result = await readMothershipExecuteResponse(response)
|
||||
const result = await readMothershipExecuteResponse(response, ctx.resolvedSecretTraceRegistry)
|
||||
return formatMothershipBlockOutput(result, chatId)
|
||||
} finally {
|
||||
if (cleanupImmediately) {
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
buildCustomBlockExecutionContext,
|
||||
runCustomBlockTool,
|
||||
} from '@/executor/handlers/workflow/custom-block-tool-runner'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
describe('buildCustomBlockExecutionContext', () => {
|
||||
it('carries consumer identity, inherits the call chain, and is fully scaffolded', () => {
|
||||
@@ -145,3 +146,16 @@ describe('buildCustomBlockExecutionContext cancellation', () => {
|
||||
expect(buildCustomBlockExecutionContext({ workspaceId: 'ws-1' }).abortSignal).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildCustomBlockExecutionContext secret provenance', () => {
|
||||
it('carries the server-only parent registry without putting it in model parameters', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry()
|
||||
|
||||
const ctx = buildCustomBlockExecutionContext(
|
||||
{ workspaceId: 'ws-1' },
|
||||
{ resolvedSecretTraceRegistry: registry }
|
||||
)
|
||||
|
||||
expect(ctx.resolvedSecretTraceRegistry).toBe(registry)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -4,6 +4,7 @@ import { generateId } from '@sim/utils/id'
|
||||
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
|
||||
import { WorkflowBlockHandler } from '@/executor/handlers/workflow/workflow-handler'
|
||||
import type { ExecutionContext } from '@/executor/types'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { SerializedBlock } from '@/serializer/types'
|
||||
import type { ToolResponse } from '@/tools/types'
|
||||
|
||||
@@ -49,7 +50,10 @@ interface CustomBlockToolParams {
|
||||
*/
|
||||
export function buildCustomBlockExecutionContext(
|
||||
context: CustomBlockExecutorContext,
|
||||
options: { abortSignal?: AbortSignal } = {}
|
||||
options: {
|
||||
abortSignal?: AbortSignal
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
} = {}
|
||||
): ExecutionContext {
|
||||
// Prefer the invoking agent run's ids so correlation and cancellation both
|
||||
// point at a real execution; fall back only when a caller could not supply them.
|
||||
@@ -66,6 +70,7 @@ export function buildCustomBlockExecutionContext(
|
||||
// Without this the child's cancellation bridge has nothing to abort on:
|
||||
// the agent tool loop owns the only signal reaching this path.
|
||||
abortSignal: options.abortSignal,
|
||||
resolvedSecretTraceRegistry: options.resolvedSecretTraceRegistry,
|
||||
environmentVariables: {},
|
||||
blockStates: new Map(),
|
||||
executedBlocks: new Set(),
|
||||
@@ -102,7 +107,10 @@ export function buildCustomBlockExecutionContext(
|
||||
*/
|
||||
export async function runCustomBlockTool(
|
||||
params: CustomBlockToolParams,
|
||||
options: { abortSignal?: AbortSignal } = {}
|
||||
options: {
|
||||
abortSignal?: AbortSignal
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
} = {}
|
||||
): Promise<ToolResponse> {
|
||||
if (!params.blockType) {
|
||||
return { success: false, output: {}, error: 'Missing custom block type' }
|
||||
@@ -110,6 +118,7 @@ export async function runCustomBlockTool(
|
||||
|
||||
const ctx = buildCustomBlockExecutionContext(params._context ?? {}, {
|
||||
abortSignal: options.abortSignal,
|
||||
resolvedSecretTraceRegistry: options.resolvedSecretTraceRegistry,
|
||||
})
|
||||
const block: SerializedBlock = {
|
||||
id: generateId(),
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { environmentUtilsMockFns, resetEnvironmentUtilsMock } from '@sim/testing'
|
||||
import { encryptionMockFns, environmentUtilsMockFns, resetEnvironmentUtilsMock } from '@sim/testing'
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, type Mock, vi } from 'vitest'
|
||||
import { getBlock } from '@/blocks/registry'
|
||||
import { BlockType } from '@/executor/constants'
|
||||
@@ -9,6 +9,10 @@ import {
|
||||
WorkflowBlockHandler,
|
||||
} from '@/executor/handlers/workflow/workflow-handler'
|
||||
import type { ExecutionContext } from '@/executor/types'
|
||||
import {
|
||||
ANONYMOUS_SECRET_TRACE_REPLACEMENT,
|
||||
ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { SerializedBlock } from '@/serializer/types'
|
||||
|
||||
const {
|
||||
@@ -24,6 +28,8 @@ const {
|
||||
mockSafeComplete,
|
||||
mockSafeCompleteWithError,
|
||||
mockSafeCompleteWithCancellation,
|
||||
mockSetResolvedSecretTraceRegistry,
|
||||
mockSetTraceLargeValueAccess,
|
||||
mockDispose,
|
||||
executorOptions,
|
||||
loggingSessionArgs,
|
||||
@@ -40,6 +46,8 @@ const {
|
||||
mockSafeComplete: vi.fn(),
|
||||
mockSafeCompleteWithError: vi.fn(),
|
||||
mockSafeCompleteWithCancellation: vi.fn(),
|
||||
mockSetResolvedSecretTraceRegistry: vi.fn(),
|
||||
mockSetTraceLargeValueAccess: vi.fn(),
|
||||
mockDispose: vi.fn(),
|
||||
executorOptions: [] as Array<Record<string, any>>,
|
||||
loggingSessionArgs: [] as Array<any[]>,
|
||||
@@ -54,6 +62,8 @@ vi.mock('@/lib/logs/execution/logging-session', () => ({
|
||||
safeComplete = mockSafeComplete
|
||||
safeCompleteWithError = mockSafeCompleteWithError
|
||||
safeCompleteWithCancellation = mockSafeCompleteWithCancellation
|
||||
setResolvedSecretTraceRegistry = mockSetResolvedSecretTraceRegistry
|
||||
setTraceLargeValueAccess = mockSetTraceLargeValueAccess
|
||||
onBlockStart = vi.fn()
|
||||
onBlockComplete = vi.fn()
|
||||
},
|
||||
@@ -63,6 +73,11 @@ vi.mock('@/lib/logs/execution/trace-spans/trace-spans', () => ({
|
||||
buildTraceSpans: mockBuildTraceSpans,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/security/encryption', () => ({
|
||||
decryptSecret: encryptionMockFns.mockDecryptSecret,
|
||||
encryptSecret: encryptionMockFns.mockEncryptSecret,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workflows/custom-blocks/child-execution', () => ({
|
||||
admitCustomBlockChildExecution: mockAdmitCustomBlockChildExecution,
|
||||
trackChildRun: mockTrackChildRun,
|
||||
@@ -1147,6 +1162,54 @@ describe('WorkflowBlockHandler', () => {
|
||||
expect(ctx.largeValueExecutionIds).toContain('grandchild-execution-id')
|
||||
})
|
||||
|
||||
it('imports only publisher secret provenance that crosses the curated output boundary', async () => {
|
||||
encryptionMockFns.mockDecryptSecret.mockResolvedValueOnce({
|
||||
decrypted: 'publisher-secret',
|
||||
})
|
||||
mockGetPersonalAndWorkspaceEnv.mockResolvedValueOnce({
|
||||
personalDecrypted: { SECRET: 'publisher-secret', UNUSED: 'unused-secret' },
|
||||
workspaceDecrypted: {},
|
||||
personalEncrypted: {
|
||||
SECRET: 'publisher-ciphertext',
|
||||
UNUSED: 'unused-ciphertext',
|
||||
},
|
||||
workspaceEncrypted: {},
|
||||
decryptionFailures: [],
|
||||
})
|
||||
mockExecutorExecute.mockImplementationOnce(async () => {
|
||||
const childRegistry = executorOptions.at(-1)?.contextExtensions
|
||||
.resolvedSecretTraceRegistry as ResolvedSecretTraceRegistry
|
||||
childRegistry.recordResolved('SECRET', 'publisher-secret')
|
||||
childRegistry.recordResolved('UNUSED', 'unused-secret')
|
||||
return {
|
||||
success: true,
|
||||
output: {},
|
||||
logs: [
|
||||
{
|
||||
blockId: 'b1',
|
||||
success: true,
|
||||
output: { content: 'value=publisher-secret' },
|
||||
},
|
||||
],
|
||||
}
|
||||
})
|
||||
const parentRegistry = new ResolvedSecretTraceRegistry()
|
||||
const result = await handler.execute(
|
||||
customBlockContext({ resolvedSecretTraceRegistry: parentRegistry }),
|
||||
customBlock(),
|
||||
{}
|
||||
)
|
||||
|
||||
expect(result).toMatchObject({ answer: 'value=publisher-secret', success: true })
|
||||
expect(parentRegistry.getActiveMatches()).toEqual([
|
||||
{
|
||||
plaintext: 'publisher-secret',
|
||||
replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT,
|
||||
},
|
||||
])
|
||||
expect(mockSetResolvedSecretTraceRegistry).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('does not duplicate ids across repeated invocations', async () => {
|
||||
const ctx = customBlockContext()
|
||||
await handler.execute(ctx, customBlock(), {})
|
||||
@@ -1179,9 +1242,19 @@ describe('WorkflowBlockHandler', () => {
|
||||
})
|
||||
|
||||
it('completes the child session and disposes the cancellation bridge', async () => {
|
||||
const executionState = {
|
||||
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
|
||||
}
|
||||
mockExecutorExecute.mockResolvedValue({
|
||||
success: true,
|
||||
output: { data: 'ok' },
|
||||
executionState,
|
||||
})
|
||||
|
||||
await handler.execute(customBlockContext(), customBlock(), {})
|
||||
|
||||
expect(mockSafeComplete).toHaveBeenCalledTimes(1)
|
||||
expect(mockSafeComplete).toHaveBeenCalledWith(expect.objectContaining({ executionState }))
|
||||
expect(mockSafeCompleteWithError).not.toHaveBeenCalled()
|
||||
expect(mockDispose).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
@@ -1189,15 +1262,22 @@ describe('WorkflowBlockHandler', () => {
|
||||
it('records a cancelled child through the cancellation path', async () => {
|
||||
// Production shape: the engine reports cancellation as `success: false`
|
||||
// plus `status: 'cancelled'` on the ExecutionResult (never on metadata).
|
||||
const executionState = {
|
||||
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
|
||||
}
|
||||
mockExecutorExecute.mockResolvedValue({
|
||||
success: false,
|
||||
output: {},
|
||||
status: 'cancelled',
|
||||
executionState,
|
||||
})
|
||||
|
||||
await handler.execute(customBlockContext(), customBlock(), {}).catch(() => {})
|
||||
|
||||
expect(mockSafeCompleteWithCancellation).toHaveBeenCalledTimes(1)
|
||||
expect(mockSafeCompleteWithCancellation).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ executionState })
|
||||
)
|
||||
expect(mockSafeComplete).not.toHaveBeenCalled()
|
||||
// Already finalized as cancelled — must not be re-completed as an error.
|
||||
expect(mockSafeCompleteWithError).not.toHaveBeenCalled()
|
||||
@@ -1407,12 +1487,14 @@ describe('WorkflowBlockHandler', () => {
|
||||
})
|
||||
|
||||
it('leaves regular workflow blocks entirely alone', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry()
|
||||
const ctx = {
|
||||
...mockContext,
|
||||
workspaceId: 'workspace-1',
|
||||
executionId: 'parent-execution-id',
|
||||
onBlockStart: vi.fn(),
|
||||
onStream: vi.fn(),
|
||||
resolvedSecretTraceRegistry: registry,
|
||||
} as unknown as ExecutionContext
|
||||
mockFetch.mockResolvedValue({
|
||||
ok: true,
|
||||
@@ -1431,6 +1513,7 @@ describe('WorkflowBlockHandler', () => {
|
||||
expect(loggingSessionArgs).toHaveLength(0)
|
||||
const extensions = executorOptions[0].contextExtensions
|
||||
expect(extensions.executionId).toBe('parent-execution-id')
|
||||
expect(extensions.resolvedSecretTraceRegistry).toBe(registry)
|
||||
expect(extensions.onStream).toBe(ctx.onStream)
|
||||
expect(extensions.childWorkflowContext).toBeDefined()
|
||||
})
|
||||
|
||||
@@ -45,6 +45,7 @@ import { buildAPIUrl, buildAuthHeaders } from '@/executor/utils/http'
|
||||
import { getIterationContext } from '@/executor/utils/iteration-context'
|
||||
import { parseJSON } from '@/executor/utils/json'
|
||||
import { lazyCleanupInputMapping } from '@/executor/utils/lazy-cleanup'
|
||||
import { createResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { isRunMetadataEnabled, resolveExecutorStartBlock } from '@/executor/utils/start-block'
|
||||
import { Serializer } from '@/serializer'
|
||||
import type { SerializedBlock } from '@/serializer/types'
|
||||
@@ -269,6 +270,7 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
/** Set for custom blocks only: the child's own execution id / log row. */
|
||||
let childExecutionId: string | undefined
|
||||
let childSession: LoggingSession | undefined
|
||||
let childResolvedSecretTraceRegistry = ctx.resolvedSecretTraceRegistry
|
||||
let childSessionStarted = false
|
||||
/** Set once the child's session reached a terminal state, so the catch doesn't re-complete it. */
|
||||
let childSessionFinalized = false
|
||||
@@ -431,6 +433,24 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
...ownerEnv.personalEncrypted,
|
||||
...ownerEnv.workspaceEncrypted,
|
||||
}
|
||||
childResolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: ownerEnv.personalEncrypted,
|
||||
workspaceEncrypted: ownerEnv.workspaceEncrypted,
|
||||
personalDecrypted: ownerEnv.personalDecrypted,
|
||||
workspaceDecrypted: ownerEnv.workspaceDecrypted,
|
||||
decryptionFailures: ownerEnv.decryptionFailures,
|
||||
scope: { userId: loadUserId, workspaceId: sourceWorkspaceId },
|
||||
})
|
||||
if (ctx.resolvedSecretTraceRegistry) {
|
||||
const crossingProvenance = ctx.resolvedSecretTraceRegistry.exportProvenanceForValue(
|
||||
childWorkflowInput,
|
||||
{ anonymous: true }
|
||||
)
|
||||
await childResolvedSecretTraceRegistry.importProvenance(crossingProvenance, {
|
||||
trusted: true,
|
||||
anonymous: true,
|
||||
})
|
||||
}
|
||||
// Custom-block children authenticate internal tool calls as the source
|
||||
// owner in the source workspace, so the consumer's snapshot would fail
|
||||
// the internal routes' actor/workspace scope match. Resolve the
|
||||
@@ -459,6 +479,7 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
// child is part of that same logical run and must not add a second.
|
||||
{ baseExecutionCharge: 0 }
|
||||
)
|
||||
childSession.setResolvedSecretTraceRegistry(childResolvedSecretTraceRegistry)
|
||||
const correlation = buildCustomBlockCorrelation({
|
||||
invokerExecutionId: ctx.executionId,
|
||||
invokerRequestId: ctx.metadata.requestId,
|
||||
@@ -509,6 +530,12 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
for (const id of [ctx.executionId, childExecutionId]) {
|
||||
if (id && !sharedLargeValueIds.includes(id)) sharedLargeValueIds.push(id)
|
||||
}
|
||||
childSession.setTraceLargeValueAccess({
|
||||
largeValueExecutionIds: sharedLargeValueIds,
|
||||
largeValueKeys: ctx.largeValueKeys,
|
||||
fileKeys: ctx.fileKeys,
|
||||
allowLargeValueWorkflowScope: ctx.allowLargeValueWorkflowScope,
|
||||
})
|
||||
}
|
||||
|
||||
// Trusted run metadata for the child's Start block. Every field describes
|
||||
@@ -570,6 +597,7 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
// internal tool calls (knowledge, guardrails, MCP, Mothership) can
|
||||
// attach the required billing attribution header.
|
||||
billingAttribution: childBillingAttribution,
|
||||
resolvedSecretTraceRegistry: childResolvedSecretTraceRegistry,
|
||||
// Fall back to the inherited metadata so a toggle-off intermediate
|
||||
// child still carries the trusted identity chain to deeper children.
|
||||
startRunMetadata: childStartRunMetadata ?? inherited,
|
||||
@@ -672,7 +700,18 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
// failures surface identically; we just reshape the successful output. The
|
||||
// child's spend is billed by its own session, not rolled onto this block.
|
||||
if (isCustomBlock) {
|
||||
return this.projectCustomBlockOutput(executionResult, exposedOutputs)
|
||||
const exposedOutput = this.projectCustomBlockOutput(executionResult, exposedOutputs)
|
||||
if (ctx.resolvedSecretTraceRegistry && childResolvedSecretTraceRegistry) {
|
||||
const crossingProvenance = childResolvedSecretTraceRegistry.exportProvenanceForValue(
|
||||
exposedOutput,
|
||||
{ anonymous: true }
|
||||
)
|
||||
await ctx.resolvedSecretTraceRegistry.importProvenance(crossingProvenance, {
|
||||
trusted: true,
|
||||
anonymous: true,
|
||||
})
|
||||
}
|
||||
return exposedOutput
|
||||
}
|
||||
|
||||
return mappedResult
|
||||
@@ -766,7 +805,12 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
// Cancellation lives on `ExecutionResult.status` — `ExecutionMetadata.status`
|
||||
// has no 'cancelled' member, so reading it there never matches.
|
||||
if (executionResult.status === 'cancelled') {
|
||||
await session.safeCompleteWithCancellation({ endedAt, totalDurationMs, traceSpans })
|
||||
await session.safeCompleteWithCancellation({
|
||||
endedAt,
|
||||
totalDurationMs,
|
||||
traceSpans,
|
||||
executionState: executionResult.executionState,
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
@@ -776,6 +820,7 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
finalOutput: executionResult.output ?? {},
|
||||
traceSpans,
|
||||
workflowInput,
|
||||
executionState: executionResult.executionState,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -792,6 +837,7 @@ export class WorkflowBlockHandler implements BlockHandler {
|
||||
totalDurationMs: totalDuration ?? 0,
|
||||
error: { message: normalized.message, stackTrace: normalized.stack },
|
||||
traceSpans,
|
||||
executionState: executionResult?.executionState,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ import type {
|
||||
PiiBlockOutputRedaction,
|
||||
SerializableExecutionState,
|
||||
} from '@/executor/execution/types'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import type { RunFromBlockContext } from '@/executor/utils/run-from-block'
|
||||
import type { AgentStreamSink, UnsubscribeAgentStreamSink } from '@/providers/stream-events'
|
||||
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
|
||||
@@ -354,6 +355,7 @@ export interface ExecutionContext {
|
||||
/** Trusted run metadata for the Start block's "Add run metadata" toggle. */
|
||||
startRunMetadata?: StartBlockRunMetadata
|
||||
environmentVariables: Record<string, string>
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
workflowVariables?: Record<string, any>
|
||||
|
||||
decisions: {
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { resolveEnvVarReferences } from '@/executor/utils/reference-validation'
|
||||
|
||||
describe('resolveEnvVarReferences provenance', () => {
|
||||
it('reports each successful exact, embedded, and deep substitution', () => {
|
||||
const onResolved = vi.fn()
|
||||
|
||||
const result = resolveEnvVarReferences(
|
||||
{
|
||||
exact: '{{TOKEN}}',
|
||||
embedded: 'Bearer {{TOKEN}} and {{OTHER}}',
|
||||
missing: '{{MISSING}}',
|
||||
},
|
||||
{ TOKEN: 'secret', OTHER: 'other-secret' },
|
||||
{ deep: true, onResolved }
|
||||
)
|
||||
|
||||
expect(result).toEqual({
|
||||
exact: 'secret',
|
||||
embedded: 'Bearer secret and other-secret',
|
||||
missing: '{{MISSING}}',
|
||||
})
|
||||
expect(onResolved.mock.calls).toEqual([
|
||||
['TOKEN', 'secret'],
|
||||
['TOKEN', 'secret'],
|
||||
['OTHER', 'other-secret'],
|
||||
])
|
||||
})
|
||||
|
||||
it('does not report disabled embedded or missing references', () => {
|
||||
const onResolved = vi.fn()
|
||||
expect(
|
||||
resolveEnvVarReferences(
|
||||
'prefix {{TOKEN}} {{MISSING}}',
|
||||
{ TOKEN: 'secret' },
|
||||
{
|
||||
allowEmbedded: false,
|
||||
onResolved,
|
||||
}
|
||||
)
|
||||
).toBe('prefix {{TOKEN}} {{MISSING}}')
|
||||
expect(onResolved).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -26,6 +26,7 @@ export interface EnvVarResolveOptions {
|
||||
onMissing?: 'keep' | 'throw' | 'empty'
|
||||
deep?: boolean
|
||||
missingKeys?: string[]
|
||||
onResolved?: (name: string, value: string) => void
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -37,7 +38,9 @@ export interface EnvVarResolveOptions {
|
||||
* - `onMissing: 'keep'` - Unknown patterns pass through (e.g., Grafana's `{{instance}}`)
|
||||
* - `deep: false` - Only processes strings by default; set `true` for nested objects
|
||||
*/
|
||||
export const ENV_VAR_RESOLVE_DEFAULTS: Required<Omit<EnvVarResolveOptions, 'missingKeys'>> = {
|
||||
export const ENV_VAR_RESOLVE_DEFAULTS: Required<
|
||||
Omit<EnvVarResolveOptions, 'missingKeys' | 'onResolved'>
|
||||
> = {
|
||||
resolveExactMatch: true,
|
||||
allowEmbedded: true,
|
||||
trimKeys: true,
|
||||
@@ -70,7 +73,10 @@ export function resolveEnvVarReferences(
|
||||
if (exactMatch) {
|
||||
const envKey = trimKeys ? exactMatch[1].trim() : exactMatch[1]
|
||||
const envValue = envVars[envKey]
|
||||
if (envValue !== undefined) return envValue
|
||||
if (envValue !== undefined) {
|
||||
if (Object.hasOwn(envVars, envKey)) options.onResolved?.(envKey, envValue)
|
||||
return envValue
|
||||
}
|
||||
if (options.missingKeys) options.missingKeys.push(envKey)
|
||||
if (onMissing === 'throw') {
|
||||
throw new Error(`Environment variable "${envKey}" was not found`)
|
||||
@@ -88,7 +94,10 @@ export function resolveEnvVarReferences(
|
||||
return value.replace(envVarPattern, (match, varName) => {
|
||||
const envKey = trimKeys ? String(varName).trim() : String(varName)
|
||||
const envValue = envVars[envKey]
|
||||
if (envValue !== undefined) return envValue
|
||||
if (envValue !== undefined) {
|
||||
if (Object.hasOwn(envVars, envKey)) options.onResolved?.(envKey, envValue)
|
||||
return envValue
|
||||
}
|
||||
if (options.missingKeys) options.missingKeys.push(envKey)
|
||||
if (onMissing === 'throw') {
|
||||
throw new Error(`Environment variable "${envKey}" was not found`)
|
||||
|
||||
@@ -0,0 +1,655 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockDecryptSecret } = vi.hoisted(() => ({
|
||||
mockDecryptSecret: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/security/encryption', () => ({
|
||||
decryptSecret: mockDecryptSecret,
|
||||
}))
|
||||
|
||||
import {
|
||||
ANONYMOUS_SECRET_TRACE_REPLACEMENT,
|
||||
createResolvedSecretTraceRegistry,
|
||||
isResolvedSecretTraceProvenanceV1,
|
||||
ResolvedSecretTraceProvenanceAccumulator,
|
||||
type ResolvedSecretTraceProvenanceV1,
|
||||
ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
describe('ResolvedSecretTraceProvenanceAccumulator', () => {
|
||||
const scope = { userId: 'user-1', workspaceId: 'workspace-1' }
|
||||
|
||||
it('unions cold, warm, and retry reports while complete', () => {
|
||||
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
|
||||
|
||||
expect(
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' }],
|
||||
scope,
|
||||
})
|
||||
).toBe(true)
|
||||
expect(
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' }],
|
||||
scope,
|
||||
})
|
||||
).toBe(true)
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' }],
|
||||
scope,
|
||||
})
|
||||
|
||||
expect(accumulator.exportProvenance()).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' },
|
||||
{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' },
|
||||
],
|
||||
scope,
|
||||
})
|
||||
})
|
||||
|
||||
it('discards accumulated and future entries once completeness is lost', () => {
|
||||
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
|
||||
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'OLD_TOKEN', encryptedValue: 'encrypted-v1' }],
|
||||
scope,
|
||||
})
|
||||
expect(
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope,
|
||||
})
|
||||
).toBe(true)
|
||||
expect(
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'NEW_TOKEN', encryptedValue: 'encrypted-v2' }],
|
||||
scope,
|
||||
})
|
||||
).toBe(true)
|
||||
|
||||
expect(accumulator.exportProvenance()).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope,
|
||||
})
|
||||
})
|
||||
|
||||
it('discards mismatched-scope reports and marks them incomplete', () => {
|
||||
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
|
||||
|
||||
expect(
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-value' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-2' },
|
||||
})
|
||||
).toBe(true)
|
||||
|
||||
expect(accumulator.exportProvenance()).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope,
|
||||
})
|
||||
})
|
||||
|
||||
it('fails closed for malformed reports and terminal incompleteness', () => {
|
||||
const accumulator = new ResolvedSecretTraceProvenanceAccumulator(scope)
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-value' }],
|
||||
scope,
|
||||
})
|
||||
|
||||
expect(accumulator.record({ version: 1 })).toBe(false)
|
||||
expect(accumulator.exportProvenance()).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope,
|
||||
})
|
||||
|
||||
accumulator.record({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'encrypted-value' }],
|
||||
scope,
|
||||
})
|
||||
accumulator.markIncomplete()
|
||||
expect(accumulator.exportProvenance().entries).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('ResolvedSecretTraceRegistry', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockDecryptSecret.mockImplementation(async (encryptedValue: string) => ({
|
||||
decrypted: `decrypted:${encryptedValue}`,
|
||||
}))
|
||||
})
|
||||
|
||||
it('starts with an inert catalog and activates only an exact successful resolution', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'API_KEY', plaintext: 'secret-value', encryptedValue: 'encrypted-value' },
|
||||
])
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([])
|
||||
expect(registry.recordResolved('API_KEY', 'wrong-value')).toBe(false)
|
||||
expect(registry.recordResolved('MISSING', 'secret-value')).toBe(false)
|
||||
expect(registry.getActiveMatches()).toEqual([])
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
|
||||
expect(registry.recordResolved('API_KEY', 'secret-value')).toBe(true)
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'secret-value', replacement: '{{API_KEY}}' },
|
||||
])
|
||||
})
|
||||
|
||||
it('uses the workspace catalog entry when personal and workspace names conflict', async () => {
|
||||
const registry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: { SHARED: 'personal-encrypted' },
|
||||
workspaceEncrypted: { SHARED: 'workspace-encrypted' },
|
||||
personalDecrypted: { SHARED: 'personal-secret' },
|
||||
workspaceDecrypted: { SHARED: 'workspace-secret' },
|
||||
})
|
||||
|
||||
expect(registry.recordResolved('SHARED', 'workspace-secret')).toBe(true)
|
||||
expect(registry.exportProvenance()).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'SHARED', encryptedValue: 'workspace-encrypted' }],
|
||||
})
|
||||
})
|
||||
|
||||
it('ignores empty decryption failures but fails closed for a resolved value outside the catalog', async () => {
|
||||
const registry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: { FAILED: 'failed-ciphertext' },
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: { FAILED: '', DECRYPTED_ONLY: 'not-catalogued' },
|
||||
workspaceDecrypted: {},
|
||||
decryptionFailures: ['FAILED'],
|
||||
})
|
||||
|
||||
expect(registry.isComplete()).toBe(true)
|
||||
expect(registry.recordResolved('FAILED', '')).toBe(false)
|
||||
expect(registry.isComplete()).toBe(true)
|
||||
expect(registry.recordResolved('DECRYPTED_ONLY', 'not-catalogued')).toBe(false)
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
})
|
||||
|
||||
it('keeps a successful workspace override when the shadowed personal value failed', async () => {
|
||||
const registry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: { SHARED: 'broken-personal-ciphertext' },
|
||||
workspaceEncrypted: { SHARED: 'workspace-ciphertext' },
|
||||
personalDecrypted: { SHARED: '' },
|
||||
workspaceDecrypted: { SHARED: 'workspace-secret' },
|
||||
decryptionFailures: ['SHARED'],
|
||||
})
|
||||
|
||||
expect(registry.recordResolved('SHARED', 'workspace-secret')).toBe(true)
|
||||
expect(registry.exportProvenance().entries).toEqual([
|
||||
{ name: 'SHARED', encryptedValue: 'workspace-ciphertext' },
|
||||
])
|
||||
})
|
||||
|
||||
it('exports encrypted active provenance without plaintext', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'raw-secret', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
registry.recordResolved('TOKEN', 'raw-secret')
|
||||
|
||||
const serialized = JSON.stringify(registry.exportProvenance())
|
||||
expect(serialized).toContain('ciphertext')
|
||||
expect(serialized).not.toContain('raw-secret')
|
||||
})
|
||||
|
||||
it('restores old encrypted values alongside the current catalog after rotation', async () => {
|
||||
mockDecryptSecret.mockResolvedValueOnce({ decrypted: 'old-secret' })
|
||||
const oldProvenance: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'old-ciphertext' }],
|
||||
}
|
||||
const registry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: { TOKEN: 'new-ciphertext' },
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: { TOKEN: 'new-secret' },
|
||||
workspaceDecrypted: {},
|
||||
restoredProvenance: oldProvenance,
|
||||
restoreTrusted: true,
|
||||
requireRestoredProvenance: true,
|
||||
})
|
||||
|
||||
registry.recordResolved('TOKEN', 'new-secret')
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'new-secret', replacement: '{{TOKEN}}' },
|
||||
{ plaintext: 'old-secret', replacement: '{{TOKEN}}' },
|
||||
])
|
||||
expect(registry.exportProvenance().entries).toEqual([
|
||||
{ name: 'TOKEN', encryptedValue: 'new-ciphertext' },
|
||||
{ name: 'TOKEN', encryptedValue: 'old-ciphertext' },
|
||||
])
|
||||
})
|
||||
|
||||
it('marks untrusted, missing, malformed, and undecryptable restoration incomplete', async () => {
|
||||
const provenance: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
|
||||
}
|
||||
const untrusted = new ResolvedSecretTraceRegistry()
|
||||
expect(await untrusted.importProvenance(provenance, { trusted: false })).toBe(false)
|
||||
expect(untrusted.isComplete()).toBe(false)
|
||||
expect(mockDecryptSecret).not.toHaveBeenCalled()
|
||||
|
||||
const missing = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: {},
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: {},
|
||||
workspaceDecrypted: {},
|
||||
requireRestoredProvenance: true,
|
||||
restoreTrusted: true,
|
||||
})
|
||||
expect(missing.isComplete()).toBe(false)
|
||||
|
||||
const malformed = new ResolvedSecretTraceRegistry()
|
||||
expect(await malformed.importProvenance({ version: 1 }, { trusted: true })).toBe(false)
|
||||
expect(malformed.isComplete()).toBe(false)
|
||||
|
||||
mockDecryptSecret.mockRejectedValueOnce(new Error('cannot decrypt'))
|
||||
const undecryptable = new ResolvedSecretTraceRegistry()
|
||||
expect(await undecryptable.importProvenance(provenance, { trusted: true })).toBe(false)
|
||||
expect(undecryptable.isComplete()).toBe(false)
|
||||
})
|
||||
|
||||
it('uses anonymous replacements for cross-scope provenance', async () => {
|
||||
mockDecryptSecret.mockResolvedValueOnce({ decrypted: 'publisher-secret' })
|
||||
const registry = new ResolvedSecretTraceRegistry()
|
||||
await registry.importProvenance(
|
||||
{
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'PUBLISHER_TOKEN', encryptedValue: 'publisher-ciphertext' }],
|
||||
},
|
||||
{ trusted: true, anonymous: true }
|
||||
)
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'publisher-secret', replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT },
|
||||
])
|
||||
expect(registry.exportProvenance().entries).toEqual([
|
||||
{ encryptedValue: 'publisher-ciphertext' },
|
||||
])
|
||||
})
|
||||
|
||||
it('preserves labels only when imported provenance has the same complete scope', async () => {
|
||||
const provenance: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
}
|
||||
const sameScope = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
const mismatchedScope = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-2',
|
||||
})
|
||||
const missingReceiverScope = new ResolvedSecretTraceRegistry()
|
||||
const missingSourceScope = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
|
||||
expect(await sameScope.importProvenance(provenance, { trusted: true })).toBe(true)
|
||||
expect(await mismatchedScope.importProvenance(provenance, { trusted: true })).toBe(true)
|
||||
expect(await missingReceiverScope.importProvenance(provenance, { trusted: true })).toBe(true)
|
||||
expect(
|
||||
await missingSourceScope.importProvenance(
|
||||
{ version: 1, complete: true, entries: provenance.entries },
|
||||
{ trusted: true }
|
||||
)
|
||||
).toBe(true)
|
||||
|
||||
expect(sameScope.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'decrypted:ciphertext', replacement: '{{TOKEN}}' },
|
||||
])
|
||||
for (const registry of [mismatchedScope, missingReceiverScope, missingSourceScope]) {
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{
|
||||
plaintext: 'decrypted:ciphertext',
|
||||
replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT,
|
||||
},
|
||||
])
|
||||
}
|
||||
})
|
||||
|
||||
it('imports all same-scope provenance across an in-process boundary', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
const provenance: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ name: 'PRESENT', encryptedValue: 'present-ciphertext' },
|
||||
{ name: 'DORMANT_IN_OUTPUT', encryptedValue: 'other-ciphertext' },
|
||||
],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
}
|
||||
|
||||
expect(
|
||||
await registry.importCrossingProvenance(
|
||||
provenance,
|
||||
{ output: 'decrypted:present-ciphertext' },
|
||||
{ trusted: true }
|
||||
)
|
||||
).toBe(true)
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'decrypted:present-ciphertext', replacement: '{{PRESENT}}' },
|
||||
{ plaintext: 'decrypted:other-ciphertext', replacement: '{{DORMANT_IN_OUTPUT}}' },
|
||||
])
|
||||
})
|
||||
|
||||
it('filters and anonymizes provenance crossing from another scope', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
const provenance: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ name: 'PRESENT', encryptedValue: 'present-ciphertext' },
|
||||
{ name: 'ABSENT', encryptedValue: 'absent-ciphertext' },
|
||||
],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-2' },
|
||||
}
|
||||
|
||||
expect(
|
||||
await registry.importCrossingProvenance(
|
||||
provenance,
|
||||
{ output: 'decrypted:present-ciphertext' },
|
||||
{ trusted: true }
|
||||
)
|
||||
).toBe(true)
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{
|
||||
plaintext: 'decrypted:present-ciphertext',
|
||||
replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT,
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
it('exports only active secrets whose exact literals cross a value boundary', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'PRESENT', plaintext: 'present-secret', encryptedValue: 'present-ciphertext' },
|
||||
{ name: 'ABSENT', plaintext: 'absent-secret', encryptedValue: 'absent-ciphertext' },
|
||||
{ name: 'UNUSED', plaintext: 'unused-secret', encryptedValue: 'unused-ciphertext' },
|
||||
])
|
||||
registry.recordResolved('PRESENT', 'present-secret')
|
||||
registry.recordResolved('ABSENT', 'absent-secret')
|
||||
|
||||
const provenance = registry.exportProvenanceForValue(
|
||||
{ nested: [{ 'key-present-secret': new Error('failed with present-secret') }] },
|
||||
{ anonymous: true }
|
||||
)
|
||||
|
||||
expect(provenance).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ encryptedValue: 'present-ciphertext' }],
|
||||
})
|
||||
})
|
||||
|
||||
it('exports active numeric, boolean, and null literals crossing a value boundary', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'NUMBER', plaintext: '1234', encryptedValue: 'number-ciphertext' },
|
||||
{ name: 'BOOLEAN', plaintext: 'false', encryptedValue: 'boolean-ciphertext' },
|
||||
{ name: 'NULL', plaintext: 'null', encryptedValue: 'null-ciphertext' },
|
||||
{ name: 'ABSENT', plaintext: '5678', encryptedValue: 'absent-ciphertext' },
|
||||
])
|
||||
registry.recordResolved('NUMBER', '1234')
|
||||
registry.recordResolved('BOOLEAN', 'false')
|
||||
registry.recordResolved('NULL', 'null')
|
||||
registry.recordResolved('ABSENT', '5678')
|
||||
|
||||
expect(
|
||||
registry.exportProvenanceForValue(
|
||||
{ number: 1234, boolean: false, nullable: null },
|
||||
{ anonymous: true }
|
||||
)
|
||||
).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ encryptedValue: 'boolean-ciphertext' },
|
||||
{ encryptedValue: 'null-ciphertext' },
|
||||
{ encryptedValue: 'number-ciphertext' },
|
||||
],
|
||||
})
|
||||
})
|
||||
|
||||
it('marks a bounded cross-boundary scan incomplete when an enumerable accessor is opaque', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'secret', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
registry.recordResolved('TOKEN', 'secret')
|
||||
const value = {}
|
||||
Object.defineProperty(value, 'opaque', {
|
||||
enumerable: true,
|
||||
get: () => 'secret',
|
||||
})
|
||||
|
||||
expect(registry.exportProvenanceForValue(value, { anonymous: true })).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
})
|
||||
})
|
||||
|
||||
it('does not claim a complete cross-boundary scan for opaque large-value refs', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'secret', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
registry.recordResolved('TOKEN', 'secret')
|
||||
|
||||
expect(
|
||||
registry.exportProvenanceForValue(
|
||||
{
|
||||
__simLargeValueRef: true,
|
||||
version: 1,
|
||||
id: 'lv_ABCDEFGHIJKL',
|
||||
kind: 'object',
|
||||
size: 1024,
|
||||
},
|
||||
{ anonymous: true }
|
||||
)
|
||||
).toEqual({ version: 1, complete: false, entries: [] })
|
||||
})
|
||||
|
||||
it('does not snapshot or enqueue an entire wide crossing object', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'needle-value', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
registry.recordResolved('TOKEN', 'needle-value')
|
||||
const wideValue: Record<string, string> = {}
|
||||
for (let index = 0; index < 30_000; index++) {
|
||||
wideValue[`field_${index}`] = 'safe-value'
|
||||
}
|
||||
const descriptorSnapshotSpy = vi.spyOn(Object, 'getOwnPropertyDescriptors')
|
||||
const provenance = registry.exportProvenanceForValue(wideValue, { anonymous: true })
|
||||
const descriptorSnapshotCalls = descriptorSnapshotSpy.mock.calls.length
|
||||
descriptorSnapshotSpy.mockRestore()
|
||||
|
||||
expect(provenance).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
})
|
||||
expect(descriptorSnapshotCalls).toBe(0)
|
||||
})
|
||||
|
||||
it('handles duplicate and empty values deterministically', () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'Z_TOKEN', plaintext: 'same', encryptedValue: 'z-ciphertext' },
|
||||
{ name: 'A_TOKEN', plaintext: 'same', encryptedValue: 'a-ciphertext' },
|
||||
{ name: 'EMPTY', plaintext: '', encryptedValue: 'empty-ciphertext' },
|
||||
{ name: 'A', plaintext: 'A', encryptedValue: 'short-ciphertext' },
|
||||
])
|
||||
registry.recordResolved('Z_TOKEN', 'same')
|
||||
registry.recordResolved('A_TOKEN', 'same')
|
||||
registry.recordResolved('EMPTY', '')
|
||||
|
||||
expect(registry.getActiveMatches()).toEqual([{ plaintext: 'same', replacement: '{{A_TOKEN}}' }])
|
||||
|
||||
registry.recordResolved('A', 'A')
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'same', replacement: '{{A_TOKEN}}' },
|
||||
{ plaintext: 'A', replacement: '{{A}}' },
|
||||
])
|
||||
})
|
||||
|
||||
it('marks the registry incomplete when active provenance exceeds its hard cap', () => {
|
||||
const entries = Array.from({ length: 10_001 }, (_, index) => ({
|
||||
name: `SECRET_${index}`,
|
||||
plaintext: `value-${index}`,
|
||||
encryptedValue: `ciphertext-${index}`,
|
||||
}))
|
||||
const registry = new ResolvedSecretTraceRegistry(entries)
|
||||
|
||||
for (const entry of entries) {
|
||||
registry.recordResolved(entry.name, entry.plaintext)
|
||||
}
|
||||
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
expect(registry.exportProvenance().entries).toEqual([])
|
||||
})
|
||||
|
||||
it('bounds provenance by serialized JSON bytes including control-character escapes', () => {
|
||||
const encryptedValue = '\u0000'.repeat(1_400_000)
|
||||
const provenance: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'TOKEN', encryptedValue }],
|
||||
}
|
||||
|
||||
expect(Buffer.byteLength(encryptedValue, 'utf8')).toBeLessThan(8 * 1024 * 1024)
|
||||
expect(Buffer.byteLength(JSON.stringify(provenance), 'utf8')).toBeGreaterThan(8 * 1024 * 1024)
|
||||
expect(isResolvedSecretTraceProvenanceV1(provenance)).toBe(false)
|
||||
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'secret', encryptedValue },
|
||||
])
|
||||
expect(registry.recordResolved('TOKEN', 'secret')).toBe(true)
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
expect(registry.exportProvenance().entries).toEqual([])
|
||||
})
|
||||
|
||||
it('rejects incomplete provenance that still carries entries', () => {
|
||||
expect(
|
||||
isResolvedSecretTraceProvenanceV1({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [{ name: 'TOKEN', encryptedValue: 'ciphertext' }],
|
||||
})
|
||||
).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects non-canonical provenance fields before applying the serialized-size bound', () => {
|
||||
const entry = { name: 'TOKEN', encryptedValue: 'ciphertext' }
|
||||
const scope = { userId: 'user-1', workspaceId: 'workspace-1' }
|
||||
|
||||
expect(
|
||||
isResolvedSecretTraceProvenanceV1({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [entry],
|
||||
scope,
|
||||
extra: 'not-transported',
|
||||
})
|
||||
).toBe(false)
|
||||
expect(
|
||||
isResolvedSecretTraceProvenanceV1({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ ...entry, extra: 'not-transported' }],
|
||||
scope,
|
||||
})
|
||||
).toBe(false)
|
||||
expect(
|
||||
isResolvedSecretTraceProvenanceV1({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [entry],
|
||||
scope: { ...scope, extra: 'not-transported' },
|
||||
})
|
||||
).toBe(false)
|
||||
|
||||
const entries = [entry]
|
||||
Object.assign(entries, { extra: 'not-transported' })
|
||||
expect(isResolvedSecretTraceProvenanceV1({ version: 1, complete: true, entries, scope })).toBe(
|
||||
false
|
||||
)
|
||||
})
|
||||
|
||||
it('stops consuming a dormant catalog when its entry cap is exceeded', () => {
|
||||
let yieldedEntries = 0
|
||||
function* catalogEntries() {
|
||||
for (let index = 0; index < 20_000; index++) {
|
||||
yieldedEntries++
|
||||
yield {
|
||||
name: `SECRET_${index}`,
|
||||
plaintext: `value-${index}`,
|
||||
encryptedValue: `ciphertext-${index}`,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const registry = new ResolvedSecretTraceRegistry(catalogEntries())
|
||||
|
||||
expect(yieldedEntries).toBe(10_001)
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
expect(registry.exportProvenance().entries).toEqual([])
|
||||
})
|
||||
|
||||
it('marks an oversized dormant catalog value incomplete without retaining it', () => {
|
||||
const oversizedPlaintext = 'x'.repeat(8 * 1024 * 1024)
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{
|
||||
name: 'OVERSIZED',
|
||||
plaintext: oversizedPlaintext,
|
||||
encryptedValue: 'ciphertext',
|
||||
},
|
||||
])
|
||||
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
expect(registry.recordResolved('OVERSIZED', oversizedPlaintext)).toBe(false)
|
||||
expect(registry.getActiveMatches()).toEqual([])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,784 @@
|
||||
import { decryptSecret } from '@/lib/core/security/encryption'
|
||||
import { isLargeArrayManifest } from '@/lib/execution/payloads/large-array-manifest-metadata'
|
||||
import { isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
|
||||
|
||||
export const ANONYMOUS_SECRET_TRACE_REPLACEMENT = '[REDACTED_SECRET]'
|
||||
|
||||
const MAX_PROVENANCE_ENTRIES = 10_000
|
||||
const MAX_SERIALIZED_PROVENANCE_BYTES = 8 * 1024 * 1024
|
||||
const MAX_TRACE_CATALOG_ENTRIES = MAX_PROVENANCE_ENTRIES
|
||||
const MAX_TRACE_CATALOG_BYTES = 8 * 1024 * 1024
|
||||
const MAX_PROVENANCE_FILTER_NODES = 50_000
|
||||
const MAX_PROVENANCE_FILTER_CHARACTERS = 16 * 1024 * 1024
|
||||
const MAX_PROVENANCE_FILTER_COMPARISONS = 1_000_000
|
||||
const ERROR_CONTENT_PROPERTY_NAMES = ['name', 'message', 'stack', 'cause', 'errors'] as const
|
||||
const PROVENANCE_PROPERTY_NAMES = new Set(['version', 'complete', 'entries', 'scope'])
|
||||
const PROVENANCE_ENTRY_PROPERTY_NAMES = new Set(['encryptedValue', 'name'])
|
||||
const PROVENANCE_SCOPE_PROPERTY_NAMES = new Set(['userId', 'workspaceId'])
|
||||
|
||||
export interface ResolvedSecretTraceCatalogEntry {
|
||||
name: string
|
||||
plaintext: string
|
||||
encryptedValue: string
|
||||
}
|
||||
|
||||
export interface ResolvedSecretTraceMatch {
|
||||
plaintext: string
|
||||
replacement: string
|
||||
}
|
||||
|
||||
export interface ResolvedSecretTraceProvenanceEntryV1 {
|
||||
encryptedValue: string
|
||||
name?: string
|
||||
}
|
||||
|
||||
export interface ResolvedSecretTraceScopeV1 {
|
||||
userId: string
|
||||
workspaceId?: string
|
||||
}
|
||||
|
||||
export interface ResolvedSecretTraceProvenanceV1 {
|
||||
version: 1
|
||||
complete: boolean
|
||||
entries: ResolvedSecretTraceProvenanceEntryV1[]
|
||||
scope?: ResolvedSecretTraceScopeV1
|
||||
}
|
||||
|
||||
interface ActiveSecretEntry extends ResolvedSecretTraceCatalogEntry {
|
||||
anonymous: boolean
|
||||
}
|
||||
|
||||
export interface ImportResolvedSecretTraceProvenanceOptions {
|
||||
trusted: boolean
|
||||
anonymous?: boolean
|
||||
}
|
||||
|
||||
export interface ExportResolvedSecretTraceProvenanceForValueOptions {
|
||||
anonymous?: boolean
|
||||
}
|
||||
|
||||
export interface CreateResolvedSecretTraceRegistryOptions {
|
||||
personalEncrypted: Record<string, string>
|
||||
workspaceEncrypted: Record<string, string>
|
||||
personalDecrypted: Record<string, string>
|
||||
workspaceDecrypted: Record<string, string>
|
||||
decryptionFailures?: readonly string[]
|
||||
restoredProvenance?: unknown
|
||||
restoreTrusted?: boolean
|
||||
requireRestoredProvenance?: boolean
|
||||
scope?: ResolvedSecretTraceScopeV1
|
||||
}
|
||||
|
||||
function compareStrings(left: string, right: string): number {
|
||||
if (left < right) return -1
|
||||
if (left > right) return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
function cloneProvenanceScope(scope: ResolvedSecretTraceScopeV1): ResolvedSecretTraceScopeV1 {
|
||||
return {
|
||||
userId: scope.userId,
|
||||
...(scope.workspaceId ? { workspaceId: scope.workspaceId } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
function serializedJsonStringByteSize(value: string): number {
|
||||
let byteSize = 2
|
||||
for (let index = 0; index < value.length; index++) {
|
||||
const codeUnit = value.charCodeAt(index)
|
||||
if (codeUnit === 0x22 || codeUnit === 0x5c) {
|
||||
byteSize += 2
|
||||
} else if (
|
||||
codeUnit === 0x08 ||
|
||||
codeUnit === 0x09 ||
|
||||
codeUnit === 0x0a ||
|
||||
codeUnit === 0x0c ||
|
||||
codeUnit === 0x0d
|
||||
) {
|
||||
byteSize += 2
|
||||
} else if (codeUnit <= 0x1f) {
|
||||
byteSize += 6
|
||||
} else if (codeUnit >= 0xd800 && codeUnit <= 0xdbff) {
|
||||
const nextCodeUnit = value.charCodeAt(index + 1)
|
||||
if (nextCodeUnit >= 0xdc00 && nextCodeUnit <= 0xdfff) {
|
||||
byteSize += 4
|
||||
index++
|
||||
} else {
|
||||
byteSize += 6
|
||||
}
|
||||
} else if (codeUnit >= 0xdc00 && codeUnit <= 0xdfff) {
|
||||
byteSize += 6
|
||||
} else if (codeUnit <= 0x7f) {
|
||||
byteSize++
|
||||
} else if (codeUnit <= 0x7ff) {
|
||||
byteSize += 2
|
||||
} else {
|
||||
byteSize += 3
|
||||
}
|
||||
}
|
||||
return byteSize
|
||||
}
|
||||
|
||||
function serializedProvenanceEntryByteSize(entry: ResolvedSecretTraceProvenanceEntryV1): number {
|
||||
let byteSize =
|
||||
Buffer.byteLength('{"encryptedValue":', 'utf8') +
|
||||
serializedJsonStringByteSize(entry.encryptedValue)
|
||||
if (entry.name !== undefined) {
|
||||
byteSize += Buffer.byteLength(',"name":', 'utf8') + serializedJsonStringByteSize(entry.name)
|
||||
}
|
||||
return byteSize + 1
|
||||
}
|
||||
|
||||
function serializedProvenanceEnvelopeByteSize(
|
||||
complete: boolean,
|
||||
scope: ResolvedSecretTraceScopeV1 | undefined
|
||||
): number {
|
||||
let byteSize = Buffer.byteLength(
|
||||
`{"version":1,"complete":${complete ? 'true' : 'false'},"entries":[]`,
|
||||
'utf8'
|
||||
)
|
||||
if (scope) {
|
||||
byteSize +=
|
||||
Buffer.byteLength(',"scope":{"userId":', 'utf8') + serializedJsonStringByteSize(scope.userId)
|
||||
if (scope.workspaceId !== undefined) {
|
||||
byteSize +=
|
||||
Buffer.byteLength(',"workspaceId":', 'utf8') +
|
||||
serializedJsonStringByteSize(scope.workspaceId)
|
||||
}
|
||||
byteSize++
|
||||
}
|
||||
return byteSize + 1
|
||||
}
|
||||
|
||||
function isSerializedProvenanceWithinLimit(
|
||||
complete: boolean,
|
||||
entries: readonly ResolvedSecretTraceProvenanceEntryV1[],
|
||||
scope: ResolvedSecretTraceScopeV1 | undefined
|
||||
): boolean {
|
||||
let byteSize = serializedProvenanceEnvelopeByteSize(complete, scope)
|
||||
for (let index = 0; index < entries.length; index++) {
|
||||
byteSize += serializedProvenanceEntryByteSize(entries[index]) + (index === 0 ? 0 : 1)
|
||||
if (byteSize > MAX_SERIALIZED_PROVENANCE_BYTES) return false
|
||||
}
|
||||
return byteSize <= MAX_SERIALIZED_PROVENANCE_BYTES
|
||||
}
|
||||
|
||||
function toProvenanceEntry(entry: ActiveSecretEntry): ResolvedSecretTraceProvenanceEntryV1 {
|
||||
return {
|
||||
encryptedValue: entry.encryptedValue,
|
||||
...(!entry.anonymous && entry.name ? { name: entry.name } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
function hasOwn(record: Record<string, string>, name: string): boolean {
|
||||
return Object.hasOwn(record, name)
|
||||
}
|
||||
|
||||
function isExactPlainDataRecord(
|
||||
value: unknown,
|
||||
allowedProperties: ReadonlySet<string>,
|
||||
requiredProperties: readonly string[]
|
||||
): value is Record<string, unknown> {
|
||||
if (value === null || typeof value !== 'object' || Array.isArray(value)) return false
|
||||
|
||||
try {
|
||||
const prototype = Object.getPrototypeOf(value)
|
||||
if (prototype !== Object.prototype && prototype !== null) return false
|
||||
|
||||
const properties = Reflect.ownKeys(value)
|
||||
for (const property of properties) {
|
||||
if (typeof property !== 'string' || !allowedProperties.has(property)) return false
|
||||
const descriptor = Object.getOwnPropertyDescriptor(value, property)
|
||||
if (!descriptor?.enumerable || !('value' in descriptor)) return false
|
||||
}
|
||||
return requiredProperties.every((property) => Object.hasOwn(value, property))
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
function isExactProvenanceEntriesArray(value: unknown): value is unknown[] {
|
||||
if (!Array.isArray(value)) return false
|
||||
|
||||
try {
|
||||
if (Object.getPrototypeOf(value) !== Array.prototype) return false
|
||||
const properties = Reflect.ownKeys(value)
|
||||
if (properties.length !== value.length + 1 || !properties.includes('length')) return false
|
||||
|
||||
for (let index = 0; index < value.length; index++) {
|
||||
const descriptor = Object.getOwnPropertyDescriptor(value, String(index))
|
||||
if (!descriptor?.enumerable || !('value' in descriptor)) return false
|
||||
}
|
||||
return true
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
function catalogEntryByteSize(entry: ResolvedSecretTraceCatalogEntry): number {
|
||||
return (
|
||||
Buffer.byteLength(entry.name, 'utf8') +
|
||||
Buffer.byteLength(entry.plaintext, 'utf8') +
|
||||
Buffer.byteLength(entry.encryptedValue, 'utf8')
|
||||
)
|
||||
}
|
||||
|
||||
function buildEffectiveCatalogEntry(
|
||||
options: CreateResolvedSecretTraceRegistryOptions,
|
||||
failedNames: ReadonlySet<string>,
|
||||
name: string,
|
||||
encryptedValue: string
|
||||
): ResolvedSecretTraceCatalogEntry | undefined {
|
||||
const plaintext = hasOwn(options.workspaceDecrypted, name)
|
||||
? options.workspaceDecrypted[name]
|
||||
: options.personalDecrypted[name]
|
||||
if (plaintext === undefined || (plaintext.length === 0 && failedNames.has(name))) {
|
||||
return undefined
|
||||
}
|
||||
return { name, plaintext, encryptedValue }
|
||||
}
|
||||
|
||||
function* iterateEffectiveCatalogEntries(
|
||||
options: CreateResolvedSecretTraceRegistryOptions,
|
||||
failedNames: ReadonlySet<string>
|
||||
): Generator<ResolvedSecretTraceCatalogEntry> {
|
||||
for (const name in options.personalEncrypted) {
|
||||
if (!hasOwn(options.personalEncrypted, name)) continue
|
||||
const encryptedValue = hasOwn(options.workspaceEncrypted, name)
|
||||
? options.workspaceEncrypted[name]
|
||||
: options.personalEncrypted[name]
|
||||
const entry = buildEffectiveCatalogEntry(options, failedNames, name, encryptedValue)
|
||||
if (entry) yield entry
|
||||
}
|
||||
|
||||
for (const name in options.workspaceEncrypted) {
|
||||
if (!hasOwn(options.workspaceEncrypted, name) || hasOwn(options.personalEncrypted, name)) {
|
||||
continue
|
||||
}
|
||||
const entry = buildEffectiveCatalogEntry(
|
||||
options,
|
||||
failedNames,
|
||||
name,
|
||||
options.workspaceEncrypted[name]
|
||||
)
|
||||
if (entry) yield entry
|
||||
}
|
||||
}
|
||||
|
||||
function isProvenanceEntry(value: unknown): value is ResolvedSecretTraceProvenanceEntryV1 {
|
||||
if (!isExactPlainDataRecord(value, PROVENANCE_ENTRY_PROPERTY_NAMES, ['encryptedValue'])) {
|
||||
return false
|
||||
}
|
||||
const entry = value
|
||||
return (
|
||||
typeof entry.encryptedValue === 'string' &&
|
||||
entry.encryptedValue.length > 0 &&
|
||||
(entry.name === undefined || (typeof entry.name === 'string' && entry.name.length > 0))
|
||||
)
|
||||
}
|
||||
|
||||
function isProvenanceScope(value: unknown): value is ResolvedSecretTraceScopeV1 {
|
||||
if (!isExactPlainDataRecord(value, PROVENANCE_SCOPE_PROPERTY_NAMES, ['userId'])) return false
|
||||
const scope = value
|
||||
return (
|
||||
typeof scope.userId === 'string' &&
|
||||
scope.userId.length > 0 &&
|
||||
(scope.workspaceId === undefined ||
|
||||
(typeof scope.workspaceId === 'string' && scope.workspaceId.length > 0))
|
||||
)
|
||||
}
|
||||
|
||||
function scopesMatch(
|
||||
left: ResolvedSecretTraceScopeV1 | undefined,
|
||||
right: ResolvedSecretTraceScopeV1 | undefined
|
||||
): boolean {
|
||||
return (
|
||||
(left === undefined && right === undefined) ||
|
||||
(left !== undefined &&
|
||||
right !== undefined &&
|
||||
left.userId === right.userId &&
|
||||
left.workspaceId === right.workspaceId)
|
||||
)
|
||||
}
|
||||
|
||||
export function isResolvedSecretTraceProvenanceV1(
|
||||
value: unknown
|
||||
): value is ResolvedSecretTraceProvenanceV1 {
|
||||
if (
|
||||
!isExactPlainDataRecord(value, PROVENANCE_PROPERTY_NAMES, ['version', 'complete', 'entries'])
|
||||
) {
|
||||
return false
|
||||
}
|
||||
const provenance = value
|
||||
if (
|
||||
provenance.version !== 1 ||
|
||||
typeof provenance.complete !== 'boolean' ||
|
||||
!isExactProvenanceEntriesArray(provenance.entries) ||
|
||||
provenance.entries.length > MAX_PROVENANCE_ENTRIES ||
|
||||
!provenance.entries.every(isProvenanceEntry) ||
|
||||
(!provenance.complete && provenance.entries.length > 0) ||
|
||||
(provenance.scope !== undefined && !isProvenanceScope(provenance.scope))
|
||||
) {
|
||||
return false
|
||||
}
|
||||
|
||||
return isSerializedProvenanceWithinLimit(
|
||||
provenance.complete,
|
||||
provenance.entries,
|
||||
provenance.scope
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Unions encrypted provenance reports emitted during one internal transport invocation.
|
||||
* It never decrypts values or projects trace content; the execution registry remains the
|
||||
* only owner of plaintext matches and replacement policy.
|
||||
*/
|
||||
export class ResolvedSecretTraceProvenanceAccumulator {
|
||||
private readonly scope?: ResolvedSecretTraceScopeV1
|
||||
private provenance: ResolvedSecretTraceProvenanceV1
|
||||
|
||||
constructor(scope?: ResolvedSecretTraceScopeV1) {
|
||||
this.scope = scope ? cloneProvenanceScope(scope) : undefined
|
||||
this.provenance = this.emptyProvenance(true)
|
||||
}
|
||||
|
||||
/** Adds one cold, warm-pool, or retry report without decrypting its entries. */
|
||||
record(provenance: unknown): boolean {
|
||||
if (!isResolvedSecretTraceProvenanceV1(provenance)) {
|
||||
this.provenance = this.emptyProvenance(false)
|
||||
return false
|
||||
}
|
||||
|
||||
const sameScope = scopesMatch(provenance.scope, this.scope)
|
||||
const complete = this.provenance.complete && provenance.complete && sameScope
|
||||
if (!complete) {
|
||||
this.provenance = this.emptyProvenance(false)
|
||||
return true
|
||||
}
|
||||
|
||||
const entries = new Map(
|
||||
this.provenance.entries.map((entry) => [
|
||||
`${entry.name ?? ''}\u0000${entry.encryptedValue}`,
|
||||
entry,
|
||||
])
|
||||
)
|
||||
for (const entry of provenance.entries) {
|
||||
const scopedEntry: ResolvedSecretTraceProvenanceEntryV1 = {
|
||||
encryptedValue: entry.encryptedValue,
|
||||
...(entry.name ? { name: entry.name } : {}),
|
||||
}
|
||||
entries.set(`${scopedEntry.name ?? ''}\u0000${scopedEntry.encryptedValue}`, scopedEntry)
|
||||
}
|
||||
|
||||
const merged: ResolvedSecretTraceProvenanceV1 = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [...entries.values()],
|
||||
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
|
||||
}
|
||||
if (!isResolvedSecretTraceProvenanceV1(merged)) {
|
||||
this.provenance = this.emptyProvenance(false)
|
||||
return false
|
||||
}
|
||||
|
||||
this.provenance = merged
|
||||
return true
|
||||
}
|
||||
|
||||
/** Marks the invocation incomplete and discards entries that can no longer be trusted. */
|
||||
markIncomplete(): void {
|
||||
this.provenance = this.emptyProvenance(false)
|
||||
}
|
||||
|
||||
exportProvenance(): ResolvedSecretTraceProvenanceV1 {
|
||||
return structuredClone(this.provenance)
|
||||
}
|
||||
|
||||
private emptyProvenance(complete: boolean): ResolvedSecretTraceProvenanceV1 {
|
||||
return {
|
||||
version: 1,
|
||||
complete,
|
||||
entries: [],
|
||||
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Tracks Secrets-tab values that were actually substituted through `{{NAME}}` during one run.
|
||||
* The current catalog is inert until a successful resolution activates an entry.
|
||||
*/
|
||||
export class ResolvedSecretTraceRegistry {
|
||||
private readonly catalog = new Map<string, ResolvedSecretTraceCatalogEntry>()
|
||||
private catalogBytes = 0
|
||||
private readonly activeEntries = new Map<string, ActiveSecretEntry>()
|
||||
private activeProvenanceEntryBytes = 0
|
||||
private complete = true
|
||||
private readonly scope?: ResolvedSecretTraceScopeV1
|
||||
private readonly completeProvenanceEnvelopeBytes: number
|
||||
|
||||
constructor(
|
||||
catalogEntries: Iterable<ResolvedSecretTraceCatalogEntry> = [],
|
||||
scope?: ResolvedSecretTraceScopeV1
|
||||
) {
|
||||
this.scope = scope ? cloneProvenanceScope(scope) : undefined
|
||||
this.completeProvenanceEnvelopeBytes = serializedProvenanceEnvelopeByteSize(true, this.scope)
|
||||
if (this.completeProvenanceEnvelopeBytes > MAX_SERIALIZED_PROVENANCE_BYTES) {
|
||||
this.markIncomplete()
|
||||
}
|
||||
let catalogEntriesSeen = 0
|
||||
for (const entry of catalogEntries) {
|
||||
catalogEntriesSeen++
|
||||
if (catalogEntriesSeen > MAX_TRACE_CATALOG_ENTRIES) {
|
||||
this.markIncomplete()
|
||||
break
|
||||
}
|
||||
if (!this.addCatalogEntry(entry)) break
|
||||
}
|
||||
}
|
||||
|
||||
/** Activates a configured secret only when the resolved runtime value matches its catalog value. */
|
||||
recordResolved(name: string, resolvedValue: string): boolean {
|
||||
if (resolvedValue.length === 0) return false
|
||||
const catalogEntry = this.catalog.get(name)
|
||||
if (!catalogEntry || catalogEntry.plaintext !== resolvedValue) {
|
||||
this.markIncomplete()
|
||||
return false
|
||||
}
|
||||
|
||||
this.addActiveEntry({ ...catalogEntry, anonymous: false })
|
||||
return true
|
||||
}
|
||||
|
||||
/** Imports encrypted provenance only from a boundary that has already established trust. */
|
||||
async importProvenance(
|
||||
provenance: unknown,
|
||||
options: ImportResolvedSecretTraceProvenanceOptions
|
||||
): Promise<boolean> {
|
||||
if (!options.trusted || !isResolvedSecretTraceProvenanceV1(provenance)) {
|
||||
this.markIncomplete()
|
||||
return false
|
||||
}
|
||||
|
||||
if (!provenance.complete) {
|
||||
this.markIncomplete()
|
||||
}
|
||||
|
||||
const sameScope = scopesMatch(provenance.scope, this.scope)
|
||||
let importedAll = true
|
||||
for (const entry of provenance.entries) {
|
||||
try {
|
||||
const { decrypted } = await decryptSecret(entry.encryptedValue)
|
||||
this.addActiveEntry({
|
||||
name: entry.name ?? '',
|
||||
plaintext: decrypted,
|
||||
encryptedValue: entry.encryptedValue,
|
||||
anonymous: options.anonymous === true || !sameScope || entry.name === undefined,
|
||||
})
|
||||
} catch {
|
||||
importedAll = false
|
||||
this.markIncomplete()
|
||||
}
|
||||
}
|
||||
|
||||
return importedAll
|
||||
}
|
||||
|
||||
/** Imports all same-scope provenance, or only anonymous literals crossing a trust boundary. */
|
||||
async importCrossingProvenance(
|
||||
provenance: unknown,
|
||||
crossingValue: unknown,
|
||||
options: { trusted: boolean }
|
||||
): Promise<boolean> {
|
||||
if (!options.trusted || !isResolvedSecretTraceProvenanceV1(provenance)) {
|
||||
return this.importProvenance(provenance, { trusted: options.trusted })
|
||||
}
|
||||
|
||||
if (scopesMatch(provenance.scope, this.scope)) {
|
||||
return this.importProvenance(provenance, { trusted: true })
|
||||
}
|
||||
|
||||
const sourceRegistry = new ResolvedSecretTraceRegistry([], provenance.scope)
|
||||
const sourceImported = await sourceRegistry.importProvenance(provenance, { trusted: true })
|
||||
const crossingProvenance = sourceRegistry.exportProvenanceForValue(crossingValue, {
|
||||
anonymous: true,
|
||||
})
|
||||
const crossingImported = await this.importProvenance(crossingProvenance, { trusted: true })
|
||||
return sourceImported && crossingImported
|
||||
}
|
||||
|
||||
/** Returns deterministic literal replacements for the terminal TraceSpan projection. */
|
||||
getActiveMatches(): readonly ResolvedSecretTraceMatch[] {
|
||||
const candidatesByPlaintext = new Map<string, ActiveSecretEntry[]>()
|
||||
for (const entry of this.activeEntries.values()) {
|
||||
if (entry.plaintext.length === 0) continue
|
||||
const candidates = candidatesByPlaintext.get(entry.plaintext) ?? []
|
||||
candidates.push(entry)
|
||||
candidatesByPlaintext.set(entry.plaintext, candidates)
|
||||
}
|
||||
|
||||
const matches = [...candidatesByPlaintext.keys()].map((plaintext) => {
|
||||
const candidates = candidatesByPlaintext.get(plaintext) ?? []
|
||||
const anonymous = candidates.some((candidate) => candidate.anonymous)
|
||||
const firstNamed = candidates
|
||||
.filter((candidate) => !candidate.anonymous)
|
||||
.sort((left, right) => compareStrings(left.name, right.name))[0]
|
||||
const replacement = anonymous
|
||||
? ANONYMOUS_SECRET_TRACE_REPLACEMENT
|
||||
: `{{${firstNamed?.name ?? ''}}}`
|
||||
|
||||
return { plaintext, replacement }
|
||||
})
|
||||
|
||||
return matches.sort(
|
||||
(left, right) =>
|
||||
right.plaintext.length - left.plaintext.length ||
|
||||
compareStrings(left.plaintext, right.plaintext) ||
|
||||
compareStrings(left.replacement, right.replacement)
|
||||
)
|
||||
}
|
||||
|
||||
isComplete(): boolean {
|
||||
return this.complete
|
||||
}
|
||||
|
||||
markIncomplete(): void {
|
||||
this.complete = false
|
||||
}
|
||||
|
||||
/** Serializes only encrypted active values; plaintext never enters execution state. */
|
||||
exportProvenance(): ResolvedSecretTraceProvenanceV1 {
|
||||
const entries = this.complete
|
||||
? this.buildProvenanceEntries([...this.activeEntries.values()])
|
||||
: []
|
||||
|
||||
return {
|
||||
version: 1,
|
||||
complete: this.complete,
|
||||
entries,
|
||||
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Exports only active provenance whose exact plaintext occurs in a cross-boundary value.
|
||||
* Bounded traversal returns incomplete provenance instead of broadening to unrelated secrets.
|
||||
*/
|
||||
exportProvenanceForValue(
|
||||
value: unknown,
|
||||
options: ExportResolvedSecretTraceProvenanceForValueOptions = {}
|
||||
): ResolvedSecretTraceProvenanceV1 {
|
||||
if (!this.complete) return { version: 1, complete: false, entries: [] }
|
||||
|
||||
const candidatesByPlaintext = new Map<string, ActiveSecretEntry>()
|
||||
const sortedActiveEntries = [...this.activeEntries.values()].sort(
|
||||
(left, right) =>
|
||||
compareStrings(left.name, right.name) ||
|
||||
compareStrings(left.encryptedValue, right.encryptedValue)
|
||||
)
|
||||
for (const entry of sortedActiveEntries) {
|
||||
if (entry.plaintext.length > 0 && !candidatesByPlaintext.has(entry.plaintext)) {
|
||||
candidatesByPlaintext.set(entry.plaintext, entry)
|
||||
}
|
||||
}
|
||||
|
||||
const matchedEntries = new Map<string, ActiveSecretEntry>()
|
||||
const pendingValues: unknown[] = [value]
|
||||
const visited = new WeakSet<object>()
|
||||
let scannedNodes = 0
|
||||
let scannedCharacters = 0
|
||||
let comparisons = 0
|
||||
let enumeratedProperties = 0
|
||||
let scanComplete = true
|
||||
|
||||
const scanString = (candidate: string): boolean => {
|
||||
scannedCharacters += candidate.length
|
||||
if (scannedCharacters > MAX_PROVENANCE_FILTER_CHARACTERS) return false
|
||||
|
||||
for (const [plaintext, entry] of candidatesByPlaintext) {
|
||||
if (matchedEntries.has(plaintext)) continue
|
||||
comparisons++
|
||||
if (comparisons > MAX_PROVENANCE_FILTER_COMPARISONS) return false
|
||||
if (candidate.includes(plaintext)) {
|
||||
matchedEntries.set(plaintext, entry)
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
const scanProperty = (key: string, descriptor: PropertyDescriptor): boolean => {
|
||||
if (scannedNodes + pendingValues.length >= MAX_PROVENANCE_FILTER_NODES) return false
|
||||
scannedNodes++
|
||||
if (!scanString(key)) return false
|
||||
if (matchedEntries.size >= candidatesByPlaintext.size) return true
|
||||
|
||||
if ('value' in descriptor) {
|
||||
if (scannedNodes + pendingValues.length >= MAX_PROVENANCE_FILTER_NODES) return false
|
||||
pendingValues.push(descriptor.value)
|
||||
} else if (descriptor.enumerable) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
while (pendingValues.length > 0 && matchedEntries.size < candidatesByPlaintext.size) {
|
||||
const current = pendingValues.pop()
|
||||
scannedNodes++
|
||||
if (scannedNodes > MAX_PROVENANCE_FILTER_NODES) {
|
||||
scanComplete = false
|
||||
break
|
||||
}
|
||||
|
||||
if (
|
||||
typeof current === 'string' ||
|
||||
typeof current === 'number' ||
|
||||
typeof current === 'boolean' ||
|
||||
current === null
|
||||
) {
|
||||
if (!scanString(String(current))) scanComplete = false
|
||||
if (!scanComplete) break
|
||||
continue
|
||||
}
|
||||
|
||||
if (typeof current !== 'object' || visited.has(current)) {
|
||||
continue
|
||||
}
|
||||
visited.add(current)
|
||||
|
||||
if (isLargeValueRef(current) || isLargeArrayManifest(current)) {
|
||||
scanComplete = false
|
||||
break
|
||||
}
|
||||
|
||||
try {
|
||||
for (const key of ERROR_CONTENT_PROPERTY_NAMES) {
|
||||
const descriptor = Object.getOwnPropertyDescriptor(current, key)
|
||||
if (descriptor && !descriptor.enumerable && !scanProperty(key, descriptor)) {
|
||||
scanComplete = false
|
||||
break
|
||||
}
|
||||
if (matchedEntries.size >= candidatesByPlaintext.size) break
|
||||
}
|
||||
if (!scanComplete || matchedEntries.size >= candidatesByPlaintext.size) break
|
||||
|
||||
for (const key in current as Record<string, unknown>) {
|
||||
enumeratedProperties++
|
||||
if (enumeratedProperties > MAX_PROVENANCE_FILTER_NODES) {
|
||||
scanComplete = false
|
||||
break
|
||||
}
|
||||
|
||||
const descriptor = Object.getOwnPropertyDescriptor(current, key)
|
||||
if (!descriptor) continue
|
||||
if (!scanProperty(key, descriptor)) {
|
||||
scanComplete = false
|
||||
break
|
||||
}
|
||||
if (matchedEntries.size >= candidatesByPlaintext.size) break
|
||||
}
|
||||
if (!scanComplete) break
|
||||
} catch {
|
||||
scanComplete = false
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
const complete = this.complete && scanComplete
|
||||
const entries = complete
|
||||
? this.buildProvenanceEntries([...matchedEntries.values()], options.anonymous)
|
||||
: []
|
||||
return {
|
||||
version: 1,
|
||||
complete,
|
||||
entries,
|
||||
...(this.scope ? { scope: cloneProvenanceScope(this.scope) } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
private addActiveEntry(entry: ActiveSecretEntry): void {
|
||||
const key = `${entry.anonymous ? 'anonymous' : entry.name}\u0000${entry.encryptedValue}`
|
||||
if (this.activeEntries.has(key)) {
|
||||
this.activeEntries.set(key, entry)
|
||||
return
|
||||
}
|
||||
|
||||
const entryBytes = serializedProvenanceEntryByteSize(toProvenanceEntry(entry))
|
||||
const separatorBytes = this.activeEntries.size === 0 ? 0 : 1
|
||||
if (
|
||||
this.activeEntries.size >= MAX_PROVENANCE_ENTRIES ||
|
||||
this.completeProvenanceEnvelopeBytes +
|
||||
this.activeProvenanceEntryBytes +
|
||||
separatorBytes +
|
||||
entryBytes >
|
||||
MAX_SERIALIZED_PROVENANCE_BYTES
|
||||
) {
|
||||
this.markIncomplete()
|
||||
return
|
||||
}
|
||||
this.activeEntries.set(key, entry)
|
||||
this.activeProvenanceEntryBytes += separatorBytes + entryBytes
|
||||
}
|
||||
|
||||
private addCatalogEntry(entry: ResolvedSecretTraceCatalogEntry): boolean {
|
||||
const existing = this.catalog.get(entry.name)
|
||||
const nextCatalogBytes =
|
||||
this.catalogBytes -
|
||||
(existing ? catalogEntryByteSize(existing) : 0) +
|
||||
catalogEntryByteSize(entry)
|
||||
const nextCatalogSize = this.catalog.size + (existing ? 0 : 1)
|
||||
if (nextCatalogSize > MAX_TRACE_CATALOG_ENTRIES || nextCatalogBytes > MAX_TRACE_CATALOG_BYTES) {
|
||||
this.markIncomplete()
|
||||
return false
|
||||
}
|
||||
|
||||
this.catalog.set(entry.name, { ...entry })
|
||||
this.catalogBytes = nextCatalogBytes
|
||||
return true
|
||||
}
|
||||
|
||||
private buildProvenanceEntries(
|
||||
activeEntries: ActiveSecretEntry[],
|
||||
forceAnonymous = false
|
||||
): ResolvedSecretTraceProvenanceEntryV1[] {
|
||||
return activeEntries
|
||||
.sort(
|
||||
(left, right) =>
|
||||
compareStrings(left.name, right.name) ||
|
||||
compareStrings(left.encryptedValue, right.encryptedValue)
|
||||
)
|
||||
.map((entry) => ({
|
||||
...toProvenanceEntry({ ...entry, anonymous: forceAnonymous || entry.anonymous }),
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
/** Builds the effective workspace-over-personal catalog and restores trusted active provenance. */
|
||||
export async function createResolvedSecretTraceRegistry(
|
||||
options: CreateResolvedSecretTraceRegistryOptions
|
||||
): Promise<ResolvedSecretTraceRegistry> {
|
||||
const failedNames = new Set(options.decryptionFailures ?? [])
|
||||
const registry = new ResolvedSecretTraceRegistry(
|
||||
iterateEffectiveCatalogEntries(options, failedNames),
|
||||
options.scope
|
||||
)
|
||||
|
||||
if (options.restoredProvenance !== undefined) {
|
||||
await registry.importProvenance(options.restoredProvenance, {
|
||||
trusted: options.restoreTrusted === true,
|
||||
})
|
||||
} else if (options.requireRestoredProvenance) {
|
||||
registry.markIncomplete()
|
||||
}
|
||||
|
||||
return registry
|
||||
}
|
||||
|
||||
/** Creates a scoped fail-closed registry when trusted catalog provenance is unavailable. */
|
||||
export function createIncompleteResolvedSecretTraceRegistry(
|
||||
scope?: ResolvedSecretTraceScopeV1
|
||||
): ResolvedSecretTraceRegistry {
|
||||
const registry = new ResolvedSecretTraceRegistry([], scope)
|
||||
registry.markIncomplete()
|
||||
return registry
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
import { BlockType } from '@/executor/constants'
|
||||
import { ExecutionState } from '@/executor/execution/state'
|
||||
import type { ExecutionContext } from '@/executor/types'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { VariableResolver } from '@/executor/variables/resolver'
|
||||
import { navigatePathAsync } from '@/executor/variables/resolvers/reference-async.server'
|
||||
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
|
||||
@@ -114,6 +115,27 @@ function createResolver(
|
||||
}
|
||||
|
||||
describe('VariableResolver function block inputs', () => {
|
||||
it('records a secret reached through workflow-variable indirection', async () => {
|
||||
const { ctx, resolver } = createResolver()
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'TOKEN', plaintext: 'resolved-secret', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
ctx.workflowVariables = {
|
||||
'var-1': { id: 'var-1', name: 'indirect', type: 'string', value: '{{TOKEN}}' },
|
||||
}
|
||||
ctx.environmentVariables = { TOKEN: 'resolved-secret' }
|
||||
ctx.resolvedSecretTraceRegistry = registry
|
||||
|
||||
const result = await resolver.resolveInputs(ctx, 'function', {
|
||||
value: '<variable.indirect>',
|
||||
})
|
||||
|
||||
expect(result.value).toBe('resolved-secret')
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'resolved-secret', replacement: '{{TOKEN}}' },
|
||||
])
|
||||
})
|
||||
|
||||
it('returns empty inputs when params are missing', async () => {
|
||||
const { block, ctx, resolver } = createResolver()
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { EnvResolver } from './env'
|
||||
import type { ResolutionContext } from './reference'
|
||||
|
||||
@@ -48,6 +49,23 @@ describe('EnvResolver', () => {
|
||||
})
|
||||
|
||||
describe('resolve', () => {
|
||||
it('records only successful Secrets-tab substitutions', () => {
|
||||
const resolver = new EnvResolver()
|
||||
const registry = new ResolvedSecretTraceRegistry([
|
||||
{ name: 'API_KEY', plaintext: 'secret-api-key', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
const ctx = createTestContext({ API_KEY: 'secret-api-key' })
|
||||
ctx.executionContext.resolvedSecretTraceRegistry = registry
|
||||
|
||||
expect(resolver.resolve('{{MISSING}}', ctx)).toBe('{{MISSING}}')
|
||||
expect(registry.getActiveMatches()).toEqual([])
|
||||
|
||||
expect(resolver.resolve('{{API_KEY}}', ctx)).toBe('secret-api-key')
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'secret-api-key', replacement: '{{API_KEY}}' },
|
||||
])
|
||||
})
|
||||
|
||||
it.concurrent('should resolve existing environment variable', () => {
|
||||
const resolver = new EnvResolver()
|
||||
const ctx = createTestContext({ API_KEY: 'secret-api-key' })
|
||||
|
||||
@@ -16,6 +16,9 @@ export class EnvResolver implements Resolver {
|
||||
if (value === undefined) {
|
||||
return reference
|
||||
}
|
||||
if (Object.hasOwn(context.executionContext.environmentVariables, varName)) {
|
||||
context.executionContext.resolvedSecretTraceRegistry?.recordResolved(varName, value)
|
||||
}
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
@@ -375,12 +375,22 @@ export function useCancelExecution(workspaceId: string) {
|
||||
export function useRetryExecution() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: async ({ workflowId, input }: { workflowId: string; input?: unknown }) => {
|
||||
mutationFn: async ({
|
||||
workflowId,
|
||||
executionId,
|
||||
}: {
|
||||
workflowId: string
|
||||
executionId: string
|
||||
}) => {
|
||||
// boundary-raw-fetch: stream response, body is a ReadableStream consumed one chunk at a time
|
||||
const res = await fetch(`/api/workflows/${workflowId}/execute`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ input, triggerType: 'manual', stream: true }),
|
||||
body: JSON.stringify({
|
||||
inputFromExecutionId: executionId,
|
||||
triggerType: 'manual',
|
||||
stream: true,
|
||||
}),
|
||||
})
|
||||
if (!res.ok) {
|
||||
const data = await res.json().catch(() => ({}))
|
||||
|
||||
@@ -210,7 +210,8 @@ export interface ExecuteStreamOptions {
|
||||
export interface ExecuteFromBlockOptions {
|
||||
workflowId: string
|
||||
startBlockId: string
|
||||
sourceSnapshot: SerializableExecutionState
|
||||
sourceSnapshot?: SerializableExecutionState
|
||||
sourceExecutionId?: string
|
||||
input?: any
|
||||
onExecutionId?: (executionId: string) => void
|
||||
callbacks?: ExecutionStreamCallbacks
|
||||
@@ -348,6 +349,7 @@ export function useExecutionStream() {
|
||||
workflowId,
|
||||
startBlockId,
|
||||
sourceSnapshot,
|
||||
sourceExecutionId,
|
||||
input,
|
||||
onExecutionId,
|
||||
callbacks = {},
|
||||
@@ -370,7 +372,11 @@ export function useExecutionStream() {
|
||||
body: JSON.stringify({
|
||||
stream: true,
|
||||
input,
|
||||
runFromBlock: { startBlockId, sourceSnapshot },
|
||||
runFromBlock: {
|
||||
startBlockId,
|
||||
...(sourceExecutionId ? { executionId: sourceExecutionId } : {}),
|
||||
...(sourceSnapshot ? { sourceSnapshot } : {}),
|
||||
},
|
||||
}),
|
||||
signal: abortController.signal,
|
||||
})
|
||||
|
||||
@@ -28,6 +28,12 @@ describe('workflow contracts', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('accepts a trusted prior-execution input reference', () => {
|
||||
expect(
|
||||
executeWorkflowBodySchema.parse({ inputFromExecutionId: 'execution-123' })
|
||||
).toMatchObject({ inputFromExecutionId: 'execution-123' })
|
||||
})
|
||||
|
||||
it('normalizes null React Flow edge handles in execution overrides', () => {
|
||||
const parsed = executeWorkflowBodySchema.parse({
|
||||
workflowStateOverride: {
|
||||
|
||||
@@ -360,6 +360,8 @@ export const executeWorkflowBodySchema = z.object({
|
||||
includeToolCalls: z.boolean().optional().default(false),
|
||||
useDraftState: z.boolean().optional(),
|
||||
input: z.any().optional(),
|
||||
/** Trusted server-side reuse of a prior execution's raw workflow input. */
|
||||
inputFromExecutionId: executionIdSchema.optional(),
|
||||
isClientSession: z.boolean().optional(),
|
||||
includeFileBase64: z.boolean().optional().default(true),
|
||||
base64MaxBytes: z.number().int().positive().optional(),
|
||||
|
||||
@@ -11,7 +11,7 @@ const { discoverServerTools, validateMcpToolsAllowed } = vi.hoisted(() => ({
|
||||
vi.mock('@/lib/mcp/service', () => ({ mcpService: { discoverServerTools } }))
|
||||
vi.mock('@/ee/access-control/utils/permission-check', () => ({ validateMcpToolsAllowed }))
|
||||
|
||||
import { buildSelectedMcpToolSchemas, buildTaggedMcpToolSchemas } from './mcp-tools'
|
||||
import { buildSelectedMcpToolSchemas, buildTaggedMcpToolSchemas } from '@/lib/copilot/mcp-tools'
|
||||
|
||||
describe('mothership MCP tool schemas', () => {
|
||||
beforeEach(() => {
|
||||
@@ -50,19 +50,130 @@ describe('mothership MCP tool schemas', () => {
|
||||
])
|
||||
})
|
||||
|
||||
it('reports tagged-server discovery provenance without placing it in tool schemas', async () => {
|
||||
const provenance = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'ws-1' },
|
||||
}
|
||||
discoverServerTools.mockImplementationOnce(
|
||||
async (
|
||||
_userId: string,
|
||||
_serverId: string,
|
||||
_workspaceId: string,
|
||||
_forceRefresh: boolean,
|
||||
report: (value: unknown) => void
|
||||
) => {
|
||||
report(provenance)
|
||||
return [
|
||||
{
|
||||
serverId: 'mcp-server-1',
|
||||
name: 'search',
|
||||
description: 'Search docs',
|
||||
inputSchema: { type: 'object' },
|
||||
},
|
||||
]
|
||||
}
|
||||
)
|
||||
const recordProvenance = vi.fn()
|
||||
|
||||
const tools = await buildTaggedMcpToolSchemas(
|
||||
'user-1',
|
||||
'ws-1',
|
||||
['mcp-server-1'],
|
||||
recordProvenance
|
||||
)
|
||||
|
||||
expect(discoverServerTools).toHaveBeenCalledWith(
|
||||
'user-1',
|
||||
'mcp-server-1',
|
||||
'ws-1',
|
||||
false,
|
||||
expect.any(Function)
|
||||
)
|
||||
expect(recordProvenance).toHaveBeenCalledWith(provenance)
|
||||
expect(JSON.stringify(tools)).not.toContain('encrypted-token')
|
||||
expect(JSON.stringify(tools)).not.toContain('resolvedSecretTraceProvenance')
|
||||
})
|
||||
|
||||
it('reports incomplete provenance when tagged-server discovery returns no report', async () => {
|
||||
discoverServerTools.mockResolvedValue([])
|
||||
const recordProvenance = vi.fn()
|
||||
|
||||
await buildTaggedMcpToolSchemas('user-1', 'ws-1', ['mcp-server-1'], recordProvenance)
|
||||
|
||||
expect(recordProvenance).toHaveBeenCalledWith({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'ws-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('uses a selected block tool cached schema without discovering the server', async () => {
|
||||
const tools = await buildSelectedMcpToolSchemas('user-1', 'ws-1', [
|
||||
{
|
||||
type: 'mcp',
|
||||
params: { serverId: 'mcp-server-1', toolName: 'search', serverName: 'Docs' },
|
||||
schema: { type: 'object', properties: { query: { type: 'string' } } },
|
||||
},
|
||||
])
|
||||
const recordProvenance = vi.fn()
|
||||
const tools = await buildSelectedMcpToolSchemas(
|
||||
'user-1',
|
||||
'ws-1',
|
||||
[
|
||||
{
|
||||
type: 'mcp',
|
||||
params: { serverId: 'mcp-server-1', toolName: 'search', serverName: 'Docs' },
|
||||
schema: { type: 'object', properties: { query: { type: 'string' } } },
|
||||
},
|
||||
],
|
||||
recordProvenance
|
||||
)
|
||||
|
||||
expect(discoverServerTools).not.toHaveBeenCalled()
|
||||
expect(recordProvenance).not.toHaveBeenCalled()
|
||||
expect(tools[0]).toMatchObject({
|
||||
name: 'mcp-server-1-search',
|
||||
input_schema: { type: 'object', properties: { query: { type: 'string' } } },
|
||||
})
|
||||
})
|
||||
|
||||
it('reports provenance from selected tools that require server discovery', async () => {
|
||||
const provenance = {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'ws-1' },
|
||||
}
|
||||
discoverServerTools.mockImplementationOnce(
|
||||
async (
|
||||
_userId: string,
|
||||
_serverId: string,
|
||||
_workspaceId: string,
|
||||
_forceRefresh: boolean,
|
||||
report: (value: unknown) => void
|
||||
) => {
|
||||
report(provenance)
|
||||
return [
|
||||
{
|
||||
serverId: 'mcp-server-1',
|
||||
name: 'search',
|
||||
inputSchema: { type: 'object' },
|
||||
},
|
||||
]
|
||||
}
|
||||
)
|
||||
const recordProvenance = vi.fn()
|
||||
|
||||
const tools = await buildSelectedMcpToolSchemas(
|
||||
'user-1',
|
||||
'ws-1',
|
||||
[
|
||||
{
|
||||
type: 'mcp',
|
||||
params: { serverId: 'mcp-server-1', toolName: 'search' },
|
||||
},
|
||||
],
|
||||
recordProvenance
|
||||
)
|
||||
|
||||
expect(recordProvenance).toHaveBeenCalledWith(provenance)
|
||||
expect(tools[0]).toMatchObject({ name: 'mcp-server-1-search' })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -8,6 +8,8 @@ import type { ToolInput } from '@/executor/handlers/agent/types'
|
||||
|
||||
const logger = createLogger('CopilotMcpTools')
|
||||
|
||||
type ResolvedSecretTraceProvenanceCallback = (provenance: unknown) => void
|
||||
|
||||
function toMothershipMcpTool(tool: {
|
||||
serverId: string
|
||||
serverName?: string
|
||||
@@ -51,11 +53,26 @@ function dedupeMcpTools(tools: ToolSchema[]): ToolSchema[] {
|
||||
async function discoverServerTools(
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
serverId: string
|
||||
serverId: string,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<McpTool[]> {
|
||||
let provenanceReported = false
|
||||
try {
|
||||
const { mcpService } = await import('@/lib/mcp/service')
|
||||
return await mcpService.discoverServerTools(userId, serverId, workspaceId)
|
||||
if (!onResolvedSecretTraceProvenance) {
|
||||
return await mcpService.discoverServerTools(userId, serverId, workspaceId)
|
||||
}
|
||||
|
||||
return await mcpService.discoverServerTools(
|
||||
userId,
|
||||
serverId,
|
||||
workspaceId,
|
||||
false,
|
||||
(provenance) => {
|
||||
provenanceReported = true
|
||||
onResolvedSecretTraceProvenance(provenance)
|
||||
}
|
||||
)
|
||||
} catch (error) {
|
||||
logger.warn('Failed to resolve tagged MCP server tools', {
|
||||
serverId,
|
||||
@@ -63,6 +80,15 @@ async function discoverServerTools(
|
||||
error: toError(error).message,
|
||||
})
|
||||
return []
|
||||
} finally {
|
||||
if (onResolvedSecretTraceProvenance && !provenanceReported) {
|
||||
onResolvedSecretTraceProvenance({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId, workspaceId },
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -73,14 +99,17 @@ async function discoverServerTools(
|
||||
export async function buildTaggedMcpToolSchemas(
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
serverIds: string[]
|
||||
serverIds: string[],
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<ToolSchema[]> {
|
||||
const uniqueServerIds = [...new Set(serverIds.filter(Boolean))]
|
||||
if (uniqueServerIds.length === 0) return []
|
||||
|
||||
await validateMcpToolsAllowed(userId, workspaceId)
|
||||
const discovered = await Promise.all(
|
||||
uniqueServerIds.map((serverId) => discoverServerTools(userId, workspaceId, serverId))
|
||||
uniqueServerIds.map((serverId) =>
|
||||
discoverServerTools(userId, workspaceId, serverId, onResolvedSecretTraceProvenance)
|
||||
)
|
||||
)
|
||||
return dedupeMcpTools(discovered.flat().map(toMothershipMcpTool))
|
||||
}
|
||||
@@ -93,7 +122,8 @@ export async function buildTaggedMcpToolSchemas(
|
||||
export async function buildSelectedMcpToolSchemas(
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
selections: ToolInput[]
|
||||
selections: ToolInput[],
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<ToolSchema[]> {
|
||||
const selected = selections.filter(
|
||||
(tool) =>
|
||||
@@ -130,7 +160,12 @@ export async function buildSelectedMcpToolSchemas(
|
||||
|
||||
let discovery = discoveredByServer.get(serverId)
|
||||
if (!discovery) {
|
||||
discovery = discoverServerTools(userId, workspaceId, serverId)
|
||||
discovery = discoverServerTools(
|
||||
userId,
|
||||
workspaceId,
|
||||
serverId,
|
||||
onResolvedSecretTraceProvenance
|
||||
)
|
||||
discoveredByServer.set(serverId, discovery)
|
||||
}
|
||||
const match = (await discovery).find((tool) => tool.name === toolName)
|
||||
|
||||
@@ -10,6 +10,7 @@ import {
|
||||
} from '@sim/testing'
|
||||
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { ExecutionContext, StreamingContext } from '@/lib/copilot/request/types'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const mockGetEffectiveDecryptedEnv = environmentUtilsMockFns.mockGetEffectiveDecryptedEnv
|
||||
|
||||
@@ -148,6 +149,44 @@ describe('runCopilotLifecycle', () => {
|
||||
mockGetMothershipSourceEnvHeaders.mockReturnValue({})
|
||||
})
|
||||
|
||||
it('threads trace provenance through server execution context only', async () => {
|
||||
const registry = {} as ResolvedSecretTraceRegistry
|
||||
const executionContext: ExecutionContext = {
|
||||
userId: 'user-1',
|
||||
workflowId: '',
|
||||
workspaceId: 'ws-1',
|
||||
decryptedEnvVars: {},
|
||||
}
|
||||
let capturedExecutionContext: ExecutionContext | undefined
|
||||
let capturedRequestBody = ''
|
||||
mockRunStreamLoop.mockImplementationOnce(
|
||||
async (
|
||||
_url: string,
|
||||
request: RequestInit,
|
||||
_streamingContext: StreamingContext,
|
||||
context: ExecutionContext
|
||||
) => {
|
||||
capturedExecutionContext = context
|
||||
capturedRequestBody = String(request.body)
|
||||
}
|
||||
)
|
||||
|
||||
await runCopilotLifecycle(
|
||||
{ message: 'hello', messageId: 'stream-private-context' },
|
||||
{
|
||||
userId: 'user-1',
|
||||
workspaceId: 'ws-1',
|
||||
executionContext,
|
||||
resolvedSecretTraceRegistry: registry,
|
||||
}
|
||||
)
|
||||
|
||||
expect(capturedExecutionContext?.resolvedSecretTraceRegistry).toBe(registry)
|
||||
expect(capturedRequestBody).not.toContain('resolvedSecretTraceRegistry')
|
||||
expect(capturedRequestBody).not.toContain('resolved-secret-provenance')
|
||||
expect(executionContext).not.toHaveProperty('resolvedSecretTraceRegistry')
|
||||
})
|
||||
|
||||
describe('tool permission feature flag', () => {
|
||||
const runMothershipTurn = () =>
|
||||
runCopilotLifecycle(
|
||||
|
||||
@@ -62,6 +62,7 @@ import {
|
||||
isHosted,
|
||||
} from '@/lib/core/config/env-flags'
|
||||
import { getEffectiveDecryptedEnv } from '@/lib/environment/utils'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('CopilotLifecycle')
|
||||
|
||||
@@ -95,6 +96,7 @@ export interface CopilotLifecycleOptions extends OrchestratorOptions {
|
||||
onGoTraceId?: (goTraceId: string) => void
|
||||
executionContext?: ExecutionContext
|
||||
billingAttribution?: BillingAttributionSnapshot
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -161,6 +163,9 @@ export async function runCopilotLifecycle(
|
||||
abortSignal: options.abortSignal,
|
||||
billingAttribution:
|
||||
options.billingAttribution ?? options.executionContext.billingAttribution,
|
||||
...(options.resolvedSecretTraceRegistry
|
||||
? { resolvedSecretTraceRegistry: options.resolvedSecretTraceRegistry }
|
||||
: {}),
|
||||
},
|
||||
}
|
||||
: {}),
|
||||
@@ -177,6 +182,7 @@ export async function runCopilotLifecycle(
|
||||
runId: resolvedRunId,
|
||||
abortSignal: lifecycleOptions.abortSignal,
|
||||
billingAttribution: lifecycleOptions.billingAttribution,
|
||||
resolvedSecretTraceRegistry: lifecycleOptions.resolvedSecretTraceRegistry,
|
||||
}))
|
||||
const shouldUseHostedBillingProtocol = isHosted && isCopilotBillingAttributionV1Enabled
|
||||
if (
|
||||
@@ -993,6 +999,7 @@ async function buildExecutionContext(
|
||||
runId?: string
|
||||
abortSignal?: AbortSignal
|
||||
billingAttribution?: BillingAttributionSnapshot
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
}
|
||||
): Promise<ExecutionContext> {
|
||||
const {
|
||||
@@ -1004,6 +1011,7 @@ async function buildExecutionContext(
|
||||
runId,
|
||||
abortSignal,
|
||||
billingAttribution,
|
||||
resolvedSecretTraceRegistry,
|
||||
} = params
|
||||
const userTimezone =
|
||||
typeof requestPayload?.userTimezone === 'string' ? requestPayload.userTimezone : undefined
|
||||
@@ -1039,6 +1047,9 @@ async function buildExecutionContext(
|
||||
execContext.runId = runId
|
||||
execContext.abortSignal = abortSignal
|
||||
if (billingAttribution) execContext.billingAttribution = billingAttribution
|
||||
if (resolvedSecretTraceRegistry) {
|
||||
execContext.resolvedSecretTraceRegistry = resolvedSecretTraceRegistry
|
||||
}
|
||||
return execContext
|
||||
}
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { DEFAULT_EXECUTION_TIMEOUT_MS } from '@/lib/execution/constants'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const { isKnownTool, isSimExecuted, isClientExecuted } = vi.hoisted(() => ({
|
||||
isKnownTool: vi.fn(),
|
||||
@@ -111,6 +112,35 @@ describe('copilot tool executor fallback', () => {
|
||||
expect(appParams._context).not.toHaveProperty('billingAttribution')
|
||||
})
|
||||
|
||||
it('passes trace provenance out-of-band without exposing it in app tool parameters', async () => {
|
||||
isKnownTool.mockReturnValue(false)
|
||||
isSimExecuted.mockReturnValue(false)
|
||||
executeAppTool.mockResolvedValue({ success: true, output: { result: 'unchanged' } })
|
||||
const registry = {} as ResolvedSecretTraceRegistry
|
||||
|
||||
const result = await executeTool(
|
||||
'gmail_read',
|
||||
{ query: 'hello' },
|
||||
{
|
||||
userId: 'user-1',
|
||||
workflowId: 'workflow-1',
|
||||
resolvedSecretTraceRegistry: registry,
|
||||
}
|
||||
)
|
||||
|
||||
expect(executeAppTool).toHaveBeenCalledWith(
|
||||
'gmail_read',
|
||||
expect.objectContaining({
|
||||
query: 'hello',
|
||||
_context: expect.not.objectContaining({ resolvedSecretTraceRegistry: expect.anything() }),
|
||||
}),
|
||||
{ resolvedSecretTraceRegistry: registry }
|
||||
)
|
||||
const appParams = executeAppTool.mock.calls[0]?.[1]
|
||||
expect(JSON.stringify(appParams)).not.toContain('resolvedSecretTraceRegistry')
|
||||
expect(result).toEqual({ success: true, output: { result: 'unchanged' } })
|
||||
})
|
||||
|
||||
it('uses the registered handler for client-routed tools when running headless (Mothership block)', async () => {
|
||||
isKnownTool.mockReturnValue(true)
|
||||
isSimExecuted.mockReturnValue(false)
|
||||
|
||||
@@ -55,7 +55,11 @@ export async function executeTool(
|
||||
(isSimExecuted(toolId) || (isClientExecuted(toolId) && hasHandler(toolId)))
|
||||
if (!canUseRegisteredHandler) {
|
||||
const appParams = buildAppToolParams(normalizedParams, context)
|
||||
return executeAppTool(toolId, appParams)
|
||||
return context.resolvedSecretTraceRegistry
|
||||
? executeAppTool(toolId, appParams, {
|
||||
resolvedSecretTraceRegistry: context.resolvedSecretTraceRegistry,
|
||||
})
|
||||
: executeAppTool(toolId, appParams)
|
||||
}
|
||||
|
||||
if (context.abortSignal?.aborted) {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
|
||||
import type { MothershipResource } from '@/lib/copilot/resources/types'
|
||||
import type { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
export interface ToolExecutionContext {
|
||||
userId: string
|
||||
@@ -24,6 +25,7 @@ export interface ToolExecutionContext {
|
||||
userTimezone?: string
|
||||
userPermission?: string
|
||||
decryptedEnvVars?: Record<string, string>
|
||||
resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
}
|
||||
|
||||
export interface ToolExecutionResult {
|
||||
|
||||
@@ -89,6 +89,32 @@ function mountedFiles() {
|
||||
|
||||
const snapshotCacheOn = (flag: string) => Promise.resolve(flag === 'table-snapshot-cache')
|
||||
|
||||
describe('executeFunctionExecute trace-secret provenance', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockExecuteTool.mockResolvedValue({ success: true })
|
||||
})
|
||||
|
||||
it('forwards the registry only through server execution options', async () => {
|
||||
const resolvedSecretTraceRegistry = { recordResolved: vi.fn() }
|
||||
|
||||
await executeFunctionExecute({ code: 'return {{API_KEY}}' }, {
|
||||
userId: 'u1',
|
||||
workspaceId: 'ws_1',
|
||||
resolvedSecretTraceRegistry,
|
||||
} as never)
|
||||
|
||||
expect(mockExecuteTool).toHaveBeenCalledWith(
|
||||
'function_execute',
|
||||
expect.not.objectContaining({ resolvedSecretTraceRegistry: expect.anything() }),
|
||||
{ resolvedSecretTraceRegistry }
|
||||
)
|
||||
const appParams = mockExecuteTool.mock.calls[0]?.[1] as Record<string, unknown>
|
||||
expect(appParams._context).not.toHaveProperty('resolvedSecretTraceRegistry')
|
||||
expect(JSON.stringify(appParams)).not.toContain('resolvedSecretTraceRegistry')
|
||||
})
|
||||
})
|
||||
|
||||
describe('executeFunctionExecute table mounts', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
|
||||
@@ -509,5 +509,9 @@ export async function executeFunctionExecute(
|
||||
enforceCredentialAccess: true,
|
||||
}
|
||||
|
||||
return executeAppTool('function_execute', enrichedParams)
|
||||
return context.resolvedSecretTraceRegistry
|
||||
? executeAppTool('function_execute', enrichedParams, {
|
||||
resolvedSecretTraceRegistry: context.resolvedSecretTraceRegistry,
|
||||
})
|
||||
: executeAppTool('function_execute', enrichedParams)
|
||||
}
|
||||
|
||||
@@ -23,6 +23,10 @@ afterAll(() => {
|
||||
|
||||
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
|
||||
import type { ExecutionContext } from '@/lib/copilot/request/types'
|
||||
import {
|
||||
ANONYMOUS_SECRET_TRACE_REPLACEMENT,
|
||||
ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const {
|
||||
ensureWorkflowAccessMock,
|
||||
@@ -39,6 +43,7 @@ const {
|
||||
checkAttributedUsageLimitsMock,
|
||||
reserveExecutionSlotMock,
|
||||
releaseExecutionSlotMock,
|
||||
decryptSecretMock,
|
||||
} = vi.hoisted(() => ({
|
||||
ensureWorkflowAccessMock: vi.fn(),
|
||||
ensureWorkspaceAccessMock: vi.fn(),
|
||||
@@ -54,6 +59,7 @@ const {
|
||||
checkAttributedUsageLimitsMock: vi.fn(),
|
||||
reserveExecutionSlotMock: vi.fn(),
|
||||
releaseExecutionSlotMock: vi.fn(),
|
||||
decryptSecretMock: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@sim/audit', () => ({
|
||||
@@ -79,6 +85,11 @@ vi.mock('@/lib/billing/calculations/usage-reservation', () => ({
|
||||
UsageReservationUnavailableError: class UsageReservationUnavailableError extends Error {},
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/security/encryption', () => ({
|
||||
decryptSecret: decryptSecretMock,
|
||||
encryptSecret: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workflows/executor/execute-workflow', () => ({
|
||||
executeWorkflow: executeWorkflowMock,
|
||||
}))
|
||||
@@ -297,6 +308,7 @@ describe('executeCreateWorkflow billing attribution', () => {
|
||||
payerUsage: { currentUsage: 1, limit: 10 },
|
||||
})
|
||||
reserveExecutionSlotMock.mockResolvedValue({ reserved: true, created: true })
|
||||
decryptSecretMock.mockResolvedValue({ decrypted: 'secret-value' })
|
||||
listFoldersMock.mockResolvedValue([])
|
||||
})
|
||||
|
||||
@@ -580,6 +592,7 @@ describe('Copilot workflow execution billing attribution', () => {
|
||||
payerUsage: { currentUsage: 1, limit: 10 },
|
||||
})
|
||||
reserveExecutionSlotMock.mockResolvedValue({ reserved: true, created: true })
|
||||
decryptSecretMock.mockResolvedValue({ decrypted: 'secret-value' })
|
||||
})
|
||||
|
||||
async function expectBillingAttributionForwarded(
|
||||
@@ -602,6 +615,178 @@ describe('Copilot workflow execution billing attribution', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('passes only input-crossing parent provenance to the child execution', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry(
|
||||
[
|
||||
{
|
||||
name: 'INPUT_SECRET',
|
||||
plaintext: 'input-secret',
|
||||
encryptedValue: 'input-ciphertext',
|
||||
},
|
||||
{
|
||||
name: 'UNRELATED_SECRET',
|
||||
plaintext: 'unrelated-secret',
|
||||
encryptedValue: 'unrelated-ciphertext',
|
||||
},
|
||||
],
|
||||
{ userId: 'user-1', workspaceId: 'workspace-1' }
|
||||
)
|
||||
registry.recordResolved('INPUT_SECRET', 'input-secret')
|
||||
registry.recordResolved('UNRELATED_SECRET', 'unrelated-secret')
|
||||
resolveTriggerRunOptionsMock.mockReturnValueOnce([
|
||||
{
|
||||
triggerBlockId: 'trigger-1',
|
||||
blockName: 'Start',
|
||||
mockPayload: { value: 'input-secret' },
|
||||
},
|
||||
])
|
||||
executeWorkflowMock.mockResolvedValueOnce({
|
||||
success: true,
|
||||
output: { ok: true },
|
||||
logs: [],
|
||||
metadata: { executionId: 'new-execution-1' },
|
||||
executionState: {
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
const result = await executeRunWorkflow(
|
||||
{ workflowId: 'workflow-1', useMockPayload: true },
|
||||
{ ...executionContext, resolvedSecretTraceRegistry: registry }
|
||||
)
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(executeWorkflowMock.mock.calls[0]?.[2]).toEqual({ value: 'input-secret' })
|
||||
expect(executeWorkflowMock.mock.calls[0]?.[4]).toEqual(
|
||||
expect.objectContaining({
|
||||
trustedInitialResolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'INPUT_SECRET', encryptedValue: 'input-ciphertext' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
})
|
||||
)
|
||||
expect(JSON.stringify(result)).not.toContain('input-ciphertext')
|
||||
expect(JSON.stringify(result)).not.toContain('unrelated-ciphertext')
|
||||
})
|
||||
|
||||
it('imports child provenance without returning private metadata to the model', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
const context: ExecutionContext = {
|
||||
...executionContext,
|
||||
resolvedSecretTraceRegistry: registry,
|
||||
}
|
||||
executeWorkflowMock.mockResolvedValueOnce({
|
||||
success: true,
|
||||
output: { value: 'secret-value' },
|
||||
logs: [],
|
||||
metadata: { executionId: 'new-execution-1' },
|
||||
executionState: {
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'API_KEY', encryptedValue: 'encrypted-secret' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
const result = await executeRunWorkflow(
|
||||
{ workflowId: 'workflow-1', useMockPayload: true },
|
||||
context
|
||||
)
|
||||
|
||||
expect(result).toMatchObject({
|
||||
success: true,
|
||||
output: { output: { value: 'secret-value' } },
|
||||
})
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'secret-value', replacement: '{{API_KEY}}' },
|
||||
])
|
||||
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
|
||||
expect(JSON.stringify(result)).not.toContain('encrypted-secret')
|
||||
})
|
||||
|
||||
it('marks provenance incomplete when child execution returns no trusted state', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry()
|
||||
const context: ExecutionContext = {
|
||||
...executionContext,
|
||||
resolvedSecretTraceRegistry: registry,
|
||||
}
|
||||
|
||||
const result = await executeRunWorkflow(
|
||||
{ workflowId: 'workflow-1', useMockPayload: true },
|
||||
context
|
||||
)
|
||||
|
||||
expect(result.success).toBe(true)
|
||||
expect(registry.isComplete()).toBe(false)
|
||||
})
|
||||
|
||||
it('filters and anonymizes cross-workspace child provenance to values that cross back', async () => {
|
||||
const registry = new ResolvedSecretTraceRegistry([], {
|
||||
userId: 'user-1',
|
||||
workspaceId: 'workspace-1',
|
||||
})
|
||||
const context: ExecutionContext = {
|
||||
...executionContext,
|
||||
resolvedSecretTraceRegistry: registry,
|
||||
}
|
||||
ensureWorkflowAccessMock.mockResolvedValueOnce({
|
||||
workflow: {
|
||||
id: 'workflow-2',
|
||||
userId: 'owner-2',
|
||||
workspaceId: 'workspace-2',
|
||||
variables: {},
|
||||
},
|
||||
})
|
||||
resolveBillingAttributionMock.mockResolvedValueOnce(childBillingAttribution)
|
||||
decryptSecretMock.mockImplementation(async (encryptedValue: string) => ({
|
||||
decrypted: encryptedValue === 'used-ciphertext' ? 'used-secret' : 'workspace-only-secret',
|
||||
}))
|
||||
executeWorkflowMock.mockResolvedValueOnce({
|
||||
success: true,
|
||||
output: { value: 'used-secret' },
|
||||
logs: [],
|
||||
metadata: { executionId: 'child-execution' },
|
||||
executionState: {
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ name: 'USED', encryptedValue: 'used-ciphertext' },
|
||||
{ name: 'WORKSPACE_ONLY', encryptedValue: 'workspace-only-ciphertext' },
|
||||
],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-2' },
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
const result = await executeRunWorkflow(
|
||||
{ workflowId: 'workflow-2', useMockPayload: true },
|
||||
context
|
||||
)
|
||||
|
||||
expect(result).toMatchObject({
|
||||
success: true,
|
||||
output: { output: { value: 'used-secret' } },
|
||||
})
|
||||
expect(registry.getActiveMatches()).toEqual([
|
||||
{ plaintext: 'used-secret', replacement: ANONYMOUS_SECRET_TRACE_REPLACEMENT },
|
||||
])
|
||||
expect(JSON.stringify(result)).not.toContain('workspace-only-ciphertext')
|
||||
expect(JSON.stringify(result)).not.toContain('__resolvedSecretTraceProvenance')
|
||||
})
|
||||
|
||||
it('passes immutable attribution when running until a block', async () => {
|
||||
await expectBillingAttributionForwarded(() =>
|
||||
executeRunWorkflowUntilBlock(
|
||||
|
||||
@@ -97,7 +97,9 @@ async function executeCopilotWorkflowTarget(params: {
|
||||
)
|
||||
|
||||
try {
|
||||
return await executeWorkflow(
|
||||
const trustedInitialResolvedSecretTraceProvenance =
|
||||
params.context.resolvedSecretTraceRegistry?.exportProvenanceForValue(params.input)
|
||||
const result = await executeWorkflow(
|
||||
params.workflow,
|
||||
generateRequestId(),
|
||||
params.input,
|
||||
@@ -105,10 +107,34 @@ async function executeCopilotWorkflowTarget(params: {
|
||||
{
|
||||
...params.options,
|
||||
billingAttribution: admission.billingAttribution,
|
||||
...(trustedInitialResolvedSecretTraceProvenance
|
||||
? { trustedInitialResolvedSecretTraceProvenance }
|
||||
: {}),
|
||||
},
|
||||
childExecutionId
|
||||
)
|
||||
if (params.context.resolvedSecretTraceRegistry) {
|
||||
await params.context.resolvedSecretTraceRegistry.importCrossingProvenance(
|
||||
result.executionState?.resolvedSecretTraceProvenance,
|
||||
{ output: result.output, logs: result.logs, error: result.error },
|
||||
{ trusted: true }
|
||||
)
|
||||
}
|
||||
return result
|
||||
} catch (error) {
|
||||
if (params.context.resolvedSecretTraceRegistry) {
|
||||
const executionResult = hasExecutionResult(error) ? error.executionResult : undefined
|
||||
await params.context.resolvedSecretTraceRegistry.importCrossingProvenance(
|
||||
executionResult?.executionState?.resolvedSecretTraceProvenance,
|
||||
{
|
||||
output: executionResult?.output,
|
||||
logs: executionResult?.logs,
|
||||
error: executionResult?.error,
|
||||
thrownMessage: toError(error).message,
|
||||
},
|
||||
{ trusted: true }
|
||||
)
|
||||
}
|
||||
if (admission.targetReservation) {
|
||||
await releaseExecutionSlot(childExecutionId)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { dbChainMockFns, resetDbChainMock } from '@sim/testing'
|
||||
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { checkWorkspaceAccessMock, materializeExecutionDataForDisplayMock } = vi.hoisted(() => ({
|
||||
checkWorkspaceAccessMock: vi.fn(),
|
||||
materializeExecutionDataForDisplayMock: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/workspaces/permissions/utils', () => ({
|
||||
checkWorkspaceAccess: checkWorkspaceAccessMock,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/logs/execution/trace-store', () => ({
|
||||
materializeExecutionDataForDisplay: materializeExecutionDataForDisplayMock,
|
||||
}))
|
||||
|
||||
import { getJobLogsServerTool } from './get-job-logs'
|
||||
|
||||
const RAW_SECRET = 'sk-job-log-secret'
|
||||
const MASKED_SECRET = '{{OPENAI_API_KEY}}'
|
||||
const CONTEXT = { userId: 'user-1', workspaceId: 'workspace-1' }
|
||||
|
||||
function jobLogRow(overrides: Record<string, unknown> = {}) {
|
||||
return {
|
||||
id: 'log-1',
|
||||
executionId: 'execution-1',
|
||||
status: 'success',
|
||||
level: 'info',
|
||||
trigger: 'schedule',
|
||||
startedAt: new Date('2026-07-31T00:00:00.000Z'),
|
||||
endedAt: new Date('2026-07-31T00:00:01.000Z'),
|
||||
totalDurationMs: 1000,
|
||||
executionData: {
|
||||
finalOutput: { result: RAW_SECRET },
|
||||
traceSpans: [{ id: 'span-1', output: { result: RAW_SECRET } }],
|
||||
},
|
||||
cost: null,
|
||||
...overrides,
|
||||
}
|
||||
}
|
||||
|
||||
describe('getJobLogsServerTool', () => {
|
||||
afterAll(resetDbChainMock)
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
resetDbChainMock()
|
||||
checkWorkspaceAccessMock.mockResolvedValue({ hasAccess: true })
|
||||
})
|
||||
|
||||
it('returns the secret-safe log projection instead of raw successful output', async () => {
|
||||
const row = jobLogRow()
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([row])
|
||||
materializeExecutionDataForDisplayMock.mockResolvedValueOnce({
|
||||
finalOutput: { result: MASKED_SECRET },
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Function 1',
|
||||
type: 'function',
|
||||
status: 'success',
|
||||
duration: 1000,
|
||||
startTime: '2026-07-31T00:00:00.000Z',
|
||||
endTime: '2026-07-31T00:00:01.000Z',
|
||||
output: { result: MASKED_SECRET },
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const result = await getJobLogsServerTool.execute(
|
||||
{ jobId: 'schedule-1', includeDetails: true },
|
||||
CONTEXT
|
||||
)
|
||||
|
||||
expect(result).toEqual([
|
||||
expect.objectContaining({
|
||||
executionId: 'execution-1',
|
||||
output: { result: MASKED_SECRET },
|
||||
}),
|
||||
])
|
||||
expect(JSON.stringify(result)).not.toContain(RAW_SECRET)
|
||||
expect(materializeExecutionDataForDisplayMock).toHaveBeenCalledWith(row.executionData, {
|
||||
workspaceId: 'workspace-1',
|
||||
workflowId: null,
|
||||
executionId: 'execution-1',
|
||||
userId: 'user-1',
|
||||
})
|
||||
})
|
||||
|
||||
it('does not fall back to raw output or errors when provenance is incomplete', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([
|
||||
jobLogRow({
|
||||
status: 'error',
|
||||
executionData: {
|
||||
finalOutput: { error: RAW_SECRET },
|
||||
error: RAW_SECRET,
|
||||
traceSpans: [{ id: 'span-1', output: { error: RAW_SECRET } }],
|
||||
},
|
||||
}),
|
||||
])
|
||||
materializeExecutionDataForDisplayMock.mockResolvedValueOnce({
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Function 1',
|
||||
type: 'function',
|
||||
status: 'error',
|
||||
duration: 1000,
|
||||
startTime: '2026-07-31T00:00:00.000Z',
|
||||
endTime: '2026-07-31T00:00:01.000Z',
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
const result = await getJobLogsServerTool.execute(
|
||||
{ jobId: 'schedule-1', includeDetails: true },
|
||||
CONTEXT
|
||||
)
|
||||
|
||||
expect(result).toEqual([
|
||||
expect.not.objectContaining({
|
||||
output: expect.anything(),
|
||||
error: expect.anything(),
|
||||
}),
|
||||
])
|
||||
expect(JSON.stringify(result)).not.toContain(RAW_SECRET)
|
||||
})
|
||||
})
|
||||
@@ -4,6 +4,7 @@ import { createLogger } from '@sim/logger'
|
||||
import { and, desc, eq } from 'drizzle-orm'
|
||||
import { GetScheduledTaskLogs } from '@/lib/copilot/generated/tool-catalog-v1'
|
||||
import type { BaseServerTool, ServerToolContext } from '@/lib/copilot/tools/server/base-tool'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
import type { TraceSpan } from '@/lib/logs/types'
|
||||
import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils'
|
||||
|
||||
@@ -188,34 +189,47 @@ export const getJobLogsServerTool: BaseServerTool<GetJobLogsArgs, JobLogEntry[]>
|
||||
.orderBy(desc(jobExecutionLogs.startedAt))
|
||||
.limit(executionId ? 1 : clampedLimit)
|
||||
|
||||
const entries: JobLogEntry[] = rows.map((row) => {
|
||||
const executionData = row.executionData as any
|
||||
const details = includeDetails ? extractOutputAndError(executionData) : null
|
||||
const entries: JobLogEntry[] = await Promise.all(
|
||||
rows.map(async (row) => {
|
||||
const executionData = await materializeExecutionDataForDisplay(
|
||||
row.executionData as Record<string, unknown> | null,
|
||||
{
|
||||
workspaceId: wsId,
|
||||
workflowId: null,
|
||||
executionId: row.executionId,
|
||||
userId: context.userId,
|
||||
}
|
||||
)
|
||||
const details = includeDetails ? extractOutputAndError(executionData) : null
|
||||
|
||||
const entry: JobLogEntry = {
|
||||
executionId: row.executionId,
|
||||
status: row.status,
|
||||
trigger: row.trigger,
|
||||
startedAt: row.startedAt.toISOString(),
|
||||
endedAt: row.endedAt ? row.endedAt.toISOString() : null,
|
||||
durationMs: row.totalDurationMs ?? null,
|
||||
}
|
||||
|
||||
if (details) {
|
||||
if (details.error) entry.error = details.error
|
||||
if (details.toolCalls.length > 0) entry.toolCalls = details.toolCalls
|
||||
if (details.output) entry.output = details.output
|
||||
if (details.cost) entry.cost = details.cost
|
||||
if (details.tokens) entry.tokens = details.tokens
|
||||
} else {
|
||||
const errorMsg = executionData?.error || executionData?.traceSpans?.[0]?.output?.error
|
||||
if (row.status === 'error' && errorMsg) {
|
||||
entry.error = typeof errorMsg === 'string' ? errorMsg : JSON.stringify(errorMsg)
|
||||
const entry: JobLogEntry = {
|
||||
executionId: row.executionId,
|
||||
status: row.status,
|
||||
trigger: row.trigger,
|
||||
startedAt: row.startedAt.toISOString(),
|
||||
endedAt: row.endedAt ? row.endedAt.toISOString() : null,
|
||||
durationMs: row.totalDurationMs ?? null,
|
||||
}
|
||||
}
|
||||
|
||||
return entry
|
||||
})
|
||||
if (details) {
|
||||
if (details.error) entry.error = details.error
|
||||
if (details.toolCalls.length > 0) entry.toolCalls = details.toolCalls
|
||||
if (details.output) entry.output = details.output
|
||||
if (details.cost) entry.cost = details.cost
|
||||
if (details.tokens) entry.tokens = details.tokens
|
||||
} else {
|
||||
const traceSpans = Array.isArray(executionData.traceSpans)
|
||||
? (executionData.traceSpans as TraceSpan[])
|
||||
: []
|
||||
const errorMsg = executionData.error || traceSpans[0]?.output?.error
|
||||
if (row.status === 'error' && errorMsg) {
|
||||
entry.error = typeof errorMsg === 'string' ? errorMsg : JSON.stringify(errorMsg)
|
||||
}
|
||||
}
|
||||
|
||||
return entry
|
||||
})
|
||||
)
|
||||
|
||||
logger.info('Job logs prepared', {
|
||||
jobId,
|
||||
|
||||
@@ -18,7 +18,6 @@
|
||||
|
||||
import { context, type Span, SpanStatusCode, trace } from '@opentelemetry/api'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { getErrorMessage, toError } from '@sim/utils/errors'
|
||||
import { TraceAttr } from '@/lib/copilot/generated/trace-attributes-v1'
|
||||
import type { TraceSpan } from '@/lib/logs/types'
|
||||
import { hostedKeyMetrics } from '@/lib/monitoring/metrics'
|
||||
@@ -335,7 +334,6 @@ export function createOTelSpansForWorkflowExecution(params: {
|
||||
endTime: string
|
||||
totalDurationMs: number
|
||||
status: 'success' | 'error'
|
||||
error?: string
|
||||
}): void {
|
||||
try {
|
||||
const tracer = getTracer()
|
||||
@@ -358,11 +356,8 @@ export function createOTelSpansForWorkflowExecution(params: {
|
||||
if (params.status === 'error') {
|
||||
rootSpan.setStatus({
|
||||
code: SpanStatusCode.ERROR,
|
||||
message: params.error || 'Workflow execution failed',
|
||||
message: 'Workflow execution failed',
|
||||
})
|
||||
if (params.error) {
|
||||
rootSpan.recordException(new Error(params.error))
|
||||
}
|
||||
} else {
|
||||
rootSpan.setStatus({ code: SpanStatusCode.OK })
|
||||
}
|
||||
@@ -387,52 +382,6 @@ export function createOTelSpansForWorkflowExecution(params: {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a real-time OpenTelemetry span for a block execution
|
||||
* Can be called from block handlers during execution for real-time tracing
|
||||
*/
|
||||
export async function traceBlockExecution<T>(
|
||||
blockType: string,
|
||||
blockId: string,
|
||||
blockName: string,
|
||||
fn: (span: Span) => Promise<T>
|
||||
): Promise<T> {
|
||||
const tracer = getTracer()
|
||||
|
||||
const blockMapping = BLOCK_TYPE_MAPPING[blockType] || {
|
||||
spanName: `block.${blockType}`,
|
||||
spanKind: 'internal',
|
||||
getAttributes: () => ({}),
|
||||
}
|
||||
|
||||
return tracer.startActiveSpan(
|
||||
blockMapping.spanName,
|
||||
{
|
||||
attributes: {
|
||||
[TraceAttr.BlockType]: blockType,
|
||||
[TraceAttr.BlockId]: blockId,
|
||||
[TraceAttr.BlockName]: blockName,
|
||||
},
|
||||
},
|
||||
async (span) => {
|
||||
try {
|
||||
const result = await fn(span)
|
||||
span.setStatus({ code: SpanStatusCode.OK })
|
||||
return result
|
||||
} catch (error) {
|
||||
span.setStatus({
|
||||
code: SpanStatusCode.ERROR,
|
||||
message: getErrorMessage(error, 'Block execution failed'),
|
||||
})
|
||||
span.recordException(toError(error))
|
||||
throw error
|
||||
} finally {
|
||||
span.end()
|
||||
}
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Track platform events (workflow creation, knowledge base operations, etc.)
|
||||
*/
|
||||
@@ -672,7 +621,6 @@ export const PlatformEvents = {
|
||||
blocksExecuted: number
|
||||
hasErrors: boolean
|
||||
totalCost?: number
|
||||
errorMessage?: string
|
||||
}) => {
|
||||
trackPlatformEvent('platform.workflow.executed', {
|
||||
'workflow.id': attrs.workflowId,
|
||||
@@ -682,7 +630,6 @@ export const PlatformEvents = {
|
||||
'execution.blocks_executed': attrs.blocksExecuted,
|
||||
'execution.has_errors': attrs.hasErrors,
|
||||
...(attrs.totalCost !== undefined && { 'execution.total_cost': attrs.totalCost }),
|
||||
...(attrs.errorMessage && { 'execution.error_message': attrs.errorMessage }),
|
||||
})
|
||||
},
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { dbReplica } from '@sim/db'
|
||||
import { jobExecutionLogs } from '@sim/db/schema'
|
||||
import { and, inArray, isNotNull } from 'drizzle-orm'
|
||||
import { MATERIALIZE_CONCURRENCY, mapWithConcurrency } from '@/lib/core/utils/concurrency'
|
||||
import {
|
||||
decodeTimeCursor,
|
||||
encodeTimeCursor,
|
||||
@@ -10,6 +11,7 @@ import {
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
|
||||
type JobLogRow = typeof jobExecutionLogs.$inferSelect
|
||||
|
||||
@@ -40,6 +42,16 @@ async function* pages(input: SourcePageInput): AsyncIterable<JobLogRow[]> {
|
||||
.limit(input.chunkSize)
|
||||
|
||||
if (rows.length === 0) return
|
||||
const displayExecutionData = await mapWithConcurrency(rows, MATERIALIZE_CONCURRENCY, (row) =>
|
||||
materializeExecutionDataForDisplay(row.executionData as Record<string, unknown> | null, {
|
||||
workspaceId: row.workspaceId,
|
||||
workflowId: null,
|
||||
executionId: row.executionId,
|
||||
})
|
||||
)
|
||||
for (let index = 0; index < rows.length; index += 1) {
|
||||
rows[index].executionData = displayExecutionData[index] as JobLogRow['executionData']
|
||||
}
|
||||
yield rows
|
||||
const last = rows[rows.length - 1]
|
||||
cursor = { ts: last.endedAt!.toISOString(), id: last.id }
|
||||
|
||||
@@ -11,7 +11,7 @@ import {
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
|
||||
type WorkflowLogRow = typeof workflowExecutionLogs.$inferSelect
|
||||
|
||||
@@ -55,7 +55,7 @@ async function* pages(input: SourcePageInput): AsyncIterable<WorkflowLogRow[]> {
|
||||
// Use the order-preserving returned array (the util's documented contract)
|
||||
// and write back, rather than mutating rows inside the mapper.
|
||||
const materialized = await mapWithConcurrency(rows, MATERIALIZE_CONCURRENCY, (row) =>
|
||||
materializeExecutionData(row.executionData as Record<string, unknown> | null, {
|
||||
materializeExecutionDataForDisplay(row.executionData as Record<string, unknown> | null, {
|
||||
workspaceId: row.workspaceId,
|
||||
workflowId: row.workflowId,
|
||||
executionId: row.executionId,
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { dbChainMockFns, encryptionMock, encryptionMockFns, resetDbChainMock } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
vi.unmock('@/lib/environment/utils')
|
||||
vi.mock('@/lib/core/security/encryption', () => encryptionMock)
|
||||
|
||||
import {
|
||||
getEffectiveDecryptedEnv,
|
||||
getEffectiveEnvironmentSnapshot,
|
||||
invalidateEffectiveDecryptedEnvCache,
|
||||
} from '@/lib/environment/utils'
|
||||
|
||||
describe('effective environment resolution cache', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
resetDbChainMock()
|
||||
encryptionMockFns.mockDecryptSecret.mockReset()
|
||||
encryptionMockFns.mockEncryptSecret.mockReset()
|
||||
invalidateEffectiveDecryptedEnvCache({ userId: 'user-1' })
|
||||
dbChainMockFns.limit.mockResolvedValue([{ variables: { API_KEY: 'encrypted-value' } }])
|
||||
encryptionMockFns.mockDecryptSecret.mockResolvedValue({ decrypted: 'runtime-value' })
|
||||
})
|
||||
|
||||
it('shares one atomic snapshot and returns defensive clones', async () => {
|
||||
const [decrypted, snapshot] = await Promise.all([
|
||||
getEffectiveDecryptedEnv('user-1'),
|
||||
getEffectiveEnvironmentSnapshot('user-1'),
|
||||
])
|
||||
|
||||
expect(decrypted).toEqual({ API_KEY: 'runtime-value' })
|
||||
expect(snapshot).toMatchObject({
|
||||
personalEncrypted: { API_KEY: 'encrypted-value' },
|
||||
personalDecrypted: { API_KEY: 'runtime-value' },
|
||||
})
|
||||
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledOnce()
|
||||
|
||||
decrypted.API_KEY = 'mutated-runtime'
|
||||
snapshot.personalEncrypted.API_KEY = 'mutated-ciphertext'
|
||||
snapshot.personalDecrypted.API_KEY = 'mutated-snapshot'
|
||||
snapshot.conflicts.push('MUTATED')
|
||||
|
||||
await expect(getEffectiveDecryptedEnv('user-1')).resolves.toEqual({
|
||||
API_KEY: 'runtime-value',
|
||||
})
|
||||
await expect(getEffectiveEnvironmentSnapshot('user-1')).resolves.toMatchObject({
|
||||
personalEncrypted: { API_KEY: 'encrypted-value' },
|
||||
personalDecrypted: { API_KEY: 'runtime-value' },
|
||||
conflicts: [],
|
||||
})
|
||||
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('evicts rejected loads and retries the canonical lookup', async () => {
|
||||
dbChainMockFns.limit.mockRejectedValueOnce(new Error('database unavailable'))
|
||||
|
||||
await expect(getEffectiveEnvironmentSnapshot('user-1')).rejects.toThrow('database unavailable')
|
||||
|
||||
dbChainMockFns.limit.mockResolvedValue([{ variables: { API_KEY: 'encrypted-value' } }])
|
||||
await expect(getEffectiveDecryptedEnv('user-1')).resolves.toEqual({
|
||||
API_KEY: 'runtime-value',
|
||||
})
|
||||
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('reloads the full snapshot after invalidation', async () => {
|
||||
await expect(getEffectiveDecryptedEnv('user-1')).resolves.toEqual({
|
||||
API_KEY: 'runtime-value',
|
||||
})
|
||||
|
||||
invalidateEffectiveDecryptedEnvCache({ userId: 'user-1' })
|
||||
dbChainMockFns.limit.mockResolvedValue([{ variables: { API_KEY: 'rotated-ciphertext' } }])
|
||||
encryptionMockFns.mockDecryptSecret.mockResolvedValue({ decrypted: 'rotated-runtime' })
|
||||
|
||||
await expect(getEffectiveEnvironmentSnapshot('user-1')).resolves.toMatchObject({
|
||||
personalEncrypted: { API_KEY: 'rotated-ciphertext' },
|
||||
personalDecrypted: { API_KEY: 'rotated-runtime' },
|
||||
})
|
||||
expect(encryptionMockFns.mockDecryptSecret).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
})
|
||||
@@ -14,26 +14,44 @@ import {
|
||||
import { checkWorkspaceAccess, type WorkspaceAccess } from '@/lib/workspaces/permissions/utils'
|
||||
|
||||
const logger = createLogger('EnvironmentUtils')
|
||||
const EFFECTIVE_DECRYPTED_ENV_CACHE_TTL_MS = 2_000
|
||||
const EFFECTIVE_DECRYPTED_ENV_CACHE_MAX_ENTRIES = 1_000
|
||||
const EFFECTIVE_ENVIRONMENT_CACHE_TTL_MS = 2_000
|
||||
const EFFECTIVE_ENVIRONMENT_CACHE_MAX_ENTRIES = 1_000
|
||||
|
||||
interface EffectiveDecryptedEnvCacheEntry {
|
||||
userId: string
|
||||
workspaceId?: string
|
||||
promise: Promise<Record<string, string>>
|
||||
export interface EnvironmentResolutionSnapshot {
|
||||
personalEncrypted: Record<string, string>
|
||||
workspaceEncrypted: Record<string, string>
|
||||
personalDecrypted: Record<string, string>
|
||||
workspaceDecrypted: Record<string, string>
|
||||
conflicts: string[]
|
||||
decryptionFailures: string[]
|
||||
}
|
||||
|
||||
const effectiveDecryptedEnvCache = new LRUCache<string, EffectiveDecryptedEnvCacheEntry>({
|
||||
max: EFFECTIVE_DECRYPTED_ENV_CACHE_MAX_ENTRIES,
|
||||
ttl: EFFECTIVE_DECRYPTED_ENV_CACHE_TTL_MS,
|
||||
interface EffectiveEnvironmentCacheEntry {
|
||||
userId: string
|
||||
workspaceId?: string
|
||||
promise: Promise<EnvironmentResolutionSnapshot>
|
||||
}
|
||||
|
||||
const effectiveEnvironmentCache = new LRUCache<string, EffectiveEnvironmentCacheEntry>({
|
||||
max: EFFECTIVE_ENVIRONMENT_CACHE_MAX_ENTRIES,
|
||||
ttl: EFFECTIVE_ENVIRONMENT_CACHE_TTL_MS,
|
||||
})
|
||||
|
||||
function getEffectiveDecryptedEnvCacheKey(userId: string, workspaceId?: string): string {
|
||||
function getEffectiveEnvironmentCacheKey(userId: string, workspaceId?: string): string {
|
||||
return JSON.stringify([userId, workspaceId ?? null])
|
||||
}
|
||||
|
||||
function cloneEnvVars(envVars: Record<string, string>): Record<string, string> {
|
||||
return { ...envVars }
|
||||
function cloneEnvironmentResolutionSnapshot(
|
||||
snapshot: EnvironmentResolutionSnapshot
|
||||
): EnvironmentResolutionSnapshot {
|
||||
return {
|
||||
personalEncrypted: { ...snapshot.personalEncrypted },
|
||||
workspaceEncrypted: { ...snapshot.workspaceEncrypted },
|
||||
personalDecrypted: { ...snapshot.personalDecrypted },
|
||||
workspaceDecrypted: { ...snapshot.workspaceDecrypted },
|
||||
conflicts: [...snapshot.conflicts],
|
||||
decryptionFailures: [...snapshot.decryptionFailures],
|
||||
}
|
||||
}
|
||||
|
||||
export function invalidateEffectiveDecryptedEnvCache(input: {
|
||||
@@ -43,13 +61,13 @@ export function invalidateEffectiveDecryptedEnvCache(input: {
|
||||
const { userId, workspaceId } = input
|
||||
if (!userId && !workspaceId) return
|
||||
|
||||
effectiveDecryptedEnvCache.forEach((entry, cacheKey) => {
|
||||
effectiveEnvironmentCache.forEach((entry, cacheKey) => {
|
||||
if (userId && entry.userId === userId) {
|
||||
effectiveDecryptedEnvCache.delete(cacheKey)
|
||||
effectiveEnvironmentCache.delete(cacheKey)
|
||||
return
|
||||
}
|
||||
if (workspaceId && entry.workspaceId === workspaceId) {
|
||||
effectiveDecryptedEnvCache.delete(cacheKey)
|
||||
effectiveEnvironmentCache.delete(cacheKey)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -94,14 +112,7 @@ export async function getPersonalAndWorkspaceEnv(
|
||||
userId: string,
|
||||
workspaceId?: string,
|
||||
options?: { workspaceAccess?: WorkspaceAccess }
|
||||
): Promise<{
|
||||
personalEncrypted: Record<string, string>
|
||||
workspaceEncrypted: Record<string, string>
|
||||
personalDecrypted: Record<string, string>
|
||||
workspaceDecrypted: Record<string, string>
|
||||
conflicts: string[]
|
||||
decryptionFailures: string[]
|
||||
}> {
|
||||
): Promise<EnvironmentResolutionSnapshot> {
|
||||
let workspaceCanAdmin = false
|
||||
if (workspaceId) {
|
||||
const access = options?.workspaceAccess ?? (await checkWorkspaceAccess(workspaceId, userId))
|
||||
@@ -357,6 +368,42 @@ export async function upsertWorkspaceEnvVars(
|
||||
return updatedKeys
|
||||
}
|
||||
|
||||
async function getCachedEnvironmentResolutionSnapshot(
|
||||
userId: string,
|
||||
workspaceId?: string
|
||||
): Promise<EnvironmentResolutionSnapshot> {
|
||||
const cacheKey = getEffectiveEnvironmentCacheKey(userId, workspaceId)
|
||||
const cached = effectiveEnvironmentCache.get(cacheKey)
|
||||
if (cached) {
|
||||
return cached.promise
|
||||
}
|
||||
|
||||
const promise = getPersonalAndWorkspaceEnv(userId, workspaceId).catch((error) => {
|
||||
effectiveEnvironmentCache.delete(cacheKey)
|
||||
throw error
|
||||
})
|
||||
|
||||
effectiveEnvironmentCache.set(cacheKey, {
|
||||
userId,
|
||||
workspaceId,
|
||||
promise,
|
||||
})
|
||||
|
||||
return promise
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a defensive clone of the cached environment snapshot used for runtime resolution.
|
||||
*/
|
||||
export async function getEffectiveEnvironmentSnapshot(
|
||||
userId: string,
|
||||
workspaceId?: string
|
||||
): Promise<EnvironmentResolutionSnapshot> {
|
||||
return cloneEnvironmentResolutionSnapshot(
|
||||
await getCachedEnvironmentResolutionSnapshot(userId, workspaceId)
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a merged decrypted env map for webhook/copilot/MCP config resolution.
|
||||
*/
|
||||
@@ -364,27 +411,9 @@ export async function getEffectiveDecryptedEnv(
|
||||
userId: string,
|
||||
workspaceId?: string
|
||||
): Promise<Record<string, string>> {
|
||||
const cacheKey = getEffectiveDecryptedEnvCacheKey(userId, workspaceId)
|
||||
const cached = effectiveDecryptedEnvCache.get(cacheKey)
|
||||
if (cached) {
|
||||
return cloneEnvVars(await cached.promise)
|
||||
}
|
||||
|
||||
const promise = getPersonalAndWorkspaceEnv(userId, workspaceId)
|
||||
.then(({ personalDecrypted, workspaceDecrypted }) => ({
|
||||
...personalDecrypted,
|
||||
...workspaceDecrypted,
|
||||
}))
|
||||
.catch((error) => {
|
||||
effectiveDecryptedEnvCache.delete(cacheKey)
|
||||
throw error
|
||||
})
|
||||
|
||||
effectiveDecryptedEnvCache.set(cacheKey, {
|
||||
const { personalDecrypted, workspaceDecrypted } = await getCachedEnvironmentResolutionSnapshot(
|
||||
userId,
|
||||
workspaceId,
|
||||
promise,
|
||||
})
|
||||
|
||||
return cloneEnvVars(await promise)
|
||||
workspaceId
|
||||
)
|
||||
return { ...personalDecrypted, ...workspaceDecrypted }
|
||||
}
|
||||
|
||||
@@ -23,6 +23,8 @@ interface CompactState {
|
||||
seen: WeakSet<object>
|
||||
}
|
||||
|
||||
const BLOCK_LOG_COMPACTION_CONCURRENCY = 4
|
||||
|
||||
function getJsonAndSize(value: unknown): { json: string; size: number } | null {
|
||||
try {
|
||||
const json = JSON.stringify(value)
|
||||
@@ -260,8 +262,15 @@ export async function compactBlockLogs(
|
||||
return logs
|
||||
}
|
||||
|
||||
return Promise.all(
|
||||
logs.map(async (log) => {
|
||||
const compactedLogs = new Array<BlockLog>(logs.length)
|
||||
let cursor = 0
|
||||
const worker = async (): Promise<void> => {
|
||||
while (true) {
|
||||
const index = cursor
|
||||
cursor += 1
|
||||
if (index >= logs.length) return
|
||||
|
||||
const log = logs[index]
|
||||
const compactedLog = { ...log }
|
||||
if ('input' in compactedLog) {
|
||||
compactedLog.input = await compactExecutionPayload(compactedLog.input, options)
|
||||
@@ -275,7 +284,12 @@ export async function compactBlockLogs(
|
||||
options
|
||||
)
|
||||
}
|
||||
return compactedLog
|
||||
})
|
||||
compactedLogs[index] = compactedLog
|
||||
}
|
||||
}
|
||||
|
||||
await Promise.all(
|
||||
Array.from({ length: Math.min(BLOCK_LOG_COMPACTION_CONCURRENCY, logs.length) }, worker)
|
||||
)
|
||||
return compactedLogs
|
||||
}
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
getPrivateToolMetadataField,
|
||||
isPrivateToolMetadataType,
|
||||
PRIVATE_TOOL_METADATA_REQUEST_HEADER,
|
||||
PRIVATE_TOOL_METADATA_RESPONSE_HEADER,
|
||||
RESOLVED_SECRET_NAMES_FIELD,
|
||||
RESOLVED_SECRET_NAMES_METADATA_V1,
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD,
|
||||
RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
requestsPrivateToolMetadata,
|
||||
responseHasPrivateToolMetadata,
|
||||
} from '@/lib/execution/private-tool-metadata'
|
||||
|
||||
describe('private tool metadata protocol', () => {
|
||||
it('keeps the versioned wire markers stable', () => {
|
||||
expect(PRIVATE_TOOL_METADATA_REQUEST_HEADER).toBe('x-sim-request-private-tool-metadata')
|
||||
expect(PRIVATE_TOOL_METADATA_RESPONSE_HEADER).toBe('x-sim-private-tool-metadata')
|
||||
expect(RESOLVED_SECRET_NAMES_METADATA_V1).toBe('resolved-secret-names-v1')
|
||||
expect(RESOLVED_SECRET_PROVENANCE_METADATA_V1).toBe('resolved-secret-provenance-v1')
|
||||
expect(RESOLVED_SECRET_NAMES_FIELD).toBe('__resolvedSecretNames')
|
||||
expect(RESOLVED_SECRET_PROVENANCE_FIELD).toBe('__resolvedSecretTraceProvenance')
|
||||
})
|
||||
|
||||
it('accepts only exact request and response markers', () => {
|
||||
const requestHeaders = new Headers({
|
||||
[PRIVATE_TOOL_METADATA_REQUEST_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1,
|
||||
})
|
||||
const responseHeaders = new Headers({
|
||||
[PRIVATE_TOOL_METADATA_RESPONSE_HEADER]: RESOLVED_SECRET_NAMES_METADATA_V1,
|
||||
})
|
||||
|
||||
expect(
|
||||
requestsPrivateToolMetadata(requestHeaders, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
|
||||
).toBe(true)
|
||||
expect(requestsPrivateToolMetadata(requestHeaders, RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(
|
||||
false
|
||||
)
|
||||
expect(responseHasPrivateToolMetadata(responseHeaders, RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(
|
||||
true
|
||||
)
|
||||
expect(
|
||||
responseHasPrivateToolMetadata(responseHeaders, RESOLVED_SECRET_PROVENANCE_METADATA_V1)
|
||||
).toBe(false)
|
||||
})
|
||||
|
||||
it('maps each marker to its private payload field', () => {
|
||||
expect(isPrivateToolMetadataType(RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(true)
|
||||
expect(isPrivateToolMetadataType(RESOLVED_SECRET_PROVENANCE_METADATA_V1)).toBe(true)
|
||||
expect(isPrivateToolMetadataType('resolved-secret-provenance-v2')).toBe(false)
|
||||
expect(isPrivateToolMetadataType(null)).toBe(false)
|
||||
expect(getPrivateToolMetadataField(RESOLVED_SECRET_NAMES_METADATA_V1)).toBe(
|
||||
RESOLVED_SECRET_NAMES_FIELD
|
||||
)
|
||||
expect(getPrivateToolMetadataField(RESOLVED_SECRET_PROVENANCE_METADATA_V1)).toBe(
|
||||
RESOLVED_SECRET_PROVENANCE_FIELD
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,44 @@
|
||||
export const PRIVATE_TOOL_METADATA_REQUEST_HEADER = 'x-sim-request-private-tool-metadata'
|
||||
export const PRIVATE_TOOL_METADATA_RESPONSE_HEADER = 'x-sim-private-tool-metadata'
|
||||
|
||||
export const RESOLVED_SECRET_NAMES_METADATA_V1 = 'resolved-secret-names-v1'
|
||||
export const RESOLVED_SECRET_PROVENANCE_METADATA_V1 = 'resolved-secret-provenance-v1'
|
||||
|
||||
export const RESOLVED_SECRET_NAMES_FIELD = '__resolvedSecretNames'
|
||||
export const RESOLVED_SECRET_PROVENANCE_FIELD = '__resolvedSecretTraceProvenance'
|
||||
|
||||
export type PrivateToolMetadataType =
|
||||
| typeof RESOLVED_SECRET_NAMES_METADATA_V1
|
||||
| typeof RESOLVED_SECRET_PROVENANCE_METADATA_V1
|
||||
|
||||
interface HeaderReader {
|
||||
get(name: string): string | null
|
||||
}
|
||||
|
||||
export function isPrivateToolMetadataType(value: string | null): value is PrivateToolMetadataType {
|
||||
return (
|
||||
value === RESOLVED_SECRET_NAMES_METADATA_V1 || value === RESOLVED_SECRET_PROVENANCE_METADATA_V1
|
||||
)
|
||||
}
|
||||
|
||||
export function requestsPrivateToolMetadata(
|
||||
headers: HeaderReader,
|
||||
expectedType: PrivateToolMetadataType
|
||||
): boolean {
|
||||
return headers.get(PRIVATE_TOOL_METADATA_REQUEST_HEADER) === expectedType
|
||||
}
|
||||
|
||||
export function responseHasPrivateToolMetadata(
|
||||
headers: HeaderReader,
|
||||
expectedType: PrivateToolMetadataType
|
||||
): boolean {
|
||||
return headers.get(PRIVATE_TOOL_METADATA_RESPONSE_HEADER) === expectedType
|
||||
}
|
||||
|
||||
export function getPrivateToolMetadataField(
|
||||
type: PrivateToolMetadataType
|
||||
): typeof RESOLVED_SECRET_NAMES_FIELD | typeof RESOLVED_SECRET_PROVENANCE_FIELD {
|
||||
return type === RESOLVED_SECRET_NAMES_METADATA_V1
|
||||
? RESOLVED_SECRET_NAMES_FIELD
|
||||
: RESOLVED_SECRET_PROVENANCE_FIELD
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import type { BlockLog } from '@/executor/types'
|
||||
|
||||
/** A server-projected BlockLog copy carrying presentation-only reconciliation hints. */
|
||||
export interface SecretSafeBlockLog extends BlockLog {
|
||||
clearLiveDisplay?: true
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
collectFunctionalBlockOutputs,
|
||||
FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE,
|
||||
FunctionalOutputsUnavailableError,
|
||||
getFunctionalBlockOutput,
|
||||
} from '@/lib/logs/execution/functional-outputs'
|
||||
|
||||
describe('functional execution outputs', () => {
|
||||
it('prefers raw execution state over projected TraceSpan output', () => {
|
||||
const data = {
|
||||
executionState: {
|
||||
blockStates: {
|
||||
'function-1': { output: { result: 1234 } },
|
||||
},
|
||||
},
|
||||
traceSpans: [
|
||||
{
|
||||
blockId: 'function-1',
|
||||
output: { result: '{{OPENAI_API_KEY}}' },
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
expect(getFunctionalBlockOutput(data, 'function-1')).toEqual({ result: 1234 })
|
||||
})
|
||||
|
||||
it('uses nested TraceSpan output for legacy rows without execution state', () => {
|
||||
const outputs = collectFunctionalBlockOutputs({
|
||||
traceSpans: [
|
||||
{
|
||||
children: [{ blockId: 'function-1', output: { result: 'legacy' } }],
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
expect(outputs.get('function-1')).toEqual({ result: 'legacy' })
|
||||
})
|
||||
|
||||
it('does not mix projected TraceSpan output into an available raw state', () => {
|
||||
const outputs = collectFunctionalBlockOutputs({
|
||||
executionState: {
|
||||
blockStates: {
|
||||
'function-1': { output: { result: 1234 } },
|
||||
},
|
||||
},
|
||||
traceSpans: [{ blockId: 'function-2', output: { result: '{{OPENAI_API_KEY}}' } }],
|
||||
})
|
||||
|
||||
expect(outputs.get('function-1')).toEqual({ result: 1234 })
|
||||
expect(outputs.has('function-2')).toBe(false)
|
||||
})
|
||||
|
||||
it('fails instead of returning projected traces when raw state was truncated', () => {
|
||||
expect(() =>
|
||||
collectFunctionalBlockOutputs({
|
||||
executionDataTruncated: true,
|
||||
traceSpans: [{ blockId: 'function-1', output: { result: '{{OPENAI_API_KEY}}' } }],
|
||||
})
|
||||
).toThrowError(new FunctionalOutputsUnavailableError())
|
||||
expect(FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE).toContain('could not be retained in full')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,62 @@
|
||||
export interface FunctionalTraceSpanSource {
|
||||
blockId?: string
|
||||
output?: unknown
|
||||
children?: FunctionalTraceSpanSource[]
|
||||
}
|
||||
|
||||
export interface FunctionalExecutionDataSource {
|
||||
traceSpans?: FunctionalTraceSpanSource[]
|
||||
executionState?: {
|
||||
blockStates?: Record<string, { output?: unknown }>
|
||||
}
|
||||
executionDataTruncated?: boolean
|
||||
}
|
||||
|
||||
export const FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE =
|
||||
'Raw block outputs are unavailable because the execution data could not be retained in full.'
|
||||
|
||||
export class FunctionalOutputsUnavailableError extends Error {
|
||||
constructor() {
|
||||
super(FUNCTIONAL_OUTPUTS_UNAVAILABLE_MESSAGE)
|
||||
this.name = 'FunctionalOutputsUnavailableError'
|
||||
}
|
||||
}
|
||||
|
||||
function collectTraceBlockOutputs(
|
||||
spans: FunctionalTraceSpanSource[] | undefined,
|
||||
outputs: Map<string, unknown>
|
||||
): void {
|
||||
if (!spans) return
|
||||
for (const span of spans) {
|
||||
if (span.blockId && span.output !== undefined && !outputs.has(span.blockId)) {
|
||||
outputs.set(span.blockId, span.output)
|
||||
}
|
||||
collectTraceBlockOutputs(span.children, outputs)
|
||||
}
|
||||
}
|
||||
|
||||
/** Returns functional block outputs, preferring raw execution state over display TraceSpans. */
|
||||
export function collectFunctionalBlockOutputs(
|
||||
data: FunctionalExecutionDataSource | undefined
|
||||
): Map<string, unknown> {
|
||||
const outputs = new Map<string, unknown>()
|
||||
const blockStates = data?.executionState?.blockStates
|
||||
if (blockStates) {
|
||||
for (const [blockId, blockState] of Object.entries(blockStates)) {
|
||||
if (blockState?.output !== undefined) outputs.set(blockId, blockState.output)
|
||||
}
|
||||
return outputs
|
||||
}
|
||||
|
||||
if (data?.executionDataTruncated) throw new FunctionalOutputsUnavailableError()
|
||||
|
||||
collectTraceBlockOutputs(data?.traceSpans, outputs)
|
||||
return outputs
|
||||
}
|
||||
|
||||
export function getFunctionalBlockOutput(
|
||||
data: FunctionalExecutionDataSource | undefined,
|
||||
blockId: string
|
||||
): unknown {
|
||||
return collectFunctionalBlockOutputs(data).get(blockId)
|
||||
}
|
||||
@@ -3,6 +3,8 @@ import { dbChainMockFns, queueTableRows, resetDbChainMock } from '@sim/testing'
|
||||
import { afterAll, beforeEach, describe, expect, test, vi } from 'vitest'
|
||||
import { recordUsage } from '@/lib/billing/core/usage-log'
|
||||
import { ExecutionLogger } from '@/lib/logs/execution/logger'
|
||||
import type { WorkflowExecutionLog } from '@/lib/logs/types'
|
||||
import type { SerializableExecutionState } from '@/executor/execution/types'
|
||||
|
||||
afterAll(resetDbChainMock)
|
||||
|
||||
@@ -242,6 +244,56 @@ describe('ExecutionLogger', () => {
|
||||
expect(completedData.billingAttribution).toEqual(billingAttribution)
|
||||
})
|
||||
|
||||
test('preserves server-only lifecycle metadata after execution-state PII masking', () => {
|
||||
const loggerInstance = new ExecutionLogger() as unknown as {
|
||||
preservePrivateExecutionStateMetadata(
|
||||
redactedState: SerializableExecutionState | undefined,
|
||||
originalState: SerializableExecutionState | undefined
|
||||
): SerializableExecutionState | undefined
|
||||
}
|
||||
const provenance = {
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'API_SECRET', encryptedValue: 'enc:original-ciphertext' }],
|
||||
}
|
||||
const trustedLargeValueAccess = {
|
||||
executionIds: ['execution-1'],
|
||||
largeValueKeys: ['execution/workspace-1/workflow-1/execution-1/value.json'],
|
||||
fileKeys: ['workspace-1/file-1'],
|
||||
}
|
||||
const originalState: SerializableExecutionState = {
|
||||
blockStates: {},
|
||||
executedBlocks: [],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: [],
|
||||
resolvedSecretTraceProvenance: provenance,
|
||||
trustedLargeValueAccess,
|
||||
}
|
||||
const redactedState: SerializableExecutionState = {
|
||||
...originalState,
|
||||
resolvedSecretTraceProvenance: {
|
||||
...provenance,
|
||||
entries: [{ name: 'API_SECRET', encryptedValue: '[MASKED]' }],
|
||||
},
|
||||
trustedLargeValueAccess: {
|
||||
executionIds: [],
|
||||
largeValueKeys: [],
|
||||
fileKeys: [],
|
||||
},
|
||||
}
|
||||
|
||||
const preserved = loggerInstance.preservePrivateExecutionStateMetadata(
|
||||
redactedState,
|
||||
originalState
|
||||
)
|
||||
|
||||
expect(preserved?.resolvedSecretTraceProvenance).toBe(provenance)
|
||||
expect(preserved?.trustedLargeValueAccess).toBe(trustedLargeValueAccess)
|
||||
expect(preserved?.blockStates).toBe(redactedState.blockStates)
|
||||
})
|
||||
|
||||
test('summarizes oversized execution data before storage', () => {
|
||||
const loggerInstance = new ExecutionLogger() as any
|
||||
const largePayload = 'x'.repeat(1_100_000)
|
||||
@@ -325,12 +377,71 @@ describe('ExecutionLogger', () => {
|
||||
activeExecutionPathLength: 0,
|
||||
pendingQueueLength: 0,
|
||||
})
|
||||
expect(compacted.traceSpans?.[0]?.children?.[0]?.input).toEqual({
|
||||
_truncated: true,
|
||||
reason: 'execution_data_size_limit',
|
||||
originalBytes: expect.any(Number),
|
||||
summary: 'object with 2 keys',
|
||||
expect(compacted.traceSpans?.[0]?.children?.[0]).not.toHaveProperty('input')
|
||||
})
|
||||
|
||||
test('retains tool-call structure when aggregate trace content exceeds the compaction cap', () => {
|
||||
const loggerInstance = new ExecutionLogger() as unknown as {
|
||||
compactExecutionDataForStorage(
|
||||
executionData: WorkflowExecutionLog['executionData'],
|
||||
executionId: string
|
||||
): WorkflowExecutionLog['executionData']
|
||||
}
|
||||
const oversizedContent = 'x'.repeat(9_000)
|
||||
const modelToolCalls = Array.from({ length: 200 }, (_, index) => ({
|
||||
id: `model-call-${index}-${'m'.repeat(40)}`,
|
||||
name: 'lookup',
|
||||
arguments: oversizedContent,
|
||||
}))
|
||||
const toolCalls = Array.from({ length: 200 }, (_, index) => ({
|
||||
id: `legacy-call-${index}-${'l'.repeat(40)}`,
|
||||
name: 'legacy_lookup',
|
||||
duration: 1,
|
||||
startTime: '2025-01-01T00:00:00.000Z',
|
||||
endTime: '2025-01-01T00:00:00.001Z',
|
||||
status: 'success' as const,
|
||||
input: oversizedContent,
|
||||
output: oversizedContent,
|
||||
error: oversizedContent,
|
||||
}))
|
||||
|
||||
const compacted = loggerInstance.compactExecutionDataForStorage(
|
||||
{
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Agent',
|
||||
type: 'agent',
|
||||
duration: 1,
|
||||
startTime: '2025-01-01T00:00:00.000Z',
|
||||
endTime: '2025-01-01T00:00:00.001Z',
|
||||
status: 'success',
|
||||
modelToolCalls,
|
||||
toolCalls,
|
||||
},
|
||||
],
|
||||
finalOutput: { data: 'y'.repeat(1_100_000) },
|
||||
},
|
||||
'execution-tool-structure'
|
||||
)
|
||||
|
||||
expect(compacted.executionDataTruncated).toBe(true)
|
||||
expect(compacted.traceSpans?.[0]?.modelToolCalls).toHaveLength(modelToolCalls.length)
|
||||
expect(compacted.traceSpans?.[0]?.modelToolCalls?.[0]).toEqual({
|
||||
id: modelToolCalls[0].id,
|
||||
name: 'lookup',
|
||||
})
|
||||
expect(compacted.traceSpans?.[0]?.toolCalls).toHaveLength(toolCalls.length)
|
||||
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).toEqual(
|
||||
expect.objectContaining({
|
||||
id: toolCalls[0].id,
|
||||
name: 'legacy_lookup',
|
||||
status: 'success',
|
||||
})
|
||||
)
|
||||
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).not.toHaveProperty('input')
|
||||
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).not.toHaveProperty('output')
|
||||
expect(compacted.traceSpans?.[0]?.toolCalls?.[0]).not.toHaveProperty('error')
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -51,8 +51,9 @@ import {
|
||||
import { snapshotService } from '@/lib/logs/execution/snapshot/service'
|
||||
import { traceSpansIndicateFailure } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
import {
|
||||
copyTraceSpansWithoutCosts,
|
||||
externalizeExecutionData,
|
||||
stripSpanCosts,
|
||||
materializeExecutionData,
|
||||
TRACE_STORE_REF_KEY,
|
||||
} from '@/lib/logs/execution/trace-store'
|
||||
import type {
|
||||
@@ -186,13 +187,87 @@ function summarizeValueForExecutionData(value: unknown, maxBytes: number): unkno
|
||||
}
|
||||
}
|
||||
|
||||
function summarizeTextForExecutionData(value: string | undefined): string | undefined {
|
||||
if (!value) return value
|
||||
const size = getJsonByteSize(value, MAX_TRACE_IO_BYTES)
|
||||
if (size === undefined || size <= MAX_TRACE_IO_BYTES) {
|
||||
return value
|
||||
function retainBoundedTraceContent<T>(value: T, maxBytes = MAX_TRACE_IO_BYTES): T | undefined {
|
||||
const size = getJsonByteSize(value, maxBytes)
|
||||
return size !== undefined && size <= maxBytes ? value : undefined
|
||||
}
|
||||
|
||||
function stripModelToolCallArguments(
|
||||
calls: NonNullable<TraceSpan['modelToolCalls']>
|
||||
): NonNullable<TraceSpan['modelToolCalls']> {
|
||||
return calls.map(({ arguments: _arguments, ...call }) => call as (typeof calls)[number])
|
||||
}
|
||||
|
||||
function compactModelToolCalls(
|
||||
calls: NonNullable<TraceSpan['modelToolCalls']>
|
||||
): NonNullable<TraceSpan['modelToolCalls']> | undefined {
|
||||
const compacted = calls.map(({ arguments: callArguments, ...call }) => {
|
||||
const retainedArguments = retainBoundedTraceContent(callArguments)
|
||||
return {
|
||||
...call,
|
||||
...(retainedArguments !== undefined ? { arguments: retainedArguments } : {}),
|
||||
} as (typeof calls)[number]
|
||||
})
|
||||
return retainBoundedTraceContent(compacted)
|
||||
}
|
||||
|
||||
function compactLegacyToolCalls(
|
||||
calls: NonNullable<TraceSpan['toolCalls']>
|
||||
): NonNullable<TraceSpan['toolCalls']> | undefined {
|
||||
const compacted = calls.map(({ input, output, error, ...call }) => ({
|
||||
...call,
|
||||
...(retainBoundedTraceContent(input) !== undefined ? { input } : {}),
|
||||
...(retainBoundedTraceContent(output) !== undefined ? { output } : {}),
|
||||
...(retainBoundedTraceContent(error) !== undefined ? { error } : {}),
|
||||
}))
|
||||
return retainBoundedTraceContent(compacted)
|
||||
}
|
||||
|
||||
function stripLegacyToolCallContent(
|
||||
calls: NonNullable<TraceSpan['toolCalls']>
|
||||
): NonNullable<TraceSpan['toolCalls']> {
|
||||
return calls.map(({ input: _input, output: _output, error: _error, ...call }) => call)
|
||||
}
|
||||
|
||||
function compactProviderTiming(
|
||||
providerTiming: NonNullable<TraceSpan['providerTiming']>
|
||||
): NonNullable<TraceSpan['providerTiming']> {
|
||||
return {
|
||||
...providerTiming,
|
||||
segments: providerTiming.segments.map(
|
||||
({ assistantContent, thinkingContent, errorMessage, toolCalls, ...segment }) => ({
|
||||
...segment,
|
||||
...(retainBoundedTraceContent(assistantContent) !== undefined ? { assistantContent } : {}),
|
||||
...(retainBoundedTraceContent(thinkingContent) !== undefined ? { thinkingContent } : {}),
|
||||
...(retainBoundedTraceContent(errorMessage) !== undefined ? { errorMessage } : {}),
|
||||
...(toolCalls
|
||||
? {
|
||||
toolCalls: compactModelToolCalls(toolCalls) ?? stripModelToolCallArguments(toolCalls),
|
||||
}
|
||||
: {}),
|
||||
})
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
function stripProviderTimingContent(
|
||||
providerTiming: NonNullable<TraceSpan['providerTiming']>
|
||||
): NonNullable<TraceSpan['providerTiming']> {
|
||||
return {
|
||||
...providerTiming,
|
||||
segments: providerTiming.segments.map(
|
||||
({
|
||||
assistantContent: _assistantContent,
|
||||
thinkingContent: _thinkingContent,
|
||||
errorMessage: _errorMessage,
|
||||
toolCalls,
|
||||
...segment
|
||||
}) => ({
|
||||
...segment,
|
||||
...(toolCalls ? { toolCalls: stripModelToolCallArguments(toolCalls) } : {}),
|
||||
})
|
||||
),
|
||||
}
|
||||
return `[Truncated ${size} byte text value due to execution log size limit]`
|
||||
}
|
||||
|
||||
function summarizeTraceSpansForExecutionData(traceSpans?: TraceSpan[]): TraceSpan[] | undefined {
|
||||
@@ -201,33 +276,39 @@ function summarizeTraceSpansForExecutionData(traceSpans?: TraceSpan[]): TraceSpa
|
||||
}
|
||||
|
||||
return traceSpans.map((span) => {
|
||||
const { input, output, children, thinking, modelToolCalls, ...rest } = span
|
||||
const {
|
||||
input,
|
||||
output,
|
||||
children,
|
||||
thinking,
|
||||
errorMessage,
|
||||
modelToolCalls,
|
||||
toolCalls,
|
||||
providerTiming,
|
||||
...rest
|
||||
} = span
|
||||
const summarized: TraceSpan = { ...rest }
|
||||
|
||||
if (input !== undefined) {
|
||||
summarized.input = summarizeValueForExecutionData(input, MAX_TRACE_IO_BYTES) as Record<
|
||||
string,
|
||||
unknown
|
||||
>
|
||||
}
|
||||
if (output !== undefined) {
|
||||
summarized.output = summarizeValueForExecutionData(output, MAX_TRACE_IO_BYTES) as Record<
|
||||
string,
|
||||
unknown
|
||||
>
|
||||
}
|
||||
const retainedInput = retainBoundedTraceContent(input)
|
||||
if (retainedInput !== undefined) summarized.input = retainedInput
|
||||
const retainedOutput = retainBoundedTraceContent(output)
|
||||
if (retainedOutput !== undefined) summarized.output = retainedOutput
|
||||
if (children?.length) {
|
||||
summarized.children = summarizeTraceSpansForExecutionData(children)
|
||||
}
|
||||
if (thinking !== undefined) {
|
||||
summarized.thinking = summarizeTextForExecutionData(thinking)
|
||||
const retainedThinking = retainBoundedTraceContent(thinking)
|
||||
if (retainedThinking !== undefined) summarized.thinking = retainedThinking
|
||||
const retainedError = retainBoundedTraceContent(errorMessage)
|
||||
if (retainedError !== undefined) summarized.errorMessage = retainedError
|
||||
if (modelToolCalls) {
|
||||
summarized.modelToolCalls =
|
||||
compactModelToolCalls(modelToolCalls) ?? stripModelToolCallArguments(modelToolCalls)
|
||||
}
|
||||
if (
|
||||
modelToolCalls !== undefined &&
|
||||
(getJsonByteSize(modelToolCalls, MAX_TRACE_IO_BYTES) ?? 0) <= MAX_TRACE_IO_BYTES
|
||||
) {
|
||||
summarized.modelToolCalls = modelToolCalls
|
||||
if (toolCalls) {
|
||||
summarized.toolCalls =
|
||||
compactLegacyToolCalls(toolCalls) ?? stripLegacyToolCallContent(toolCalls)
|
||||
}
|
||||
if (providerTiming) summarized.providerTiming = compactProviderTiming(providerTiming)
|
||||
|
||||
return summarized
|
||||
})
|
||||
@@ -244,11 +325,17 @@ function summarizeTraceSpansWithoutIo(traceSpans?: TraceSpan[]): TraceSpan[] | u
|
||||
output: _output,
|
||||
children,
|
||||
thinking: _thinking,
|
||||
modelToolCalls: _modelToolCalls,
|
||||
errorMessage: _errorMessage,
|
||||
modelToolCalls,
|
||||
toolCalls,
|
||||
providerTiming,
|
||||
...rest
|
||||
} = span
|
||||
return {
|
||||
...rest,
|
||||
...(modelToolCalls ? { modelToolCalls: stripModelToolCallArguments(modelToolCalls) } : {}),
|
||||
...(toolCalls ? { toolCalls: stripLegacyToolCallContent(toolCalls) } : {}),
|
||||
...(providerTiming ? { providerTiming: stripProviderTimingContent(providerTiming) } : {}),
|
||||
...(children?.length ? { children: summarizeTraceSpansWithoutIo(children) } : {}),
|
||||
}
|
||||
})
|
||||
@@ -698,6 +785,73 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
})
|
||||
}
|
||||
|
||||
/** Restores server-only lifecycle metadata after broad execution-state PII masking. */
|
||||
private preservePrivateExecutionStateMetadata(
|
||||
redactedState: SerializableExecutionState | undefined,
|
||||
originalState: SerializableExecutionState | undefined
|
||||
): SerializableExecutionState | undefined {
|
||||
const provenance = originalState?.resolvedSecretTraceProvenance
|
||||
const trustedLargeValueAccess = originalState?.trustedLargeValueAccess
|
||||
return redactedState
|
||||
? {
|
||||
...redactedState,
|
||||
...(provenance !== undefined ? { resolvedSecretTraceProvenance: provenance } : {}),
|
||||
...(trustedLargeValueAccess !== undefined ? { trustedLargeValueAccess } : {}),
|
||||
}
|
||||
: redactedState
|
||||
}
|
||||
|
||||
async loadTraceSpansForProjection(params: {
|
||||
executionId: string
|
||||
workflowId: string
|
||||
workspaceId: string | null
|
||||
traceSpans: TraceSpan[]
|
||||
isResume?: boolean
|
||||
}): Promise<TraceSpan[]> {
|
||||
let sourceSpans = params.traceSpans
|
||||
if (params.isResume && sourceSpans.length === 0) {
|
||||
const [existingLog] = await execDb
|
||||
.select({ executionData: workflowExecutionLogs.executionData })
|
||||
.from(workflowExecutionLogs)
|
||||
.where(eq(workflowExecutionLogs.executionId, params.executionId))
|
||||
.limit(1)
|
||||
const executionData = await materializeExecutionData(
|
||||
existingLog?.executionData as Record<string, unknown> | null,
|
||||
{
|
||||
workspaceId: params.workspaceId,
|
||||
workflowId: params.workflowId,
|
||||
executionId: params.executionId,
|
||||
}
|
||||
)
|
||||
if (Array.isArray(executionData.traceSpans)) {
|
||||
sourceSpans = executionData.traceSpans as TraceSpan[]
|
||||
}
|
||||
}
|
||||
|
||||
return sourceSpans
|
||||
}
|
||||
|
||||
async prepareTraceSpansForProjection(params: {
|
||||
workflowId: string
|
||||
executionId: string
|
||||
workspaceId: string | null
|
||||
userId?: string | null
|
||||
traceSpans: TraceSpan[]
|
||||
}): Promise<TraceSpan[]> {
|
||||
const filtered = filterForDisplay(params.traceSpans)
|
||||
const redacted = redactApiKeys(filtered)
|
||||
const pii = await this.applyPiiRedaction(
|
||||
params.workspaceId,
|
||||
{ traceSpans: redacted },
|
||||
{
|
||||
workflowId: params.workflowId,
|
||||
executionId: params.executionId,
|
||||
userId: params.userId ?? undefined,
|
||||
}
|
||||
)
|
||||
return pii.traceSpans as TraceSpan[]
|
||||
}
|
||||
|
||||
async completeWorkflowExecution(params: {
|
||||
executionId: string
|
||||
endedAt: string
|
||||
@@ -785,11 +939,7 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
const status = statusOverride ?? (hasErrors ? 'failed' : 'completed')
|
||||
|
||||
// For resume executions, rebuild trace spans from the aggregated logs
|
||||
const mergedTraceSpans = isResume
|
||||
? traceSpans && traceSpans.length > 0
|
||||
? traceSpans
|
||||
: existingExecutionData?.traceSpans || []
|
||||
: traceSpans
|
||||
const mergedTraceSpans = traceSpans
|
||||
|
||||
const executionCost = {
|
||||
total: costSummary.totalCost,
|
||||
@@ -826,13 +976,12 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
builtExecutionData.workflowInput
|
||||
)
|
||||
|
||||
const filteredTraceSpans = filterForDisplay(builtExecutionData.traceSpans)
|
||||
const preparedTraceSpans = builtExecutionData.traceSpans
|
||||
const filteredFinalOutput = filterForDisplay(builtExecutionData.finalOutput)
|
||||
const filteredWorkflowInput =
|
||||
builtExecutionData.workflowInput !== undefined
|
||||
? filterForDisplay(builtExecutionData.workflowInput)
|
||||
: undefined
|
||||
const redactedTraceSpans = redactApiKeys(filteredTraceSpans)
|
||||
const redactedFinalOutput = redactApiKeys(filteredFinalOutput)
|
||||
const redactedWorkflowInput =
|
||||
filteredWorkflowInput !== undefined ? redactApiKeys(filteredWorkflowInput) : undefined
|
||||
@@ -840,7 +989,7 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
const pii = await this.applyPiiRedaction(
|
||||
existingLog?.workspaceId ?? null,
|
||||
{
|
||||
traceSpans: redactedTraceSpans,
|
||||
traceSpans: [],
|
||||
finalOutput: redactedFinalOutput,
|
||||
...(redactedWorkflowInput !== undefined ? { workflowInput: redactedWorkflowInput } : {}),
|
||||
...(builtExecutionData.error !== undefined ? { error: builtExecutionData.error } : {}),
|
||||
@@ -899,9 +1048,14 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
? Math.max(0, Math.round(rawDurationMs))
|
||||
: 0
|
||||
|
||||
const safeExecutionState = this.preservePrivateExecutionStateMetadata(
|
||||
pii.executionState as SerializableExecutionState | undefined,
|
||||
builtExecutionData.executionState
|
||||
)
|
||||
|
||||
const cleanExecutionData: ExecutionData = {
|
||||
...builtExecutionData,
|
||||
traceSpans: pii.traceSpans as TraceSpan[],
|
||||
traceSpans: copyTraceSpansWithoutCosts(preparedTraceSpans),
|
||||
finalOutput: pii.finalOutput as BlockOutputData,
|
||||
...(pii.workflowInput !== undefined ? { workflowInput: pii.workflowInput } : {}),
|
||||
...(pii.error !== undefined ? { error: pii.error as string } : {}),
|
||||
@@ -909,9 +1063,7 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
? { completionFailure: pii.completionFailure as string }
|
||||
: {}),
|
||||
...(pii.trigger !== undefined ? { trigger: pii.trigger as ExecutionTrigger } : {}),
|
||||
...(pii.executionState !== undefined
|
||||
? { executionState: pii.executionState as SerializableExecutionState }
|
||||
: {}),
|
||||
...(safeExecutionState !== undefined ? { executionState: safeExecutionState } : {}),
|
||||
...(pii.environment !== undefined
|
||||
? { environment: pii.environment as ExecutionEnvironment }
|
||||
: {}),
|
||||
@@ -920,8 +1072,6 @@ export class ExecutionLogger implements IExecutionLoggerService {
|
||||
: {}),
|
||||
}
|
||||
|
||||
stripSpanCosts((cleanExecutionData as Record<string, unknown>).traceSpans)
|
||||
|
||||
// Bounded in-memory form. Returned to callers (notification delivery/events)
|
||||
// and reused as the inline-storage fallback below. This is a no-op for
|
||||
// payloads already within MAX_EXECUTION_DATA_BYTES.
|
||||
|
||||
@@ -9,14 +9,27 @@ const dbMocks = vi.hoisted(() => ({
|
||||
|
||||
const {
|
||||
completeWorkflowExecutionMock,
|
||||
loadTraceSpansForProjectionMock,
|
||||
prepareTraceSpansForProjectionMock,
|
||||
startWorkflowExecutionMock,
|
||||
loadWorkflowStateForExecutionMock,
|
||||
releaseExecutionSlotMock,
|
||||
createOTelSpansMock,
|
||||
workflowExecutedMock,
|
||||
} = vi.hoisted(() => ({
|
||||
completeWorkflowExecutionMock: vi.fn(),
|
||||
loadTraceSpansForProjectionMock: vi.fn(),
|
||||
prepareTraceSpansForProjectionMock: vi.fn(),
|
||||
startWorkflowExecutionMock: vi.fn(),
|
||||
loadWorkflowStateForExecutionMock: vi.fn(),
|
||||
releaseExecutionSlotMock: vi.fn(),
|
||||
createOTelSpansMock: vi.fn(),
|
||||
workflowExecutedMock: vi.fn(),
|
||||
}))
|
||||
|
||||
const { materializeLargeValueRefMock, storeLargeValueMock } = vi.hoisted(() => ({
|
||||
materializeLargeValueRefMock: vi.fn(),
|
||||
storeLargeValueMock: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('drizzle-orm', () => ({
|
||||
@@ -29,6 +42,8 @@ vi.mock('@/lib/logs/execution/logger', () => ({
|
||||
executionLogger: {
|
||||
startWorkflowExecution: startWorkflowExecutionMock,
|
||||
completeWorkflowExecution: completeWorkflowExecutionMock,
|
||||
loadTraceSpansForProjection: loadTraceSpansForProjectionMock,
|
||||
prepareTraceSpansForProjection: prepareTraceSpansForProjectionMock,
|
||||
},
|
||||
}))
|
||||
|
||||
@@ -36,6 +51,16 @@ vi.mock('@/lib/billing/calculations/usage-reservation', () => ({
|
||||
releaseExecutionSlot: releaseExecutionSlotMock,
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/telemetry', () => ({
|
||||
createOTelSpansForWorkflowExecution: createOTelSpansMock,
|
||||
PlatformEvents: { workflowExecuted: workflowExecutedMock },
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/execution/payloads/store', () => ({
|
||||
materializeLargeValueRef: materializeLargeValueRefMock,
|
||||
storeLargeValue: storeLargeValueMock,
|
||||
}))
|
||||
|
||||
const {
|
||||
setLastStartedBlockMock,
|
||||
setLastCompletedBlockMock,
|
||||
@@ -73,10 +98,105 @@ vi.mock('@/lib/logs/execution/logging-factory', () => ({
|
||||
}))
|
||||
|
||||
import { calculateCostSummary } from '@/lib/logs/execution/logging-factory'
|
||||
import type {
|
||||
ResolvedSecretTraceMatch,
|
||||
ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
import { LoggingSession } from './logging-session'
|
||||
|
||||
afterAll(resetDbChainMock)
|
||||
|
||||
function createSecretRegistry(
|
||||
matches: ResolvedSecretTraceMatch[],
|
||||
complete = true
|
||||
): ResolvedSecretTraceRegistry {
|
||||
return {
|
||||
isComplete: () => complete,
|
||||
getActiveMatches: () => matches,
|
||||
exportProvenance: () => ({ version: 1, complete, entries: [] }),
|
||||
} as unknown as ResolvedSecretTraceRegistry
|
||||
}
|
||||
|
||||
describe('LoggingSession terminal provenance', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
resetDbChainMock()
|
||||
dbChainMockFns.limit.mockResolvedValue([])
|
||||
completeWorkflowExecutionMock.mockResolvedValue({})
|
||||
releaseExecutionSlotMock.mockResolvedValue(undefined)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[
|
||||
'error',
|
||||
(session: LoggingSession) => session.completeWithError({ error: { message: 'failed' } }),
|
||||
],
|
||||
['cancellation', (session: LoggingSession) => session.completeWithCancellation()],
|
||||
['pause', (session: LoggingSession) => session.completeWithPause()],
|
||||
])('persists complete zero-entry provenance on %s finalization', async (_name, finalize) => {
|
||||
const session = new LoggingSession('workflow-1', `execution-${_name}`, 'manual')
|
||||
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
|
||||
|
||||
await finalize(session)
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
executionState: expect.objectContaining({
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [],
|
||||
},
|
||||
}),
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it.each(['cancellation', 'pause'] as const)(
|
||||
'preserves raw execution state on %s finalization',
|
||||
async (finalization) => {
|
||||
const session = new LoggingSession('workflow-1', `execution-state-${finalization}`, 'manual')
|
||||
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
|
||||
const executionState = {
|
||||
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
}
|
||||
|
||||
if (finalization === 'cancellation') {
|
||||
await session.completeWithCancellation({ executionState })
|
||||
} else {
|
||||
await session.completeWithPause({ executionState })
|
||||
}
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
executionState: expect.objectContaining({
|
||||
blockStates: executionState.blockStates,
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [],
|
||||
},
|
||||
}),
|
||||
})
|
||||
)
|
||||
}
|
||||
)
|
||||
})
|
||||
|
||||
beforeEach(() => {
|
||||
loadTraceSpansForProjectionMock.mockImplementation(
|
||||
async ({ traceSpans }: { traceSpans: unknown[] }) => traceSpans
|
||||
)
|
||||
prepareTraceSpansForProjectionMock.mockImplementation(
|
||||
async ({ traceSpans }: { traceSpans: unknown[] }) => traceSpans
|
||||
)
|
||||
})
|
||||
|
||||
describe('LoggingSession start snapshots', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
@@ -224,6 +344,7 @@ describe('LoggingSession completion retries', () => {
|
||||
|
||||
it('starts a new error completion attempt after a non-error completion and fallback both fail', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-3', 'api', 'req-1')
|
||||
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
|
||||
|
||||
completeWorkflowExecutionMock
|
||||
.mockRejectedValueOnce(new Error('success finalize failed'))
|
||||
@@ -250,6 +371,7 @@ describe('LoggingSession completion retries', () => {
|
||||
|
||||
it('preserves successful final output during fallback completion', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-5', 'api', 'req-1')
|
||||
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
|
||||
|
||||
completeWorkflowExecutionMock
|
||||
.mockRejectedValueOnce(new Error('success finalize failed'))
|
||||
@@ -268,6 +390,441 @@ describe('LoggingSession completion retries', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('projects only TraceSpans while preserving functional completion values', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-safe', 'api', 'req-1')
|
||||
const secret = 'sk-demo / trace?token=7f3a91'
|
||||
const rawFinalOutput = {
|
||||
result: {
|
||||
resolvedAtRuntime: true,
|
||||
echoed: `prefix:${secret}:suffix`,
|
||||
encoded: encodeURIComponent(secret),
|
||||
ordinary: 'us-east-1',
|
||||
},
|
||||
}
|
||||
const rawTraceSpans = [
|
||||
{
|
||||
id: 'span-safe',
|
||||
name: 'Function',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-01T00:00:00.000Z',
|
||||
endTime: '2026-07-01T00:00:00.001Z',
|
||||
status: 'success',
|
||||
output: { echoed: secret, encoded: encodeURIComponent(secret) },
|
||||
},
|
||||
]
|
||||
const rawWorkflowInput = { prompt: `use ${secret}` }
|
||||
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' }])
|
||||
)
|
||||
completeWorkflowExecutionMock.mockResolvedValue({})
|
||||
|
||||
await session.safeComplete({
|
||||
finalOutput: rawFinalOutput,
|
||||
traceSpans: rawTraceSpans as any,
|
||||
workflowInput: rawWorkflowInput,
|
||||
})
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
finalOutput: rawFinalOutput,
|
||||
workflowInput: rawWorkflowInput,
|
||||
traceSpans: [
|
||||
expect.objectContaining({
|
||||
output: {
|
||||
echoed: '{{OPENAI_API_KEY}}',
|
||||
encoded: encodeURIComponent(secret),
|
||||
},
|
||||
}),
|
||||
],
|
||||
})
|
||||
)
|
||||
expect(rawFinalOutput.result.echoed).toBe(`prefix:${secret}:suffix`)
|
||||
expect(rawTraceSpans[0].output.echoed).toBe(secret)
|
||||
expect(calculateCostSummary).toHaveBeenCalledWith(rawTraceSpans, undefined)
|
||||
|
||||
const persistedSpans = completeWorkflowExecutionMock.mock.calls[0]?.[0].traceSpans
|
||||
expect(createOTelSpansMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ traceSpans: persistedSpans })
|
||||
)
|
||||
expect(createOTelSpansMock.mock.calls[0]?.[0].traceSpans).toBe(persistedSpans)
|
||||
})
|
||||
|
||||
it('projects secrets before display filters can truncate a long active literal', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-long-secret', 'api', 'req-1')
|
||||
const secret = `secret-${'x'.repeat(16_000)}`
|
||||
const sourceTraceSpans = [
|
||||
{
|
||||
id: 'span-long-secret',
|
||||
name: 'Function',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-01T00:00:00.000Z',
|
||||
endTime: '2026-07-01T00:00:00.001Z',
|
||||
output: { result: secret },
|
||||
},
|
||||
]
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: secret, replacement: '{{LONG_SECRET}}' }])
|
||||
)
|
||||
prepareTraceSpansForProjectionMock.mockImplementationOnce(
|
||||
async ({ traceSpans }: { traceSpans: Array<{ output?: { result?: string } }> }) => {
|
||||
expect(traceSpans[0]?.output?.result).toBe('{{LONG_SECRET}}')
|
||||
return traceSpans
|
||||
}
|
||||
)
|
||||
|
||||
await session.safeComplete({ traceSpans: sourceTraceSpans as any })
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
traceSpans: [expect.objectContaining({ output: { result: '{{LONG_SECRET}}' } })],
|
||||
})
|
||||
)
|
||||
expect(sourceTraceSpans[0].output.result).toBe(secret)
|
||||
})
|
||||
|
||||
it('fails closed when generic log transforms reintroduce a secret literal for persistence and OTel', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-invariant', 'api', 'req-1')
|
||||
const sourceTraceSpans = [
|
||||
{
|
||||
id: 'span-invariant',
|
||||
name: 'Function',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-01T00:00:00.000Z',
|
||||
endTime: '2026-07-01T00:00:00.001Z',
|
||||
status: 'success',
|
||||
output: { apiKey: 'ordinary-value' },
|
||||
},
|
||||
]
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: 'E', replacement: '{{X}}' }])
|
||||
)
|
||||
prepareTraceSpansForProjectionMock.mockImplementationOnce(
|
||||
async ({ traceSpans }: { traceSpans: Array<Record<string, unknown>> }) =>
|
||||
traceSpans.map((span) => ({ ...span, output: { apiKey: '[REDACTED]' } }))
|
||||
)
|
||||
|
||||
await session.safeComplete({ traceSpans: sourceTraceSpans as any })
|
||||
|
||||
const persistedSpans = completeWorkflowExecutionMock.mock.calls[0]?.[0].traceSpans
|
||||
expect(persistedSpans).toEqual([
|
||||
expect.objectContaining({
|
||||
id: 'span-invariant',
|
||||
status: 'success',
|
||||
}),
|
||||
])
|
||||
expect(persistedSpans[0]).not.toHaveProperty('output')
|
||||
expect(createOTelSpansMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ traceSpans: persistedSpans })
|
||||
)
|
||||
expect(createOTelSpansMock.mock.calls[0]?.[0].traceSpans).toBe(persistedSpans)
|
||||
expect(sourceTraceSpans[0].output).toEqual({ apiKey: 'ordinary-value' })
|
||||
})
|
||||
|
||||
it('hydrates post-transform refs before sharing structural-only spans with persistence and OTel', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-ref-invariant', 'api', 'req-1')
|
||||
const sourceTraceSpans = [
|
||||
{
|
||||
id: 'span-ref-invariant',
|
||||
name: 'Function',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-01T00:00:00.000Z',
|
||||
endTime: '2026-07-01T00:00:00.001Z',
|
||||
status: 'success',
|
||||
output: { apiKey: 'ordinary-value' },
|
||||
},
|
||||
]
|
||||
const ref = {
|
||||
__simLargeValueRef: true,
|
||||
version: 1,
|
||||
id: 'lv_bbbbbbbbbbbb',
|
||||
kind: 'string',
|
||||
size: 32,
|
||||
preview: '{{X}}',
|
||||
} as const
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: 'E', replacement: '{{X}}' }])
|
||||
)
|
||||
prepareTraceSpansForProjectionMock.mockImplementationOnce(
|
||||
async ({ traceSpans }: { traceSpans: Array<Record<string, unknown>> }) =>
|
||||
traceSpans.map((span) => ({ ...span, output: { payload: ref } }))
|
||||
)
|
||||
materializeLargeValueRefMock.mockResolvedValue({ value: 'hidden-E' })
|
||||
|
||||
await session.safeComplete({ traceSpans: sourceTraceSpans as any })
|
||||
|
||||
const persistedSpans = completeWorkflowExecutionMock.mock.calls[0]?.[0].traceSpans
|
||||
expect(materializeLargeValueRefMock).toHaveBeenCalledWith(
|
||||
ref,
|
||||
expect.objectContaining({
|
||||
executionId: 'execution-ref-invariant',
|
||||
trackReference: false,
|
||||
})
|
||||
)
|
||||
expect(storeLargeValueMock).not.toHaveBeenCalled()
|
||||
expect(persistedSpans).toEqual([
|
||||
expect.objectContaining({
|
||||
id: 'span-ref-invariant',
|
||||
status: 'success',
|
||||
}),
|
||||
])
|
||||
expect(persistedSpans[0]).not.toHaveProperty('output')
|
||||
expect(createOTelSpansMock.mock.calls[0]?.[0].traceSpans).toBe(persistedSpans)
|
||||
expect(sourceTraceSpans[0].output).toEqual({ apiKey: 'ordinary-value' })
|
||||
})
|
||||
|
||||
it('projects synthetic error spans without copying the raw error into OTel metadata', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-error-safe', 'api', 'req-1')
|
||||
const secret = 'sk-demo-error-7f3a91'
|
||||
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' }])
|
||||
)
|
||||
completeWorkflowExecutionMock.mockResolvedValue({})
|
||||
const rawExecutionState = {
|
||||
blockStates: { 'function-1': { output: { result: secret } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
}
|
||||
|
||||
await session.safeCompleteWithError({
|
||||
error: { message: `Function failed with ${secret}` },
|
||||
executionState: rawExecutionState,
|
||||
})
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
finalOutput: { error: `Function failed with ${secret}` },
|
||||
traceSpans: [
|
||||
expect.objectContaining({
|
||||
output: { error: 'Function failed with {{OPENAI_API_KEY}}' },
|
||||
}),
|
||||
],
|
||||
completionFailure: `Function failed with ${secret}`,
|
||||
executionState: expect.objectContaining({
|
||||
blockStates: { 'function-1': { output: { result: secret } } },
|
||||
}),
|
||||
})
|
||||
)
|
||||
expect(createOTelSpansMock).toHaveBeenCalledWith(
|
||||
expect.not.objectContaining({ error: expect.anything() })
|
||||
)
|
||||
expect(workflowExecutedMock).toHaveBeenCalledWith(
|
||||
expect.not.objectContaining({ errorMessage: expect.anything() })
|
||||
)
|
||||
})
|
||||
|
||||
it('keeps fallback functional output unchanged', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-fallback-safe', 'api', 'req-1')
|
||||
const secret = 'sk-demo-fallback-7f3a91'
|
||||
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: secret, replacement: '{{OPENAI_API_KEY}}' }])
|
||||
)
|
||||
completeWorkflowExecutionMock
|
||||
.mockRejectedValueOnce(new Error('primary persistence failed'))
|
||||
.mockResolvedValueOnce({})
|
||||
const executionState = {
|
||||
blockStates: { 'function-1': { output: { result: secret } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
}
|
||||
|
||||
await session.safeComplete({
|
||||
finalOutput: { echoed: secret },
|
||||
executionState,
|
||||
})
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({
|
||||
finalOutput: { echoed: secret },
|
||||
finalizationPath: 'fallback_completed',
|
||||
executionState: expect.objectContaining({ blockStates: executionState.blockStates }),
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('persists structural-only spans when installed provenance is incomplete', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-incomplete', 'api', 'req-1')
|
||||
session.setResolvedSecretTraceRegistry(createSecretRegistry([], false))
|
||||
completeWorkflowExecutionMock.mockResolvedValue({})
|
||||
|
||||
await session.safeComplete({
|
||||
finalOutput: { raw: 'functional-data' },
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Agent',
|
||||
type: 'agent',
|
||||
duration: 1,
|
||||
startTime: '2026-07-01T00:00:00.000Z',
|
||||
endTime: '2026-07-01T00:00:00.001Z',
|
||||
status: 'success',
|
||||
output: { raw: 'unknown-provenance' },
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
finalOutput: { raw: 'functional-data' },
|
||||
traceSpans: [
|
||||
expect.not.objectContaining({
|
||||
output: expect.anything(),
|
||||
}),
|
||||
],
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('fails closed to structural-only spans when provenance was not installed', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-no-registry', 'api', 'req-1')
|
||||
completeWorkflowExecutionMock.mockResolvedValue({})
|
||||
|
||||
await session.safeComplete({
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Function',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-01T00:00:00.000Z',
|
||||
endTime: '2026-07-01T00:00:00.001Z',
|
||||
output: { unknown: 'provenance' },
|
||||
},
|
||||
],
|
||||
})
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
finalOutput: {},
|
||||
traceSpans: [expect.not.objectContaining({ output: expect.anything() })],
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('projects live block errors and terminal block logs without mutating raw callback data', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-display-safe', 'manual', 'req-1')
|
||||
const secret = '1234'
|
||||
const rawError = `Reference Error: Line 1: return blah +${secret} - blah is not defined`
|
||||
const rawLog = {
|
||||
blockId: 'function-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
startedAt: '2026-07-01T00:00:00.000Z',
|
||||
endedAt: '2026-07-01T00:00:00.001Z',
|
||||
durationMs: 1,
|
||||
success: false,
|
||||
executionOrder: 1,
|
||||
input: { code: `return blah +${secret}` },
|
||||
output: { error: rawError },
|
||||
error: rawError,
|
||||
}
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: secret, replacement: '{{NUMBER_SECRET}}' }])
|
||||
)
|
||||
|
||||
const display = await session.projectDisplayContent({
|
||||
input: rawLog.input,
|
||||
output: rawLog.output,
|
||||
error: rawError,
|
||||
})
|
||||
const [displayLog] = await session.projectBlockLogsForDisplay([rawLog])
|
||||
|
||||
expect(display).toEqual({
|
||||
input: { code: 'return blah +{{NUMBER_SECRET}}' },
|
||||
output: {
|
||||
error: 'Reference Error: Line 1: return blah +{{NUMBER_SECRET}} - blah is not defined',
|
||||
},
|
||||
error: 'Reference Error: Line 1: return blah +{{NUMBER_SECRET}} - blah is not defined',
|
||||
clearLiveDisplay: true,
|
||||
})
|
||||
expect(displayLog.input).toEqual(display.input)
|
||||
expect(displayLog.output).toEqual(display.output)
|
||||
expect(displayLog.error).toBe(display.error)
|
||||
expect(displayLog.clearLiveDisplay).toBe(true)
|
||||
expect(rawLog.input.code).toBe(`return blah +${secret}`)
|
||||
expect(rawLog.output.error).toBe(rawError)
|
||||
expect(rawLog.error).toBe(rawError)
|
||||
})
|
||||
|
||||
it('projects large terminal log sets in bounded batches without dropping rows', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-display-batches', 'manual', 'req-1')
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: 'raw-secret', replacement: '{{TOKEN}}' }])
|
||||
)
|
||||
const rawLogs = Array.from({ length: 129 }, (_, index) => ({
|
||||
blockId: `function-${index}`,
|
||||
blockName: `Function ${index}`,
|
||||
blockType: 'function',
|
||||
startedAt: '2026-07-01T00:00:00.000Z',
|
||||
endedAt: '2026-07-01T00:00:00.001Z',
|
||||
durationMs: 1,
|
||||
success: true,
|
||||
executionOrder: index,
|
||||
output: { value: `row-${index}:raw-secret` },
|
||||
}))
|
||||
|
||||
const displayLogs = await session.projectBlockLogsForDisplay(rawLogs)
|
||||
|
||||
expect(displayLogs).toHaveLength(rawLogs.length)
|
||||
expect(displayLogs[0].output).toEqual({ value: 'row-0:{{TOKEN}}' })
|
||||
expect(displayLogs[128].output).toEqual({ value: 'row-128:{{TOKEN}}' })
|
||||
expect(rawLogs[128].output.value).toBe('row-128:raw-secret')
|
||||
})
|
||||
|
||||
it('projects a numeric Function result produced by a resolved numeric secret', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-numeric-secret', 'manual', 'req-1')
|
||||
session.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: '1234', replacement: '{{OPENAI_API_KEY}}' }])
|
||||
)
|
||||
const rawLog = {
|
||||
blockId: 'function-1',
|
||||
blockName: 'Function 1',
|
||||
blockType: 'function',
|
||||
startedAt: '2026-07-01T00:00:00.000Z',
|
||||
endedAt: '2026-07-01T00:00:00.001Z',
|
||||
durationMs: 1,
|
||||
success: true,
|
||||
executionOrder: 1,
|
||||
input: { code: 'return 1234' },
|
||||
output: { result: 1234, stdout: '' },
|
||||
}
|
||||
|
||||
const [displayLog] = await session.projectBlockLogsForDisplay([rawLog])
|
||||
|
||||
expect(displayLog.input).toEqual({ code: 'return {{OPENAI_API_KEY}}' })
|
||||
expect(displayLog.output).toEqual({ result: '{{OPENAI_API_KEY}}', stdout: '' })
|
||||
expect(rawLog.output.result).toBe(1234)
|
||||
})
|
||||
|
||||
it('suppresses live deltas once a resolved secret is active', async () => {
|
||||
const active = new LoggingSession('workflow-1', 'execution-live-active', 'manual', 'req-1')
|
||||
active.setResolvedSecretTraceRegistry(
|
||||
createSecretRegistry([{ plaintext: 'split-secret', replacement: '{{SECRET}}' }])
|
||||
)
|
||||
|
||||
const inactive = new LoggingSession('workflow-1', 'execution-live-inactive', 'manual', 'req-1')
|
||||
inactive.setResolvedSecretTraceRegistry(createSecretRegistry([]))
|
||||
|
||||
await expect(active.projectLiveDisplayText('chunk', 'split-')).resolves.toEqual({
|
||||
clearLiveDisplay: true,
|
||||
})
|
||||
await expect(inactive.projectLiveDisplayText('chunk', 'ordinary text')).resolves.toEqual({
|
||||
chunk: 'ordinary text',
|
||||
})
|
||||
})
|
||||
|
||||
it('derives fallback cost from trace spans when the primary completion fails', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-6', 'api', 'req-1') as any
|
||||
|
||||
@@ -328,6 +885,7 @@ describe('LoggingSession completion retries', () => {
|
||||
|
||||
it('persists failed error semantics when completeWithError receives non-error trace spans', async () => {
|
||||
const session = new LoggingSession('workflow-1', 'execution-4', 'api', 'req-1')
|
||||
session.setResolvedSecretTraceRegistry(createSecretRegistry([]))
|
||||
const traceSpans = [
|
||||
{
|
||||
id: 'span-1',
|
||||
@@ -431,6 +989,14 @@ describe('LoggingSession completion retries', () => {
|
||||
completeWorkflowExecutionMock
|
||||
.mockRejectedValueOnce(new Error('pause finalize failed'))
|
||||
.mockResolvedValueOnce({})
|
||||
const executionState = {
|
||||
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
}
|
||||
|
||||
await expect(
|
||||
session.safeCompleteWithPause({
|
||||
@@ -438,11 +1004,18 @@ describe('LoggingSession completion retries', () => {
|
||||
totalDurationMs: 10,
|
||||
traceSpans: [],
|
||||
workflowInput: { hello: 'world' },
|
||||
executionState,
|
||||
})
|
||||
).resolves.toBeUndefined()
|
||||
|
||||
expect(session.hasCompleted()).toBe(true)
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledTimes(2)
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenLastCalledWith(
|
||||
expect.objectContaining({
|
||||
finalizationPath: 'paused',
|
||||
executionState: expect.objectContaining({ blockStates: executionState.blockStates }),
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
it('persists last started block independently from cost accumulation', async () => {
|
||||
@@ -667,11 +1240,22 @@ describe('completeWithError cancelled-status guard', () => {
|
||||
dbChainMockFns.limit.mockRejectedValueOnce(new Error('DB connection lost'))
|
||||
completeWorkflowExecutionMock.mockResolvedValue({})
|
||||
const session = new LoggingSession('workflow-1', 'execution-1', 'api', 'req-1')
|
||||
const executionState = {
|
||||
blockStates: { 'function-1': { output: { result: 'raw-secret-value' } } },
|
||||
executedBlocks: ['function-1'],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: ['function-1'],
|
||||
}
|
||||
|
||||
await session.safeCompleteWithError({ error: { message: 'block failed' } })
|
||||
await session.safeCompleteWithError({
|
||||
error: { message: 'block failed' },
|
||||
executionState,
|
||||
})
|
||||
|
||||
expect(completeWorkflowExecutionMock).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ finalizationPath: 'force_failed' })
|
||||
expect.objectContaining({ finalizationPath: 'force_failed', executionState })
|
||||
)
|
||||
expect(session.hasCompleted()).toBe(true)
|
||||
})
|
||||
|
||||
@@ -6,6 +6,8 @@ import { and, eq, sql } from 'drizzle-orm'
|
||||
import { releaseExecutionSlot } from '@/lib/billing/calculations/usage-reservation'
|
||||
import type { BillingAttributionSnapshot } from '@/lib/billing/core/billing-attribution'
|
||||
import { isRetryableInfrastructureError } from '@/lib/core/errors/retryable-infrastructure'
|
||||
import type { LargeValueStoreContext } from '@/lib/execution/payloads/store'
|
||||
import type { SecretSafeBlockLog } from '@/lib/logs/execution/display-types'
|
||||
import { executionLogger } from '@/lib/logs/execution/logger'
|
||||
import {
|
||||
type CostSummaryOptions,
|
||||
@@ -21,6 +23,10 @@ import {
|
||||
setLastCompletedBlock,
|
||||
setLastStartedBlock,
|
||||
} from '@/lib/logs/execution/progress-markers'
|
||||
import {
|
||||
enforceTraceSpanSecretInvariant,
|
||||
projectTraceSpansForSecrets,
|
||||
} from '@/lib/logs/execution/trace-secret-projection'
|
||||
import { traceSpansIndicateFailure } from '@/lib/logs/execution/trace-spans/trace-spans'
|
||||
import type {
|
||||
ExecutionEnvironment,
|
||||
@@ -32,6 +38,8 @@ import type {
|
||||
WorkflowState,
|
||||
} from '@/lib/logs/types'
|
||||
import type { SerializableExecutionState } from '@/executor/execution/types'
|
||||
import type { BlockLog } from '@/executor/types'
|
||||
import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
type TriggerData = Record<string, unknown> & {
|
||||
correlation?: NonNullable<ExecutionTrigger['data']>['correlation']
|
||||
@@ -83,11 +91,32 @@ function buildCompletedMarkerPersistenceQuery(params: {
|
||||
|
||||
/** Progress-marker and status writes on `workflow_execution_logs` use the exec pool. */
|
||||
const execDb = dbFor('exec')
|
||||
const BLOCK_LOG_PROJECTION_BATCH_SIZE = 64
|
||||
|
||||
function structuralBlockLog(log: BlockLog): BlockLog {
|
||||
const {
|
||||
input: _input,
|
||||
output: _output,
|
||||
error: _error,
|
||||
childTraceSpans: _childTraceSpans,
|
||||
...structural
|
||||
} = log
|
||||
return structural
|
||||
}
|
||||
|
||||
const logger = createLogger('LoggingSession')
|
||||
|
||||
type CompletionAttempt = 'complete' | 'error' | 'cancelled' | 'paused'
|
||||
|
||||
export interface SecretSafeDisplayContent {
|
||||
input?: unknown
|
||||
output?: unknown
|
||||
error?: string
|
||||
text?: string
|
||||
chunk?: string
|
||||
clearLiveDisplay?: true
|
||||
}
|
||||
|
||||
export interface SessionStartParams {
|
||||
userId?: string
|
||||
/** Explicit initiating actor for callers that do not populate `userId`. */
|
||||
@@ -120,12 +149,14 @@ export interface SessionErrorCompleteParams {
|
||||
}
|
||||
traceSpans?: TraceSpan[]
|
||||
skipCost?: boolean
|
||||
executionState?: SerializableExecutionState
|
||||
}
|
||||
|
||||
export interface SessionCancelledParams {
|
||||
endedAt?: string
|
||||
totalDurationMs?: number
|
||||
traceSpans?: TraceSpan[]
|
||||
executionState?: SerializableExecutionState
|
||||
}
|
||||
|
||||
export interface SessionPausedParams {
|
||||
@@ -133,6 +164,7 @@ export interface SessionPausedParams {
|
||||
totalDurationMs?: number
|
||||
traceSpans?: TraceSpan[]
|
||||
workflowInput?: any
|
||||
executionState?: SerializableExecutionState
|
||||
}
|
||||
|
||||
export interface LoggingSessionOptions {
|
||||
@@ -166,6 +198,8 @@ export class LoggingSession {
|
||||
private costOptions?: CostSummaryOptions
|
||||
private pendingProgressWrites = new Set<Promise<void>>()
|
||||
private postExecutionPromise: Promise<void> | null = null
|
||||
private resolvedSecretTraceRegistry?: ResolvedSecretTraceRegistry
|
||||
private traceLargeValueAccess: LargeValueStoreContext = {}
|
||||
|
||||
constructor(
|
||||
workflowId: string,
|
||||
@@ -186,6 +220,187 @@ export class LoggingSession {
|
||||
: undefined
|
||||
}
|
||||
|
||||
/** Installs the run-scoped provenance used only at the terminal TraceSpan boundary. */
|
||||
setResolvedSecretTraceRegistry(registry: ResolvedSecretTraceRegistry): void {
|
||||
this.resolvedSecretTraceRegistry = registry
|
||||
}
|
||||
|
||||
/** Adds the trusted execution-ref scope needed to rewrite offloaded trace content. */
|
||||
setTraceLargeValueAccess(context: LargeValueStoreContext): void {
|
||||
this.traceLargeValueAccess = context
|
||||
}
|
||||
|
||||
private getSecretProjectionStore(): LargeValueStoreContext {
|
||||
return {
|
||||
...this.traceLargeValueAccess,
|
||||
workspaceId: this.environment?.workspaceId,
|
||||
workflowId: this.workflowId,
|
||||
executionId: this.executionId,
|
||||
userId: this.actorUserId ?? this.environment?.userId,
|
||||
}
|
||||
}
|
||||
|
||||
private async projectRawTraceSpans(traceSpans: TraceSpan[]): Promise<TraceSpan[]> {
|
||||
return projectTraceSpansForSecrets(traceSpans, {
|
||||
registry: this.resolvedSecretTraceRegistry,
|
||||
store: this.getSecretProjectionStore(),
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Produces a display-only copy of known observability content through the same
|
||||
* projector used for persisted TraceSpans. Runtime values and callback payloads
|
||||
* remain untouched, and an unavailable projection yields no content fields.
|
||||
*/
|
||||
async projectDisplayContent(
|
||||
content: SecretSafeDisplayContent
|
||||
): Promise<SecretSafeDisplayContent> {
|
||||
try {
|
||||
const envelope: Record<string, unknown> = {}
|
||||
for (const key of ['input', 'output', 'error', 'text', 'chunk'] as const) {
|
||||
if (Object.hasOwn(content, key)) envelope[key] = content[key]
|
||||
}
|
||||
|
||||
const now = new Date().toISOString()
|
||||
const [projectedSpan] = await this.projectRawTraceSpans([
|
||||
{
|
||||
id: 'secret-safe-display-projection',
|
||||
name: 'Display Projection',
|
||||
type: 'display',
|
||||
duration: 0,
|
||||
startTime: now,
|
||||
endTime: now,
|
||||
output: envelope,
|
||||
},
|
||||
])
|
||||
const projected = this.readProjectedDisplayContent(projectedSpan?.output)
|
||||
return this.shouldClearLiveDisplay() ? { ...projected, clearLiveDisplay: true } : projected
|
||||
} catch {
|
||||
logger.warn('Display secret projection failed; omitting display content')
|
||||
return {}
|
||||
}
|
||||
}
|
||||
|
||||
private readProjectedDisplayContent(
|
||||
projectedEnvelope: TraceSpan['output'] | undefined
|
||||
): SecretSafeDisplayContent {
|
||||
if (!projectedEnvelope) return {}
|
||||
|
||||
const projected: SecretSafeDisplayContent = {}
|
||||
if (Object.hasOwn(projectedEnvelope, 'input')) projected.input = projectedEnvelope.input
|
||||
if (Object.hasOwn(projectedEnvelope, 'output')) projected.output = projectedEnvelope.output
|
||||
if (typeof projectedEnvelope.error === 'string') projected.error = projectedEnvelope.error
|
||||
if (typeof projectedEnvelope.text === 'string') projected.text = projectedEnvelope.text
|
||||
if (typeof projectedEnvelope.chunk === 'string') projected.chunk = projectedEnvelope.chunk
|
||||
return projected
|
||||
}
|
||||
|
||||
/**
|
||||
* Projects terminal reconciliation logs without changing the executor-owned
|
||||
* BlockLogs. Child traces use the identical TraceSpan projection boundary.
|
||||
*/
|
||||
async projectBlockLogsForDisplay(blockLogs: BlockLog[]): Promise<SecretSafeBlockLog[]> {
|
||||
const now = new Date().toISOString()
|
||||
const displayLogs: SecretSafeBlockLog[] = []
|
||||
const clearLiveDisplay = this.shouldClearLiveDisplay()
|
||||
|
||||
for (let offset = 0; offset < blockLogs.length; offset += BLOCK_LOG_PROJECTION_BATCH_SIZE) {
|
||||
const batch = blockLogs.slice(offset, offset + BLOCK_LOG_PROJECTION_BATCH_SIZE)
|
||||
let projectedLogs: TraceSpan[]
|
||||
try {
|
||||
projectedLogs = await this.projectRawTraceSpans(
|
||||
batch.map((log, index) => ({
|
||||
id: `secret-safe-block-log-${offset + index}`,
|
||||
name: 'Block Log Display Projection',
|
||||
type: 'display',
|
||||
duration: 0,
|
||||
startTime: now,
|
||||
endTime: now,
|
||||
output: {
|
||||
...(log.input !== undefined ? { input: log.input } : {}),
|
||||
...(log.output !== undefined ? { output: log.output } : {}),
|
||||
...(log.error !== undefined ? { error: log.error } : {}),
|
||||
},
|
||||
...(log.childTraceSpans ? { children: log.childTraceSpans } : {}),
|
||||
}))
|
||||
)
|
||||
} catch {
|
||||
logger.warn('Block-log secret projection failed; retaining structural logs only')
|
||||
displayLogs.push(...batch.map(structuralBlockLog))
|
||||
continue
|
||||
}
|
||||
|
||||
for (let index = 0; index < batch.length; index += 1) {
|
||||
const log = batch[index]
|
||||
const display = this.readProjectedDisplayContent(projectedLogs[index]?.output)
|
||||
displayLogs.push({
|
||||
...structuralBlockLog(log),
|
||||
...(clearLiveDisplay ? { clearLiveDisplay: true as const } : {}),
|
||||
...(Object.hasOwn(display, 'input')
|
||||
? { input: display.input as Record<string, unknown> }
|
||||
: {}),
|
||||
...(Object.hasOwn(display, 'output')
|
||||
? { output: display.output as BlockLog['output'] }
|
||||
: {}),
|
||||
...(display.error !== undefined ? { error: display.error } : {}),
|
||||
...(projectedLogs[index]?.children
|
||||
? { childTraceSpans: projectedLogs[index].children }
|
||||
: {}),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
return displayLogs
|
||||
}
|
||||
|
||||
/**
|
||||
* Live deltas may split one literal across multiple events. Once provenance is
|
||||
* active (or incomplete), suppress their display copy instead of attempting a
|
||||
* per-chunk replacement that could miss the split value.
|
||||
*/
|
||||
async projectLiveDisplayText(
|
||||
field: 'text' | 'chunk',
|
||||
value: string
|
||||
): Promise<SecretSafeDisplayContent> {
|
||||
if (
|
||||
!this.resolvedSecretTraceRegistry?.isComplete() ||
|
||||
this.resolvedSecretTraceRegistry.getActiveMatches().length > 0
|
||||
) {
|
||||
return { clearLiveDisplay: true }
|
||||
}
|
||||
return this.projectDisplayContent({ [field]: value })
|
||||
}
|
||||
|
||||
private shouldClearLiveDisplay(): boolean {
|
||||
return (
|
||||
!this.resolvedSecretTraceRegistry?.isComplete() ||
|
||||
this.resolvedSecretTraceRegistry.getActiveMatches().length > 0
|
||||
)
|
||||
}
|
||||
|
||||
private async projectTraceSpans(traceSpans: TraceSpan[]): Promise<TraceSpan[]> {
|
||||
const sourceTraceSpans = await executionLogger.loadTraceSpansForProjection({
|
||||
executionId: this.executionId,
|
||||
workflowId: this.workflowId,
|
||||
workspaceId: this.environment?.workspaceId ?? null,
|
||||
traceSpans,
|
||||
isResume: this.isResume,
|
||||
})
|
||||
const secretSafeTraceSpans = await this.projectRawTraceSpans(sourceTraceSpans)
|
||||
const preparedTraceSpans = await executionLogger.prepareTraceSpansForProjection({
|
||||
executionId: this.executionId,
|
||||
workflowId: this.workflowId,
|
||||
workspaceId: this.environment?.workspaceId ?? null,
|
||||
userId: this.actorUserId ?? this.environment?.userId,
|
||||
traceSpans: secretSafeTraceSpans,
|
||||
})
|
||||
const invariantSafeTraceSpans = await enforceTraceSpanSecretInvariant(preparedTraceSpans, {
|
||||
registry: this.resolvedSecretTraceRegistry,
|
||||
store: this.getSecretProjectionStore(),
|
||||
})
|
||||
return invariantSafeTraceSpans
|
||||
}
|
||||
|
||||
async onBlockStart(
|
||||
blockId: string,
|
||||
blockName: string,
|
||||
@@ -303,6 +518,7 @@ export class LoggingSession {
|
||||
level?: 'info' | 'error'
|
||||
status?: 'completed' | 'failed' | 'cancelled' | 'pending'
|
||||
}): Promise<void> {
|
||||
const executionState = this.withResolvedSecretTraceProvenance(params.executionState)
|
||||
await executionLogger.completeWorkflowExecution({
|
||||
executionId: this.executionId,
|
||||
endedAt: params.endedAt,
|
||||
@@ -311,7 +527,7 @@ export class LoggingSession {
|
||||
finalOutput: params.finalOutput,
|
||||
traceSpans: params.traceSpans,
|
||||
workflowInput: params.workflowInput,
|
||||
executionState: params.executionState,
|
||||
executionState,
|
||||
finalizationPath: params.finalizationPath,
|
||||
completionFailure: params.completionFailure,
|
||||
isResume: this.isResume,
|
||||
@@ -337,6 +553,27 @@ export class LoggingSession {
|
||||
}
|
||||
}
|
||||
|
||||
private withResolvedSecretTraceProvenance(
|
||||
executionState?: SerializableExecutionState
|
||||
): SerializableExecutionState | undefined {
|
||||
if (!this.resolvedSecretTraceRegistry) return executionState
|
||||
|
||||
const resolvedSecretTraceProvenance = this.resolvedSecretTraceRegistry.exportProvenance()
|
||||
if (executionState) {
|
||||
return { ...executionState, resolvedSecretTraceProvenance }
|
||||
}
|
||||
|
||||
return {
|
||||
blockStates: {},
|
||||
executedBlocks: [],
|
||||
blockLogs: [],
|
||||
decisions: { router: {}, condition: {} },
|
||||
completedLoops: [],
|
||||
activeExecutionPath: [],
|
||||
resolvedSecretTraceProvenance,
|
||||
}
|
||||
}
|
||||
|
||||
async onBlockComplete(
|
||||
blockId: string,
|
||||
blockName: string,
|
||||
@@ -371,6 +608,14 @@ export class LoggingSession {
|
||||
} = params
|
||||
this.actorUserId = billingAttribution?.actorUserId ?? actorUserId ?? userId ?? null
|
||||
this.billingAttribution = billingAttribution
|
||||
if (!this.resolvedSecretTraceRegistry) {
|
||||
const scopeUserId = userId ?? this.actorUserId
|
||||
this.resolvedSecretTraceRegistry = new ResolvedSecretTraceRegistry(
|
||||
[],
|
||||
scopeUserId ? { userId: scopeUserId, workspaceId } : undefined
|
||||
)
|
||||
if (skipLogCreation) this.resolvedSecretTraceRegistry.markIncomplete()
|
||||
}
|
||||
|
||||
try {
|
||||
this.trigger = createTriggerObject(this.triggerType, triggerData)
|
||||
@@ -419,35 +664,38 @@ export class LoggingSession {
|
||||
}
|
||||
this.completing = true
|
||||
|
||||
const { endedAt, totalDurationMs, finalOutput, traceSpans, workflowInput, executionState } =
|
||||
params
|
||||
const { endedAt, totalDurationMs, workflowInput, executionState } = params
|
||||
const finalOutput = params.finalOutput || {}
|
||||
const rawTraceSpans = params.traceSpans || []
|
||||
|
||||
try {
|
||||
const costSummary = calculateCostSummary(traceSpans || [], this.costOptions)
|
||||
const costSummary = calculateCostSummary(rawTraceSpans, this.costOptions)
|
||||
const endTime = endedAt || new Date().toISOString()
|
||||
const duration = totalDurationMs || 0
|
||||
const hasErrors = traceSpansIndicateFailure(rawTraceSpans)
|
||||
const traceSpans = await this.projectTraceSpans(rawTraceSpans)
|
||||
|
||||
await this.completeExecutionWithFinalization({
|
||||
endedAt: endTime,
|
||||
totalDurationMs: duration,
|
||||
costSummary,
|
||||
finalOutput: finalOutput || {},
|
||||
traceSpans: traceSpans || [],
|
||||
finalOutput,
|
||||
traceSpans,
|
||||
workflowInput,
|
||||
executionState,
|
||||
finalizationPath: 'completed',
|
||||
level: hasErrors ? 'error' : 'info',
|
||||
status: hasErrors ? 'failed' : 'completed',
|
||||
})
|
||||
|
||||
this.completed = true
|
||||
|
||||
if (traceSpans && traceSpans.length > 0) {
|
||||
if (traceSpans.length > 0) {
|
||||
try {
|
||||
const { PlatformEvents, createOTelSpansForWorkflowExecution } = await import(
|
||||
'@/lib/core/telemetry'
|
||||
)
|
||||
|
||||
const hasErrors = traceSpansIndicateFailure(traceSpans)
|
||||
|
||||
PlatformEvents.workflowExecuted({
|
||||
workflowId: this.workflowId,
|
||||
durationMs: duration,
|
||||
@@ -513,13 +761,14 @@ export class LoggingSession {
|
||||
return
|
||||
}
|
||||
|
||||
const { endedAt, totalDurationMs, error, traceSpans, skipCost } = params
|
||||
const { endedAt, totalDurationMs, error, skipCost } = params
|
||||
const rawTraceSpans = params.traceSpans || []
|
||||
|
||||
const endTime = endedAt ? new Date(endedAt) : new Date()
|
||||
const durationMs = typeof totalDurationMs === 'number' ? totalDurationMs : 0
|
||||
const startTime = new Date(endTime.getTime() - Math.max(1, durationMs))
|
||||
|
||||
const hasProvidedSpans = Array.isArray(traceSpans) && traceSpans.length > 0
|
||||
const hasProvidedSpans = rawTraceSpans.length > 0
|
||||
|
||||
// calculateCostSummary([]) / (undefined) already returns the base-charge
|
||||
// summary, so the no-spans branch needs no separate literal.
|
||||
@@ -535,7 +784,7 @@ export class LoggingSession {
|
||||
models: {},
|
||||
charges: {},
|
||||
}
|
||||
: calculateCostSummary(traceSpans, this.costOptions)
|
||||
: calculateCostSummary(rawTraceSpans, this.costOptions)
|
||||
|
||||
const message = error?.message || 'Run failed before starting blocks'
|
||||
|
||||
@@ -551,7 +800,7 @@ export class LoggingSession {
|
||||
output: { error: message },
|
||||
}
|
||||
|
||||
const spans = hasProvidedSpans ? traceSpans : [errorSpan]
|
||||
const spans = await this.projectTraceSpans(hasProvidedSpans ? rawTraceSpans : [errorSpan])
|
||||
|
||||
await this.completeExecutionWithFinalization({
|
||||
endedAt: endTime.toISOString(),
|
||||
@@ -559,6 +808,7 @@ export class LoggingSession {
|
||||
costSummary,
|
||||
finalOutput: { error: message },
|
||||
traceSpans: spans,
|
||||
executionState: params.executionState,
|
||||
level: 'error',
|
||||
status: 'failed',
|
||||
finalizationPath: 'force_failed',
|
||||
@@ -578,7 +828,6 @@ export class LoggingSession {
|
||||
trigger: this.triggerType,
|
||||
blocksExecuted: spans.length,
|
||||
hasErrors: true,
|
||||
errorMessage: message,
|
||||
})
|
||||
|
||||
createOTelSpansForWorkflowExecution({
|
||||
@@ -591,7 +840,6 @@ export class LoggingSession {
|
||||
endTime: endTime.toISOString(),
|
||||
totalDurationMs: Math.max(1, durationMs),
|
||||
status: 'error',
|
||||
error: message,
|
||||
})
|
||||
} catch (_e) {
|
||||
// Silently fail
|
||||
@@ -622,7 +870,8 @@ export class LoggingSession {
|
||||
this.completing = true
|
||||
|
||||
try {
|
||||
const { endedAt, totalDurationMs, traceSpans } = params
|
||||
const { endedAt, totalDurationMs } = params
|
||||
const rawTraceSpans = params.traceSpans || []
|
||||
|
||||
const endTime = endedAt ? new Date(endedAt) : new Date()
|
||||
const durationMs = typeof totalDurationMs === 'number' ? totalDurationMs : 0
|
||||
@@ -646,14 +895,16 @@ export class LoggingSession {
|
||||
|
||||
// calculateCostSummary handles empty/undefined spans by returning the
|
||||
// base-charge summary, so no separate no-spans literal is needed.
|
||||
const costSummary = calculateCostSummary(traceSpans, this.costOptions)
|
||||
const costSummary = calculateCostSummary(rawTraceSpans, this.costOptions)
|
||||
const traceSpans = await this.projectTraceSpans(rawTraceSpans)
|
||||
|
||||
await this.completeExecutionWithFinalization({
|
||||
endedAt: endTime.toISOString(),
|
||||
totalDurationMs: Math.max(1, durationMs),
|
||||
costSummary,
|
||||
finalOutput: { cancelled: true },
|
||||
traceSpans: traceSpans || [],
|
||||
traceSpans,
|
||||
executionState: params.executionState,
|
||||
finalizationPath: 'cancelled',
|
||||
status: 'cancelled',
|
||||
})
|
||||
@@ -669,11 +920,11 @@ export class LoggingSession {
|
||||
durationMs: Math.max(1, durationMs),
|
||||
status: 'cancelled',
|
||||
trigger: this.triggerType,
|
||||
blocksExecuted: traceSpans?.length || 0,
|
||||
blocksExecuted: traceSpans.length,
|
||||
hasErrors: false,
|
||||
})
|
||||
|
||||
if (traceSpans && traceSpans.length > 0) {
|
||||
if (traceSpans.length > 0) {
|
||||
const startTime = new Date(endTime.getTime() - Math.max(1, durationMs))
|
||||
createOTelSpansForWorkflowExecution({
|
||||
workflowId: this.workflowId,
|
||||
@@ -716,7 +967,8 @@ export class LoggingSession {
|
||||
this.completing = true
|
||||
|
||||
try {
|
||||
const { endedAt, totalDurationMs, traceSpans, workflowInput } = params
|
||||
const { endedAt, totalDurationMs, workflowInput } = params
|
||||
const rawTraceSpans = params.traceSpans || []
|
||||
|
||||
const endTime = endedAt ? new Date(endedAt) : new Date()
|
||||
const durationMs = typeof totalDurationMs === 'number' ? totalDurationMs : 0
|
||||
@@ -740,15 +992,17 @@ export class LoggingSession {
|
||||
|
||||
// calculateCostSummary handles empty/undefined spans by returning the
|
||||
// base-charge summary, so no separate no-spans literal is needed.
|
||||
const costSummary = calculateCostSummary(traceSpans, this.costOptions)
|
||||
const costSummary = calculateCostSummary(rawTraceSpans, this.costOptions)
|
||||
const traceSpans = await this.projectTraceSpans(rawTraceSpans)
|
||||
|
||||
await this.completeExecutionWithFinalization({
|
||||
endedAt: endTime.toISOString(),
|
||||
totalDurationMs: Math.max(1, durationMs),
|
||||
costSummary,
|
||||
finalOutput: { paused: true },
|
||||
traceSpans: traceSpans || [],
|
||||
traceSpans,
|
||||
workflowInput,
|
||||
executionState: params.executionState,
|
||||
finalizationPath: 'paused',
|
||||
status: 'pending',
|
||||
})
|
||||
@@ -764,12 +1018,12 @@ export class LoggingSession {
|
||||
durationMs: Math.max(1, durationMs),
|
||||
status: 'paused',
|
||||
trigger: this.triggerType,
|
||||
blocksExecuted: traceSpans?.length || 0,
|
||||
blocksExecuted: traceSpans.length,
|
||||
hasErrors: false,
|
||||
totalCost: costSummary.totalCost || 0,
|
||||
})
|
||||
|
||||
if (traceSpans && traceSpans.length > 0) {
|
||||
if (traceSpans.length > 0) {
|
||||
const startTime = new Date(endTime.getTime() - Math.max(1, durationMs))
|
||||
createOTelSpansForWorkflowExecution({
|
||||
workflowId: this.workflowId,
|
||||
@@ -947,6 +1201,7 @@ export class LoggingSession {
|
||||
isError: false,
|
||||
finalizationPath: 'fallback_completed',
|
||||
finalOutput: params.finalOutput || {},
|
||||
executionState: params.executionState,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -976,6 +1231,7 @@ export class LoggingSession {
|
||||
finalOutput: {
|
||||
error: params?.error?.message || `Execution failed to store trace spans: ${errorMsg}`,
|
||||
},
|
||||
executionState: params?.executionState,
|
||||
status: 'failed',
|
||||
})
|
||||
}
|
||||
@@ -1005,6 +1261,7 @@ export class LoggingSession {
|
||||
isError: false,
|
||||
finalizationPath: 'cancelled',
|
||||
finalOutput: { cancelled: true },
|
||||
executionState: params?.executionState,
|
||||
status: 'cancelled',
|
||||
})
|
||||
}
|
||||
@@ -1032,6 +1289,7 @@ export class LoggingSession {
|
||||
isError: false,
|
||||
finalizationPath: 'paused',
|
||||
finalOutput: { paused: true },
|
||||
executionState: params?.executionState,
|
||||
status: 'pending',
|
||||
})
|
||||
}
|
||||
@@ -1128,6 +1386,7 @@ export class LoggingSession {
|
||||
isError: boolean
|
||||
finalizationPath: ExecutionFinalizationPath
|
||||
finalOutput?: Record<string, unknown>
|
||||
executionState?: SerializableExecutionState
|
||||
status?: 'completed' | 'failed' | 'cancelled' | 'pending'
|
||||
}): Promise<void> {
|
||||
if (this.completed || this.completing) {
|
||||
@@ -1155,6 +1414,7 @@ export class LoggingSession {
|
||||
costSummary,
|
||||
finalOutput,
|
||||
traceSpans: [],
|
||||
executionState: params.executionState,
|
||||
finalizationPath: params.finalizationPath,
|
||||
completionFailure: params.errorMessage,
|
||||
level: params.isError ? 'error' : 'info',
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { decryptSecretMock } = vi.hoisted(() => ({
|
||||
decryptSecretMock: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/core/security/encryption', () => ({
|
||||
decryptSecret: decryptSecretMock,
|
||||
}))
|
||||
|
||||
import { projectExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
|
||||
const CONTEXT = {
|
||||
workspaceId: 'workspace-1',
|
||||
workflowId: 'workflow-1',
|
||||
executionId: 'execution-1',
|
||||
userId: 'user-1',
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
decryptSecretMock.mockResolvedValue({ decrypted: '1234' })
|
||||
})
|
||||
|
||||
describe('projectExecutionDataForDisplay', () => {
|
||||
it('projects persisted output, input, errors, and spans from trusted provenance', async () => {
|
||||
const executionData = {
|
||||
finalOutput: { result: 1234, derived: 1239 },
|
||||
workflowInput: { nested: { token: 'prefix-1234-suffix' } },
|
||||
completionFailure: 'Function failed with 1234',
|
||||
errorDetails: { blockId: 'function-1', error: 'Invalid token 1234' },
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Function 1',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-31T00:00:00.000Z',
|
||||
endTime: '2026-07-31T00:00:00.001Z',
|
||||
output: { result: 1234 },
|
||||
},
|
||||
],
|
||||
executionState: {
|
||||
blockStates: { 'function-1': { output: { result: 1234 } } },
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'OPENAI_API_KEY', encryptedValue: 'ciphertext' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
const displayData = await projectExecutionDataForDisplay(executionData, CONTEXT)
|
||||
|
||||
expect(displayData.finalOutput).toEqual({
|
||||
result: '{{OPENAI_API_KEY}}',
|
||||
derived: 1239,
|
||||
})
|
||||
expect(displayData.workflowInput).toEqual({
|
||||
nested: { token: 'prefix-{{OPENAI_API_KEY}}-suffix' },
|
||||
})
|
||||
expect(displayData.completionFailure).toBe('Function failed with {{OPENAI_API_KEY}}')
|
||||
expect(displayData.errorDetails).toEqual({
|
||||
blockId: 'function-1',
|
||||
error: 'Invalid token {{OPENAI_API_KEY}}',
|
||||
})
|
||||
expect(displayData.traceSpans).toEqual([
|
||||
expect.objectContaining({ output: { result: '{{OPENAI_API_KEY}}' } }),
|
||||
])
|
||||
expect(displayData).not.toHaveProperty('executionState')
|
||||
expect(executionData.finalOutput).toEqual({ result: 1234, derived: 1239 })
|
||||
expect(executionData.executionState.resolvedSecretTraceProvenance.entries).toEqual([
|
||||
{ name: 'OPENAI_API_KEY', encryptedValue: 'ciphertext' },
|
||||
])
|
||||
expect(JSON.stringify(displayData)).not.toContain('1234')
|
||||
})
|
||||
|
||||
it('omits log content and keeps only structural traces without trusted provenance', async () => {
|
||||
const displayData = await projectExecutionDataForDisplay(
|
||||
{
|
||||
finalOutput: { result: 'unknown-secret' },
|
||||
workflowInput: { token: 'unknown-secret' },
|
||||
completionFailure: 'unknown-secret',
|
||||
traceSpans: [
|
||||
{
|
||||
id: 'span-1',
|
||||
name: 'Function 1',
|
||||
type: 'function',
|
||||
duration: 1,
|
||||
startTime: '2026-07-31T00:00:00.000Z',
|
||||
endTime: '2026-07-31T00:00:00.001Z',
|
||||
output: { result: 'unknown-secret' },
|
||||
},
|
||||
],
|
||||
},
|
||||
CONTEXT
|
||||
)
|
||||
|
||||
expect(displayData).not.toHaveProperty('finalOutput')
|
||||
expect(displayData).not.toHaveProperty('workflowInput')
|
||||
expect(displayData).not.toHaveProperty('completionFailure')
|
||||
expect(displayData.traceSpans).toEqual([
|
||||
expect.not.objectContaining({ output: expect.anything() }),
|
||||
])
|
||||
})
|
||||
|
||||
it('preserves direct literals when trusted provenance has no activated secrets', async () => {
|
||||
const displayData = await projectExecutionDataForDisplay(
|
||||
{
|
||||
finalOutput: { result: 'direct-literal' },
|
||||
executionState: {
|
||||
resolvedSecretTraceProvenance: {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
},
|
||||
},
|
||||
},
|
||||
CONTEXT
|
||||
)
|
||||
|
||||
expect(displayData.finalOutput).toEqual({ result: 'direct-literal' })
|
||||
})
|
||||
|
||||
it('omits malformed trace content even when it was present on the stored row', async () => {
|
||||
const displayData = await projectExecutionDataForDisplay(
|
||||
{
|
||||
traceSpans: { output: 'unsafe' },
|
||||
},
|
||||
CONTEXT
|
||||
)
|
||||
|
||||
expect(displayData).not.toHaveProperty('traceSpans')
|
||||
})
|
||||
})
|
||||
@@ -1,7 +1,14 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { omit } from '@sim/utils/object'
|
||||
import { isLargeValueRef } from '@/lib/execution/payloads/large-value-ref'
|
||||
import { materializeLargeValueRef, storeLargeValue } from '@/lib/execution/payloads/store'
|
||||
import { projectTraceSpansForSecrets } from '@/lib/logs/execution/trace-secret-projection'
|
||||
import type { TraceSpan } from '@/lib/logs/types'
|
||||
import {
|
||||
isResolvedSecretTraceProvenanceV1,
|
||||
ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('TraceStore')
|
||||
|
||||
@@ -26,10 +33,11 @@ const INLINE_MARKER_KEYS = ['hasTraceSpans', 'traceSpanCount'] as const
|
||||
* Read-path context. Resolves an externalized payload by storage key, authorized
|
||||
* via the (already-authorized) workspace — no owner needed.
|
||||
*/
|
||||
interface TraceStoreReadContext {
|
||||
export interface TraceStoreReadContext {
|
||||
workspaceId: string | null
|
||||
workflowId: string | null
|
||||
executionId: string
|
||||
userId?: string
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -69,6 +77,14 @@ export function stripSpanCosts(spans: unknown): void {
|
||||
}
|
||||
}
|
||||
|
||||
/** Creates a persistence-owned span tree with per-span cost fields removed. */
|
||||
export function copyTraceSpansWithoutCosts(spans?: TraceSpan[]): TraceSpan[] | undefined {
|
||||
return spans?.map(({ cost: _cost, children, ...span }) => ({
|
||||
...span,
|
||||
...(children ? { children: copyTraceSpansWithoutCosts(children) } : {}),
|
||||
}))
|
||||
}
|
||||
|
||||
/**
|
||||
* Externalizes heavy `execution_data` to object storage as a single large value
|
||||
* (reusing the execution-context large-value store + its reference/dependency/GC
|
||||
@@ -177,3 +193,105 @@ export async function materializeExecutionData(
|
||||
return markers
|
||||
}
|
||||
}
|
||||
|
||||
const LOG_DISPLAY_CONTENT_KEYS = [
|
||||
'finalOutput',
|
||||
'workflowInput',
|
||||
'blockInput',
|
||||
'blockExecutions',
|
||||
'error',
|
||||
'errorDetails',
|
||||
'completionFailure',
|
||||
'message',
|
||||
] as const
|
||||
|
||||
const LOG_DISPLAY_PROJECTION_SPAN_ID = 'secret-safe-log-display-projection'
|
||||
|
||||
/**
|
||||
* Materializes trusted execution data and returns its log-facing projection.
|
||||
* Functional readers must continue using {@link materializeExecutionData}.
|
||||
*/
|
||||
export async function materializeExecutionDataForDisplay(
|
||||
executionData: Record<string, unknown> | null | undefined,
|
||||
context: TraceStoreReadContext
|
||||
): Promise<Record<string, unknown>> {
|
||||
const materialized = await materializeExecutionData(executionData, context)
|
||||
return projectExecutionDataForDisplay(materialized, context)
|
||||
}
|
||||
|
||||
/**
|
||||
* Projects execution-log content with the encrypted provenance saved by the
|
||||
* trusted executor. Missing or malformed provenance deliberately yields a
|
||||
* structural-only log instead of returning content that cannot be proven safe.
|
||||
*/
|
||||
export async function projectExecutionDataForDisplay(
|
||||
executionData: Record<string, unknown>,
|
||||
context: TraceStoreReadContext
|
||||
): Promise<Record<string, unknown>> {
|
||||
const executionState =
|
||||
executionData.executionState &&
|
||||
typeof executionData.executionState === 'object' &&
|
||||
!Array.isArray(executionData.executionState)
|
||||
? (executionData.executionState as Record<string, unknown>)
|
||||
: undefined
|
||||
const provenance = executionState?.resolvedSecretTraceProvenance
|
||||
let registry: ResolvedSecretTraceRegistry | undefined
|
||||
|
||||
if (isResolvedSecretTraceProvenanceV1(provenance)) {
|
||||
registry = new ResolvedSecretTraceRegistry([], provenance.scope)
|
||||
await registry.importProvenance(provenance, { trusted: true })
|
||||
}
|
||||
|
||||
const envelope: Record<string, unknown> = {}
|
||||
for (const key of LOG_DISPLAY_CONTENT_KEYS) {
|
||||
if (Object.hasOwn(executionData, key)) envelope[key] = executionData[key]
|
||||
}
|
||||
|
||||
const now = new Date().toISOString()
|
||||
const syntheticSpan: TraceSpan = {
|
||||
id: LOG_DISPLAY_PROJECTION_SPAN_ID,
|
||||
name: 'Log Display Projection',
|
||||
type: 'display',
|
||||
duration: 0,
|
||||
startTime: now,
|
||||
endTime: now,
|
||||
output: envelope,
|
||||
}
|
||||
const sourceTraceSpans = Array.isArray(executionData.traceSpans)
|
||||
? (executionData.traceSpans as TraceSpan[])
|
||||
: []
|
||||
const projectedSpans = await projectTraceSpansForSecrets([syntheticSpan, ...sourceTraceSpans], {
|
||||
registry,
|
||||
allowLargeValueWrites: false,
|
||||
store: {
|
||||
workspaceId: context.workspaceId ?? undefined,
|
||||
workflowId: context.workflowId ?? undefined,
|
||||
executionId: context.executionId,
|
||||
userId: context.userId,
|
||||
trackReference: false,
|
||||
},
|
||||
})
|
||||
|
||||
const displayData = omit(executionData, [
|
||||
...LOG_DISPLAY_CONTENT_KEYS,
|
||||
'executionState',
|
||||
'traceSpans',
|
||||
]) as Record<string, unknown>
|
||||
|
||||
const projectedEnvelope = projectedSpans.find(
|
||||
(span) => span.id === LOG_DISPLAY_PROJECTION_SPAN_ID
|
||||
)?.output
|
||||
if (projectedEnvelope) {
|
||||
for (const key of LOG_DISPLAY_CONTENT_KEYS) {
|
||||
if (Object.hasOwn(projectedEnvelope, key)) displayData[key] = projectedEnvelope[key]
|
||||
}
|
||||
}
|
||||
|
||||
if (Array.isArray(executionData.traceSpans)) {
|
||||
displayData.traceSpans = projectedSpans.filter(
|
||||
(span) => span.id !== LOG_DISPLAY_PROJECTION_SPAN_ID
|
||||
)
|
||||
}
|
||||
|
||||
return displayData
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@ import {
|
||||
pickLatestCompletedMarker,
|
||||
pickLatestStartedMarker,
|
||||
} from '@/lib/logs/execution/progress-markers'
|
||||
import { materializeExecutionData } from '@/lib/logs/execution/trace-store'
|
||||
import { materializeExecutionDataForDisplay } from '@/lib/logs/execution/trace-store'
|
||||
import { checkWorkspaceAccess } from '@/lib/workspaces/permissions/utils'
|
||||
|
||||
type LookupColumn = 'id' | 'executionId'
|
||||
@@ -170,9 +170,14 @@ export async function fetchLogDetail({
|
||||
|
||||
// Trace spans / heavy execution data may live in object storage; resolve the
|
||||
// pointer here (no-op for inline / pre-externalization rows).
|
||||
const executionData = await materializeExecutionData(
|
||||
const executionData = await materializeExecutionDataForDisplay(
|
||||
log.executionData as Record<string, unknown> | null,
|
||||
{ workspaceId, workflowId: log.workflowId, executionId: log.executionId }
|
||||
{
|
||||
workspaceId,
|
||||
workflowId: log.workflowId,
|
||||
executionId: log.executionId,
|
||||
userId,
|
||||
}
|
||||
)
|
||||
|
||||
const liveMarkers =
|
||||
@@ -248,7 +253,15 @@ export async function fetchLogDetail({
|
||||
const jobLog = jobRows[0]
|
||||
if (!jobLog) return null
|
||||
|
||||
const execData = (jobLog.executionData as Record<string, unknown> | null) ?? {}
|
||||
const execData = await materializeExecutionDataForDisplay(
|
||||
jobLog.executionData as Record<string, unknown> | null,
|
||||
{
|
||||
workspaceId,
|
||||
workflowId: null,
|
||||
executionId: jobLog.executionId,
|
||||
userId,
|
||||
}
|
||||
)
|
||||
return {
|
||||
id: jobLog.id,
|
||||
workflowId: null,
|
||||
|
||||
@@ -433,6 +433,22 @@ export interface SnapshotCreationResult {
|
||||
}
|
||||
|
||||
export interface ExecutionLoggerService {
|
||||
loadTraceSpansForProjection(params: {
|
||||
executionId: string
|
||||
workflowId: string
|
||||
workspaceId: string | null
|
||||
traceSpans: TraceSpan[]
|
||||
isResume?: boolean
|
||||
}): Promise<TraceSpan[]>
|
||||
|
||||
prepareTraceSpansForProjection(params: {
|
||||
executionId: string
|
||||
workflowId: string
|
||||
workspaceId: string | null
|
||||
userId?: string | null
|
||||
traceSpans: TraceSpan[]
|
||||
}): Promise<TraceSpan[]>
|
||||
|
||||
startWorkflowExecution(params: {
|
||||
workflowId: string
|
||||
workspaceId: string
|
||||
|
||||
@@ -74,6 +74,7 @@ export class McpClient {
|
||||
private authProvider?: McpClientOptions['authProvider']
|
||||
private isConnected = false
|
||||
private closeGuardedTransport?: () => Promise<void>
|
||||
private readonly resolvedSecretTraceProvenance?: McpClientOptions['resolvedSecretTraceProvenance']
|
||||
|
||||
constructor(options: McpClientOptions) {
|
||||
this.config = options.config
|
||||
@@ -84,6 +85,7 @@ export class McpClient {
|
||||
}
|
||||
this.onToolsChanged = options.onToolsChanged
|
||||
this.authProvider = options.authProvider
|
||||
this.resolvedSecretTraceProvenance = options.resolvedSecretTraceProvenance
|
||||
const resolvedIP = options.resolvedIP
|
||||
|
||||
this.connectionStatus = { connected: false }
|
||||
@@ -133,6 +135,12 @@ export class McpClient {
|
||||
}
|
||||
}
|
||||
|
||||
getResolvedSecretTraceProvenance(): McpClientOptions['resolvedSecretTraceProvenance'] {
|
||||
return this.resolvedSecretTraceProvenance
|
||||
? structuredClone(this.resolvedSecretTraceProvenance)
|
||||
: undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize connection to MCP server.
|
||||
* If an `onToolsChanged` callback was provided, registers a notification handler
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockGetEffectiveEnvironmentSnapshot } = vi.hoisted(() => ({
|
||||
mockGetEffectiveEnvironmentSnapshot: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/environment/utils', () => ({
|
||||
getEffectiveEnvironmentSnapshot: mockGetEffectiveEnvironmentSnapshot,
|
||||
}))
|
||||
|
||||
import { resolveMcpConfigEnvVars } from '@/lib/mcp/resolve-config'
|
||||
|
||||
const BASE_CONFIG = {
|
||||
id: 'server-1',
|
||||
name: 'Server',
|
||||
transport: 'streamable-http' as const,
|
||||
url: 'https://{{MCP_HOST}}/mcp',
|
||||
headers: {
|
||||
Authorization: 'Bearer {{MCP_TOKEN}}',
|
||||
'X-Direct': 'literal-secret',
|
||||
},
|
||||
}
|
||||
|
||||
describe('resolveMcpConfigEnvVars secret provenance', () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
|
||||
personalEncrypted: {
|
||||
MCP_HOST: 'personal-host-encrypted',
|
||||
MCP_TOKEN: 'personal-token-encrypted',
|
||||
UNUSED: 'unused-encrypted',
|
||||
},
|
||||
workspaceEncrypted: { MCP_TOKEN: 'workspace-token-encrypted' },
|
||||
personalDecrypted: {
|
||||
MCP_HOST: 'api.example.com',
|
||||
MCP_TOKEN: 'personal-token',
|
||||
UNUSED: 'literal-secret',
|
||||
},
|
||||
workspaceDecrypted: { MCP_TOKEN: 'workspace-token' },
|
||||
conflicts: ['MCP_TOKEN'],
|
||||
decryptionFailures: [],
|
||||
})
|
||||
})
|
||||
|
||||
it('returns encrypted provenance only for successful {{NAME}} substitutions', async () => {
|
||||
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
|
||||
|
||||
expect(result.config.url).toBe('https://api.example.com/mcp')
|
||||
expect(result.config.headers).toEqual({
|
||||
Authorization: 'Bearer workspace-token',
|
||||
'X-Direct': 'literal-secret',
|
||||
})
|
||||
expect(result.resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ name: 'MCP_HOST', encryptedValue: 'personal-host-encrypted' },
|
||||
{ name: 'MCP_TOKEN', encryptedValue: 'workspace-token-encrypted' },
|
||||
],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
expect(JSON.stringify(result.resolvedSecretTraceProvenance)).not.toContain('workspace-token"')
|
||||
expect(JSON.stringify(result.resolvedSecretTraceProvenance)).not.toContain('literal-secret')
|
||||
})
|
||||
|
||||
it('marks provenance incomplete when a resolved value has no encrypted catalog entry', async () => {
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
|
||||
personalEncrypted: {},
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: { MCP_HOST: 'api.example.com', MCP_TOKEN: 'token' },
|
||||
workspaceDecrypted: {},
|
||||
conflicts: [],
|
||||
decryptionFailures: [],
|
||||
})
|
||||
|
||||
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
|
||||
|
||||
expect(result.resolvedSecretTraceProvenance).toEqual({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('does not let an unused configured-secret failure affect invocation provenance', async () => {
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValue({
|
||||
personalEncrypted: {
|
||||
MCP_HOST: 'host-encrypted',
|
||||
MCP_TOKEN: 'token-encrypted',
|
||||
UNUSED: 'broken-encrypted',
|
||||
},
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: {
|
||||
MCP_HOST: 'api.example.com',
|
||||
MCP_TOKEN: 'token',
|
||||
UNUSED: '',
|
||||
},
|
||||
workspaceDecrypted: {},
|
||||
conflicts: [],
|
||||
decryptionFailures: ['UNUSED'],
|
||||
})
|
||||
|
||||
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
|
||||
|
||||
expect(result.resolvedSecretTraceProvenance.complete).toBe(true)
|
||||
expect(result.resolvedSecretTraceProvenance.entries).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('reports successful substitutions before strict missing-reference failure', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
const config = {
|
||||
...BASE_CONFIG,
|
||||
headers: {
|
||||
Authorization: 'Bearer {{MCP_TOKEN}}',
|
||||
'X-Missing': '{{NOT_CONFIGURED}}',
|
||||
},
|
||||
}
|
||||
|
||||
await expect(
|
||||
resolveMcpConfigEnvVars(config, 'user-1', 'workspace-1', {
|
||||
onResolvedSecretTraceProvenance: onProvenance,
|
||||
})
|
||||
).rejects.toThrow('NOT_CONFIGURED')
|
||||
|
||||
expect(onProvenance).toHaveBeenCalledWith({
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [
|
||||
{ name: 'MCP_HOST', encryptedValue: 'personal-host-encrypted' },
|
||||
{ name: 'MCP_TOKEN', encryptedValue: 'workspace-token-encrypted' },
|
||||
],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
})
|
||||
})
|
||||
|
||||
it('reports incomplete provenance when the Secrets catalog cannot be loaded', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
mockGetEffectiveEnvironmentSnapshot.mockRejectedValueOnce(new Error('database unavailable'))
|
||||
|
||||
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1', {
|
||||
onResolvedSecretTraceProvenance: onProvenance,
|
||||
})
|
||||
|
||||
const expected = {
|
||||
version: 1 as const,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: 'user-1', workspaceId: 'workspace-1' },
|
||||
}
|
||||
expect(result.resolvedSecretTraceProvenance).toEqual(expected)
|
||||
expect(onProvenance).toHaveBeenCalledWith(expected)
|
||||
expect(result.config).toEqual(BASE_CONFIG)
|
||||
})
|
||||
|
||||
it('uses one atomic cached snapshot for runtime values and encrypted provenance', async () => {
|
||||
mockGetEffectiveEnvironmentSnapshot.mockResolvedValueOnce({
|
||||
personalEncrypted: { MCP_HOST: 'old-host-ciphertext', MCP_TOKEN: 'old-token-ciphertext' },
|
||||
workspaceEncrypted: {},
|
||||
personalDecrypted: { MCP_HOST: 'old.example.com', MCP_TOKEN: 'old-token' },
|
||||
workspaceDecrypted: {},
|
||||
conflicts: [],
|
||||
decryptionFailures: [],
|
||||
})
|
||||
|
||||
const result = await resolveMcpConfigEnvVars(BASE_CONFIG, 'user-1', 'workspace-1')
|
||||
|
||||
expect(mockGetEffectiveEnvironmentSnapshot).toHaveBeenCalledOnce()
|
||||
expect(mockGetEffectiveEnvironmentSnapshot).toHaveBeenCalledWith('user-1', 'workspace-1')
|
||||
expect(result.config).toMatchObject({
|
||||
url: 'https://old.example.com/mcp',
|
||||
headers: expect.objectContaining({ Authorization: 'Bearer old-token' }),
|
||||
})
|
||||
expect(result.resolvedSecretTraceProvenance.entries).toEqual([
|
||||
{ name: 'MCP_HOST', encryptedValue: 'old-host-ciphertext' },
|
||||
{ name: 'MCP_TOKEN', encryptedValue: 'old-token-ciphertext' },
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -5,15 +5,26 @@
|
||||
*/
|
||||
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { getEffectiveDecryptedEnv } from '@/lib/environment/utils'
|
||||
import { getErrorMessage } from '@sim/utils/errors'
|
||||
import {
|
||||
type EnvironmentResolutionSnapshot,
|
||||
getEffectiveEnvironmentSnapshot,
|
||||
} from '@/lib/environment/utils'
|
||||
import type { McpServerConfig } from '@/lib/mcp/types'
|
||||
import { resolveEnvVarReferences } from '@/executor/utils/reference-validation'
|
||||
import {
|
||||
createIncompleteResolvedSecretTraceRegistry,
|
||||
createResolvedSecretTraceRegistry,
|
||||
type ResolvedSecretTraceProvenanceV1,
|
||||
type ResolvedSecretTraceRegistry,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('McpResolveConfig')
|
||||
|
||||
export interface ResolveMcpConfigOptions {
|
||||
/** If true, throws an error when env vars are missing. Default: true */
|
||||
strict?: boolean
|
||||
onResolvedSecretTraceProvenance?: (provenance: ResolvedSecretTraceProvenanceV1) => void
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -31,22 +42,55 @@ export async function resolveMcpConfigEnvVars(
|
||||
userId: string,
|
||||
workspaceId?: string,
|
||||
options: ResolveMcpConfigOptions = {}
|
||||
): Promise<{ config: McpServerConfig; missingVars: string[] }> {
|
||||
): Promise<{
|
||||
config: McpServerConfig
|
||||
missingVars: string[]
|
||||
resolvedSecretTraceProvenance: ResolvedSecretTraceProvenanceV1
|
||||
}> {
|
||||
const { strict = true } = options
|
||||
const allMissingVars: string[] = []
|
||||
const scope = { userId, ...(workspaceId ? { workspaceId } : {}) }
|
||||
|
||||
let envVars: Record<string, string> = {}
|
||||
let env: EnvironmentResolutionSnapshot
|
||||
try {
|
||||
envVars = await getEffectiveDecryptedEnv(userId, workspaceId)
|
||||
env = await getEffectiveEnvironmentSnapshot(userId, workspaceId)
|
||||
} catch (error) {
|
||||
logger.error('Failed to fetch environment variables for MCP config:', error)
|
||||
return { config, missingVars: [] }
|
||||
const resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(scope)
|
||||
const provenance = resolvedSecretTraceRegistry.exportProvenance()
|
||||
options.onResolvedSecretTraceProvenance?.(provenance)
|
||||
return {
|
||||
config,
|
||||
missingVars: [],
|
||||
resolvedSecretTraceProvenance: provenance,
|
||||
}
|
||||
}
|
||||
const envVars = { ...env.personalDecrypted, ...env.workspaceDecrypted }
|
||||
|
||||
let resolvedSecretTraceRegistry: ResolvedSecretTraceRegistry
|
||||
try {
|
||||
resolvedSecretTraceRegistry = await createResolvedSecretTraceRegistry({
|
||||
personalEncrypted: env.personalEncrypted,
|
||||
workspaceEncrypted: env.workspaceEncrypted,
|
||||
personalDecrypted: env.personalDecrypted,
|
||||
workspaceDecrypted: env.workspaceDecrypted,
|
||||
decryptionFailures: env.decryptionFailures,
|
||||
scope,
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn('Failed to build MCP trace secret catalog; provenance will be incomplete', {
|
||||
error: getErrorMessage(error),
|
||||
})
|
||||
resolvedSecretTraceRegistry = createIncompleteResolvedSecretTraceRegistry(scope)
|
||||
}
|
||||
|
||||
const resolveValue = (value: string): string => {
|
||||
const missingVars: string[] = []
|
||||
const resolved = resolveEnvVarReferences(value, envVars, {
|
||||
missingKeys: missingVars,
|
||||
onResolved: (name, resolvedValue) => {
|
||||
resolvedSecretTraceRegistry.recordResolved(name, resolvedValue)
|
||||
},
|
||||
}) as string
|
||||
allMissingVars.push(...missingVars)
|
||||
return resolved
|
||||
@@ -66,7 +110,9 @@ export async function resolveMcpConfigEnvVars(
|
||||
resolvedConfig.headers = resolvedHeaders
|
||||
}
|
||||
|
||||
// Handle missing vars based on strict mode
|
||||
const resolvedSecretTraceProvenance = resolvedSecretTraceRegistry.exportProvenance()
|
||||
options.onResolvedSecretTraceProvenance?.(resolvedSecretTraceProvenance)
|
||||
|
||||
if (allMissingVars.length > 0) {
|
||||
const uniqueMissing = Array.from(new Set(allMissingVars))
|
||||
|
||||
@@ -81,5 +127,9 @@ export async function resolveMcpConfigEnvVars(
|
||||
})
|
||||
}
|
||||
|
||||
return { config: resolvedConfig, missingVars: allMissingVars }
|
||||
return {
|
||||
config: resolvedConfig,
|
||||
missingVars: allMissingVars,
|
||||
resolvedSecretTraceProvenance,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
const {
|
||||
MockMcpClient,
|
||||
mockCallTool,
|
||||
mockListTools,
|
||||
mockConnect,
|
||||
mockDisconnect,
|
||||
mockAcquire,
|
||||
@@ -24,12 +25,24 @@ const {
|
||||
poolClient,
|
||||
} = vi.hoisted(() => {
|
||||
const mockCallTool = vi.fn()
|
||||
const mockListTools = vi.fn()
|
||||
const mockConnect = vi.fn()
|
||||
const mockDisconnect = vi.fn()
|
||||
const mockRelease = vi.fn(async () => {})
|
||||
const poolClient = { callTool: mockCallTool, disconnect: vi.fn() }
|
||||
const poolClient = {
|
||||
callTool: mockCallTool,
|
||||
listTools: mockListTools,
|
||||
disconnect: vi.fn(),
|
||||
getResolvedSecretTraceProvenance: vi.fn(() => ({
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: 'user-1', workspaceId: 'ws-1' },
|
||||
})),
|
||||
}
|
||||
return {
|
||||
mockCallTool,
|
||||
mockListTools,
|
||||
mockConnect,
|
||||
mockDisconnect,
|
||||
mockRelease,
|
||||
@@ -50,9 +63,10 @@ const {
|
||||
connect: mockConnect,
|
||||
disconnect: mockDisconnect,
|
||||
callTool: mockCallTool,
|
||||
listTools: vi.fn(async () => []),
|
||||
listTools: mockListTools,
|
||||
hasListChangedCapability: vi.fn(() => false),
|
||||
onClose: vi.fn(),
|
||||
getResolvedSecretTraceProvenance: poolClient.getResolvedSecretTraceProvenance,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -117,6 +131,7 @@ describe('McpService connection reuse wiring', () => {
|
||||
mockResolveEnvVars.mockImplementation(async (config: unknown) => ({ config }))
|
||||
mockAcquire.mockResolvedValue({ client: poolClient, release: mockRelease })
|
||||
mockCallTool.mockResolvedValue({ content: [] })
|
||||
mockListTools.mockResolvedValue([])
|
||||
})
|
||||
|
||||
afterAll(() => {
|
||||
@@ -137,6 +152,161 @@ describe('McpService connection reuse wiring', () => {
|
||||
expect(mockResolveEnvVars).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('emits the pooled connection provenance on every invocation', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
|
||||
await mcpService.executeTool(
|
||||
USER_ID,
|
||||
'server-1',
|
||||
{ name: 'first', arguments: {} },
|
||||
WORKSPACE_ID,
|
||||
undefined,
|
||||
onProvenance
|
||||
)
|
||||
await mcpService.executeTool(
|
||||
USER_ID,
|
||||
'server-1',
|
||||
{ name: 'second', arguments: {} },
|
||||
WORKSPACE_ID,
|
||||
undefined,
|
||||
onProvenance
|
||||
)
|
||||
|
||||
expect(onProvenance).toHaveBeenCalledTimes(2)
|
||||
expect(onProvenance).toHaveBeenNthCalledWith(1, {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
})
|
||||
expect(onProvenance).toHaveBeenNthCalledWith(2, {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
})
|
||||
})
|
||||
|
||||
it('emits cold-connection provenance once even though the connected client retains it', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
const provenance = {
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
}
|
||||
mockResolveEnvVars.mockImplementationOnce(
|
||||
async (
|
||||
config: unknown,
|
||||
_userId: unknown,
|
||||
_workspaceId: unknown,
|
||||
options: {
|
||||
onResolvedSecretTraceProvenance?: (value: typeof provenance) => void
|
||||
}
|
||||
) => {
|
||||
options.onResolvedSecretTraceProvenance?.(provenance)
|
||||
return { config, resolvedSecretTraceProvenance: provenance }
|
||||
}
|
||||
)
|
||||
mockAcquire.mockImplementationOnce(
|
||||
async ({ create }: { create: () => Promise<typeof poolClient> }) => ({
|
||||
client: await create(),
|
||||
release: mockRelease,
|
||||
})
|
||||
)
|
||||
|
||||
await mcpService.executeTool(
|
||||
USER_ID,
|
||||
'server-1',
|
||||
{ name: 'do', arguments: {} },
|
||||
WORKSPACE_ID,
|
||||
undefined,
|
||||
onProvenance
|
||||
)
|
||||
|
||||
expect(mockResolveEnvVars).toHaveBeenCalledTimes(1)
|
||||
expect(onProvenance).toHaveBeenCalledTimes(1)
|
||||
expect(onProvenance).toHaveBeenCalledWith(provenance)
|
||||
})
|
||||
|
||||
it('emits cold-connection provenance for tools/list', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
const provenance = {
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
}
|
||||
mockResolveEnvVars.mockImplementationOnce(
|
||||
async (
|
||||
config: unknown,
|
||||
_userId: unknown,
|
||||
_workspaceId: unknown,
|
||||
options: {
|
||||
onResolvedSecretTraceProvenance?: (value: typeof provenance) => void
|
||||
}
|
||||
) => {
|
||||
options.onResolvedSecretTraceProvenance?.(provenance)
|
||||
return { config, resolvedSecretTraceProvenance: provenance }
|
||||
}
|
||||
)
|
||||
mockAcquire.mockImplementationOnce(
|
||||
async ({ create }: { create: () => Promise<typeof poolClient> }) => ({
|
||||
client: await create(),
|
||||
release: mockRelease,
|
||||
})
|
||||
)
|
||||
|
||||
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
|
||||
|
||||
expect(mockResolveEnvVars).toHaveBeenCalledTimes(1)
|
||||
expect(mockListTools).toHaveBeenCalledTimes(1)
|
||||
expect(onProvenance).toHaveBeenCalledTimes(1)
|
||||
expect(onProvenance).toHaveBeenCalledWith(provenance)
|
||||
})
|
||||
|
||||
it('emits retained provenance on every warm-pool tools/list invocation', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
|
||||
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
|
||||
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
|
||||
|
||||
expect(mockResolveEnvVars).not.toHaveBeenCalled()
|
||||
expect(mockListTools).toHaveBeenCalledTimes(2)
|
||||
expect(onProvenance).toHaveBeenCalledTimes(2)
|
||||
expect(onProvenance).toHaveBeenNthCalledWith(1, {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
})
|
||||
expect(onProvenance).toHaveBeenNthCalledWith(2, {
|
||||
version: 1,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
})
|
||||
})
|
||||
|
||||
it('reports incomplete provenance when a pooled tools/list client has no retained report', async () => {
|
||||
const onProvenance = vi.fn()
|
||||
const legacyClient = {
|
||||
...poolClient,
|
||||
getResolvedSecretTraceProvenance: undefined,
|
||||
}
|
||||
mockAcquire.mockResolvedValueOnce({ client: legacyClient, release: mockRelease })
|
||||
|
||||
await mcpService.discoverServerTools(USER_ID, 'server-1', WORKSPACE_ID, true, onProvenance)
|
||||
|
||||
expect(mockListTools).toHaveBeenCalledTimes(1)
|
||||
expect(onProvenance).toHaveBeenCalledWith({
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
})
|
||||
})
|
||||
|
||||
it('bypasses the pool for calls carrying per-request headers', async () => {
|
||||
await mcpService.executeTool(USER_ID, 'server-1', { name: 'do', arguments: {} }, WORKSPACE_ID, {
|
||||
Authorization: 'Bearer per-call',
|
||||
@@ -226,4 +396,48 @@ describe('McpService connection reuse wiring', () => {
|
||||
expect(mockRelease).toHaveBeenCalledWith(true, false)
|
||||
expect(mockAcquire).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('reports both retained and rotated provenance when an auth retry rebuilds the client', async () => {
|
||||
const staleProvenance = {
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token-v1' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
}
|
||||
const rotatedProvenance = {
|
||||
version: 1 as const,
|
||||
complete: true,
|
||||
entries: [{ name: 'MCP_TOKEN', encryptedValue: 'encrypted-token-v2' }],
|
||||
scope: { userId: USER_ID, workspaceId: WORKSPACE_ID },
|
||||
}
|
||||
const staleClient = {
|
||||
...poolClient,
|
||||
getResolvedSecretTraceProvenance: vi.fn(() => staleProvenance),
|
||||
}
|
||||
const rotatedClient = {
|
||||
...poolClient,
|
||||
getResolvedSecretTraceProvenance: vi.fn(() => rotatedProvenance),
|
||||
}
|
||||
mockAcquire
|
||||
.mockResolvedValueOnce({ client: staleClient, release: mockRelease })
|
||||
.mockResolvedValueOnce({ client: rotatedClient, release: mockRelease })
|
||||
mockCallTool
|
||||
.mockRejectedValueOnce(new UnauthorizedError('stale key'))
|
||||
.mockResolvedValueOnce({ content: [] })
|
||||
const onProvenance = vi.fn()
|
||||
|
||||
await mcpService.executeTool(
|
||||
USER_ID,
|
||||
'server-1',
|
||||
{ name: 'do', arguments: {} },
|
||||
WORKSPACE_ID,
|
||||
undefined,
|
||||
onProvenance
|
||||
)
|
||||
|
||||
expect(onProvenance.mock.calls.map(([provenance]) => provenance)).toEqual([
|
||||
staleProvenance,
|
||||
rotatedProvenance,
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
+141
-21
@@ -42,6 +42,10 @@ import {
|
||||
type McpTransport,
|
||||
} from '@/lib/mcp/types'
|
||||
import { MCP_CLIENT_CONSTANTS, MCP_CONSTANTS } from '@/lib/mcp/utils'
|
||||
import {
|
||||
isResolvedSecretTraceProvenanceV1,
|
||||
type ResolvedSecretTraceProvenanceV1,
|
||||
} from '@/executor/utils/resolved-secret-trace-registry'
|
||||
|
||||
const logger = createLogger('McpService')
|
||||
|
||||
@@ -55,6 +59,59 @@ function failureCacheKey(workspaceId: string, serverId: string): string {
|
||||
|
||||
const FAILURE_CACHE_SENTINEL: McpTool[] = []
|
||||
|
||||
type ResolvedSecretTraceProvenanceCallback = (provenance: ResolvedSecretTraceProvenanceV1) => void
|
||||
|
||||
function reportRetainedClientProvenance(
|
||||
provenance: unknown,
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
callback?: ResolvedSecretTraceProvenanceCallback
|
||||
): void {
|
||||
if (!callback) return
|
||||
callback(
|
||||
isResolvedSecretTraceProvenanceV1(provenance)
|
||||
? provenance
|
||||
: {
|
||||
version: 1,
|
||||
complete: false,
|
||||
entries: [],
|
||||
scope: { userId, workspaceId },
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
function isSameProvenance(
|
||||
left: ResolvedSecretTraceProvenanceV1,
|
||||
right: ResolvedSecretTraceProvenanceV1
|
||||
): boolean {
|
||||
if (
|
||||
left.complete !== right.complete ||
|
||||
left.scope?.userId !== right.scope?.userId ||
|
||||
left.scope?.workspaceId !== right.scope?.workspaceId ||
|
||||
left.entries.length !== right.entries.length
|
||||
) {
|
||||
return false
|
||||
}
|
||||
|
||||
return left.entries.every((entry, index) => {
|
||||
const other = right.entries[index]
|
||||
return entry.name === other.name && entry.encryptedValue === other.encryptedValue
|
||||
})
|
||||
}
|
||||
|
||||
function createInvocationProvenanceReporter(
|
||||
callback?: ResolvedSecretTraceProvenanceCallback
|
||||
): ResolvedSecretTraceProvenanceCallback | undefined {
|
||||
if (!callback) return undefined
|
||||
|
||||
let lastReported: ResolvedSecretTraceProvenanceV1 | undefined
|
||||
return (provenance) => {
|
||||
if (lastReported && isSameProvenance(lastReported, provenance)) return
|
||||
lastReported = provenance
|
||||
callback(provenance)
|
||||
}
|
||||
}
|
||||
|
||||
type DiscoveryOutcome =
|
||||
| { kind: 'cached'; tools: McpTool[] }
|
||||
| { kind: 'fetched'; tools: McpTool[] }
|
||||
@@ -212,14 +269,25 @@ class McpService {
|
||||
private async resolveConfigEnvVars(
|
||||
config: McpServerConfig,
|
||||
userId: string,
|
||||
workspaceId?: string
|
||||
): Promise<{ config: McpServerConfig; resolvedIP: string | null }> {
|
||||
const { config: resolvedConfig } = await resolveMcpConfigEnvVars(config, userId, workspaceId, {
|
||||
strict: true,
|
||||
})
|
||||
workspaceId?: string,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<{
|
||||
config: McpServerConfig
|
||||
resolvedIP: string | null
|
||||
resolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceV1
|
||||
}> {
|
||||
const { config: resolvedConfig, resolvedSecretTraceProvenance } = await resolveMcpConfigEnvVars(
|
||||
config,
|
||||
userId,
|
||||
workspaceId,
|
||||
{
|
||||
strict: true,
|
||||
onResolvedSecretTraceProvenance,
|
||||
}
|
||||
)
|
||||
validateMcpDomain(resolvedConfig.url)
|
||||
const resolvedIP = await validateMcpServerSsrf(resolvedConfig.url)
|
||||
return { config: resolvedConfig, resolvedIP }
|
||||
return { config: resolvedConfig, resolvedIP, resolvedSecretTraceProvenance }
|
||||
}
|
||||
|
||||
private async getServerConfig(
|
||||
@@ -298,7 +366,8 @@ class McpService {
|
||||
private async createClient(
|
||||
config: McpServerConfig,
|
||||
resolvedIP: string | null,
|
||||
userId?: string
|
||||
userId?: string,
|
||||
resolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceV1
|
||||
): Promise<McpClient> {
|
||||
const securityPolicy = {
|
||||
requireConsent: true,
|
||||
@@ -312,6 +381,7 @@ class McpService {
|
||||
config,
|
||||
securityPolicy,
|
||||
resolvedIP: resolvedIP ?? undefined,
|
||||
resolvedSecretTraceProvenance,
|
||||
})
|
||||
await client.connect()
|
||||
return client
|
||||
@@ -343,6 +413,7 @@ class McpService {
|
||||
securityPolicy,
|
||||
authProvider,
|
||||
resolvedIP: resolvedIP ?? undefined,
|
||||
resolvedSecretTraceProvenance,
|
||||
})
|
||||
await client.connect()
|
||||
return client
|
||||
@@ -367,18 +438,24 @@ class McpService {
|
||||
config: McpServerConfig,
|
||||
userId: string,
|
||||
workspaceId: string,
|
||||
extraHeaders?: Record<string, string>
|
||||
extraHeaders?: Record<string, string>,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): () => Promise<McpClient> {
|
||||
return async () => {
|
||||
const { config: resolvedConfig, resolvedIP } = await this.resolveConfigEnvVars(
|
||||
const {
|
||||
config: resolvedConfig,
|
||||
resolvedIP,
|
||||
resolvedSecretTraceProvenance,
|
||||
} = await this.resolveConfigEnvVars(
|
||||
config,
|
||||
userId,
|
||||
workspaceId
|
||||
workspaceId,
|
||||
onResolvedSecretTraceProvenance
|
||||
)
|
||||
if (extraHeaders) {
|
||||
resolvedConfig.headers = { ...resolvedConfig.headers, ...extraHeaders }
|
||||
}
|
||||
return this.createClient(resolvedConfig, resolvedIP, userId)
|
||||
return this.createClient(resolvedConfig, resolvedIP, userId, resolvedSecretTraceProvenance)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -392,7 +469,8 @@ class McpService {
|
||||
private async fetchServerTools(
|
||||
config: McpServerConfig,
|
||||
userId: string,
|
||||
workspaceId: string
|
||||
workspaceId: string,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<McpTool[]> {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try {
|
||||
@@ -402,8 +480,16 @@ class McpService {
|
||||
serverId: config.id,
|
||||
allowPool: true,
|
||||
},
|
||||
this.buildClient(config, userId, workspaceId),
|
||||
(client) => client.listTools()
|
||||
this.buildClient(config, userId, workspaceId, undefined, onResolvedSecretTraceProvenance),
|
||||
(client) => {
|
||||
reportRetainedClientProvenance(
|
||||
client.getResolvedSecretTraceProvenance?.(),
|
||||
userId,
|
||||
workspaceId,
|
||||
onResolvedSecretTraceProvenance
|
||||
)
|
||||
return client.listTools()
|
||||
}
|
||||
)
|
||||
} catch (error) {
|
||||
if (attempt === 0 && isAuthError(error) && config.authType !== 'oauth') continue
|
||||
@@ -462,10 +548,12 @@ class McpService {
|
||||
serverId: string,
|
||||
toolCall: McpToolCall,
|
||||
workspaceId: string,
|
||||
extraHeaders?: Record<string, string>
|
||||
extraHeaders?: Record<string, string>,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<McpToolResult> {
|
||||
const requestId = generateRequestId()
|
||||
const maxRetries = 2
|
||||
const reportProvenance = createInvocationProvenanceReporter(onResolvedSecretTraceProvenance)
|
||||
|
||||
for (let attempt = 0; attempt < maxRetries; attempt++) {
|
||||
try {
|
||||
@@ -485,8 +573,22 @@ class McpService {
|
||||
serverId,
|
||||
allowPool: !hasExtraHeaders,
|
||||
},
|
||||
this.buildClient(config, userId, workspaceId, hasExtraHeaders ? extraHeaders : undefined),
|
||||
(client) => client.callTool(toolCall)
|
||||
this.buildClient(
|
||||
config,
|
||||
userId,
|
||||
workspaceId,
|
||||
hasExtraHeaders ? extraHeaders : undefined,
|
||||
reportProvenance
|
||||
),
|
||||
(client) => {
|
||||
reportRetainedClientProvenance(
|
||||
client.getResolvedSecretTraceProvenance?.(),
|
||||
userId,
|
||||
workspaceId,
|
||||
reportProvenance
|
||||
)
|
||||
return client.callTool(toolCall)
|
||||
}
|
||||
)
|
||||
logger.info(`[${requestId}] Successfully executed tool ${toolCall.name}`)
|
||||
return result
|
||||
@@ -874,8 +976,19 @@ class McpService {
|
||||
userId: string,
|
||||
serverId: string,
|
||||
workspaceId: string,
|
||||
forceRefresh = false
|
||||
forceRefresh = false,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<McpTool[]> {
|
||||
if (onResolvedSecretTraceProvenance) {
|
||||
return this.discoverServerToolsImpl(
|
||||
userId,
|
||||
serverId,
|
||||
workspaceId,
|
||||
forceRefresh,
|
||||
createInvocationProvenanceReporter(onResolvedSecretTraceProvenance)
|
||||
)
|
||||
}
|
||||
|
||||
const inflightKey = `${workspaceId}:${serverId}:${userId}:${forceRefresh ? 'force' : 'cache'}`
|
||||
const existing = this.inflightServerDiscovery.get(inflightKey)
|
||||
if (existing) return existing
|
||||
@@ -884,7 +997,8 @@ class McpService {
|
||||
userId,
|
||||
serverId,
|
||||
workspaceId,
|
||||
forceRefresh
|
||||
forceRefresh,
|
||||
undefined
|
||||
).finally(() => {
|
||||
this.inflightServerDiscovery.delete(inflightKey)
|
||||
})
|
||||
@@ -896,7 +1010,8 @@ class McpService {
|
||||
userId: string,
|
||||
serverId: string,
|
||||
workspaceId: string,
|
||||
forceRefresh: boolean
|
||||
forceRefresh: boolean,
|
||||
onResolvedSecretTraceProvenance?: ResolvedSecretTraceProvenanceCallback
|
||||
): Promise<McpTool[]> {
|
||||
const requestId = generateRequestId()
|
||||
const discoveryStartedAt = new Date()
|
||||
@@ -934,7 +1049,12 @@ class McpService {
|
||||
}
|
||||
authType = config.authType
|
||||
|
||||
const tools = await this.fetchServerTools(config, userId, workspaceId)
|
||||
const tools = await this.fetchServerTools(
|
||||
config,
|
||||
userId,
|
||||
workspaceId,
|
||||
onResolvedSecretTraceProvenance
|
||||
)
|
||||
logger.info(`[${requestId}] Discovered ${tools.length} tools from server ${config.name}`)
|
||||
await Promise.allSettled([
|
||||
this.cacheAdapter
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user