opensource/sim
3
0
Fork 0
mirror of https://github.com/simstudioai/sim.git synced 2026-08-29 02:27:35 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(agent-stream): thinking and tool streaming (#5671)

Browse Source 
* feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas

Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): clear stuck streaming UI and format db snapshot

Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle
assistant streaming/tool flags when SSE ends without a terminal frame,
without clobbering Stop's finalized content.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): satisfy biome format and import order

Auto-format the sim package for CI lint:check, and repair the Anthropic
streaming tool-loop payload after an unsafe delete-to-undefined rewrite.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): keep drained answer on abort and update migration journal test

Treat AbortError from reader.cancel as a cancelled pump result so soft-complete
retains answerText. Point the workspace storage migration journal assertion at
0261_chat_include_thinking.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(chat): keep Stop notice when server emits cancel error

Ignore terminal SSE error frames after the user aborts so
"Client cancelled request" cannot overwrite "Response stopped by user".

Co-authored-by: Cursor <cursoragent@cursor.com>

* improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll

Add left-to-right shimmer on live thinking label/body, keep scroll working by
shimmering an inner node, and follow the answer only while near the bottom.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): stop pump on client disconnect; soft-complete agents only

Abort the agent stream pump when the projected HTTP body is cancelled so
provider work does not continue after disconnect. Limit AbortError soft-success
to Agent blocks so Function/HTTP cancels still fail in logs.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): persist includeThinking across pause snapshots

Paused chat runs with Include thinking enabled were dropping the flag when
serializing the pause snapshot, so resume always rebuilt streams without
thinking/tool SSE frames.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): keep drained answer text when stream times out

Persist pump answerText onto the streaming execution before throwing on
timeout, and carry that partial content into the failed block output so
logs match what the client already saw.

Co-authored-by: Cursor <cursoragent@cursor.com>

* improvement(chat): auto-collapse tools chrome when tool streaming ends

Match thinking UX: open while tools run, collapse when finished, and keep
the panel open only if the user manually reopens it.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): settle canvas stream chrome on failure paths

Clear agentStreamActive and settle running tool chips when blocks error,
timeouts cancel runs, or execution ends without stream:done so the output
panel does not stay on live Thinking/Using tools chrome.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(lint): organize imports in terminal console store

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(agent-stream): mark open tools cancelled on HITL pause

Pause can interrupt a tool loop without tool end events; settling those
chips as success incorrectly showed unfinished tools as complete.

Co-authored-by: Cursor <cursoragent@cursor.com>

* chore(db): drop branch-local 0261 migration ahead of staging merge

* chore(db): regenerate include_thinking migration as 0266 post staging merge

* fix(providers): resolve type errors in streaming tool loop call sites

* fix(agent-stream): gate agent events opt-in and correct provider loop behavior

- streamToolCalls and provider thinking requests now require run-level
  agentEvents opt-in (canvas on, chat dual-gated, API off) so existing
  runs keep pre-agent-events behavior exactly
- OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400
- streaming loops run tool postProcess again (firecrawl/exa async results)
- bedrock live loop falls back to silent path for responseFormat
- deepseek: reasoning_content pass-back unconditional, 'none' sends disabled
- groq: x_groq.usage fallback, reasoning params gated, qwen none disables
- gemini: functionCall parts echoed verbatim, local ids only for events
- truncated turns (max_tokens/length) no longer execute partial tool calls
- MAX_TOOL_ITERATIONS exit flushes last turn text as final answer
- iterations reports actual model calls; shared loop plumbing extracted

* refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene

- canonical ChatStreamFrame union + type guards consumed by server emitters
  and the chat client; stream_error restored to legacy log-only handling
- strip thinking/tool args from providerTiming on public final envelopes
- shared tool-chip lifecycle module for chat, canvas, and console store
- shared sink-to-execution-events forwarder replaces the copy-pasted
  adapter in the execute route and HITL manager; LIVE_ONLY event set shared
- stream:thinking payload field renamed data->text; canvas thinking batched
- abort reasons carried as AbortError DOMExceptions so raw fetch consumers
  classify correctly; thinking cap renamed to chars and scope-documented
- kimi wired for agent events like the other compat providers
- deleted dead exports/step-N comments; fixtures match real wire shapes;
  loop tests use explicit mocks instead of importOriginal

* test(agent-stream): cover the dual-gated execution path and typed abort reasons

- chat route tests assert agentEvents reaches executeWorkflow only when
  policy and protocol header agree
- execution-limits tests assert AbortError-typed reasons
- executor metadata type carries agentEvents

* fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm

* docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page

- capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts,
  explicit for the Anthropic family where visibility varies per generation;
  getThinkingStreamVisibility exposes the derivation for docs and UI alike
- scripts/sync-agent-stream-docs.ts regenerates the support tables between
  markers in workflows/blocks/agent.mdx from the model registry and
  STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata
- wired agent-stream-docs:check into CI next to the other sync gates

* feat(anthropic): request summarized thinking display for omitted-default Claude models

The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default
thinking.display to omitted — empty thinking blocks, no deltas. On
agent-events runs Sim now opts back in with display: 'summarized', driven
by the registry's streamed metadata; legacy runs keep the exact
pre-agent-events request shape. Registry, generated docs, and the family
capability table updated accordingly.

* docs(skills): cover thinking.streamed and agent-stream docs sync in model skills

* chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types

- adaptive thinking, display, and output_config are now SDK-typed; the only
  remaining custom payload field is output_format (beta-header structured
  outputs, which the SDK models as output_config.format instead)
- anthropic stream events narrow on the SDK's discriminated unions instead
  of anonymous casts; compat deltas type content/tool_calls from the OpenAI
  SDK with vendor reasoning fields as an explicit optional extension
- @sim/auth exposes an explicit VerifyAuth contract so its declarations no
  longer reference better-auth's nested zod instance (TS2883 under fresh
  install layouts); realtime consumer aligned
- docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the
  same zod instance (docs type-check was latently broken)
- knowledge embedding tests made hermetic against local .env keys and
  hosted rotation fallback

* refactor(providers): replace legacy as-any stream casts with annotated typed casts

* refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts

Audit of all 26 providers for the agent-events feature confirmed every
streaming execution declares agent-events-v1 and every adapter emits
AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy
createOpenAICompatibleStream byte helper is deleted, and the remaining
streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are
annotated typed casts matching the groq/deepseek fix.

* feat(streaming): stream answer text live during tool loops via turn_end protocol

The live tool loops buffered all answer text per model turn (classification
of intermediate vs final is only known at turn end), so gated surfaces saw
thinking stream, then dead air with the thinking chrome stuck open, then the
whole answer at once.

Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end`
event per turn. The pump buffers pending text and projects it to the byte
path (answerText/logs/memory/legacy clients) only on a final turn_end, so
all settled semantics are unchanged. Gated surfaces render the pending text
as it streams and reconcile with a reset when a turn resolves to tools:

- public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`;
  byte-path frame emission is suppressed to avoid duplicates (kept for
  response-format transformed streams via clientStreamTransformed)
- canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the
  execute route and HITL resume readers stop re-emitting byte chunks; panel
  chat tracks per-block segments and replaces content on flush
- chat client: per-block text segments, chunk_reset handling, and thinking
  chrome now settles on tool start as well as first answer chunk

* fix(streaming): address validated review findings across provider gating and reset reconciliation

Three-reviewer pass over the branch, findings validated against staging:

- agent-handler forwards agentEvents to executeProviderRequest — the flag was
  computed but dropped in the field-by-field copy, so provider-side thinking
  requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized
  display) never activated on opted-in runs
- openai: restore summary:'auto' alongside explicit reasoning effort — staging
  always paired them; gating summary purely on agentEvents changed legacy
  payloads
- gemini: Gemini 2 + tools + responseFormat falls back to the silent path;
  the live loop never applied the deferred responseSchema for AUTO tools
- openai-compat loop: malformed tool-argument JSON fails the call instead of
  executing with defaulted {} args (staging parsed inside the execution try)
- openai-compat parser: a vendor id arriving after a synthesized start no
  longer renames the call (start/end ids stayed consistent)
- stream-pump: abort closes the byte projection so a drain blocked on
  backpressure cannot deadlock teardown
- chunk_reset removes the block from the client text order (deployed chat +
  panel chat) so a reset block re-registers at arrival position — fixes
  separator/order corruption when parallel blocks stream around a reset
- resume route echoes the negotiated X-Sim-Stream-Protocol response header
  (parity with the chat route); docs: [DONE] wire shape + final-vs-error
  terminal semantics corrected

* chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate

CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23,
younger than the 7-day supply-chain gate). The pin is exact and was vetted
for the agent-events streaming work; following the existing bunfig pattern,
the exclusion ages out on 2026-07-30 and should be dropped then.

* chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit

The audit only recognizes the annotation on the line directly above the cast;
two annotations had drifted behind intervening code lines (groq stream params,
deepseek loop messages) and the OpenAI reasoning-summary widening cast was
never annotated. No behavior change.

* fix(chat): settle straggler tool chips as error when final reports failure

A failed run can still terminate with a `final` frame carrying success: false;
running chips previously settled green regardless of the outcome.

* fix(canvas): wire agent stream chrome into run-from-block

Run-from-block executions emit the same live stream:thinking/stream:tool
events as full runs but registered none of the handlers, so the terminal
never showed thinking or tool chips on that path. The per-run chrome
(batched thinking writes + tool chip lifecycle + settlement on stream done,
block error, and every terminal execution state) is extracted into a shared
createAgentStreamChrome factory consumed by both paths.

---------

Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
This commit is contained in:
Bill Leoutsakos
2026-07-23 19:39:03 -07:00
committed by GitHub
parent 03adc8fe1d
commit d24bc7eccb
165 changed files with 30705 additions and 826 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/add-model/SKILL.md
+12
View File
@@ -52,6 +52,7 @@ Use a precise WebFetch prompt: *"Extract for {model_id}: exact model id string,
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped to thinking), `gemini/core.ts` | **Dead on xai, deepseek, mistral, groq, cerebras, openrouter, fireworks, bedrock, vertex** unless their core consumes it — re-grep before assuming |
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` only | Dead elsewhere |
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` | Dead elsewhere |
| `thinking.streamed` | Docs generator + `getThinkingStreamVisibility` (`models.ts`); `anthropic/core.ts` uses `'summary'` to request `display: 'summarized'` on agent-events runs | **Mandatory on Anthropic-family thinking models** (`agent-stream-docs:check` fails without it); other families fall back to provider defaults |
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` | Dead on openai, xai, google, vertex, bedrock, azure-openai, deepseek, mistral, groq, cerebras |
| `maxOutputTokens` | Read by UI + executor for token estimation | Always meaningful — set if provider documents a cap |
| `computerUse` | `anthropic/core.ts` | Dead elsewhere |
@@ -146,6 +147,15 @@ If anything matches, run the affected provider tests and update assertions as ne
The Consumption Matrix (Step 2) tells you which capability *flags* are honored by existing provider code. But if the new model needs **net-new** request handling that the provider doesn't implement yet — a new beta header (e.g. Anthropic's `anthropic-beta` structured-outputs header in `anthropic/index.ts`), a new thinking/reasoning encoding, a Responses-API quirk — you must edit `apps/sim/providers/<provider>/core.ts` / `index.ts`. Setting a flag whose behavior isn't implemented is a silent no-op. When you do edit provider code, reuse the shared helpers rather than hand-rolling: streaming responses are assembled via `createStreamingExecution` (`@/providers/streaming-execution`) and tool schemas via `adaptOpenAIChatToolSchema` / `adaptAnthropicToolSchema` (`@/providers/tool-schema-adapter`).
### Thinking/reasoning models: `streamed` visibility + generated docs
If the entry has `capabilities.thinking` or `capabilities.reasoningEffort`, it appears in the autogenerated "Streamed thinking and tool calls" table on the Agent block docs page:
- **Anthropic-family (`anthropic`, `azure-anthropic`) thinking models MUST declare `capabilities.thinking.streamed`** (`'full' | 'summary' | 'none'`) — visibility varies per Claude generation. Verify against the provider's thinking/display docs (e.g. Anthropic's "controlling thinking display" page and per-model "what's new" notes): generations that default `thinking.display` to `omitted` (Opus 4.7+, Sonnet 5, Fable 5) are `'summary'` — Sim opts back in with `display: 'summarized'` on agent-events runs; older generations that return full thinking deltas are `'full'`. `bun run agent-stream-docs:check` (CI) fails if the field is missing.
- Other families usually omit the field and inherit the provider default in `getThinkingStreamVisibility` (Gemini/OpenAI → summaries, Bedrock → none, OpenAI-compat vendors → full raw CoT). Set it explicitly only when the model deviates from its family.
- After inserting the entry, run `bun run agent-stream-docs:generate` and commit the regenerated `apps/docs/content/docs/en/workflows/blocks/agent.mdx` — CI diffs it.
- Include the `streamed` value (with its source URL) in the verification report when set.
### Wrong family entirely?
- **Embedding or rerank model** → it does NOT go in the `models[]` array. Use `EMBEDDING_MODEL_PRICING` / `RERANK_MODEL_PRICING` in `models.ts` instead.
@@ -155,6 +165,7 @@ The Consumption Matrix (Step 2) tells you which capability *flags* are honored b
```bash
bun run lint
bun run agent-stream-docs:generate # only when the entry has thinking/reasoningEffort
```
Lint must pass before reporting done. **If lint fails:** read the error, fix the syntax/typing issue in the entry you just wrote (do not delete the entry — it's the work product), re-run lint, and note the fix in a "Lint adjustments" line in the verification report. Never report done with lint failing.
@@ -201,6 +212,7 @@ Omitting a field is **not the same as verifying it**. Any field you cannot confi
- ❌ Trusting a marketing email (xAI's grok-4.3 email claimed "3 reasoning efforts" but the API rejects `reasoning_effort` — verified by official docs only)
- ❌ Setting `nativeStructuredOutputs: true` on xai/openai/google (dead — only anthropic/fireworks/openrouter consume it)
- ❌ Setting `thinking` on non-Anthropic/non-Gemini providers
- ❌ Adding an Anthropic-family thinking model without `thinking.streamed` (CI `agent-stream-docs:check` fails), or skipping `bun run agent-stream-docs:generate` after adding any thinking/reasoning model
- ❌ Setting `verbosity` on anything other than OpenAI gpt-5.x
- ❌ Copying `pricing.updatedAt` from a sibling instead of using today's date
- ❌ Inventing a `cachedInput` price by dividing input by 4 (varies by provider — find an explicit number)
.agents/skills/validate-model/SKILL.md
+1
View File
@@ -89,6 +89,7 @@ For each model, evaluate every row. Statuses: ✓ matches docs, ✗ disagrees,
- [ ] `reasoningEffort.values` — list matches docs; **omitted** for always-reasoning models that reject the parameter (e.g., grok-4.3, where xAI docs explicitly state `reasoning_effort` is not supported). Verify per model — some always-reasoning models (e.g., OpenAI's o-series) DO accept `reasoning_effort` and should keep the flag.
- [ ] `verbosity.values` — only on OpenAI gpt-5.x family; values match docs
- [ ] `thinking.levels` + `thinking.default` — only on Anthropic/Gemini; values match docs
- [ ] `thinking.streamed` — REQUIRED on Anthropic-family thinking models (`'full'` for generations returning full thinking deltas, `'summary'` for omitted-display generations like Opus 4.7+/Sonnet 5/Fable 5 where Sim requests `display: 'summarized'`); verify against the provider's thinking-display docs. After any change, run `bun run agent-stream-docs:generate` so the Agent block docs table stays in sync (CI diffs it)
- [ ] `nativeStructuredOutputs` — only on anthropic/fireworks/openrouter; provider must document Structured Outputs / JSON-mode for this model
- [ ] `toolUsageControl` — provider supports `tool_choice` semantics
- [ ] `computerUse` — provider implements computer-use loop AND model is a computer-use SKU
.claude/commands/add-model.md
+12
View File
@@ -51,6 +51,7 @@ Use a precise WebFetch prompt: *"Extract for {model_id}: exact model id string,
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped to thinking), `gemini/core.ts` | **Dead on xai, deepseek, mistral, groq, cerebras, openrouter, fireworks, bedrock, vertex** unless their core consumes it — re-grep before assuming |
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` only | Dead elsewhere |
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` | Dead elsewhere |
| `thinking.streamed` | Docs generator + `getThinkingStreamVisibility` (`models.ts`); `anthropic/core.ts` uses `'summary'` to request `display: 'summarized'` on agent-events runs | **Mandatory on Anthropic-family thinking models** (`agent-stream-docs:check` fails without it); other families fall back to provider defaults |
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` | Dead on openai, xai, google, vertex, bedrock, azure-openai, deepseek, mistral, groq, cerebras |
| `maxOutputTokens` | Read by UI + executor for token estimation | Always meaningful — set if provider documents a cap |
| `computerUse` | `anthropic/core.ts` | Dead elsewhere |
@@ -145,6 +146,15 @@ If anything matches, run the affected provider tests and update assertions as ne
The Consumption Matrix (Step 2) tells you which capability *flags* are honored by existing provider code. But if the new model needs **net-new** request handling that the provider doesn't implement yet — a new beta header (e.g. Anthropic's `anthropic-beta` structured-outputs header in `anthropic/index.ts`), a new thinking/reasoning encoding, a Responses-API quirk — you must edit `apps/sim/providers/<provider>/core.ts` / `index.ts`. Setting a flag whose behavior isn't implemented is a silent no-op. When you do edit provider code, reuse the shared helpers rather than hand-rolling: streaming responses are assembled via `createStreamingExecution` (`@/providers/streaming-execution`) and tool schemas via `adaptOpenAIChatToolSchema` / `adaptAnthropicToolSchema` (`@/providers/tool-schema-adapter`).
### Thinking/reasoning models: `streamed` visibility + generated docs
If the entry has `capabilities.thinking` or `capabilities.reasoningEffort`, it appears in the autogenerated "Streamed thinking and tool calls" table on the Agent block docs page:
- **Anthropic-family (`anthropic`, `azure-anthropic`) thinking models MUST declare `capabilities.thinking.streamed`** (`'full' | 'summary' | 'none'`) — visibility varies per Claude generation. Verify against the provider's thinking/display docs (e.g. Anthropic's "controlling thinking display" page and per-model "what's new" notes): generations that default `thinking.display` to `omitted` (Opus 4.7+, Sonnet 5, Fable 5) are `'summary'` — Sim opts back in with `display: 'summarized'` on agent-events runs; older generations that return full thinking deltas are `'full'`. `bun run agent-stream-docs:check` (CI) fails if the field is missing.
- Other families usually omit the field and inherit the provider default in `getThinkingStreamVisibility` (Gemini/OpenAI → summaries, Bedrock → none, OpenAI-compat vendors → full raw CoT). Set it explicitly only when the model deviates from its family.
- After inserting the entry, run `bun run agent-stream-docs:generate` and commit the regenerated `apps/docs/content/docs/en/workflows/blocks/agent.mdx` — CI diffs it.
- Include the `streamed` value (with its source URL) in the verification report when set.
### Wrong family entirely?
- **Embedding or rerank model** → it does NOT go in the `models[]` array. Use `EMBEDDING_MODEL_PRICING` / `RERANK_MODEL_PRICING` in `models.ts` instead.
@@ -154,6 +164,7 @@ The Consumption Matrix (Step 2) tells you which capability *flags* are honored b
```bash
bun run lint
bun run agent-stream-docs:generate # only when the entry has thinking/reasoningEffort
```
Lint must pass before reporting done. **If lint fails:** read the error, fix the syntax/typing issue in the entry you just wrote (do not delete the entry — it's the work product), re-run lint, and note the fix in a "Lint adjustments" line in the verification report. Never report done with lint failing.
@@ -200,6 +211,7 @@ Omitting a field is **not the same as verifying it**. Any field you cannot confi
- ❌ Trusting a marketing email (xAI's grok-4.3 email claimed "3 reasoning efforts" but the API rejects `reasoning_effort` — verified by official docs only)
- ❌ Setting `nativeStructuredOutputs: true` on xai/openai/google (dead — only anthropic/fireworks/openrouter consume it)
- ❌ Setting `thinking` on non-Anthropic/non-Gemini providers
- ❌ Adding an Anthropic-family thinking model without `thinking.streamed` (CI `agent-stream-docs:check` fails), or skipping `bun run agent-stream-docs:generate` after adding any thinking/reasoning model
- ❌ Setting `verbosity` on anything other than OpenAI gpt-5.x
- ❌ Copying `pricing.updatedAt` from a sibling instead of using today's date
- ❌ Inventing a `cachedInput` price by dividing input by 4 (varies by provider — find an explicit number)
.claude/commands/validate-model.md
+1
View File
@@ -88,6 +88,7 @@ For each model, evaluate every row. Statuses: ✓ matches docs, ✗ disagrees,
- [ ] `reasoningEffort.values` — list matches docs; **omitted** for always-reasoning models that reject the parameter (e.g., grok-4.3, where xAI docs explicitly state `reasoning_effort` is not supported). Verify per model — some always-reasoning models (e.g., OpenAI's o-series) DO accept `reasoning_effort` and should keep the flag.
- [ ] `verbosity.values` — only on OpenAI gpt-5.x family; values match docs
- [ ] `thinking.levels` + `thinking.default` — only on Anthropic/Gemini; values match docs
- [ ] `thinking.streamed` — REQUIRED on Anthropic-family thinking models (`'full'` for generations returning full thinking deltas, `'summary'` for omitted-display generations like Opus 4.7+/Sonnet 5/Fable 5 where Sim requests `display: 'summarized'`); verify against the provider's thinking-display docs. After any change, run `bun run agent-stream-docs:generate` so the Agent block docs table stays in sync (CI diffs it)
- [ ] `nativeStructuredOutputs` — only on anthropic/fireworks/openrouter; provider must document Structured Outputs / JSON-mode for this model
- [ ] `toolUsageControl` — provider supports `tool_choice` semantics
- [ ] `computerUse` — provider implements computer-use loop AND model is a computer-use SKU
.cursor/commands/add-model.md
+12
View File
@@ -46,6 +46,7 @@ Use a precise WebFetch prompt: *"Extract for {model_id}: exact model id string,
| `reasoningEffort` | `openai/core.ts`, `azure-openai`, `anthropic/core.ts` (mapped to thinking), `gemini/core.ts` | **Dead on xai, deepseek, mistral, groq, cerebras, openrouter, fireworks, bedrock, vertex** unless their core consumes it — re-grep before assuming |
| `verbosity` | `openai/core.ts`, `azure-openai/index.ts` only | Dead elsewhere |
| `thinking` | `anthropic/core.ts`, `gemini/core.ts` | Dead elsewhere |
| `thinking.streamed` | Docs generator + `getThinkingStreamVisibility` (`models.ts`); `anthropic/core.ts` uses `'summary'` to request `display: 'summarized'` on agent-events runs | **Mandatory on Anthropic-family thinking models** (`agent-stream-docs:check` fails without it); other families fall back to provider defaults |
| `nativeStructuredOutputs` | `anthropic/core.ts`, `fireworks/index.ts`, `openrouter/index.ts` | Dead on openai, xai, google, vertex, bedrock, azure-openai, deepseek, mistral, groq, cerebras |
| `maxOutputTokens` | Read by UI + executor for token estimation | Always meaningful — set if provider documents a cap |
| `computerUse` | `anthropic/core.ts` | Dead elsewhere |
@@ -140,6 +141,15 @@ If anything matches, run the affected provider tests and update assertions as ne
The Consumption Matrix (Step 2) tells you which capability *flags* are honored by existing provider code. But if the new model needs **net-new** request handling that the provider doesn't implement yet — a new beta header (e.g. Anthropic's `anthropic-beta` structured-outputs header in `anthropic/index.ts`), a new thinking/reasoning encoding, a Responses-API quirk — you must edit `apps/sim/providers/<provider>/core.ts` / `index.ts`. Setting a flag whose behavior isn't implemented is a silent no-op. When you do edit provider code, reuse the shared helpers rather than hand-rolling: streaming responses are assembled via `createStreamingExecution` (`@/providers/streaming-execution`) and tool schemas via `adaptOpenAIChatToolSchema` / `adaptAnthropicToolSchema` (`@/providers/tool-schema-adapter`).
### Thinking/reasoning models: `streamed` visibility + generated docs
If the entry has `capabilities.thinking` or `capabilities.reasoningEffort`, it appears in the autogenerated "Streamed thinking and tool calls" table on the Agent block docs page:
- **Anthropic-family (`anthropic`, `azure-anthropic`) thinking models MUST declare `capabilities.thinking.streamed`** (`'full' | 'summary' | 'none'`) — visibility varies per Claude generation. Verify against the provider's thinking/display docs (e.g. Anthropic's "controlling thinking display" page and per-model "what's new" notes): generations that default `thinking.display` to `omitted` (Opus 4.7+, Sonnet 5, Fable 5) are `'summary'` — Sim opts back in with `display: 'summarized'` on agent-events runs; older generations that return full thinking deltas are `'full'`. `bun run agent-stream-docs:check` (CI) fails if the field is missing.
- Other families usually omit the field and inherit the provider default in `getThinkingStreamVisibility` (Gemini/OpenAI → summaries, Bedrock → none, OpenAI-compat vendors → full raw CoT). Set it explicitly only when the model deviates from its family.
- After inserting the entry, run `bun run agent-stream-docs:generate` and commit the regenerated `apps/docs/content/docs/en/workflows/blocks/agent.mdx` — CI diffs it.
- Include the `streamed` value (with its source URL) in the verification report when set.
### Wrong family entirely?
- **Embedding or rerank model** → it does NOT go in the `models[]` array. Use `EMBEDDING_MODEL_PRICING` / `RERANK_MODEL_PRICING` in `models.ts` instead.
@@ -149,6 +159,7 @@ The Consumption Matrix (Step 2) tells you which capability *flags* are honored b
```bash
bun run lint
bun run agent-stream-docs:generate # only when the entry has thinking/reasoningEffort
```
Lint must pass before reporting done. **If lint fails:** read the error, fix the syntax/typing issue in the entry you just wrote (do not delete the entry — it's the work product), re-run lint, and note the fix in a "Lint adjustments" line in the verification report. Never report done with lint failing.
@@ -195,6 +206,7 @@ Omitting a field is **not the same as verifying it**. Any field you cannot confi
- ❌ Trusting a marketing email (xAI's grok-4.3 email claimed "3 reasoning efforts" but the API rejects `reasoning_effort` — verified by official docs only)
- ❌ Setting `nativeStructuredOutputs: true` on xai/openai/google (dead — only anthropic/fireworks/openrouter consume it)
- ❌ Setting `thinking` on non-Anthropic/non-Gemini providers
- ❌ Adding an Anthropic-family thinking model without `thinking.streamed` (CI `agent-stream-docs:check` fails), or skipping `bun run agent-stream-docs:generate` after adding any thinking/reasoning model
- ❌ Setting `verbosity` on anything other than OpenAI gpt-5.x
- ❌ Copying `pricing.updatedAt` from a sibling instead of using today's date
- ❌ Inventing a `cachedInput` price by dividing input by 4 (varies by provider — find an explicit number)
.cursor/commands/validate-model.md
+1
View File
@@ -83,6 +83,7 @@ For each model, evaluate every row. Statuses: ✓ matches docs, ✗ disagrees,
- [ ] `reasoningEffort.values` — list matches docs; **omitted** for always-reasoning models that reject the parameter (e.g., grok-4.3, where xAI docs explicitly state `reasoning_effort` is not supported). Verify per model — some always-reasoning models (e.g., OpenAI's o-series) DO accept `reasoning_effort` and should keep the flag.
- [ ] `verbosity.values` — only on OpenAI gpt-5.x family; values match docs
- [ ] `thinking.levels` + `thinking.default` — only on Anthropic/Gemini; values match docs
- [ ] `thinking.streamed` — REQUIRED on Anthropic-family thinking models (`'full'` for generations returning full thinking deltas, `'summary'` for omitted-display generations like Opus 4.7+/Sonnet 5/Fable 5 where Sim requests `display: 'summarized'`); verify against the provider's thinking-display docs. After any change, run `bun run agent-stream-docs:generate` so the Agent block docs table stays in sync (CI diffs it)
- [ ] `nativeStructuredOutputs` — only on anthropic/fireworks/openrouter; provider must document Structured Outputs / JSON-mode for this model
- [ ] `toolUsageControl` — provider supports `tool_choice` semantics
- [ ] `computerUse` — provider implements computer-use loop AND model is a computer-use SKU
.github/workflows/test-build.yml
+3
View File
@@ -140,6 +140,9 @@ jobs:
- name: Verify skill projections are in sync
run: bun run skills:check
- name: Verify agent stream capability docs are in sync
run: bun run agent-stream-docs:check
- name: Migration safety (zero-downtime) audit
run: |
if [ "${{ github.event_name }}" = "pull_request" ]; then
apps/docs/content/docs/en/workflows/blocks/agent.mdx
+25
View File
@@ -96,6 +96,31 @@ After the agent runs, later blocks read its result by name:
When a response format is set, its fields are readable directly, like `<agent.sentiment>`.
## Streamed thinking and tool calls
While an agent runs, Sim can stream its thinking and tool lifecycle live — the canvas terminal always shows them, and a [deployed chat](/docs/workflows/deployment/agent-events) shows them when its **Include thinking** setting and the client's protocol opt-in agree. What actually streams depends on the model: models marked with full deltas or summaries stream thinking when a Thinking level or Reasoning effort is set (DeepSeek reasoner models always reason); a model marked "Not streamed" thinks internally but its provider withholds the text.
{/* agent-stream-capabilities:begin — generated by `bun run agent-stream-docs:generate`; do not edit between markers */}
Live tool-call chips stream for **Anthropic, Azure Anthropic, Google, Vertex AI, DeepSeek, Groq, AWS Bedrock** models. Other providers run tools without live chips — tool results still appear in the block output when the run completes.
| Provider | Streamed thinking | Models |
|----------|-------------------|--------|
| OpenAI | Summaries only — Requires OpenAI organization verification; falls back to no summaries. | `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5-pro`, `gpt-5.5`, `gpt-5.4-pro`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-5.2-pro`, `gpt-5.2`, `gpt-5.1`, `gpt-5-pro`, `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `o4-mini`, `o3`, `o3-mini`, `o1` |
| Anthropic | Full thinking deltas | `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-opus-4-5`, `claude-opus-4-1`, `claude-sonnet-4-5`, `claude-haiku-4-5` |
| Anthropic | Summaries only — These generations omit full thinking; Sim requests summarized thinking on streaming runs. | `claude-fable-5`, `claude-sonnet-5`, `claude-opus-4-8`, `claude-opus-4-7` |
| Azure OpenAI | Summaries only — Requires OpenAI organization verification; falls back to no summaries. | `azure/gpt-5.4`, `azure/gpt-5.4-mini`, `azure/gpt-5.4-nano`, `azure/gpt-5.2`, `azure/gpt-5.1`, `azure/gpt-5.1-codex`, `azure/gpt-5`, `azure/gpt-5-mini`, `azure/gpt-5-nano`, `azure/o3`, `azure/o4-mini` |
| Azure Anthropic | Full thinking deltas | `azure-anthropic/claude-opus-4-6`, `azure-anthropic/claude-opus-4-5`, `azure-anthropic/claude-sonnet-4-5`, `azure-anthropic/claude-opus-4-1`, `azure-anthropic/claude-haiku-4-5` |
| Google | Summaries only | `gemini-3.6-flash`, `gemini-3.5-flash-lite`, `gemini-3.5-flash`, `gemini-3.1-pro-preview`, `gemini-3.1-flash-lite`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite` |
| Vertex AI | Summaries only | `vertex/gemini-3.5-flash`, `vertex/gemini-3.1-pro-preview`, `vertex/gemini-3.1-flash-lite`, `vertex/gemini-3-flash-preview`, `vertex/gemini-2.5-pro`, `vertex/gemini-2.5-flash`, `vertex/gemini-2.5-flash-lite` |
| DeepSeek | Full thinking deltas | `deepseek-v4-pro`, `deepseek-v4-flash`, `deepseek-chat`, `deepseek-reasoner` |
| Groq | Full thinking deltas | `groq/openai/gpt-oss-120b`, `groq/openai/gpt-oss-20b`, `groq/openai/gpt-oss-safeguard-20b`, `groq/qwen/qwen3.6-27b` |
| Meta | Full thinking deltas | `muse-spark-1.1` |
| Kimi | Full thinking deltas | `kimi-k2.6` |
| Z.ai | Full thinking deltas | `glm-5.2`, `glm-5.1`, `glm-5`, `glm-5-turbo`, `glm-4.7`, `glm-4.6`, `glm-4.5`, `glm-4.5-air` |
{/* agent-stream-capabilities:end */}
## Example
A workflow that reads an incoming customer message and classifies it:
apps/docs/content/docs/en/workflows/deployment/agent-events.mdx
+100
View File
@@ -0,0 +1,100 @@
---
title: Agent stream events
description: Protocol for streaming thinking and tool lifecycle from Agent blocks
---
import { Callout } from 'fumadocs-ui/components/callout'
import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
Agent blocks can emit more than answer text while they run: **provider-exposed thinking** (or reasoning summaries) and a **tool-call lifecycle** (name + status). Sim delivers those as typed events on an opt-in stream protocol.
<Callout type="info">
Sim does **not** invent thinking for providers that do not stream it. Bedrock Converse, many OpenAI-compat models, and non-reasoning chat models stay text-only (plus tools when a live tool loop is wired).
</Callout>
## Dual gate (public chat / simple SSE)
Thinking and tool frames leave the public chat or workflow simple-SSE path only when **both** are true:
1. Deployment policy: chat `includeThinking` is enabled (default **off**).
2. Request opts in with header:
```http
X-Sim-Stream-Protocol: agent-events-v1
```
Legacy clients that omit the header keep today’s text-only SSE even if the deployment has thinking enabled. The hosted chat UI always sends the header for its own deployments.
## Simple SSE frame shapes
Answer text stays on `chunk`. Thinking and tools never reuse `chunk` (so older clients that append every `chunk` into the answer cannot leak thinking).
| Frame | Meaning |
|-------|---------|
| `{ "blockId", "chunk": "…" }` | Answer text. Legacy clients receive settled final-turn text; opted-in clients receive it live as the model generates (see below) |
| `{ "blockId", "event": "chunk_reset" }` | Opted-in only: discard the block’s streamed answer text — it belonged to a turn that resolved to tool calls |
| `{ "blockId", "event": "thinking", "data": "…" }` | Thinking / reasoning summary delta |
| `{ "blockId", "event": "tool", "phase": "start"\|"end", "id", "name", "status?" }` | Tool lifecycle (no args / results) |
| `{ "event": "final", "data": … }` | Terminal result envelope for a settled execution. `data.success` may be `false` with `data.error` when the workflow itself failed |
| `{ "event": "error", "error": "…" }` | Terminal stream failure (timeout, client abort, processing error) — followed by `[DONE]`, never by `final` |
| `{ "event": "stream_error", "blockId?", "error" }` | Non-terminal mid-block read issue; the stream keeps going |
| `data: "[DONE]"` | Stream closed (JSON-encoded sentinel; always follows the terminal `final` or `error` frame) |
### Live answer text and intermediate turns
During a live tool loop, the model can’t be classified mid-turn: text it emits may turn out to be the final answer or preamble before a tool call (the stop reason arrives only at turn end).
- **Opted-in clients** (protocol header + `includeThinking`) receive answer text as `chunk` frames **live**, token by token. If the turn then resolves to tool calls, a `chunk_reset` frame tells the client to discard that block’s streamed text — the final turn re-streams live after tools settle. Append `chunk`, honor `chunk_reset`, and the displayed answer always converges to the block’s final content.
- **Legacy clients** (no header) never see provisional text: only settled final-turn text is emitted as `chunk`, delivered when the turn completes.
Logs, memory, and the block’s `content` output always contain final-turn text only — intermediate preamble is never persisted.
### Abort
Client disconnect or Stop aborts the provider stream. In-flight tools settle as `cancelled`. Cancel is distinct from execution timeout.
### Reconnect
Canvas execution-events `stream:chunk`, `stream:chunk_reset`, `stream:thinking`, and `stream:tool` are **live-only** (not buffered for reconnect replay), same as answer chunks. Guaranteed `seq` replay is out of scope.
## Canvas (draft Run)
When you click **Run** in the builder, the execution-events SSE path forwards the same sink. The canvas is always opted in — it does not send (or need) the `X-Sim-Stream-Protocol` header; the dual gate applies only to the public chat / simple SSE surface:
- `stream:thinking` — `{ blockId, text }`
- `stream:tool` — `{ blockId, phase, id, name, status? }`
- `stream:chunk` — answer text, live on agent-events runs
- `stream:chunk_reset` — `{ blockId }`; discard the block’s streamed text (intermediate turn)
The terminal output panel shows Thinking / Tools chrome above the block output when those events arrive. PII redaction on block output still disables live forwarding (executor rule).
## Chat deployment toggle
In **Deploy → Chat**, enable **Include thinking** so public chat can expose thinking/tool frames (still requires the protocol header). Redeploy / update the chat after changing Agent models or tools.
## Capability honesty (high level)
Per-model support is generated from the model registry on the [Agent block page](/docs/workflows/blocks/agent#streamed-thinking-and-tool-calls); the table below summarizes by provider family.
| Family | Thinking | Live tools |
|--------|----------|------------|
| Anthropic / Azure Anthropic | Yes (incl. redacted blocks in traces). The newest Claude generations omit full thinking; Sim requests summarized thinking for them on streaming runs | Yes |
| Gemini / Vertex | Yes when a thinking level is set (thought summaries requested on agent-events runs) | Yes |
| OpenAI Responses | Reasoning **summaries** when streamed (requires OpenAI organization verification; unverified orgs fall back to no summaries) | Silent tool loop today (answer streams; chips may be absent) |
| OpenAI-compat (Groq, DeepSeek, …) | Only if vendor streams `reasoning` / `reasoning_content` | Live loop where wired (e.g. Groq, DeepSeek) |
| Bedrock | Not invented | Yes when streaming tool loop is used |
## Example (public chat)
<Tabs items={['cURL']}>
<Tab value="cURL">
```bash
curl -N -X POST 'http://localhost:3000/api/chat/your-slug' \
-H 'Content-Type: application/json' \
-H 'X-Sim-Stream-Protocol: agent-events-v1' \
-d '{"input":"Think briefly, then say hi"}'
```
</Tab>
</Tabs>
See also [Chat deployment](/docs/workflows/deployment/chat) for access control and the Include thinking setting.
apps/docs/content/docs/en/workflows/deployment/chat.mdx
+1
View File
@@ -32,6 +32,7 @@ Configure the following fields, then click **Launch Chat**:
| **Output** | Output fields from your workflow blocks returned as the chat response. At least one must be selected. |
| **Welcome Message** | Greeting shown before the user sends their first message. Defaults to `"Hi there! How can I help you today?"`. |
| **Access Control** | Controls who can access the chat. See [Access Control](#access-control) below. |
| **Include thinking** | When enabled, the hosted chat can show provider-exposed thinking and tool lifecycle (name + status). Requires the chat client to send `X-Sim-Stream-Protocol: agent-events-v1` (the hosted UI always does). Default is off. See [Agent stream events](/docs/workflows/deployment/agent-events). |
### Output Selection
apps/docs/content/docs/en/workflows/deployment/meta.json
+1 -1
View File
@@ -1,4 +1,4 @@
{
"title": "Deployment",
"pages": ["index", "api", "chat", "mcp"]
"pages": ["index", "api", "chat", "agent-events", "mcp"]
}
apps/docs/package.json
+1 -1
View File
@@ -41,7 +41,7 @@
"tailwind-merge": "^3.0.2",
"reactflow": "^11.11.4",
"framer-motion": "^12.5.0",
"zod": "^4.3.6"
"zod": "4.3.6"
},
"devDependencies": {
"@sim/tsconfig": "workspace:*",
apps/realtime/src/middleware/auth.ts
+1 -1
View File
@@ -69,7 +69,7 @@ export async function authenticateSocket(socket: AuthenticatedSocket, next: (err
// Store user info in socket for later use
socket.userId = session.user.id
socket.userName = session.user.name || session.user.email || 'Unknown User'
socket.userEmail = session.user.email
socket.userEmail = session.user.email ?? undefined
socket.userImage = session.user.image || null
socket.activeOrganizationId = session.session.activeOrganizationId || undefined
apps/sim/app/(interfaces)/chat/[identifier]/chat.tsx
+43 -29
View File
@@ -4,6 +4,10 @@ import { type RefObject, useCallback, useEffect, useMemo, useRef, useState } fro
import { createLogger } from '@sim/logger'
import { generateId } from '@sim/utils/id'
import { noop } from '@/lib/core/utils/request'
import {
AGENT_STREAM_PROTOCOL_HEADER,
AGENT_STREAM_PROTOCOL_V1,
} from '@/lib/workflows/streaming/agent-stream-protocol'
import {
ChatErrorState,
ChatHeader,
@@ -95,8 +99,9 @@ export default function ChatClient({ identifier }: { identifier: string }) {
const [conversationId] = useState(() => generateId())
const [showScrollButton, setShowScrollButton] = useState(false)
const [userHasScrolled, setUserHasScrolled] = useState(false)
const isUserScrollingRef = useRef(false)
/** ChatGPT-style: follow new tokens only while the viewport is near the bottom. */
const stickToBottomRef = useRef(true)
const ignoreScrollRef = useRef(false)
const [isVoiceFirstMode, setIsVoiceFirstMode] = useState(false)
@@ -131,10 +136,31 @@ export default function ChatClient({ identifier }: { identifier: string }) {
const audioContextRef = useRef<AudioContext | null>(null)
const { isPlayingAudio, streamTextToAudio, stopAudio } = useAudioStreaming(audioContextRef)
const scrollToBottom = useCallback(() => {
if (messagesEndRef.current) {
messagesEndRef.current.scrollIntoView({ behavior: 'smooth' })
const NEAR_BOTTOM_THRESHOLD_PX = 100
/**
* ChatGPT-style scroll. Without `force`, no-ops when the user has scrolled away.
* With `force` (jump button), re-pins to bottom.
*/
const scrollToBottom = useCallback((options?: { behavior?: ScrollBehavior; force?: boolean }) => {
const behavior = options?.behavior ?? 'smooth'
const force = options?.force === true
if (!force && !stickToBottomRef.current) return
if (!messagesEndRef.current) return
if (force) {
stickToBottomRef.current = true
setShowScrollButton(false)
}
ignoreScrollRef.current = true
messagesEndRef.current.scrollIntoView({ behavior })
window.setTimeout(
() => {
ignoreScrollRef.current = false
},
behavior === 'smooth' ? 400 : 50
)
}, [])
const scrollToMessage = useCallback(
@@ -165,39 +191,23 @@ export default function ChatClient({ identifier }: { identifier: string }) {
[messagesContainerRef]
)
const isStreamingResponseRef = useRef(isStreamingResponse)
isStreamingResponseRef.current = isStreamingResponse
useEffect(() => {
const container = messagesContainerRef.current
if (!container) return
const handleScroll = () => {
if (ignoreScrollRef.current) return
const { scrollTop, scrollHeight, clientHeight } = container
const distanceFromBottom = scrollHeight - scrollTop - clientHeight
setShowScrollButton(distanceFromBottom > 100)
if (isStreamingResponseRef.current && !isUserScrollingRef.current) {
setUserHasScrolled(true)
}
const nearBottom = distanceFromBottom <= NEAR_BOTTOM_THRESHOLD_PX
stickToBottomRef.current = nearBottom
setShowScrollButton(!nearBottom)
}
container.addEventListener('scroll', handleScroll, { passive: true })
return () => container.removeEventListener('scroll', handleScroll)
}, [chatConfig, isVoiceFirstMode, authRequired])
useEffect(() => {
if (isStreamingResponse) {
setUserHasScrolled(false)
isUserScrollingRef.current = true
const timeoutId = setTimeout(() => {
isUserScrollingRef.current = false
}, 1000)
return () => clearTimeout(timeoutId)
}
}, [isStreamingResponse])
const handleSendMessage = async (
messageParam?: string,
isVoiceInput = false,
@@ -220,7 +230,8 @@ export default function ChatClient({ identifier }: { identifier: string }) {
filesCount: files?.length,
})
setUserHasScrolled(false)
stickToBottomRef.current = true
setShowScrollButton(false)
const userMessage: ChatMessage = {
id: generateId(),
@@ -244,7 +255,9 @@ export default function ChatClient({ identifier }: { identifier: string }) {
scrollToMessage(userMessage.id, true)
}, 100)
// One AbortController for fetch + SSE body reads so Stop cancels server work too.
const abortController = new AbortController()
abortControllerRef.current = abortController
const timeoutId = setTimeout(() => {
abortController.abort()
}, CHAT_REQUEST_TIMEOUT_MS)
@@ -282,6 +295,7 @@ export default function ChatClient({ identifier }: { identifier: string }) {
headers: {
'Content-Type': 'application/json',
'X-Requested-With': 'XMLHttpRequest',
[AGENT_STREAM_PROTOCOL_HEADER]: AGENT_STREAM_PROTOCOL_V1,
},
body: JSON.stringify(payload),
credentials: 'same-origin',
@@ -316,8 +330,7 @@ export default function ChatClient({ identifier }: { identifier: string }) {
response,
setMessages,
setIsLoading,
scrollToBottom,
userHasScrolled,
() => scrollToBottom({ behavior: 'auto' }),
{
voiceSettings: {
isVoiceEnabled: shouldPlayAudio,
@@ -326,6 +339,7 @@ export default function ChatClient({ identifier }: { identifier: string }) {
},
audioStreamHandler: audioHandler,
outputConfigs: chatConfig?.outputConfigs,
abortController,
}
)
} catch (error) {
@@ -437,7 +451,7 @@ export default function ChatClient({ identifier }: { identifier: string }) {
showScrollButton={showScrollButton}
messagesContainerRef={messagesContainerRef as RefObject<HTMLDivElement>}
messagesEndRef={messagesEndRef as RefObject<HTMLDivElement>}
scrollToBottom={scrollToBottom}
scrollToBottom={() => scrollToBottom({ behavior: 'smooth', force: true })}
scrollToMessage={scrollToMessage}
chatConfig={chatConfig}
/>
apps/sim/app/(interfaces)/chat/components/message/message.test.tsx
+162 -5
View File
@@ -1,11 +1,24 @@
/**
* @vitest-environment node
* @vitest-environment jsdom
*/
import { describe, expect, it, vi } from 'vitest'
import { act } from 'react'
import { createRoot, type Root } from 'react-dom/client'
import { afterEach, describe, expect, it, vi } from 'vitest'
vi.mock('@sim/emcn', () => ({
Button: ({ children, ...props }: { children?: React.ReactNode; [key: string]: unknown }) => (
<button type='button' {...props}>
{children}
</button>
),
Duplicate: () => null,
Tooltip: {},
Tooltip: {
Provider: ({ children }: { children?: React.ReactNode }) => <>{children}</>,
Root: ({ children }: { children?: React.ReactNode }) => <>{children}</>,
Trigger: ({ children }: { children?: React.ReactNode }) => <>{children}</>,
Content: ({ children }: { children?: React.ReactNode }) => <>{children}</>,
},
cn: (...args: unknown[]) => args.filter(Boolean).join(' '),
}))
vi.mock('@/app/(interfaces)/chat/components/message/components/file-download', () => ({
@@ -14,10 +27,14 @@ vi.mock('@/app/(interfaces)/chat/components/message/components/file-download', (
}))
vi.mock('@/app/(interfaces)/chat/components/message/components/markdown-renderer', () => ({
default: () => null,
default: ({ content }: { content: string }) => <div data-testid='answer'>{content}</div>,
}))
import { escapeHtml } from '@/app/(interfaces)/chat/components/message/message'
import {
type ChatMessage,
ClientChatMessage,
escapeHtml,
} from '@/app/(interfaces)/chat/components/message/message'
describe('escapeHtml', () => {
it('escapes all five HTML-significant characters', () => {
@@ -41,3 +58,143 @@ describe('escapeHtml', () => {
expect(escapeHtml('')).toBe('')
})
})
function renderMessage(message: ChatMessage): { container: HTMLDivElement; unmount: () => void } {
;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true
const container = document.createElement('div')
document.body.appendChild(container)
const root: Root = createRoot(container)
act(() => {
root.render(<ClientChatMessage message={message} />)
})
return {
container,
unmount: () => {
act(() => {
root.unmount()
})
container.remove()
},
}
}
describe('ClientChatMessage thinking chrome (Step 6)', () => {
const mounts: Array<() => void> = []
afterEach(() => {
while (mounts.length) {
mounts.pop()?.()
}
})
it('does not show thinking chrome when thinking is absent or empty', () => {
const without = renderMessage({
id: '1',
type: 'assistant',
content: 'Hello',
timestamp: new Date(),
})
mounts.push(without.unmount)
expect(without.container.textContent).not.toContain('Thinking')
const empty = renderMessage({
id: '2',
type: 'assistant',
content: 'Hello',
thinking: '',
timestamp: new Date(),
})
mounts.push(empty.unmount)
expect(empty.container.textContent).not.toContain('Thinking')
})
it('shows collapsible thinking chrome above the answer after first thinking', () => {
const { container, unmount } = renderMessage({
id: '3',
type: 'assistant',
content: 'Answer text',
thinking: 'Internal reasoning',
isThinkingStreaming: true,
timestamp: new Date(),
})
mounts.push(unmount)
expect(container.textContent).toContain('Thinking…')
expect(container.textContent).toContain('Internal reasoning')
expect(container.textContent).toContain('Answer text')
})
it('labels completed thinking as Thought for a moment', () => {
const { container, unmount } = renderMessage({
id: '4',
type: 'assistant',
content: 'Answer text',
thinking: 'Internal reasoning',
isThinkingStreaming: false,
timestamp: new Date(),
})
mounts.push(unmount)
expect(container.textContent).toContain('Thought for a moment')
expect(container.textContent).toContain('Answer text')
})
})
describe('ClientChatMessage tool chrome (Step 8)', () => {
const mounts: Array<() => void> = []
afterEach(() => {
while (mounts.length) {
mounts.pop()?.()
}
})
it('does not show tool chrome when toolCalls are absent or empty', () => {
const without = renderMessage({
id: '1',
type: 'assistant',
content: 'Hello',
timestamp: new Date(),
})
mounts.push(without.unmount)
expect(without.container.textContent).not.toContain('Tools')
expect(without.container.textContent).not.toContain('Using tools')
const empty = renderMessage({
id: '2',
type: 'assistant',
content: 'Hello',
toolCalls: [],
timestamp: new Date(),
})
mounts.push(empty.unmount)
expect(empty.container.textContent).not.toContain('Tools')
})
it('shows humanized tool names only (no args) while tools are running', () => {
const { container, unmount } = renderMessage({
id: '3',
type: 'assistant',
content: 'Answer',
isToolStreaming: true,
toolCalls: [
{
key: 'agent-1:toolu_1',
blockId: 'agent-1',
id: 'toolu_1',
name: 'http_request',
displayName: 'Http Request',
status: 'running',
},
],
timestamp: new Date(),
})
mounts.push(unmount)
expect(container.textContent).toContain('Using tools…')
expect(container.textContent).toContain('Http Request')
expect(container.textContent).not.toContain('toolu_1')
expect(container.textContent).not.toContain('args')
expect(container.textContent).toContain('Answer')
})
})
apps/sim/app/(interfaces)/chat/components/message/message.tsx
+50 -3
View File
@@ -3,6 +3,14 @@
import { memo, useState } from 'react'
import { Button, cn, Duplicate, Tooltip } from '@sim/emcn'
import { Check, File as FileIcon, FileText, Image as ImageIcon } from 'lucide-react'
import {
AgentStreamThinkingChrome,
AgentStreamToolCallsChrome,
} from '@/components/agent-stream/agent-stream-chrome'
import type {
AgentStreamToolCall,
AgentStreamToolStatus,
} from '@/components/agent-stream/tool-call-lifecycle'
import {
ChatFileDownload,
ChatFileDownloadAll,
@@ -27,6 +35,15 @@ export interface ChatFile {
context?: string
}
/** Lifecycle status for a tool chip (agent-events-v1). No args/results. */
export type ChatToolCallStatus = AgentStreamToolStatus
/** Chat surface tool chip — the shared lifecycle chip plus its block id. */
export interface ChatToolCall extends AgentStreamToolCall {
blockId: string
displayName: string
}
export interface ChatMessage {
id: string
content: string | Record<string, unknown>
@@ -34,6 +51,14 @@ export interface ChatMessage {
timestamp: Date
isInitialMessage?: boolean
isStreaming?: boolean
/** Model thinking text (agent-events-v1). Chrome only when non-empty. */
thinking?: string
/** True while thinking deltas are still arriving (before first answer chunk / final). */
isThinkingStreaming?: boolean
/** Tool lifecycle chips (name + status only). Chrome only when non-empty. */
toolCalls?: ChatToolCall[]
/** True while any tool chip is still `running`. */
isToolStreaming?: boolean
attachments?: ChatAttachment[]
files?: ChatFile[]
}
@@ -81,15 +106,21 @@ function openAttachmentPreview(name: string, dataUrl: string): void {
setTimeout(() => URL.revokeObjectURL(blobUrl), 60_000)
}
function toolCallsFingerprint(toolCalls: ChatToolCall[] | undefined): string {
if (!toolCalls?.length) return ''
return toolCalls.map((t) => `${t.key}:${t.status}`).join('|')
}
export const ClientChatMessage = memo(
function ClientChatMessage({ message }: { message: ChatMessage }) {
const [isCopied, setIsCopied] = useState(false)
const isJsonObject = typeof message.content === 'object' && message.content !== null
// Since tool calls are now handled via SSE events and stored in message.toolCalls,
// we can use the content directly without parsing
// Answer text is streamed separately from thinking / tool lifecycle events.
const cleanTextContent = message.content
const hasThinking = typeof message.thinking === 'string' && message.thinking.length > 0
const hasToolCalls = Array.isArray(message.toolCalls) && message.toolCalls.length > 0
const content =
message.type === 'user' ? (
@@ -207,8 +238,19 @@ export const ClientChatMessage = memo(
<div className='px-4 pt-5 pb-2' data-message-id={message.id}>
<div className='mx-auto max-w-3xl'>
<div className='flex flex-col space-y-3'>
{/* Direct content rendering - tool calls are now handled via SSE events */}
<div>
{hasThinking && (
<AgentStreamThinkingChrome
thinking={message.thinking!}
isStreaming={message.isThinkingStreaming}
/>
)}
{hasToolCalls && (
<AgentStreamToolCallsChrome
toolCalls={message.toolCalls!}
isStreaming={message.isToolStreaming}
/>
)}
<div className='break-words text-base'>
{isJsonObject ? (
<pre className='text-[var(--text-primary)]'>
@@ -274,7 +316,12 @@ export const ClientChatMessage = memo(
return (
prevProps.message.id === nextProps.message.id &&
prevProps.message.content === nextProps.message.content &&
prevProps.message.thinking === nextProps.message.thinking &&
prevProps.message.isStreaming === nextProps.message.isStreaming &&
prevProps.message.isThinkingStreaming === nextProps.message.isThinkingStreaming &&
prevProps.message.isToolStreaming === nextProps.message.isToolStreaming &&
toolCallsFingerprint(prevProps.message.toolCalls) ===
toolCallsFingerprint(nextProps.message.toolCalls) &&
prevProps.message.isInitialMessage === nextProps.message.isInitialMessage &&
prevProps.message.files?.length === nextProps.message.files?.length
)
apps/sim/app/(interfaces)/chat/hooks/use-chat-streaming.test.tsx
+691
View File
@@ -0,0 +1,691 @@
/**
* @vitest-environment jsdom
*/
import { act } from 'react'
import { createRoot, type Root } from 'react-dom/client'
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
const { mockReadSSEEvents } = vi.hoisted(() => ({
mockReadSSEEvents: vi.fn(),
}))
vi.mock('@/lib/core/utils/sse', () => ({
readSSEEvents: mockReadSSEEvents,
}))
vi.mock('@sim/utils/id', () => ({
generateId: () => 'msg-assistant-1',
}))
import { isChatChunkFrame } from '@/lib/workflows/streaming/agent-stream-protocol'
import type { ChatMessage } from '@/app/(interfaces)/chat/components/message/message'
import { useChatStreaming } from '@/app/(interfaces)/chat/hooks/use-chat-streaming'
describe('isChatChunkFrame', () => {
it('accepts plain answer chunks without an event type', () => {
expect(isChatChunkFrame({ blockId: 'a1', chunk: 'hello' })).toBe(true)
})
it('rejects thinking / stream_error / tool frames even if chunk is present', () => {
expect(
isChatChunkFrame({ blockId: 'a1', chunk: 'leak', event: 'thinking', data: 'thought' })
).toBe(false)
expect(isChatChunkFrame({ blockId: 'a1', chunk: 'x', event: 'stream_error' })).toBe(false)
expect(isChatChunkFrame({ blockId: 'a1', chunk: 'x', event: 'tool' })).toBe(false)
expect(isChatChunkFrame({ blockId: 'a1', chunk: 'x', event: 'final' })).toBe(false)
})
it('rejects frames missing blockId or empty chunk', () => {
expect(isChatChunkFrame({ chunk: 'hello' })).toBe(false)
expect(isChatChunkFrame({ blockId: 'a1', chunk: '' })).toBe(false)
})
})
interface HookHandle {
latest: () => ReturnType<typeof useChatStreaming>
unmount: () => void
}
function renderStreamingHook(): HookHandle {
;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true
const container = document.createElement('div')
const root: Root = createRoot(container)
let latest!: ReturnType<typeof useChatStreaming>
function Probe() {
latest = useChatStreaming()
return null
}
act(() => {
root.render(<Probe />)
})
return {
latest: () => latest,
unmount: () => {
act(() => {
root.unmount()
})
},
}
}
function makeSseResponse(): Response {
return {
body: new ReadableStream(),
} as Response
}
async function flushUiBatch() {
await act(async () => {
await new Promise<void>((resolve) => {
requestAnimationFrame(() => resolve())
})
await new Promise<void>((resolve) => {
setTimeout(resolve, 60)
})
})
}
describe('useChatStreaming thinking + abort', () => {
let handle: HookHandle
let messages: ChatMessage[]
let setMessages: React.Dispatch<React.SetStateAction<ChatMessage[]>>
beforeEach(() => {
vi.clearAllMocks()
messages = []
setMessages = ((updater: React.SetStateAction<ChatMessage[]>) => {
messages = typeof updater === 'function' ? updater(messages) : updater
}) as React.Dispatch<React.SetStateAction<ChatMessage[]>>
handle = renderStreamingHook()
// Run rAF immediately so UI batching is deterministic in tests.
vi.spyOn(window, 'requestAnimationFrame').mockImplementation((cb) => {
cb(performance.now())
return 1
})
})
afterEach(() => {
handle.unmount()
vi.restoreAllMocks()
})
it('routes thinking to message.thinking and answer chunks to content only', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
data: 'Let me reason. ',
})
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
data: 'More thought.',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'Final answer.',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.thinking).toBe('Let me reason. More thought.')
expect(assistant?.content).toBe('Final answer.')
expect(assistant?.isStreaming).toBe(false)
expect(assistant?.isThinkingStreaming).toBe(false)
})
it('clears a block’s live text on chunk_reset and keeps the re-streamed final turn', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
// Turn 1: live preamble, then tools follow → reset.
await options.onEvent({ blockId: 'agent-1', chunk: 'Let me check the weather…' })
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 't1',
name: 'get_weather',
})
await options.onEvent({ blockId: 'agent-1', event: 'chunk_reset' })
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'end',
id: 't1',
name: 'get_weather',
status: 'success',
})
// Turn 2: final answer streams live.
await options.onEvent({ blockId: 'agent-1', chunk: 'It is ' })
await options.onEvent({ blockId: 'agent-1', chunk: '68°F.' })
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.content).toBe('It is 68°F.')
expect(assistant?.content).not.toContain('Let me check')
})
it('re-registers a reset block at the end so multi-block order matches arrival', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
// Agent A streams provisional text, then resets (tools follow).
await options.onEvent({ blockId: 'agent-a', chunk: 'Checking the weather…' })
await options.onEvent({ blockId: 'agent-a', event: 'chunk_reset' })
// Another block streams while A's tools run.
await options.onEvent({ blockId: 'block-b', chunk: 'B output' })
// A's final turn re-streams; the server bakes in the cross-block separator.
await options.onEvent({ blockId: 'agent-a', chunk: '\n\nIt is 68°F.' })
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.content).toBe('B output\n\nIt is 68°F.')
})
it('settles thinking chrome when a tool starts', async () => {
let midStreamThinking: boolean | undefined
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({ blockId: 'agent-1', event: 'thinking', data: 'planning…' })
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 't1',
name: 'get_weather',
})
// UI flush is synchronous in tests (rAF mocked) — capture mid-stream state.
midStreamThinking = messages.find((m) => m.type === 'assistant')?.isThinkingStreaming
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'end',
id: 't1',
name: 'get_weather',
status: 'success',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
expect(midStreamThinking).toBe(false)
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.thinking).toBe('planning…')
})
it('ignores non-terminal stream_error frames and keeps streaming', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
data: 'Working…',
})
await options.onEvent({
blockId: 'agent-1',
event: 'stream_error',
error: 'partial provider glitch',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'Recovered answer.',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
// Error text never pollutes the thinking lane (legacy parity: log-only).
expect(assistant?.thinking).toBe('Working…')
expect(assistant?.content).toBe('Recovered answer.')
expect(assistant?.isStreaming).toBe(false)
})
it('clears streaming flags when SSE ends without a terminal final/error frame', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
data: 'Halfway…',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'Partial answer',
})
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 't1',
name: 'search',
})
// Stream closes abruptly — no final or error event.
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.content).toBe('Partial answer')
expect(assistant?.thinking).toBe('Halfway…')
expect(assistant?.isStreaming).toBe(false)
expect(assistant?.isThinkingStreaming).toBe(false)
expect(assistant?.isToolStreaming).toBe(false)
expect(assistant?.toolCalls?.some((t) => t.status === 'error')).toBe(true)
})
it('does not append thinking payload into answer when mislabeled as chunk', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
chunk: 'SHOULD_NOT_APPEND',
data: 'real thought',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'ok',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.content).toBe('ok')
expect(assistant?.thinking).toBe('real thought')
})
it('TTS audioStreamHandler receives answer text only', async () => {
const audioStreamHandler = vi.fn().mockResolvedValue(undefined)
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
data: 'secret internal monologue that must not be spoken.',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'Hello world.',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle
.latest()
.handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn(), {
voiceSettings: {
isVoiceEnabled: true,
voiceId: 'voice-1',
autoPlayResponses: true,
},
audioStreamHandler,
})
})
await flushUiBatch()
expect(audioStreamHandler).toHaveBeenCalled()
for (const call of audioStreamHandler.mock.calls) {
expect(String(call[0])).not.toContain('secret')
expect(String(call[0])).not.toContain('monologue')
}
expect(audioStreamHandler.mock.calls.some((c) => String(c[0]).includes('Hello'))).toBe(true)
})
it('stopStreaming preserves thinking and aborts the shared controller', async () => {
const abortController = new AbortController()
let resolveStream!: () => void
const streamDone = new Promise<void>((resolve) => {
resolveStream = resolve
})
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'thinking',
data: 'partial thought',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'partial answer',
})
// Hold the stream open until Stop aborts.
await new Promise<void>((resolve) => {
options.signal?.addEventListener('abort', () => resolve(), { once: true })
// Also allow test cleanup if abort never fires.
streamDone.then(() => resolve())
})
})
const streamPromise = act(async () => {
await handle
.latest()
.handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn(), {
abortController,
})
})
await flushUiBatch()
expect(messages.find((m) => m.id === 'msg-assistant-1')?.thinking).toBe('partial thought')
act(() => {
handle.latest().stopStreaming(setMessages)
})
resolveStream()
await streamPromise
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(abortController.signal.aborted).toBe(true)
expect(assistant?.thinking).toBe('partial thought')
expect(String(assistant?.content)).toContain('partial answer')
expect(String(assistant?.content)).toContain('Response stopped by user')
expect(assistant?.isStreaming).toBe(false)
expect(assistant?.isThinkingStreaming).toBe(false)
})
it('does not replace Stop notice with server Client cancelled request error', async () => {
const abortController = new AbortController()
let resolveStream!: () => void
const streamDone = new Promise<void>((resolve) => {
resolveStream = resolve
})
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
chunk: 'partial answer',
})
await new Promise<void>((resolve) => {
options.signal?.addEventListener(
'abort',
() => {
// Server still emits terminal cancel error while the reader finishes.
void options.onEvent({
event: 'error',
error: 'Client cancelled request',
})
resolve()
},
{ once: true }
)
streamDone.then(() => resolve())
})
})
const streamPromise = act(async () => {
await handle
.latest()
.handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn(), {
abortController,
})
})
await flushUiBatch()
act(() => {
handle.latest().stopStreaming(setMessages)
})
resolveStream()
await streamPromise
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(String(assistant?.content)).toContain('partial answer')
expect(String(assistant?.content)).toContain('Response stopped by user')
expect(String(assistant?.content)).not.toContain('Client cancelled request')
expect(assistant?.isStreaming).toBe(false)
})
it('leaves thinking undefined when no thinking events arrive', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
chunk: 'just text',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.thinking).toBeUndefined()
expect(assistant?.content).toBe('just text')
})
})
describe('useChatStreaming tool lifecycle', () => {
let handle: HookHandle
let messages: ChatMessage[]
let setMessages: React.Dispatch<React.SetStateAction<ChatMessage[]>>
beforeEach(() => {
vi.clearAllMocks()
messages = []
setMessages = ((updater: React.SetStateAction<ChatMessage[]>) => {
messages = typeof updater === 'function' ? updater(messages) : updater
}) as React.Dispatch<React.SetStateAction<ChatMessage[]>>
handle = renderStreamingHook()
vi.spyOn(window, 'requestAnimationFrame').mockImplementation((cb) => {
cb(performance.now())
return 1
})
})
afterEach(() => {
handle.unmount()
vi.restoreAllMocks()
})
it('maps tool start/end into keyed chips without touching answer content', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 'toolu_1',
name: 'http_request',
})
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'end',
id: 'toolu_1',
name: 'http_request',
status: 'success',
})
await options.onEvent({
blockId: 'agent-1',
chunk: 'https://httpbin.org/get',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
const assistant = messages.find((m) => m.id === 'msg-assistant-1')
expect(assistant?.content).toBe('https://httpbin.org/get')
expect(assistant?.toolCalls).toEqual([
{
key: 'agent-1:toolu_1',
blockId: 'agent-1',
id: 'toolu_1',
name: 'http_request',
displayName: 'Http Request',
status: 'success',
},
])
expect(assistant?.toolCalls?.[0]).not.toHaveProperty('args')
expect(assistant?.toolCalls?.[0]).not.toHaveProperty('result')
expect(assistant?.isToolStreaming).toBe(false)
})
it('tracks parallel tools and cancels running chips on Stop', async () => {
const abortController = new AbortController()
let resolveStream!: () => void
const streamDone = new Promise<void>((resolve) => {
resolveStream = resolve
})
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 'toolu_1',
name: 'http_request',
})
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 'toolu_2',
name: 'function_execute',
})
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'end',
id: 'toolu_2',
name: 'function_execute',
status: 'success',
})
await new Promise<void>((resolve) => {
options.signal?.addEventListener('abort', () => resolve(), { once: true })
streamDone.then(() => resolve())
})
})
const streamPromise = act(async () => {
await handle
.latest()
.handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn(), {
abortController,
})
})
await flushUiBatch()
expect(messages.find((m) => m.id === 'msg-assistant-1')?.toolCalls).toHaveLength(2)
act(() => {
handle.latest().stopStreaming(setMessages)
})
resolveStream()
await streamPromise
const tools = messages.find((m) => m.id === 'msg-assistant-1')?.toolCalls
expect(tools?.find((t) => t.id === 'toolu_1')?.status).toBe('cancelled')
expect(tools?.find((t) => t.id === 'toolu_2')?.status).toBe('success')
expect(messages.find((m) => m.id === 'msg-assistant-1')?.isToolStreaming).toBe(false)
})
it('settles straggler running tools to success on final', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 'toolu_open',
name: 'http_request',
})
await options.onEvent({
event: 'final',
data: { success: true, output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
expect(messages.find((m) => m.id === 'msg-assistant-1')?.toolCalls?.[0]?.status).toBe('success')
})
it('settles straggler running tools to error when final reports failure', async () => {
mockReadSSEEvents.mockImplementation(async (_source, options) => {
await options.onEvent({
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 'toolu_open',
name: 'http_request',
})
// Failed runs can still terminate with `final` carrying success: false.
await options.onEvent({
event: 'final',
data: { success: false, error: 'Workflow failed', output: {} },
})
})
await act(async () => {
await handle.latest().handleStreamedResponse(makeSseResponse(), setMessages, vi.fn(), vi.fn())
})
await flushUiBatch()
expect(messages.find((m) => m.id === 'msg-assistant-1')?.toolCalls?.[0]?.status).toBe('error')
})
})
apps/sim/app/(interfaces)/chat/hooks/use-chat-streaming.ts
+300 -40
View File
@@ -3,9 +3,29 @@
import { useRef, useState } from 'react'
import { createLogger } from '@sim/logger'
import { generateId } from '@sim/utils/id'
import {
anyToolCallRunning,
applyToolCallPhase,
settleRunningToolCalls,
snapshotToolCalls,
toolCallKey,
} from '@/components/agent-stream/tool-call-lifecycle'
import { readSSEEvents } from '@/lib/core/utils/sse'
import { isUserFileWithMetadata } from '@/lib/core/utils/user-file'
import type { ChatFile, ChatMessage } from '@/app/(interfaces)/chat/components/message/message'
import {
isChatChunkFrame,
isChatChunkResetFrame,
isChatErrorFrame,
isChatFinalFrame,
isChatStreamErrorFrame,
isChatThinkingFrame,
isChatToolFrame,
} from '@/lib/workflows/streaming/agent-stream-protocol'
import type {
ChatFile,
ChatMessage,
ChatToolCall,
} from '@/app/(interfaces)/chat/components/message/message'
import { CHAT_ERROR_MESSAGES } from '@/app/(interfaces)/chat/constants'
const logger = createLogger('UseChatStreaming')
@@ -64,23 +84,40 @@ export interface StreamingOptions {
onAudioEnd?: () => void
audioStreamHandler?: (text: string) => Promise<void>
outputConfigs?: Array<{ blockId: string; path?: string }>
/**
* Shared AbortController for fetch + SSE body reads. When provided (preferred),
* Stop aborts the in-flight request server-side as well as the reader.
*/
abortController?: AbortController
}
/** Client-side view of the `final` frame's opaque `data` payload. */
interface ChatFinalData {
success?: boolean
error?: string | { message?: string }
output?: Record<string, Record<string, any>>
}
export function useChatStreaming() {
const [isStreamingResponse, setIsStreamingResponse] = useState(false)
const abortControllerRef = useRef<AbortController | null>(null)
const accumulatedTextRef = useRef<string>('')
const accumulatedThinkingRef = useRef<string>('')
const accumulatedToolCallsRef = useRef<ChatToolCall[]>([])
const lastStreamedPositionRef = useRef<number>(0)
const audioStreamingActiveRef = useRef<boolean>(false)
const lastDisplayedPositionRef = useRef<number>(0) // Track displayed text in synced mode
const lastDisplayedPositionRef = useRef<number>(0)
const stopStreaming = (setMessages: React.Dispatch<React.SetStateAction<ChatMessage[]>>) => {
if (abortControllerRef.current) {
// Abort the fetch request
abortControllerRef.current.abort()
abortControllerRef.current = null
const latestContent = accumulatedTextRef.current
const latestThinking = accumulatedThinkingRef.current
const latestTools = accumulatedToolCallsRef.current.map((tool) =>
tool.status === 'running' ? { ...tool, status: 'cancelled' as const } : tool
)
setMessages((prev) => {
const lastMessage = prev[prev.length - 1]
@@ -92,7 +129,16 @@ export function useChatStreaming() {
return [
...prev.slice(0, -1),
{ ...lastMessage, content: updatedContent, isStreaming: false },
{
...lastMessage,
content: updatedContent,
// Preserve any thinking / tools received before Stop.
thinking: latestThinking || lastMessage.thinking,
toolCalls: latestTools.length > 0 ? latestTools : lastMessage.toolCalls,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
},
]
}
@@ -101,6 +147,8 @@ export function useChatStreaming() {
setIsStreamingResponse(false)
accumulatedTextRef.current = ''
accumulatedThinkingRef.current = ''
accumulatedToolCallsRef.current = []
lastStreamedPositionRef.current = 0
lastDisplayedPositionRef.current = 0
audioStreamingActiveRef.current = false
@@ -112,15 +160,18 @@ export function useChatStreaming() {
setMessages: React.Dispatch<React.SetStateAction<ChatMessage[]>>,
setIsLoading: React.Dispatch<React.SetStateAction<boolean>>,
scrollToBottom: () => void,
userHasScrolled?: boolean,
streamingOptions?: StreamingOptions
) => {
logger.info('[useChatStreaming] handleStreamedResponse called')
// Set streaming state
setIsStreamingResponse(true)
abortControllerRef.current = new AbortController()
// Check if we should stream audio
// Prefer a shared controller from the caller (fetch + reader). Otherwise create one.
if (streamingOptions?.abortController) {
abortControllerRef.current = streamingOptions.abortController
} else if (!abortControllerRef.current) {
abortControllerRef.current = new AbortController()
}
const shouldPlayAudio =
streamingOptions?.voiceSettings?.isVoiceEnabled &&
streamingOptions?.voiceSettings?.autoPlayResponses &&
@@ -132,8 +183,28 @@ export function useChatStreaming() {
return
}
/**
* Answer text tracked per block so a `chunk_reset` (dual-gated streams:
* a live-streamed turn resolved to tool calls) can clear one block's
* contribution. `accumulatedText` is re-derived on every mutation —
* cross-block separators arrive baked into the chunks.
*/
const blockTextOrder: string[] = []
const blockTextSegments = new Map<string, string>()
let accumulatedText = ''
const recomputeAccumulatedText = () => {
accumulatedText = blockTextOrder.map((id) => blockTextSegments.get(id) ?? '').join('')
accumulatedTextRef.current = accumulatedText
}
let accumulatedThinking = ''
let isThinkingStreaming = false
let lastAudioPosition = 0
const toolCallsMap = new Map<string, ChatToolCall>()
const toolCallOrder: string[] = []
const syncToolCallsRef = () => {
accumulatedToolCallsRef.current = snapshotToolCalls(toolCallOrder, toolCallsMap) ?? []
}
const messageIdMap = new Map<string, string>()
const messageId = generateId()
@@ -156,14 +227,29 @@ export function useChatStreaming() {
if (!uiDirty) return
uiDirty = false
lastUIFlush = performance.now()
const snapshot = accumulatedText
const contentSnapshot = accumulatedText
const thinkingSnapshot = accumulatedThinking
const thinkingStreamingSnapshot = isThinkingStreaming
const toolCallsSnapshot = snapshotToolCalls(toolCallOrder, toolCallsMap)
const toolStreamingSnapshot = anyToolCallRunning(toolCallsMap)
setMessages((prev) =>
prev.map((msg) => {
if (msg.id !== messageId) return msg
if (!msg.isStreaming) return msg
return { ...msg, content: snapshot }
return {
...msg,
content: contentSnapshot,
thinking: thinkingSnapshot || undefined,
isThinkingStreaming: thinkingStreamingSnapshot,
toolCalls: toolCallsSnapshot,
isToolStreaming: toolStreamingSnapshot,
}
})
)
// Caller supplies a stick-to-bottom-aware scroller (no-ops if user scrolled away).
requestAnimationFrame(() => {
scrollToBottom()
})
}
const scheduleUIFlush = () => {
@@ -192,35 +278,59 @@ export function useChatStreaming() {
setIsLoading(false)
let terminated = false
// Capture before Stop nulls abortControllerRef; needed when the reader
// resolves on abort instead of throwing AbortError.
const streamAbortSignal = abortControllerRef.current!.signal
try {
await readSSEEvents<{
blockId?: string
chunk?: string
event?: string
error?: string
data?: {
success: boolean
error?: string | { message?: string }
output?: Record<string, Record<string, any>>
}
}>(response.body, {
signal: abortControllerRef.current.signal,
await readSSEEvents<Record<string, unknown>>(response.body, {
signal: streamAbortSignal,
onParseError: (_data, parseError) => {
logger.error('Error parsing stream data:', parseError)
},
onEvent: async (json) => {
const { blockId, chunk: contentChunk, event: eventType } = json
if (isChatErrorFrame(json)) {
// User Stop aborts the fetch; the server often still emits a terminal
// `{ event: 'error', error: 'Client cancelled request' }` before the
// SSE reader finishes. Do not overwrite the stop notice.
if (streamAbortSignal.aborted) {
settleRunningToolCalls(toolCallsMap, 'cancelled')
syncToolCallsRef()
const toolsSnapshot = snapshotToolCalls(toolCallOrder, toolCallsMap)
setMessages((prev) =>
prev.map((msg) =>
msg.id === messageId
? {
...msg,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
thinking: accumulatedThinking || msg.thinking,
toolCalls: toolsSnapshot ?? msg.toolCalls,
}
: msg
)
)
setIsLoading(false)
terminated = true
return true
}
if (eventType === 'error' || json.event === 'error') {
const errorMessage = json.error || CHAT_ERROR_MESSAGES.GENERIC_ERROR
settleRunningToolCalls(toolCallsMap, 'error')
syncToolCallsRef()
const toolsSnapshot = snapshotToolCalls(toolCallOrder, toolCallsMap)
setMessages((prev) =>
prev.map((msg) =>
msg.id === messageId
? {
...msg,
content: errorMessage,
thinking: accumulatedThinking || msg.thinking,
toolCalls: toolsSnapshot ?? msg.toolCalls,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
type: 'assistant' as const,
}
: msg
@@ -231,9 +341,70 @@ export function useChatStreaming() {
return true
}
if (eventType === 'final' && json.data) {
if (isChatStreamErrorFrame(json)) {
// Non-terminal mid-block read issue: keep streaming. The legacy
// client ignored these frames; log only — never repurpose the
// thinking lane for error text.
logger.warn('[useChatStreaming] Non-terminal stream_error', {
blockId: json.blockId,
error: json.error || 'A streaming error occurred',
})
return false
}
if (isChatThinkingFrame(json)) {
if (!messageIdMap.has(json.blockId)) {
messageIdMap.set(json.blockId, messageId)
}
accumulatedThinking += json.data
accumulatedThinkingRef.current = accumulatedThinking
isThinkingStreaming = true
uiDirty = true
scheduleUIFlush()
return false
}
if (isChatToolFrame(json)) {
const { blockId } = json
if (!messageIdMap.has(blockId)) {
messageIdMap.set(blockId, messageId)
}
// Tools starting means the turn's thinking phase is over — settle
// the thinking chrome (it re-opens if more thinking streams later).
if (json.phase === 'start' && isThinkingStreaming) {
isThinkingStreaming = false
}
applyToolCallPhase(
toolCallsMap,
toolCallOrder,
{
key: toolCallKey(blockId, json.id),
id: json.id,
name: json.name,
phase: json.phase,
status: json.status,
},
(tool): ChatToolCall => ({
...tool,
blockId,
displayName: tool.displayName ?? tool.name,
})
)
syncToolCallsRef()
uiDirty = true
scheduleUIFlush()
return false
}
if (isChatFinalFrame(json)) {
flushUI()
const finalData = json.data
const finalData = json.data as ChatFinalData
isThinkingStreaming = false
// A failed run can still terminate with `final` (success: false) —
// straggler running chips must not settle green in that case.
settleRunningToolCalls(toolCallsMap, finalData.success === false ? 'error' : 'success')
syncToolCallsRef()
const toolsSnapshot = snapshotToolCalls(toolCallOrder, toolCallsMap)
const outputConfigs = streamingOptions?.outputConfigs
const formattedOutputs: string[] = []
@@ -358,7 +529,11 @@ export function useChatStreaming() {
? {
...msg,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
content: finalContent ?? msg.content,
thinking: accumulatedThinking || msg.thinking,
toolCalls: toolsSnapshot ?? msg.toolCalls,
files: extractedFiles.length > 0 ? extractedFiles : undefined,
}
: msg
@@ -366,6 +541,8 @@ export function useChatStreaming() {
)
accumulatedTextRef.current = ''
accumulatedThinkingRef.current = ''
accumulatedToolCallsRef.current = []
lastStreamedPositionRef.current = 0
lastDisplayedPositionRef.current = 0
audioStreamingActiveRef.current = false
@@ -374,13 +551,46 @@ export function useChatStreaming() {
return true
}
if (blockId && contentChunk) {
if (isChatChunkResetFrame(json)) {
// The block's live-streamed text belonged to an intermediate turn
// (tool calls follow); drop it — the final turn re-streams after.
// Remove the block from the order too: its re-streamed text
// re-registers at the end, keeping render order = arrival order
// (the server re-computes the cross-block separator on re-stream).
const { blockId } = json
if (blockTextSegments.has(blockId)) {
blockTextSegments.delete(blockId)
const orderIndex = blockTextOrder.indexOf(blockId)
if (orderIndex !== -1) {
blockTextOrder.splice(orderIndex, 1)
}
recomputeAccumulatedText()
// Spoken audio cannot be unplayed; clamp so slicing stays valid.
lastAudioPosition = Math.min(lastAudioPosition, accumulatedText.length)
uiDirty = true
scheduleUIFlush()
}
return false
}
// Answer text only — never append thinking/tool/unknown chunk frames blindly.
if (isChatChunkFrame(json)) {
const { blockId, chunk: contentChunk } = json
if (!messageIdMap.has(blockId)) {
messageIdMap.set(blockId, messageId)
}
accumulatedText += contentChunk
accumulatedTextRef.current = accumulatedText
// First answer chunk settles thinking chrome (still visible, no longer “live”).
if (isThinkingStreaming) {
isThinkingStreaming = false
}
if (!blockTextSegments.has(blockId)) {
blockTextOrder.push(blockId)
blockTextSegments.set(blockId, '')
}
blockTextSegments.set(blockId, blockTextSegments.get(blockId)! + contentChunk)
recomputeAccumulatedText()
logger.debug('[useChatStreaming] Received chunk', {
blockId,
chunkLength: contentChunk.length,
@@ -416,17 +626,47 @@ export function useChatStreaming() {
}
}
}
} else if (blockId && eventType === 'end') {
setMessages((prev) =>
prev.map((msg) => (msg.id === messageId ? { ...msg, isStreaming: false } : msg))
)
}
},
})
if (!terminated) {
flushUI()
// Stream closed without a terminal final/error frame (abrupt disconnect,
// or only non-terminal stream_error). Clear live chrome so the UI does not
// stay stuck in a streaming/loading state.
const wasAborted = streamAbortSignal.aborted
settleRunningToolCalls(toolCallsMap, wasAborted ? 'cancelled' : 'error')
syncToolCallsRef()
isThinkingStreaming = false
const toolsSnapshot = snapshotToolCalls(toolCallOrder, toolCallsMap)
setMessages((prev) =>
prev.map((msg) => {
if (msg.id !== messageId) return msg
// stopStreaming already wrote the stop notice into content; do not clobber it.
if (wasAborted) {
return {
...msg,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
thinking: accumulatedThinking || msg.thinking,
toolCalls: toolsSnapshot ?? msg.toolCalls,
}
}
return {
...msg,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
content: accumulatedText || msg.content,
thinking: accumulatedThinking || msg.thinking,
toolCalls: toolsSnapshot ?? msg.toolCalls,
}
})
)
if (
!wasAborted &&
shouldPlayAudio &&
streamingOptions?.audioStreamHandler &&
accumulatedText.length > lastAudioPosition
@@ -442,10 +682,31 @@ export function useChatStreaming() {
}
}
} catch (error) {
logger.error('Error processing stream:', error)
// Stop / timeout abort the shared fetch controller; body read then throws AbortError.
// Match chat.tsx + use-audio-streaming: expected cancel, not a hard failure.
if (error instanceof Error && error.name === 'AbortError') {
logger.info('Stream aborted by user or timeout')
settleRunningToolCalls(toolCallsMap, 'cancelled')
} else {
logger.error('Error processing stream:', error)
settleRunningToolCalls(toolCallsMap, 'error')
}
syncToolCallsRef()
flushUI()
const toolsSnapshot = snapshotToolCalls(toolCallOrder, toolCallsMap)
setMessages((prev) =>
prev.map((msg) => (msg.id === messageId ? { ...msg, isStreaming: false } : msg))
prev.map((msg) =>
msg.id === messageId
? {
...msg,
isStreaming: false,
isThinkingStreaming: false,
isToolStreaming: false,
thinking: accumulatedThinking || msg.thinking,
toolCalls: toolsSnapshot ?? msg.toolCalls,
}
: msg
)
)
} finally {
if (uiRAF !== null) cancelAnimationFrame(uiRAF)
@@ -453,11 +714,10 @@ export function useChatStreaming() {
setIsStreamingResponse(false)
abortControllerRef.current = null
if (!userHasScrolled) {
setTimeout(() => {
scrollToBottom()
}, 300)
}
// Stick-to-bottom-aware; no-ops if the user scrolled away mid-stream.
setTimeout(() => {
scrollToBottom()
}, 300)
if (shouldPlayAudio) {
streamingOptions?.onAudioEnd?.()
apps/sim/app/api/chat/[identifier]/otp/route.ts
+2
View File
@@ -160,6 +160,7 @@ export const PUT = withRouteHandler(
authType: chat.authType,
password: chat.password,
outputConfigs: chat.outputConfigs,
includeThinking: chat.includeThinking,
})
.from(chat)
.where(
@@ -209,6 +210,7 @@ export const PUT = withRouteHandler(
customizations: deployment.customizations,
authType: deployment.authType,
outputConfigs: deployment.outputConfigs,
includeThinking: deployment.includeThinking ?? false,
})
setChatAuthCookie(response, deployment.id, deployment.authType, deployment.password)
apps/sim/app/api/chat/[identifier]/route.test.ts
+83
View File
@@ -36,6 +36,7 @@ function createMockNextRequest(
method,
headers: headersObj,
nextUrl: parsedUrl,
signal: AbortSignal.timeout(60_000),
cookies: {
get: vi.fn().mockReturnValue(undefined),
},
@@ -106,6 +107,7 @@ vi.mock('@/lib/uploads', () => ({
vi.mock('@/lib/workflows/streaming/streaming', () => ({
createStreamingResponse: vi.fn().mockImplementation(async () => createMockStream()),
agentStreamProtocolResponseHeaders: vi.fn().mockReturnValue({}),
}))
vi.mock('@/lib/workflows/executor/execute-workflow', () => ({
@@ -124,6 +126,7 @@ vi.mock('@/lib/core/utils/sse', () => ({
vi.mock('@/lib/core/security/encryption', () => encryptionMock)
import { preprocessExecution } from '@/lib/execution/preprocessing'
import { executeWorkflow } from '@/lib/workflows/executor/execute-workflow'
import { createStreamingResponse } from '@/lib/workflows/streaming/streaming'
import { GET, POST } from '@/app/api/chat/[identifier]/route'
@@ -142,6 +145,7 @@ describe('Chat Identifier API Route', () => {
primaryColor: '#000000',
},
outputConfigs: [{ blockId: 'block-1', path: 'output' }],
includeThinking: false,
},
]
@@ -395,14 +399,93 @@ describe('Chat Identifier API Route', () => {
expect(createStreamingResponse).toHaveBeenCalledWith(
expect.objectContaining({
executeFn: expect.any(Function),
requestSignal: expect.any(AbortSignal),
requestHeaders: expect.anything(),
streamConfig: expect.objectContaining({
isSecureMode: true,
workflowTriggerType: 'chat',
includeThinking: false,
}),
})
)
}, 10000)
it('enables agent events for the execution only when policy and protocol header agree', async () => {
const thinkingChatResult = [{ ...mockChatResult[0], includeThinking: true }]
dbChainMockFns.select.mockImplementation((fields: Record<string, unknown>) => {
if (fields && fields.isDeployed !== undefined) {
return {
from: vi.fn().mockReturnValue({
where: vi.fn().mockReturnValue({
limit: vi.fn().mockReturnValue(mockWorkflowResult),
}),
}),
}
}
return {
from: vi.fn().mockReturnValue({
where: vi.fn().mockReturnValue({
limit: vi.fn().mockReturnValue(thinkingChatResult),
}),
}),
}
})
const req = createMockNextRequest(
'POST',
{ input: 'Hello world' },
{ 'X-Sim-Stream-Protocol': 'agent-events-v1' }
)
const response = await POST(req, { params: Promise.resolve({ identifier: 'test-chat' }) })
expect(response.status).toBe(200)
const options = vi.mocked(createStreamingResponse).mock.calls[0][0]
expect(options.streamConfig).toMatchObject({ includeThinking: true })
await options.executeFn({
onStream: vi.fn(),
onBlockComplete: vi.fn(),
abortSignal: new AbortController().signal,
})
const executeOptions = vi.mocked(executeWorkflow).mock.calls[0][4]
expect(executeOptions).toMatchObject({ includeThinking: true, agentEvents: true })
}, 10000)
it('keeps agent events off when the protocol header is missing, even with policy on', async () => {
const thinkingChatResult = [{ ...mockChatResult[0], includeThinking: true }]
dbChainMockFns.select.mockImplementation((fields: Record<string, unknown>) => {
if (fields && fields.isDeployed !== undefined) {
return {
from: vi.fn().mockReturnValue({
where: vi.fn().mockReturnValue({
limit: vi.fn().mockReturnValue(mockWorkflowResult),
}),
}),
}
}
return {
from: vi.fn().mockReturnValue({
where: vi.fn().mockReturnValue({
limit: vi.fn().mockReturnValue(thinkingChatResult),
}),
}),
}
})
const req = createMockNextRequest('POST', { input: 'Hello world' })
const response = await POST(req, { params: Promise.resolve({ identifier: 'test-chat' }) })
expect(response.status).toBe(200)
const options = vi.mocked(createStreamingResponse).mock.calls[0][0]
await options.executeFn({
onStream: vi.fn(),
onBlockComplete: vi.fn(),
abortSignal: new AbortController().signal,
})
const executeOptions = vi.mocked(executeWorkflow).mock.calls[0][4]
expect(executeOptions).toMatchObject({ includeThinking: true, agentEvents: false })
}, 10000)
it('should handle streaming response body correctly', async () => {
const req = createMockNextRequest('POST', { input: 'Hello world' })
const params = Promise.resolve({ identifier: 'test-chat' })
apps/sim/app/api/chat/[identifier]/route.ts
+27 -2
View File
@@ -27,6 +27,7 @@ interface ChatConfigSource {
customizations: unknown
authType: string | null
outputConfigs: unknown
includeThinking?: boolean | null
}
function toChatConfigResponse(deployment: ChatConfigSource) {
@@ -37,6 +38,7 @@ function toChatConfigResponse(deployment: ChatConfigSource) {
customizations: deployment.customizations,
authType: deployment.authType,
outputConfigs: deployment.outputConfigs,
includeThinking: deployment.includeThinking ?? false,
}
}
@@ -80,6 +82,7 @@ export const POST = withRouteHandler(
password: chat.password,
allowedEmails: chat.allowedEmails,
outputConfigs: chat.outputConfigs,
includeThinking: chat.includeThinking,
})
.from(chat)
.where(and(eq(chat.identifier, identifier), isNull(chat.archivedAt)))
@@ -213,7 +216,12 @@ export const POST = withRouteHandler(
}
}
const { createStreamingResponse } = await import('@/lib/workflows/streaming/streaming')
const { createStreamingResponse, agentStreamProtocolResponseHeaders } = await import(
'@/lib/workflows/streaming/streaming'
)
const { shouldEmitAgentStreamEvents } = await import(
'@/lib/workflows/streaming/agent-stream-protocol'
)
const { executeWorkflow } = await import('@/lib/workflows/executor/execute-workflow')
const { SSE_HEADERS } = await import('@/lib/core/utils/sse')
@@ -269,17 +277,25 @@ export const POST = withRouteHandler(
variables: (workflowRecord?.variables as Record<string, unknown>) ?? undefined,
}
const includeThinking = deployment.includeThinking ?? false
const agentEvents = shouldEmitAgentStreamEvents({
includeThinking,
requestHeaders: request.headers,
})
const stream = await createStreamingResponse({
requestId,
streamConfig: {
selectedOutputs,
isSecureMode: true,
workflowTriggerType: 'chat',
includeThinking,
},
executionId,
workspaceId,
workflowId: deployment.workflowId,
userId: resolvedActorUserId,
requestSignal: request.signal,
requestHeaders: request.headers,
executeFn: async ({ onStream, onBlockComplete, abortSignal }) =>
executeWorkflow(
workflowForExecution,
@@ -297,6 +313,8 @@ export const POST = withRouteHandler(
abortSignal,
executionMode: 'stream',
billingAttribution,
includeThinking,
agentEvents,
},
executionId
),
@@ -304,7 +322,13 @@ export const POST = withRouteHandler(
const streamResponse = new NextResponse(stream, {
status: 200,
headers: SSE_HEADERS,
headers: {
...SSE_HEADERS,
...agentStreamProtocolResponseHeaders({
includeThinking,
requestHeaders: request.headers,
}),
},
})
return streamResponse
} catch (error: any) {
@@ -341,6 +365,7 @@ export const GET = withRouteHandler(
password: chat.password,
allowedEmails: chat.allowedEmails,
outputConfigs: chat.outputConfigs,
includeThinking: chat.includeThinking,
})
.from(chat)
.where(and(eq(chat.identifier, identifier), isNull(chat.archivedAt)))
apps/sim/app/api/chat/manage/[id]/route.ts
+6
View File
@@ -117,6 +117,7 @@ export const PATCH = withRouteHandler(
password,
allowedEmails,
outputConfigs,
includeThinking,
} = validatedData
if (workflowId && workflowId !== existingChat[0].workflowId) {
@@ -250,6 +251,10 @@ export const PATCH = withRouteHandler(
updateData.outputConfigs = outputConfigs
}
if (includeThinking !== undefined) {
updateData.includeThinking = includeThinking
}
const emailCount = Array.isArray(updateData.allowedEmails)
? updateData.allowedEmails.length
: undefined
@@ -263,6 +268,7 @@ export const PATCH = withRouteHandler(
hasPassword: updateData.password !== undefined,
emailCount,
outputConfigsCount,
includeThinking: updateData.includeThinking,
})
await db.update(chat).set(updateData).where(eq(chat.id, chatId))
apps/sim/app/api/chat/route.ts
+2
View File
@@ -68,6 +68,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
password,
allowedEmails = [],
outputConfigs = [],
includeThinking = false,
} = parsed.data.body
if (authType === 'password' && !password) {
@@ -127,6 +128,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
password,
allowedEmails,
outputConfigs,
includeThinking,
workspaceId: workflowRecord.workspaceId,
})
apps/sim/app/api/knowledge/search/utils.test.ts
+16 -3
View File
@@ -308,10 +308,23 @@ describe('Knowledge Search Utils', () => {
it('should throw error when no API configuration provided', async () => {
const { env } = await import('@/lib/core/config/env')
Object.keys(env).forEach((key) => delete (env as any)[key])
// The env object lazily reads process.env, so a developer's local .env
// keys survive the deletion above — stub the direct key empty and fail
// the hosted rotation fallback for hermeticity on any machine.
vi.stubEnv('OPENAI_API_KEY', '')
const apiKeysModule = await import('@/lib/core/config/api-keys')
const rotationSpy = vi.spyOn(apiKeysModule, 'getRotatingApiKey').mockImplementation(() => {
throw new Error('No rotation keys configured')
})
await expect(generateSearchEmbedding('test query')).rejects.toThrow(
'OPENAI_API_KEY is not configured'
)
try {
await expect(generateSearchEmbedding('test query')).rejects.toThrow(
'OPENAI_API_KEY is not configured'
)
} finally {
rotationSpy.mockRestore()
vi.unstubAllEnvs()
}
})
it('should handle Azure OpenAI API errors properly', async () => {
apps/sim/app/api/knowledge/utils.test.ts
+16 -3
View File
@@ -345,10 +345,23 @@ describe('Knowledge Utils', () => {
it('should throw error when no API configuration provided', async () => {
const { env } = await import('@/lib/core/config/env')
Object.keys(env).forEach((key) => delete (env as any)[key])
// The env object lazily reads process.env, so a developer's local .env
// keys survive the deletion above — stub the direct key empty and fail
// the hosted rotation fallback for hermeticity on any machine.
vi.stubEnv('OPENAI_API_KEY', '')
const apiKeysModule = await import('@/lib/core/config/api-keys')
const rotationSpy = vi.spyOn(apiKeysModule, 'getRotatingApiKey').mockImplementation(() => {
throw new Error('No rotation keys configured')
})
await expect(generateEmbeddings(['test text'])).rejects.toThrow(
'OPENAI_API_KEY is not configured'
)
try {
await expect(generateEmbeddings(['test text'])).rejects.toThrow(
'OPENAI_API_KEY is not configured'
)
} finally {
rotationSpy.mockRestore()
vi.unstubAllEnvs()
}
})
})
})
apps/sim/app/api/providers/route.ts
+4 -2
View File
@@ -29,6 +29,7 @@ import {
} from '@/ee/access-control/utils/permission-check'
import type { StreamingExecution } from '@/executor/types'
import { executeProviderRequest } from '@/providers'
import { projectStreamingExecutionToByteStream } from '@/providers/stream-pump'
const logger = createLogger('ProvidersAPI')
@@ -274,8 +275,9 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
logger.info(`[${requestId}] Received StreamingExecution from provider`)
// Extract the stream and execution data
const stream = streamingExec.stream
const executionData = streamingExec.execution
// agent-events-v1 is an object stream — project final-turn answer bytes for HTTP.
const byteStream = projectStreamingExecutionToByteStream(streamingExec)
// Attach the execution data as a custom header
// We need to safely serialize the execution data to avoid circular references
@@ -324,7 +326,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
}
// Return the stream with execution data in a header
return new Response(stream, {
return new Response(byteStream, {
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
apps/sim/app/api/resume/[workflowId]/[executionId]/[contextId]/route.ts
+12 -1
View File
@@ -20,7 +20,10 @@ import { getBaseUrl } from '@/lib/core/utils/urls'
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { preprocessExecution } from '@/lib/execution/preprocessing'
import { PauseResumeManager } from '@/lib/workflows/executor/human-in-the-loop-manager'
import { createStreamingResponse } from '@/lib/workflows/streaming/streaming'
import {
agentStreamProtocolResponseHeaders,
createStreamingResponse,
} from '@/lib/workflows/streaming/streaming'
import { validateWorkflowAccess } from '@/app/api/workflows/middleware'
import type { ResumeExecutionPayload } from '@/background/resume-execution'
import { ExecutionSnapshot } from '@/executor/execution/snapshot'
@@ -257,12 +260,15 @@ export const POST = withRouteHandler(
streamConfig: {
selectedOutputs: persistedSnapshot.selectedOutputs,
timeoutMs: preprocessResult.executionTimeout?.sync,
includeThinking: persistedSnapshot.metadata.includeThinking === true,
},
executionId: enqueueResult.resumeExecutionId,
workspaceId: workflow.workspaceId || undefined,
workflowId,
userId: enqueueResult.userId,
allowLargeValueWorkflowScope: true,
requestSignal: request.signal,
requestHeaders: request.headers,
executeFn: async ({ onStream, onBlockComplete, abortSignal }) =>
PauseResumeManager.startResumeExecution({
...resumeArgs,
@@ -275,6 +281,11 @@ export const POST = withRouteHandler(
return new NextResponse(stream, {
headers: {
...SSE_HEADERS,
// Echo the negotiated stream protocol (same as the public chat route).
...agentStreamProtocolResponseHeaders({
includeThinking: persistedSnapshot.metadata.includeThinking === true,
requestHeaders: request.headers,
}),
'X-Execution-Id': enqueueResult.resumeExecutionId,
},
})
apps/sim/app/api/workflows/[id]/chat/status/route.test.ts
+2
View File
@@ -74,6 +74,7 @@ describe('Workflow Chat Status Route', () => {
authType: 'public',
allowedEmails: [],
outputConfigs: [{ blockId: 'agent-1', path: 'content' }],
includeThinking: false,
password: 'secret',
isActive: true,
},
@@ -88,5 +89,6 @@ describe('Workflow Chat Status Route', () => {
expect(data.deployment.id).toBe('chat-1')
expect(data.deployment.hasPassword).toBe(true)
expect(data.deployment.outputConfigs).toEqual([{ blockId: 'agent-1', path: 'content' }])
expect(data.deployment.includeThinking).toBe(false)
})
})
apps/sim/app/api/workflows/[id]/chat/status/route.ts
+2
View File
@@ -52,6 +52,7 @@ export const GET = withRouteHandler(
authType: chat.authType,
allowedEmails: chat.allowedEmails,
outputConfigs: chat.outputConfigs,
includeThinking: chat.includeThinking,
password: chat.password,
isActive: chat.isActive,
})
@@ -71,6 +72,7 @@ export const GET = withRouteHandler(
authType: deploymentResults[0].authType,
allowedEmails: deploymentResults[0].allowedEmails,
outputConfigs: deploymentResults[0].outputConfigs,
includeThinking: deploymentResults[0].includeThinking ?? false,
hasPassword: Boolean(deploymentResults[0].password),
}
: null
apps/sim/app/api/workflows/[id]/execute/route.ts
+34 -2
View File
@@ -71,7 +71,11 @@ import {
import { getCustomBlockRowsForWorkspace } from '@/lib/workflows/custom-blocks/operations'
import { executeWorkflow } from '@/lib/workflows/executor/execute-workflow'
import { executeWorkflowCore } from '@/lib/workflows/executor/execution-core'
import { type ExecutionEvent, encodeSSEEvent } from '@/lib/workflows/executor/execution-events'
import {
type ExecutionEvent,
encodeSSEEvent,
LIVE_ONLY_EXECUTION_EVENT_TYPES,
} from '@/lib/workflows/executor/execution-events'
import {
claimExecutionId,
type ExecutionIdClaim,
@@ -84,6 +88,10 @@ import {
loadWorkflowDeploymentVersionState,
loadWorkflowFromNormalizedTables,
} from '@/lib/workflows/persistence/utils'
import {
forwardAgentStreamToExecutionEvents,
shouldForwardAnswerTextFromSink,
} from '@/lib/workflows/streaming/forward-agent-stream-events'
import { createStreamingResponse } from '@/lib/workflows/streaming/streaming'
import { createHttpResponseFromBlock, workflowHasResponseBlock } from '@/lib/workflows/utils'
import { getWorkspaceBillingSettings } from '@/lib/workspaces/utils'
@@ -1437,6 +1445,8 @@ async function handleExecutePost(
includeFileBase64,
base64MaxBytes,
timeoutMs: preprocessResult.executionTimeout?.sync,
// Workflow API has no chat includeThinking policy — thinking frames stay off.
includeThinking: false,
},
executionId,
largeValueExecutionIds,
@@ -1446,6 +1456,8 @@ async function handleExecutePost(
workflowId,
userId: actorUserId,
allowLargeValueWorkflowScope,
requestSignal: req.signal,
requestHeaders: req.headers,
executeFn: async ({ onStream, onBlockComplete, abortSignal }) =>
executeWorkflow(
streamWorkflow,
@@ -1518,7 +1530,7 @@ async function handleExecutePost(
event: ExecutionEvent,
terminalStatus?: TerminalExecutionStreamStatus
) => {
const isBuffered = event.type !== 'stream:chunk' && event.type !== 'stream:done'
const isBuffered = !LIVE_ONLY_EXECUTION_EVENT_TYPES.has(event.type)
let eventToSend = event
if (isBuffered) {
try {
@@ -1716,6 +1728,21 @@ async function handleExecutePost(
const onStream = async (streamingExec: StreamingExecution) => {
const blockId = (streamingExec.execution as any).blockId
// Live answer text rides the sink when available (pending deltas
// stream as the model generates; chunk_reset clears intermediate
// turns). The byte stream is then drained without re-emitting
// chunks — its text is the same final-turn content.
const answerTextFromSink = shouldForwardAnswerTextFromSink(streamingExec)
// Sync window: attach sink before first await so pump delivers thinking/tools.
const unsubscribe = forwardAgentStreamToExecutionEvents(streamingExec, {
blockId,
executionId,
workflowId,
sendEvent,
forwardAnswerText: answerTextFromSink,
})
const reader = streamingExec.stream.getReader()
const decoder = new TextDecoder()
const cancelReader = () => {
@@ -1732,6 +1759,8 @@ async function handleExecutePost(
if (timeoutController.signal.aborted || isStreamClosed) break
if (done) break
if (answerTextFromSink) continue
const chunk = decoder.decode(value, { stream: true })
await sendEvent({
type: 'stream:chunk',
@@ -1756,6 +1785,7 @@ async function handleExecutePost(
reqLogger.error('Error streaming block content:', error)
}
} finally {
unsubscribe()
timeoutController.signal.removeEventListener('abort', cancelReader)
try {
await reader.cancel().catch(() => {})
@@ -1785,6 +1815,8 @@ async function handleExecutePost(
allowLargeValueWorkflowScope,
callChain,
executionMode: 'sync',
// Canvas execution-events runs are the primary agent-events surface.
agentEvents: true,
}
const sseExecutionVariables = cachedWorkflowData?.variables ?? workflow.variables ?? {}
apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/chat/chat.tsx
+57 -9
View File
@@ -238,6 +238,7 @@ export function Chat() {
selectedWorkflowOutputs,
setSelectedWorkflowOutput,
appendMessageContent,
setMessageContent,
finalizeMessageStream,
getConversationId,
clearChat,
@@ -256,6 +257,7 @@ export function Chat() {
selectedWorkflowOutputs: s.selectedWorkflowOutputs,
setSelectedWorkflowOutput: s.setSelectedWorkflowOutput,
appendMessageContent: s.appendMessageContent,
setMessageContent: s.setMessageContent,
finalizeMessageStream: s.finalizeMessageStream,
getConversationId: s.getConversationId,
clearChat: s.clearChat,
@@ -495,10 +497,21 @@ export function Chat() {
async (stream: ReadableStream<Uint8Array>, responseMessageId: string) => {
const reader = stream.getReader()
streamReaderRef.current = reader
/**
* Answer text tracked per block so a `chunk_reset` frame (a live-streamed
* turn resolved to tool calls) can drop one block's contribution. Each
* flush replaces the message content with the joined segments.
*/
const blockOrder: string[] = []
const blockSegments = new Map<string, string>()
let accumulatedContent = ''
const recomputeContent = () => {
accumulatedContent = blockOrder.map((id) => blockSegments.get(id) ?? '').join('')
}
const BATCH_MAX_MS = 50
let pendingChunks = ''
let contentDirty = false
let batchRAF: number | null = null
let batchTimer: ReturnType<typeof setTimeout> | null = null
let lastFlush = 0
@@ -512,9 +525,9 @@ export function Chat() {
clearTimeout(batchTimer)
batchTimer = null
}
if (pendingChunks) {
appendMessageContent(responseMessageId, pendingChunks)
pendingChunks = ''
if (contentDirty) {
setMessageContent(responseMessageId, accumulatedContent)
contentDirty = false
}
lastFlush = performance.now()
}
@@ -534,12 +547,17 @@ export function Chat() {
let finalError: string | null = null
try {
await readSSEEvents<{ event?: string; data?: ExecutionResult; chunk?: string }>(reader, {
await readSSEEvents<{
event?: string
data?: ExecutionResult
chunk?: string
blockId?: string
}>(reader, {
onParseError: (_data, e) => {
logger.error('Error parsing stream data:', e)
},
onEvent: (json) => {
const { event, data: eventData, chunk: contentChunk } = json
const { event, data: eventData, chunk: contentChunk, blockId } = json
if (event === 'final' && eventData) {
if ('success' in eventData && !eventData.success) {
@@ -548,9 +566,32 @@ export function Chat() {
return true
}
if (event === 'chunk_reset' && blockId) {
// Drop the block's provisional text and its order slot — the
// final turn re-registers at the end, keeping render order =
// arrival order (separators are recomputed on re-stream).
if (blockSegments.has(blockId)) {
blockSegments.delete(blockId)
const orderIndex = blockOrder.indexOf(blockId)
if (orderIndex !== -1) {
blockOrder.splice(orderIndex, 1)
}
recomputeContent()
contentDirty = true
scheduleFlush()
}
return
}
if (contentChunk) {
accumulatedContent += contentChunk
pendingChunks += contentChunk
const segmentKey = blockId ?? ''
if (!blockSegments.has(segmentKey)) {
blockOrder.push(segmentKey)
blockSegments.set(segmentKey, '')
}
blockSegments.set(segmentKey, blockSegments.get(segmentKey)! + contentChunk)
recomputeContent()
contentDirty = true
scheduleFlush()
}
},
@@ -578,7 +619,14 @@ export function Chat() {
focusInput(100)
}
},
[appendMessageContent, finalizeMessageStream, focusInput, selectedOutputs, activeWorkflowId]
[
appendMessageContent,
setMessageContent,
finalizeMessageStream,
focusInput,
selectedOutputs,
activeWorkflowId,
]
)
/**
apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/deploy/components/deploy-modal/components/chat/chat.tsx
+20
View File
@@ -11,6 +11,7 @@ import {
Label,
Loader,
Skeleton,
Switch,
TagInput,
type TagItem,
Textarea,
@@ -84,6 +85,7 @@ const initialFormData: ChatFormData = {
emails: [],
welcomeMessage: 'Hi there! How can I help you today?',
selectedOutputBlocks: [],
includeThinking: false,
}
export function ChatDeploy({
@@ -194,6 +196,7 @@ export function ChatDeploy({
(config: { blockId: string; path: string }) => `${config.blockId}_${config.path}`
)
: [],
includeThinking: existingChat.includeThinking ?? false,
})
if (existingChat.customizations?.imageUrl) {
@@ -369,6 +372,23 @@ export function ChatDeploy({
)}
</div>
<div className='flex items-center justify-between gap-3'>
<div className='min-w-0'>
<Label className='block pl-0.5 font-medium text-[var(--text-primary)] text-small'>
Include thinking
</Label>
<p className='mt-[6.5px] text-[var(--text-secondary)] text-xs'>
Allow this chat to stream model thinking when the client opts in. Off by default.
</p>
</div>
<Switch
checked={formData.includeThinking}
disabled={chatSubmitting}
onCheckedChange={(checked) => updateField('includeThinking', checked)}
aria-label='Include thinking'
/>
</div>
<AuthSelector
key={`${existingChat?.id ?? 'new'}-${formInitCounter}`}
authType={formData.authType}
apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/terminal/components/output-panel/output-panel.tsx
+29
View File
@@ -27,6 +27,10 @@ import {
X,
} from 'lucide-react'
import Link from 'next/link'
import {
AgentStreamThinkingChrome,
AgentStreamToolCallsChrome,
} from '@/components/agent-stream/agent-stream-chrome'
import {
OutputContextMenu,
StructuredOutput,
@@ -567,6 +571,31 @@ export const OutputPanel = React.memo(function OutputPanel({
className={clsx('flex-1 overflow-y-auto', !wrapText && 'overflow-x-auto')}
onContextMenu={handleOutputPanelContextMenu}
>
{!showInput &&
(selectedEntry.agentStreamThinking ||
(selectedEntry.agentStreamToolCalls &&
selectedEntry.agentStreamToolCalls.length > 0)) && (
<div className='border-[var(--border)] border-b px-3 pt-3'>
{selectedEntry.agentStreamThinking ? (
<AgentStreamThinkingChrome
thinking={selectedEntry.agentStreamThinking}
isStreaming={Boolean(
selectedEntry.isRunning && selectedEntry.agentStreamActive
)}
/>
) : null}
{selectedEntry.agentStreamToolCalls &&
selectedEntry.agentStreamToolCalls.length > 0 ? (
<AgentStreamToolCallsChrome
toolCalls={selectedEntry.agentStreamToolCalls}
isStreaming={Boolean(
selectedEntry.isRunning &&
selectedEntry.agentStreamToolCalls.some((t) => t.status === 'running')
)}
/>
) : null}
</div>
)}
{shouldShowCodeDisplay ? (
<OutputCodeContent
code={selectedEntry.input.code}
apps/sim/app/workspace/[workspaceId]/w/[workflowId]/hooks/use-workflow-execution.ts
+190 -7
View File
@@ -7,11 +7,23 @@ import { generateId } from '@sim/utils/id'
import { useQueryClient } from '@tanstack/react-query'
import { useParams } from 'next/navigation'
import { useShallow } from 'zustand/react/shallow'
import {
type AgentStreamToolCall,
applyToolCallPhase,
settleRunningToolCalls,
snapshotToolCalls,
toolCallKey,
} from '@/components/agent-stream/tool-call-lifecycle'
import { requestJson } from '@/lib/api/client/request'
import { cancelWorkflowExecutionContract, workflowLogContract } from '@/lib/api/contracts/workflows'
import { buildTraceSpans } from '@/lib/logs/execution/trace-spans/trace-spans'
import { processStreamingBlockLogs } from '@/lib/tokenization'
import type { ExecutionPausedData } from '@/lib/workflows/executor/execution-events'
import type {
ExecutionPausedData,
StreamDoneData,
StreamThinkingData,
StreamToolData,
} from '@/lib/workflows/executor/execution-events'
import { collectInputFormatFiles, isFileFieldType } from '@/lib/workflows/input-format'
import {
extractTriggerMockPayload,
@@ -209,6 +221,122 @@ function buildInputFormatInput(inputFormatValue: unknown): Record<string, any> |
return Object.keys(testInput).length > 0 ? testInput : undefined
}
/**
* Thinking deltas arrive per token; batch console writes (same cadence as the
* chat surface) so the terminal does not re-render per delta.
*/
const AGENT_STREAM_THINKING_FLUSH_MS = 50
type UpdateConsoleFn = ReturnType<(typeof useTerminalConsoleStore)['getState']>['updateConsole']
interface AgentStreamChromeOptions {
executionIdRef: { current: string }
updateConsole: UpdateConsoleFn
}
/**
* Per-run terminal chrome for live agent stream events: accumulates thinking
* text (batched) and tool chips per block, and settles running chips when a
* block's stream ends, a block errors, or the execution terminates. Shared by
* the full-run and run-from-block paths so both render identical chrome.
*/
function createAgentStreamChrome({ executionIdRef, updateConsole }: AgentStreamChromeOptions) {
const thinkingByBlock = new Map<string, string>()
const toolCallsByBlock = new Map<string, Map<string, AgentStreamToolCall>>()
const toolOrderByBlock = new Map<string, string[]>()
const thinkingFlushTimers = new Map<string, ReturnType<typeof setTimeout>>()
const flushThinking = (blockId: string) => {
const timer = thinkingFlushTimers.get(blockId)
if (timer !== undefined) {
clearTimeout(timer)
thinkingFlushTimers.delete(blockId)
}
const thinking = thinkingByBlock.get(blockId)
if (thinking === undefined) return
updateConsole(
blockId,
{ agentStreamThinking: thinking, agentStreamActive: true },
executionIdRef.current
)
}
const settleBlock = (blockId: string, status: 'success' | 'error' | 'cancelled') => {
flushThinking(blockId)
const map = toolCallsByBlock.get(blockId)
const order = toolOrderByBlock.get(blockId)
if (map && order) {
settleRunningToolCalls(map, status)
updateConsole(
blockId,
{
agentStreamActive: false,
agentStreamToolCalls: snapshotToolCalls(order, map),
},
executionIdRef.current
)
} else {
updateConsole(blockId, { agentStreamActive: false }, executionIdRef.current)
}
}
const settleAll = (status: 'success' | 'error' | 'cancelled') => {
const blockIds = new Set<string>([...thinkingByBlock.keys(), ...toolCallsByBlock.keys()])
for (const blockId of blockIds) {
settleBlock(blockId, status)
}
}
const onStreamThinking = (data: StreamThinkingData) => {
const prev = thinkingByBlock.get(data.blockId) ?? ''
thinkingByBlock.set(data.blockId, prev + data.text)
if (!thinkingFlushTimers.has(data.blockId)) {
thinkingFlushTimers.set(
data.blockId,
setTimeout(() => flushThinking(data.blockId), AGENT_STREAM_THINKING_FLUSH_MS)
)
}
}
const onStreamTool = (data: StreamToolData) => {
if (!toolCallsByBlock.has(data.blockId)) {
toolCallsByBlock.set(data.blockId, new Map())
toolOrderByBlock.set(data.blockId, [])
}
const map = toolCallsByBlock.get(data.blockId)!
const order = toolOrderByBlock.get(data.blockId)!
applyToolCallPhase(
map,
order,
{
key: toolCallKey(data.blockId, data.id),
id: data.id,
name: data.name,
phase: data.phase,
status: data.status,
},
(tool) => tool
)
updateConsole(
data.blockId,
{
agentStreamToolCalls: snapshotToolCalls(order, map),
agentStreamActive: true,
},
executionIdRef.current
)
}
const onStreamDone = (data: StreamDoneData) => {
logger.info('Stream done for block:', data.blockId)
settleBlock(data.blockId, 'success')
}
return { flushThinking, settleBlock, settleAll, onStreamThinking, onStreamTool, onStreamDone }
}
export function useWorkflowExecution() {
const { workspaceId: routeWorkspaceId } = useParams<{ workspaceId: string }>()
const hydrationWorkspaceId = useWorkflowRegistry((s) => s.hydration.workspaceId)
@@ -625,6 +753,19 @@ export function useWorkflowExecution() {
streamReadingPromises.push(promise)
}
/**
* Intermediate-turn reconciliation: drop the block's streamed text
* (chunk_reset frame) and remove its bookkeeping entirely so
* separator counting ignores it and the final turn (or, if none
* re-streams, onBlockComplete's output fallback) starts clean.
*/
const onStreamReset = (blockId: string) => {
if (!streamedChunks.has(blockId)) return
streamedChunks.delete(blockId)
processedFirstChunk.delete(blockId)
safeEnqueue(encodeSSE({ blockId, event: 'chunk_reset' }))
}
// Handle non-streaming blocks (like Function blocks)
const onBlockComplete = async (blockId: string, output: any) => {
// Skip if this block already had streaming content (avoid duplicates)
@@ -682,7 +823,9 @@ export function useWorkflowExecution() {
onStream,
executionId,
onBlockComplete,
'chat'
'chat',
undefined,
onStreamReset
)
// Check if execution was cancelled
@@ -846,7 +989,8 @@ export function useWorkflowExecution() {
executionId?: string,
onBlockComplete?: (blockId: string, output: any) => Promise<void>,
overrideTriggerType?: 'chat' | 'manual' | 'api',
stopAfterBlockId?: string
stopAfterBlockId?: string,
onStreamReset?: (blockId: string) => void
): Promise<ExecutionResult | StreamingExecution> => {
// Use diff workflow for execution when available, regardless of canvas view state
const executionWorkflowState = null as {
@@ -1063,6 +1207,9 @@ export function useWorkflowExecution() {
const activeBlocksSet = new Set<string>()
const activeBlockRefCounts = new Map<string, number>()
const streamedChunks = new Map<string, string[]>()
const agentStreamChrome = createAgentStreamChrome({ executionIdRef, updateConsole })
const settleAgentStreamChrome = agentStreamChrome.settleBlock
const settleAllAgentStreamChrome = agentStreamChrome.settleAll
const accumulatedBlockLogs: BlockLog[] = []
const accumulatedBlockStates = new Map<string, BlockState>()
const executedBlockIds = new Set<string>()
@@ -1134,7 +1281,11 @@ export function useWorkflowExecution() {
onBlockStarted: blockHandlers.onBlockStarted,
onBlockCompleted: blockHandlers.onBlockCompleted,
onBlockError: blockHandlers.onBlockError,
onBlockError: (data) => {
// Failures often skip stream:done — settle thinking/tool chrome here.
settleAgentStreamChrome(data.blockId, 'error')
blockHandlers.onBlockError(data)
},
onBlockChildWorkflowStarted: blockHandlers.onBlockChildWorkflowStarted,
onStreamChunk: (data) => {
@@ -1167,10 +1318,19 @@ export function useWorkflowExecution() {
}
},
onStreamDone: (data) => {
logger.info('Stream done for block:', data.blockId)
onStreamChunkReset: (data) => {
// Live-streamed text belonged to an intermediate turn (tools
// follow); the final turn re-streams as regular chunks.
streamedChunks.delete(data.blockId)
if (onStreamReset && isExecutingFromChat) {
onStreamReset(data.blockId)
}
},
onStreamThinking: agentStreamChrome.onStreamThinking,
onStreamTool: agentStreamChrome.onStreamTool,
onStreamDone: agentStreamChrome.onStreamDone,
onExecutionCompleted: (data) => {
executionFinished = true
if (
@@ -1181,6 +1341,8 @@ export function useWorkflowExecution() {
)
return
settleAllAgentStreamChrome(data.success ? 'success' : 'error')
if (activeWorkflowId) {
setCurrentExecutionId(activeWorkflowId, null)
reconcileFinalBlockLogs(
@@ -1278,6 +1440,9 @@ export function useWorkflowExecution() {
)
return
// HITL pause mid tool-loop — open tools never got an end event.
settleAllAgentStreamChrome('cancelled')
if (activeWorkflowId) {
setCurrentExecutionId(activeWorkflowId, null)
reconcileFinalBlockLogs(
@@ -1322,6 +1487,8 @@ export function useWorkflowExecution() {
)
return
settleAllAgentStreamChrome('error')
if (activeWorkflowId) {
setCurrentExecutionId(activeWorkflowId, null)
}
@@ -1364,6 +1531,8 @@ export function useWorkflowExecution() {
)
return
settleAllAgentStreamChrome('cancelled')
if (activeWorkflowId) {
setCurrentExecutionId(activeWorkflowId, null)
}
@@ -1807,6 +1976,7 @@ export function useWorkflowExecution() {
const executedBlockIds = new Set<string>()
const activeBlocksSet = new Set<string>()
const activeBlockRefCounts = new Map<string, number>()
const agentStreamChrome = createAgentStreamChrome({ executionIdRef, updateConsole })
const isCurrentRunFromBlockExecution = () => {
return (
Boolean(executionIdRef.current) &&
@@ -1860,11 +2030,20 @@ export function useWorkflowExecution() {
onBlockStarted: blockHandlers.onBlockStarted,
onBlockCompleted: blockHandlers.onBlockCompleted,
onBlockError: blockHandlers.onBlockError,
onBlockError: (data) => {
// Failures often skip stream:done — settle thinking/tool chrome here.
agentStreamChrome.settleBlock(data.blockId, 'error')
blockHandlers.onBlockError(data)
},
onBlockChildWorkflowStarted: blockHandlers.onBlockChildWorkflowStarted,
onStreamThinking: agentStreamChrome.onStreamThinking,
onStreamTool: agentStreamChrome.onStreamTool,
onStreamDone: agentStreamChrome.onStreamDone,
onExecutionCompleted: (data) => {
if (!isCurrentRunFromBlockExecution()) return
agentStreamChrome.settleAll(data.success ? 'success' : 'error')
const executionId = executionIdRef.current
reconcileFinalBlockLogs(updateConsole, workflowId, executionId, data.finalBlockLogs)
finishRunningEntries(workflowId, executionId)
@@ -1899,6 +2078,8 @@ export function useWorkflowExecution() {
onExecutionPaused: (data) => {
if (!isCurrentRunFromBlockExecution()) return
// HITL pause mid tool-loop — open tools never got an end event.
agentStreamChrome.settleAll('cancelled')
const executionId = executionIdRef.current
reconcileFinalBlockLogs(updateConsole, workflowId, executionId, data.finalBlockLogs)
finishRunningEntries(workflowId, executionId)
@@ -1918,6 +2099,7 @@ export function useWorkflowExecution() {
onExecutionError: (data) => {
if (!isCurrentRunFromBlockExecution()) return
agentStreamChrome.settleAll('error')
const executionId = executionIdRef.current
const isWorkflowModified =
data.error?.includes('Block not found in workflow') ||
@@ -1944,6 +2126,7 @@ export function useWorkflowExecution() {
onExecutionCancelled: (data) => {
if (!isCurrentRunFromBlockExecution()) return
agentStreamChrome.settleAll('cancelled')
const executionId = executionIdRef.current
handleExecutionCancelledConsole({
workflowId,
apps/sim/components/agent-stream/agent-stream-chrome.test.tsx
+286
View File
@@ -0,0 +1,286 @@
/**
* @vitest-environment jsdom
*/
import { act } from 'react'
import { createRoot, type Root } from 'react-dom/client'
import { afterEach, describe, expect, it, vi } from 'vitest'
vi.mock('@sim/emcn', () => ({
cn: (...args: unknown[]) => args.filter(Boolean).join(' '),
}))
vi.mock('@/lib/copilot/tools/tool-display', () => ({
humanizeToolName: (name: string) => name,
}))
vi.mock('@/components/ui', () => ({
ShimmerText: ({
as: Comp = 'span',
children,
className,
...props
}: {
as?: 'span' | 'div'
children: React.ReactNode
className?: string
[key: string]: unknown
}) => {
const Tag = Comp
return (
<Tag data-shimmer='true' className={className} {...props}>
{children}
</Tag>
)
},
}))
import {
AgentStreamThinkingChrome,
AgentStreamToolCallsChrome,
} from '@/components/agent-stream/agent-stream-chrome'
import type { AgentStreamToolCall } from '@/components/agent-stream/tool-call-lifecycle'
function renderChrome(props: { thinking: string; isStreaming?: boolean }): {
container: HTMLDivElement
rerender: (next: { thinking: string; isStreaming?: boolean }) => void
unmount: () => void
} {
;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true
const container = document.createElement('div')
document.body.appendChild(container)
const root: Root = createRoot(container)
const mount = (p: { thinking: string; isStreaming?: boolean }) => {
act(() => {
root.render(<AgentStreamThinkingChrome thinking={p.thinking} isStreaming={p.isStreaming} />)
})
}
mount(props)
return {
container,
rerender: (next) => mount(next),
unmount: () => {
act(() => {
root.unmount()
})
container.remove()
},
}
}
describe('AgentStreamThinkingChrome', () => {
const mounts: Array<() => void> = []
afterEach(() => {
while (mounts.length) {
mounts.pop()?.()
}
})
it('opens while streaming with Thinking… label and scrollable body', () => {
const { container, unmount } = renderChrome({
thinking: 'step one',
isStreaming: true,
})
mounts.push(unmount)
const toggle = container.querySelector(
'[data-testid="agent-stream-thinking-toggle"]'
) as HTMLButtonElement
const body = container.querySelector(
'[data-testid="agent-stream-thinking-body"]'
) as HTMLDivElement
expect(toggle.getAttribute('aria-expanded')).toBe('true')
expect(toggle.textContent).toContain('Thinking…')
expect(
container
.querySelector('[data-testid="agent-stream-thinking-label"]')
?.getAttribute('data-shimmer')
).toBe('true')
expect(body.className).toContain('max-h-40')
expect(body.className).toContain('overflow-y-auto')
expect(body.getAttribute('data-shimmer')).toBeNull()
expect(
container
.querySelector('[data-testid="agent-stream-thinking-shimmer"]')
?.getAttribute('data-shimmer')
).toBe('true')
expect(body.textContent).toContain('step one')
})
it('auto-collapses when streaming ends and shows Thought for a moment', () => {
const { container, rerender, unmount } = renderChrome({
thinking: 'long internal chain',
isStreaming: true,
})
mounts.push(unmount)
rerender({ thinking: 'long internal chain', isStreaming: false })
const toggle = container.querySelector(
'[data-testid="agent-stream-thinking-toggle"]'
) as HTMLButtonElement
const body = container.querySelector(
'[data-testid="agent-stream-thinking-body"]'
) as HTMLDivElement
expect(toggle.getAttribute('aria-expanded')).toBe('false')
expect(toggle.textContent).toContain('Thought for a moment')
expect(
container
.querySelector('[data-testid="agent-stream-thinking-label"]')
?.getAttribute('data-shimmer')
).toBeNull()
expect(container.querySelector('[data-testid="agent-stream-thinking-shimmer"]')).toBeNull()
expect(body.className).toContain('text-[var(--text-muted)]')
})
it('stays open after manual reopen once collapsed', () => {
const { container, rerender, unmount } = renderChrome({
thinking: 'reason\n'.repeat(40),
isStreaming: true,
})
mounts.push(unmount)
rerender({ thinking: 'reason\n'.repeat(40), isStreaming: false })
const toggle = container.querySelector(
'[data-testid="agent-stream-thinking-toggle"]'
) as HTMLButtonElement
expect(toggle.getAttribute('aria-expanded')).toBe('false')
const body = container.querySelector(
'[data-testid="agent-stream-thinking-body"]'
) as HTMLDivElement
Object.defineProperty(body, 'scrollTop', { value: 80, writable: true, configurable: true })
act(() => {
toggle.click()
})
expect(toggle.getAttribute('aria-expanded')).toBe('true')
expect(body.scrollTop).toBe(0)
expect(body.textContent).toContain('reason')
// Re-render with same done state should not force-close a user pin.
rerender({ thinking: 'reason\n'.repeat(40), isStreaming: false })
expect(toggle.getAttribute('aria-expanded')).toBe('true')
})
it('re-opens when a new streaming phase starts', () => {
const { container, rerender, unmount } = renderChrome({
thinking: 'first',
isStreaming: false,
})
mounts.push(unmount)
const toggle = container.querySelector(
'[data-testid="agent-stream-thinking-toggle"]'
) as HTMLButtonElement
// Initial non-streaming starts closed.
expect(toggle.getAttribute('aria-expanded')).toBe('false')
rerender({ thinking: 'first then more', isStreaming: true })
expect(toggle.getAttribute('aria-expanded')).toBe('true')
expect(toggle.textContent).toContain('Thinking…')
})
})
const sampleTools: AgentStreamToolCall[] = [
{
key: 'agent-1:t1',
id: 't1',
name: 'http_request',
displayName: 'Http Request',
status: 'success',
},
]
function renderToolsChrome(props: { toolCalls?: AgentStreamToolCall[]; isStreaming?: boolean }): {
container: HTMLDivElement
rerender: (next: { toolCalls?: AgentStreamToolCall[]; isStreaming?: boolean }) => void
unmount: () => void
} {
;(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true
const container = document.createElement('div')
document.body.appendChild(container)
const root: Root = createRoot(container)
const mount = (p: { toolCalls?: AgentStreamToolCall[]; isStreaming?: boolean }) => {
act(() => {
root.render(
<AgentStreamToolCallsChrome
toolCalls={p.toolCalls ?? sampleTools}
isStreaming={p.isStreaming}
/>
)
})
}
mount(props)
return {
container,
rerender: (next) => mount(next),
unmount: () => {
act(() => {
root.unmount()
})
container.remove()
},
}
}
describe('AgentStreamToolCallsChrome', () => {
const mounts: Array<() => void> = []
afterEach(() => {
while (mounts.length) {
mounts.pop()?.()
}
})
it('opens while tools are streaming and auto-collapses when they finish', () => {
const { container, rerender, unmount } = renderToolsChrome({
isStreaming: true,
toolCalls: [{ ...sampleTools[0], status: 'running' }],
})
mounts.push(unmount)
const toggle = container.querySelector(
'[data-testid="agent-stream-tools-toggle"]'
) as HTMLButtonElement
expect(toggle.getAttribute('aria-expanded')).toBe('true')
expect(toggle.textContent).toContain('Using tools…')
expect(container.textContent).toContain('Http Request')
rerender({ isStreaming: false, toolCalls: sampleTools })
expect(toggle.getAttribute('aria-expanded')).toBe('false')
expect(toggle.textContent).toContain('Tools')
expect(container.textContent).not.toContain('Http Request')
})
it('stays open after manual reopen once collapsed', () => {
const { container, rerender, unmount } = renderToolsChrome({ isStreaming: true })
mounts.push(unmount)
rerender({ isStreaming: false })
const toggle = container.querySelector(
'[data-testid="agent-stream-tools-toggle"]'
) as HTMLButtonElement
expect(toggle.getAttribute('aria-expanded')).toBe('false')
act(() => {
toggle.click()
})
expect(toggle.getAttribute('aria-expanded')).toBe('true')
expect(container.textContent).toContain('Http Request')
rerender({ isStreaming: false })
expect(toggle.getAttribute('aria-expanded')).toBe('true')
})
})
apps/sim/components/agent-stream/agent-stream-chrome.tsx
+246
View File
@@ -0,0 +1,246 @@
'use client'
import { useEffect, useLayoutEffect, useRef, useState } from 'react'
import { cn } from '@sim/emcn'
import { Check, ChevronDown, Circle, Square, X } from 'lucide-react'
import type {
AgentStreamToolCall,
AgentStreamToolStatus,
} from '@/components/agent-stream/tool-call-lifecycle'
import { ShimmerText } from '@/components/ui'
import { humanizeToolName } from '@/lib/copilot/tools/tool-display'
/** Distance from bottom (px) within which we keep following new thinking text. */
const STICK_TO_BOTTOM_THRESHOLD_PX = 24
/**
* Open / pinned-open / auto-collapse state shared by both chrome panels:
* streaming forces the panel open, the panel auto-collapses when streaming
* ends unless the user pinned it open, and manual toggles while idle pin it.
*/
function useAutoCollapseOpen(isStreaming: boolean, onOpen?: (streaming: boolean) => void) {
const [open, setOpen] = useState(!!isStreaming)
const [userPinnedOpen, setUserPinnedOpen] = useState(false)
const wasStreamingRef = useRef(!!isStreaming)
useEffect(() => {
const wasStreaming = wasStreamingRef.current
wasStreamingRef.current = !!isStreaming
if (isStreaming) {
setOpen(true)
setUserPinnedOpen(false)
return
}
if (wasStreaming && !isStreaming && !userPinnedOpen) {
setOpen(false)
}
}, [isStreaming, userPinnedOpen])
const toggle = () => {
setOpen((prev) => {
const next = !prev
if (!isStreaming) {
setUserPinnedOpen(next)
} else {
setUserPinnedOpen(false)
}
if (next) {
onOpen?.(!!isStreaming)
}
return next
})
}
return { open, toggle }
}
export interface AgentStreamThinkingChromeProps {
thinking: string
isStreaming?: boolean
}
export function AgentStreamThinkingChrome({
thinking,
isStreaming = false,
}: AgentStreamThinkingChromeProps) {
const [stickToBottom, setStickToBottom] = useState(true)
const [overflowing, setOverflowing] = useState(false)
const scrollRef = useRef<HTMLDivElement>(null)
/** After a manual reopen of completed thoughts, jump to the top once. */
const reopenFromTopRef = useRef(false)
const { open, toggle } = useAutoCollapseOpen(isStreaming, (streaming) => {
if (streaming) {
setStickToBottom(true)
} else {
// ChatGPT-style: reopen completed thoughts from the top.
setStickToBottom(false)
reopenFromTopRef.current = true
}
})
useEffect(() => {
if (isStreaming) {
setStickToBottom(true)
}
}, [isStreaming])
useLayoutEffect(() => {
const el = scrollRef.current
if (!el || !open) return
setOverflowing(el.scrollHeight > el.clientHeight + 1)
if (reopenFromTopRef.current && !isStreaming) {
el.scrollTop = 0
reopenFromTopRef.current = false
return
}
if (isStreaming && stickToBottom) {
el.scrollTop = el.scrollHeight
}
}, [thinking, open, isStreaming, stickToBottom])
const handleScroll = () => {
const el = scrollRef.current
if (!el) return
const distanceFromBottom = el.scrollHeight - el.scrollTop - el.clientHeight
const nearBottom = distanceFromBottom <= STICK_TO_BOTTOM_THRESHOLD_PX
setStickToBottom(nearBottom)
setOverflowing(el.scrollHeight > el.clientHeight + 1)
}
const label = isStreaming ? 'Thinking…' : 'Thought for a moment'
return (
<div className='mb-3'>
<button
type='button'
className='flex items-center gap-1 text-[var(--text-muted)] text-sm transition-colors hover:text-[var(--text-secondary)]'
onClick={toggle}
aria-expanded={open}
data-testid='agent-stream-thinking-toggle'
>
<ChevronDown
className={cn(
'size-[14px] transition-transform duration-150',
open ? 'rotate-0' : '-rotate-90'
)}
strokeWidth={2}
/>
{isStreaming ? (
<ShimmerText
className='text-sm [--shimmer-rest:var(--text-muted)]'
data-testid='agent-stream-thinking-label'
>
{label}
</ShimmerText>
) : (
<span data-testid='agent-stream-thinking-label'>{label}</span>
)}
</button>
<div
className={cn(
'grid transition-[grid-template-rows,opacity] duration-200 ease-out',
open ? 'grid-rows-[1fr] opacity-100' : 'grid-rows-[0fr] opacity-0'
)}
aria-hidden={!open}
>
<div className='min-h-0 overflow-hidden'>
<div className='relative mt-2'>
<div
ref={scrollRef}
onScroll={handleScroll}
data-testid='agent-stream-thinking-body'
className={cn(
'max-h-40 overflow-y-auto border-[var(--border)] border-l pl-3',
'whitespace-pre-wrap break-words text-sm leading-relaxed',
!isStreaming && 'text-[var(--text-muted)]'
)}
>
{/* Shimmer on an inner node — never on the scroll shell. background-clip:text
on overflow-y-auto breaks scroll/overflow in Chromium. */}
{isStreaming ? (
<ShimmerText
as='div'
className='[--shimmer-rest:var(--text-muted)]'
data-testid='agent-stream-thinking-shimmer'
>
{thinking}
</ShimmerText>
) : (
thinking
)}
</div>
{open && isStreaming && overflowing && (
<div
aria-hidden
className='pointer-events-none absolute inset-x-0 bottom-0 h-8 bg-gradient-to-t from-[var(--bg)] to-transparent'
/>
)}
</div>
</div>
</div>
</div>
)
}
function ToolStatusIcon({ status }: { status: AgentStreamToolStatus }) {
if (status === 'success') {
return <Check className='size-[14px] shrink-0' strokeWidth={2} aria-hidden />
}
if (status === 'error') {
return <X className='size-[14px] shrink-0' strokeWidth={2} aria-hidden />
}
if (status === 'cancelled') {
return <Square className='size-3 shrink-0 fill-current' strokeWidth={0} aria-hidden />
}
return <Circle className='size-3 shrink-0' strokeWidth={2} aria-hidden />
}
export interface AgentStreamToolCallsChromeProps {
toolCalls: AgentStreamToolCall[]
isStreaming?: boolean
}
export function AgentStreamToolCallsChrome({
toolCalls,
isStreaming,
}: AgentStreamToolCallsChromeProps) {
const { open, toggle } = useAutoCollapseOpen(!!isStreaming)
return (
<div className='mb-3'>
<button
type='button'
className='flex items-center gap-1 text-[var(--text-muted)] text-sm transition-colors hover:text-[var(--text-secondary)]'
onClick={toggle}
aria-expanded={open}
data-testid='agent-stream-tools-toggle'
>
<ChevronDown
className={cn('size-[14px] transition-transform', open ? 'rotate-0' : '-rotate-90')}
strokeWidth={2}
/>
<span>{isStreaming ? 'Using tools…' : 'Tools'}</span>
</button>
{open && (
<ul className='mt-2 space-y-1.5 text-[var(--text-muted)] text-sm'>
{toolCalls.map((tool) => (
<li key={tool.key} className='flex items-center gap-2'>
<ToolStatusIcon status={tool.status} />
<span className='truncate'>
{tool.displayName || humanizeToolName(tool.name)}
{tool.status === 'running' ? '…' : ''}
</span>
</li>
))}
</ul>
)}
</div>
)
}
apps/sim/components/agent-stream/tool-call-lifecycle.ts
+117
View File
@@ -0,0 +1,117 @@
/**
* Shared client-side reducer for agent-stream tool chips.
*
* The public chat hook, the canvas execution hook, and the terminal console
* store all consume the same tool lifecycle (keyed ordered upsert on
* start/end, settle-running-on-terminal). This module is the single
* implementation so the three surfaces cannot drift.
*/
import { humanizeToolName } from '@/lib/copilot/tools/tool-display'
export type AgentStreamToolStatus = 'running' | 'success' | 'error' | 'cancelled'
export interface AgentStreamToolCall {
key: string
id: string
name: string
displayName?: string
status: AgentStreamToolStatus
}
/** Terminal statuses a running chip can settle to. */
export type AgentStreamToolTerminalStatus = Exclude<AgentStreamToolStatus, 'running'>
/** Canonical chip key — unique per block and tool call within an execution. */
export function toolCallKey(blockId: string, id: string): string {
return `${blockId}:${id}`
}
/** Normalizes a wire `status` into a terminal chip status (default success). */
export function resolveToolCallEndStatus(status?: string): AgentStreamToolTerminalStatus {
return status === 'error' || status === 'cancelled' ? status : 'success'
}
/**
* Applies a tool lifecycle phase to a keyed map + insertion-order list.
* `extend` lets callers add surface-specific fields (e.g. chat's `blockId`).
*/
export function applyToolCallPhase<T extends AgentStreamToolCall>(
map: Map<string, T>,
order: string[],
event: { key: string; id: string; name: string; phase: 'start' | 'end'; status?: string },
extend: (tool: AgentStreamToolCall) => T
): void {
const { key } = event
if (event.phase === 'start') {
if (!map.has(key)) {
order.push(key)
}
map.set(
key,
extend({
key,
id: event.id,
name: event.name,
displayName: humanizeToolName(event.name),
status: 'running',
})
)
return
}
const endStatus = resolveToolCallEndStatus(event.status)
const existing = map.get(key)
if (!existing) {
order.push(key)
map.set(
key,
extend({
key,
id: event.id,
name: event.name,
displayName: humanizeToolName(event.name),
status: endStatus,
})
)
return
}
map.set(key, { ...existing, status: endStatus })
}
/** Settles every still-running chip in the map to a terminal status. */
export function settleRunningToolCalls<T extends AgentStreamToolCall>(
map: Map<string, T>,
status: AgentStreamToolTerminalStatus
): void {
for (const [key, tool] of map) {
if (tool.status === 'running') {
map.set(key, { ...tool, status })
}
}
}
/** List variant of {@link settleRunningToolCalls} for immutable store entries. */
export function settleRunningToolCallList<T extends AgentStreamToolCall>(
toolCalls: T[] | undefined,
status: AgentStreamToolTerminalStatus
): T[] | undefined {
return toolCalls?.map((tool) => (tool.status === 'running' ? { ...tool, status } : tool))
}
/** Ordered snapshot of the map, or undefined when no chips exist. */
export function snapshotToolCalls<T extends AgentStreamToolCall>(
order: string[],
map: Map<string, T>
): T[] | undefined {
if (order.length === 0) return undefined
return order.map((key) => map.get(key)).filter((tool): tool is T => Boolean(tool))
}
/** True while any chip is still running. */
export function anyToolCallRunning<T extends AgentStreamToolCall>(map: Map<string, T>): boolean {
for (const tool of map.values()) {
if (tool.status === 'running') return true
}
return false
}
apps/sim/components/ui/shimmer-text.module.css
+4 -1
View File
@@ -21,8 +21,11 @@
}
@keyframes shimmer-sweep {
from {
background-position: 100% 0;
}
to {
background-position: 200% 0;
background-position: -100% 0;
}
}
apps/sim/components/ui/shimmer-text.tsx
+16 -4
View File
@@ -1,10 +1,12 @@
import type { ComponentPropsWithoutRef, ElementType } from 'react'
import { cn } from '@sim/emcn'
import styles from '@/components/ui/shimmer-text.module.css'
interface ShimmerTextProps {
type ShimmerTextProps<T extends ElementType = 'span'> = {
as?: T
children: React.ReactNode
className?: string
}
} & Omit<ComponentPropsWithoutRef<T>, 'as' | 'children' | 'className'>
/**
* Sweeping-highlight shimmer over a text phrase — the same treatment as the
@@ -12,6 +14,16 @@ interface ShimmerTextProps {
* Size and weight come from the consumer's className; the gradient replaces
* the text color, so color classes are ignored while shimmering.
*/
export function ShimmerText({ children, className }: ShimmerTextProps) {
return <span className={cn(styles.shimmer, className)}>{children}</span>
export function ShimmerText<T extends ElementType = 'span'>({
as,
children,
className,
...props
}: ShimmerTextProps<T>) {
const Comp = as ?? 'span'
return (
<Comp className={cn(styles.shimmer, className)} {...props}>
{children}
</Comp>
)
}
apps/sim/executor/execution/block-executor.test.ts
+366
View File
@@ -26,6 +26,14 @@ vi.mock('@/lib/uploads', () => ({
},
}))
vi.mock('@/lib/logs/execution/pii-redaction', async (importOriginal) => {
const actual = await importOriginal<typeof import('@/lib/logs/execution/pii-redaction')>()
return {
...actual,
redactObjectStrings: vi.fn(actual.redactObjectStrings),
}
})
function createBlock(): SerializedBlock {
return {
id: 'function-block-1',
@@ -379,4 +387,362 @@ describe('BlockExecutor', () => {
)
expect(state.getBlockOutput(block.id)).toEqual(output)
})
it('does not soft-succeed non-agent blocks on user AbortError', async () => {
const block = createBlock()
const workflow: SerializedWorkflow = {
version: '1',
blocks: [block],
connections: [],
loops: {},
parallels: {},
}
const state = new ExecutionState()
const resolver = new VariableResolver(workflow, {}, state)
const abortController = new AbortController()
const handler: BlockHandler = {
canHandle: () => true,
execute: async () => {
abortController.abort('user')
throw new DOMException('The operation was aborted.', 'AbortError')
},
}
const executor = new BlockExecutor(
[handler],
resolver,
{
workspaceId: 'workspace-1',
executionId: 'execution-1',
userId: 'user-1',
metadata: {
requestId: 'request-1',
executionId: 'execution-1',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
userId: 'user-1',
triggerType: 'manual',
useDraftState: false,
startTime: new Date().toISOString(),
},
},
state
)
const ctx = createContext(state)
ctx.abortSignal = abortController.signal
await expect(executor.execute(ctx, createNode(block), block)).rejects.toThrow(/abort/i)
const output = state.getBlockOutput(block.id)
expect(output?.error).toBeTruthy()
expect(output).not.toEqual({ content: '' })
})
})
describe('BlockExecutor streaming pump', () => {
function createAgentBlock(): SerializedBlock {
return {
id: 'agent-block-1',
metadata: { id: BlockType.AGENT, name: 'Agent' },
position: { x: 0, y: 0 },
config: { tool: BlockType.AGENT, params: {} },
inputs: {},
outputs: {},
enabled: true,
}
}
function createExecutor(handler: BlockHandler) {
const block = createAgentBlock()
const workflow: SerializedWorkflow = {
version: '1',
blocks: [block],
connections: [],
loops: {},
parallels: {},
}
const state = new ExecutionState()
const resolver = new VariableResolver(workflow, {}, state)
const executor = new BlockExecutor(
[handler],
resolver,
{
workspaceId: 'workspace-1',
executionId: 'execution-1',
userId: 'user-1',
metadata: {
requestId: 'request-1',
executionId: 'execution-1',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
userId: 'user-1',
triggerType: 'manual',
useDraftState: false,
startTime: new Date().toISOString(),
},
},
state
)
return { executor, block, state }
}
function createAgentEventsStreamingHandler(options: {
events: Array<Record<string, unknown>>
attachThinkingOnDrain?: string
failAfterText?: string
onFullContent?: (content: string) => void | Promise<void>
}): BlockHandler {
return {
canHandle: () => true,
execute: async () => {
const timeSegment: Record<string, unknown> = {
type: 'model',
name: 'claude-test',
startTime: Date.now(),
endTime: Date.now(),
duration: 1,
}
const output = {
content: '',
model: 'claude-test',
tokens: { input: 1, output: 2, total: 3 },
providerTiming: {
startTime: new Date().toISOString(),
endTime: new Date().toISOString(),
duration: 1,
timeSegments: [timeSegment],
},
cost: { input: 0, output: 0, total: 0 },
}
const stream = new ReadableStream({
start(controller) {
if (options.failAfterText) {
controller.enqueue({
type: 'text_delta',
text: options.failAfterText,
turn: 'final',
})
controller.error(new Error('provider reset'))
return
}
for (const event of options.events) {
controller.enqueue(event)
}
if (options.attachThinkingOnDrain) {
timeSegment.thinkingContent = options.attachThinkingOnDrain
}
controller.close()
},
})
return {
stream,
streamFormat: 'agent-events-v1' as const,
execution: {
success: true,
output,
logs: [],
metadata: {
startTime: new Date().toISOString(),
endTime: new Date().toISOString(),
duration: 1,
},
},
onFullContent: options.onFullContent,
}
},
}
}
it('projects answer text to onStream and content; sink gets full timeline', async () => {
const onFullContent = vi.fn()
const handler = createAgentEventsStreamingHandler({
events: [
{ type: 'thinking_delta', text: 'hmm ' },
{ type: 'thinking_delta', text: 'yes' },
{ type: 'text_delta', text: 'Hello ', turn: 'final' },
{ type: 'text_delta', text: 'world', turn: 'final' },
],
attachThinkingOnDrain: 'hmm yes',
onFullContent,
})
const { executor, block, state } = createExecutor(handler)
const ctx = createContext(state)
const forwarded: string[] = []
const sinkEvents: Array<Record<string, unknown>> = []
ctx.onStream = async (streamingExec) => {
expect(streamingExec.streamFormat).toBe('text')
streamingExec.subscribe?.({
onEvent: async (event) => {
sinkEvents.push(event as Record<string, unknown>)
},
})
const reader = streamingExec.stream.getReader()
const decoder = new TextDecoder()
while (true) {
const { done, value } = await reader.read()
if (done) break
forwarded.push(decoder.decode(value, { stream: true }))
}
}
await executor.execute(ctx, createNode(block), block)
expect(forwarded.join('')).toBe('Hello world')
expect(state.getBlockOutput(block.id)?.content).toBe('Hello world')
expect(onFullContent).toHaveBeenCalledWith('Hello world')
expect(sinkEvents).toEqual([
{ type: 'thinking_delta', text: 'hmm ' },
{ type: 'thinking_delta', text: 'yes' },
{ type: 'text_delta', text: 'Hello ', turn: 'final' },
{ type: 'text_delta', text: 'world', turn: 'final' },
])
expect(state.getBlockOutput(block.id)?.providerTiming?.timeSegments?.[0]?.thinkingContent).toBe(
'hmm yes'
)
})
it('drains without onStream and still persists answer content', async () => {
const handler = createAgentEventsStreamingHandler({
events: [{ type: 'text_delta', text: 'offline answer', turn: 'final' }],
})
const { executor, block, state } = createExecutor(handler)
const ctx = createContext(state)
await executor.execute(ctx, createNode(block), block)
expect(state.getBlockOutput(block.id)?.content).toBe('offline answer')
})
it('throws on mid-stream provider error (no truncated success)', async () => {
const handler = createAgentEventsStreamingHandler({
failAfterText: 'partial',
})
const { executor, block, state } = createExecutor(handler)
const ctx = createContext(state)
ctx.onStream = async (streamingExec) => {
const reader = streamingExec.stream.getReader()
try {
while (true) {
const { done } = await reader.read()
if (done) break
}
} catch {
// consumer may see the error; block must still fail
}
}
await expect(executor.execute(ctx, createNode(block), block)).rejects.toThrow('provider reset')
expect(state.getBlockOutput(block.id)?.content).not.toBe('partial')
})
it('soft-completes on user abort with drained answer text (no failed block)', async () => {
const abortController = new AbortController()
const handler = createAgentEventsStreamingHandler({
events: [
{ type: 'text_delta', text: 'partial answer', turn: 'final' },
{ type: 'thinking_delta', text: 'more' },
],
})
const { executor, block, state } = createExecutor(handler)
const ctx = createContext(state)
ctx.abortSignal = abortController.signal
ctx.onStream = async (streamingExec) => {
streamingExec.subscribe?.({ onEvent: async () => {} })
const reader = streamingExec.stream.getReader()
try {
// Drain the first projected answer chunk, then Stop — pump must keep it.
const first = await reader.read()
expect(first.done).toBe(false)
abortController.abort('user')
while (true) {
const { done } = await reader.read()
if (done) break
}
} catch {
// abort may cancel the text stream
}
}
await executor.execute(ctx, createNode(block), block)
const output = state.getBlockOutput(block.id)
expect(output?.error).toBeUndefined()
// Soft-complete must keep text already projected before Stop — not empty content.
expect(output?.content).toBe('partial answer')
expect(output).not.toMatchObject({ error: expect.any(String) })
})
it('fails on timeout but keeps drained answer text in block output', async () => {
const abortController = new AbortController()
const handler = createAgentEventsStreamingHandler({
events: [
{ type: 'text_delta', text: 'partial before timeout', turn: 'final' },
{ type: 'thinking_delta', text: 'more' },
],
})
const { executor, block, state } = createExecutor(handler)
const ctx = createContext(state)
ctx.abortSignal = abortController.signal
ctx.onStream = async (streamingExec) => {
streamingExec.subscribe?.({ onEvent: async () => {} })
const reader = streamingExec.stream.getReader()
try {
const first = await reader.read()
expect(first.done).toBe(false)
abortController.abort('timeout')
while (true) {
const { done } = await reader.read()
if (done) break
}
} catch {
// timeout may cancel the text stream
}
}
await expect(executor.execute(ctx, createNode(block), block)).rejects.toThrow(/timed out/i)
const output = state.getBlockOutput(block.id)
expect(output?.error).toBeTruthy()
expect(output?.content).toBe('partial before timeout')
})
it('with PII redaction: no live forward and strips thinking from traces', async () => {
const { redactObjectStrings } = await import('@/lib/logs/execution/pii-redaction')
vi.mocked(redactObjectStrings).mockImplementation(async (value) => {
if (typeof value === 'string') {
return `[masked]${value}` as never
}
// Object walk is exercised elsewhere; keep streaming-stage string mask as-is.
return value as never
})
const handler = createAgentEventsStreamingHandler({
events: [
{ type: 'thinking_delta', text: 'secret thought' },
{ type: 'text_delta', text: 'alice@example.com said hi', turn: 'final' },
],
attachThinkingOnDrain: 'secret thought',
})
const { executor, block, state } = createExecutor(handler)
const ctx = createContext(state)
const onStream = vi.fn()
ctx.onStream = onStream
ctx.piiBlockOutputRedaction = {
enabled: true,
entityTypes: ['EMAIL_ADDRESS'],
language: 'en',
}
await executor.execute(ctx, createNode(block), block)
expect(onStream).not.toHaveBeenCalled()
expect(state.getBlockOutput(block.id)?.content).toBe('[masked]alice@example.com said hi')
expect(
state.getBlockOutput(block.id)?.providerTiming?.timeSegments?.[0]?.thinkingContent
).toBeUndefined()
})
})
apps/sim/executor/execution/block-executor.ts
+173 -91
View File
@@ -1,5 +1,6 @@
import { createLogger, type Logger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { isTimeoutAbortReason } from '@/lib/core/execution-limits/types'
import { redactApiKeys } from '@/lib/core/security/redaction'
import { normalizeStringArray } from '@/lib/core/utils/arrays'
import { getBaseUrl } from '@/lib/core/utils/urls'
@@ -59,6 +60,7 @@ import {
FUNCTION_BLOCK_DISPLAY_CODE_KEY,
type VariableResolver,
} from '@/executor/variables/resolver'
import { createAgentStreamPump } from '@/providers/stream-pump'
import type { SerializedBlock } from '@/serializer/types'
import { SYSTEM_SUBBLOCK_IDS } from '@/triggers/constants'
@@ -176,6 +178,7 @@ export class BlockExecutor {
}
cleanupSelfReference?.()
let streamingPartialOutput: Record<string, any> | undefined
try {
const output = handler.executeWithNode
? await handler.executeWithNode(ctx, block, resolvedInputs, nodeMetadata)
@@ -188,21 +191,24 @@ export class BlockExecutor {
if (isStreamingExecution) {
const streamingExec = output as StreamingExecution
// The stream must still be drained to populate `execution.output`, but
// forwarding raw chunks to the client (or persisting them to memory)
// before redaction would leak PII. When block-output redaction is on we
// drain in buffer-only mode (no `onStream`, content masked before it's
// stored); the masked final output reaches the client via block-complete.
if (ctx.onStream) {
// Always drain via the agent stream pump (tokens/cost/timing callbacks),
// even with no `onStream`. When block-output redaction is on we do not
// live-forward chunks; content is masked before persist and the masked
// final output reaches the client via block-complete.
try {
await this.handleStreamingExecution(
ctx,
node,
block,
streamingExec,
resolvedInputs,
normalizeStringArray(ctx.selectedOutputs),
!ctx.piiBlockOutputRedaction?.enabled
normalizeStringArray(ctx.selectedOutputs)
)
} catch (streamError) {
// Timeout / drain failures may still have projected answer text — keep it
// for the failed block output so logs match what the client already saw.
streamingPartialOutput = streamingExec.execution?.output
throw streamError
}
normalizedOutput = this.normalizeOutput(
@@ -315,7 +321,8 @@ export class BlockExecutor {
blockLog,
inputsForLog,
isSentinel,
'execution'
'execution',
streamingPartialOutput
)
}
}
@@ -372,7 +379,8 @@ export class BlockExecutor {
blockLog: BlockLog | undefined,
inputsForLog: Record<string, any>,
isSentinel: boolean,
phase: 'input_resolution' | 'execution'
phase: 'input_resolution' | 'execution',
streamingPartialOutput?: Record<string, any>
): Promise<NormalizedBlockOutput> {
const endedAt = new Date().toISOString()
const duration = performance.now() - startTime
@@ -383,10 +391,65 @@ export class BlockExecutor {
? inputsForLog
: ((block.config?.params as Record<string, any> | undefined) ?? {})
// Routine user Stop on Agent streams: don't paint a failed agent block
// (workflow is already cancelled). Timeouts abort with reason `'timeout'`.
// Non-agent blocks (HTTP, Function, etc.) still fail normally on AbortError
// so logs don't show a green empty success.
const isAbort =
(error instanceof DOMException && error.name === 'AbortError') ||
(error instanceof Error && error.name === 'AbortError')
const isTimeout = isTimeoutAbortReason(ctx.abortSignal?.reason)
const isAgentBlock = block.metadata?.id === BlockType.AGENT
if (isAbort && !isTimeout && ctx.abortSignal?.aborted && isAgentBlock) {
const softOutput: NormalizedBlockOutput = {
content: '',
}
this.setNodeOutput(node, softOutput, duration)
if (blockLog) {
blockLog.endedAt = endedAt
blockLog.durationMs = duration
blockLog.success = true
blockLog.error = undefined
blockLog.input = this.sanitizeInputsForLog(input, block.metadata?.id)
blockLog.output = filterOutputForLog(block.metadata?.id || '', softOutput, { block })
}
this.execLogger.info('Block stream aborted by client; soft-completing', {
blockId: node.id,
blockType: block.metadata?.id,
})
if (!isSentinel && blockLog) {
this.fireBlockCompleteCallback(
blockStartPromise,
ctx,
node,
block,
this.sanitizeInputsForLog(input, block.metadata?.id),
filterOutputForLog(block.metadata?.id || '', softOutput, { block }),
duration,
blockLog.startedAt,
blockLog.executionOrder,
blockLog.endedAt
)
}
return softOutput
}
const errorOutput: NormalizedBlockOutput = {
error: errorMessage,
}
// Keep any answer text already drained before timeout/failure so logs match
// what was projected to the client.
const partialContent = streamingPartialOutput?.content
if (typeof partialContent === 'string' && partialContent) {
errorOutput.content = partialContent
}
if (ChildWorkflowError.isChildWorkflowError(error)) {
errorOutput.childWorkflowName = error.childWorkflowName
if (error.childWorkflowSnapshotId) {
@@ -774,119 +837,125 @@ export class BlockExecutor {
block: SerializedBlock,
streamingExec: StreamingExecution,
resolvedInputs: Record<string, any>,
selectedOutputs: string[],
forwardToClient = true
selectedOutputs: string[]
): Promise<void> {
const blockId = node.id
const piiEnabled = Boolean(ctx.piiBlockOutputRedaction?.enabled)
// Live-forward only when a client stream exists and PII redaction is off.
const forwardToClient = Boolean(ctx.onStream) && !piiEnabled
const responseFormat =
resolvedInputs?.responseFormat ??
(block.config?.params as Record<string, any> | undefined)?.responseFormat ??
(block.config as Record<string, any> | undefined)?.responseFormat
const sourceReader = streamingExec.stream.getReader()
const decoder = new TextDecoder()
const accumulated: string[] = []
let drainError: unknown
let sourceFullyDrained = false
const streamFormat = streamingExec.streamFormat ?? 'text'
const pump = createAgentStreamPump({
source: streamingExec.stream,
streamFormat,
// No live consumer → sink-mode so we never buffer into an unread text stream.
sinkMode: !forwardToClient,
abortSignal: ctx.abortSignal,
})
if (forwardToClient) {
const clientSource = new ReadableStream<Uint8Array>({
async pull(controller) {
try {
const { done, value } = await sourceReader.read()
if (done) {
const tail = decoder.decode()
if (tail) accumulated.push(tail)
sourceFullyDrained = true
controller.close()
return
}
accumulated.push(decoder.decode(value, { stream: true }))
controller.enqueue(value)
} catch (error) {
drainError = error
controller.error(error)
}
},
async cancel(reason) {
try {
await sourceReader.cancel(reason)
} catch {}
},
})
let onStreamPromise: Promise<void> | undefined
let processedClientStream: ReadableStream<Uint8Array> | undefined
const processedClientStream = streamingResponseFormatProcessor.processStream(
clientSource,
if (forwardToClient && ctx.onStream && pump.textStream) {
processedClientStream = streamingResponseFormatProcessor.processStream(
pump.textStream,
blockId,
selectedOutputs,
responseFormat
)
try {
await ctx.onStream?.({
// Start onStream without awaiting so a sync `subscribe(sink)` can run before
// the first provider pull, then read the projected text stream concurrently
// with `pump.run()`.
onStreamPromise = ctx
.onStream({
...streamingExec,
stream: processedClientStream,
execution: streamingExec.execution,
streamFormat: 'text',
subscribe: pump.subscribe,
// processStream returns the input stream identity when no
// response-format extraction applies.
clientStreamTransformed: processedClientStream !== pump.textStream,
})
.catch(async (error) => {
this.execLogger.error('Error in onStream callback', { blockId, error })
await processedClientStream?.cancel().catch(() => {})
})
} catch (error) {
this.execLogger.error('Error in onStream callback', { blockId, error })
await processedClientStream.cancel().catch(() => {})
} finally {
try {
sourceReader.releaseLock()
} catch {}
}
} else {
// Buffer-only drain: consume the source so `execution.output` is complete,
// but never forward raw chunks to the client (block-output redaction is on).
try {
while (true) {
const { done, value } = await sourceReader.read()
if (done) {
const tail = decoder.decode()
if (tail) accumulated.push(tail)
sourceFullyDrained = true
break
}
accumulated.push(decoder.decode(value, { stream: true }))
}
} catch (error) {
drainError = error
} finally {
try {
sourceReader.releaseLock()
} catch {}
}
}
if (drainError) {
this.execLogger.error('Error reading stream for block', { blockId, error: drainError })
let pumpResult
try {
pumpResult = await pump.run()
} catch (error) {
this.execLogger.error('Error reading stream for block', { blockId, error })
if (onStreamPromise) {
await onStreamPromise.catch(() => {})
}
throw error instanceof Error ? error : new Error(String(error))
}
if (onStreamPromise) {
await onStreamPromise
}
// Timeout still fails the block, but keep any drained answer text so logs
// match what was already projected to the client before the deadline.
// User Stop soft-completes below so logs don't show a scary red agent block
// for a routine cancel (workflow status remains `cancelled` via abort).
if (pumpResult.cancelled && pumpResult.cancelReason === 'timeout') {
const truncated = pumpResult.answerText
if (truncated && streamingExec.execution?.output) {
streamingExec.execution.output.content = truncated
}
this.execLogger.warn('Stream timed out; persisting drained answer before failing block', {
blockId,
hasContent: Boolean(truncated),
})
throw new DOMException('Provider request timed out', 'AbortError')
}
// Provider onComplete may have attached thinking to timing segments during drain.
// Under PII redaction, never retain raw thinking in traces.
if (piiEnabled) {
stripThinkingContentFromOutput(streamingExec.execution?.output)
}
// User/unknown cancel: persist truncated answer when present, then return.
if (pumpResult.cancelled) {
const truncated = pumpResult.answerText
if (truncated && streamingExec.execution?.output) {
streamingExec.execution.output.content = truncated
}
this.execLogger.info('Stream cancelled by client; soft-completing agent block', {
blockId,
cancelReason: pumpResult.cancelReason,
hasContent: Boolean(truncated),
})
return
}
// If the onStream consumer exited before the source drained (e.g. it caught
// an internal error and returned normally), `accumulated` holds a truncated
// response. Persisting that to memory or setting it as the block output
// would corrupt downstream state — skip and log instead.
if (!sourceFullyDrained) {
// If the pump did not fully drain (should be rare when not cancelled), skip
// persistence of potentially truncated answer text.
if (!pumpResult.fullyDrained) {
this.execLogger.warn(
'Stream consumer exited before source drained; skipping content persistence',
{
blockId,
}
{ blockId }
)
return
}
let fullContent = accumulated.join('')
let fullContent = pumpResult.answerText
if (!fullContent) {
return
}
if (!forwardToClient && ctx.piiBlockOutputRedaction?.enabled) {
// Mask before the content is written to `execution.output` or persisted to
// memory via `onFullContent`, so the streamed agent response can't leak PII
// through either path. The block-output redaction below is then idempotent.
if (piiEnabled && ctx.piiBlockOutputRedaction) {
// Mask before writing to `execution.output` or `onFullContent`.
fullContent = await redactObjectStrings(fullContent, {
entityTypes: ctx.piiBlockOutputRedaction.entityTypes,
language: ctx.piiBlockOutputRedaction.language,
@@ -931,3 +1000,16 @@ export class BlockExecutor {
}
}
}
/** Removes retained thinking from provider timing segments (PII safe default). */
function stripThinkingContentFromOutput(output: unknown): void {
if (!output || typeof output !== 'object') return
const providerTiming = (output as { providerTiming?: { timeSegments?: unknown } }).providerTiming
const segments = providerTiming?.timeSegments
if (!Array.isArray(segments)) return
for (const segment of segments) {
if (segment && typeof segment === 'object' && 'thinkingContent' in segment) {
;(segment as { thinkingContent?: string }).thinkingContent = undefined
}
}
}
apps/sim/executor/execution/snapshot-serializer.test.ts
+23
View File
@@ -189,4 +189,27 @@ describe('serializePauseSnapshot', () => {
expect(serialized.metadata.billingAttribution).toEqual(billingAttribution)
})
it('preserves includeThinking on pause so chat resume can emit thinking SSE', () => {
const context = createContext({
metadata: {
...createContext().metadata,
includeThinking: true,
executionMode: 'stream',
},
})
const snapshot = serializePauseSnapshot(context, ['next-block'])
const serialized = JSON.parse(snapshot.snapshot)
expect(serialized.metadata.includeThinking).toBe(true)
expect(serialized.metadata.executionMode).toBe('stream')
})
it('omits includeThinking when the live run did not enable it', () => {
const snapshot = serializePauseSnapshot(createContext(), ['next-block'])
const serialized = JSON.parse(snapshot.snapshot)
expect(serialized.metadata.includeThinking).toBeUndefined()
})
})
apps/sim/executor/execution/snapshot-serializer.ts
+4
View File
@@ -252,6 +252,10 @@ export function serializePauseSnapshot(
startTime: metadataFromContext?.startTime ?? new Date().toISOString(),
isClientSession: metadataFromContext?.isClientSession,
executionMode: metadataFromContext?.executionMode,
// Preserve deployed-chat thinking gate across HITL pause/resume.
includeThinking: metadataFromContext?.includeThinking === true ? true : undefined,
// Preserve the run-level agent-events opt-in across HITL pause/resume.
agentEvents: metadataFromContext?.agentEvents === true ? true : undefined,
}
const snapshot = new ExecutionSnapshot(
apps/sim/executor/execution/types.ts
+12
View File
@@ -47,6 +47,18 @@ export interface ExecutionMetadata {
callChain?: string[]
correlation?: AsyncExecutionCorrelation
executionMode?: 'sync' | 'stream' | 'async'
/**
* Deployed-chat thinking policy half of the SSE dual gate. Persisted so HITL
* resume can re-enable thinking frames without hardcoding false.
*/
includeThinking?: boolean
/**
* Run-level agent-events opt-in. True only on surfaces that consume thinking
* and tool lifecycle events (canvas Run, dual-gated public chat). Enables the
* live streaming tool loops and provider thinking-summary requests; when
* unset, providers behave exactly as they did before agent events existed.
*/
agentEvents?: boolean
}
export interface SerializableExecutionState {
apps/sim/executor/handlers/agent/agent-handler.test.ts
+80
View File
@@ -1875,6 +1875,52 @@ describe('AgentBlockHandler', () => {
expect(providerCallArgs.billingAttribution).toEqual(billingAttribution)
})
it('forwards agentEvents to executeProviderRequest on opted-in streaming runs', async () => {
const inputs = {
model: 'gpt-4o',
userPrompt: 'Stream this',
apiKey: 'test-api-key',
}
const streamingContext = {
...mockContext,
stream: true,
selectedOutputs: ['test-agent-block'],
metadata: { ...mockContext.metadata, agentEvents: true },
} as ExecutionContext
mockGetProviderFromModel.mockReturnValue('openai')
await handler.execute(streamingContext, mockBlock, inputs)
expect(mockExecuteProviderRequest).toHaveBeenCalled()
const providerCallArgs = mockExecuteProviderRequest.mock.calls[0][1]
expect(providerCallArgs.stream).toBe(true)
expect(providerCallArgs.agentEvents).toBe(true)
})
it('does not set agentEvents on runs without the run-level opt-in', async () => {
const inputs = {
model: 'gpt-4o',
userPrompt: 'Stream this',
apiKey: 'test-api-key',
}
const streamingContext = {
...mockContext,
stream: true,
selectedOutputs: ['test-agent-block'],
} as ExecutionContext
mockGetProviderFromModel.mockReturnValue('openai')
await handler.execute(streamingContext, mockBlock, inputs)
expect(mockExecuteProviderRequest).toHaveBeenCalled()
const providerCallArgs = mockExecuteProviderRequest.mock.calls[0][1]
expect(providerCallArgs.agentEvents).toBe(false)
})
it('should handle multiple MCP tools from the same server efficiently', async () => {
const fetchCalls: any[] = []
@@ -2290,4 +2336,38 @@ describe('AgentBlockHandler', () => {
})
})
})
describe('wrapStreamForMemoryPersistence envelope', () => {
it('preserves streamFormat and subscribe via object spread', () => {
const handler = new AgentBlockHandler()
const subscribe = vi.fn()
const streamingExec: StreamingExecution = {
stream: new ReadableStream(),
streamFormat: 'agent-events-v1',
subscribe,
execution: {
success: true,
output: { content: '' },
logs: [],
metadata: { startTime: '', endTime: '', duration: 0 },
},
}
const wrapped = (
handler as unknown as {
wrapStreamForMemoryPersistence: (
ctx: ExecutionContext,
inputs: Record<string, unknown>,
exec: StreamingExecution
) => StreamingExecution
}
).wrapStreamForMemoryPersistence({} as ExecutionContext, {}, streamingExec)
expect(wrapped.streamFormat).toBe('agent-events-v1')
expect(wrapped.subscribe).toBe(subscribe)
expect(wrapped.stream).toBe(streamingExec.stream)
expect(wrapped.execution).toBe(streamingExec.execution)
expect(typeof wrapped.onFullContent).toBe('function')
})
})
})
apps/sim/executor/handlers/agent/agent-handler.ts
+16 -2
View File
@@ -46,6 +46,7 @@ import {
shouldUseLargeFilePath,
supportsFileAttachments,
} from '@/providers/attachments'
import { supportsStreamingToolCalls } from '@/providers/streaming-tool-loop-shared'
import { getProviderFromModel, transformBlockTool } from '@/providers/utils'
import type { SerializedBlock } from '@/serializer/types'
import { filterSchemaForLLM, type ToolSchema } from '@/tools/params'
@@ -955,6 +956,18 @@ export class AgentBlockHandler implements BlockHandler {
verbosity: inputs.verbosity,
thinkingLevel: inputs.thinkingLevel,
previousInteractionId: inputs.previousInteractionId,
/**
* Agent-events opt-in and live tool lifecycle. Both are gated on the
* run-level {@link ExecutionMetadata.agentEvents} flag so runs without an
* agent-events consumer keep the exact pre-agent-events provider
* behavior (legacy loops, unchanged request payloads).
*/
agentEvents: streaming && ctx.metadata?.agentEvents === true,
streamToolCalls:
streaming &&
ctx.metadata?.agentEvents === true &&
formattedTools.length > 0 &&
supportsStreamingToolCalls(providerId),
}
}
@@ -1029,6 +1042,8 @@ export class AgentBlockHandler implements BlockHandler {
verbosity: providerRequest.verbosity,
thinkingLevel: providerRequest.thinkingLevel,
previousInteractionId: providerRequest.previousInteractionId,
agentEvents: providerRequest.agentEvents,
streamToolCalls: providerRequest.streamToolCalls,
abortSignal: ctx.abortSignal,
})
@@ -1088,8 +1103,7 @@ export class AgentBlockHandler implements BlockHandler {
streamingExec: StreamingExecution
): StreamingExecution {
return {
stream: streamingExec.stream,
execution: streamingExec.execution,
...streamingExec,
onFullContent: async (content: string) => {
if (!content.trim()) return
try {
apps/sim/executor/types.ts
+31
View File
@@ -10,6 +10,7 @@ import type {
SerializableExecutionState,
} from '@/executor/execution/types'
import type { RunFromBlockContext } from '@/executor/utils/run-from-block'
import type { AgentStreamSink, UnsubscribeAgentStreamSink } from '@/providers/stream-events'
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
import type { SubflowType } from '@/stores/workflows/workflow/types'
@@ -314,6 +315,11 @@ interface ExecutionMetadata {
resumeFromSnapshot?: boolean
resumeTerminalNoop?: boolean
executionMode?: 'sync' | 'stream' | 'async'
/**
* Run-level agent-events opt-in (see the snapshot ExecutionMetadata).
* Gates streaming tool loops and provider thinking-summary requests.
*/
agentEvents?: boolean
}
export interface BlockState {
@@ -522,7 +528,32 @@ export interface ExecutionResult {
}
export interface StreamingExecution {
/**
* Provider stream payload. Format is declared by {@link streamFormat}:
* - `'text'` (default): UTF-8 answer bytes (`ReadableStream<Uint8Array>`)
* - `'agent-events-v1'`: in-process `ReadableStream` of `AgentStreamEvent` objects
*
* Never sniff the payload; always read {@link streamFormat}.
* After the executor pump, {@link stream} is always projected UTF-8 answer text.
*/
stream: ReadableStream
/**
* Discriminator for {@link stream}. Defaults to `'text'` when omitted so
* existing providers remain byte-stream consumers without changes.
*/
streamFormat?: 'text' | 'agent-events-v1'
/**
* Optional sink subscription installed synchronously during `onStream` before
* the executor pump starts draining. Late subscribers receive future events only.
*/
subscribe?: (sink: AgentStreamSink) => UnsubscribeAgentStreamSink
/**
* True when {@link stream} is a response-format projection (selected JSON
* fields extracted from structured output) rather than raw answer text. Sink
* `text_delta` events then do NOT match the byte stream, so consumers must
* keep sourcing answer text from {@link stream} instead of the sink.
*/
clientStreamTransformed?: boolean
execution: ExecutionResult & { isStreaming?: boolean }
/**
* Invoked with the assembled response text after the stream drains. Lets agent
apps/sim/hooks/queries/chats.ts
+3
View File
@@ -175,6 +175,8 @@ export interface ChatFormData {
emails: string[]
welcomeMessage: string
selectedOutputBlocks: string[]
/** When true, thinking may be streamed to clients that opt into agent-events-v1. Default false. */
includeThinking: boolean
}
/**
@@ -263,6 +265,7 @@ function buildChatPayload(
allowedEmails:
formData.authType === 'email' || formData.authType === 'sso' ? formData.emails : [],
outputConfigs,
includeThinking: formData.includeThinking,
}
}
apps/sim/hooks/use-execution-stream.test.ts
+55
View File
@@ -53,6 +53,61 @@ describe('processSSEStream', () => {
expect(order).toEqual(['handler:start', 'handler:end', 'event-id'])
})
it('routes stream:thinking and stream:tool without requiring event ids', async () => {
const onStreamThinking = vi.fn()
const onStreamTool = vi.fn()
const onStreamChunk = vi.fn()
const onEventId = vi.fn()
const events: ExecutionEvent[] = [
{
type: 'stream:thinking',
timestamp: new Date().toISOString(),
executionId: 'exec-1',
workflowId: 'wf-1',
data: { blockId: 'agent-1', text: 'reasoning ' },
},
{
type: 'stream:tool',
timestamp: new Date().toISOString(),
executionId: 'exec-1',
workflowId: 'wf-1',
data: {
blockId: 'agent-1',
phase: 'start',
id: 'tool_1',
name: 'http_request',
},
},
{
type: 'stream:chunk',
timestamp: new Date().toISOString(),
executionId: 'exec-1',
workflowId: 'wf-1',
data: { blockId: 'agent-1', chunk: 'answer' },
},
]
await processSSEStream(
streamEvents(events).getReader(),
{ onStreamThinking, onStreamTool, onStreamChunk, onEventId },
'test'
)
expect(onStreamThinking).toHaveBeenCalledWith({
blockId: 'agent-1',
text: 'reasoning ',
})
expect(onStreamTool).toHaveBeenCalledWith({
blockId: 'agent-1',
phase: 'start',
id: 'tool_1',
name: 'http_request',
})
expect(onStreamChunk).toHaveBeenCalledWith({ blockId: 'agent-1', chunk: 'answer' })
expect(onEventId).not.toHaveBeenCalled()
})
it('propagates callback failures without acknowledging the event id', async () => {
const event: ExecutionEvent = {
type: 'block:started',
apps/sim/hooks/use-execution-stream.ts
+15
View File
@@ -14,7 +14,10 @@ import type {
ExecutionPausedData,
ExecutionStartedData,
StreamChunkData,
StreamChunkResetData,
StreamDoneData,
StreamThinkingData,
StreamToolData,
} from '@/lib/workflows/executor/execution-events'
import type { SerializableExecutionState } from '@/executor/execution/types'
@@ -121,9 +124,18 @@ export async function processSSEStream(
case 'stream:chunk':
await callbacks.onStreamChunk?.(event.data)
break
case 'stream:chunk_reset':
await callbacks.onStreamChunkReset?.(event.data)
break
case 'stream:done':
await callbacks.onStreamDone?.(event.data)
break
case 'stream:thinking':
await callbacks.onStreamThinking?.(event.data)
break
case 'stream:tool':
await callbacks.onStreamTool?.(event.data)
break
default:
logger.warn('Unknown event type:', (event as any).type)
}
@@ -164,7 +176,10 @@ export interface ExecutionStreamCallbacks {
onBlockError?: (data: BlockErrorData) => void | Promise<void>
onBlockChildWorkflowStarted?: (data: BlockChildWorkflowStartedData) => void | Promise<void>
onStreamChunk?: (data: StreamChunkData) => void | Promise<void>
onStreamChunkReset?: (data: StreamChunkResetData) => void | Promise<void>
onStreamDone?: (data: StreamDoneData) => void | Promise<void>
onStreamThinking?: (data: StreamThinkingData) => void | Promise<void>
onStreamTool?: (data: StreamToolData) => void | Promise<void>
onEventId?: (eventId: number) => void | Promise<void>
}
apps/sim/lib/api/contracts/chats.include-thinking.test.ts
+76
View File
@@ -0,0 +1,76 @@
/**
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import {
createChatBodySchema,
deployedChatConfigSchema,
updateChatBodySchema,
} from '@/lib/api/contracts/chats'
import { chatDetailSchema } from '@/lib/api/contracts/deployments'
describe('chat includeThinking contracts (Step 4)', () => {
it('create defaults includeThinking to false', () => {
const parsed = createChatBodySchema.parse({
workflowId: 'wf-1',
identifier: 'my-chat',
title: 'Support',
customizations: {
primaryColor: 'var(--brand-hover)',
welcomeMessage: 'Hi',
},
})
expect(parsed.includeThinking).toBe(false)
})
it('create accepts includeThinking true', () => {
const parsed = createChatBodySchema.parse({
workflowId: 'wf-1',
identifier: 'my-chat',
title: 'Support',
customizations: {
primaryColor: 'var(--brand-hover)',
welcomeMessage: 'Hi',
},
includeThinking: true,
})
expect(parsed.includeThinking).toBe(true)
})
it('update accepts includeThinking toggle', () => {
expect(updateChatBodySchema.parse({ includeThinking: true }).includeThinking).toBe(true)
expect(updateChatBodySchema.parse({ includeThinking: false }).includeThinking).toBe(false)
expect(updateChatBodySchema.parse({ title: 'x' }).includeThinking).toBeUndefined()
})
it('chat detail and deployed config expose includeThinking (default false)', () => {
const detail = chatDetailSchema.parse({
id: 'chat-1',
identifier: 'my-chat',
title: 'Support',
description: '',
authType: 'public',
allowedEmails: [],
outputConfigs: [],
isActive: true,
chatUrl: 'http://localhost/chat/my-chat',
hasPassword: false,
})
expect(detail.includeThinking).toBe(false)
const detailOn = chatDetailSchema.parse({
...detail,
includeThinking: true,
})
expect(detailOn.includeThinking).toBe(true)
const config = deployedChatConfigSchema.parse({
id: 'chat-1',
title: 'Support',
description: '',
customizations: {},
authType: 'public',
})
expect(config.includeThinking).toBe(false)
})
})
apps/sim/lib/api/contracts/chats.ts
+9 -2
View File
@@ -41,6 +41,8 @@ export const createChatBodySchema = z.object({
password: z.string().optional(),
allowedEmails: z.array(z.string()).optional().default([]),
outputConfigs: z.array(chatOutputConfigSchema).optional().default([]),
/** When true, clients may receive thinking SSE if they also send the protocol header. Default off. */
includeThinking: z.boolean().optional().default(false),
})
export type CreateChatBody = z.input<typeof createChatBodySchema>
@@ -58,6 +60,7 @@ export const updateChatBodySchema = z.object({
password: z.string().optional(),
allowedEmails: z.array(z.string()).optional(),
outputConfigs: z.array(chatOutputConfigSchema).optional(),
includeThinking: z.boolean().optional(),
})
export type UpdateChatBody = z.input<typeof updateChatBodySchema>
@@ -101,6 +104,8 @@ export const deployedChatConfigSchema = z.object({
(value) => value ?? undefined,
z.array(deployedChatOutputConfigSchema).optional()
),
/** Policy for thinking SSE; clients still need the X-Sim-Stream-Protocol opt-in. */
includeThinking: z.preprocess((value) => value ?? false, z.boolean()),
})
export type DeployedChatConfig = z.output<typeof deployedChatConfigSchema>
@@ -209,8 +214,10 @@ export const deployedChatPostContract = defineRouteContract({
params: chatIdentifierParamsSchema,
body: deployedChatPostBodySchema,
response: {
mode: 'json',
schema: deployedChatConfigSchema,
// Message posts return SSE (`text/event-stream`). Auth-only POSTs use
// authenticateDeployedChatContract (JSON). Terminal frames: `final` or one
// `error`, then `[DONE]`. Thinking frames require includeThinking + protocol header.
mode: 'stream',
},
})
apps/sim/lib/api/contracts/deployments.ts
+1
View File
@@ -176,6 +176,7 @@ export const chatDetailSchema = z.object({
})
)
),
includeThinking: z.preprocess((value) => value ?? false, z.boolean()),
customizations: z.preprocess(
(value) => value ?? undefined,
z
apps/sim/lib/copilot/tools/handlers/deployment/deploy.ts
+7
View File
@@ -308,6 +308,7 @@ export async function executeDeployChat(
allowedEmails: (existing[0].allowedEmails as string[]) || [],
outputConfigs:
(existing[0].outputConfigs as Array<{ blockId: string; path: string }>) || [],
includeThinking: existing[0].includeThinking ?? false,
welcomeMessage:
(existing[0].customizations as { welcomeMessage?: string } | null)
?.welcomeMessage || 'Hi there! How can I help you today?',
@@ -395,6 +396,10 @@ export async function executeDeployChat(
blockId: string
path: string
}>
const resolvedIncludeThinking =
typeof params.includeThinking === 'boolean'
? params.includeThinking
: (existingDeployment?.includeThinking ?? false)
const welcomeMessage =
typeof params.welcomeMessage === 'string'
? params.welcomeMessage
@@ -438,6 +443,7 @@ export async function executeDeployChat(
password: params.password,
allowedEmails: resolvedAllowedEmails,
outputConfigs: resolvedOutputConfigs,
includeThinking: resolvedIncludeThinking,
workspaceId: workflowRecord.workspaceId,
})
@@ -490,6 +496,7 @@ export async function executeDeployChat(
authType: resolvedAuthType,
allowedEmails: resolvedAllowedEmails,
outputConfigs: resolvedOutputConfigs,
includeThinking: resolvedIncludeThinking,
welcomeMessage: welcomeMessage || 'Hi there! How can I help you today?',
primaryColor:
params.customizations?.primaryColor ||
apps/sim/lib/copilot/tools/handlers/deployment/manage.ts
+2
View File
@@ -61,6 +61,7 @@ export async function executeCheckDeploymentStatus(
authType: chat.authType,
allowedEmails: chat.allowedEmails,
outputConfigs: chat.outputConfigs,
includeThinking: chat.includeThinking,
password: chat.password,
customizations: chat.customizations,
})
@@ -103,6 +104,7 @@ export async function executeCheckDeploymentStatus(
authType: chatDeploy[0]?.authType || null,
allowedEmails: chatDeploy[0]?.allowedEmails || null,
outputConfigs: chatDeploy[0]?.outputConfigs || null,
includeThinking: chatDeploy[0]?.includeThinking ?? false,
welcomeMessage: chatCustomizations.welcomeMessage || null,
primaryColor: chatCustomizations.primaryColor || null,
hasPassword: Boolean(chatDeploy[0]?.password),
apps/sim/lib/copilot/tools/handlers/param-types.ts
+1
View File
@@ -158,6 +158,7 @@ export interface DeployChatParams {
subdomain?: string
allowedEmails?: string[]
outputConfigs?: unknown[]
includeThinking?: boolean
}
export interface DeployMcpParams {
apps/sim/lib/core/execution-limits/types.test.ts
+32
View File
@@ -35,6 +35,7 @@ declare module '@/lib/core/execution-limits/types?execution-limits-test' {
import {
createTimeoutAbortController,
getExecutionTimeout,
isTimeoutAbortReason,
} from '@/lib/core/execution-limits/types?execution-limits-test'
afterAll(resetEnvFlagsMock)
@@ -80,4 +81,35 @@ describe('getExecutionTimeout', () => {
vi.useRealTimers()
}
})
it('aborts with an AbortError carrying the timeout reason when the timer fires', () => {
vi.useFakeTimers()
try {
const controller = createTimeoutAbortController(1000)
vi.advanceTimersByTime(1000)
expect(controller.signal.aborted).toBe(true)
expect(controller.isTimedOut()).toBe(true)
const reason = controller.signal.reason as DOMException
expect(reason).toBeInstanceOf(DOMException)
expect(reason.name).toBe('AbortError')
expect(reason.message).toBe('timeout')
expect(isTimeoutAbortReason(reason)).toBe(true)
controller.cleanup()
} finally {
vi.useRealTimers()
}
})
it('manual abort uses an AbortError carrying the user reason', () => {
const controller = createTimeoutAbortController(60_000)
controller.abort()
expect(controller.signal.aborted).toBe(true)
expect(controller.isTimedOut()).toBe(false)
const reason = controller.signal.reason as DOMException
expect(reason).toBeInstanceOf(DOMException)
expect(reason.name).toBe('AbortError')
expect(reason.message).toBe('user')
expect(isTimeoutAbortReason(reason)).toBe(false)
controller.cleanup()
})
})
apps/sim/lib/core/execution-limits/types.ts
+17 -2
View File
@@ -144,6 +144,19 @@ export interface TimeoutAbortController {
timeoutMs: number | undefined
}
/**
* True when an abort signal's reason marks an execution timeout. Abort reasons
* are `DOMException('timeout' | 'user', 'AbortError')` so code that passes the
* signal straight into `fetch` still sees a standard AbortError, while pumps
* and executors can discriminate timeout from user Stop via the message.
*/
export function isTimeoutAbortReason(reason: unknown): boolean {
if (reason === 'timeout') return true
return (
reason instanceof DOMException && reason.name === 'AbortError' && reason.message === 'timeout'
)
}
export function createTimeoutAbortController(timeoutMs?: number): TimeoutAbortController {
const abortController = new AbortController()
let isTimedOut = false
@@ -152,7 +165,8 @@ export function createTimeoutAbortController(timeoutMs?: number): TimeoutAbortCo
if (timeoutMs) {
timeoutId = setTimeout(() => {
isTimedOut = true
abortController.abort()
// AbortError with a typed message — see isTimeoutAbortReason.
abortController.abort(new DOMException('timeout', 'AbortError'))
}, timeoutMs)
}
@@ -162,7 +176,8 @@ export function createTimeoutAbortController(timeoutMs?: number): TimeoutAbortCo
cleanup: () => {
if (timeoutId) clearTimeout(timeoutId)
},
abort: () => abortController.abort(),
// Manual abort is user/client cancellation (disconnect, Stop, registerManualExecutionAborter).
abort: () => abortController.abort(new DOMException('user', 'AbortError')),
timeoutMs,
}
}
apps/sim/lib/workflows/executor/execute-workflow.ts
+9
View File
@@ -54,6 +54,13 @@ export interface ExecuteWorkflowOptions {
executionMode?: 'sync' | 'stream' | 'async'
/** Immutable actor/payer decision captured by preprocessing. */
billingAttribution?: BillingAttributionSnapshot
/** Deployed-chat thinking policy; persisted on the snapshot for resume. */
includeThinking?: boolean
/**
* Run-level agent-events opt-in (see {@link ExecutionMetadata.agentEvents}).
* Callers set this only when the surface consumes thinking/tool events.
*/
agentEvents?: boolean
}
export interface WorkflowInfo {
@@ -111,6 +118,8 @@ export async function executeWorkflow(
largeValueKeys: streamConfig?.largeValueKeys,
fileKeys: streamConfig?.fileKeys,
executionMode: streamConfig?.executionMode,
includeThinking: streamConfig?.includeThinking === true ? true : undefined,
agentEvents: streamConfig?.agentEvents === true ? true : undefined,
}
const snapshot = new ExecutionSnapshot(
apps/sim/lib/workflows/executor/execution-events.ts
+69
View File
@@ -17,7 +17,26 @@ export type ExecutionEventType =
| 'block:error'
| 'block:childWorkflowStarted'
| 'stream:chunk'
/** Live-only: clears a block's streamed answer text (intermediate turn). */
| 'stream:chunk_reset'
| 'stream:done'
/** Live-only agent thinking delta (not buffered for reconnect replay). */
| 'stream:thinking'
/** Live-only tool lifecycle (not buffered for reconnect replay). */
| 'stream:tool'
/**
* Event types that are live-only: forwarded to connected clients but excluded
* from reconnect replay buffers (same rule as answer chunks — guaranteed `seq`
* replay for stream events is out of scope).
*/
export const LIVE_ONLY_EXECUTION_EVENT_TYPES: ReadonlySet<ExecutionEventType> = new Set([
'stream:chunk',
'stream:chunk_reset',
'stream:done',
'stream:thinking',
'stream:tool',
])
/**
* Base event structure for SSE
@@ -208,6 +227,20 @@ interface StreamChunkEvent extends BaseExecutionEvent {
}
}
/**
* Live-only reconciliation for agent-events runs: the answer text streamed so
* far for `blockId` belonged to an intermediate turn (tool calls follow).
* Clients discard the block's accumulated streamed text; the final turn's
* text re-streams as regular `stream:chunk` events after tools settle.
*/
interface StreamChunkResetEvent extends BaseExecutionEvent {
type: 'stream:chunk_reset'
workflowId: string
data: {
blockId: string
}
}
/**
* Stream done event
*/
@@ -219,6 +252,36 @@ interface StreamDoneEvent extends BaseExecutionEvent {
}
}
/**
* Live thinking delta from an agent-events provider sink (canvas / draft runs).
* Builder runs show provider-exposed signals when the sink is attached
* (executor already disables the sink under PII redaction).
*/
interface StreamThinkingEvent extends BaseExecutionEvent {
type: 'stream:thinking'
workflowId: string
data: {
blockId: string
text: string
}
}
/**
* Live tool lifecycle from an agent-events provider sink.
* Name + status only — never args or results.
*/
interface StreamToolEvent extends BaseExecutionEvent {
type: 'stream:tool'
workflowId: string
data: {
blockId: string
phase: 'start' | 'end'
id: string
name: string
status?: 'success' | 'error' | 'cancelled'
}
}
/**
* Union type of all execution events
*/
@@ -233,7 +296,10 @@ export type ExecutionEvent =
| BlockErrorEvent
| BlockChildWorkflowStartedEvent
| StreamChunkEvent
| StreamChunkResetEvent
| StreamDoneEvent
| StreamThinkingEvent
| StreamToolEvent
export type ExecutionStartedData = ExecutionStartedEvent['data']
export type ExecutionCompletedData = ExecutionCompletedEvent['data']
@@ -245,7 +311,10 @@ export type BlockCompletedData = BlockCompletedEvent['data']
export type BlockErrorData = BlockErrorEvent['data']
export type BlockChildWorkflowStartedData = BlockChildWorkflowStartedEvent['data']
export type StreamChunkData = StreamChunkEvent['data']
export type StreamChunkResetData = StreamChunkResetEvent['data']
export type StreamDoneData = StreamDoneEvent['data']
export type StreamThinkingData = StreamThinkingEvent['data']
export type StreamToolData = StreamToolEvent['data']
/**
* Helper to create SSE formatted message
apps/sim/lib/workflows/executor/human-in-the-loop-manager.ts
+24 -2
View File
@@ -25,7 +25,10 @@ import { preprocessExecution } from '@/lib/execution/preprocessing'
import { LoggingSession } from '@/lib/logs/execution/logging-session'
import { cleanupExecutionBase64Cache } from '@/lib/uploads/utils/user-file-base64.server'
import { executeWorkflowCore } from '@/lib/workflows/executor/execution-core'
import type { ExecutionEvent } from '@/lib/workflows/executor/execution-events'
import {
type ExecutionEvent,
LIVE_ONLY_EXECUTION_EVENT_TYPES,
} from '@/lib/workflows/executor/execution-events'
import {
createPausedExecutionResumeMetadata,
parsePausedExecutionResumeMetadata,
@@ -35,6 +38,10 @@ import {
normalizeAutomaticResumeWaitingReason,
resolveAutomaticResumeAdmissionFailure,
} from '@/lib/workflows/executor/resume-policy'
import {
forwardAgentStreamToExecutionEvents,
shouldForwardAnswerTextFromSink,
} from '@/lib/workflows/streaming/forward-agent-stream-events'
import { ExecutionSnapshot } from '@/executor/execution/snapshot'
import type {
ChildWorkflowContext,
@@ -1276,7 +1283,7 @@ export class PauseResumeManager {
event: ExecutionEvent,
terminalStatus?: TerminalExecutionStreamStatus
) => {
const isBuffered = event.type !== 'stream:chunk' && event.type !== 'stream:done'
const isBuffered = !LIVE_ONLY_EXECUTION_EVENT_TYPES.has(event.type)
if (isBuffered) {
const entry = terminalStatus
? await eventWriter.writeTerminal(event, terminalStatus).catch((error) => {
@@ -1418,12 +1425,26 @@ export class PauseResumeManager {
? streamingExec.execution.blockId
: undefined
const blockId = typeof blockIdValue === 'string' ? blockIdValue : ''
// Live answer text rides the sink when available; the byte stream is
// then drained without re-emitting chunks (same final-turn content).
const answerTextFromSink = shouldForwardAnswerTextFromSink(streamingExec)
const unsubscribe = forwardAgentStreamToExecutionEvents(streamingExec, {
blockId,
executionId: resumeExecutionId,
workflowId,
sendEvent: writeBufferedEvent,
forwardAnswerText: answerTextFromSink,
})
const reader = streamingExec.stream.getReader()
const decoder = new TextDecoder()
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
if (answerTextFromSink) continue
const chunk = decoder.decode(value, { stream: true })
await writeBufferedEvent({
type: 'stream:chunk',
@@ -1447,6 +1468,7 @@ export class PauseResumeManager {
error: toError(streamError).message,
})
} finally {
unsubscribe()
try {
await reader.cancel().catch(() => {})
} catch {}
apps/sim/lib/workflows/orchestration/chat-deploy.ts
+5
View File
@@ -29,6 +29,8 @@ export interface ChatDeployPayload {
password?: string | null
allowedEmails?: string[]
outputConfigs?: Array<{ blockId: string; path: string }>
/** When true, public SSE may expose thinking if the client also opts into agent-events-v1. */
includeThinking?: boolean
workspaceId?: string | null
}
@@ -60,6 +62,7 @@ export async function performChatDeploy(
password,
allowedEmails = [],
outputConfigs = [],
includeThinking = false,
} = params
const customizations = {
@@ -141,6 +144,7 @@ export async function performChatDeploy(
password: passwordToStore,
allowedEmails: authType === 'email' || authType === 'sso' ? allowedEmails : [],
outputConfigs,
includeThinking,
updatedAt: new Date(),
})
.where(eq(chat.id, chatId))
@@ -159,6 +163,7 @@ export async function performChatDeploy(
password: encryptedPassword,
allowedEmails: authType === 'email' || authType === 'sso' ? allowedEmails : [],
outputConfigs,
includeThinking,
createdAt: new Date(),
updatedAt: new Date(),
})
apps/sim/lib/workflows/streaming/agent-stream-protocol.test.ts
+85
View File
@@ -0,0 +1,85 @@
/**
* @vitest-environment node
*/
import { describe, expect, it } from 'vitest'
import {
AGENT_STREAM_PROTOCOL_HEADER,
AGENT_STREAM_PROTOCOL_V1,
isChatChunkFrame,
isChatChunkResetFrame,
shouldEmitAgentStreamEvents,
} from '@/lib/workflows/streaming/agent-stream-protocol'
function headers(init?: Record<string, string>): Headers {
return new Headers(init)
}
describe('chunk_reset frame guard', () => {
it('identifies reset frames and keeps them out of the chunk guard', () => {
const reset = { blockId: 'agent-1', event: 'chunk_reset' }
expect(isChatChunkResetFrame(reset)).toBe(true)
// A reset must never be appended as answer text.
expect(isChatChunkFrame(reset)).toBe(false)
expect(isChatChunkResetFrame({ event: 'chunk_reset' })).toBe(false)
expect(isChatChunkResetFrame({ blockId: 'agent-1', chunk: 'text' })).toBe(false)
})
})
describe('shouldEmitAgentStreamEvents', () => {
it('defaults to false when policy is off and header is missing', () => {
expect(
shouldEmitAgentStreamEvents({
includeThinking: false,
requestHeaders: headers(),
})
).toBe(false)
expect(
shouldEmitAgentStreamEvents({
includeThinking: undefined,
requestHeaders: headers(),
})
).toBe(false)
})
it('requires both includeThinking and protocol header', () => {
expect(
shouldEmitAgentStreamEvents({
includeThinking: true,
requestHeaders: headers(),
})
).toBe(false)
expect(
shouldEmitAgentStreamEvents({
includeThinking: false,
requestHeaders: headers({ [AGENT_STREAM_PROTOCOL_HEADER]: AGENT_STREAM_PROTOCOL_V1 }),
})
).toBe(false)
expect(
shouldEmitAgentStreamEvents({
includeThinking: true,
requestHeaders: headers({ [AGENT_STREAM_PROTOCOL_HEADER]: AGENT_STREAM_PROTOCOL_V1 }),
})
).toBe(true)
})
it('accepts case-insensitive header values and comma lists', () => {
expect(
shouldEmitAgentStreamEvents({
includeThinking: true,
requestHeaders: headers({ [AGENT_STREAM_PROTOCOL_HEADER]: ' Agent-Events-V1 ' }),
})
).toBe(true)
expect(
shouldEmitAgentStreamEvents({
includeThinking: true,
requestHeaders: headers({
[AGENT_STREAM_PROTOCOL_HEADER]: 'text, agent-events-v1',
}),
})
).toBe(true)
})
})
apps/sim/lib/workflows/streaming/agent-stream-protocol.ts
+188
View File
@@ -0,0 +1,188 @@
/**
* Public agent stream protocol: header negotiation and the wire frame
* vocabulary for the public chat / simple SSE surface.
*
* Exposure rule (locked) for public chat / simple SSE:
* emit thinking/tool SSE frames iff
* deployment.includeThinking === true
* AND request opts into agent-events-v1 via {@link AGENT_STREAM_PROTOCOL_HEADER}
*
* Canvas draft runs (execution-events) forward the same sink as live-only
* `stream:thinking` / `stream:tool` events without the includeThinking gate;
* the executor still disables the sink when block-output PII redaction is on.
*
* Legacy clients omitting the header stay text-only even when the deployment
* has thinking enabled. Deployed chat UI always sends the header when loading
* its own deployment.
*
* See docs: workflows/deployment/agent-events.
*/
import type { ToolCallEndStatus } from '@/providers/stream-events'
export const AGENT_STREAM_PROTOCOL_HEADER = 'x-sim-stream-protocol' as const
export const AGENT_STREAM_PROTOCOL_V1 = 'agent-events-v1' as const
export type AgentStreamProtocol = typeof AGENT_STREAM_PROTOCOL_V1
/**
* Answer text. The only frame legacy clients append to the answer.
*
* Legacy clients (no protocol header) receive only settled final-turn text.
* Dual-gated clients receive answer text live as it streams — including text
* from a turn that may later resolve to tool calls — reconciled by
* {@link ChatStreamChunkResetFrame} when a turn turns out to be intermediate.
*/
export interface ChatStreamChunkFrame {
blockId: string
chunk: string
}
/**
* Dual-gated only: the live-streamed answer text for `blockId` belonged to an
* intermediate turn (tool calls follow). Clients discard the block's
* accumulated answer text; the final turn re-streams after tools settle.
*/
export interface ChatStreamChunkResetFrame {
blockId: string
event: 'chunk_reset'
}
/** Thinking / reasoning-summary delta. Dual-gated; never reuses `chunk`. */
export interface ChatStreamThinkingFrame {
blockId: string
event: 'thinking'
data: string
}
/** Tool lifecycle (name + status only — never args or results). Dual-gated. */
export interface ChatStreamToolFrame {
blockId: string
event: 'tool'
phase: 'start' | 'end'
id: string
name: string
status?: ToolCallEndStatus
}
/** Terminal success envelope, followed by `[DONE]`. */
export interface ChatStreamFinalFrame {
event: 'final'
data: Record<string, unknown>
}
/** Terminal failure, followed by `[DONE]`. Never followed by `final`. */
export interface ChatStreamErrorFrame {
blockId?: string
event: 'error'
error: string
}
/** Non-terminal mid-block read issue; the stream keeps going. */
export interface ChatStreamStreamErrorFrame {
blockId?: string
event: 'stream_error'
error: string
}
/**
* Every JSON frame the public chat / simple SSE stream can carry (the stream
* additionally ends with a literal `[DONE]` marker). The server emitters and
* the chat client both consume this union so the two cannot drift.
*/
export type ChatStreamFrame =
| ChatStreamChunkFrame
| ChatStreamChunkResetFrame
| ChatStreamThinkingFrame
| ChatStreamToolFrame
| ChatStreamFinalFrame
| ChatStreamErrorFrame
| ChatStreamStreamErrorFrame
function isRecord(value: unknown): value is Record<string, unknown> {
return Boolean(value) && typeof value === 'object'
}
/**
* Answer text frame: `{ blockId, chunk }` with no `event` discriminator.
* Positively defined so thinking/tool/terminal frames can never be appended
* into the answer by a client that checks this first.
*/
export function isChatChunkFrame(value: unknown): value is ChatStreamChunkFrame {
if (!isRecord(value)) return false
return (
typeof value.blockId === 'string' &&
typeof value.chunk === 'string' &&
value.chunk.length > 0 &&
value.event === undefined
)
}
export function isChatChunkResetFrame(value: unknown): value is ChatStreamChunkResetFrame {
if (!isRecord(value)) return false
return value.event === 'chunk_reset' && typeof value.blockId === 'string'
}
export function isChatThinkingFrame(value: unknown): value is ChatStreamThinkingFrame {
if (!isRecord(value)) return false
return (
value.event === 'thinking' &&
typeof value.blockId === 'string' &&
typeof value.data === 'string'
)
}
export function isChatToolFrame(value: unknown): value is ChatStreamToolFrame {
if (!isRecord(value)) return false
return (
value.event === 'tool' &&
typeof value.blockId === 'string' &&
(value.phase === 'start' || value.phase === 'end') &&
typeof value.id === 'string' &&
value.id.length > 0 &&
typeof value.name === 'string' &&
value.name.length > 0
)
}
export function isChatFinalFrame(value: unknown): value is ChatStreamFinalFrame {
if (!isRecord(value)) return false
return value.event === 'final' && isRecord(value.data)
}
export function isChatErrorFrame(value: unknown): value is ChatStreamErrorFrame {
if (!isRecord(value)) return false
return value.event === 'error'
}
export function isChatStreamErrorFrame(value: unknown): value is ChatStreamStreamErrorFrame {
if (!isRecord(value)) return false
return value.event === 'stream_error'
}
/**
* Returns true when both the deployment policy and the request protocol opt-in
* are present. Simple SSE checks this before emitting thinking/tool frames.
*/
export function shouldEmitAgentStreamEvents(options: {
includeThinking: boolean | null | undefined
requestHeaders: Headers | { get(name: string): string | null }
}): boolean {
if (options.includeThinking !== true) {
return false
}
const raw = options.requestHeaders.get(AGENT_STREAM_PROTOCOL_HEADER)
if (!raw) {
return false
}
// Allow comma-separated values / surrounding whitespace from proxies.
const tokens = raw
.split(',')
.map((token) => token.trim().toLowerCase())
.filter(Boolean)
return tokens.includes(AGENT_STREAM_PROTOCOL_V1)
}
apps/sim/lib/workflows/streaming/forward-agent-stream-events.test.ts
+174
View File
@@ -0,0 +1,174 @@
/**
* @vitest-environment node
*/
import { describe, expect, it, vi } from 'vitest'
import {
forwardAgentStreamToExecutionEvents,
shouldForwardAnswerTextFromSink,
} from '@/lib/workflows/streaming/forward-agent-stream-events'
import type { StreamingExecution } from '@/executor/types'
import type { AgentStreamEvent } from '@/providers/stream-events'
function makeStreamingExec(
onSubscribe: (handler: (event: AgentStreamEvent) => void | Promise<void>) => void,
unsubscribe = vi.fn()
): StreamingExecution {
return {
stream: new ReadableStream(),
execution: { success: true, output: {} },
subscribe: (sink: { onEvent: (event: AgentStreamEvent) => void | Promise<void> }) => {
onSubscribe(sink.onEvent)
return unsubscribe
},
} as StreamingExecution
}
describe('forwardAgentStreamToExecutionEvents', () => {
it('subscribes in the sync window and maps sink events to execution events', async () => {
let sinkHandler: ((event: AgentStreamEvent) => void | Promise<void>) | undefined
const unsubscribe = vi.fn()
const sendEvent = vi.fn()
const unsub = forwardAgentStreamToExecutionEvents(
makeStreamingExec((handler) => {
sinkHandler = handler
}, unsubscribe),
{ blockId: 'agent-1', executionId: 'exec-1', workflowId: 'wf-1', sendEvent }
)
expect(sinkHandler).toBeTypeOf('function')
await sinkHandler!({ type: 'thinking_delta', text: 'plan ' })
await sinkHandler!({ type: 'tool_call_start', id: 't1', name: 'http_request' })
await sinkHandler!({
type: 'tool_call_end',
id: 't1',
name: 'http_request',
status: 'success',
})
await sinkHandler!({ type: 'text_delta', text: 'hi', turn: 'final' })
expect(sendEvent).toHaveBeenCalledTimes(3)
expect(sendEvent.mock.calls[0][0]).toMatchObject({
type: 'stream:thinking',
executionId: 'exec-1',
workflowId: 'wf-1',
data: { blockId: 'agent-1', text: 'plan ' },
})
expect(sendEvent.mock.calls[1][0]).toMatchObject({
type: 'stream:tool',
data: { blockId: 'agent-1', phase: 'start', id: 't1', name: 'http_request' },
})
expect(sendEvent.mock.calls[2][0]).toMatchObject({
type: 'stream:tool',
data: { blockId: 'agent-1', phase: 'end', id: 't1', name: 'http_request', status: 'success' },
})
unsub()
expect(unsubscribe).toHaveBeenCalled()
})
it('does not forward text deltas by default (answer text rides the byte stream)', async () => {
let sinkHandler: ((event: AgentStreamEvent) => void | Promise<void>) | undefined
const sendEvent = vi.fn()
forwardAgentStreamToExecutionEvents(
makeStreamingExec((handler) => {
sinkHandler = handler
}),
{ blockId: 'agent-1', executionId: 'exec-1', workflowId: 'wf-1', sendEvent }
)
await sinkHandler!({ type: 'text_delta', text: 'answer', turn: 'final' })
await sinkHandler!({ type: 'text_delta', text: 'preamble', turn: 'intermediate' })
await sinkHandler!({ type: 'turn_end', turn: 'intermediate' })
expect(sendEvent).not.toHaveBeenCalled()
})
it('forwardAnswerText streams live text and resets intermediate turns', async () => {
let sinkHandler: ((event: AgentStreamEvent) => void | Promise<void>) | undefined
const sendEvent = vi.fn()
forwardAgentStreamToExecutionEvents(
makeStreamingExec((handler) => {
sinkHandler = handler
}),
{
blockId: 'agent-1',
executionId: 'exec-1',
workflowId: 'wf-1',
sendEvent,
forwardAnswerText: true,
}
)
// Turn 1: preamble text, then tools follow → reset.
await sinkHandler!({ type: 'text_delta', text: 'Let me check…', turn: 'pending' })
await sinkHandler!({ type: 'turn_end', turn: 'intermediate' })
// Turn 2: final answer.
await sinkHandler!({ type: 'text_delta', text: 'Answer', turn: 'pending' })
await sinkHandler!({ type: 'turn_end', turn: 'final' })
// Intermediate-tagged deltas never forward.
await sinkHandler!({ type: 'text_delta', text: 'hidden', turn: 'intermediate' })
const calls = sendEvent.mock.calls.map(([event]) => ({ type: event.type, data: event.data }))
expect(calls).toEqual([
{ type: 'stream:chunk', data: { blockId: 'agent-1', chunk: 'Let me check…' } },
{ type: 'stream:chunk_reset', data: { blockId: 'agent-1' } },
{ type: 'stream:chunk', data: { blockId: 'agent-1', chunk: 'Answer' } },
])
})
it('skips chunk_reset when no text was forwarded for the turn', async () => {
let sinkHandler: ((event: AgentStreamEvent) => void | Promise<void>) | undefined
const sendEvent = vi.fn()
forwardAgentStreamToExecutionEvents(
makeStreamingExec((handler) => {
sinkHandler = handler
}),
{
blockId: 'agent-1',
executionId: 'exec-1',
workflowId: 'wf-1',
sendEvent,
forwardAnswerText: true,
}
)
// Tool-only turn (no text) resolves intermediate — nothing to clear.
await sinkHandler!({ type: 'turn_end', turn: 'intermediate' })
expect(sendEvent).not.toHaveBeenCalled()
})
it('no-ops when subscribe is absent', () => {
const streamingExec = {
stream: new ReadableStream(),
execution: { success: true, output: {} },
} as StreamingExecution
const unsub = forwardAgentStreamToExecutionEvents(streamingExec, {
blockId: 'agent-1',
executionId: 'exec-1',
workflowId: 'wf-1',
sendEvent: vi.fn(),
})
expect(() => unsub()).not.toThrow()
})
})
describe('shouldForwardAnswerTextFromSink', () => {
it('requires a sink and an untransformed client stream', () => {
const base = {
stream: new ReadableStream(),
execution: { success: true, output: {} },
} as StreamingExecution
const subscribe = () => () => {}
expect(shouldForwardAnswerTextFromSink(base)).toBe(false)
expect(shouldForwardAnswerTextFromSink({ ...base, subscribe })).toBe(true)
expect(
shouldForwardAnswerTextFromSink({ ...base, subscribe, clientStreamTransformed: true })
).toBe(false)
})
})
apps/sim/lib/workflows/streaming/forward-agent-stream-events.ts
+117
View File
@@ -0,0 +1,117 @@
/**
* Bridges an agent-events provider sink onto the execution-events SSE
* vocabulary (`stream:thinking` / `stream:tool`, and optionally live answer
* text as `stream:chunk` + `stream:chunk_reset`). Shared by the workflow
* execute route and the HITL resume manager so the mapping cannot drift.
*
* Must be called in the caller's sync window (before awaiting the text
* reader) so the executor pump registers the sink before pulling provider
* chunks.
*/
import type { ExecutionEvent } from '@/lib/workflows/executor/execution-events'
import type { StreamingExecution } from '@/executor/types'
export interface ForwardAgentStreamEventsOptions {
blockId: string
executionId: string
workflowId: string
sendEvent: (event: ExecutionEvent) => void | Promise<void>
/**
* When true, answer text deltas forward live as `stream:chunk` events and an
* intermediate `turn_end` forwards as `stream:chunk_reset`. The caller MUST
* then stop emitting `stream:chunk` from the block's byte stream, or clients
* receive the final turn's text twice. Never enable for response-format
* projected streams ({@link StreamingExecution.clientStreamTransformed}).
*/
forwardAnswerText?: boolean
}
/**
* Returns true when the caller should source `stream:chunk` events from the
* sink (via {@link forwardAgentStreamToExecutionEvents} with
* `forwardAnswerText`) instead of the block's byte stream.
*/
export function shouldForwardAnswerTextFromSink(streamingExec: StreamingExecution): boolean {
return Boolean(streamingExec.subscribe) && streamingExec.clientStreamTransformed !== true
}
/**
* Subscribes to the streaming execution's agent-events sink and forwards
* thinking deltas and tool lifecycle as execution events. With
* {@link ForwardAgentStreamEventsOptions.forwardAnswerText}, answer text also
* forwards live (`pending` deltas stream as the model generates; a
* `chunk_reset` clears turns that resolve to tool calls). Returns an
* unsubscribe function (no-op when the execution has no sink).
*/
export function forwardAgentStreamToExecutionEvents(
streamingExec: StreamingExecution,
options: ForwardAgentStreamEventsOptions
): () => void {
if (!streamingExec.subscribe) {
return () => {}
}
const { blockId, executionId, workflowId, sendEvent, forwardAnswerText = false } = options
let emittedSinceReset = false
return streamingExec.subscribe({
onEvent: async (event) => {
if (event.type === 'thinking_delta') {
await sendEvent({
type: 'stream:thinking',
timestamp: new Date().toISOString(),
executionId,
workflowId,
data: { blockId, text: event.text },
})
return
}
if (event.type === 'tool_call_start') {
await sendEvent({
type: 'stream:tool',
timestamp: new Date().toISOString(),
executionId,
workflowId,
data: { blockId, phase: 'start', id: event.id, name: event.name },
})
return
}
if (event.type === 'tool_call_end') {
await sendEvent({
type: 'stream:tool',
timestamp: new Date().toISOString(),
executionId,
workflowId,
data: { blockId, phase: 'end', id: event.id, name: event.name, status: event.status },
})
return
}
if (!forwardAnswerText) {
return
}
if (event.type === 'text_delta') {
if (event.turn === 'intermediate' || !event.text) return
emittedSinceReset = true
await sendEvent({
type: 'stream:chunk',
timestamp: new Date().toISOString(),
executionId,
workflowId,
data: { blockId, chunk: event.text },
})
return
}
if (event.type === 'turn_end' && event.turn === 'intermediate' && emittedSinceReset) {
emittedSinceReset = false
await sendEvent({
type: 'stream:chunk_reset',
timestamp: new Date().toISOString(),
executionId,
workflowId,
data: { blockId },
})
}
},
})
}
apps/sim/lib/workflows/streaming/streaming.test.ts
+526
View File
@@ -604,3 +604,529 @@ describe('createStreamingResponse', () => {
await expect(readSSEStream(stream)).resolves.toBe('ok')
})
})
describe('createStreamingResponse agent-events-v1', () => {
beforeEach(() => {
vi.clearAllMocks()
clearLargeValueCacheForTests()
})
function createAgentStreamExecuteFn(options: {
thinking?: string[]
answer: string
fail?: boolean
tools?: Array<
| { type: 'tool_call_start'; id: string; name: string }
| { type: 'tool_call_end'; id: string; name: string; status: string }
>
}) {
return async ({
onStream,
abortSignal,
}: {
onStream: (streamingExec: any) => Promise<void>
onBlockComplete: (blockId: string, output: unknown) => Promise<void>
abortSignal: AbortSignal
}) => {
let textController!: ReadableStreamDefaultController<Uint8Array>
let sink: { onEvent: (event: unknown) => void | Promise<void> } | undefined
const textStream = new ReadableStream<Uint8Array>({
start(controller) {
textController = controller
},
})
const onStreamPromise = onStream({
stream: textStream,
streamFormat: 'text',
subscribe: (nextSink: { onEvent: (event: unknown) => void | Promise<void> }) => {
sink = nextSink
return () => {
sink = undefined
}
},
execution: {
blockId: 'agent-1',
success: true,
output: { content: options.answer },
logs: [],
metadata: {},
},
})
if (options.fail) {
textController.error(new Error('provider reset'))
await onStreamPromise.catch(() => {})
throw new Error('provider reset')
}
for (const text of options.thinking ?? []) {
await sink?.onEvent({ type: 'thinking_delta', text })
}
for (const toolEvent of options.tools ?? []) {
await sink?.onEvent(toolEvent)
}
// Mirror the pump: text dispatches to the sink first, then projects to bytes.
await sink?.onEvent({ type: 'text_delta', text: options.answer, turn: 'final' })
textController.enqueue(new TextEncoder().encode(options.answer))
textController.close()
await onStreamPromise
expect(abortSignal).toBeDefined()
return {
success: true,
output: { content: options.answer },
logs: [
{
blockId: 'agent-1',
output: { content: '' },
startedAt: new Date().toISOString(),
endedAt: new Date().toISOString(),
durationMs: 1,
success: true,
},
],
} as any
}
}
async function collectSSEPayloads(stream: ReadableStream<Uint8Array>): Promise<string[]> {
const reader = stream.getReader()
const decoder = new TextDecoder()
let buffer = ''
while (true) {
const { done, value } = await reader.read()
if (done) {
buffer += decoder.decode()
break
}
buffer += decoder.decode(value, { stream: true })
}
return buffer
.split('\n\n')
.map((chunk) => chunk.trim())
.filter((chunk) => chunk.startsWith('data: '))
.map((chunk) => chunk.slice(6))
}
it('legacy path without protocol header stays text-only (no thinking frames)', async () => {
const stream = await createStreamingResponse({
requestId: 'request-1',
streamConfig: {
includeThinking: true,
selectedOutputs: ['agent-1_content'],
},
// No requestHeaders → gate closed
executeFn: createAgentStreamExecuteFn({
thinking: ['secret thought'],
answer: 'Hello',
}),
})
const events = await collectSSEEvents(stream)
expect(events.some((event) => event.event === 'thinking')).toBe(false)
expect(events).toContainEqual({ blockId: 'agent-1', chunk: 'Hello' })
expect(events.some((event) => event.event === 'final')).toBe(true)
})
it('header + includeThinking emits thinking on data and answer on chunk', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: true,
selectedOutputs: ['agent-1_content'],
},
executeFn: createAgentStreamExecuteFn({
thinking: ['hmm ', 'yes'],
answer: 'Answer',
}),
})
const events = await collectSSEEvents(stream)
expect(events.filter((event) => event.event === 'thinking')).toEqual([
{ blockId: 'agent-1', event: 'thinking', data: 'hmm ' },
{ blockId: 'agent-1', event: 'thinking', data: 'yes' },
])
expect(events).toContainEqual({ blockId: 'agent-1', chunk: 'Answer' })
expect(events.some((event) => event.event === 'final')).toBe(true)
})
it('dual gate emits tool start/end frames without putting tools on chunk', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: true,
selectedOutputs: ['agent-1_content'],
},
executeFn: createAgentStreamExecuteFn({
answer: 'Done',
tools: [
{ type: 'tool_call_start', id: 'toolu_1', name: 'get_weather' },
{
type: 'tool_call_end',
id: 'toolu_1',
name: 'get_weather',
status: 'success',
},
],
}),
})
const events = await collectSSEEvents(stream)
expect(events.filter((event) => event.event === 'tool')).toEqual([
{
blockId: 'agent-1',
event: 'tool',
phase: 'start',
id: 'toolu_1',
name: 'get_weather',
},
{
blockId: 'agent-1',
event: 'tool',
phase: 'end',
id: 'toolu_1',
name: 'get_weather',
status: 'success',
},
])
expect(events).toContainEqual({ blockId: 'agent-1', chunk: 'Done' })
expect(
events.some(
(event) =>
typeof event.chunk === 'string' &&
(String(event.chunk).includes('toolu_1') || String(event.chunk).includes('get_weather'))
)
).toBe(false)
})
it('dual gate streams pending text live and resets intermediate turns', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: true,
selectedOutputs: ['agent-1_content'],
},
executeFn: async ({ onStream }) => {
let textController!: ReadableStreamDefaultController<Uint8Array>
let sink: { onEvent: (event: unknown) => void | Promise<void> } | undefined
const textStream = new ReadableStream<Uint8Array>({
start(controller) {
textController = controller
},
})
const onStreamPromise = onStream({
stream: textStream,
streamFormat: 'text',
subscribe: (nextSink: { onEvent: (event: unknown) => void | Promise<void> }) => {
sink = nextSink
return () => {}
},
execution: {
blockId: 'agent-1',
success: true,
output: { content: 'Final answer' },
logs: [],
metadata: {},
},
} as any)
// Turn 1: live preamble, then tools follow → intermediate turn_end.
await sink?.onEvent({ type: 'text_delta', text: 'Checking…', turn: 'pending' })
await sink?.onEvent({ type: 'tool_call_start', id: 'toolu_1', name: 'get_weather' })
await sink?.onEvent({ type: 'turn_end', turn: 'intermediate' })
await sink?.onEvent({
type: 'tool_call_end',
id: 'toolu_1',
name: 'get_weather',
status: 'success',
})
// Turn 2: live final answer; pump projects it to bytes at turn_end.
await sink?.onEvent({ type: 'text_delta', text: 'Final ', turn: 'pending' })
await sink?.onEvent({ type: 'text_delta', text: 'answer', turn: 'pending' })
await sink?.onEvent({ type: 'turn_end', turn: 'final' })
textController.enqueue(new TextEncoder().encode('Final answer'))
textController.close()
await onStreamPromise
return {
success: true,
output: { content: 'Final answer' },
logs: [
{
blockId: 'agent-1',
output: { content: '' },
startedAt: new Date().toISOString(),
endedAt: new Date().toISOString(),
durationMs: 1,
success: true,
},
],
} as any
},
})
const events = await collectSSEEvents(stream)
// Live text arrives as chunk frames in stream order, with a reset between turns.
const answerFlow = events
.filter((event) => event.chunk !== undefined || event.event === 'chunk_reset')
.map((event) => (event.event === 'chunk_reset' ? 'RESET' : event.chunk))
expect(answerFlow).toEqual(['Checking…', 'RESET', 'Final ', 'answer'])
// The byte-path flush of the same final text must not duplicate chunk frames.
expect(events.filter((event) => event.chunk !== undefined).map((event) => event.chunk)).toEqual(
['Checking…', 'Final ', 'answer']
)
})
it('dual gate keeps byte-path chunks for response-format transformed streams', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: true,
selectedOutputs: ['agent-1_content'],
},
executeFn: async ({ onStream }) => {
let textController!: ReadableStreamDefaultController<Uint8Array>
let sink: { onEvent: (event: unknown) => void | Promise<void> } | undefined
const textStream = new ReadableStream<Uint8Array>({
start(controller) {
textController = controller
},
})
const onStreamPromise = onStream({
stream: textStream,
streamFormat: 'text',
subscribe: (nextSink: { onEvent: (event: unknown) => void | Promise<void> }) => {
sink = nextSink
return () => {}
},
clientStreamTransformed: true,
execution: {
blockId: 'agent-1',
success: true,
output: { content: '{"answer":"extracted"}' },
logs: [],
metadata: {},
},
} as any)
// Sink text must NOT become chunk frames — bytes are a different projection.
await sink?.onEvent({ type: 'text_delta', text: '{"answer":"', turn: 'pending' })
await sink?.onEvent({ type: 'text_delta', text: 'extracted"}', turn: 'pending' })
await sink?.onEvent({ type: 'turn_end', turn: 'final' })
textController.enqueue(new TextEncoder().encode('extracted'))
textController.close()
await onStreamPromise
return {
success: true,
output: { content: '{"answer":"extracted"}' },
logs: [
{
blockId: 'agent-1',
output: { content: '' },
startedAt: new Date().toISOString(),
endedAt: new Date().toISOString(),
durationMs: 1,
success: true,
},
],
} as any
},
})
const events = await collectSSEEvents(stream)
expect(events.filter((event) => event.chunk !== undefined).map((event) => event.chunk)).toEqual(
['extracted']
)
expect(events.some((event) => event.event === 'chunk_reset')).toBe(false)
})
it('protocol header without includeThinking does not emit tool frames', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: false,
selectedOutputs: ['agent-1_content'],
},
executeFn: createAgentStreamExecuteFn({
answer: 'Answer',
tools: [{ type: 'tool_call_start', id: 'toolu_1', name: 'get_weather' }],
}),
})
const events = await collectSSEEvents(stream)
expect(events.some((event) => event.event === 'tool')).toBe(false)
expect(events).toContainEqual({ blockId: 'agent-1', chunk: 'Answer' })
})
it('protocol header without includeThinking does not emit thinking', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: false,
selectedOutputs: ['agent-1_content'],
},
executeFn: createAgentStreamExecuteFn({
thinking: ['should not appear'],
answer: 'Answer',
}),
})
const events = await collectSSEEvents(stream)
expect(events.some((event) => event.event === 'thinking')).toBe(false)
expect(events).toContainEqual({ blockId: 'agent-1', chunk: 'Answer' })
})
it('provider failure emits one terminal error, no final, then [DONE]', async () => {
const stream = await createStreamingResponse({
requestId: 'request-1',
streamConfig: {},
executeFn: createAgentStreamExecuteFn({
answer: 'partial',
fail: true,
}),
})
const payloads = await collectSSEPayloads(stream)
const events = payloads
.filter((payload) => payload !== '[DONE]' && payload !== '"[DONE]"')
.map((payload) => JSON.parse(payload) as Record<string, unknown>)
expect(events.filter((event) => event.event === 'error')).toHaveLength(1)
expect(events.some((event) => event.event === 'final')).toBe(false)
expect(payloads.some((payload) => payload === '[DONE]' || payload === '"[DONE]"')).toBe(true)
})
it('requestSignal abort propagates to executeFn abortSignal', async () => {
const requestAbort = new AbortController()
let sawAbort = false
const stream = await createStreamingResponse({
requestId: 'request-1',
requestSignal: requestAbort.signal,
streamConfig: {},
executeFn: async ({ abortSignal }) => {
requestAbort.abort()
sawAbort = abortSignal.aborted
return {
success: false,
status: 'cancelled',
output: {},
logs: [],
} as any
},
})
const events = await collectSSEEvents(stream)
expect(sawAbort).toBe(true)
expect(events.some((event) => event.event === 'final')).toBe(false)
expect(events).toContainEqual({ event: 'error', error: 'Client cancelled request' })
})
it('thinking never enters streamedChunks / log content rewrite', async () => {
const headers = new Headers({
'x-sim-stream-protocol': 'agent-events-v1',
})
let rewrittenContent: string | undefined
const stream = await createStreamingResponse({
requestId: 'request-1',
requestHeaders: headers,
streamConfig: {
includeThinking: true,
selectedOutputs: ['agent-1_content'],
},
executeFn: async ({ onStream }) => {
let textController!: ReadableStreamDefaultController<Uint8Array>
let sink: { onEvent: (event: unknown) => void | Promise<void> } | undefined
const textStream = new ReadableStream<Uint8Array>({
start(controller) {
textController = controller
},
})
const onStreamPromise = onStream({
stream: textStream,
streamFormat: 'text',
subscribe: (nextSink: any) => {
sink = nextSink
return () => {
sink = undefined
}
},
execution: {
blockId: 'agent-1',
success: true,
output: { content: 'visible' },
logs: [],
metadata: {},
},
} as any)
await sink?.onEvent({ type: 'thinking_delta', text: 'PRIVATE_THINKING' })
textController.enqueue(new TextEncoder().encode('visible'))
textController.close()
await onStreamPromise
return {
success: true,
output: {},
logs: [
{
blockId: 'agent-1',
output: { content: '' },
startedAt: new Date().toISOString(),
endedAt: new Date().toISOString(),
durationMs: 1,
success: true,
},
],
} as any
},
})
const events = await collectSSEEvents(stream)
const answerChunks = events.filter((event) => typeof event.chunk === 'string')
expect(answerChunks.every((event) => !String(event.chunk).includes('PRIVATE_THINKING'))).toBe(
true
)
expect(events).toContainEqual({
blockId: 'agent-1',
event: 'thinking',
data: 'PRIVATE_THINKING',
})
// Force consumption of stream so log rewrite runs
expect(events.some((event) => event.event === 'final')).toBe(true)
void rewrittenContent
})
})
apps/sim/lib/workflows/streaming/streaming.ts
+262 -42
View File
@@ -1,5 +1,6 @@
import { createLogger } from '@sim/logger'
import { getErrorMessage } from '@sim/utils/errors'
import { omit } from '@sim/utils/object'
import { createTimeoutAbortController, getTimeoutErrorMessage } from '@/lib/core/execution-limits'
import {
extractBlockIdFromOutputId,
@@ -24,8 +25,22 @@ import {
cleanupExecutionBase64Cache,
hydrateUserFilesWithBase64,
} from '@/lib/uploads/utils/user-file-base64.server'
import {
AGENT_STREAM_PROTOCOL_HEADER,
AGENT_STREAM_PROTOCOL_V1,
type ChatStreamChunkFrame,
type ChatStreamChunkResetFrame,
type ChatStreamErrorFrame,
type ChatStreamFinalFrame,
type ChatStreamStreamErrorFrame,
type ChatStreamThinkingFrame,
type ChatStreamToolFrame,
shouldEmitAgentStreamEvents,
} from '@/lib/workflows/streaming/agent-stream-protocol'
import type { BlockLog, ExecutionResult, StreamingExecution } from '@/executor/types'
import { navigatePathAsync } from '@/executor/variables/resolvers/reference-async.server'
import type { ToolCallEndStatus } from '@/providers/stream-events'
import { DEFAULT_MAX_THINKING_CHARS } from '@/providers/stream-pump'
/**
* Extended streaming execution type that includes blockId on the execution.
@@ -41,6 +56,21 @@ const DANGEROUS_KEYS = ['__proto__', 'constructor', 'prototype']
const SELECTED_OUTPUT_TOO_LARGE_MESSAGE =
'Selected output is too large to inline; select a nested field or use pagination/preview.'
/**
* Simple SSE stream contract — frame shapes are the `ChatStreamFrame` union in
* `agent-stream-protocol.ts`, consumed by both these emitters and the chat client:
* - Answer text: `{ blockId, chunk }` only (`chunk` is forever answer text).
* Legacy clients get settled final-turn text; dual-gated clients get answer
* text live from the agent-events sink, reconciled by
* `{ blockId, event: 'chunk_reset' }` when a turn resolves to tool calls.
* - Thinking (opt-in): `{ blockId, event: 'thinking', data }` — never uses `chunk`.
* - Success terminal: `{ event: 'final', data }` then `[DONE]`.
* - Failure terminal: exactly one `{ event: 'error', ... }` then `[DONE]`. No `final` after failure.
* - Mid-block read issues may emit non-terminal `{ event: 'stream_error', blockId, error }`.
* - Thinking never enters `streamedChunks` / log rewrite / tokenization — the
* log/tokenization source is always the byte stream (final-turn text only).
*/
interface StreamingConfig {
selectedOutputs?: string[]
isSecureMode?: boolean
@@ -48,6 +78,11 @@ interface StreamingConfig {
includeFileBase64?: boolean
base64MaxBytes?: number
timeoutMs?: number
/**
* Deployment policy for thinking/tool SSE. Still requires the client to send
* {@link AGENT_STREAM_PROTOCOL_HEADER}: {@link AGENT_STREAM_PROTOCOL_V1}.
*/
includeThinking?: boolean
}
export type StreamingExecutorFn = (callbacks: {
@@ -67,9 +102,33 @@ export interface StreamingResponseOptions {
workspaceId?: string
workflowId?: string
userId?: string
/** Incoming fetch/request abort — combined with the stream timeout. */
requestSignal?: AbortSignal
/** Used with {@link StreamingConfig.includeThinking} for dual-gate thinking SSE. */
requestHeaders?: Headers | { get(name: string): string | null }
executeFn: StreamingExecutorFn
}
/**
* Extra response headers when the dual-gate agent stream protocol is active.
* Callers should merge these into the SSE response alongside {@link SSE_HEADERS}.
*/
export function agentStreamProtocolResponseHeaders(options: {
includeThinking?: boolean | null
requestHeaders?: Headers | { get(name: string): string | null }
}): Record<string, string> {
if (!options.requestHeaders) return {}
if (
!shouldEmitAgentStreamEvents({
includeThinking: options.includeThinking,
requestHeaders: options.requestHeaders,
})
) {
return {}
}
return { [AGENT_STREAM_PROTOCOL_HEADER]: AGENT_STREAM_PROTOCOL_V1 }
}
interface StreamingState {
streamedChunks: Map<string, string[]>
processedOutputs: Set<string>
@@ -193,6 +252,47 @@ function assertSelectedOutputBytes(value: unknown): number {
return bytes
}
/**
* Strips model internals from `providerTiming.timeSegments` before an output
* rides a simple-SSE `final` envelope: `thinkingContent`, intermediate
* `assistantContent`, and tool-call arguments would otherwise reach public
* chat clients wholesale, bypassing the dual-gated thinking/tool frames.
* Timing numbers and tool names stay — they carry no model internals.
*/
function sanitizeProviderTimingForEnvelope(
output: Record<string, unknown>
): Record<string, unknown> {
const providerTiming = output.providerTiming as { timeSegments?: unknown } | undefined
if (!providerTiming || !Array.isArray(providerTiming.timeSegments)) {
return output
}
return {
...output,
providerTiming: {
...providerTiming,
timeSegments: providerTiming.timeSegments.map((segment) => {
if (!segment || typeof segment !== 'object') return segment
const { toolCalls, ...rest } = omit(segment as Record<string, unknown>, [
'thinkingContent',
'assistantContent',
]) as Record<string, unknown> & { toolCalls?: unknown }
return {
...rest,
...(Array.isArray(toolCalls)
? {
toolCalls: toolCalls.map((toolCall) =>
toolCall && typeof toolCall === 'object'
? omit(toolCall as Record<string, unknown>, ['arguments'])
: toolCall
),
}
: {}),
}
}),
},
}
}
async function buildMinimalResult(
result: ExecutionResult,
selectedOutputs: string[] | undefined,
@@ -220,7 +320,7 @@ async function buildMinimalResult(
}
if (result.status === 'paused') {
minimalResult.output = result.output || {}
minimalResult.output = sanitizeProviderTimingForEnvelope(result.output || {})
return compactExecutionPayload(minimalResult, {
...durableContext,
preserveUserFileBase64: includeFileBase64,
@@ -229,7 +329,7 @@ async function buildMinimalResult(
}
if (!selectedOutputs?.length) {
minimalResult.output = result.output || {}
minimalResult.output = sanitizeProviderTimingForEnvelope(result.output || {})
return compactExecutionPayload(minimalResult, {
...durableContext,
preserveUserFileBase64: includeFileBase64,
@@ -351,14 +451,31 @@ export async function createStreamingResponse(
options: StreamingResponseOptions
): Promise<ReadableStream> {
const { requestId, streamConfig, executionId, executeFn } = options
const durableContext = {
workspaceId: options.workspaceId,
workflowId: options.workflowId,
executionId,
userId: options.userId,
requireDurable: Boolean(options.workspaceId && options.workflowId && executionId),
}
const timeoutController = createTimeoutAbortController(streamConfig.timeoutMs)
const emitAgentEvents =
Boolean(options.requestHeaders) &&
shouldEmitAgentStreamEvents({
includeThinking: streamConfig.includeThinking,
requestHeaders: options.requestHeaders!,
})
const maxThinkingChars = DEFAULT_MAX_THINKING_CHARS
let requestAborted = false
const onRequestAbort = () => {
requestAborted = true
timeoutController.abort()
}
if (options.requestSignal) {
if (options.requestSignal.aborted) {
onRequestAbort()
} else {
options.requestSignal.addEventListener('abort', onRequestAbort, { once: true })
}
}
const cleanupRequestAbort = () => {
options.requestSignal?.removeEventListener('abort', onRequestAbort)
}
return new ReadableStream({
async start(controller) {
@@ -370,6 +487,7 @@ export async function createStreamingResponse(
selectedOutputBytes: 0,
streamedSelectedOutputKeys: new Set(),
}
let thinkingCharsEmitted = 0
const sendChunk = (
blockId: string,
@@ -386,12 +504,47 @@ export async function createStreamingResponse(
state.selectedOutputBytes = nextSelectedOutputBytes
state.streamedSelectedOutputKeys.add(options.selectedOutputKey)
}
controller.enqueue(encodeSSE({ blockId, chunk }))
const frame: ChatStreamChunkFrame = { blockId, chunk }
controller.enqueue(encodeSSE(frame))
state.processedOutputs.add(blockId)
}
const sendThinking = (blockId: string, text: string) => {
if (!text || thinkingCharsEmitted >= maxThinkingChars) return
const remaining = maxThinkingChars - thinkingCharsEmitted
const forwarded = text.length > remaining ? text.slice(0, remaining) : text
thinkingCharsEmitted += forwarded.length
// Never push thinking into streamedChunks — logs stay answer-text only.
const frame: ChatStreamThinkingFrame = {
blockId,
event: 'thinking',
data: forwarded,
}
controller.enqueue(encodeSSE(frame))
}
const sendTool = (
blockId: string,
phase: 'start' | 'end',
id: string,
name: string,
status?: ToolCallEndStatus
) => {
const frame: ChatStreamToolFrame = {
blockId,
event: 'tool',
phase,
id,
name,
...(phase === 'end' && status ? { status } : {}),
}
controller.enqueue(encodeSSE(frame))
}
/**
* Callback for handling streaming execution events.
* Subscribe synchronously before the first await so the executor pump
* can attach sinks before pulling provider chunks.
*/
const onStreamCallback = async (streamingExec: StreamingExecutionWithBlockId) => {
const blockId = streamingExec.execution?.blockId
@@ -400,9 +553,62 @@ export async function createStreamingResponse(
return
}
/**
* Dual-gated clients get answer text live from the sink (pending deltas
* stream as the model generates; `chunk_reset` clears an intermediate
* turn). The byte stream then only feeds `streamedChunks` for logs.
* Response-format projections rewrite the bytes, so those blocks keep
* the byte stream as the frame source.
*/
const sinkAnswerText =
emitAgentEvents &&
Boolean(streamingExec.subscribe) &&
streamingExec.clientStreamTransformed !== true
/** False until the first chunk since block start or since a reset. */
let emittedSinceReset = false
const emitAnswerChunk = (text: string) => {
if (!text) return
if (!emittedSinceReset) {
// sendChunk adds the cross-block separator + output bookkeeping.
sendChunk(blockId, text)
emittedSinceReset = true
} else {
const frame: ChatStreamChunkFrame = { blockId, chunk: text }
controller.enqueue(encodeSSE(frame))
}
}
let unsubscribe: (() => void) | undefined
if (emitAgentEvents && streamingExec.subscribe) {
unsubscribe = streamingExec.subscribe({
onEvent: async (event) => {
if (event.type === 'thinking_delta') {
sendThinking(blockId, event.text)
} else if (event.type === 'tool_call_start') {
sendTool(blockId, 'start', event.id, event.name)
} else if (event.type === 'tool_call_end') {
sendTool(blockId, 'end', event.id, event.name, event.status)
} else if (sinkAnswerText && event.type === 'text_delta') {
if (event.turn !== 'intermediate') {
emitAnswerChunk(event.text)
}
} else if (sinkAnswerText && event.type === 'turn_end') {
if (event.turn === 'intermediate' && emittedSinceReset) {
const frame: ChatStreamChunkResetFrame = { blockId, event: 'chunk_reset' }
controller.enqueue(encodeSSE(frame))
// Re-arm separator bookkeeping so re-streamed text starts clean.
emittedSinceReset = false
state.processedOutputs.delete(blockId)
}
}
},
})
}
const reader = streamingExec.stream.getReader()
const decoder = new TextDecoder()
let isFirstChunk = true
try {
while (true) {
@@ -418,22 +624,20 @@ export async function createStreamingResponse(
}
state.streamedChunks.get(blockId)!.push(textChunk)
if (isFirstChunk) {
sendChunk(blockId, textChunk)
isFirstChunk = false
} else {
controller.enqueue(encodeSSE({ blockId, chunk: textChunk }))
if (!sinkAnswerText) {
emitAnswerChunk(textChunk)
}
}
} catch (error) {
logger.error(`[${requestId}] Error reading stream for block ${blockId}:`, error)
controller.enqueue(
encodeSSE({
event: 'stream_error',
blockId,
error: getErrorMessage(error, 'Stream reading error'),
})
)
const frame: ChatStreamStreamErrorFrame = {
event: 'stream_error',
blockId,
error: getErrorMessage(error, 'Stream reading error'),
}
controller.enqueue(encodeSSE(frame))
} finally {
unsubscribe?.()
}
}
@@ -521,13 +725,12 @@ export async function createStreamingResponse(
})
const errorMessage = getSelectedOutputErrorMessage(error)
state.selectedOutputError ??= errorMessage
controller.enqueue(
encodeSSE({
event: 'error',
blockId,
error: errorMessage,
})
)
const frame: ChatStreamErrorFrame = {
event: 'error',
blockId,
error: errorMessage,
}
controller.enqueue(encodeSSE(frame))
break
}
}
@@ -555,7 +758,8 @@ export async function createStreamingResponse(
if (
result.status === 'cancelled' &&
timeoutController.isTimedOut() &&
timeoutController.timeoutMs
timeoutController.timeoutMs &&
!requestAborted
) {
const timeoutErrorMessage = getTimeoutErrorMessage(null, timeoutController.timeoutMs)
logger.info(`[${requestId}] Streaming execution timed out`, {
@@ -564,7 +768,17 @@ export async function createStreamingResponse(
if (result._streamingMetadata?.loggingSession) {
await result._streamingMetadata.loggingSession.markAsFailed(timeoutErrorMessage)
}
controller.enqueue(encodeSSE({ event: 'error', error: timeoutErrorMessage }))
const frame: ChatStreamErrorFrame = { event: 'error', error: timeoutErrorMessage }
controller.enqueue(encodeSSE(frame))
} else if (result.status === 'cancelled' && requestAborted) {
logger.info(`[${requestId}] Streaming execution aborted by client disconnect`)
if (result._streamingMetadata?.loggingSession) {
// LoggingSession has no cancelled status; match workflow execute route wording.
await result._streamingMetadata.loggingSession.markAsFailed('Client cancelled request')
}
// No `final` after abort; clients that already disconnected ignore these.
const frame: ChatStreamErrorFrame = { event: 'error', error: 'Client cancelled request' }
controller.enqueue(encodeSSE(frame))
} else {
await completeLoggingSession(result)
@@ -591,18 +805,18 @@ export async function createStreamingResponse(
}
)
controller.enqueue(
encodeSSE({
event: 'final',
data: {
...minimalResult,
...(result.status === 'paused' && { status: 'paused' }),
},
})
)
const frame: ChatStreamFinalFrame = {
event: 'final',
data: {
...minimalResult,
...(result.status === 'paused' && { status: 'paused' }),
},
}
controller.enqueue(encodeSSE(frame))
}
}
// Terminal marker: always follows success `final` or a single terminal `error`.
controller.enqueue(encodeSSE('[DONE]'))
if (executionId) {
@@ -616,7 +830,10 @@ export async function createStreamingResponse(
streamConfig.selectedOutputs?.length && isExecutionResourceLimitError(error)
? SELECTED_OUTPUT_TOO_LARGE_MESSAGE
: getErrorMessage(error, 'Stream processing error')
controller.enqueue(encodeSSE({ event: 'error', error: errorMessage }))
const frame: ChatStreamErrorFrame = { event: 'error', error: errorMessage }
controller.enqueue(encodeSSE(frame))
// Same terminal rule as timeout/abort: one error, then [DONE], never `final`.
controller.enqueue(encodeSSE('[DONE]'))
if (executionId) {
await cleanupExecutionBase64Cache(executionId)
@@ -624,12 +841,15 @@ export async function createStreamingResponse(
controller.close()
} finally {
cleanupRequestAbort()
timeoutController.cleanup()
}
},
async cancel(reason) {
logger.info(`[${requestId}] Streaming response cancelled`, { reason })
requestAborted = true
timeoutController.abort()
cleanupRequestAbort()
timeoutController.cleanup()
if (executionId) {
try {
apps/sim/package.json
+1 -1
View File
@@ -36,7 +36,7 @@
"dependencies": {
"@1password/sdk": "0.3.1",
"@a2a-js/sdk": "1.0.0-alpha.0",
"@anthropic-ai/sdk": "0.71.2",
"@anthropic-ai/sdk": "0.114.0",
"@aws-sdk/client-appconfigdata": "3.1032.0",
"@aws-sdk/client-athena": "3.1032.0",
"@aws-sdk/client-bedrock-runtime": "3.1032.0",
apps/sim/providers/__fixtures__/anthropic/fixtures.test.ts
+194
View File
@@ -0,0 +1,194 @@
/**
* @vitest-environment node
*
* Fixture gate: Anthropic stream fixtures parse and match expected assembled
* thinking/text/tool/signature content. No provider adapters are exercised yet.
*/
import { describe, expect, it } from 'vitest'
import {
anthropicRedactedThinkingAssembledContent,
anthropicRedactedThinkingExpectedText,
anthropicRedactedThinkingExpectedTraceThinking,
anthropicRedactedThinkingStreamEvents,
anthropicThinkingTextToolAssembledContent,
anthropicThinkingTextToolExpectedText,
anthropicThinkingTextToolExpectedThinking,
anthropicThinkingTextToolStreamEvents,
} from '@/providers/__fixtures__/anthropic'
type StreamEvent = {
type: string
index?: number
delta?: {
type: string
thinking?: string
text?: string
signature?: string
partial_json?: string
}
content_block?: {
type: string
thinking?: string
text?: string
data?: string
id?: string
name?: string
input?: unknown
signature?: string
}
}
function assembleAnthropicContentFromStream(events: readonly StreamEvent[]) {
const blocks: Array<Record<string, unknown>> = []
for (const event of events) {
if (event.type === 'content_block_start' && event.content_block) {
const block = event.content_block
if (block.type === 'thinking') {
blocks.push({ type: 'thinking', thinking: block.thinking ?? '', signature: '' })
} else if (block.type === 'redacted_thinking') {
blocks.push({ type: 'redacted_thinking', data: block.data ?? '' })
} else if (block.type === 'text') {
blocks.push({ type: 'text', text: block.text ?? '' })
} else if (block.type === 'tool_use') {
blocks.push({
type: 'tool_use',
id: block.id,
name: block.name,
inputJson: '',
})
}
continue
}
if (event.type !== 'content_block_delta' || event.index === undefined || !event.delta) {
continue
}
const target = blocks[event.index]
if (!target) continue
const delta = event.delta
if (delta.type === 'thinking_delta' && typeof delta.thinking === 'string') {
target.thinking = `${target.thinking ?? ''}${delta.thinking}`
} else if (delta.type === 'signature_delta' && typeof delta.signature === 'string') {
target.signature = delta.signature
} else if (delta.type === 'text_delta' && typeof delta.text === 'string') {
target.text = `${target.text ?? ''}${delta.text}`
} else if (delta.type === 'input_json_delta' && typeof delta.partial_json === 'string') {
target.inputJson = `${target.inputJson ?? ''}${delta.partial_json}`
}
}
return blocks.map((block) => {
if (block.type === 'tool_use') {
const inputJson = typeof block.inputJson === 'string' ? block.inputJson : '{}'
return {
type: 'tool_use',
id: block.id,
name: block.name,
input: JSON.parse(inputJson || '{}'),
}
}
if (block.type === 'thinking') {
return {
type: 'thinking',
thinking: block.thinking,
signature: block.signature,
}
}
return block
})
}
function extractTextDeltas(events: readonly StreamEvent[]): string {
return events
.filter(
(e) =>
e.type === 'content_block_delta' &&
e.delta?.type === 'text_delta' &&
typeof e.delta.text === 'string'
)
.map((e) => e.delta!.text!)
.join('')
}
function extractThinkingDeltas(events: readonly StreamEvent[]): string {
return events
.filter(
(e) =>
e.type === 'content_block_delta' &&
e.delta?.type === 'thinking_delta' &&
typeof e.delta.thinking === 'string'
)
.map((e) => e.delta!.thinking!)
.join('')
}
function assertValidStreamSequence(events: readonly StreamEvent[]) {
expect(events[0]?.type).toBe('message_start')
expect(events[events.length - 1]?.type).toBe('message_stop')
const open = new Set<number>()
for (const event of events) {
if (event.type === 'content_block_start' && event.index !== undefined) {
expect(open.has(event.index)).toBe(false)
open.add(event.index)
}
if (event.type === 'content_block_stop' && event.index !== undefined) {
expect(open.has(event.index)).toBe(true)
open.delete(event.index)
}
}
expect(open.size).toBe(0)
}
describe('Anthropic stream fixtures', () => {
it('parses thinking → text → tool_use stream and preserves signature', () => {
assertValidStreamSequence(anthropicThinkingTextToolStreamEvents)
expect(extractThinkingDeltas(anthropicThinkingTextToolStreamEvents)).toBe(
anthropicThinkingTextToolExpectedThinking
)
expect(extractTextDeltas(anthropicThinkingTextToolStreamEvents)).toBe(
anthropicThinkingTextToolExpectedText
)
const assembled = assembleAnthropicContentFromStream(anthropicThinkingTextToolStreamEvents)
expect(assembled).toEqual([...anthropicThinkingTextToolAssembledContent])
const thinkingBlock = assembled.find((b) => b.type === 'thinking') as {
signature?: string
}
expect(thinkingBlock?.signature).toMatch(/^EpAB/)
})
it('parses redacted_thinking + signed thinking + text and matches trace mapping', () => {
assertValidStreamSequence(anthropicRedactedThinkingStreamEvents)
expect(extractTextDeltas(anthropicRedactedThinkingStreamEvents)).toBe(
anthropicRedactedThinkingExpectedText
)
const assembled = assembleAnthropicContentFromStream(anthropicRedactedThinkingStreamEvents)
expect(assembled).toEqual([...anthropicRedactedThinkingAssembledContent])
// Mirrors enrichLastModelSegmentFromAnthropicResponse: redacted → "[redacted]"
const traceThinking = assembled
.filter((b) => b.type === 'thinking' || b.type === 'redacted_thinking')
.map((b) => (b.type === 'thinking' ? b.thinking : '[redacted]'))
.join('\n\n')
expect(traceThinking).toBe(anthropicRedactedThinkingExpectedTraceThinking)
})
it('documents that live stream today would only surface text_delta bytes', () => {
// Baseline behavior of createReadableStreamFromAnthropicStream: only text_delta
// is enqueued. This test locks the fixture expectation for later adapter work.
const textOnlyFromStream = extractTextDeltas(anthropicThinkingTextToolStreamEvents)
const thinkingFromStream = extractThinkingDeltas(anthropicThinkingTextToolStreamEvents)
expect(textOnlyFromStream).toBe(anthropicThinkingTextToolExpectedText)
expect(thinkingFromStream.length).toBeGreaterThan(0)
expect(textOnlyFromStream).not.toContain('I should check the weather')
})
})
apps/sim/providers/__fixtures__/anthropic/index.ts
+17
View File
@@ -0,0 +1,17 @@
/**
* Re-exports Anthropic stream fixtures used by agent-stream-events work.
* Keep fixture data in dedicated modules so adapters can import without pulling tests.
*/
export {
anthropicRedactedThinkingAssembledContent,
anthropicRedactedThinkingExpectedText,
anthropicRedactedThinkingExpectedTraceThinking,
anthropicRedactedThinkingStreamEvents,
} from '@/providers/__fixtures__/anthropic/redacted-thinking-signature'
export {
anthropicThinkingTextToolAssembledContent,
anthropicThinkingTextToolExpectedText,
anthropicThinkingTextToolExpectedThinking,
anthropicThinkingTextToolStreamEvents,
} from '@/providers/__fixtures__/anthropic/thinking-text-tool'
apps/sim/providers/__fixtures__/anthropic/redacted-thinking-signature.ts
+98
View File
@@ -0,0 +1,98 @@
/**
* Anthropic stream + assembled message covering redacted_thinking blocks.
*
* When Anthropic redacts thinking, the stream emits a redacted_thinking content
* block (opaque `data`) instead of thinking_delta text. Multi-turn tool loops
* must round-trip that block (and any adjacent signed thinking blocks) back
* into subsequent Messages API requests unchanged.
*/
export const anthropicRedactedThinkingStreamEvents = [
{
type: 'message_start',
message: {
id: 'msg_fixture_redacted_thinking',
type: 'message',
role: 'assistant',
content: [],
model: 'claude-sonnet-4-5',
stop_reason: null,
stop_sequence: null,
usage: { input_tokens: 20, output_tokens: 0 },
},
},
{
type: 'content_block_start',
index: 0,
content_block: {
type: 'redacted_thinking',
data: 'fixture-redacted-thinking-opaque-blob-001',
},
},
{ type: 'content_block_stop', index: 0 },
{
type: 'content_block_start',
index: 1,
content_block: {
type: 'thinking',
thinking: '',
},
},
{
type: 'content_block_delta',
index: 1,
delta: { type: 'thinking_delta', thinking: 'Visible follow-up reasoning after redaction.' },
},
{
type: 'content_block_delta',
index: 1,
delta: {
type: 'signature_delta',
signature: 'EpABCkYICBgCKkDfixture-visible-thinking-signature-def456',
},
},
{ type: 'content_block_stop', index: 1 },
{
type: 'content_block_start',
index: 2,
content_block: { type: 'text', text: '' },
},
{
type: 'content_block_delta',
index: 2,
delta: { type: 'text_delta', text: 'Here is the answer after redacted thinking.' },
},
{ type: 'content_block_stop', index: 2 },
{
type: 'message_delta',
delta: { stop_reason: 'end_turn', stop_sequence: null },
usage: { output_tokens: 64 },
},
{ type: 'message_stop' },
] as const
/**
* Assembled content that must be preserved when appending the assistant turn
* to Anthropic history (see anthropic/core.ts thinking/redacted_thinking filters).
*/
export const anthropicRedactedThinkingAssembledContent = [
{
type: 'redacted_thinking',
data: 'fixture-redacted-thinking-opaque-blob-001',
},
{
type: 'thinking',
thinking: 'Visible follow-up reasoning after redaction.',
signature: 'EpABCkYICBgCKkDfixture-visible-thinking-signature-def456',
},
{
type: 'text',
text: 'Here is the answer after redacted thinking.',
},
] as const
/** What enrichLastModelSegmentFromAnthropicResponse maps redacted blocks to today. */
export const anthropicRedactedThinkingExpectedTraceThinking =
'[redacted]\n\nVisible follow-up reasoning after redaction.'
export const anthropicRedactedThinkingExpectedText = 'Here is the answer after redacted thinking.'
apps/sim/providers/__fixtures__/anthropic/thinking-text-tool.ts
+118
View File
@@ -0,0 +1,118 @@
/**
* Anthropic Messages API SSE-style stream events for a single assistant turn that:
* 1. Streams extended thinking (thinking_delta + signature_delta)
* 2. Streams answer text (text_delta)
* 3. Streams a tool_use block (input_json_delta)
*
* Shapes mirror Anthropic RawMessageStreamEvent fields used by
* `createReadableStreamFromAnthropicStream`. The adapter emits thinking_delta +
* text_delta AgentStreamEvents; tool_use deltas are ignored until the streaming
* tool loop.
*/
export const anthropicThinkingTextToolStreamEvents = [
{
type: 'message_start',
message: {
id: 'msg_fixture_thinking_text_tool',
type: 'message',
role: 'assistant',
content: [],
model: 'claude-sonnet-4-5',
stop_reason: null,
stop_sequence: null,
usage: { input_tokens: 42, output_tokens: 0 },
},
},
{
type: 'content_block_start',
index: 0,
content_block: { type: 'thinking', thinking: '', signature: '' },
},
{
type: 'content_block_delta',
index: 0,
delta: { type: 'thinking_delta', thinking: 'I should check the weather before answering. ' },
},
{
type: 'content_block_delta',
index: 0,
delta: { type: 'thinking_delta', thinking: 'Calling get_weather for SF.' },
},
{
type: 'content_block_delta',
index: 0,
delta: {
type: 'signature_delta',
signature: 'EpABCkYICBgCKkDfixture-thinking-signature-abc123xyz',
},
},
{ type: 'content_block_stop', index: 0 },
{
type: 'content_block_start',
index: 1,
content_block: { type: 'text', text: '' },
},
{
type: 'content_block_delta',
index: 1,
delta: { type: 'text_delta', text: 'Let me check the weather in San Francisco.' },
},
{ type: 'content_block_stop', index: 1 },
{
type: 'content_block_start',
index: 2,
content_block: {
type: 'tool_use',
id: 'toolu_fixture_01Weather',
name: 'get_weather',
input: {},
},
},
{
type: 'content_block_delta',
index: 2,
delta: { type: 'input_json_delta', partial_json: '{"city":' },
},
{
type: 'content_block_delta',
index: 2,
delta: { type: 'input_json_delta', partial_json: '"San Francisco"}' },
},
{ type: 'content_block_stop', index: 2 },
{
type: 'message_delta',
delta: { stop_reason: 'tool_use', stop_sequence: null },
usage: { output_tokens: 128 },
},
{ type: 'message_stop' },
] as const
/**
* Expected assembled assistant Message.content after draining the stream above.
* Used for history round-trip tests (thinking block must keep its signature).
*/
export const anthropicThinkingTextToolAssembledContent = [
{
type: 'thinking',
thinking: 'I should check the weather before answering. Calling get_weather for SF.',
signature: 'EpABCkYICBgCKkDfixture-thinking-signature-abc123xyz',
},
{
type: 'text',
text: 'Let me check the weather in San Francisco.',
},
{
type: 'tool_use',
id: 'toolu_fixture_01Weather',
name: 'get_weather',
input: { city: 'San Francisco' },
},
] as const
/** Concatenated thinking text (what traces should store as thinkingContent). */
export const anthropicThinkingTextToolExpectedThinking =
'I should check the weather before answering. Calling get_weather for SF.'
/** Concatenated answer text (what output.content / live stream should contain today). */
export const anthropicThinkingTextToolExpectedText = 'Let me check the weather in San Francisco.'
apps/sim/providers/__fixtures__/openai-compat/index.ts
+69
View File
@@ -0,0 +1,69 @@
/**
* OpenAI-compat stream fixtures — capability-honest reasoning deltas.
*/
export const openaiCompatReasoningAndTextChunks = [
{
choices: [
{
delta: {
reasoning_content: 'I should compute carefully. ',
},
},
],
},
{
choices: [
{
delta: {
reasoning_content: 'Answer is 4.',
content: '2+2=',
},
},
],
},
{
choices: [{ delta: { content: '4' } }],
},
// Usage arrives on a trailing chunk with empty choices (stream_options.include_usage).
{
choices: [],
usage: { prompt_tokens: 10, completion_tokens: 8, total_tokens: 18 },
},
] as const
export const openaiCompatTextOnlyChunks = [
{
choices: [{ delta: { content: 'Hello' } }],
},
{
choices: [{ delta: { content: ' world' } }],
},
// Usage arrives on a trailing chunk with empty choices (stream_options.include_usage).
{
choices: [],
usage: { prompt_tokens: 3, completion_tokens: 2, total_tokens: 5 },
},
] as const
export const openaiCompatToolCallStartChunks = [
{
choices: [
{
delta: {
tool_calls: [
{ index: 0, id: 'call_abc', function: { name: 'http_request', arguments: '' } },
],
},
},
],
},
{
choices: [
{
delta: {
tool_calls: [{ index: 0, function: { arguments: '{"url":' } }],
},
},
],
},
] as const
apps/sim/providers/agent-events-smokes.test.ts
+67
View File
@@ -0,0 +1,67 @@
/**
* @vitest-environment node
*
* Agent-events capability smokes: Anthropic thinking+tool fixture, openai-compat thinking,
* and a non-thinking text-only stream — capability-honest expectations.
*/
import { describe, expect, it } from 'vitest'
import { anthropicThinkingTextToolStreamEvents } from '@/providers/__fixtures__/anthropic'
import {
openaiCompatReasoningAndTextChunks,
openaiCompatTextOnlyChunks,
} from '@/providers/__fixtures__/openai-compat'
import { createReadableStreamFromAnthropicStream } from '@/providers/anthropic/utils'
import { createOpenAICompatibleAgentEventStream } from '@/providers/openai-compat/stream-events'
import type { AgentStreamEvent } from '@/providers/stream-events'
async function collectEvents(
stream: ReadableStream<AgentStreamEvent>
): Promise<AgentStreamEvent[]> {
const events: AgentStreamEvent[] = []
const reader = stream.getReader()
while (true) {
const { done, value } = await reader.read()
if (done) break
events.push(value)
}
return events
}
describe('agent-events provider smokes', () => {
it('Anthropic fixture emits thinking then text (tools ignored on simple util)', async () => {
const stream = createReadableStreamFromAnthropicStream(
(async function* () {
yield* anthropicThinkingTextToolStreamEvents as any
})()
)
const events = await collectEvents(stream)
expect(events.some((e) => e.type === 'thinking_delta')).toBe(true)
expect(events.some((e) => e.type === 'text_delta')).toBe(true)
// Simple Anthropic util does not emit tool_call_* (loop does).
expect(events.some((e) => e.type === 'tool_call_start')).toBe(false)
})
it('openai-compat reasoning model emits thinking_delta', async () => {
const stream = createOpenAICompatibleAgentEventStream(
(async function* () {
yield* openaiCompatReasoningAndTextChunks as any
})(),
{ providerName: 'DeepSeek' }
)
const events = await collectEvents(stream)
expect(events.filter((e) => e.type === 'thinking_delta').length).toBeGreaterThan(0)
expect(events.some((e) => e.type === 'text_delta')).toBe(true)
})
it('openai-compat non-thinking model stays text-only', async () => {
const stream = createOpenAICompatibleAgentEventStream(
(async function* () {
yield* openaiCompatTextOnlyChunks as any
})(),
{ providerName: 'OpenAI' }
)
const events = await collectEvents(stream)
expect(events.every((e) => e.type === 'text_delta')).toBe(true)
expect(events.some((e) => e.type === 'thinking_delta')).toBe(false)
})
})
apps/sim/providers/anthropic/core.thinking.test.ts
+55
View File
@@ -0,0 +1,55 @@
/**
* @vitest-environment node
*
* Anthropic thinking config: the summarized-display opt-in is requested only
* on agent-events runs and only for models whose registry marks summarized
* streaming (the omitted-display Claude generations). Legacy runs keep the
* exact pre-agent-events request shape.
*/
import { describe, expect, it } from 'vitest'
import { buildThinkingConfig } from '@/providers/anthropic/core'
describe('buildThinkingConfig', () => {
it('requests summarized display for omitted-display models on agent-events runs', () => {
for (const model of [
'claude-fable-5',
'claude-sonnet-5',
'claude-opus-4-8',
'claude-opus-4-7',
]) {
const config = buildThinkingConfig(model, 'high', true)
expect(config?.thinking).toEqual({ type: 'adaptive', display: 'summarized' })
expect(config?.outputConfig).toEqual({ effort: 'high' })
}
})
it('never adds display on legacy runs (no agent events)', () => {
for (const model of [
'claude-fable-5',
'claude-sonnet-5',
'claude-opus-4-8',
'claude-opus-4-7',
]) {
const config = buildThinkingConfig(model, 'high', false)
expect(config?.thinking).toEqual({ type: 'adaptive' })
}
})
it('never adds display for adaptive models that already stream full thinking', () => {
for (const model of ['claude-opus-4-6', 'claude-sonnet-4-6']) {
const config = buildThinkingConfig(model, 'high', true)
expect(config?.thinking).toEqual({ type: 'adaptive' })
}
})
it('keeps budget-token models on the extended thinking path', () => {
const config = buildThinkingConfig('claude-sonnet-4-5', 'high', true)
expect(config?.thinking).toMatchObject({ type: 'enabled' })
expect(config?.thinking).not.toHaveProperty('display')
})
it('returns null for unknown levels and non-thinking models', () => {
expect(buildThinkingConfig('claude-fable-5', 'not-a-level', true)).toBeNull()
expect(buildThinkingConfig('gpt-4o', 'high', true)).toBeNull()
})
})
apps/sim/providers/anthropic/core.ts
+117 -15
View File
@@ -3,8 +3,14 @@ import { transformJSONSchema } from '@anthropic-ai/sdk/lib/transform-json-schema
import type { RawMessageStreamEvent } from '@anthropic-ai/sdk/resources/messages/messages'
import type { Logger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import type { BlockTokens, IterationToolCall, StreamingExecution } from '@/executor/types'
import type {
BlockTokens,
IterationToolCall,
NormalizedBlockOutput,
StreamingExecution,
} from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import { createAnthropicStreamingToolLoopStream } from '@/providers/anthropic/streaming-tool-loop'
import {
checkForForcedToolUsage,
createReadableStreamFromAnthropicStream,
@@ -45,12 +51,12 @@ export interface AnthropicProviderConfig {
/**
* Custom payload type extending the SDK's base message creation params.
* Adds fields not yet in the SDK: adaptive thinking, output_format, output_config.
* Message params plus `output_format`: Sim's structured outputs ride the
* anthropic-beta header with a top-level `output_format` field, which the SDK
* does not model (it exposes the newer `output_config.format` shape instead).
*/
interface AnthropicPayload extends Omit<Anthropic.Messages.MessageStreamParams, 'thinking'> {
thinking?: Anthropic.Messages.ThinkingConfigParam | { type: 'adaptive' }
interface AnthropicPayload extends Anthropic.Messages.MessageStreamParams {
output_format?: { type: 'json_schema'; schema: Record<string, unknown> }
output_config?: { effort: string }
}
/**
@@ -119,14 +125,21 @@ function supportsAdaptiveThinking(modelId: string): boolean {
* - Opus 4.6, Sonnet 4.6: Uses adaptive thinking with effort parameter
* - Other models: Uses budget_tokens-based extended thinking
*
* The newest Claude generations default `thinking.display` to `omitted`
* (empty thinking blocks, no thinking deltas). Their registry entries mark
* `capabilities.thinking.streamed: 'summary'`, and for those models Sim opts
* back in with `display: 'summarized'` — but only on agent-events runs, so
* legacy runs keep the exact pre-agent-events request shape.
*
* Returns both the thinking config and optional output_config for adaptive thinking.
*/
function buildThinkingConfig(
export function buildThinkingConfig(
modelId: string,
thinkingLevel: string
thinkingLevel: string,
agentEvents: boolean
): {
thinking: { type: 'enabled'; budget_tokens: number } | { type: 'adaptive' }
outputConfig?: { effort: string }
thinking: Anthropic.Messages.ThinkingConfigParam
outputConfig?: Anthropic.Messages.OutputConfig
} | null {
const capability = getThinkingCapability(modelId)
if (!capability || !capability.levels.includes(thinkingLevel)) {
@@ -135,9 +148,14 @@ function buildThinkingConfig(
// Models with effort support use adaptive thinking
if (supportsAdaptiveThinking(modelId)) {
const requestSummarizedDisplay = agentEvents && capability.streamed === 'summary'
return {
thinking: { type: 'adaptive' },
outputConfig: { effort: thinkingLevel },
thinking: {
type: 'adaptive',
...(requestSummarizedDisplay ? { display: 'summarized' as const } : {}),
},
// Levels are validated against the model's capability list above.
outputConfig: { effort: thinkingLevel as Anthropic.Messages.OutputConfig['effort'] },
}
}
@@ -337,7 +355,11 @@ export async function executeAnthropicProviderRequest(
// Add extended thinking configuration if supported and requested
// The 'none' sentinel means "disable thinking" — skip configuration entirely.
if (request.thinkingLevel && request.thinkingLevel !== 'none') {
const thinkingConfig = buildThinkingConfig(request.model, request.thinkingLevel)
const thinkingConfig = buildThinkingConfig(
request.model,
request.thinkingLevel,
request.agentEvents === true
)
if (thinkingConfig) {
payload.thinking = thinkingConfig.thinking
if (thinkingConfig.outputConfig) {
@@ -403,6 +425,56 @@ export async function executeAnthropicProviderRequest(
const shouldStreamToolCalls = request.streamToolCalls ?? false
if (request.stream && shouldStreamToolCalls && anthropicTools && anthropicTools.length > 0) {
logger.info(`Using streaming tool loop for ${providerLabel} request`)
const providerStartTime = Date.now()
const providerStartTimeISO = new Date(providerStartTime).toISOString()
const timeSegments: TimeSegment[] = []
const forcedTools = preparedTools?.forcedTools || []
return createStreamingExecution({
model: request.model,
providerStartTime,
providerStartTimeISO,
timing: {
kind: 'accumulated',
modelTime: 0,
toolsTime: 0,
firstResponseTime: 0,
iterations: 1,
timeSegments,
},
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { total: 0.0, input: 0.0, output: 0.0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createAnthropicStreamingToolLoopStream({
anthropic,
payload,
request,
messages,
logger,
timeSegments,
forcedTools,
onComplete: (result) => {
output.content = result.content
output.tokens = result.tokens
output.cost = result.cost
output.toolCalls = result.toolCalls as NormalizedBlockOutput['toolCalls']
if (output.providerTiming) {
output.providerTiming.modelTime = result.modelTime
output.providerTiming.toolsTime = result.toolsTime
output.providerTiming.firstResponseTime = result.firstResponseTime
output.providerTiming.iterations = result.iterations
}
finalizeTiming()
},
}),
})
}
if (request.stream && (!anthropicTools || anthropicTools.length === 0)) {
logger.info(`Using streaming response for ${providerLabel} request (no tools)`)
@@ -425,10 +497,11 @@ export async function executeAnthropicProviderRequest(
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { total: 0.0, input: 0.0, output: 0.0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromAnthropicStream(
streamResponse as AsyncIterable<RawMessageStreamEvent>,
(content, usage) => {
({ content, usage, thinking }) => {
output.content = content
output.tokens = {
input: usage.input_tokens,
@@ -443,6 +516,13 @@ export async function executeAnthropicProviderRequest(
total: costResult.total,
}
if (thinking) {
const segment = output.providerTiming?.timeSegments?.[0]
if (segment) {
segment.thinkingContent = thinking
}
}
finalizeTiming()
}
),
@@ -805,10 +885,11 @@ export async function executeAnthropicProviderRequest(
},
toolCalls: toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromAnthropicStream(
streamResponse as AsyncIterable<RawMessageStreamEvent>,
(streamContent, usage) => {
({ content: streamContent, usage, thinking }) => {
output.content = streamContent
output.tokens = {
input: tokens.input + usage.input_tokens,
@@ -829,6 +910,16 @@ export async function executeAnthropicProviderRequest(
total: accumulatedCost.total + streamCost.total + tc,
}
if (thinking) {
const segments = output.providerTiming?.timeSegments
const lastModel = segments
? [...segments].reverse().find((segment) => segment.type === 'model')
: undefined
if (lastModel) {
lastModel.thinkingContent = thinking
}
}
finalizeTiming()
}
),
@@ -1228,10 +1319,11 @@ export async function executeAnthropicProviderRequest(
},
toolCalls: toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromAnthropicStream(
streamResponse as AsyncIterable<RawMessageStreamEvent>,
(streamContent, usage) => {
({ content: streamContent, usage, thinking }) => {
output.content = streamContent
output.tokens = {
input: tokens.input + usage.input_tokens,
@@ -1252,6 +1344,16 @@ export async function executeAnthropicProviderRequest(
total: cost.total + streamCost.total + tc2,
}
if (thinking) {
const segments = output.providerTiming?.timeSegments
const lastModel = segments
? [...segments].reverse().find((segment) => segment.type === 'model')
: undefined
if (lastModel) {
lastModel.thinkingContent = thinking
}
}
finalizeTiming()
}
),
apps/sim/providers/anthropic/streaming-tool-loop.test.ts
+341
View File
@@ -0,0 +1,341 @@
/**
* @vitest-environment node
*
* Anthropic streaming tool loop — live tool_call_start/end, live `pending`
* text classified by turn_end, abort → cancelled, per-turn usage accumulation.
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
import {
anthropicThinkingTextToolExpectedThinking,
anthropicThinkingTextToolStreamEvents,
} from '@/providers/__fixtures__/anthropic'
import { createAnthropicStreamingToolLoopStream } from '@/providers/anthropic/streaming-tool-loop'
import type { AgentStreamEvent } from '@/providers/stream-events'
import type { TimeSegment } from '@/providers/types'
const { mockExecuteTool, mockPrepareToolExecution } = vi.hoisted(() => ({
mockExecuteTool: vi.fn(),
mockPrepareToolExecution: vi.fn(),
}))
vi.mock('@/tools', () => ({
executeTool: mockExecuteTool,
}))
vi.mock('@/providers/utils', () => ({
prepareToolExecution: mockPrepareToolExecution,
calculateCost: () => ({ input: 0.01, output: 0.02, total: 0.03 }),
sumToolCosts: () => 0,
trackForcedToolUsage: () => ({ hasUsedForcedTool: false, usedForcedTools: [] }),
}))
async function collectEvents(
stream: ReadableStream<AgentStreamEvent>
): Promise<AgentStreamEvent[]> {
const events: AgentStreamEvent[] = []
const reader = stream.getReader()
while (true) {
const { done, value } = await reader.read()
if (done) break
events.push(value)
}
return events
}
function makeFinalMessage(overrides: {
content: unknown[]
usage?: { input_tokens: number; output_tokens: number }
stop_reason?: string | null
}) {
return {
id: 'msg_test',
type: 'message',
role: 'assistant',
model: 'claude-sonnet-4-5',
content: overrides.content,
stop_reason: overrides.stop_reason ?? null,
stop_sequence: null,
usage: overrides.usage ?? { input_tokens: 10, output_tokens: 20 },
}
}
function makeMessageStream(events: unknown[], finalMessage: ReturnType<typeof makeFinalMessage>) {
return {
async *[Symbol.asyncIterator]() {
for (const event of events) {
yield event
}
},
finalMessage: async () => finalMessage,
}
}
describe('createAnthropicStreamingToolLoopStream', () => {
const logger = {
info: vi.fn(),
warn: vi.fn(),
error: vi.fn(),
debug: vi.fn(),
} as any
beforeEach(() => {
vi.clearAllMocks()
mockPrepareToolExecution.mockReturnValue({
toolParams: { city: 'San Francisco' },
executionParams: { city: 'San Francisco' },
})
mockExecuteTool.mockResolvedValue({
success: true,
output: { temp: 68 },
})
})
it('emits tool_call_start/end, live pending text with turn_end classification, and accumulates usage', async () => {
const toolTurnEvents = anthropicThinkingTextToolStreamEvents
const toolTurnMessage = makeFinalMessage({
content: [
{
type: 'thinking',
thinking: anthropicThinkingTextToolExpectedThinking,
signature: 'EpABCkYICBgCKkDfixture-thinking-signature-abc123xyz',
},
{ type: 'text', text: 'Let me check the weather in San Francisco.' },
{
type: 'tool_use',
id: 'toolu_fixture_01Weather',
name: 'get_weather',
input: { city: 'San Francisco' },
},
],
usage: { input_tokens: 42, output_tokens: 30 },
stop_reason: 'tool_use',
})
const finalTurnEvents = [
{
type: 'message_start',
message: {
usage: { input_tokens: 100, output_tokens: 0 },
},
},
{
type: 'content_block_start',
index: 0,
content_block: { type: 'text', text: '' },
},
{
type: 'content_block_delta',
index: 0,
delta: { type: 'text_delta', text: 'It is 68°F in San Francisco.' },
},
{ type: 'content_block_stop', index: 0 },
{
type: 'message_delta',
delta: { stop_reason: 'end_turn' },
usage: { output_tokens: 12 },
},
{ type: 'message_stop' },
]
const finalTurnMessage = makeFinalMessage({
content: [{ type: 'text', text: 'It is 68°F in San Francisco.' }],
usage: { input_tokens: 100, output_tokens: 12 },
stop_reason: 'end_turn',
})
let streamCall = 0
const anthropic = {
messages: {
stream: vi.fn(() => {
streamCall++
if (streamCall === 1) {
return makeMessageStream(toolTurnEvents as unknown[], toolTurnMessage)
}
return makeMessageStream(finalTurnEvents, finalTurnMessage)
}),
},
} as any
const timeSegments: TimeSegment[] = []
const onComplete = vi.fn()
const stream = createAnthropicStreamingToolLoopStream({
anthropic,
payload: {
model: 'claude-sonnet-4-5',
max_tokens: 1024,
messages: [{ role: 'user', content: 'Weather?' }],
tools: [
{
name: 'get_weather',
description: 'Get weather',
input_schema: { type: 'object', properties: {} },
},
],
} as any,
request: {
model: 'claude-sonnet-4-5',
apiKey: 'test',
tools: [{ id: 'get_weather', name: 'get_weather', params: {}, parameters: {} }],
} as any,
messages: [{ role: 'user', content: 'Weather?' }],
logger,
timeSegments,
onComplete,
})
const events = await collectEvents(stream)
expect(events.filter((e) => e.type === 'thinking_delta').length).toBeGreaterThan(0)
expect(events).toContainEqual({
type: 'tool_call_start',
id: 'toolu_fixture_01Weather',
name: 'get_weather',
})
expect(events).toContainEqual({
type: 'tool_call_end',
id: 'toolu_fixture_01Weather',
name: 'get_weather',
status: 'success',
})
// All text streams live as `pending`; the pump classifies via turn_end.
const textEvents = events.filter((e) => e.type === 'text_delta')
expect(textEvents.every((e) => e.turn === 'pending')).toBe(true)
expect(textEvents.some((e) => e.text.includes('Let me check'))).toBe(true)
expect(textEvents.some((e) => e.text.includes('68°F'))).toBe(true)
const turnEnds = events.filter((e) => e.type === 'turn_end')
expect(turnEnds.map((e) => e.turn)).toEqual(['intermediate', 'final'])
// Ordering: the tool turn's pending text precedes its intermediate turn_end,
// and the final turn's text precedes the final turn_end.
const eventKinds = events.map((e) =>
e.type === 'turn_end' ? `turn_end:${e.turn}` : e.type === 'text_delta' ? 'text' : e.type
)
expect(eventKinds.indexOf('text')).toBeLessThan(eventKinds.indexOf('turn_end:intermediate'))
expect(eventKinds.lastIndexOf('text')).toBeLessThan(eventKinds.indexOf('turn_end:final'))
// Assistant history must keep thinking signature for multi-iteration round-trip.
const secondPayload = anthropic.messages.stream.mock.calls[1][0]
const assistantMsg = secondPayload.messages.find((m: any) => m.role === 'assistant')
expect(assistantMsg.content.some((b: any) => b.type === 'thinking' && b.signature)).toBe(true)
expect(assistantMsg.content.some((b: any) => b.type === 'tool_use')).toBe(true)
expect(onComplete).toHaveBeenCalledTimes(1)
expect(onComplete.mock.calls[0][0].tokens).toEqual({
input: 142,
output: 42,
total: 184,
})
expect(onComplete.mock.calls[0][0].content).toContain('68°F')
expect(mockExecuteTool).toHaveBeenCalled()
})
it('settles in-flight tools as cancelled on abort', async () => {
const abortController = new AbortController()
const toolStartEvents = [
{
type: 'message_start',
message: { usage: { input_tokens: 5, output_tokens: 0 } },
},
{
type: 'content_block_start',
index: 0,
content_block: {
type: 'tool_use',
id: 'toolu_abort',
name: 'get_weather',
input: {},
},
},
{
type: 'content_block_delta',
index: 0,
delta: { type: 'input_json_delta', partial_json: '{}' },
},
{ type: 'content_block_stop', index: 0 },
{
type: 'message_delta',
delta: { stop_reason: 'tool_use' },
usage: { output_tokens: 3 },
},
{ type: 'message_stop' },
]
mockExecuteTool.mockImplementation(async () => {
abortController.abort()
throw new DOMException('Stream aborted', 'AbortError')
})
const anthropic = {
messages: {
stream: vi.fn(() =>
makeMessageStream(
toolStartEvents,
makeFinalMessage({
content: [
{
type: 'tool_use',
id: 'toolu_abort',
name: 'get_weather',
input: {},
},
],
stop_reason: 'tool_use',
})
)
),
},
} as any
const stream = createAnthropicStreamingToolLoopStream({
anthropic,
payload: {
model: 'claude-sonnet-4-5',
max_tokens: 1024,
messages: [{ role: 'user', content: 'x' }],
tools: [
{
name: 'get_weather',
description: 'd',
input_schema: { type: 'object', properties: {} },
},
],
} as any,
request: {
model: 'claude-sonnet-4-5',
apiKey: 'test',
tools: [{ id: 'get_weather', name: 'get_weather', params: {}, parameters: {} }],
abortSignal: abortController.signal,
} as any,
messages: [{ role: 'user', content: 'x' }],
logger,
timeSegments: [],
onComplete: vi.fn(),
})
const captured: AgentStreamEvent[] = []
const reader = stream.getReader()
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
captured.push(value)
}
} catch {
// expected — stream errors after abort settlement
}
expect(captured).toContainEqual({
type: 'tool_call_start',
id: 'toolu_abort',
name: 'get_weather',
})
expect(captured).toContainEqual({
type: 'tool_call_end',
id: 'toolu_abort',
name: 'get_weather',
status: 'cancelled',
})
})
})
apps/sim/providers/anthropic/streaming-tool-loop.ts
+544
View File
@@ -0,0 +1,544 @@
/**
* Live Anthropic streaming tool loop.
*
* Each model turn is streamed via `messages.stream` + `finalMessage()` so thinking
* signatures round-trip correctly. Thinking, `tool_call_start`, and `pending`
* text deltas emit live; a `turn_end` event classifies the turn as
* `intermediate` (tool-use turns) or `final` so the pump projects only final
* text to the answer channel. Tool ends emit in actual completion order; abort
* settles in-flight tools as `cancelled`.
*/
import type Anthropic from '@anthropic-ai/sdk'
import type { Logger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import type { BlockTokens, IterationToolCall } from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import { checkForForcedToolUsage } from '@/providers/anthropic/utils'
import type { AgentStreamEvent, ToolCallEndStatus } from '@/providers/stream-events'
import {
isAbortError,
type StreamingToolLoopComplete,
settleOpenTools,
} from '@/providers/streaming-tool-loop-shared'
import { enrichLastModelSegment } from '@/providers/trace-enrichment'
import type { ProviderRequest, TimeSegment } from '@/providers/types'
import { calculateCost, prepareToolExecution, sumToolCosts } from '@/providers/utils'
import { executeTool } from '@/tools'
/**
* Message params plus `output_format`, shared with `core.ts`: Sim's structured
* outputs ride the anthropic-beta header with a top-level `output_format`
* field, which the SDK does not model (it exposes `output_config.format`).
*/
export type AnthropicStreamingToolLoopPayload = Anthropic.Messages.MessageStreamParams & {
output_format?: { type: 'json_schema'; schema: Record<string, unknown> }
}
export interface CreateAnthropicStreamingToolLoopStreamOptions {
anthropic: Anthropic
payload: AnthropicStreamingToolLoopPayload
request: ProviderRequest
messages: Anthropic.Messages.MessageParam[]
logger: Logger
/** Shared mutable segments; same array reference passed into createStreamingExecution. */
timeSegments: TimeSegment[]
/** Forced tool names from prepareToolsWithUsageControl (may be empty). */
forcedTools?: string[]
onComplete: (result: StreamingToolLoopComplete) => void
}
function buildSegmentTokens(usage: Anthropic.Messages.Usage): BlockTokens {
const input = usage.input_tokens ?? 0
const output = usage.output_tokens ?? 0
const cacheRead = usage.cache_read_input_tokens ?? 0
const cacheWrite = usage.cache_creation_input_tokens ?? 0
return {
input,
output,
total: input + output + cacheRead + cacheWrite,
...(cacheRead > 0 && { cacheRead }),
...(cacheWrite > 0 && { cacheWrite }),
}
}
function enrichModelSegment(
timeSegments: TimeSegment[],
response: Anthropic.Messages.Message,
textContent: string,
model: string
): void {
const thinkingBlocks = response.content.filter(
(item): item is Anthropic.Messages.ThinkingBlock | Anthropic.Messages.RedactedThinkingBlock =>
item.type === 'thinking' || item.type === 'redacted_thinking'
)
const thinkingContent = thinkingBlocks
.map((b) => (b.type === 'thinking' ? b.thinking : '[redacted]'))
.join('\n\n')
const toolUseBlocks = response.content.filter(
(item): item is Anthropic.Messages.ToolUseBlock => item.type === 'tool_use'
)
const toolCalls: IterationToolCall[] = toolUseBlocks.map((t) => ({
id: t.id,
name: t.name,
arguments:
t.input && typeof t.input === 'object' && !Array.isArray(t.input)
? (t.input as Record<string, unknown>)
: {},
}))
const segmentTokens = response.usage ? buildSegmentTokens(response.usage) : undefined
let cost: { input: number; output: number; total: number } | undefined
if (
segmentTokens &&
typeof segmentTokens.input === 'number' &&
typeof segmentTokens.output === 'number'
) {
const useCached = (segmentTokens.cacheRead ?? 0) > 0
const full = calculateCost(model, segmentTokens.input, segmentTokens.output, useCached)
cost = { input: full.input, output: full.output, total: full.total }
}
enrichLastModelSegment(timeSegments, {
assistantContent: textContent || undefined,
thinkingContent: thinkingContent || undefined,
toolCalls: toolCalls.length > 0 ? toolCalls : undefined,
finishReason: response.stop_reason ?? undefined,
tokens: segmentTokens,
cost,
provider: 'anthropic',
})
}
/**
* Multi-turn Anthropic tool loop as an `agent-events-v1` object stream.
*/
export function createAnthropicStreamingToolLoopStream(
options: CreateAnthropicStreamingToolLoopStreamOptions
): ReadableStream<AgentStreamEvent> {
const { anthropic, payload, request, messages, logger, timeSegments, onComplete } = options
const forcedToolNames = options.forcedTools ?? []
return new ReadableStream<AgentStreamEvent>({
async start(controller) {
const currentMessages = [...messages]
const originalToolChoice = payload.tool_choice
let usedForcedTools: string[] = []
let hasUsedForcedTool = false
let content = ''
let iterationCount = 0
let modelCalls = 0
let sawFinalTurn = false
let modelTime = 0
let toolsTime = 0
let firstResponseTime = 0
const tokens = { input: 0, output: 0, total: 0 }
const toolCalls: unknown[] = []
const toolResults: Record<string, unknown>[] = []
/** Tools that received start but not yet end (abort settlement). */
const openToolStarts = new Map<string, string>()
const streamOptions = request.abortSignal ? { signal: request.abortSignal } : undefined
try {
while (iterationCount < MAX_TOOL_ITERATIONS) {
if (request.abortSignal?.aborted) {
const abortErr = new DOMException('Stream aborted', 'AbortError')
settleOpenTools(controller, openToolStarts, 'cancelled')
throw abortErr
}
const turnPayload: AnthropicStreamingToolLoopPayload = {
...payload,
messages: currentMessages,
}
// Streaming tool loop always streams each turn; never pass stream:true twice.
;(turnPayload as { stream?: boolean }).stream = undefined
// Forced tool_choice vs thinking — same rules as silent loop.
const thinkingEnabled = !!payload.thinking
if (
!thinkingEnabled &&
typeof originalToolChoice === 'object' &&
hasUsedForcedTool &&
forcedToolNames.length > 0
) {
const remainingTools = forcedToolNames.filter((tool) => !usedForcedTools.includes(tool))
if (remainingTools.length > 0) {
turnPayload.tool_choice = { type: 'tool', name: remainingTools[0] }
} else {
turnPayload.tool_choice = undefined
}
} else if (
!thinkingEnabled &&
hasUsedForcedTool &&
typeof originalToolChoice === 'object'
) {
turnPayload.tool_choice = undefined
}
const modelStart = Date.now()
const messageStream = anthropic.messages.stream(turnPayload, streamOptions)
const textChunks: string[] = []
let inputTokens = 0
let outputTokens = 0
try {
for await (const event of messageStream) {
if (event.type === 'message_start') {
inputTokens = event.message.usage?.input_tokens ?? 0
continue
}
if (event.type === 'message_delta') {
outputTokens = event.usage?.output_tokens ?? outputTokens
continue
}
if (event.type === 'content_block_start') {
const block = event.content_block
if (block.type === 'tool_use' && block.id && block.name) {
openToolStarts.set(block.id, block.name)
controller.enqueue({
type: 'tool_call_start',
id: block.id,
name: block.name,
})
}
continue
}
if (event.type !== 'content_block_delta') {
continue
}
const delta = event.delta
if (delta.type === 'thinking_delta' && typeof delta.thinking === 'string') {
controller.enqueue({ type: 'thinking_delta', text: delta.thinking })
continue
}
if (delta.type === 'text_delta' && typeof delta.text === 'string') {
textChunks.push(delta.text)
// Live pending text: sinks render it now; the pump projects it
// to the answer only when this turn's turn_end says 'final'.
controller.enqueue({ type: 'text_delta', text: delta.text, turn: 'pending' })
}
}
const finalMessage = await messageStream.finalMessage()
const modelEnd = Date.now()
const thisModelTime = modelEnd - modelStart
modelTime += thisModelTime
modelCalls++
if (iterationCount === 0) {
firstResponseTime = thisModelTime
}
timeSegments.push({
type: 'model',
name: request.model,
startTime: modelStart,
endTime: modelEnd,
duration: thisModelTime,
})
// Prefer finalMessage.usage when present (includes cache fields).
const turnInput = finalMessage.usage?.input_tokens ?? inputTokens
const turnOutput = finalMessage.usage?.output_tokens ?? outputTokens
tokens.input += turnInput
tokens.output += turnOutput
tokens.total += turnInput + turnOutput
const textContent = finalMessage.content
.filter((item): item is Anthropic.Messages.TextBlock => item.type === 'text')
.map((item) => item.text)
.join('\n')
const toolUses = finalMessage.content.filter(
(item): item is Anthropic.Messages.ToolUseBlock => item.type === 'tool_use'
)
/**
* Only execute tools when the model actually stopped to call them.
* On `max_tokens` / `malformed_tool_use` the assembled inputs are
* truncated best-effort JSON — running tools on them would execute
* with wrong or partial arguments.
*/
const toolsExecutable = finalMessage.stop_reason === 'tool_use'
if (toolUses.length > 0 && !toolsExecutable) {
logger.warn('Skipping tool execution for incomplete turn', {
stopReason: finalMessage.stop_reason,
toolCount: toolUses.length,
})
settleOpenTools(controller, openToolStarts, 'error')
}
const executableToolUses = toolsExecutable ? toolUses : []
const turnTag = executableToolUses.length > 0 ? 'intermediate' : 'final'
// If the SDK assembled text but we somehow missed deltas, still emit it
// before the boundary so the turn_end classification covers it.
if (textChunks.length === 0 && textContent) {
controller.enqueue({ type: 'text_delta', text: textContent, turn: 'pending' })
}
controller.enqueue({ type: 'turn_end', turn: turnTag })
if (textChunks.length > 0 || textContent) {
// Streamed deltas are the answer bytes; fall back to assembled text.
// Intermediate text is kept so a MAX_TOOL_ITERATIONS exit still has content.
content = textChunks.length > 0 ? textChunks.join('') : textContent
}
enrichModelSegment(timeSegments, finalMessage, textContent, request.model)
const forcedCheck = checkForForcedToolUsage(
finalMessage,
turnPayload.tool_choice,
forcedToolNames,
usedForcedTools
)
if (forcedCheck) {
hasUsedForcedTool = forcedCheck.hasUsedForcedTool
usedForcedTools = forcedCheck.usedForcedTools
}
if (executableToolUses.length === 0) {
sawFinalTurn = true
break
}
const toolsStartTime = Date.now()
// Emit ends in completion order; keep Promise.all result order (= start order) for history.
const orderedResults = await Promise.all(
executableToolUses.map(async (toolUse) => {
const toolCallStartTime = Date.now()
const toolName = toolUse.name
const toolArgs = (toolUse.input ?? {}) as Record<string, unknown>
try {
if (request.abortSignal?.aborted) {
throw new DOMException('Stream aborted', 'AbortError')
}
const tool = request.tools?.find((t) => t.id === toolName)
if (!tool) {
const value = {
toolUse,
toolName,
toolArgs,
toolParams: {} as Record<string, unknown>,
result: {
success: false as const,
output: undefined,
error: `Tool not found: ${toolName}`,
},
startTime: toolCallStartTime,
endTime: Date.now(),
duration: Date.now() - toolCallStartTime,
status: 'error' as ToolCallEndStatus,
}
openToolStarts.delete(toolUse.id)
controller.enqueue({
type: 'tool_call_end',
id: toolUse.id,
name: toolName,
status: 'error',
})
return value
}
const { toolParams, executionParams } = prepareToolExecution(
tool,
toolArgs,
request
)
const result = await executeTool(toolName, executionParams, {
signal: request.abortSignal,
})
const toolCallEndTime = Date.now()
const value = {
toolUse,
toolName,
toolArgs,
toolParams,
result,
startTime: toolCallStartTime,
endTime: toolCallEndTime,
duration: toolCallEndTime - toolCallStartTime,
status: (result.success ? 'success' : 'error') as ToolCallEndStatus,
}
openToolStarts.delete(toolUse.id)
controller.enqueue({
type: 'tool_call_end',
id: toolUse.id,
name: toolName,
status: value.status,
})
return value
} catch (error) {
const toolCallEndTime = Date.now()
const cancelled = isAbortError(error) || !!request.abortSignal?.aborted
if (!cancelled) {
logger.error('Error processing tool call:', { error, toolName })
}
const value = {
toolUse,
toolName,
toolArgs,
toolParams: {} as Record<string, unknown>,
result: {
success: false as const,
output: undefined,
error: getErrorMessage(error, 'Tool execution failed'),
},
startTime: toolCallStartTime,
endTime: toolCallEndTime,
duration: toolCallEndTime - toolCallStartTime,
status: (cancelled ? 'cancelled' : 'error') as ToolCallEndStatus,
}
openToolStarts.delete(toolUse.id)
controller.enqueue({
type: 'tool_call_end',
id: toolUse.id,
name: toolName,
status: value.status,
})
return value
}
})
)
const toolUseBlocks: Anthropic.Messages.ToolUseBlockParam[] = []
const toolResultBlocks: Anthropic.Messages.ToolResultBlockParam[] = []
for (const value of orderedResults) {
const {
toolUse,
toolName,
toolArgs,
toolParams,
result,
startTime,
endTime,
duration,
} = value
timeSegments.push({
type: 'tool',
name: toolName,
startTime,
endTime,
duration,
toolCallId: toolUse.id,
})
let resultContent: unknown
if (result.success && result.output) {
toolResults.push(result.output as Record<string, unknown>)
resultContent = result.output
} else {
resultContent = {
error: true,
message: result.error || 'Tool execution failed',
tool: toolName,
}
}
toolCalls.push({
name: toolName,
arguments: toolParams,
startTime: new Date(startTime).toISOString(),
endTime: new Date(endTime).toISOString(),
duration,
result: resultContent,
success: result.success,
})
toolUseBlocks.push({
type: 'tool_use',
id: toolUse.id,
name: toolName,
input: toolArgs,
})
toolResultBlocks.push({
type: 'tool_result',
tool_use_id: toolUse.id,
content: JSON.stringify(resultContent),
})
}
const thinkingBlocks = finalMessage.content.filter(
(
item
): item is
| Anthropic.Messages.ThinkingBlock
| Anthropic.Messages.RedactedThinkingBlock =>
item.type === 'thinking' || item.type === 'redacted_thinking'
)
if (toolUseBlocks.length > 0) {
currentMessages.push({
role: 'assistant',
content: [
...thinkingBlocks,
...toolUseBlocks,
] as Anthropic.Messages.ContentBlockParam[],
})
}
if (toolResultBlocks.length > 0) {
currentMessages.push({
role: 'user',
content: toolResultBlocks as Anthropic.Messages.ContentBlockParam[],
})
}
toolsTime += Date.now() - toolsStartTime
iterationCount++
if (request.abortSignal?.aborted) {
settleOpenTools(controller, openToolStarts, 'cancelled')
throw new DOMException('Stream aborted', 'AbortError')
}
} catch (error) {
settleOpenTools(controller, openToolStarts, isAbortError(error) ? 'cancelled' : 'error')
throw error
}
}
/**
* MAX_TOOL_ITERATIONS exit: every turn was tagged intermediate, so the
* answer channel would otherwise be empty. Flush the last turn's text
* as the final answer so legacy consumers still receive content.
*/
if (!sawFinalTurn && content) {
controller.enqueue({ type: 'text_delta', text: content, turn: 'final' })
}
const modelCost = calculateCost(request.model, tokens.input, tokens.output)
const toolCostTotal = sumToolCosts(toolResults)
const cost = {
input: modelCost.input,
output: modelCost.output,
total: modelCost.total + (toolCostTotal || 0),
...(toolCostTotal ? { toolCost: toolCostTotal } : {}),
}
onComplete({
content,
tokens,
cost,
toolCalls:
toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
modelTime,
toolsTime,
firstResponseTime,
iterations: modelCalls,
})
controller.close()
} catch (error) {
const cancelled = isAbortError(error)
settleOpenTools(controller, openToolStarts, cancelled ? 'cancelled' : 'error')
controller.error(toError(error))
}
},
})
}
apps/sim/providers/anthropic/utils.test.ts
+106
View File
@@ -0,0 +1,106 @@
/**
* @vitest-environment node
*
* Anthropic adapter emits AgentStreamEvent objects (thinking + text)
* from Messages stream fixtures; tool_use deltas are handled by the tool loop.
*/
import { describe, expect, it, vi } from 'vitest'
import {
anthropicRedactedThinkingExpectedText,
anthropicRedactedThinkingExpectedTraceThinking,
anthropicRedactedThinkingStreamEvents,
anthropicThinkingTextToolExpectedText,
anthropicThinkingTextToolExpectedThinking,
anthropicThinkingTextToolStreamEvents,
} from '@/providers/__fixtures__/anthropic'
import { createReadableStreamFromAnthropicStream } from '@/providers/anthropic/utils'
import type { AgentStreamEvent } from '@/providers/stream-events'
async function collectEvents(
stream: ReadableStream<AgentStreamEvent>
): Promise<AgentStreamEvent[]> {
const events: AgentStreamEvent[] = []
const reader = stream.getReader()
while (true) {
const { done, value } = await reader.read()
if (done) break
events.push(value)
}
return events
}
describe('createReadableStreamFromAnthropicStream', () => {
it('emits thinking_delta then text_delta and ignores tool_use (thinking+text+tool fixture)', async () => {
const onComplete = vi.fn()
const stream = createReadableStreamFromAnthropicStream(
(async function* () {
yield* anthropicThinkingTextToolStreamEvents
})() as AsyncIterable<any>,
onComplete
)
const events = await collectEvents(stream)
expect(events.filter((e) => e.type === 'thinking_delta').map((e) => e.text)).toEqual([
'I should check the weather before answering. ',
'Calling get_weather for SF.',
])
expect(events.filter((e) => e.type === 'text_delta')).toEqual([
{
type: 'text_delta',
text: anthropicThinkingTextToolExpectedText,
turn: 'final',
},
])
expect(events.some((e) => e.type === 'tool_call_start' || e.type === 'tool_call_end')).toBe(
false
)
expect(onComplete).toHaveBeenCalledTimes(1)
expect(onComplete.mock.calls[0][0]).toMatchObject({
content: anthropicThinkingTextToolExpectedText,
thinking: anthropicThinkingTextToolExpectedThinking,
usage: { input_tokens: 42, output_tokens: expect.any(Number) },
})
})
it('records [redacted] for redacted_thinking blocks and streams text', async () => {
const onComplete = vi.fn()
const stream = createReadableStreamFromAnthropicStream(
(async function* () {
yield* anthropicRedactedThinkingStreamEvents
})() as AsyncIterable<any>,
onComplete
)
const events = await collectEvents(stream)
expect(events.filter((e) => e.type === 'thinking_delta').map((e) => e.text)).toEqual([
'Visible follow-up reasoning after redaction.',
])
expect(events.filter((e) => e.type === 'text_delta')).toEqual([
{
type: 'text_delta',
text: anthropicRedactedThinkingExpectedText,
turn: 'final',
},
])
expect(onComplete.mock.calls[0][0]).toMatchObject({
content: anthropicRedactedThinkingExpectedText,
thinking: anthropicRedactedThinkingExpectedTraceThinking,
})
})
it('errors the readable stream when the source throws', async () => {
const stream = createReadableStreamFromAnthropicStream(
(async function* () {
yield {
type: 'content_block_delta',
delta: { type: 'text_delta', text: 'partial' },
}
throw new Error('provider reset')
})() as AsyncIterable<any>
)
await expect(collectEvents(stream)).rejects.toThrow('provider reset')
})
})
apps/sim/providers/anthropic/utils.ts
+81 -26
View File
@@ -1,11 +1,7 @@
import type {
RawMessageDeltaEvent,
RawMessageStartEvent,
RawMessageStreamEvent,
Usage,
} from '@anthropic-ai/sdk/resources'
import type { RawMessageStreamEvent } from '@anthropic-ai/sdk/resources'
import { createLogger } from '@sim/logger'
import { randomFloat } from '@sim/utils/random'
import type { AgentStreamEvent } from '@/providers/stream-events'
import { trackForcedToolUsage } from '@/providers/utils'
const logger = createLogger('AnthropicUtils')
@@ -15,34 +11,93 @@ export interface AnthropicStreamUsage {
output_tokens: number
}
export interface AnthropicStreamComplete {
content: string
usage: AnthropicStreamUsage
/** Assembled thinking text for traces (redacted blocks become `[redacted]`). */
thinking: string
}
/**
* Converts an Anthropic Messages stream into an in-process
* {@link AgentStreamEvent} object stream (`thinking_delta` + `text_delta`).
* Tool_use / input_json deltas are ignored here — use
* {@link createAnthropicStreamingToolLoopStream} for the live tool loop.
*/
export function createReadableStreamFromAnthropicStream(
anthropicStream: AsyncIterable<RawMessageStreamEvent>,
onComplete?: (content: string, usage: AnthropicStreamUsage) => void
): ReadableStream<Uint8Array> {
let fullContent = ''
let inputTokens = 0
let outputTokens = 0
return new ReadableStream({
onComplete?: (result: AnthropicStreamComplete) => void
): ReadableStream<AgentStreamEvent> {
return new ReadableStream<AgentStreamEvent>({
async start(controller) {
try {
for await (const event of anthropicStream) {
if (event.type === 'message_start') {
const startEvent = event as RawMessageStartEvent
const usage: Usage = startEvent.message.usage
inputTokens = usage.input_tokens
} else if (event.type === 'message_delta') {
const deltaEvent = event as RawMessageDeltaEvent
outputTokens = deltaEvent.usage.output_tokens
} else if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
const text = event.delta.text
fullContent += text
controller.enqueue(new TextEncoder().encode(text))
let fullContent = ''
const thinkingBlocks: string[] = []
let currentThinking = ''
let inputTokens = 0
let outputTokens = 0
const flushThinkingBlock = () => {
if (currentThinking) {
thinkingBlocks.push(currentThinking)
currentThinking = ''
}
}
for await (const event of anthropicStream) {
if (event.type === 'message_start') {
inputTokens = event.message.usage.input_tokens
continue
}
if (event.type === 'message_delta') {
outputTokens = event.usage.output_tokens
continue
}
if (event.type === 'content_block_start') {
if (event.content_block.type === 'redacted_thinking') {
flushThinkingBlock()
thinkingBlocks.push('[redacted]')
} else if (event.content_block.type === 'thinking') {
flushThinkingBlock()
}
continue
}
if (event.type === 'content_block_stop') {
flushThinkingBlock()
continue
}
if (event.type !== 'content_block_delta') {
continue
}
const delta = event.delta
if (delta.type === 'thinking_delta' && typeof delta.thinking === 'string') {
currentThinking += delta.thinking
controller.enqueue({ type: 'thinking_delta', text: delta.thinking })
continue
}
if (delta.type === 'text_delta' && typeof delta.text === 'string') {
flushThinkingBlock()
fullContent += delta.text
controller.enqueue({ type: 'text_delta', text: delta.text, turn: 'final' })
}
}
flushThinkingBlock()
if (onComplete) {
onComplete(fullContent, { input_tokens: inputTokens, output_tokens: outputTokens })
onComplete({
content: fullContent,
usage: { input_tokens: inputTokens, output_tokens: outputTokens },
// Match enrichLastModelSegmentFromAnthropicResponse: join blocks with blank lines.
thinking: thinkingBlocks.filter(Boolean).join('\n\n'),
})
}
controller.close()
apps/sim/providers/azure-openai/index.ts
+2
View File
@@ -201,6 +201,7 @@ async function executeChatCompletionsRequest(
timing: { kind: 'simple', segmentName: request.model },
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { input: 0, output: 0, total: 0 },
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromAzureOpenAIStream(streamResponse, (content, usage) => {
output.content = content
@@ -517,6 +518,7 @@ async function executeChatCompletionsRequest(
count: toolCalls.length,
}
: undefined,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromAzureOpenAIStream(streamResponse, (content, usage) => {
output.content = content
apps/sim/providers/azure-openai/utils.ts
+13 -6
View File
@@ -3,17 +3,24 @@ import type OpenAI from 'openai'
import type { ChatCompletionChunk } from 'openai/resources/chat/completions'
import type { CompletionUsage } from 'openai/resources/completions'
import type { Stream } from 'openai/streaming'
import { checkForForcedToolUsageOpenAI, createOpenAICompatibleStream } from '@/providers/utils'
import { createOpenAICompatibleAgentEventStream } from '@/providers/openai-compat/stream-events'
import type { AgentStreamEvent } from '@/providers/stream-events'
import { checkForForcedToolUsageOpenAI } from '@/providers/utils'
/**
* Creates a ReadableStream from an Azure OpenAI streaming response.
* Uses the shared OpenAI-compatible streaming utility.
* Creates an agent-events stream from an Azure OpenAI streaming response.
* Uses the shared OpenAI-compatible agent event streaming utility.
*/
export function createReadableStreamFromAzureOpenAIStream(
azureOpenAIStream: Stream<ChatCompletionChunk>,
onComplete?: (content: string, usage: CompletionUsage) => void
): ReadableStream {
return createOpenAICompatibleStream(azureOpenAIStream, 'Azure OpenAI', onComplete)
onComplete?: (content: string, usage: CompletionUsage, thinking?: string) => void
): ReadableStream<AgentStreamEvent> {
return createOpenAICompatibleAgentEventStream(azureOpenAIStream, {
providerName: 'Azure OpenAI',
onComplete: onComplete
? (result) => onComplete(result.content, result.usage, result.thinking)
: undefined,
})
}
/**
apps/sim/providers/baseten/index.ts
+2
View File
@@ -167,6 +167,7 @@ export const basetenProvider: ProviderConfig = {
timing: { kind: 'simple', segmentName: request.model },
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { input: 0, output: 0, total: 0 },
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromOpenAIStream(streamResponse, (content, usage) => {
output.content = content
@@ -469,6 +470,7 @@ export const basetenProvider: ProviderConfig = {
},
toolCalls:
toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
streamFormat: 'agent-events-v1',
createStream: ({ output }) =>
createReadableStreamFromOpenAIStream(streamResponse, (content, usage) => {
output.content = content
apps/sim/providers/baseten/utils.ts
+13 -6
View File
@@ -1,6 +1,8 @@
import type { ChatCompletionChunk } from 'openai/resources/chat/completions'
import type { CompletionUsage } from 'openai/resources/completions'
import { checkForForcedToolUsageOpenAI, createOpenAICompatibleStream } from '@/providers/utils'
import { createOpenAICompatibleAgentEventStream } from '@/providers/openai-compat/stream-events'
import type { AgentStreamEvent } from '@/providers/stream-events'
import { checkForForcedToolUsageOpenAI } from '@/providers/utils'
/**
* Checks if a model supports native structured outputs (json_schema).
@@ -11,14 +13,19 @@ export async function supportsNativeStructuredOutputs(_modelId: string): Promise
}
/**
* Creates a ReadableStream from a Baseten streaming response.
* Uses the shared OpenAI-compatible streaming utility.
* Creates an agent-events stream from a Baseten streaming response.
* Uses the shared OpenAI-compatible agent event streaming utility.
*/
export function createReadableStreamFromOpenAIStream(
openaiStream: AsyncIterable<ChatCompletionChunk>,
onComplete?: (content: string, usage: CompletionUsage) => void
): ReadableStream<Uint8Array> {
return createOpenAICompatibleStream(openaiStream, 'Baseten', onComplete)
onComplete?: (content: string, usage: CompletionUsage, thinking?: string) => void
): ReadableStream<AgentStreamEvent> {
return createOpenAICompatibleAgentEventStream(openaiStream, {
providerName: 'Baseten',
onComplete: onComplete
? (result) => onComplete(result.content, result.usage, result.thinking)
: undefined,
})
}
/**
apps/sim/providers/bedrock/index.ts
+64 -2
View File
@@ -15,9 +15,10 @@ import {
} from '@aws-sdk/client-bedrock-runtime'
import { createLogger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import type { IterationToolCall, StreamingExecution } from '@/executor/types'
import type { IterationToolCall, NormalizedBlockOutput, StreamingExecution } from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import { buildBedrockMessageContent } from '@/providers/attachments'
import { createBedrockStreamingToolLoopStream } from '@/providers/bedrock/streaming-tool-loop'
import {
checkForForcedToolUsage,
createReadableStreamFromBedrockStream,
@@ -347,7 +348,66 @@ export const bedrockProvider: ProviderConfig = {
inferenceConfig.maxTokens = Number.parseInt(String(request.maxTokens))
}
const shouldStreamToolCalls = request.streamToolCalls ?? false
/**
* The live tool loop cannot honor responseFormat — structured output on
* Bedrock rides a final forced `structured_output` tool call that only the
* silent loop performs — so those requests fall back to the silent path.
*/
const shouldStreamToolCalls = (request.streamToolCalls ?? false) && !request.responseFormat
if (request.stream && shouldStreamToolCalls && bedrockTools && bedrockTools.length > 0) {
logger.info('Using streaming tool loop for Bedrock request')
const providerStartTime = Date.now()
const providerStartTimeISO = new Date(providerStartTime).toISOString()
const timeSegments: TimeSegment[] = []
const forcedTools = preparedTools?.forcedTools || []
return createStreamingExecution({
model: request.model,
providerStartTime,
providerStartTimeISO,
timing: {
kind: 'accumulated',
modelTime: 0,
toolsTime: 0,
firstResponseTime: 0,
iterations: 1,
timeSegments,
},
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { total: 0.0, input: 0.0, output: 0.0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createBedrockStreamingToolLoopStream({
client,
modelId: bedrockModelId,
request,
messages,
system: systemPromptWithSchema.length > 0 ? systemPromptWithSchema : undefined,
inferenceConfig,
bedrockTools,
toolChoice,
logger,
timeSegments,
forcedTools,
onComplete: (result) => {
output.content = result.content
output.tokens = result.tokens
output.cost = result.cost
output.toolCalls = result.toolCalls as NormalizedBlockOutput['toolCalls']
if (output.providerTiming) {
output.providerTiming.modelTime = result.modelTime
output.providerTiming.toolsTime = result.toolsTime
output.providerTiming.firstResponseTime = result.firstResponseTime
output.providerTiming.iterations = result.iterations
}
finalizeTiming()
},
}),
})
}
if (request.stream && (!bedrockTools || bedrockTools.length === 0)) {
logger.info('Using streaming response for Bedrock request (no tools)')
@@ -380,6 +440,7 @@ export const bedrockProvider: ProviderConfig = {
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { total: 0.0, input: 0.0, output: 0.0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromBedrockStream(bedrockStream, (content, usage) => {
output.content = content
@@ -878,6 +939,7 @@ export const bedrockProvider: ProviderConfig = {
toolCalls:
toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromBedrockStream(bedrockStream, (streamContent, usage) => {
output.content = streamContent
apps/sim/providers/bedrock/streaming-tool-loop.test.ts
+143
View File
@@ -0,0 +1,143 @@
/**
* @vitest-environment node
*/
import { describe, expect, it, vi } from 'vitest'
import { createBedrockStreamingToolLoopStream } from '@/providers/bedrock/streaming-tool-loop'
import type { AgentStreamEvent } from '@/providers/stream-events'
async function collectEvents(
stream: ReadableStream<AgentStreamEvent>
): Promise<AgentStreamEvent[]> {
const events: AgentStreamEvent[] = []
const reader = stream.getReader()
while (true) {
const { done, value } = await reader.read()
if (done) break
events.push(value)
}
return events
}
vi.mock('@/tools', () => ({
executeTool: vi.fn(async () => ({
success: true,
output: { ok: true },
})),
}))
vi.mock('@/providers/utils', () => ({
prepareToolExecution: vi.fn(() => ({
toolParams: { url: 'https://example.com' },
executionParams: { url: 'https://example.com' },
})),
calculateCost: vi.fn(() => ({
input: 0.01,
output: 0.02,
total: 0.03,
pricing: { input: 1, output: 2, updatedAt: new Date().toISOString() },
})),
sumToolCosts: vi.fn(() => 0),
trackForcedToolUsage: () => ({ hasUsedForcedTool: false, usedForcedTools: [] }),
}))
describe('createBedrockStreamingToolLoopStream', () => {
it('emits tool_call_start/end and final text; no invented thinking', async () => {
const turns = [
(async function* () {
yield {
contentBlockStart: {
contentBlockIndex: 0,
start: { toolUse: { toolUseId: 'tooluse_1', name: 'http_request' } },
},
}
yield {
contentBlockDelta: {
contentBlockIndex: 0,
delta: { toolUse: { input: '{"url":"https://example.com"}' } },
},
}
yield {
metadata: { usage: { inputTokens: 11, outputTokens: 4 } },
}
yield { messageStop: { stopReason: 'tool_use' } }
})(),
(async function* () {
yield {
contentBlockDelta: {
contentBlockIndex: 0,
delta: { text: 'Request completed.' },
},
}
yield {
metadata: { usage: { inputTokens: 22, outputTokens: 6 } },
}
yield { messageStop: { stopReason: 'end_turn' } }
})(),
]
let turnIdx = 0
const client = {
send: vi.fn(async () => ({ stream: turns[turnIdx++] })),
}
const onComplete = vi.fn()
const stream = createBedrockStreamingToolLoopStream({
client: client as any,
modelId: 'us.anthropic.claude-sonnet-4-5-20250929-v1:0',
request: {
model: 'bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0',
tools: [
{
id: 'http_request',
name: 'http_request',
description: 'HTTP',
parameters: { type: 'object', properties: {}, required: [] },
},
],
} as any,
messages: [{ role: 'user', content: [{ text: 'call it' }] }],
inferenceConfig: { temperature: 0.7 },
bedrockTools: [
{
toolSpec: {
name: 'http_request',
description: 'HTTP',
inputSchema: { json: { type: 'object', properties: {} } },
},
},
],
toolChoice: { auto: {} },
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() } as any,
timeSegments: [],
onComplete,
})
const events = await collectEvents(stream)
expect(events.some((e) => e.type === 'thinking_delta')).toBe(false)
expect(events.filter((e) => e.type === 'tool_call_start')).toEqual([
{ type: 'tool_call_start', id: 'tooluse_1', name: 'http_request' },
])
expect(events.filter((e) => e.type === 'tool_call_end')).toEqual([
{ type: 'tool_call_end', id: 'tooluse_1', name: 'http_request', status: 'success' },
])
// Text streams live as `pending`; the turn_end sequence classifies turns.
expect(
events
.filter((e) => e.type === 'text_delta' && e.turn === 'pending')
.map((e) => e.text)
.join('')
).toBe('Request completed.')
expect(events.filter((e) => e.type === 'turn_end').map((e) => e.turn)).toEqual([
'intermediate',
'final',
])
expect(onComplete).toHaveBeenCalledWith(
expect.objectContaining({
content: 'Request completed.',
toolCalls: expect.objectContaining({ count: 1 }),
})
)
})
})
apps/sim/providers/bedrock/streaming-tool-loop.ts
+535
View File
@@ -0,0 +1,535 @@
/**
* Live Bedrock ConverseStream tool loop.
*
* Capability-honest: text + tool_call_start/end only — Sim does not request
* Bedrock reasoning, so no thinking is invented. Text emits live as `pending`
* deltas and a `turn_end` event classifies each turn, so the pump projects
* only final-turn text to the answer channel. Abort → cancelled.
*/
import {
type Message as BedrockMessage,
type BedrockRuntimeClient,
type ContentBlock,
type ConversationRole,
ConverseStreamCommand,
type SystemContentBlock,
type Tool,
type ToolConfiguration,
type ToolResultBlock,
type ToolUseBlock,
} from '@aws-sdk/client-bedrock-runtime'
import type { Logger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import { checkForForcedToolUsage, generateToolUseId } from '@/providers/bedrock/utils'
import type { AgentStreamEvent, ToolCallEndStatus } from '@/providers/stream-events'
import {
isAbortError,
type StreamingToolLoopComplete,
settleOpenTools,
} from '@/providers/streaming-tool-loop-shared'
import { enrichLastModelSegment } from '@/providers/trace-enrichment'
import type { ProviderRequest, TimeSegment } from '@/providers/types'
import { calculateCost, prepareToolExecution, sumToolCosts } from '@/providers/utils'
import { executeTool } from '@/tools'
export interface CreateBedrockStreamingToolLoopStreamOptions {
client: BedrockRuntimeClient
modelId: string
request: ProviderRequest
messages: BedrockMessage[]
system?: SystemContentBlock[]
inferenceConfig: { temperature: number; maxTokens?: number }
bedrockTools: Tool[]
toolChoice: ToolConfiguration['toolChoice']
logger: Logger
timeSegments: TimeSegment[]
forcedTools?: string[]
onComplete: (result: StreamingToolLoopComplete) => void
}
interface AssembledToolUse {
toolUseId: string
name: string
inputJson: string
}
type ToolUseInput = NonNullable<ToolUseBlock['input']>
function parseToolInput(inputJson: string): Record<string, ToolUseInput> {
if (!inputJson.trim()) return {}
try {
const parsed = JSON.parse(inputJson)
return parsed && typeof parsed === 'object' && !Array.isArray(parsed)
? (parsed as Record<string, ToolUseInput>)
: {}
} catch {
return {}
}
}
async function drainBedrockTurn(
stream: AsyncIterable<import('@aws-sdk/client-bedrock-runtime').ConverseStreamOutput>,
controller: ReadableStreamDefaultController<AgentStreamEvent>,
openTools: Map<string, string>
): Promise<{
text: string
toolUses: AssembledToolUse[]
inputTokens: number
outputTokens: number
stopReason?: string
}> {
let text = ''
const toolsByIndex = new Map<number, AssembledToolUse>()
let currentIndex: number | undefined
let inputTokens = 0
let outputTokens = 0
let stopReason: string | undefined
for await (const event of stream) {
if (event.contentBlockStart) {
currentIndex = event.contentBlockStart.contentBlockIndex
const start = event.contentBlockStart.start
if (start && 'toolUse' in start && start.toolUse) {
const id = start.toolUse.toolUseId || generateToolUseId(start.toolUse.name || 'tool')
const name = start.toolUse.name || ''
if (typeof currentIndex === 'number') {
toolsByIndex.set(currentIndex, { toolUseId: id, name, inputJson: '' })
}
if (id && name && !openTools.has(id)) {
openTools.set(id, name)
controller.enqueue({ type: 'tool_call_start', id, name })
}
}
continue
}
if (event.contentBlockDelta) {
const idx = event.contentBlockDelta.contentBlockIndex ?? currentIndex
const delta = event.contentBlockDelta.delta
if (delta?.text) {
text += delta.text
// Live pending text: sinks render it now; the pump projects it to the
// answer only when this turn's turn_end says 'final'.
controller.enqueue({ type: 'text_delta', text: delta.text, turn: 'pending' })
}
if (delta && 'toolUse' in delta && delta.toolUse?.input && typeof idx === 'number') {
const pending = toolsByIndex.get(idx)
if (pending) {
pending.inputJson += delta.toolUse.input
}
}
continue
}
if (event.metadata?.usage) {
inputTokens = event.metadata.usage.inputTokens ?? inputTokens
outputTokens = event.metadata.usage.outputTokens ?? outputTokens
continue
}
if (event.messageStop?.stopReason) {
stopReason = event.messageStop.stopReason
}
}
return {
text,
toolUses: [...toolsByIndex.values()],
inputTokens,
outputTokens,
stopReason,
}
}
/**
* Multi-turn Bedrock ConverseStream tool loop as agent-events-v1.
*/
export function createBedrockStreamingToolLoopStream(
options: CreateBedrockStreamingToolLoopStreamOptions
): ReadableStream<AgentStreamEvent> {
const {
client,
modelId,
request,
messages: initialMessages,
system,
inferenceConfig,
bedrockTools,
logger,
timeSegments,
onComplete,
} = options
const forcedTools = options.forcedTools ?? []
const originalToolChoice = options.toolChoice
return new ReadableStream<AgentStreamEvent>({
async start(controller) {
const currentMessages = [...initialMessages]
let toolChoice = originalToolChoice
let usedForcedTools: string[] = []
let hasUsedForcedTool = false
let content = ''
let iterationCount = 0
let modelCalls = 0
let sawFinalTurn = false
let modelTime = 0
let toolsTime = 0
let firstResponseTime = 0
const tokens = { input: 0, output: 0, total: 0 }
let costInput = 0
let costOutput = 0
let costTotal = 0
let latestPricing: ReturnType<typeof calculateCost>['pricing'] | undefined
const toolCalls: unknown[] = []
const toolResults: Record<string, unknown>[] = []
const openToolStarts = new Map<string, string>()
try {
while (iterationCount < MAX_TOOL_ITERATIONS) {
if (request.abortSignal?.aborted) {
settleOpenTools(controller, openToolStarts, 'cancelled')
throw new DOMException('Stream aborted', 'AbortError')
}
const toolConfig: ToolConfiguration | undefined = bedrockTools.length
? { tools: bedrockTools, toolChoice }
: undefined
const modelStart = Date.now()
const command = new ConverseStreamCommand({
modelId,
messages: currentMessages,
system: system && system.length > 0 ? system : undefined,
inferenceConfig,
toolConfig,
})
const streamResponse = await client.send(
command,
request.abortSignal ? { abortSignal: request.abortSignal } : undefined
)
if (!streamResponse.stream) {
throw new Error('No stream returned from Bedrock')
}
const drained = await drainBedrockTurn(streamResponse.stream, controller, openToolStarts)
const modelEnd = Date.now()
const thisModelTime = modelEnd - modelStart
modelTime += thisModelTime
modelCalls++
if (iterationCount === 0) {
firstResponseTime = thisModelTime
}
timeSegments.push({
type: 'model',
name: request.model,
startTime: modelStart,
endTime: modelEnd,
duration: thisModelTime,
})
tokens.input += drained.inputTokens
tokens.output += drained.outputTokens
tokens.total += drained.inputTokens + drained.outputTokens
const turnCost = calculateCost(request.model, drained.inputTokens, drained.outputTokens)
costInput += turnCost.input
costOutput += turnCost.output
costTotal += turnCost.total
latestPricing = turnCost.pricing
/**
* Only execute tools when the model actually stopped to call them.
* On `max_tokens` / `malformed_tool_use` the accumulated input JSON
* is truncated and would parse to empty or partial arguments.
*/
const toolsExecutable = drained.stopReason === 'tool_use'
if (drained.toolUses.length > 0 && !toolsExecutable) {
logger.warn('Skipping tool execution for incomplete turn', {
stopReason: drained.stopReason,
toolCount: drained.toolUses.length,
})
settleOpenTools(controller, openToolStarts, 'error')
}
const executableToolUses = toolsExecutable ? drained.toolUses : []
const turnTag = executableToolUses.length > 0 ? 'intermediate' : 'final'
controller.enqueue({ type: 'turn_end', turn: turnTag })
if (drained.text) {
content = drained.text
}
const assembledToolUses = executableToolUses.map((t) => ({
toolUseId: t.toolUseId,
name: t.name,
input: parseToolInput(t.inputJson),
}))
enrichLastModelSegment(timeSegments, {
assistantContent: drained.text || undefined,
toolCalls:
assembledToolUses.length > 0
? assembledToolUses.map((t) => ({
id: t.toolUseId || '',
name: t.name || '',
arguments: t.input,
}))
: undefined,
finishReason: drained.stopReason,
tokens: {
input: drained.inputTokens,
output: drained.outputTokens,
total: drained.inputTokens + drained.outputTokens,
},
cost: {
input: turnCost.input,
output: turnCost.output,
total: turnCost.total,
},
provider: 'bedrock',
})
const forcedCheck = checkForForcedToolUsage(
assembledToolUses.map((t) => ({ name: t.name || '' })),
toolChoice,
forcedTools,
usedForcedTools
)
if (forcedCheck) {
hasUsedForcedTool = forcedCheck.hasUsedForcedTool
usedForcedTools = forcedCheck.usedForcedTools
}
if (assembledToolUses.length === 0) {
sawFinalTurn = true
break
}
const toolsStartTime = Date.now()
const orderedResults = await Promise.all(
assembledToolUses.map(async (toolUse) => {
const toolCallStartTime = Date.now()
const toolName = toolUse.name || ''
const toolArgs: Record<string, unknown> = toolUse.input
const toolUseId = toolUse.toolUseId || generateToolUseId(toolName)
try {
if (request.abortSignal?.aborted) {
throw new DOMException('Stream aborted', 'AbortError')
}
const tool = request.tools?.find((t) => t.id === toolName)
if (!tool) {
const value = {
toolUse,
toolUseId,
toolName,
toolArgs,
toolParams: {} as Record<string, unknown>,
result: {
success: false as const,
output: undefined,
error: `Tool not found: ${toolName}`,
},
startTime: toolCallStartTime,
endTime: Date.now(),
duration: Date.now() - toolCallStartTime,
status: 'error' as ToolCallEndStatus,
}
openToolStarts.delete(toolUseId)
controller.enqueue({
type: 'tool_call_end',
id: toolUseId,
name: toolName,
status: 'error',
})
return value
}
const { toolParams, executionParams } = prepareToolExecution(
tool,
toolArgs,
request
)
const result = await executeTool(toolName, executionParams, {
signal: request.abortSignal,
})
const toolCallEndTime = Date.now()
const status: ToolCallEndStatus = result.success ? 'success' : 'error'
openToolStarts.delete(toolUseId)
controller.enqueue({
type: 'tool_call_end',
id: toolUseId,
name: toolName,
status,
})
return {
toolUse,
toolUseId,
toolName,
toolArgs,
toolParams,
result,
startTime: toolCallStartTime,
endTime: toolCallEndTime,
duration: toolCallEndTime - toolCallStartTime,
status,
}
} catch (error) {
const toolCallEndTime = Date.now()
const cancelled = isAbortError(error) || !!request.abortSignal?.aborted
if (!cancelled) {
logger.error('Error processing tool call:', { error, toolName })
}
const status: ToolCallEndStatus = cancelled ? 'cancelled' : 'error'
openToolStarts.delete(toolUseId)
controller.enqueue({
type: 'tool_call_end',
id: toolUseId,
name: toolName,
status,
})
return {
toolUse,
toolUseId,
toolName,
toolArgs,
toolParams: {} as Record<string, unknown>,
result: {
success: false as const,
output: undefined,
error: getErrorMessage(error, 'Tool execution failed'),
},
startTime: toolCallStartTime,
endTime: toolCallEndTime,
duration: toolCallEndTime - toolCallStartTime,
status,
}
}
})
)
toolsTime += Date.now() - toolsStartTime
const assistantContent: ContentBlock[] = assembledToolUses.map((toolUse) => ({
toolUse: {
toolUseId: toolUse.toolUseId,
name: toolUse.name,
input: toolUse.input,
},
}))
currentMessages.push({
role: 'assistant' as ConversationRole,
content: assistantContent,
})
const toolResultContent: ContentBlock[] = []
for (const value of orderedResults) {
const { toolUseId, toolName, toolParams, result, startTime, endTime, duration } = value
timeSegments.push({
type: 'tool',
name: toolName,
startTime,
endTime,
duration,
toolCallId: toolUseId,
})
let resultContent: unknown
if (result.success && result.output) {
toolResults.push(result.output as Record<string, unknown>)
resultContent = result.output
} else {
resultContent = {
error: true,
message: result.error || 'Tool execution failed',
tool: toolName,
}
}
toolCalls.push({
name: toolName,
arguments: toolParams,
startTime: new Date(startTime).toISOString(),
endTime: new Date(endTime).toISOString(),
duration,
result: resultContent,
success: result.success,
})
const toolResultBlock: ToolResultBlock = {
toolUseId,
content: [{ text: JSON.stringify(resultContent) }],
}
toolResultContent.push({ toolResult: toolResultBlock })
}
if (toolResultContent.length > 0) {
currentMessages.push({
role: 'user' as ConversationRole,
content: toolResultContent,
})
}
if (
typeof originalToolChoice === 'object' &&
hasUsedForcedTool &&
forcedTools.length > 0
) {
const remainingTools = forcedTools.filter((tool) => !usedForcedTools.includes(tool))
toolChoice =
remainingTools.length > 0 ? { tool: { name: remainingTools[0] } } : { auto: {} }
} else if (hasUsedForcedTool && typeof originalToolChoice === 'object') {
toolChoice = { auto: {} }
}
iterationCount += 1
}
/**
* MAX_TOOL_ITERATIONS exit: every turn was tagged intermediate, so the
* answer channel would otherwise be empty. Flush the last turn's text
* as the final answer so legacy consumers still receive content.
*/
if (!sawFinalTurn && content) {
controller.enqueue({ type: 'text_delta', text: content, turn: 'final' })
}
const toolCost = sumToolCosts(toolResults)
onComplete({
content,
tokens,
cost: {
input: costInput,
output: costOutput,
toolCost: toolCost || undefined,
total: costTotal + toolCost,
pricing: latestPricing,
},
toolCalls:
toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
modelTime,
toolsTime,
firstResponseTime,
iterations: modelCalls,
})
controller.close()
} catch (error) {
if (isAbortError(error) || request.abortSignal?.aborted) {
settleOpenTools(controller, openToolStarts, 'cancelled')
} else {
settleOpenTools(controller, openToolStarts, 'error')
logger.error('Bedrock streaming tool loop failed', {
error: toError(error).message,
})
}
controller.error(error)
}
},
})
}
apps/sim/providers/bedrock/utils.stream.test.ts
+49
View File
@@ -0,0 +1,49 @@
/**
* @vitest-environment node
*/
import { describe, expect, it, vi } from 'vitest'
import { createReadableStreamFromBedrockStream } from '@/providers/bedrock/utils'
import type { AgentStreamEvent } from '@/providers/stream-events'
async function collectEvents(
stream: ReadableStream<AgentStreamEvent>
): Promise<AgentStreamEvent[]> {
const events: AgentStreamEvent[] = []
const reader = stream.getReader()
while (true) {
const { done, value } = await reader.read()
if (done) break
events.push(value)
}
return events
}
describe('createReadableStreamFromBedrockStream', () => {
it('emits text only — no tool events (never executed on this path) and no invented thinking', async () => {
const onComplete = vi.fn()
const stream = createReadableStreamFromBedrockStream(
(async function* () {
yield {
contentBlockStart: {
start: {
toolUse: { toolUseId: 'tooluse_1', name: 'http_request' },
},
},
} as any
yield {
contentBlockDelta: { delta: { text: 'Done' } },
} as any
yield {
metadata: { usage: { inputTokens: 2, outputTokens: 3 } },
} as any
})(),
onComplete
)
const events = await collectEvents(stream)
expect(events).toEqual([{ type: 'text_delta', text: 'Done', turn: 'final' }])
expect(events.some((e) => e.type === 'thinking_delta')).toBe(false)
expect(events.some((e) => e.type === 'tool_call_start')).toBe(false)
expect(onComplete).toHaveBeenCalledWith('Done', { inputTokens: 2, outputTokens: 3 })
})
})
apps/sim/providers/bedrock/utils.ts
+10 -2
View File
@@ -1,6 +1,7 @@
import type { ConverseStreamOutput } from '@aws-sdk/client-bedrock-runtime'
import { createLogger } from '@sim/logger'
import { randomFloat } from '@sim/utils/random'
import type { AgentStreamEvent } from '@/providers/stream-events'
import { trackForcedToolUsage } from '@/providers/utils'
const logger = createLogger('BedrockUtils')
@@ -10,10 +11,17 @@ export interface BedrockStreamUsage {
outputTokens: number
}
/**
* Bedrock ConverseStream → agent-events-v1 for the legacy (non-tool-loop)
* streaming path. Text deltas only: tools on this path are never executed, so
* emitting `tool_call_start` here would leave a chip running forever with no
* matching end. Sim does not request Bedrock reasoning, so there is no
* thinking to forward either.
*/
export function createReadableStreamFromBedrockStream(
bedrockStream: AsyncIterable<ConverseStreamOutput>,
onComplete?: (content: string, usage: BedrockStreamUsage) => void
): ReadableStream<Uint8Array> {
): ReadableStream<AgentStreamEvent> {
let fullContent = ''
let inputTokens = 0
let outputTokens = 0
@@ -25,7 +33,7 @@ export function createReadableStreamFromBedrockStream(
if (event.contentBlockDelta?.delta?.text) {
const text = event.contentBlockDelta.delta.text
fullContent += text
controller.enqueue(new TextEncoder().encode(text))
controller.enqueue({ type: 'text_delta', text, turn: 'final' })
} else if (event.metadata?.usage) {
inputTokens = event.metadata.usage.inputTokens ?? 0
outputTokens = event.metadata.usage.outputTokens ?? 0
apps/sim/providers/cerebras/index.ts
+2
View File
@@ -133,6 +133,7 @@ export const cerebrasProvider: ProviderConfig = {
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { input: 0, output: 0, total: 0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output }) =>
createReadableStreamFromCerebrasStream(streamResponse, (content, usage) => {
output.content = content
@@ -492,6 +493,7 @@ export const cerebrasProvider: ProviderConfig = {
}
: undefined,
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output }) =>
createReadableStreamFromCerebrasStream(streamResponse, (content, usage) => {
output.content = content
apps/sim/providers/cerebras/utils.ts
+12 -6
View File
@@ -1,5 +1,6 @@
import type { CompletionUsage } from 'openai/resources/completions'
import { createOpenAICompatibleStream } from '@/providers/utils'
import { createOpenAICompatibleAgentEventStream } from '@/providers/openai-compat/stream-events'
import type { AgentStreamEvent } from '@/providers/stream-events'
interface CerebrasChunk {
choices?: Array<{
@@ -15,12 +16,17 @@ interface CerebrasChunk {
}
/**
* Creates a ReadableStream from a Cerebras streaming response.
* Uses the shared OpenAI-compatible streaming utility.
* Creates an agent-events stream from a Cerebras streaming response.
* Uses the shared OpenAI-compatible agent event streaming utility.
*/
export function createReadableStreamFromCerebrasStream(
cerebrasStream: AsyncIterable<CerebrasChunk>,
onComplete?: (content: string, usage: CompletionUsage) => void
): ReadableStream<Uint8Array> {
return createOpenAICompatibleStream(cerebrasStream as any, 'Cerebras', onComplete)
onComplete?: (content: string, usage: CompletionUsage, thinking?: string) => void
): ReadableStream<AgentStreamEvent> {
return createOpenAICompatibleAgentEventStream(cerebrasStream as any, {
providerName: 'Cerebras',
onComplete: onComplete
? (result) => onComplete(result.content, result.usage, result.thinking)
: undefined,
})
}
apps/sim/providers/deepseek/index.test.ts
+99
View File
@@ -0,0 +1,99 @@
/**
* @vitest-environment node
*/
import { beforeEach, describe, expect, it, vi } from 'vitest'
import type { ProviderRequest } from '@/providers/types'
const { mockCreate } = vi.hoisted(() => ({
mockCreate: vi.fn(),
}))
vi.mock('openai', () => ({
default: vi.fn().mockImplementation(
class {
chat = { completions: { create: mockCreate } }
}
),
}))
vi.mock('@/providers', () => ({ MAX_TOOL_ITERATIONS: 5 }))
vi.mock('@/providers/models', () => ({
getProviderModels: vi.fn(() => ['deepseek-chat']),
getProviderDefaultModel: vi.fn(() => 'deepseek-chat'),
}))
vi.mock('@/providers/attachments', () => ({
formatMessagesForProvider: vi.fn((messages) => messages),
}))
vi.mock('@/providers/deepseek/utils', () => ({
createReadableStreamFromDeepseekStream: vi.fn(),
}))
vi.mock('@/providers/openai-compat/streaming-tool-loop', () => ({
createOpenAICompatStreamingToolLoopStream: vi.fn(),
}))
vi.mock('@/providers/streaming-execution', () => ({
createStreamingExecution: vi.fn((args) => args),
}))
vi.mock('@/providers/trace-enrichment', () => ({
enrichLastModelSegmentFromChatCompletions: vi.fn(),
}))
vi.mock('@/providers/utils', () => ({
calculateCost: vi.fn(() => ({ input: 0, output: 0, total: 0 })),
prepareToolExecution: vi.fn((_tool, args) => ({ toolParams: args, executionParams: args })),
prepareToolsWithUsageControl: vi.fn(() => ({
tools: [],
toolChoice: undefined,
forcedTools: [],
hasFilteredTools: false,
})),
sumToolCosts: vi.fn(() => 0),
trackForcedToolUsage: vi.fn(() => ({ hasUsedForcedTool: false, usedForcedTools: [] })),
}))
vi.mock('@/tools', () => ({ executeTool: vi.fn() }))
import { deepseekProvider } from '@/providers/deepseek/index'
function request(overrides: Partial<ProviderRequest> = {}): ProviderRequest {
return {
model: 'deepseek-chat',
apiKey: 'test-key',
messages: [{ role: 'user', content: 'hi' }],
...overrides,
}
}
describe('deepseekProvider thinking payload', () => {
beforeEach(() => {
mockCreate.mockReset()
mockCreate.mockResolvedValue({
choices: [{ message: { content: 'ok', tool_calls: [] } }],
usage: { prompt_tokens: 1, completion_tokens: 1, total_tokens: 2 },
})
})
it('sets thinking: { type: enabled } when thinkingLevel is enabled', async () => {
await deepseekProvider.executeRequest(request({ thinkingLevel: 'enabled' }))
expect(mockCreate).toHaveBeenCalled()
const payload = mockCreate.mock.calls[0][0]
expect(payload.thinking).toEqual({ type: 'enabled' })
})
it('sets thinking: { type: disabled } when thinkingLevel is none (API default is enabled)', async () => {
await deepseekProvider.executeRequest(request({ thinkingLevel: 'none' }))
const payload = mockCreate.mock.calls[0][0]
expect(payload.thinking).toEqual({ type: 'disabled' })
})
it('omits thinking when thinkingLevel is unset', async () => {
await deepseekProvider.executeRequest(request())
const payload = mockCreate.mock.calls[0][0]
expect(payload.thinking).toBeUndefined()
})
})
apps/sim/providers/deepseek/index.ts
+160 -39
View File
@@ -1,11 +1,12 @@
import { createLogger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import OpenAI from 'openai'
import type { StreamingExecution } from '@/executor/types'
import type { NormalizedBlockOutput, StreamingExecution } from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import { formatMessagesForProvider } from '@/providers/attachments'
import { createReadableStreamFromDeepseekStream } from '@/providers/deepseek/utils'
import { getProviderDefaultModel, getProviderModels } from '@/providers/models'
import { createOpenAICompatStreamingToolLoopStream } from '@/providers/openai-compat/streaming-tool-loop'
import { createStreamingExecution } from '@/providers/streaming-execution'
import { adaptOpenAIChatToolSchema } from '@/providers/tool-schema-adapter'
import { enrichLastModelSegmentFromChatCompletions } from '@/providers/trace-enrichment'
@@ -84,6 +85,17 @@ export const deepseekProvider: ProviderConfig = {
if (request.temperature !== undefined) payload.temperature = request.temperature
if (request.maxTokens != null) payload.max_tokens = request.maxTokens
/**
* DeepSeek Think mode: reasoning_content streams when enabled (or inherent
* on reasoner). The API default is enabled, so 'none' must explicitly send
* `disabled`; unset sends nothing to preserve the legacy request shape.
*/
if (request.thinkingLevel && request.thinkingLevel !== 'none') {
payload.thinking = { type: 'enabled' }
} else if (request.thinkingLevel === 'none') {
payload.thinking = { type: 'disabled' }
}
let preparedTools: ReturnType<typeof prepareToolsWithUsageControl> | null = null
if (tools?.length) {
@@ -111,6 +123,67 @@ export const deepseekProvider: ProviderConfig = {
}
}
const shouldStreamToolCalls = request.streamToolCalls ?? false
if (request.stream && shouldStreamToolCalls && payload.tools?.length) {
logger.info('Using streaming tool loop for DeepSeek request')
const timeSegments: TimeSegment[] = []
const forcedTools = preparedTools?.forcedTools || []
return createStreamingExecution({
model: request.model,
providerStartTime,
providerStartTimeISO,
timing: {
kind: 'accumulated',
modelTime: 0,
toolsTime: 0,
firstResponseTime: 0,
iterations: 1,
timeSegments,
},
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { total: 0.0, input: 0.0, output: 0.0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createOpenAICompatStreamingToolLoopStream({
providerName: 'Deepseek',
request,
basePayload: payload,
messages:
// double-cast-allowed: formatMessagesForProvider returns loosely-typed provider messages that are wire-compatible with the OpenAI chat.completions message params the shared loop expects
formattedMessages as unknown as OpenAI.Chat.Completions.ChatCompletionMessageParam[],
createStream: async (params, options) =>
deepseek.chat.completions.create({ ...params, stream: true }, options),
logger,
timeSegments,
forcedTools,
/**
* DeepSeek requires reasoning_content passed back on tool-call
* turns whenever the API returns it (thinking defaults to enabled
* server-side); it is ignored on non-tool turns, so preserving
* unconditionally is always safe.
*/
preserveAssistantReasoning: true,
onComplete: (result) => {
output.content = result.content
output.tokens = result.tokens
output.cost = result.cost
output.toolCalls = result.toolCalls as NormalizedBlockOutput['toolCalls']
if (output.providerTiming) {
output.providerTiming.modelTime = result.modelTime
output.providerTiming.toolsTime = result.toolsTime
output.providerTiming.firstResponseTime = result.firstResponseTime
output.providerTiming.iterations = result.iterations
}
finalizeTiming()
},
}),
})
}
if (request.stream && (!tools || tools.length === 0)) {
logger.info('Using streaming response for DeepSeek request (no tools)')
@@ -130,26 +203,38 @@ export const deepseekProvider: ProviderConfig = {
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { input: 0, output: 0, total: 0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output }) =>
createReadableStreamFromDeepseekStream(streamResponse as any, (content, usage) => {
output.content = content
output.tokens = {
input: usage.prompt_tokens,
output: usage.completion_tokens,
total: usage.total_tokens,
}
createReadableStreamFromDeepseekStream(
// double-cast-allowed: payload is untyped so the SDK cannot resolve the streaming overload; the stream yields OpenAI ChatCompletionChunk objects
streamResponse as unknown as AsyncIterable<OpenAI.Chat.Completions.ChatCompletionChunk>,
(content, usage, thinking) => {
output.content = content
output.tokens = {
input: usage.prompt_tokens,
output: usage.completion_tokens,
total: usage.total_tokens,
}
const costResult = calculateCost(
request.model,
usage.prompt_tokens,
usage.completion_tokens
)
output.cost = {
input: costResult.input,
output: costResult.output,
total: costResult.total,
const costResult = calculateCost(
request.model,
usage.prompt_tokens,
usage.completion_tokens
)
output.cost = {
input: costResult.input,
output: costResult.output,
total: costResult.total,
}
if (thinking) {
const segment = output.providerTiming?.timeSegments?.[0]
if (segment) {
segment.thinkingContent = thinking
}
}
}
}),
),
})
return streamingResult
@@ -281,7 +366,17 @@ export const deepseekProvider: ProviderConfig = {
const executionResults = await Promise.allSettled(toolExecutionPromises)
currentMessages.push({
const assistantMessage = currentResponse.choices[0]?.message
const assistantHistory: {
role: string
content: string | null
tool_calls: Array<{
id: string
type: string
function: { name: string; arguments: string }
}>
reasoning_content?: string
} = {
role: 'assistant',
content: null,
tool_calls: toolCallsInResponse.map((tc) => ({
@@ -292,7 +387,21 @@ export const deepseekProvider: ProviderConfig = {
arguments: tc.function.arguments,
},
})),
})
}
/**
* DeepSeek requires reasoning_content passed back on tool-call turns
* whenever the API returns it (thinking defaults to enabled
* server-side, so this applies even without an explicit thinking
* level); it is ignored on non-tool turns.
*/
if (assistantMessage) {
const reasoningContent = (assistantMessage as { reasoning_content?: string })
.reasoning_content
if (typeof reasoningContent === 'string' && reasoningContent.length > 0) {
assistantHistory.reasoning_content = reasoningContent
}
}
currentMessages.push(assistantHistory)
for (const settledResult of executionResults) {
if (settledResult.status === 'rejected' || !settledResult.value) continue
@@ -480,28 +589,40 @@ export const deepseekProvider: ProviderConfig = {
}
: undefined,
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output }) =>
createReadableStreamFromDeepseekStream(streamResponse as any, (content, usage) => {
output.content = content
output.tokens = {
input: tokens.input + usage.prompt_tokens,
output: tokens.output + usage.completion_tokens,
total: tokens.total + usage.total_tokens,
}
createReadableStreamFromDeepseekStream(
// double-cast-allowed: payload is untyped so the SDK cannot resolve the streaming overload; the stream yields OpenAI ChatCompletionChunk objects
streamResponse as unknown as AsyncIterable<OpenAI.Chat.Completions.ChatCompletionChunk>,
(content, usage, thinking) => {
output.content = content
output.tokens = {
input: tokens.input + usage.prompt_tokens,
output: tokens.output + usage.completion_tokens,
total: tokens.total + usage.total_tokens,
}
const streamCost = calculateCost(
request.model,
usage.prompt_tokens,
usage.completion_tokens
)
const tc = sumToolCosts(toolResults)
output.cost = {
input: accumulatedCost.input + streamCost.input,
output: accumulatedCost.output + streamCost.output,
toolCost: tc || undefined,
total: accumulatedCost.total + streamCost.total + tc,
const streamCost = calculateCost(
request.model,
usage.prompt_tokens,
usage.completion_tokens
)
const tc = sumToolCosts(toolResults)
output.cost = {
input: accumulatedCost.input + streamCost.input,
output: accumulatedCost.output + streamCost.output,
toolCost: tc || undefined,
total: accumulatedCost.total + streamCost.total + tc,
}
if (thinking) {
const lastModel = [...timeSegments].reverse().find((s) => s.type === 'model')
if (lastModel) {
lastModel.thinkingContent = thinking
}
}
}
}),
),
})
return streamingResult
apps/sim/providers/deepseek/utils.ts
+12 -6
View File
@@ -1,14 +1,20 @@
import type { ChatCompletionChunk } from 'openai/resources/chat/completions'
import type { CompletionUsage } from 'openai/resources/completions'
import { createOpenAICompatibleStream } from '@/providers/utils'
import { createOpenAICompatibleAgentEventStream } from '@/providers/openai-compat/stream-events'
import type { AgentStreamEvent } from '@/providers/stream-events'
/**
* Creates a ReadableStream from a DeepSeek streaming response.
* Uses the shared OpenAI-compatible streaming utility.
* Creates an agent-events stream from a DeepSeek streaming response.
* Uses the shared OpenAI-compatible agent event streaming utility.
*/
export function createReadableStreamFromDeepseekStream(
deepseekStream: AsyncIterable<ChatCompletionChunk>,
onComplete?: (content: string, usage: CompletionUsage) => void
): ReadableStream<Uint8Array> {
return createOpenAICompatibleStream(deepseekStream, 'Deepseek', onComplete)
onComplete?: (content: string, usage: CompletionUsage, thinking?: string) => void
): ReadableStream<AgentStreamEvent> {
return createOpenAICompatibleAgentEventStream(deepseekStream, {
providerName: 'Deepseek',
onComplete: onComplete
? (result) => onComplete(result.content, result.usage, result.thinking)
: undefined,
})
}
apps/sim/providers/fireworks/index.ts
+2
View File
@@ -167,6 +167,7 @@ export const fireworksProvider: ProviderConfig = {
timing: { kind: 'simple', segmentName: request.model },
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { input: 0, output: 0, total: 0 },
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createReadableStreamFromOpenAIStream(streamResponse, (content, usage) => {
output.content = content
@@ -469,6 +470,7 @@ export const fireworksProvider: ProviderConfig = {
},
toolCalls:
toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
streamFormat: 'agent-events-v1',
createStream: ({ output }) =>
createReadableStreamFromOpenAIStream(streamResponse, (content, usage) => {
output.content = content
apps/sim/providers/fireworks/utils.ts
+13 -6
View File
@@ -1,6 +1,8 @@
import type { ChatCompletionChunk } from 'openai/resources/chat/completions'
import type { CompletionUsage } from 'openai/resources/completions'
import { checkForForcedToolUsageOpenAI, createOpenAICompatibleStream } from '@/providers/utils'
import { createOpenAICompatibleAgentEventStream } from '@/providers/openai-compat/stream-events'
import type { AgentStreamEvent } from '@/providers/stream-events'
import { checkForForcedToolUsageOpenAI } from '@/providers/utils'
/**
* Checks if a model supports native structured outputs (json_schema).
@@ -11,14 +13,19 @@ export async function supportsNativeStructuredOutputs(_modelId: string): Promise
}
/**
* Creates a ReadableStream from a Fireworks streaming response.
* Uses the shared OpenAI-compatible streaming utility.
* Creates an agent-events stream from a Fireworks streaming response.
* Uses the shared OpenAI-compatible agent event streaming utility.
*/
export function createReadableStreamFromOpenAIStream(
openaiStream: AsyncIterable<ChatCompletionChunk>,
onComplete?: (content: string, usage: CompletionUsage) => void
): ReadableStream<Uint8Array> {
return createOpenAICompatibleStream(openaiStream, 'Fireworks', onComplete)
onComplete?: (content: string, usage: CompletionUsage, thinking?: string) => void
): ReadableStream<AgentStreamEvent> {
return createOpenAICompatibleAgentEventStream(openaiStream, {
providerName: 'Fireworks',
onComplete: onComplete
? (result) => onComplete(result.content, result.usage, result.thinking)
: undefined,
})
}
/**
apps/sim/providers/gemini/core.ts
+91 -8
View File
@@ -13,8 +13,9 @@ import {
} from '@google/genai'
import { createLogger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import type { IterationToolCall, StreamingExecution } from '@/executor/types'
import type { IterationToolCall, NormalizedBlockOutput, StreamingExecution } from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import { createGeminiStreamingToolLoopStream } from '@/providers/gemini/streaming-tool-loop'
import {
checkForForcedToolUsage,
cleanSchemaForGemini,
@@ -28,6 +29,8 @@ import {
mapToThinkingLevel,
supportsDisablingGemini25Thinking,
} from '@/providers/google/utils'
import { createStreamingExecution } from '@/providers/streaming-execution'
import { ensureToolCallId } from '@/providers/tool-call-id'
import { enrichLastModelSegment } from '@/providers/trace-enrichment'
import type {
FunctionCallResponse,
@@ -228,7 +231,7 @@ async function executeToolCallsBatch(
startTime: r.startTime,
endTime: r.endTime,
duration: r.duration,
toolCallId: r.part.functionCall?.id ?? undefined,
toolCallId: ensureToolCallId(r.part.functionCall?.id, 'gemini'),
})
totalToolsTime += r.duration
@@ -955,8 +958,10 @@ export async function executeGeminiRequest(
}
// Gemini 3.x takes thinkingLevel directly; Gemini 2.5-series rejects it and needs thinkingBudget.
// includeThoughts is required for thought parts to appear; it is requested
// only on agent-events runs so legacy runs keep the pre-agent-events payload.
if (request.thinkingLevel && request.thinkingLevel !== 'none') {
const thinkingConfig: ThinkingConfig = { includeThoughts: false }
const thinkingConfig: ThinkingConfig = { includeThoughts: request.agentEvents === true }
if (isGemini3Model(model)) {
thinkingConfig.thinkingLevel = mapToThinkingLevel(request.thinkingLevel)
} else {
@@ -1019,8 +1024,69 @@ export async function executeGeminiRequest(
}
const initialCallTime = Date.now()
/**
* Gemini 2 cannot combine responseSchema with tools, so structured output
* is applied on a final schema-configured request after tools settle — the
* silent path does this; the live loop would break as soon as a turn has
* no calls and skip the schema. Gemini 3 carries responseJsonSchema
* alongside tools, so its live loop keeps structured output.
*/
const responseFormatNeedsFinalPass = Boolean(request.responseFormat) && !isGemini3Model(model)
const shouldStreamToolCalls =
(request.streamToolCalls ?? false) && !responseFormatNeedsFinalPass
const shouldStream = request.stream && !tools?.length
// Live streaming tool loop
if (request.stream && shouldStreamToolCalls && tools?.length) {
logger.info('Using streaming tool loop for Gemini request')
const timeSegments: TimeSegment[] = []
const forcedTools = preparedTools?.forcedTools ?? []
return createStreamingExecution({
model,
providerStartTime,
providerStartTimeISO,
timing: {
kind: 'accumulated',
modelTime: 0,
toolsTime: 0,
firstResponseTime: 0,
iterations: 1,
timeSegments,
},
initialTokens: { input: 0, output: 0, total: 0 },
initialCost: { total: 0.0, input: 0.0, output: 0.0 },
isStreaming: true,
streamFormat: 'agent-events-v1',
createStream: ({ output, finalizeTiming }) =>
createGeminiStreamingToolLoopStream({
ai,
model,
baseConfig: geminiConfig,
contents,
request,
logger,
timeSegments,
forcedTools,
toolConfig,
onComplete: (result) => {
output.content = result.content
output.tokens = result.tokens
output.cost = result.cost
output.toolCalls = result.toolCalls as NormalizedBlockOutput['toolCalls']
if (output.providerTiming) {
output.providerTiming.modelTime = result.modelTime
output.providerTiming.toolsTime = result.toolsTime
output.providerTiming.firstResponseTime = result.firstResponseTime
output.providerTiming.iterations = result.iterations
}
finalizeTiming()
},
}),
})
}
// Streaming without tools
if (shouldStream) {
logger.info('Handling Gemini streaming response')
@@ -1042,7 +1108,7 @@ export async function executeGeminiRequest(
const stream = createReadableStreamFromGeminiStream(
streamGenerator,
(content: string, usage: GeminiUsage) => {
(content: string, usage: GeminiUsage, thinking?: string) => {
streamingResult.execution.output.content = content
streamingResult.execution.output.tokens = {
input: usage.promptTokenCount,
@@ -1057,6 +1123,13 @@ export async function executeGeminiRequest(
)
streamingResult.execution.output.cost = costResult
if (thinking) {
const segment = streamingResult.execution.output.providerTiming?.timeSegments?.[0]
if (segment) {
segment.thinkingContent = thinking
}
}
const streamEndTime = Date.now()
if (streamingResult.execution.output.providerTiming) {
streamingResult.execution.output.providerTiming.endTime = new Date(
@@ -1073,7 +1146,7 @@ export async function executeGeminiRequest(
}
)
return { ...streamingResult, stream }
return { ...streamingResult, stream, streamFormat: 'agent-events-v1' as const }
}
// Non-streaming request
@@ -1193,7 +1266,7 @@ export async function executeGeminiRequest(
const stream = createReadableStreamFromGeminiStream(
streamGenerator,
(streamContent: string, usage: GeminiUsage) => {
(streamContent: string, usage: GeminiUsage, thinking?: string) => {
streamingResult.execution.output.content = streamContent
streamingResult.execution.output.tokens = {
input: accumulatedTokens.input + usage.promptTokenCount,
@@ -1215,6 +1288,16 @@ export async function executeGeminiRequest(
pricing: streamCost.pricing,
}
if (thinking) {
const segments = streamingResult.execution.output.providerTiming?.timeSegments
const lastModel = segments
? [...segments].reverse().find((s) => s.type === 'model')
: undefined
if (lastModel) {
lastModel.thinkingContent = thinking
}
}
if (streamingResult.execution.output.providerTiming) {
streamingResult.execution.output.providerTiming.endTime = new Date().toISOString()
streamingResult.execution.output.providerTiming.duration =
@@ -1223,7 +1306,7 @@ export async function executeGeminiRequest(
}
)
return { ...streamingResult, stream }
return { ...streamingResult, stream, streamFormat: 'agent-events-v1' as const }
}
// Non-streaming: get next response
@@ -1318,7 +1401,7 @@ function enrichLastModelSegmentFromGeminiResponse(
Boolean(p.functionCall)
)
.map((p) => ({
id: p.functionCall.id ?? '',
id: ensureToolCallId(p.functionCall.id, 'gemini'),
name: p.functionCall.name ?? '',
arguments: (p.functionCall.args ?? {}) as Record<string, unknown>,
}))
apps/sim/providers/gemini/streaming-tool-loop.test.ts
+168
View File
@@ -0,0 +1,168 @@
/**
* @vitest-environment node
*/
import { describe, expect, it, vi } from 'vitest'
import { createGeminiStreamingToolLoopStream } from '@/providers/gemini/streaming-tool-loop'
import type { AgentStreamEvent } from '@/providers/stream-events'
import { resetLocalToolIdCounterForTests } from '@/providers/tool-call-id'
async function collectEvents(
stream: ReadableStream<AgentStreamEvent>
): Promise<AgentStreamEvent[]> {
const events: AgentStreamEvent[] = []
const reader = stream.getReader()
while (true) {
const { done, value } = await reader.read()
if (done) break
events.push(value)
}
return events
}
vi.mock('@/tools', () => ({
executeTool: vi.fn(async () => ({
success: true,
output: { ok: true, url: 'https://httpbin.org/get' },
})),
}))
vi.mock('@/providers/utils', () => ({
prepareToolExecution: vi.fn(() => ({
toolParams: { url: 'https://httpbin.org/get' },
executionParams: { url: 'https://httpbin.org/get' },
})),
calculateCost: vi.fn(() => ({
input: 0.01,
output: 0.02,
total: 0.03,
pricing: { input: 1, output: 2, updatedAt: new Date().toISOString() },
})),
sumToolCosts: vi.fn(() => 0),
isGemini3Model: vi.fn(() => false),
trackForcedToolUsage: () => ({ hasUsedForcedTool: false, usedForcedTools: [] }),
}))
describe('createGeminiStreamingToolLoopStream', () => {
it('emits thinking, tool lifecycle, then final answer; allocates local tool ids', async () => {
resetLocalToolIdCounterForTests()
const turns = [
// Turn 1: thinking + functionCall (no id)
(async function* () {
yield {
candidates: [
{
content: {
parts: [
{ text: 'I should call the API. ', thought: true },
{
functionCall: {
name: 'http_request',
args: { url: 'https://httpbin.org/get' },
},
},
],
},
finishReason: 'STOP',
},
],
usageMetadata: {
promptTokenCount: 10,
candidatesTokenCount: 5,
totalTokenCount: 15,
},
} as any
})(),
// Turn 2: final answer
(async function* () {
yield {
candidates: [
{
content: {
parts: [{ text: 'Done: https://httpbin.org/get' }],
},
finishReason: 'STOP',
},
],
usageMetadata: {
promptTokenCount: 20,
candidatesTokenCount: 8,
totalTokenCount: 28,
},
} as any
})(),
]
let turnIdx = 0
const ai = {
models: {
generateContentStream: vi.fn(async () => turns[turnIdx++]),
},
}
const onComplete = vi.fn()
const timeSegments: any[] = []
const stream = createGeminiStreamingToolLoopStream({
ai: ai as any,
model: 'gemini-2.5-flash',
baseConfig: {},
contents: [{ role: 'user', parts: [{ text: 'fetch it' }] }],
request: {
model: 'gemini-2.5-flash',
tools: [
{
id: 'http_request',
name: 'http_request',
description: 'HTTP',
parameters: { type: 'object', properties: {}, required: [] },
},
],
} as any,
logger: { info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() } as any,
timeSegments,
onComplete,
})
const events = await collectEvents(stream)
expect(events.filter((e) => e.type === 'thinking_delta').map((e) => e.text)).toEqual([
'I should call the API. ',
])
const starts = events.filter((e) => e.type === 'tool_call_start')
expect(starts).toHaveLength(1)
expect(starts[0]).toMatchObject({ name: 'http_request' })
expect((starts[0] as { id: string }).id).toMatch(/^gemini_/)
const ends = events.filter((e) => e.type === 'tool_call_end')
expect(ends).toEqual([
{
type: 'tool_call_end',
id: (starts[0] as { id: string }).id,
name: 'http_request',
status: 'success',
},
])
// Text streams live as `pending`; the turn_end sequence classifies turns.
const textEvents = events.filter((e) => e.type === 'text_delta')
expect(textEvents.every((e) => e.type === 'text_delta' && e.turn === 'pending')).toBe(true)
expect(
textEvents
.filter((e) => e.type === 'text_delta')
.map((e) => e.text)
.join('')
).toContain('Done:')
expect(events.filter((e) => e.type === 'turn_end').map((e) => e.turn)).toEqual([
'intermediate',
'final',
])
expect(onComplete).toHaveBeenCalledWith(
expect.objectContaining({
content: expect.stringContaining('Done:'),
toolCalls: expect.objectContaining({ count: 1 }),
})
)
})
})
apps/sim/providers/gemini/streaming-tool-loop.ts
+536
View File
@@ -0,0 +1,536 @@
/**
* Live Gemini streaming tool loop.
*
* Each model turn uses generateContentStream. Thought parts → thinking_delta
* live; functionCall parts → tool_call_start (with local ids when the model
* omits them); text parts → `pending` text deltas live, classified by a
* `turn_end` event as intermediate vs final. Tool ends emit in completion
* order; abort → cancelled.
*
* Function-call parts are echoed back into request history verbatim — Google
* requires signatures/ids to round-trip exactly as received, so local ids are
* used only for agent events and trace segments, never injected into history.
*/
import {
type Content,
FunctionCallingConfigMode,
type GenerateContentConfig,
type GenerateContentResponse,
type GoogleGenAI,
type Part,
type Schema,
type ToolConfig,
} from '@google/genai'
import type { Logger } from '@sim/logger'
import { getErrorMessage, toError } from '@sim/utils/errors'
import type { IterationToolCall } from '@/executor/types'
import { MAX_TOOL_ITERATIONS } from '@/providers'
import {
checkForForcedToolUsage,
cleanSchemaForGemini,
convertUsageMetadata,
ensureStructResponse,
} from '@/providers/google/utils'
import type { AgentStreamEvent, ToolCallEndStatus } from '@/providers/stream-events'
import {
isAbortError,
type StreamingToolLoopComplete,
settleOpenTools,
} from '@/providers/streaming-tool-loop-shared'
import { ensureToolCallId } from '@/providers/tool-call-id'
import { enrichLastModelSegment } from '@/providers/trace-enrichment'
import type { ProviderRequest, TimeSegment } from '@/providers/types'
import {
calculateCost,
isGemini3Model,
prepareToolExecution,
sumToolCosts,
} from '@/providers/utils'
import { executeTool } from '@/tools'
import type { GeminiUsage } from './types'
export interface CreateGeminiStreamingToolLoopStreamOptions {
ai: GoogleGenAI
model: string
baseConfig: GenerateContentConfig
contents: Content[]
request: ProviderRequest
logger: Logger
timeSegments: TimeSegment[]
forcedTools?: string[]
toolConfig?: ToolConfig
onComplete: (result: StreamingToolLoopComplete) => void
}
/**
* A streamed functionCall part paired with the execution-local id used on the
* agent-events stream. The part itself stays verbatim for history echo.
*/
interface StreamedFunctionCall {
part: Part
localId: string
}
function buildNextConfig(
baseConfig: GenerateContentConfig,
currentToolConfig: ToolConfig | undefined,
usedForcedTools: string[],
forcedTools: string[],
request: ProviderRequest,
logger: Logger,
model: string
): GenerateContentConfig {
const nextConfig = { ...baseConfig }
const allForcedToolsUsed = forcedTools.length > 0 && usedForcedTools.length === forcedTools.length
if (allForcedToolsUsed && request.responseFormat) {
nextConfig.tools = undefined
nextConfig.toolConfig = undefined
if (isGemini3Model(model)) {
logger.info('Gemini 3: Stripping tools after forced tool execution, schema already set')
} else {
nextConfig.responseMimeType = 'application/json'
nextConfig.responseSchema = cleanSchemaForGemini(request.responseFormat.schema) as Schema
logger.info('Using structured output for final response after tool execution')
}
} else if (currentToolConfig) {
nextConfig.toolConfig = currentToolConfig
} else {
nextConfig.toolConfig = { functionCallingConfig: { mode: FunctionCallingConfigMode.AUTO } }
}
return nextConfig
}
/**
* Drain one generateContentStream turn into live agent events + aggregated parts.
*/
async function drainGeminiTurn(
stream: AsyncGenerator<GenerateContentResponse>,
controller: ReadableStreamDefaultController<AgentStreamEvent>,
openTools: Map<string, string>
): Promise<{
text: string
thinking: string
functionCalls: StreamedFunctionCall[]
usage: GeminiUsage
finishReason?: string
}> {
let text = ''
let thinking = ''
const functionCalls: StreamedFunctionCall[] = []
const seenKeys = new Set<string>()
let usage: GeminiUsage = { promptTokenCount: 0, candidatesTokenCount: 0, totalTokenCount: 0 }
let finishReason: string | undefined
for await (const chunk of stream) {
if (chunk.usageMetadata) {
usage = convertUsageMetadata(chunk.usageMetadata)
}
const candidate = chunk.candidates?.[0]
if (candidate?.finishReason) {
finishReason = String(candidate.finishReason)
}
const parts = candidate?.content?.parts
if (!Array.isArray(parts)) {
const fallback = chunk.text
if (fallback) {
text += fallback
controller.enqueue({ type: 'text_delta', text: fallback, turn: 'pending' })
}
continue
}
for (const part of parts) {
if (part.functionCall) {
const localId = ensureToolCallId(part.functionCall.id, 'gemini')
const name = part.functionCall.name ?? ''
if (!seenKeys.has(localId) && name) {
seenKeys.add(localId)
functionCalls.push({ part, localId })
if (!openTools.has(localId)) {
openTools.set(localId, name)
controller.enqueue({ type: 'tool_call_start', id: localId, name })
}
}
continue
}
if (!part.text) continue
if (part.thought === true) {
thinking += part.text
controller.enqueue({ type: 'thinking_delta', text: part.text })
} else {
text += part.text
// Live pending text: sinks render it now; the pump projects it to the
// answer only when this turn's turn_end says 'final'.
controller.enqueue({ type: 'text_delta', text: part.text, turn: 'pending' })
}
}
}
return { text, thinking, functionCalls, usage, finishReason }
}
/**
* Multi-turn Gemini tool loop as an agent-events-v1 object stream.
*/
export function createGeminiStreamingToolLoopStream(
options: CreateGeminiStreamingToolLoopStreamOptions
): ReadableStream<AgentStreamEvent> {
const {
ai,
model,
baseConfig,
contents: initialContents,
request,
logger,
timeSegments,
onComplete,
} = options
const forcedTools = options.forcedTools ?? []
return new ReadableStream<AgentStreamEvent>({
async start(controller) {
let contents = [...initialContents]
let currentToolConfig = options.toolConfig
let usedForcedTools: string[] = []
let content = ''
let iterationCount = 0
let modelCalls = 0
let sawFinalTurn = false
let modelTime = 0
let toolsTime = 0
let firstResponseTime = 0
const tokens = { input: 0, output: 0, total: 0 }
let costInput = 0
let costOutput = 0
let costTotal = 0
let latestPricing: ReturnType<typeof calculateCost>['pricing'] | undefined
const toolCalls: unknown[] = []
const toolResults: Record<string, unknown>[] = []
const openToolStarts = new Map<string, string>()
try {
while (iterationCount < MAX_TOOL_ITERATIONS) {
if (request.abortSignal?.aborted) {
settleOpenTools(controller, openToolStarts, 'cancelled')
throw new DOMException('Stream aborted', 'AbortError')
}
const turnConfig = buildNextConfig(
baseConfig,
currentToolConfig,
usedForcedTools,
forcedTools,
request,
logger,
model
)
const modelStart = Date.now()
const streamGenerator = await ai.models.generateContentStream({
model,
contents,
config: turnConfig,
})
const drained = await drainGeminiTurn(streamGenerator, controller, openToolStarts)
const modelEnd = Date.now()
const thisModelTime = modelEnd - modelStart
modelTime += thisModelTime
modelCalls++
if (iterationCount === 0) {
firstResponseTime = thisModelTime
}
timeSegments.push({
type: 'model',
name: model,
startTime: modelStart,
endTime: modelEnd,
duration: thisModelTime,
})
tokens.input += drained.usage.promptTokenCount
tokens.output += drained.usage.candidatesTokenCount
tokens.total += drained.usage.totalTokenCount
const turnCost = calculateCost(
model,
drained.usage.promptTokenCount,
drained.usage.candidatesTokenCount
)
costInput += turnCost.input
costOutput += turnCost.output
costTotal += turnCost.total
latestPricing = turnCost.pricing
const turnTag = drained.functionCalls.length > 0 ? 'intermediate' : 'final'
controller.enqueue({ type: 'turn_end', turn: turnTag })
if (drained.text) {
content = drained.text
}
const toolCallsForEnrich: IterationToolCall[] = drained.functionCalls
.filter((fc) => Boolean(fc.part.functionCall))
.map((fc) => ({
id: fc.localId,
name: fc.part.functionCall?.name ?? '',
arguments: (fc.part.functionCall?.args ?? {}) as Record<string, unknown>,
}))
enrichLastModelSegment(timeSegments, {
assistantContent: drained.text || undefined,
thinkingContent: drained.thinking || undefined,
toolCalls: toolCallsForEnrich.length > 0 ? toolCallsForEnrich : undefined,
finishReason: drained.finishReason,
tokens: {
input: drained.usage.promptTokenCount,
output: drained.usage.candidatesTokenCount,
total: drained.usage.totalTokenCount,
},
cost: {
input: turnCost.input,
output: turnCost.output,
total: turnCost.total,
},
provider: 'google',
})
const forcedCheck = checkForForcedToolUsage(
drained.functionCalls
.map((fc) => fc.part.functionCall)
.filter((fc): fc is NonNullable<typeof fc> => Boolean(fc)),
currentToolConfig,
forcedTools,
usedForcedTools
)
if (forcedCheck) {
usedForcedTools = forcedCheck.usedForcedTools
currentToolConfig = forcedCheck.nextToolConfig
}
if (drained.functionCalls.length === 0) {
sawFinalTurn = true
break
}
const toolsStartTime = Date.now()
const orderedResults = await Promise.all(
drained.functionCalls.map(async ({ part, localId }) => {
const functionCall = part.functionCall!
const toolCallId = localId
const toolName = functionCall.name ?? ''
const toolArgs = (functionCall.args ?? {}) as Record<string, unknown>
const toolCallStartTime = Date.now()
try {
if (request.abortSignal?.aborted) {
throw new DOMException('Stream aborted', 'AbortError')
}
const tool = request.tools?.find((t) => t.id === toolName)
if (!tool) {
const value = {
part,
toolCallId,
toolName,
toolArgs,
toolParams: {} as Record<string, unknown>,
resultContent: {
error: true,
message: `Tool ${toolName} not found`,
tool: toolName,
},
result: undefined as
| { success: boolean; output?: unknown; error?: string }
| undefined,
startTime: toolCallStartTime,
endTime: Date.now(),
duration: Date.now() - toolCallStartTime,
status: 'error' as ToolCallEndStatus,
success: false,
}
openToolStarts.delete(toolCallId)
controller.enqueue({
type: 'tool_call_end',
id: toolCallId,
name: toolName,
status: 'error',
})
return value
}
const { toolParams, executionParams } = prepareToolExecution(
tool,
toolArgs,
request
)
const result = await executeTool(toolName, executionParams, {
signal: request.abortSignal,
})
const toolCallEndTime = Date.now()
const resultContent: Record<string, unknown> = result.success
? ensureStructResponse(result.output)
: {
error: true,
message: result.error || 'Tool execution failed',
tool: toolName,
}
const status: ToolCallEndStatus = result.success ? 'success' : 'error'
openToolStarts.delete(toolCallId)
controller.enqueue({
type: 'tool_call_end',
id: toolCallId,
name: toolName,
status,
})
return {
part,
toolCallId,
toolName,
toolArgs,
toolParams,
resultContent,
result,
startTime: toolCallStartTime,
endTime: toolCallEndTime,
duration: toolCallEndTime - toolCallStartTime,
status,
success: result.success,
}
} catch (error) {
const toolCallEndTime = Date.now()
const cancelled = isAbortError(error) || !!request.abortSignal?.aborted
if (!cancelled) {
logger.error('Error processing function call:', {
error: toError(error).message,
functionName: toolName,
})
}
const status: ToolCallEndStatus = cancelled ? 'cancelled' : 'error'
openToolStarts.delete(toolCallId)
controller.enqueue({
type: 'tool_call_end',
id: toolCallId,
name: toolName,
status,
})
return {
part,
toolCallId,
toolName,
toolArgs,
toolParams: {} as Record<string, unknown>,
resultContent: {
error: true,
message: getErrorMessage(error, 'Tool execution failed'),
tool: toolName,
},
result: undefined,
startTime: toolCallStartTime,
endTime: toolCallEndTime,
duration: toolCallEndTime - toolCallStartTime,
status,
success: false,
}
}
})
)
toolsTime += Date.now() - toolsStartTime
/**
* Echo the model's functionCall parts verbatim (signatures and any
* model-provided ids must round-trip untouched). A functionResponse
* id is attached only when the model itself provided one.
*/
const modelParts: Part[] = orderedResults.map((r) => r.part)
const userParts: Part[] = orderedResults.map((r) => ({
functionResponse: {
name: r.toolName,
response: r.resultContent,
...(r.part.functionCall?.id ? { id: r.part.functionCall.id } : {}),
},
}))
contents = [
...contents,
{ role: 'model', parts: modelParts },
{ role: 'user', parts: userParts },
]
for (const r of orderedResults) {
toolCalls.push({
name: r.toolName,
arguments: r.toolParams,
startTime: new Date(r.startTime).toISOString(),
endTime: new Date(r.endTime).toISOString(),
duration: r.duration,
result: r.resultContent,
success: r.success,
})
if (r.success && r.result?.output) {
toolResults.push(r.result.output as Record<string, unknown>)
}
timeSegments.push({
type: 'tool',
name: r.toolName,
startTime: r.startTime,
endTime: r.endTime,
duration: r.duration,
toolCallId: r.toolCallId,
})
}
iterationCount += 1
}
/**
* MAX_TOOL_ITERATIONS exit: every turn was tagged intermediate, so the
* answer channel would otherwise be empty. Flush the last turn's text
* as the final answer so legacy consumers still receive content.
*/
if (!sawFinalTurn && content) {
controller.enqueue({ type: 'text_delta', text: content, turn: 'final' })
}
const toolCost = sumToolCosts(toolResults)
onComplete({
content,
tokens,
cost: {
input: costInput,
output: costOutput,
toolCost: toolCost || undefined,
total: costTotal + toolCost,
pricing: latestPricing,
},
toolCalls:
toolCalls.length > 0 ? { list: toolCalls, count: toolCalls.length } : undefined,
modelTime,
toolsTime,
firstResponseTime,
iterations: modelCalls,
})
controller.close()
} catch (error) {
if (isAbortError(error) || request.abortSignal?.aborted) {
settleOpenTools(controller, openToolStarts, 'cancelled')
} else {
settleOpenTools(controller, openToolStarts, 'error')
logger.error('Gemini streaming tool loop failed', {
error: toError(error).message,
})
}
controller.error(error)
}
},
})
}

Some files were not shown because too many files have changed in this diff Show More