mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
feat(agent-stream): thinking and tool streaming (#5671)
* feat(agent-stream): add agent-events thinking/tool streaming for chat and canvas Ship the agent-events-v1 protocol with provider tool loops, dual-gated chat thinking, DeepSeek/Groq/OpenAI reasoning wiring, and ChatGPT-like thinking chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): clear stuck streaming UI and format db snapshot Biome was failing CI on migrations/meta/0261_snapshot.json. Also settle assistant streaming/tool flags when SSE ends without a terminal frame, without clobbering Stop's finalized content. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): satisfy biome format and import order Auto-format the sim package for CI lint:check, and repair the Anthropic streaming tool-loop payload after an unsafe delete-to-undefined rewrite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer on abort and update migration journal test Treat AbortError from reader.cancel as a cancelled pump result so soft-complete retains answerText. Point the workspace storage migration journal assertion at 0261_chat_include_thinking. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(chat): keep Stop notice when server emits cancel error Ignore terminal SSE error frames after the user aborts so "Client cancelled request" cannot overwrite "Response stopped by user". Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): ChatGPT-style thinking shimmer and stick-to-bottom scroll Add left-to-right shimmer on live thinking label/body, keep scroll working by shimmering an inner node, and follow the answer only while near the bottom. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): stop pump on client disconnect; soft-complete agents only Abort the agent stream pump when the projected HTTP body is cancelled so provider work does not continue after disconnect. Limit AbortError soft-success to Agent blocks so Function/HTTP cancels still fail in logs. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): persist includeThinking across pause snapshots Paused chat runs with Include thinking enabled were dropping the flag when serializing the pause snapshot, so resume always rebuilt streams without thinking/tool SSE frames. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): keep drained answer text when stream times out Persist pump answerText onto the streaming execution before throwing on timeout, and carry that partial content into the failed block output so logs match what the client already saw. Co-authored-by: Cursor <cursoragent@cursor.com> * improvement(chat): auto-collapse tools chrome when tool streaming ends Match thinking UX: open while tools run, collapse when finished, and keep the panel open only if the user manually reopens it. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): settle canvas stream chrome on failure paths Clear agentStreamActive and settle running tool chips when blocks error, timeouts cancel runs, or execution ends without stream:done so the output panel does not stay on live Thinking/Using tools chrome. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(lint): organize imports in terminal console store Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agent-stream): mark open tools cancelled on HITL pause Pause can interrupt a tool loop without tool end events; settling those chips as success incorrectly showed unfinished tools as complete. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(db): drop branch-local 0261 migration ahead of staging merge * chore(db): regenerate include_thinking migration as 0266 post staging merge * fix(providers): resolve type errors in streaming tool loop call sites * fix(agent-stream): gate agent events opt-in and correct provider loop behavior - streamToolCalls and provider thinking requests now require run-level agentEvents opt-in (canvas on, chat dual-gated, API off) so existing runs keep pre-agent-events behavior exactly - OpenAI reasoning summaries opt-in + strip-and-retry on unverified-org 400 - streaming loops run tool postProcess again (firecrawl/exa async results) - bedrock live loop falls back to silent path for responseFormat - deepseek: reasoning_content pass-back unconditional, 'none' sends disabled - groq: x_groq.usage fallback, reasoning params gated, qwen none disables - gemini: functionCall parts echoed verbatim, local ids only for events - truncated turns (max_tokens/length) no longer execute partial tool calls - MAX_TOOL_ITERATIONS exit flushes last turn text as final answer - iterations reports actual model calls; shared loop plumbing extracted * refactor(agent-stream): consolidate protocol, dedupe client/server plumbing, hygiene - canonical ChatStreamFrame union + type guards consumed by server emitters and the chat client; stream_error restored to legacy log-only handling - strip thinking/tool args from providerTiming on public final envelopes - shared tool-chip lifecycle module for chat, canvas, and console store - shared sink-to-execution-events forwarder replaces the copy-pasted adapter in the execute route and HITL manager; LIVE_ONLY event set shared - stream:thinking payload field renamed data->text; canvas thinking batched - abort reasons carried as AbortError DOMExceptions so raw fetch consumers classify correctly; thinking cap renamed to chars and scope-documented - kimi wired for agent events like the other compat providers - deleted dead exports/step-N comments; fixtures match real wire shapes; loop tests use explicit mocks instead of importOriginal * test(agent-stream): cover the dual-gated execution path and typed abort reasons - chat route tests assert agentEvents reaches executeWorkflow only when policy and protocol header agree - execution-limits tests assert AbortError-typed reasons - executor metadata type carries agentEvents * fix(deploy-modal): align include-thinking spacing with the modal's 6.5px rhythm * docs(agent-stream): autogenerate per-model thinking/tool stream support on the Agent block page - capabilities.thinking.streamed ('full' | 'summary' | 'none') on models.ts, explicit for the Anthropic family where visibility varies per generation; getThinkingStreamVisibility exposes the derivation for docs and UI alike - scripts/sync-agent-stream-docs.ts regenerates the support tables between markers in workflows/blocks/agent.mdx from the model registry and STREAMING_TOOL_CALL_PROVIDERS; --check fails on drift or missing metadata - wired agent-stream-docs:check into CI next to the other sync gates * feat(anthropic): request summarized thinking display for omitted-default Claude models The newest Claude generations (Fable 5, Sonnet 5, Opus 4.8/4.7) default thinking.display to omitted — empty thinking blocks, no deltas. On agent-events runs Sim now opts back in with display: 'summarized', driven by the registry's streamed metadata; legacy runs keep the exact pre-agent-events request shape. Registry, generated docs, and the family capability table updated accordingly. * docs(skills): cover thinking.streamed and agent-stream docs sync in model skills * chore(deps): upgrade @anthropic-ai/sdk to 0.114.0 and adopt official types - adaptive thinking, display, and output_config are now SDK-typed; the only remaining custom payload field is output_format (beta-header structured outputs, which the SDK models as output_config.format instead) - anthropic stream events narrow on the SDK's discriminated unions instead of anonymous casts; compat deltas type content/tool_calls from the OpenAI SDK with vendor reasoning fields as an explicit optional extension - @sim/auth exposes an explicit VerifyAuth contract so its declarations no longer reference better-auth's nested zod instance (TS2883 under fresh install layouts); realtime consumer aligned - docs app zod pinned to the repo's exact 4.3.6 so ai SDK types bind the same zod instance (docs type-check was latently broken) - knowledge embedding tests made hermetic against local .env keys and hosted rotation fallback * refactor(providers): replace legacy as-any stream casts with annotated typed casts * refactor(providers): finish provider audit — remove dead byte-stream helper, annotate remaining legacy casts Audit of all 26 providers for the agent-events feature confirmed every streaming execution declares agent-events-v1 and every adapter emits AgentStreamEvent objects. Cleanup from the audit: the unconsumed legacy createOpenAICompatibleStream byte helper is deleted, and the remaining streamResponse-as-any casts (xai, nvidia, kimi, meta, zai, sakana) are annotated typed casts matching the groq/deepseek fix. * feat(streaming): stream answer text live during tool loops via turn_end protocol The live tool loops buffered all answer text per model turn (classification of intermediate vs final is only known at turn end), so gated surfaces saw thinking stream, then dead air with the thinking chrome stuck open, then the whole answer at once. Loops now emit text deltas live as `turn: 'pending'` plus a `turn_end` event per turn. The pump buffers pending text and projects it to the byte path (answerText/logs/memory/legacy clients) only on a final turn_end, so all settled semantics are unchanged. Gated surfaces render the pending text as it streams and reconcile with a reset when a turn resolves to tools: - public chat: live `chunk` frames from the sink + dual-gated `chunk_reset`; byte-path frame emission is suppressed to avoid duplicates (kept for response-format transformed streams via clientStreamTransformed) - canvas: forwarder emits live `stream:chunk` + `stream:chunk_reset`; the execute route and HITL resume readers stop re-emitting byte chunks; panel chat tracks per-block segments and replaces content on flush - chat client: per-block text segments, chunk_reset handling, and thinking chrome now settles on tool start as well as first answer chunk * fix(streaming): address validated review findings across provider gating and reset reconciliation Three-reviewer pass over the branch, findings validated against staging: - agent-handler forwards agentEvents to executeProviderRequest — the flag was computed but dropped in the field-by-field copy, so provider-side thinking requests (OpenAI summaries, Gemini includeThoughts, Anthropic summarized display) never activated on opted-in runs - openai: restore summary:'auto' alongside explicit reasoning effort — staging always paired them; gating summary purely on agentEvents changed legacy payloads - gemini: Gemini 2 + tools + responseFormat falls back to the silent path; the live loop never applied the deferred responseSchema for AUTO tools - openai-compat loop: malformed tool-argument JSON fails the call instead of executing with defaulted {} args (staging parsed inside the execution try) - openai-compat parser: a vendor id arriving after a synthesized start no longer renames the call (start/end ids stayed consistent) - stream-pump: abort closes the byte projection so a drain blocked on backpressure cannot deadlock teardown - chunk_reset removes the block from the client text order (deployed chat + panel chat) so a reset block re-registers at arrival position — fixes separator/order corruption when parallel blocks stream around a reset - resume route echoes the negotiated X-Sim-Stream-Protocol response header (parity with the chat route); docs: [DONE] wire shape + final-vs-error terminal semantics corrected * chore(deps): exempt pinned @anthropic-ai/sdk 0.114.0 from the release-age gate CI's bun install --frozen-lockfile blocks 0.114.0 (published 2026-07-23, younger than the 7-day supply-chain gate). The pin is exact and was vetted for the agent-events streaming work; following the existing bunfig pattern, the exclusion ages out on 2026-07-30 and should be dropped then. * chore(providers): fix double-cast-allowed annotation placement for the strict boundary audit The audit only recognizes the annotation on the line directly above the cast; two annotations had drifted behind intervening code lines (groq stream params, deepseek loop messages) and the OpenAI reasoning-summary widening cast was never annotated. No behavior change. * fix(chat): settle straggler tool chips as error when final reports failure A failed run can still terminate with a `final` frame carrying success: false; running chips previously settled green regardless of the outcome. * fix(canvas): wire agent stream chrome into run-from-block Run-from-block executions emit the same live stream:thinking/stream:tool events as full runs but registered none of the handlers, so the terminal never showed thinking or tool chips on that path. The per-run chrome (batched thinking writes + tool chip lifecycle + settlement on stream done, block error, and every terminal execution state) is extracted into a shared createAgentStreamChrome factory consumed by both paths. --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>
This commit is contained in:
Bill Leoutsakos
co-authored by
Bill Leoutsakos
Cursor
Vikhyath Mondreti
Bill Leoutsakos
Cursor
Vikhyath Mondreti
parent
03adc8fe1d
commit
d24bc7eccb
@@ -96,6 +96,31 @@ After the agent runs, later blocks read its result by name:
|
||||
|
||||
When a response format is set, its fields are readable directly, like `<agent.sentiment>`.
|
||||
|
||||
## Streamed thinking and tool calls
|
||||
|
||||
While an agent runs, Sim can stream its thinking and tool lifecycle live — the canvas terminal always shows them, and a [deployed chat](/docs/workflows/deployment/agent-events) shows them when its **Include thinking** setting and the client's protocol opt-in agree. What actually streams depends on the model: models marked with full deltas or summaries stream thinking when a Thinking level or Reasoning effort is set (DeepSeek reasoner models always reason); a model marked "Not streamed" thinks internally but its provider withholds the text.
|
||||
|
||||
{/* agent-stream-capabilities:begin — generated by `bun run agent-stream-docs:generate`; do not edit between markers */}
|
||||
|
||||
Live tool-call chips stream for **Anthropic, Azure Anthropic, Google, Vertex AI, DeepSeek, Groq, AWS Bedrock** models. Other providers run tools without live chips — tool results still appear in the block output when the run completes.
|
||||
|
||||
| Provider | Streamed thinking | Models |
|
||||
|----------|-------------------|--------|
|
||||
| OpenAI | Summaries only — Requires OpenAI organization verification; falls back to no summaries. | `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5-pro`, `gpt-5.5`, `gpt-5.4-pro`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-5.2-pro`, `gpt-5.2`, `gpt-5.1`, `gpt-5-pro`, `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `o4-mini`, `o3`, `o3-mini`, `o1` |
|
||||
| Anthropic | Full thinking deltas | `claude-opus-4-6`, `claude-sonnet-4-6`, `claude-opus-4-5`, `claude-opus-4-1`, `claude-sonnet-4-5`, `claude-haiku-4-5` |
|
||||
| Anthropic | Summaries only — These generations omit full thinking; Sim requests summarized thinking on streaming runs. | `claude-fable-5`, `claude-sonnet-5`, `claude-opus-4-8`, `claude-opus-4-7` |
|
||||
| Azure OpenAI | Summaries only — Requires OpenAI organization verification; falls back to no summaries. | `azure/gpt-5.4`, `azure/gpt-5.4-mini`, `azure/gpt-5.4-nano`, `azure/gpt-5.2`, `azure/gpt-5.1`, `azure/gpt-5.1-codex`, `azure/gpt-5`, `azure/gpt-5-mini`, `azure/gpt-5-nano`, `azure/o3`, `azure/o4-mini` |
|
||||
| Azure Anthropic | Full thinking deltas | `azure-anthropic/claude-opus-4-6`, `azure-anthropic/claude-opus-4-5`, `azure-anthropic/claude-sonnet-4-5`, `azure-anthropic/claude-opus-4-1`, `azure-anthropic/claude-haiku-4-5` |
|
||||
| Google | Summaries only | `gemini-3.6-flash`, `gemini-3.5-flash-lite`, `gemini-3.5-flash`, `gemini-3.1-pro-preview`, `gemini-3.1-flash-lite`, `gemini-3-flash-preview`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite` |
|
||||
| Vertex AI | Summaries only | `vertex/gemini-3.5-flash`, `vertex/gemini-3.1-pro-preview`, `vertex/gemini-3.1-flash-lite`, `vertex/gemini-3-flash-preview`, `vertex/gemini-2.5-pro`, `vertex/gemini-2.5-flash`, `vertex/gemini-2.5-flash-lite` |
|
||||
| DeepSeek | Full thinking deltas | `deepseek-v4-pro`, `deepseek-v4-flash`, `deepseek-chat`, `deepseek-reasoner` |
|
||||
| Groq | Full thinking deltas | `groq/openai/gpt-oss-120b`, `groq/openai/gpt-oss-20b`, `groq/openai/gpt-oss-safeguard-20b`, `groq/qwen/qwen3.6-27b` |
|
||||
| Meta | Full thinking deltas | `muse-spark-1.1` |
|
||||
| Kimi | Full thinking deltas | `kimi-k2.6` |
|
||||
| Z.ai | Full thinking deltas | `glm-5.2`, `glm-5.1`, `glm-5`, `glm-5-turbo`, `glm-4.7`, `glm-4.6`, `glm-4.5`, `glm-4.5-air` |
|
||||
|
||||
{/* agent-stream-capabilities:end */}
|
||||
|
||||
## Example
|
||||
|
||||
A workflow that reads an incoming customer message and classifies it:
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: Agent stream events
|
||||
description: Protocol for streaming thinking and tool lifecycle from Agent blocks
|
||||
---
|
||||
|
||||
import { Callout } from 'fumadocs-ui/components/callout'
|
||||
import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
|
||||
|
||||
Agent blocks can emit more than answer text while they run: **provider-exposed thinking** (or reasoning summaries) and a **tool-call lifecycle** (name + status). Sim delivers those as typed events on an opt-in stream protocol.
|
||||
|
||||
<Callout type="info">
|
||||
Sim does **not** invent thinking for providers that do not stream it. Bedrock Converse, many OpenAI-compat models, and non-reasoning chat models stay text-only (plus tools when a live tool loop is wired).
|
||||
</Callout>
|
||||
|
||||
## Dual gate (public chat / simple SSE)
|
||||
|
||||
Thinking and tool frames leave the public chat or workflow simple-SSE path only when **both** are true:
|
||||
|
||||
1. Deployment policy: chat `includeThinking` is enabled (default **off**).
|
||||
2. Request opts in with header:
|
||||
|
||||
```http
|
||||
X-Sim-Stream-Protocol: agent-events-v1
|
||||
```
|
||||
|
||||
Legacy clients that omit the header keep today’s text-only SSE even if the deployment has thinking enabled. The hosted chat UI always sends the header for its own deployments.
|
||||
|
||||
## Simple SSE frame shapes
|
||||
|
||||
Answer text stays on `chunk`. Thinking and tools never reuse `chunk` (so older clients that append every `chunk` into the answer cannot leak thinking).
|
||||
|
||||
| Frame | Meaning |
|
||||
|-------|---------|
|
||||
| `{ "blockId", "chunk": "…" }` | Answer text. Legacy clients receive settled final-turn text; opted-in clients receive it live as the model generates (see below) |
|
||||
| `{ "blockId", "event": "chunk_reset" }` | Opted-in only: discard the block’s streamed answer text — it belonged to a turn that resolved to tool calls |
|
||||
| `{ "blockId", "event": "thinking", "data": "…" }` | Thinking / reasoning summary delta |
|
||||
| `{ "blockId", "event": "tool", "phase": "start"\|"end", "id", "name", "status?" }` | Tool lifecycle (no args / results) |
|
||||
| `{ "event": "final", "data": … }` | Terminal result envelope for a settled execution. `data.success` may be `false` with `data.error` when the workflow itself failed |
|
||||
| `{ "event": "error", "error": "…" }` | Terminal stream failure (timeout, client abort, processing error) — followed by `[DONE]`, never by `final` |
|
||||
| `{ "event": "stream_error", "blockId?", "error" }` | Non-terminal mid-block read issue; the stream keeps going |
|
||||
| `data: "[DONE]"` | Stream closed (JSON-encoded sentinel; always follows the terminal `final` or `error` frame) |
|
||||
|
||||
### Live answer text and intermediate turns
|
||||
|
||||
During a live tool loop, the model can’t be classified mid-turn: text it emits may turn out to be the final answer or preamble before a tool call (the stop reason arrives only at turn end).
|
||||
|
||||
- **Opted-in clients** (protocol header + `includeThinking`) receive answer text as `chunk` frames **live**, token by token. If the turn then resolves to tool calls, a `chunk_reset` frame tells the client to discard that block’s streamed text — the final turn re-streams live after tools settle. Append `chunk`, honor `chunk_reset`, and the displayed answer always converges to the block’s final content.
|
||||
- **Legacy clients** (no header) never see provisional text: only settled final-turn text is emitted as `chunk`, delivered when the turn completes.
|
||||
|
||||
Logs, memory, and the block’s `content` output always contain final-turn text only — intermediate preamble is never persisted.
|
||||
|
||||
### Abort
|
||||
|
||||
Client disconnect or Stop aborts the provider stream. In-flight tools settle as `cancelled`. Cancel is distinct from execution timeout.
|
||||
|
||||
### Reconnect
|
||||
|
||||
Canvas execution-events `stream:chunk`, `stream:chunk_reset`, `stream:thinking`, and `stream:tool` are **live-only** (not buffered for reconnect replay), same as answer chunks. Guaranteed `seq` replay is out of scope.
|
||||
|
||||
## Canvas (draft Run)
|
||||
|
||||
When you click **Run** in the builder, the execution-events SSE path forwards the same sink. The canvas is always opted in — it does not send (or need) the `X-Sim-Stream-Protocol` header; the dual gate applies only to the public chat / simple SSE surface:
|
||||
|
||||
- `stream:thinking` — `{ blockId, text }`
|
||||
- `stream:tool` — `{ blockId, phase, id, name, status? }`
|
||||
- `stream:chunk` — answer text, live on agent-events runs
|
||||
- `stream:chunk_reset` — `{ blockId }`; discard the block’s streamed text (intermediate turn)
|
||||
|
||||
The terminal output panel shows Thinking / Tools chrome above the block output when those events arrive. PII redaction on block output still disables live forwarding (executor rule).
|
||||
|
||||
## Chat deployment toggle
|
||||
|
||||
In **Deploy → Chat**, enable **Include thinking** so public chat can expose thinking/tool frames (still requires the protocol header). Redeploy / update the chat after changing Agent models or tools.
|
||||
|
||||
## Capability honesty (high level)
|
||||
|
||||
Per-model support is generated from the model registry on the [Agent block page](/docs/workflows/blocks/agent#streamed-thinking-and-tool-calls); the table below summarizes by provider family.
|
||||
|
||||
| Family | Thinking | Live tools |
|
||||
|--------|----------|------------|
|
||||
| Anthropic / Azure Anthropic | Yes (incl. redacted blocks in traces). The newest Claude generations omit full thinking; Sim requests summarized thinking for them on streaming runs | Yes |
|
||||
| Gemini / Vertex | Yes when a thinking level is set (thought summaries requested on agent-events runs) | Yes |
|
||||
| OpenAI Responses | Reasoning **summaries** when streamed (requires OpenAI organization verification; unverified orgs fall back to no summaries) | Silent tool loop today (answer streams; chips may be absent) |
|
||||
| OpenAI-compat (Groq, DeepSeek, …) | Only if vendor streams `reasoning` / `reasoning_content` | Live loop where wired (e.g. Groq, DeepSeek) |
|
||||
| Bedrock | Not invented | Yes when streaming tool loop is used |
|
||||
|
||||
## Example (public chat)
|
||||
|
||||
<Tabs items={['cURL']}>
|
||||
<Tab value="cURL">
|
||||
```bash
|
||||
curl -N -X POST 'http://localhost:3000/api/chat/your-slug' \
|
||||
-H 'Content-Type: application/json' \
|
||||
-H 'X-Sim-Stream-Protocol: agent-events-v1' \
|
||||
-d '{"input":"Think briefly, then say hi"}'
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
See also [Chat deployment](/docs/workflows/deployment/chat) for access control and the Include thinking setting.
|
||||
@@ -32,6 +32,7 @@ Configure the following fields, then click **Launch Chat**:
|
||||
| **Output** | Output fields from your workflow blocks returned as the chat response. At least one must be selected. |
|
||||
| **Welcome Message** | Greeting shown before the user sends their first message. Defaults to `"Hi there! How can I help you today?"`. |
|
||||
| **Access Control** | Controls who can access the chat. See [Access Control](#access-control) below. |
|
||||
| **Include thinking** | When enabled, the hosted chat can show provider-exposed thinking and tool lifecycle (name + status). Requires the chat client to send `X-Sim-Stream-Protocol: agent-events-v1` (the hosted UI always does). Default is off. See [Agent stream events](/docs/workflows/deployment/agent-events). |
|
||||
|
||||
### Output Selection
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{
|
||||
"title": "Deployment",
|
||||
"pages": ["index", "api", "chat", "mcp"]
|
||||
"pages": ["index", "api", "chat", "agent-events", "mcp"]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user