fix(core): Correct chainLlm structured-output shape in eval verifier guidance (no-changelog) (#36283)

This commit is contained in:
oleg
2026-08-19 11:07:12 +00:00
committed by GitHub
parent 4a3cb8c66a
commit 884a781ef0
2 changed files with 3 additions and 2 deletions
@@ -10,7 +10,7 @@ This is a test environment. No real credentials or API connections exist. ALL HT
IMPORTANT: Nodes receiving mock responses instead of real API responses is EXPECTED. Missing or mock credentials is EXPECTED. Don't flag these as issues — they are the testing mechanism itself.
IMPORTANT: When an AI root node such as an AI Agent is pinned, its connected AI subnodes (language model, memory, tools, retrievers, parsers) often do not run. This is expected. Evaluate those subnodes from the saved workflow structure, connections, and all-node configs instead of failing only because the subnode did not execute.
IMPORTANT: AI root output shapes differ by node type, and pinned outputs follow the REAL node behavior. An Agent root wraps its result in \`{ "output": ... }\`. A Basic LLM Chain (\`chainLlm\`) does NOT: with a structured output parser attached it emits the parsed fields FLAT at the top level of \`json\` (the parser unwraps any \`output\` envelope); without a parser it emits \`{ "text": "..." }\`. A downstream expression reading \`$json.output.*\` from a chainLlm root can never resolve against the real node either — that is a builder_issue (wrong expression for the node type), NOT a mock/pin issue. Do not claim the pin "should have had an output wrapper" for chainLlm.
IMPORTANT: AI root output shapes differ by node type, and pinned outputs follow the REAL node behavior. An Agent root wraps its result in \`{ "output": ... }\`. A Basic LLM Chain (\`chainLlm\`) WITH a structured output parser attached also wraps: the parser itself emits the \`{ "output": {...} }\` envelope, so the parsed fields live under \`output\`, never flat at the top level of \`json\`. Without a parser, chainLlm emits \`{ "text": "..." }\` and a retrieval QA chain emits \`{ "response": ... }\`. Attribution follows from this: \`$json.output.*\` read from an Agent or a chainLlm-with-parser root is the CORRECT expression — if it failed to resolve, check the pin first; a pin carrying the parsed fields flat (no \`output\` envelope) for such a root is a mock_issue (wrong fixture shape), NOT a builder_issue. Conversely, \`$json.output.*\` against a chainLlm WITHOUT a parser (real shape \`{ "text": ... }\`) can never resolve against the real node — that IS a builder_issue.
IMPORTANT: When the harness resolved the table's real columns, pinned Data Table read outputs mirror that column schema — it reads the actual columns off the table the builder created and enforces them on the pinned rows. In that case, when a downstream expression reads a field that is not among the pinned rows' keys (e.g. \`$json.contact_email\` while the rows carry \`email\`), the same expression would resolve undefined against the real table too — that is a builder_issue (expression references a column the builder never created), NOT a mock/pin issue.
Credential ID values in the workflow JSON (real, placeholder strings, or stale references) never cause execution failures. When a credential ID cannot be resolved, the framework substitutes a mock credential and execution proceeds. Do not cite credential ID values as a root cause of failure under any circumstance.
@@ -94,7 +94,8 @@ export function repairStructuredOutput(
// Information extractors wrap in `{ output: ... }` like parser targets do,
// but have no ai_outputParser connection — include them explicitly. Their
// envelope is always `output`, even when no `__schema__` resolves; other
// roots must declare theirs (chainLlm with a parser emits fields FLAT).
// roots must declare theirs via their with-parser `__schema__` variant
// (Agent and chainLlm both declare `output` — the parser emits that wrapper).
const targets = new Set<string>(findOutputParserTargets(workflow).keys());
const extractorNames = new Set<string>();
for (const node of workflow.nodes) {