mirror of
https://github.com/simstudioai/sim.git
synced 2026-10-16 14:50:06 +08:00
perf(tools): read tool metadata instead of the registry on client paths (#6155)
* perf(tools): read tool metadata instead of the registry on client paths Cuts the last four edges that pulled `@/tools/registry` into the workspace shell. Every workspace route drops ~4,700 modules: route before after /w (canvas) 6,592 1,908 -71% /logs 6,227 1,543 -75% /tables 5,903 1,217 -79% /files 5,996 1,310 -78% workspace layout 5,751 1,063 -82% Dev cold compile of the canvas, n=3, cache cleared between runs: before 32.3s / 31.4s / 30.1s RSS 9.0-12.5 GB after 22.4s / 22.2s / 21.6s RSS 7.8-9.2 GB That lands where the `dev:minimal` escape hatch measured (20.0s / 6.7 GB) without its downside — `dev:minimal` swaps in curated registries that drop ~250 services, whereas this keeps every tool working. Rewired: - `block-outputs` -> `getToolOutputsMetadata` (needed `outputs`) - `serializer` -> `getToolParams` (needed `params`) - `validation` -> `hasToolId` (needed existence only) - `tools/params` -> `getToolMetadata` (needed `params`, `oauth`, `name`) `tools/params.ts` was the stubborn one: `mcp-dynamic-args.tsx` imports only `formatParameterLabel` from it, so the whole registry rode in behind a string helper — the same shape as the `mergeToolParameters` edge cut earlier. Adds a third generated artifact, `tool-ids.ts` (~110 KB). Resolution needs only the key set, so `@/tools/metadata` and `@/tools/metadata-outputs` both resolve through it and stay independent of each other, and an existence check costs ~110 KB instead of ~4 MB. Behaviour preservation was the risk here: `getTool` resolves an unversioned name onto its newest version, and a plain key lookup would have silently reported 246 versioned tools as missing. `resolveToolId` is reproduced against the id set and differentially tested — 4,404 probes (every id, every stripped base name, and an unknown) comparing old vs new resolution and existence: 0 mismatches. `ToolWithParameters.toolConfig` and `SubBlocksForToolInput.toolConfig` narrow from `ToolConfig` to `ToolMetadata`. The only external reader is `tool-input.tsx`, which uses `.name`. * docs(tools): point the boundary skill at the three metadata modules The skill still routed `hasToolMetadata` and `getToolIds` to `@/tools/metadata`, but this PR moved id resolution into `@/tools/tool-ids`. Left as-is it would send the next caller to the 4 MB module for an existence check that costs 110 KB — the exact mistake the skill exists to prevent. Also records the two properties a caller can silently get wrong: lookups guard with `Object.hasOwn` (a bare bracket lookup returns inherited prototype members), and they resolve unversioned names (246 tools are versioned, and a plain lookup reports them missing rather than crashing). * fix(tools): cut the settings-route registry edge and fix serializer test mocks Two findings from review, both real. The settings route still reached the registry: settings/[section]/page.tsx -> settings.tsx -> (dynamic import) ee/access-control/components/access-control.tsx -> group-detail.tsx -> tools/utils.ts -> tools/registry.ts It reads `getTool(id)?.name` — metadata — so it moves to `getToolMetadata`. The earlier audit missed it because it walked only from the canvas route, and the edge hides behind a dynamic `import()` that a static walk skips. Serializer tests mocked the wrong module. `Serializer` now reads params via `getToolParams` from `@/tools/metadata`, but the tests still only mocked `@/tools/utils`, so they controlled nothing and passed because the real generated artifacts happen to agree with the fixtures. Adds `toolsMetadataMock` to `@sim/testing/mocks`, backed by the same `mockToolConfigs` as `toolsUtilsMock` so a test mocking both sees one consistent tool universe, and mocks it in the three serializer suites. Verified the mock is now load-bearing: pointing it at a sentinel param makes the three user-only-required validation tests fail, and restoring it returns all 110 serializer tests to green. Before this they passed either way. * fix(tools): freeze the tool id array handed out by getToolIds `getToolIds()` returned the module's internal array by reference, so a caller doing `getToolIds().sort()` would reorder it in place and silently corrupt every later lookup — the in-place-mutation footgun `.claude/rules/sim-react-performance.md` calls out. Frozen rather than copied: the array is consumed in loops, so copying would allocate on every call. Freezing makes the mutation throw instead of corrupt, and `[...getToolIds()].sort()` still works. Return type is now `readonly string[]`, so the mistake is a compile error rather than a runtime surprise. No caller mutates it today; this is closing the hole, not fixing a live bug. * test(tools): enforce that the two tool-id resolvers never diverge `resolveToolId` now exists twice on purpose — `@/tools/utils` resolves against the live registry (so a tool added before regeneration still resolves at runtime), `@/tools/tool-ids` against the generated id list (so client code resolves without importing 4,300 tools). Nothing structurally kept them in step; a change to versioning logic in one would silently drift from the other. `tool-metadata:check` now asserts they agree across every id, every stripped base name, and an unknown — 4,404 probes — and only after the staleness check passes, so a missing regeneration reports as staleness rather than as drift. Verified it fails: breaking resolution for `gmail*` exits 1; restoring it passes. It cannot live in a vitest suite. `vitest.setup.ts` globally mocks `@/tools/registry` to an empty map, so `getTool` resolves nothing there — a parity test written as a spec passes or fails for the wrong reason. Both facts are recorded where the code is. Both resolvers stay exported. An earlier pass here un-exported the `@/tools/utils` one as dead; `tools/utils.server.ts` imports it through a multi-line import that a grep missed, and `tsc` caught it. Its doc now says which resolver a caller should reach for instead of leaving two identically-named functions unexplained.
This commit is contained in:
@@ -19,17 +19,20 @@ You keep the 4,300-tool executable registry out of module graphs that don't exec
|
||||
|
||||
| you need | import | notes |
|
||||
| --- | --- | --- |
|
||||
| whether a tool id exists | `hasToolMetadata` from `@/tools/metadata` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | separate module on purpose — see below |
|
||||
| every tool id | `getToolIds` from `@/tools/metadata` | |
|
||||
| whether a tool id exists | `hasToolId` from `@/tools/tool-ids` | ~110 KB — the cheapest module |
|
||||
| to resolve an unversioned name | `resolveToolId` from `@/tools/tool-ids` | |
|
||||
| every tool id | `getToolIds` from `@/tools/tool-ids` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | ~4 MB |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | ~4 MB, separate on purpose |
|
||||
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
|
||||
|
||||
Outputs live in their own module because they are roughly two thirds of the generated data and have a single consumer. Importing `@/tools/metadata` must never pull them — do not "helpfully" re-export one from the other.
|
||||
Three modules, cheapest first. Ids are their own artifact because resolution and existence checks need only the key set; outputs are their own because they are the larger half of the data with a single consumer. `@/tools/metadata` and `@/tools/metadata-outputs` both resolve ids through `@/tools/tool-ids`, which is what keeps them independent of each other — do not "helpfully" re-export one from another, or every caller pays for all three.
|
||||
|
||||
All lookups guard with `Object.hasOwn`. `JSON.parse` yields an object with the normal prototype, so a bare bracket lookup returns inherited members: `getToolMetadata('constructor')` returned a *function* typed as tool metadata before that was fixed.
|
||||
|
||||
## The generated artifacts
|
||||
|
||||
`apps/sim/tools/generated/tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
`apps/sim/tools/generated/tool-ids.ts`, `tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # after adding/changing a tool
|
||||
@@ -43,6 +46,20 @@ Three non-obvious properties, each of which was measured and is easy to undo by
|
||||
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
|
||||
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
|
||||
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
|
||||
- **Lookups resolve versions.** `getTool` maps an unversioned name onto the newest version, and 246 tools are versioned. A plain key lookup would silently report them missing — a quiet correctness bug, not a crash. `resolveToolId` reproduces that against the id set and is differentially tested against the original.
|
||||
|
||||
## Testing code that reads tool metadata
|
||||
|
||||
Mock the module the code under test actually reads. `vi.mock('@/tools/utils', () => toolsUtilsMock)` only controls `getTool`; code that reads `params`/`outputs`/`name` goes through `@/tools/metadata`, so mocking `tools/utils` there is a **no-op that still passes** — because the real generated artifacts happen to agree with the mock fixtures. The test looks green while controlling nothing.
|
||||
|
||||
```ts
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock) // executable lookup
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock) // params / outputs / name
|
||||
```
|
||||
|
||||
Both are backed by the same `mockToolConfigs`, so mocking both gives one consistent tool universe. If you are unsure whether a mock is load-bearing, change a fixture value to a sentinel and confirm the test fails.
|
||||
|
||||
## How to verify an edge actually got cut
|
||||
|
||||
@@ -59,9 +76,9 @@ Reference points measured on this repo:
|
||||
| `tools/registry.ts` reachable | ~4,900 |
|
||||
| `tools/merge-params.ts` (leaf) | 2 |
|
||||
| `providers/utils.ts` after cutting its `params` edge | 22 |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,591, of which 4,689 are the registry |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,592 before, 1,908 after |
|
||||
|
||||
The canvas route reaches the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moves the module count by ~1. They must all be cut before anything improves; measure the route, not the file you edited.
|
||||
The canvas route reached the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moved the module count by ~1. They all had to go before anything improved; measure the route, not the file you edited.
|
||||
|
||||
## When adding a new caller
|
||||
|
||||
|
||||
@@ -18,17 +18,20 @@ You keep the 4,300-tool executable registry out of module graphs that don't exec
|
||||
|
||||
| you need | import | notes |
|
||||
| --- | --- | --- |
|
||||
| whether a tool id exists | `hasToolMetadata` from `@/tools/metadata` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | separate module on purpose — see below |
|
||||
| every tool id | `getToolIds` from `@/tools/metadata` | |
|
||||
| whether a tool id exists | `hasToolId` from `@/tools/tool-ids` | ~110 KB — the cheapest module |
|
||||
| to resolve an unversioned name | `resolveToolId` from `@/tools/tool-ids` | |
|
||||
| every tool id | `getToolIds` from `@/tools/tool-ids` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | ~4 MB |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | ~4 MB, separate on purpose |
|
||||
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
|
||||
|
||||
Outputs live in their own module because they are roughly two thirds of the generated data and have a single consumer. Importing `@/tools/metadata` must never pull them — do not "helpfully" re-export one from the other.
|
||||
Three modules, cheapest first. Ids are their own artifact because resolution and existence checks need only the key set; outputs are their own because they are the larger half of the data with a single consumer. `@/tools/metadata` and `@/tools/metadata-outputs` both resolve ids through `@/tools/tool-ids`, which is what keeps them independent of each other — do not "helpfully" re-export one from another, or every caller pays for all three.
|
||||
|
||||
All lookups guard with `Object.hasOwn`. `JSON.parse` yields an object with the normal prototype, so a bare bracket lookup returns inherited members: `getToolMetadata('constructor')` returned a *function* typed as tool metadata before that was fixed.
|
||||
|
||||
## The generated artifacts
|
||||
|
||||
`apps/sim/tools/generated/tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
`apps/sim/tools/generated/tool-ids.ts`, `tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # after adding/changing a tool
|
||||
@@ -42,6 +45,20 @@ Three non-obvious properties, each of which was measured and is easy to undo by
|
||||
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
|
||||
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
|
||||
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
|
||||
- **Lookups resolve versions.** `getTool` maps an unversioned name onto the newest version, and 246 tools are versioned. A plain key lookup would silently report them missing — a quiet correctness bug, not a crash. `resolveToolId` reproduces that against the id set and is differentially tested against the original.
|
||||
|
||||
## Testing code that reads tool metadata
|
||||
|
||||
Mock the module the code under test actually reads. `vi.mock('@/tools/utils', () => toolsUtilsMock)` only controls `getTool`; code that reads `params`/`outputs`/`name` goes through `@/tools/metadata`, so mocking `tools/utils` there is a **no-op that still passes** — because the real generated artifacts happen to agree with the mock fixtures. The test looks green while controlling nothing.
|
||||
|
||||
```ts
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock) // executable lookup
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock) // params / outputs / name
|
||||
```
|
||||
|
||||
Both are backed by the same `mockToolConfigs`, so mocking both gives one consistent tool universe. If you are unsure whether a mock is load-bearing, change a fixture value to a sentinel and confirm the test fails.
|
||||
|
||||
## How to verify an edge actually got cut
|
||||
|
||||
@@ -58,9 +75,9 @@ Reference points measured on this repo:
|
||||
| `tools/registry.ts` reachable | ~4,900 |
|
||||
| `tools/merge-params.ts` (leaf) | 2 |
|
||||
| `providers/utils.ts` after cutting its `params` edge | 22 |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,591, of which 4,689 are the registry |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,592 before, 1,908 after |
|
||||
|
||||
The canvas route reaches the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moves the module count by ~1. They must all be cut before anything improves; measure the route, not the file you edited.
|
||||
The canvas route reached the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moved the module count by ~1. They all had to go before anything improved; measure the route, not the file you edited.
|
||||
|
||||
## When adding a new caller
|
||||
|
||||
|
||||
@@ -14,17 +14,20 @@ You keep the 4,300-tool executable registry out of module graphs that don't exec
|
||||
|
||||
| you need | import | notes |
|
||||
| --- | --- | --- |
|
||||
| whether a tool id exists | `hasToolMetadata` from `@/tools/metadata` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | separate module on purpose — see below |
|
||||
| every tool id | `getToolIds` from `@/tools/metadata` | |
|
||||
| whether a tool id exists | `hasToolId` from `@/tools/tool-ids` | ~110 KB — the cheapest module |
|
||||
| to resolve an unversioned name | `resolveToolId` from `@/tools/tool-ids` | |
|
||||
| every tool id | `getToolIds` from `@/tools/tool-ids` | |
|
||||
| a tool's params | `getToolParams` / `getToolMetadata` from `@/tools/metadata` | ~4 MB |
|
||||
| a tool's declared outputs | `getToolOutputsMetadata` from `@/tools/metadata-outputs` | ~4 MB, separate on purpose |
|
||||
| to **execute** a tool | `getTool` from `@/tools/utils`, or `@/tools/utils.server` | server paths only |
|
||||
|
||||
Outputs live in their own module because they are roughly two thirds of the generated data and have a single consumer. Importing `@/tools/metadata` must never pull them — do not "helpfully" re-export one from the other.
|
||||
Three modules, cheapest first. Ids are their own artifact because resolution and existence checks need only the key set; outputs are their own because they are the larger half of the data with a single consumer. `@/tools/metadata` and `@/tools/metadata-outputs` both resolve ids through `@/tools/tool-ids`, which is what keeps them independent of each other — do not "helpfully" re-export one from another, or every caller pays for all three.
|
||||
|
||||
All lookups guard with `Object.hasOwn`. `JSON.parse` yields an object with the normal prototype, so a bare bracket lookup returns inherited members: `getToolMetadata('constructor')` returned a *function* typed as tool metadata before that was fixed.
|
||||
|
||||
## The generated artifacts
|
||||
|
||||
`apps/sim/tools/generated/tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
`apps/sim/tools/generated/tool-ids.ts`, `tool-metadata.ts` and `tool-outputs.ts` are produced by `scripts/sync-tool-metadata.ts`:
|
||||
|
||||
```bash
|
||||
bun run tool-metadata:generate # after adding/changing a tool
|
||||
@@ -38,6 +41,20 @@ Three non-obvious properties, each of which was measured and is easy to undo by
|
||||
- **The data is a JSON string parsed at runtime, not an imported `.json` and not an object literal.** With `resolveJsonModule` (which this repo enables), a `.json` import makes TypeScript infer a literal type for all 4,300+ entries and takes `tsc --noEmit` from **12.6s to 8m07s** — a 38x regression. An ambient `declare module` does *not* short-circuit it, and an object literal costs the same. A single string literal is one cheap token for both the compiler and the bundler, and `JSON.parse` beats evaluating the equivalent literal at runtime. Do not "clean this up" into a `.json` import.
|
||||
- **The generator refuses to emit function values.** If you add a field to `METADATA_FIELDS` that contains a closure, generation fails loudly rather than shipping executable config to the client. `hosting` and `schemaEnrichment` are excluded for exactly this reason (`hosting.enabled`, `pricing`, and `enrichSchema` are functions) — they are server-only.
|
||||
- **Empty param entries are stripped.** The registry contains one (`stt_deepgram_v2`), which crashes callers that read `param.type` while iterating.
|
||||
- **Lookups resolve versions.** `getTool` maps an unversioned name onto the newest version, and 246 tools are versioned. A plain key lookup would silently report them missing — a quiet correctness bug, not a crash. `resolveToolId` reproduces that against the id set and is differentially tested against the original.
|
||||
|
||||
## Testing code that reads tool metadata
|
||||
|
||||
Mock the module the code under test actually reads. `vi.mock('@/tools/utils', () => toolsUtilsMock)` only controls `getTool`; code that reads `params`/`outputs`/`name` goes through `@/tools/metadata`, so mocking `tools/utils` there is a **no-op that still passes** — because the real generated artifacts happen to agree with the mock fixtures. The test looks green while controlling nothing.
|
||||
|
||||
```ts
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock) // executable lookup
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock) // params / outputs / name
|
||||
```
|
||||
|
||||
Both are backed by the same `mockToolConfigs`, so mocking both gives one consistent tool universe. If you are unsure whether a mock is load-bearing, change a fixture value to a sentinel and confirm the test fails.
|
||||
|
||||
## How to verify an edge actually got cut
|
||||
|
||||
@@ -54,9 +71,9 @@ Reference points measured on this repo:
|
||||
| `tools/registry.ts` reachable | ~4,900 |
|
||||
| `tools/merge-params.ts` (leaf) | 2 |
|
||||
| `providers/utils.ts` after cutting its `params` edge | 22 |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,591, of which 4,689 are the registry |
|
||||
| `app/workspace/[workspaceId]/w/page.tsx` (canvas) | 6,592 before, 1,908 after |
|
||||
|
||||
The canvas route reaches the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moves the module count by ~1. They must all be cut before anything improves; measure the route, not the file you edited.
|
||||
The canvas route reached the registry through **four** redundant edges — `providers/utils` (via `tools/params`), `lib/workflows/blocks/block-outputs`, `lib/workflows/sanitization/validation`, and `serializer/index`. Cutting any one alone moved the module count by ~1. They all had to go before anything improved; measure the route, not the file you edited.
|
||||
|
||||
## When adding a new caller
|
||||
|
||||
|
||||
@@ -76,7 +76,7 @@ import {
|
||||
import type { ProviderId } from '@/providers/types'
|
||||
import { getAllProviderIds, getProviderFromModel } from '@/providers/utils'
|
||||
import type { ProviderName } from '@/stores/providers'
|
||||
import { getTool } from '@/tools/utils'
|
||||
import { getToolMetadata } from '@/tools/metadata'
|
||||
|
||||
const logger = createLogger('AccessControlGroupDetail')
|
||||
|
||||
@@ -733,7 +733,7 @@ function BlockToolRow({
|
||||
const checkboxId = `block-${block.type}`
|
||||
|
||||
const toolItems = useMemo<DenylistGridItem[]>(
|
||||
() => (block.tools?.access ?? []).map((id) => ({ id, label: getTool(id)?.name ?? id })),
|
||||
() => (block.tools?.access ?? []).map((id) => ({ id, label: getToolMetadata(id)?.name ?? id })),
|
||||
[block.tools?.access]
|
||||
)
|
||||
const isExpandable = toolItems.length > 1
|
||||
|
||||
@@ -22,7 +22,7 @@ import {
|
||||
type OutputCondition,
|
||||
type OutputFieldDefinition,
|
||||
} from '@/blocks/types'
|
||||
import { getTool } from '@/tools/utils'
|
||||
import { getToolOutputsMetadata } from '@/tools/metadata-outputs'
|
||||
import { getTrigger, isTriggerValid } from '@/triggers'
|
||||
|
||||
const logger = createLogger('BlockOutputs')
|
||||
@@ -681,13 +681,13 @@ export function getToolOutputs(
|
||||
const toolId = blockConfig.tools.config.tool(params)
|
||||
if (!toolId) return {}
|
||||
|
||||
const toolConfig = getTool(toolId)
|
||||
if (!toolConfig?.outputs) return {}
|
||||
const toolOutputs = getToolOutputsMetadata(toolId)
|
||||
if (!toolOutputs) return {}
|
||||
if (includeHidden) {
|
||||
return toolConfig.outputs
|
||||
return toolOutputs
|
||||
}
|
||||
return Object.fromEntries(
|
||||
Object.entries(toolConfig.outputs).filter(([_, def]) => !isHiddenFromDisplay(def))
|
||||
Object.entries(toolOutputs).filter(([_, def]) => !isHiddenFromDisplay(def))
|
||||
)
|
||||
} catch (error) {
|
||||
logger.warn('Failed to get tool outputs', { error })
|
||||
|
||||
@@ -4,7 +4,7 @@ import { isRecordLike } from '@sim/utils/object'
|
||||
import { getBlock } from '@/blocks/registry'
|
||||
import { isCustomTool, isMcpTool } from '@/executor/constants'
|
||||
import type { BlockState, WorkflowState } from '@/stores/workflows/workflow/types'
|
||||
import { getTool } from '@/tools/utils'
|
||||
import { hasToolId } from '@/tools/tool-ids'
|
||||
|
||||
const logger = createLogger('WorkflowValidation')
|
||||
|
||||
@@ -305,8 +305,7 @@ export function validateToolReference(
|
||||
|
||||
if (!isCustomTool(toolId) && !isMcpTool(toolId)) {
|
||||
// For built-in tools, verify they exist
|
||||
const tool = getTool(toolId)
|
||||
if (!tool) {
|
||||
if (!hasToolId(toolId)) {
|
||||
return `Block ${blockName || 'unknown'} (${blockType}): references non-existent tool '${toolId}'`
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
* - a deleted *input* on a live custom block no longer leaks its stale value into
|
||||
* the child `inputMapping` (Bug 1).
|
||||
*/
|
||||
import { toolsUtilsMock } from '@sim/testing/mocks'
|
||||
import { toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
// Build the custom-block configs INSIDE the factory (hoisted) so no top-level
|
||||
@@ -33,6 +33,7 @@ vi.mock('@/blocks', async () => {
|
||||
return { getBlock, getAllBlocks: () => Object.values(mockBlockConfigs) }
|
||||
})
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock)
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock)
|
||||
|
||||
import { extractBlockParams, Serializer } from '@/serializer/index'
|
||||
|
||||
|
||||
@@ -5,12 +5,13 @@
|
||||
* (collectBlockFieldIssues / extractBlockParams) — the single source of truth
|
||||
* shared by the serializer's required-field validation and the copilot lint.
|
||||
*/
|
||||
import { blocksMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { svcConfig } = vi.hoisted(() => ({ svcConfig: { value: null as any } }))
|
||||
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock)
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock)
|
||||
vi.mock('@/blocks', () => ({
|
||||
...blocksMock,
|
||||
getBlock: (type: string) => (type === 'svc' ? svcConfig.value : blocksMock.getBlock(type)),
|
||||
|
||||
@@ -18,13 +18,14 @@ import {
|
||||
createMinimalWorkflowState,
|
||||
createMissingMetadataWorkflow,
|
||||
} from '@sim/testing/factories'
|
||||
import { blocksMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
import { blocksMock, toolsMetadataMock, toolsUtilsMock } from '@sim/testing/mocks'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { Serializer } from '@/serializer/index'
|
||||
import type { SerializedWorkflow } from '@/serializer/types'
|
||||
|
||||
vi.mock('@/blocks', () => blocksMock)
|
||||
vi.mock('@/tools/utils', () => toolsUtilsMock)
|
||||
vi.mock('@/tools/metadata', () => toolsMetadataMock)
|
||||
|
||||
describe('Serializer', () => {
|
||||
describe('serializeWorkflow', () => {
|
||||
|
||||
@@ -21,7 +21,7 @@ import type { SubBlockConfig } from '@/blocks/types'
|
||||
import type { SerializedBlock, SerializedWorkflow } from '@/serializer/types'
|
||||
import type { BlockState, Loop, Parallel } from '@/stores/workflows/workflow/types'
|
||||
import { generateLoopBlocks, generateParallelBlocks } from '@/stores/workflows/workflow/utils'
|
||||
import { getTool } from '@/tools/utils'
|
||||
import { getToolParams } from '@/tools/metadata'
|
||||
|
||||
const logger = createLogger('Serializer')
|
||||
|
||||
@@ -637,13 +637,13 @@ export function collectBlockFieldIssues(
|
||||
// Get the tool configuration to check parameter visibility
|
||||
const toolAccess = blockConfig.tools?.access
|
||||
const currentToolId = toolAccess?.length > 0 ? selectToolId(blockConfig, params) : null
|
||||
const currentTool = currentToolId ? getTool(currentToolId) : null
|
||||
const currentToolParams = currentToolId ? getToolParams(currentToolId) : undefined
|
||||
|
||||
// Validate tool parameters (for blocks with tools).
|
||||
// Lookup contract: a tool param's value lives under its own paramId in `params`.
|
||||
// Block subBlocks align via either `id === paramId` or `canonicalParamId === paramId`.
|
||||
if (currentTool) {
|
||||
Object.entries(currentTool.params || {}).forEach(([paramId, paramConfig]: [string, any]) => {
|
||||
if (currentToolParams) {
|
||||
Object.entries(currentToolParams).forEach(([paramId, paramConfig]: [string, any]) => {
|
||||
if (paramConfig.required && paramConfig.visibility === 'user-only') {
|
||||
const matchingConfigs =
|
||||
blockConfig.subBlocks?.filter(
|
||||
@@ -699,7 +699,7 @@ export function collectBlockFieldIssues(
|
||||
}
|
||||
|
||||
// Validate required subBlocks not covered by tool params (e.g., blocks with empty tools.access)
|
||||
const validatedByTool = new Set(currentTool ? Object.keys(currentTool.params || {}) : [])
|
||||
const validatedByTool = new Set(currentToolParams ? Object.keys(currentToolParams) : [])
|
||||
|
||||
blockConfig.subBlocks?.forEach((subBlockConfig: SubBlockConfig) => {
|
||||
if (validatedByTool.has(subBlockConfig.id)) {
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -1,4 +1,5 @@
|
||||
import rawOutputs from '@/tools/generated/tool-outputs'
|
||||
import { resolveToolId } from '@/tools/tool-ids'
|
||||
import type { ToolConfig } from '@/tools/types'
|
||||
|
||||
/**
|
||||
@@ -28,5 +29,6 @@ const outputs: Record<string, ToolOutputs> = rawOutputs as Record<string, ToolOu
|
||||
* `Object.hasOwn` rather than a bare lookup — see `getToolMetadata` for why.
|
||||
*/
|
||||
export function getToolOutputsMetadata(toolId: string): ToolOutputs | undefined {
|
||||
return Object.hasOwn(outputs, toolId) ? outputs[toolId] : undefined
|
||||
const resolved = resolveToolId(toolId)
|
||||
return Object.hasOwn(outputs, resolved) ? outputs[resolved] : undefined
|
||||
}
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { getToolIds, getToolMetadata, getToolParams, hasToolMetadata } from '@/tools/metadata'
|
||||
import { getToolMetadata, getToolParams } from '@/tools/metadata'
|
||||
import { getToolOutputsMetadata } from '@/tools/metadata-outputs'
|
||||
import { getToolIds, hasToolId, resolveToolId } from '@/tools/tool-ids'
|
||||
|
||||
/**
|
||||
* Guards the properties the generated artifacts are relied on for. The
|
||||
@@ -22,7 +23,7 @@ describe('generated tool metadata', () => {
|
||||
})
|
||||
|
||||
it('reports unknown tools as absent without throwing', () => {
|
||||
expect(hasToolMetadata('definitely_not_a_tool')).toBe(false)
|
||||
expect(hasToolId('definitely_not_a_tool')).toBe(false)
|
||||
expect(getToolMetadata('definitely_not_a_tool')).toBeUndefined()
|
||||
expect(getToolParams('definitely_not_a_tool')).toBeUndefined()
|
||||
expect(getToolOutputsMetadata('definitely_not_a_tool')).toBeUndefined()
|
||||
@@ -43,10 +44,44 @@ describe('generated tool metadata', () => {
|
||||
expect(getToolMetadata(key)).toBeUndefined()
|
||||
expect(getToolParams(key)).toBeUndefined()
|
||||
expect(getToolOutputsMetadata(key)).toBeUndefined()
|
||||
expect(hasToolMetadata(key)).toBe(false)
|
||||
expect(hasToolId(key)).toBe(false)
|
||||
}
|
||||
)
|
||||
|
||||
/**
|
||||
* `getTool` resolves an unversioned name onto the newest version, and callers
|
||||
* migrated off it depend on that. A plain key lookup would silently report
|
||||
* versioned tools as missing.
|
||||
*/
|
||||
describe('version resolution', () => {
|
||||
/**
|
||||
* Only a versioned id whose base name is *not* itself registered exercises
|
||||
* resolution — where both exist, the base name resolves to itself.
|
||||
*/
|
||||
const versionedId = getToolIds().find(
|
||||
(id) => /_v[2-9]\d*$/.test(id) && !getToolIds().includes(id.replace(/_v\d+$/, ''))
|
||||
)
|
||||
|
||||
it('has at least one versioned tool to exercise', () => {
|
||||
expect(versionedId).toBeDefined()
|
||||
})
|
||||
|
||||
it('maps an unversioned name onto the newest version', () => {
|
||||
const baseName = (versionedId as string).replace(/_v\d+$/, '')
|
||||
expect(getToolIds()).not.toContain(baseName)
|
||||
expect(resolveToolId(baseName)).toBe(versionedId)
|
||||
expect(hasToolId(baseName)).toBe(true)
|
||||
expect(getToolMetadata(baseName)?.id).toBe(getToolMetadata(versionedId as string)?.id)
|
||||
expect(getToolOutputsMetadata(baseName)).toEqual(
|
||||
getToolOutputsMetadata(versionedId as string)
|
||||
)
|
||||
})
|
||||
|
||||
it('returns an unknown name unchanged', () => {
|
||||
expect(resolveToolId('definitely_not_a_tool')).toBe('definitely_not_a_tool')
|
||||
})
|
||||
})
|
||||
|
||||
/**
|
||||
* The registry contains a null param entry (`stt_deepgram_v2`), which crashes
|
||||
* any consumer that iterates params unguarded. The generator strips those, so
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import rawMetadata from '@/tools/generated/tool-metadata'
|
||||
import { resolveToolId } from '@/tools/tool-ids'
|
||||
import type { OAuthConfig, ToolConfig } from '@/tools/types'
|
||||
|
||||
/**
|
||||
@@ -15,7 +16,10 @@ import type { OAuthConfig, ToolConfig } from '@/tools/types'
|
||||
*
|
||||
* Outputs live in `@/tools/metadata-outputs`, not here: they are the larger half
|
||||
* of the data and have a single consumer, so keeping them in a separate module
|
||||
* means callers that only need params don't pay for them.
|
||||
* means callers that only need params don't pay for them. Callers that need
|
||||
* neither should use `@/tools/tool-ids`, which is ~40x smaller again.
|
||||
*
|
||||
* Lookups resolve unversioned names the same way `getTool` does.
|
||||
*/
|
||||
export interface ToolMetadata {
|
||||
id: string
|
||||
@@ -33,16 +37,6 @@ export interface ToolMetadata {
|
||||
*/
|
||||
const metadata: Record<string, ToolMetadata> = rawMetadata as Record<string, ToolMetadata>
|
||||
|
||||
/** Every registered tool id, including versioned variants. */
|
||||
export function getToolIds(): string[] {
|
||||
return Object.keys(metadata)
|
||||
}
|
||||
|
||||
/** Whether `toolId` names a built-in tool. Cheaper than resolving its metadata. */
|
||||
export function hasToolMetadata(toolId: string): boolean {
|
||||
return Object.hasOwn(metadata, toolId)
|
||||
}
|
||||
|
||||
/**
|
||||
* Serializable metadata for a built-in tool, or `undefined` if unknown.
|
||||
*
|
||||
@@ -52,7 +46,8 @@ export function hasToolMetadata(toolId: string): boolean {
|
||||
* metadata.
|
||||
*/
|
||||
export function getToolMetadata(toolId: string): ToolMetadata | undefined {
|
||||
return Object.hasOwn(metadata, toolId) ? metadata[toolId] : undefined
|
||||
const resolved = resolveToolId(toolId)
|
||||
return Object.hasOwn(metadata, resolved) ? metadata[resolved] : undefined
|
||||
}
|
||||
|
||||
/** Declared parameters for a built-in tool, or `undefined` if unknown. */
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { afterAll, describe, expect, it, vi } from 'vitest'
|
||||
import { mergeToolParameters } from '@/tools/merge-params'
|
||||
import * as toolMetadata from '@/tools/metadata'
|
||||
import {
|
||||
createExecutionToolSchema,
|
||||
createLLMToolSchema,
|
||||
@@ -15,7 +16,6 @@ import {
|
||||
validateToolParameters,
|
||||
} from '@/tools/params'
|
||||
import type { HttpMethod, ParameterVisibility } from '@/tools/types'
|
||||
import * as toolsUtils from '@/tools/utils'
|
||||
|
||||
const mockToolConfig = {
|
||||
id: 'test_tool',
|
||||
@@ -58,11 +58,13 @@ const mockToolConfig = {
|
||||
|
||||
/**
|
||||
* Spy on the real module namespace instead of vi.mock: under `isolate: false`
|
||||
* `@/tools/params` may already be cached bound to the real `@/tools/utils`
|
||||
* `@/tools/params` may already be cached bound to the real `@/tools/metadata`
|
||||
* module, so patching the shared namespace is the only wiring that always
|
||||
* applies.
|
||||
*/
|
||||
const getToolSpy = vi.spyOn(toolsUtils, 'getTool').mockImplementation(((toolId: string) => {
|
||||
const getToolSpy = vi.spyOn(toolMetadata, 'getToolMetadata').mockImplementation(((
|
||||
toolId: string
|
||||
) => {
|
||||
if (toolId === 'test_tool') {
|
||||
return mockToolConfig
|
||||
}
|
||||
@@ -76,7 +78,7 @@ const getToolSpy = vi.spyOn(toolsUtils, 'getTool').mockImplementation(((toolId:
|
||||
}
|
||||
}
|
||||
return null
|
||||
}) as unknown as typeof toolsUtils.getTool)
|
||||
}) as unknown as typeof toolMetadata.getToolMetadata)
|
||||
|
||||
afterAll(() => {
|
||||
getToolSpy.mockRestore()
|
||||
|
||||
@@ -18,6 +18,7 @@ import type {
|
||||
GenerationType,
|
||||
} from '@/blocks/types'
|
||||
import { isNonEmpty } from '@/tools/merge-params'
|
||||
import { getToolMetadata, type ToolMetadata } from '@/tools/metadata'
|
||||
import { safeAssign } from '@/tools/safe-assign'
|
||||
import type {
|
||||
OAuthConfig,
|
||||
@@ -25,7 +26,6 @@ import type {
|
||||
ToolConfig,
|
||||
ToolParameterItemSchema,
|
||||
} from '@/tools/types'
|
||||
import { getTool } from '@/tools/utils'
|
||||
|
||||
const logger = createLogger('ToolsParams')
|
||||
type ToolParamDefinition = ToolConfig['params'][string]
|
||||
@@ -173,7 +173,7 @@ export interface ToolParameterConfig {
|
||||
}
|
||||
|
||||
export interface ToolWithParameters {
|
||||
toolConfig: ToolConfig
|
||||
toolConfig: ToolMetadata
|
||||
allParameters: ToolParameterConfig[]
|
||||
userInputParameters: ToolParameterConfig[] // Parameters shown to user
|
||||
requiredParameters: ToolParameterConfig[] // Must be filled by user or LLM
|
||||
@@ -301,7 +301,7 @@ export function getToolParametersConfig(
|
||||
blockConfigOverride?: Pick<ToolInputBlockConfig, 'subBlocks'>
|
||||
): ToolWithParameters | null {
|
||||
try {
|
||||
const toolConfig = getTool(toolId)
|
||||
const toolConfig = getToolMetadata(toolId)
|
||||
if (!toolConfig) {
|
||||
logger.warn(`Tool not found: ${toolId}`)
|
||||
return null
|
||||
@@ -971,7 +971,7 @@ const EXCLUDED_SUBBLOCK_TYPES = new Set([
|
||||
])
|
||||
|
||||
export interface SubBlocksForToolInput {
|
||||
toolConfig: ToolConfig
|
||||
toolConfig: ToolMetadata
|
||||
subBlocks: BlockSubBlockConfig[]
|
||||
oauthConfig?: OAuthConfig
|
||||
}
|
||||
@@ -992,7 +992,7 @@ export function getSubBlocksForToolInput(
|
||||
blockConfigOverride?: Pick<ToolInputBlockConfig, 'subBlocks'>
|
||||
): SubBlocksForToolInput | null {
|
||||
try {
|
||||
const toolConfig = getTool(toolId)
|
||||
const toolConfig = getToolMetadata(toolId)
|
||||
if (!toolConfig) {
|
||||
logger.warn(`Tool not found: ${toolId}`)
|
||||
return null
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
import { stripVersionSuffix } from '@sim/utils/string'
|
||||
import rawToolIds from '@/tools/generated/tool-ids'
|
||||
|
||||
/**
|
||||
* Tool id resolution, without importing the executable registry.
|
||||
*
|
||||
* Resolving a tool name and checking whether one exists need only the registry's
|
||||
* key set — never a `ToolConfig` — so this module carries just the ids (~100 KB
|
||||
* versus ~4 MB for params or outputs). `@/tools/metadata` and
|
||||
* `@/tools/metadata-outputs` both resolve through here, which is what keeps them
|
||||
* independent of each other.
|
||||
*
|
||||
* Semantics mirror `resolveToolId`/`getTool` in `@/tools/utils` exactly,
|
||||
* including returning the input unchanged when nothing matches. See
|
||||
* `.agents/skills/tool-registry-boundary/SKILL.md`.
|
||||
*/
|
||||
/**
|
||||
* Frozen because {@link getToolIds} hands it out directly. Returning a copy
|
||||
* would allocate on every call in the loops that consume it; freezing makes an
|
||||
* in-place `sort()`/`push()` by a caller throw rather than silently corrupt
|
||||
* every later lookup.
|
||||
*/
|
||||
const toolIds: readonly string[] = Object.freeze(rawToolIds)
|
||||
|
||||
const toolIdSet = new Set(toolIds)
|
||||
|
||||
/**
|
||||
* Base name -> newest versioned id, built once.
|
||||
*
|
||||
* `@/tools/utils` rebuilds this on every unresolved lookup; the id set is static
|
||||
* at runtime, so caching it is behaviour-identical and drops the repeated O(n)
|
||||
* scan.
|
||||
*/
|
||||
let latestByBaseName: Map<string, string> | null = null
|
||||
|
||||
function getLatestByBaseName(): Map<string, string> {
|
||||
if (latestByBaseName) return latestByBaseName
|
||||
|
||||
const versions = new Map<string, { toolId: string; version: number }>()
|
||||
for (const toolId of toolIds) {
|
||||
const baseName = stripVersionSuffix(toolId)
|
||||
const versionMatch = toolId.match(/_v(\d+)$/)
|
||||
const version = versionMatch ? Number.parseInt(versionMatch[1], 10) : 1
|
||||
const previous = versions.get(baseName)
|
||||
if (!previous || version > previous.version) {
|
||||
versions.set(baseName, { toolId, version })
|
||||
}
|
||||
}
|
||||
|
||||
latestByBaseName = new Map()
|
||||
for (const [baseName, { toolId }] of versions) {
|
||||
latestByBaseName.set(baseName, toolId)
|
||||
}
|
||||
return latestByBaseName
|
||||
}
|
||||
|
||||
/** Every registered tool id, including versioned variants. Frozen — copy before sorting. */
|
||||
export function getToolIds(): readonly string[] {
|
||||
return toolIds
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves a tool name to its registered id, mapping an unversioned name onto
|
||||
* the newest version (`notion_search` -> `notion_search_v2`). Returns the input
|
||||
* unchanged when nothing matches, matching `@/tools/utils`.
|
||||
*/
|
||||
export function resolveToolId(toolName: string): string {
|
||||
if (toolIdSet.has(toolName)) return toolName
|
||||
return getLatestByBaseName().get(toolName) ?? toolName
|
||||
}
|
||||
|
||||
/** Whether `toolId` names a built-in tool, resolving unversioned names. */
|
||||
export function hasToolId(toolId: string): boolean {
|
||||
return toolIdSet.has(resolveToolId(toolId))
|
||||
}
|
||||
@@ -58,6 +58,13 @@ export function getLatestVersionTools(
|
||||
/**
|
||||
* Resolves a tool name to its actual tool ID in the registry.
|
||||
* Handles both stripped names (e.g., 'notion_search') and versioned names (e.g., 'notion_search_v2').
|
||||
*
|
||||
* Server-side counterpart to `resolveToolId` in `@/tools/tool-ids`. Both exist
|
||||
* deliberately: this one reads the live registry, so a tool added but not yet
|
||||
* regenerated stays resolvable; that one resolves against the generated id list
|
||||
* without pulling 4,300 tools into a client graph. Client code wants that one.
|
||||
* `tool-metadata:check` asserts the two never diverge.
|
||||
*
|
||||
* @param toolName The tool name to resolve (may or may not have version suffix)
|
||||
* @returns The actual tool ID in the registry, or the original name if not found
|
||||
*/
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
import type { ToolResponse } from '@/tools/types'
|
||||
|
||||
/** Params shared by every Zoho tool: the OAuth token and the account's region. */
|
||||
export interface ZohoBaseParams {
|
||||
accessToken: string
|
||||
dataCenter?: string
|
||||
}
|
||||
|
||||
/** Params shared by every Zoho Desk tool — Desk requires an `orgId` header. */
|
||||
export interface ZohoDeskBaseParams extends ZohoBaseParams {
|
||||
orgId: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Pagination block returned by CRM list/search endpoints.
|
||||
* @see https://www.zoho.com/crm/developer/docs/api/v8/get-records.html
|
||||
*/
|
||||
export interface ZohoCrmPageInfo {
|
||||
page: number | null
|
||||
perPage: number | null
|
||||
count: number | null
|
||||
moreRecords: boolean
|
||||
}
|
||||
|
||||
export const ZOHO_CRM_PAGE_INFO_OUTPUT = {
|
||||
type: 'object' as const,
|
||||
description: 'Pagination metadata returned by Zoho CRM',
|
||||
properties: {
|
||||
page: { type: 'number' as const, description: 'Current page number', optional: true },
|
||||
perPage: { type: 'number' as const, description: 'Records requested per page', optional: true },
|
||||
count: {
|
||||
type: 'number' as const,
|
||||
description: 'Number of records in this page',
|
||||
optional: true,
|
||||
},
|
||||
moreRecords: { type: 'boolean' as const, description: 'Whether further pages are available' },
|
||||
},
|
||||
}
|
||||
|
||||
/** A CRM write result entry, as returned in the `data` array of insert/update/upsert. */
|
||||
export interface ZohoCrmWriteResult {
|
||||
id: string | null
|
||||
code: string | null
|
||||
status: string | null
|
||||
message: string | null
|
||||
}
|
||||
|
||||
export const ZOHO_CRM_WRITE_RESULT_OUTPUT = {
|
||||
type: 'array' as const,
|
||||
description: 'Per-record results returned by Zoho CRM',
|
||||
properties: {
|
||||
id: { type: 'string' as const, description: 'Record ID', optional: true },
|
||||
code: {
|
||||
type: 'string' as const,
|
||||
description: 'Zoho result code (e.g. SUCCESS)',
|
||||
optional: true,
|
||||
},
|
||||
status: { type: 'string' as const, description: 'Result status', optional: true },
|
||||
message: {
|
||||
type: 'string' as const,
|
||||
description: 'Human-readable result message',
|
||||
optional: true,
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetRecordsParams extends ZohoBaseParams {
|
||||
module: string
|
||||
recordId?: string
|
||||
fields?: string
|
||||
page?: string | number
|
||||
perPage?: string | number
|
||||
sortBy?: string
|
||||
sortOrder?: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetRecordsResponse extends ToolResponse {
|
||||
output: {
|
||||
records: Record<string, unknown>[]
|
||||
record?: Record<string, unknown> | null
|
||||
info: ZohoCrmPageInfo
|
||||
}
|
||||
}
|
||||
|
||||
export interface ZohoCrmCreateRecordsParams extends ZohoBaseParams {
|
||||
module: string
|
||||
records: unknown
|
||||
trigger?: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmWriteResponse extends ToolResponse {
|
||||
output: {
|
||||
results: ZohoCrmWriteResult[]
|
||||
}
|
||||
}
|
||||
|
||||
export interface ZohoCrmUpdateRecordParams extends ZohoBaseParams {
|
||||
module: string
|
||||
recordId: string
|
||||
record: unknown
|
||||
trigger?: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmUpsertRecordsParams extends ZohoBaseParams {
|
||||
module: string
|
||||
records: unknown
|
||||
duplicateCheckFields?: string
|
||||
trigger?: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmDeleteRecordParams extends ZohoBaseParams {
|
||||
module: string
|
||||
recordId: string
|
||||
wfTrigger?: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmSearchRecordsParams extends ZohoBaseParams {
|
||||
module: string
|
||||
criteria?: string
|
||||
email?: string
|
||||
phone?: string
|
||||
word?: string
|
||||
fields?: string
|
||||
page?: string | number
|
||||
perPage?: string | number
|
||||
}
|
||||
|
||||
export interface ZohoCrmCoqlQueryParams extends ZohoBaseParams {
|
||||
selectQuery: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmCoqlQueryResponse extends ToolResponse {
|
||||
output: {
|
||||
records: Record<string, unknown>[]
|
||||
info: ZohoCrmPageInfo
|
||||
}
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetModulesParams extends ZohoBaseParams {}
|
||||
|
||||
export interface ZohoCrmGetModulesResponse extends ToolResponse {
|
||||
output: {
|
||||
modules: Array<{
|
||||
apiName: string | null
|
||||
moduleName: string | null
|
||||
id: string | null
|
||||
pluralLabel: string | null
|
||||
singularLabel: string | null
|
||||
creatable: boolean | null
|
||||
editable: boolean | null
|
||||
deletable: boolean | null
|
||||
}>
|
||||
}
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetFieldsParams extends ZohoBaseParams {
|
||||
module: string
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetFieldsResponse extends ToolResponse {
|
||||
output: {
|
||||
fields: Array<{
|
||||
apiName: string | null
|
||||
displayLabel: string | null
|
||||
dataType: string | null
|
||||
id: string | null
|
||||
required: boolean | null
|
||||
readOnly: boolean | null
|
||||
length: number | null
|
||||
}>
|
||||
}
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetUsersParams extends ZohoBaseParams {
|
||||
type?: string
|
||||
page?: string | number
|
||||
perPage?: string | number
|
||||
}
|
||||
|
||||
export interface ZohoCrmGetUsersResponse extends ToolResponse {
|
||||
output: {
|
||||
users: Record<string, unknown>[]
|
||||
info: ZohoCrmPageInfo
|
||||
}
|
||||
}
|
||||
|
||||
export interface ZohoCrmAddNoteParams extends ZohoBaseParams {
|
||||
parentId: string
|
||||
seModule: string
|
||||
noteTitle?: string
|
||||
noteContent: string
|
||||
}
|
||||
|
||||
export type ZohoCrmResponse =
|
||||
| ZohoCrmGetRecordsResponse
|
||||
| ZohoCrmWriteResponse
|
||||
| ZohoCrmCoqlQueryResponse
|
||||
| ZohoCrmGetModulesResponse
|
||||
| ZohoCrmGetFieldsResponse
|
||||
| ZohoCrmGetUsersResponse
|
||||
@@ -0,0 +1,192 @@
|
||||
import { truncate } from '@sim/utils/string'
|
||||
|
||||
/**
|
||||
* Zoho data centers Sim supports.
|
||||
*
|
||||
* Zoho is multi-DC: a user's account lives in exactly one region, and every API
|
||||
* host is region-scoped. Zoho's guidance is to never hardcode a single region —
|
||||
* the token response carries an `api_domain` naming the caller's CRM host.
|
||||
* Sim cannot persist that value (the shared Better Auth `account` table has no
|
||||
* column for provider-specific token fields), so the region is selected on the
|
||||
* block instead and resolved to hosts here.
|
||||
*
|
||||
* Only regions whose CRM *and* Desk hosts are confirmed in Zoho's own
|
||||
* documentation are listed. CA and SA are deliberately absent: Zoho's accounts
|
||||
* docs give `accounts.zohocloud.ca` while the Desk SDK uses `accounts.zoho.ca`,
|
||||
* and neither source states their CRM API host.
|
||||
*
|
||||
* @see https://www.zoho.com/crm/developer/docs/api/v8/multi-dc.html
|
||||
*/
|
||||
export const ZOHO_DATA_CENTERS = {
|
||||
us: { crm: 'https://www.zohoapis.com', desk: 'https://desk.zoho.com' },
|
||||
eu: { crm: 'https://www.zohoapis.eu', desk: 'https://desk.zoho.eu' },
|
||||
in: { crm: 'https://www.zohoapis.in', desk: 'https://desk.zoho.in' },
|
||||
au: { crm: 'https://www.zohoapis.com.au', desk: 'https://desk.zoho.com.au' },
|
||||
jp: { crm: 'https://www.zohoapis.jp', desk: 'https://desk.zoho.jp' },
|
||||
} as const
|
||||
|
||||
export type ZohoDataCenter = keyof typeof ZOHO_DATA_CENTERS
|
||||
|
||||
const DEFAULT_DATA_CENTER: ZohoDataCenter = 'us'
|
||||
|
||||
/** Zoho CRM API version pinned across every CRM tool. */
|
||||
export const ZOHO_CRM_API_VERSION = 'v8'
|
||||
|
||||
function resolveDataCenter(dataCenter?: string): ZohoDataCenter {
|
||||
const normalized = dataCenter?.trim().toLowerCase()
|
||||
if (normalized && normalized in ZOHO_DATA_CENTERS) {
|
||||
return normalized as ZohoDataCenter
|
||||
}
|
||||
return DEFAULT_DATA_CENTER
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the CRM API base URL (including version segment) for a data center.
|
||||
* Falls back to the US region when unset or unrecognized.
|
||||
*/
|
||||
export function getCrmBaseUrl(dataCenter?: string): string {
|
||||
return `${ZOHO_DATA_CENTERS[resolveDataCenter(dataCenter)].crm}/crm/${ZOHO_CRM_API_VERSION}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the Desk API base URL (including the `/api/v1` segment) for a data
|
||||
* center. Falls back to the US region when unset or unrecognized.
|
||||
*/
|
||||
export function getDeskBaseUrl(dataCenter?: string): string {
|
||||
return `${ZOHO_DATA_CENTERS[resolveDataCenter(dataCenter)].desk}/api/v1`
|
||||
}
|
||||
|
||||
/**
|
||||
* Zoho authenticates with a bespoke scheme rather than `Bearer`.
|
||||
* @see https://www.zoho.com/crm/developer/docs/api/v8/access-refresh.html
|
||||
*/
|
||||
export function buildZohoHeaders(accessToken: string, orgId?: string): Record<string, string> {
|
||||
const headers: Record<string, string> = {
|
||||
Authorization: `Zoho-oauthtoken ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
}
|
||||
if (orgId?.trim()) {
|
||||
headers.orgId = orgId.trim()
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
/**
|
||||
* Trims a required identifier and throws when it is missing or whitespace-only,
|
||||
* so a blank value can never collapse into an empty URL path segment and send a
|
||||
* malformed request to Zoho.
|
||||
*/
|
||||
export function requireZohoId(value: string | undefined, label: string): string {
|
||||
const trimmed = value?.trim()
|
||||
if (!trimmed) {
|
||||
throw new Error(`${label} is required.`)
|
||||
}
|
||||
return trimmed
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a JSON object supplied either as an object (from the LLM) or as a JSON
|
||||
* string (from a block input), and rejects anything that is not a plain object.
|
||||
*/
|
||||
export function parseJsonObject(
|
||||
value: unknown,
|
||||
label: string
|
||||
): Record<string, unknown> | undefined {
|
||||
if (value === undefined || value === null || value === '') return undefined
|
||||
let parsed: unknown = value
|
||||
if (typeof value === 'string') {
|
||||
try {
|
||||
parsed = JSON.parse(value)
|
||||
} catch {
|
||||
throw new Error(`${label} must be valid JSON.`)
|
||||
}
|
||||
}
|
||||
if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) {
|
||||
throw new Error(`${label} must be a JSON object.`)
|
||||
}
|
||||
return parsed as Record<string, unknown>
|
||||
}
|
||||
|
||||
/**
|
||||
* Coerces a numeric-ish param into a positive integer, clamped to `max`.
|
||||
* Returns undefined when unset or unparseable so callers can omit the param.
|
||||
*/
|
||||
export function toPositiveInt(value: unknown, max: number): number | undefined {
|
||||
if (value === undefined || value === null || value === '') return undefined
|
||||
const parsed = Number(value)
|
||||
if (!Number.isFinite(parsed) || parsed < 1) return undefined
|
||||
return Math.min(Math.floor(parsed), max)
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends only the defined entries of `params` to a URL's query string, so unset
|
||||
* optional params never surface as empty values Zoho would reject.
|
||||
*/
|
||||
export function buildQuery(params: Record<string, string | number | undefined>): string {
|
||||
const search = new URLSearchParams()
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
if (value === undefined || value === '') continue
|
||||
search.set(key, String(value))
|
||||
}
|
||||
const query = search.toString()
|
||||
return query ? `?${query}` : ''
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a Zoho response body as JSON, tolerating the empty bodies Zoho returns
|
||||
* for "no matching records" — CRM answers `204 No Content` on an empty list or
|
||||
* search, and Desk does the same, so a bare `response.json()` would throw on a
|
||||
* perfectly successful call.
|
||||
*/
|
||||
export async function readZohoJson(response: Response): Promise<any> {
|
||||
if (response.status === 204) return {}
|
||||
const text = await response.text()
|
||||
if (!text.trim()) return {}
|
||||
try {
|
||||
return JSON.parse(text)
|
||||
} catch {
|
||||
return { message: truncate(text, 500) }
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts a descriptive message from a Zoho error payload.
|
||||
*
|
||||
* CRM returns `{ code, message, status, details }`; Desk returns
|
||||
* `{ errorCode, message }`. Both are handled, with an HTTP-status fallback.
|
||||
*/
|
||||
export function extractZohoErrorMessage(
|
||||
data: unknown,
|
||||
status: number,
|
||||
defaultMessage: string
|
||||
): string {
|
||||
const payload = data as
|
||||
| { message?: unknown; code?: unknown; errorCode?: unknown; data?: unknown }
|
||||
| undefined
|
||||
|
||||
const nested = Array.isArray(payload?.data) ? payload.data[0] : undefined
|
||||
const record = (nested ?? payload) as
|
||||
| { message?: unknown; code?: unknown; errorCode?: unknown }
|
||||
| undefined
|
||||
|
||||
if (record && typeof record.message === 'string' && record.message.trim()) {
|
||||
const code = record.code ?? record.errorCode
|
||||
const suffix = typeof code === 'string' && code.trim() ? ` [${code}]` : ''
|
||||
return `Zoho API Error (${status}): ${record.message}${suffix}`
|
||||
}
|
||||
|
||||
switch (status) {
|
||||
case 400:
|
||||
return `Zoho API Error (400): Bad Request — the request was malformed or missing required parameters.`
|
||||
case 401:
|
||||
return `Zoho API Error (401): Unauthorized — the access token is invalid or expired. Please reconnect your Zoho account.`
|
||||
case 403:
|
||||
return `Zoho API Error (403): Forbidden — your Zoho account lacks permission, or the required OAuth scope was not granted.`
|
||||
case 404:
|
||||
return `Zoho API Error (404): Not Found — the requested record does not exist or is not visible to you.`
|
||||
case 429:
|
||||
return `Zoho API Error (429): Rate limit exceeded — too many API calls. Please retry later.`
|
||||
default:
|
||||
return `${defaultMessage} (HTTP ${status})`
|
||||
}
|
||||
}
|
||||
@@ -338,7 +338,25 @@ export const blocksMock = {
|
||||
|
||||
/**
|
||||
* Pre-configured tools/utils mock for use with vi.mock('@/tools/utils', () => toolsUtilsMock).
|
||||
*
|
||||
* Only mocks the *executable* lookup. Code that reads a tool's shape now goes
|
||||
* through `@/tools/metadata`, so mocking this alone leaves such code reading the
|
||||
* real generated artifacts — see {@link toolsMetadataMock}.
|
||||
*/
|
||||
export const toolsUtilsMock = {
|
||||
getTool: createMockGetTool(),
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-configured metadata mock for use with
|
||||
* `vi.mock('@/tools/metadata', () => toolsMetadataMock)`.
|
||||
*
|
||||
* Backed by the same `mockToolConfigs` as {@link toolsUtilsMock}, so a test that
|
||||
* mocks both sees one consistent tool universe. Mock this wherever the code under
|
||||
* test reads `params`, `outputs` or `name` — mocking `@/tools/utils` there is a
|
||||
* no-op that only appears to work because the real artifacts happen to agree.
|
||||
*/
|
||||
export const toolsMetadataMock = {
|
||||
getToolMetadata: createMockGetTool(),
|
||||
getToolParams: (toolId: string) => createMockGetTool()(toolId)?.params,
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ export {
|
||||
createMockGetTool,
|
||||
mockBlockConfigs,
|
||||
mockToolConfigs,
|
||||
toolsMetadataMock,
|
||||
toolsUtilsMock,
|
||||
} from './blocks.mock'
|
||||
// Copilot HTTP mocks (for @/lib/copilot/request/http)
|
||||
|
||||
@@ -17,9 +17,14 @@
|
||||
* a single consumer — keeping it separate means callers that only need `params`
|
||||
* or an id check don't pay for it:
|
||||
*
|
||||
* tools/generated/tool-ids.ts every registered tool id
|
||||
* tools/generated/tool-metadata.ts id -> { name, description, version, params, oauth }
|
||||
* tools/generated/tool-outputs.ts id -> outputs
|
||||
*
|
||||
* Ids are their own artifact because resolving a possibly-unversioned tool name
|
||||
* needs only the key set, and an existence check needs nothing more — so those
|
||||
* callers load ~100 KB instead of ~4 MB.
|
||||
*
|
||||
* Each artifact holds its data as one JSON string parsed at runtime — see
|
||||
* `serialize()` for why an imported `.json` or an object literal is not viable
|
||||
* at this size.
|
||||
@@ -32,11 +37,14 @@ import { mkdir, readFile, writeFile } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { tools } from '../apps/sim/tools/registry'
|
||||
import { hasToolId } from '../apps/sim/tools/tool-ids'
|
||||
import type { ToolConfig } from '../apps/sim/tools/types'
|
||||
import { getTool } from '../apps/sim/tools/utils'
|
||||
|
||||
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url))
|
||||
const ROOT = resolve(SCRIPT_DIR, '..')
|
||||
const GENERATED_DIR = resolve(ROOT, 'apps/sim/tools/generated')
|
||||
const IDS_PATH = resolve(GENERATED_DIR, 'tool-ids.ts')
|
||||
const METADATA_PATH = resolve(GENERATED_DIR, 'tool-metadata.ts')
|
||||
const OUTPUTS_PATH = resolve(GENERATED_DIR, 'tool-outputs.ts')
|
||||
|
||||
@@ -141,6 +149,12 @@ function build(registry: ToolRecord) {
|
||||
}
|
||||
|
||||
return {
|
||||
ids: serializeValue(
|
||||
Object.keys(metadata),
|
||||
'toolIds',
|
||||
'/** Every registered tool id, including versioned variants. */',
|
||||
'string[]'
|
||||
),
|
||||
metadata: serialize(
|
||||
metadata,
|
||||
'toolMetadata',
|
||||
@@ -192,12 +206,16 @@ function toJsStringLiteral(json: string): string {
|
||||
* generated artifact nothing reads by eye and CI verifies wholesale.
|
||||
*/
|
||||
function serialize(entries: Record<string, unknown>, exportName: string, doc: string): string {
|
||||
const literal = toJsStringLiteral(JSON.stringify(entries))
|
||||
return serializeValue(entries, exportName, doc, 'Record<string, unknown>')
|
||||
}
|
||||
|
||||
function serializeValue(value: unknown, exportName: string, doc: string, type: string): string {
|
||||
const literal = toJsStringLiteral(JSON.stringify(value))
|
||||
return `// Generated by scripts/sync-tool-metadata.ts — do not edit.
|
||||
// Regenerate with: bun run tool-metadata:generate
|
||||
|
||||
${doc}
|
||||
const ${exportName}: Record<string, unknown> = JSON.parse(
|
||||
const ${exportName}: ${type} = JSON.parse(
|
||||
${literal}
|
||||
)
|
||||
|
||||
@@ -205,24 +223,57 @@ export default ${exportName}
|
||||
`
|
||||
}
|
||||
|
||||
/**
|
||||
* Asserts the two tool-id resolvers agree.
|
||||
*
|
||||
* `@/tools/utils` resolves against the live registry; `@/tools/tool-ids` against
|
||||
* the generated id list. Both exist deliberately — the registry-backed one keeps
|
||||
* a newly-added tool resolvable before regeneration, the list-backed one lets
|
||||
* client code resolve without importing 4,300 tools. Nothing structurally keeps
|
||||
* the two in step, so it is checked here rather than left to trust.
|
||||
*
|
||||
* Runs only after the staleness check passes, since a stale id list would
|
||||
* otherwise report a divergence that is really just a missing regeneration. It
|
||||
* cannot live in a vitest suite: `vitest.setup.ts` globally mocks
|
||||
* `@/tools/registry` to an empty map, so `getTool` resolves nothing there.
|
||||
*/
|
||||
function assertResolverParity() {
|
||||
const ids = Object.keys(tools)
|
||||
const probes = new Set([...ids, ...ids.map((id) => id.replace(/_v\d+$/, '')), '__not_a_tool__'])
|
||||
const divergent: string[] = []
|
||||
for (const probe of probes) {
|
||||
if (Boolean(getTool(probe)) !== hasToolId(probe)) divergent.push(probe)
|
||||
}
|
||||
if (divergent.length > 0) {
|
||||
throw new Error(
|
||||
`Tool id resolvers disagree on ${divergent.length} of ${probes.size} names ` +
|
||||
`(e.g. ${divergent.slice(0, 5).join(', ')}).\n` +
|
||||
'resolveToolId in apps/sim/tools/utils.ts and apps/sim/tools/tool-ids.ts have drifted.'
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const checkOnly = process.argv.includes('--check')
|
||||
const { metadata, outputs, toolCount } = build(tools as ToolRecord)
|
||||
const { ids, metadata, outputs, toolCount } = build(tools as ToolRecord)
|
||||
|
||||
if (checkOnly) {
|
||||
const [existingMetadata, existingOutputs] = await Promise.all([
|
||||
const [existingIds, existingMetadata, existingOutputs] = await Promise.all([
|
||||
readFile(IDS_PATH, 'utf8').catch(() => null),
|
||||
readFile(METADATA_PATH, 'utf8').catch(() => null),
|
||||
readFile(OUTPUTS_PATH, 'utf8').catch(() => null),
|
||||
])
|
||||
if (existingMetadata !== metadata || existingOutputs !== outputs) {
|
||||
if (existingIds !== ids || existingMetadata !== metadata || existingOutputs !== outputs) {
|
||||
throw new Error('Generated tool metadata is stale. Run: bun run tool-metadata:generate')
|
||||
}
|
||||
console.log(`✓ tool metadata in sync (${toolCount} tools)`)
|
||||
assertResolverParity()
|
||||
console.log(`✓ tool metadata in sync (${toolCount} tools), resolvers agree`)
|
||||
return
|
||||
}
|
||||
|
||||
await mkdir(GENERATED_DIR, { recursive: true })
|
||||
await Promise.all([
|
||||
writeFile(IDS_PATH, ids, 'utf8'),
|
||||
writeFile(METADATA_PATH, metadata, 'utf8'),
|
||||
writeFile(OUTPUTS_PATH, outputs, 'utf8'),
|
||||
])
|
||||
|
||||
Reference in New Issue
Block a user