mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-24 15:45:35 +08:00
chore(skills): reinforce skill to not guess integration outputs (#4122)
This commit is contained in:
@@ -14,6 +14,20 @@ When the user asks you to create a block:
|
||||
2. Configure all subBlocks with proper types, conditions, and dependencies
|
||||
3. Wire up tools correctly
|
||||
|
||||
## Hard Rule: No Guessed Tool Outputs
|
||||
|
||||
Blocks depend on tool outputs. If the underlying tool response schema is not documented or live-verified, you MUST tell the user instead of guessing block outputs.
|
||||
|
||||
- Do NOT invent block outputs for undocumented tool responses
|
||||
- Do NOT describe unknown JSON shapes as if they were confirmed
|
||||
- Do NOT wire fields into the block just because they seem likely to exist
|
||||
|
||||
If the tool outputs are not known, do one of these instead:
|
||||
1. Ask the user for sample tool responses
|
||||
2. Ask the user for test credentials so the tool responses can be verified
|
||||
3. Limit the block to operations whose outputs are documented
|
||||
4. Leave uncertain outputs out and explicitly tell the user what remains unknown
|
||||
|
||||
## Block Configuration Structure
|
||||
|
||||
```typescript
|
||||
@@ -575,6 +589,8 @@ Use `type: 'json'` with a descriptive string when:
|
||||
- It represents a list/array of items
|
||||
- The shape varies by operation
|
||||
|
||||
If the output shape is unknown because the underlying tool response is undocumented, you MUST tell the user and stop. Unknown is not the same as variable. Never guess block outputs.
|
||||
|
||||
## V2 Block Pattern
|
||||
|
||||
When creating V2 blocks (alongside legacy V1):
|
||||
@@ -829,3 +845,4 @@ After creating the block, you MUST validate it against every tool it references:
|
||||
- Type coercions in `tools.config.params` for any params that need conversion (Number(), Boolean(), JSON.parse())
|
||||
3. **Verify block outputs** cover the key fields returned by all tools
|
||||
4. **Verify conditions** — each subBlock should only show for the operations that actually use it
|
||||
5. **If any tool outputs are still unknown**, explicitly tell the user instead of guessing block outputs
|
||||
|
||||
@@ -15,6 +15,21 @@ When the user asks you to create a connector:
|
||||
3. Create the connector directory and config
|
||||
4. Register it in the connector registry
|
||||
|
||||
## Hard Rule: No Guessed Response Or Document Schemas
|
||||
|
||||
If the service docs do not clearly show the document list response, document fetch response, pagination shape, or metadata fields, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT invent document fields
|
||||
- Do NOT guess pagination cursors or next-page fields
|
||||
- Do NOT infer metadata/tag mappings from unrelated endpoints
|
||||
- Do NOT fabricate `ExternalDocument` content structure from partial docs
|
||||
|
||||
If the source schema is unknown, do one of these instead:
|
||||
1. Ask the user for sample API responses
|
||||
2. Ask the user for test credentials so you can verify live payloads
|
||||
3. Implement only the documented parts of the connector
|
||||
4. Leave the connector incomplete and explicitly say which fields remain unknown
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Create files in `apps/sim/connectors/{service}/`:
|
||||
@@ -92,6 +107,8 @@ export const {service}Connector: ConnectorConfig = {
|
||||
}
|
||||
```
|
||||
|
||||
Only map fields in `listDocuments`, `getDocument`, `validateConfig`, and `mapTags` when the source payload shape is documented or live-verified. If not, tell the user and stop rather than guessing.
|
||||
|
||||
### API key connector example
|
||||
|
||||
```typescript
|
||||
|
||||
@@ -29,6 +29,21 @@ Before writing any code:
|
||||
- Required vs optional parameters
|
||||
- Response structures
|
||||
|
||||
### Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the official docs do not clearly show the response JSON shape for an endpoint, you MUST stop and tell the user exactly which outputs are unknown.
|
||||
|
||||
- Do NOT guess response field names
|
||||
- Do NOT infer nested JSON paths from related endpoints
|
||||
- Do NOT invent output properties just because they seem likely
|
||||
- Do NOT implement `transformResponse` against unverified payload shapes
|
||||
|
||||
If response schemas are missing or incomplete, do one of the following before proceeding:
|
||||
1. Ask the user for sample responses
|
||||
2. Ask the user for test credentials so you can verify the live payload
|
||||
3. Reduce the scope to only endpoints whose response shapes are documented
|
||||
4. Leave the tool unimplemented and explicitly report why
|
||||
|
||||
## Step 2: Create Tools
|
||||
|
||||
### Directory Structure
|
||||
@@ -103,6 +118,7 @@ export const {service}{Action}Tool: ToolConfig<Params, Response> = {
|
||||
- Set `optional: true` for outputs that may not exist
|
||||
- Never output raw JSON dumps - extract meaningful fields
|
||||
- When using `type: 'json'` and you know the object shape, define `properties` with the inner fields so downstream consumers know the structure. Only use bare `type: 'json'` when the shape is truly dynamic
|
||||
- If you do not know the response JSON shape from docs or verified examples, you MUST tell the user and stop. Never guess outputs or response mappings.
|
||||
|
||||
## Step 3: Create Block
|
||||
|
||||
@@ -450,6 +466,8 @@ If creating V2 versions (API-aligned outputs):
|
||||
- [ ] Verified block subBlocks cover all required tool params with correct conditions
|
||||
- [ ] Verified block outputs match what the tools actually return
|
||||
- [ ] Verified `tools.config.params` correctly maps and coerces all param types
|
||||
- [ ] Verified every tool output and `transformResponse` path against documented or live-verified JSON responses
|
||||
- [ ] If any response schema remained unknown, explicitly told the user instead of guessing
|
||||
|
||||
## Example Command
|
||||
|
||||
|
||||
@@ -14,6 +14,21 @@ When the user asks you to create tools for a service:
|
||||
2. Create the tools directory structure
|
||||
3. Generate properly typed tool configurations
|
||||
|
||||
## Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the docs do not clearly show the response JSON for a tool, you MUST tell the user exactly which outputs are unknown and stop short of guessing.
|
||||
|
||||
- Do NOT invent response field names
|
||||
- Do NOT infer nested paths from nearby endpoints
|
||||
- Do NOT guess array item shapes
|
||||
- Do NOT write `transformResponse` against unverified payloads
|
||||
|
||||
If the response shape is unknown, do one of these instead:
|
||||
1. Ask the user for sample responses
|
||||
2. Ask the user for test credentials so you can verify live responses
|
||||
3. Implement only the endpoints whose outputs are documented
|
||||
4. Leave the tool unimplemented and explicitly say why
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Create files in `apps/sim/tools/{service}/`:
|
||||
@@ -187,6 +202,8 @@ items: {
|
||||
|
||||
Only use bare `type: 'json'` without `properties` when the shape is truly dynamic or unknown.
|
||||
|
||||
If the response shape is unknown because the docs do not provide it, you MUST tell the user and stop. Unknown is not the same as dynamic. Never guess outputs.
|
||||
|
||||
## Critical Rules for transformResponse
|
||||
|
||||
### Handle Nullable Fields
|
||||
@@ -441,7 +458,9 @@ After creating all tools, you MUST validate every tool before finishing:
|
||||
- All output fields match what the API actually returns
|
||||
- No fields are missing from outputs that the API provides
|
||||
- No extra fields are defined in outputs that the API doesn't return
|
||||
- Every output field and JSON path is backed by docs or live-verified sample responses
|
||||
3. **Verify consistency** across tools:
|
||||
- Shared types in `types.ts` match all tools that use them
|
||||
- Tool IDs in the barrel export match the tool file definitions
|
||||
- Error handling is consistent (error checks, meaningful messages)
|
||||
4. **If any response schema is still unknown**, explicitly tell the user instead of guessing
|
||||
|
||||
@@ -14,6 +14,21 @@ You are an expert at creating webhook triggers for Sim. You understand the trigg
|
||||
3. Create a provider handler if custom auth, formatting, or subscriptions are needed
|
||||
4. Register triggers and connect them to the block
|
||||
|
||||
## Hard Rule: No Guessed Webhook Payload Schemas
|
||||
|
||||
If the service docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing trigger outputs or `formatInput` mappings.
|
||||
|
||||
- Do NOT invent payload field names
|
||||
- Do NOT guess nested event object paths
|
||||
- Do NOT infer output fields from the UI or marketing docs
|
||||
- Do NOT write `formatInput` against unverified webhook bodies
|
||||
|
||||
If the payload shape is unknown, do one of these instead:
|
||||
1. Ask the user for sample webhook payloads
|
||||
2. Ask the user for a test webhook source so you can inspect a real event
|
||||
3. Implement only the event registration/setup portions whose payloads are documented
|
||||
4. Leave the trigger unimplemented and explicitly say which payload fields are unknown
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
|
||||
@@ -52,6 +52,20 @@ Fetch the official API docs for the service. This is the **source of truth** for
|
||||
|
||||
Use Context7 (resolve-library-id → query-docs) or WebFetch to retrieve documentation. If both fail, note which claims are based on training knowledge vs verified docs.
|
||||
|
||||
### Hard Rule: No Guessed Source Schemas
|
||||
|
||||
If the service docs do not clearly show document list responses, document fetch responses, metadata fields, or pagination shapes, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT infer document fields from unrelated endpoints
|
||||
- Do NOT guess pagination cursors or response wrappers
|
||||
- Do NOT assume metadata keys that are not documented
|
||||
- Do NOT treat probable shapes as validated
|
||||
|
||||
If a schema is unknown, validation must explicitly recommend:
|
||||
1. sample API responses,
|
||||
2. live test credentials, or
|
||||
3. trimming the connector to only documented fields.
|
||||
|
||||
## Step 3: Validate API Endpoints
|
||||
|
||||
For **every** API call in the connector (`listDocuments`, `getDocument`, `validateConfig`, and any helper functions), verify against the API docs:
|
||||
@@ -93,6 +107,7 @@ For **every** API call in the connector (`listDocuments`, `getDocument`, `valida
|
||||
- [ ] Field names extracted match what the API actually returns
|
||||
- [ ] Nullable fields are handled with `?? null` or `|| undefined`
|
||||
- [ ] Error responses are checked before accessing data fields
|
||||
- [ ] Every extracted field and pagination value is backed by official docs or live-verified sample payloads
|
||||
|
||||
## Step 4: Validate OAuth Scopes (if OAuth connector)
|
||||
|
||||
@@ -304,6 +319,7 @@ After fixing, confirm:
|
||||
1. `bun run lint` passes
|
||||
2. TypeScript compiles clean
|
||||
3. Re-read all modified files to verify fixes are correct
|
||||
4. Any remaining unknown source schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
|
||||
@@ -41,6 +41,20 @@ Fetch the official API docs for the service. This is the **source of truth** for
|
||||
- Pagination patterns (which param name, which response field)
|
||||
- Rate limits and error formats
|
||||
|
||||
### Hard Rule: No Guessed Response Schemas
|
||||
|
||||
If the official docs do not clearly show the response JSON shape for an endpoint, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT assume field names from nearby endpoints
|
||||
- Do NOT infer nested JSON paths without evidence
|
||||
- Do NOT treat "likely" fields as confirmed outputs
|
||||
- Do NOT accept implementation guesses as valid just because they are defensive
|
||||
|
||||
If a response schema is unknown, the validation must explicitly call that out and require:
|
||||
1. sample responses from the user,
|
||||
2. live test credentials for verification, or
|
||||
3. trimming the tool/block down to only documented fields.
|
||||
|
||||
## Step 3: Validate Tools
|
||||
|
||||
For **every** tool file, check:
|
||||
@@ -81,6 +95,7 @@ For **every** tool file, check:
|
||||
- [ ] All optional arrays use `?? []`
|
||||
- [ ] Error cases are handled: checks for missing/empty data and returns meaningful error
|
||||
- [ ] Does NOT do raw JSON dumps — extracts meaningful, individual fields
|
||||
- [ ] Every extracted field is backed by official docs or live-verified sample payloads
|
||||
|
||||
### Outputs
|
||||
- [ ] All output fields match what the API actually returns
|
||||
@@ -267,6 +282,7 @@ After fixing, confirm:
|
||||
1. `bun run lint` passes with no fixes needed
|
||||
2. TypeScript compiles clean (no type errors)
|
||||
3. Re-read all modified files to verify fixes are correct
|
||||
4. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
|
||||
@@ -44,6 +44,20 @@ Fetch the service's official webhook documentation. This is the **source of trut
|
||||
- Webhook subscription API (create/delete endpoints, if applicable)
|
||||
- Retry behavior and delivery guarantees
|
||||
|
||||
### Hard Rule: No Guessed Webhook Payload Schemas
|
||||
|
||||
If the official docs do not clearly show the webhook payload JSON for an event, you MUST tell the user instead of guessing.
|
||||
|
||||
- Do NOT invent payload field names
|
||||
- Do NOT infer nested payload paths without evidence
|
||||
- Do NOT treat likely event shapes as verified
|
||||
- Do NOT accept `formatInput` mappings that are not backed by docs or live payloads
|
||||
|
||||
If a payload schema is unknown, validation must explicitly recommend:
|
||||
1. sample webhook payloads,
|
||||
2. a live test webhook source, or
|
||||
3. trimming the trigger to only documented outputs.
|
||||
|
||||
## Step 3: Validate Trigger Definitions
|
||||
|
||||
### utils.ts
|
||||
@@ -93,6 +107,7 @@ Fetch the service's official webhook documentation. This is the **source of trut
|
||||
- [ ] Nested output paths exist at the correct depth (e.g., `resource.id` actually has `resource: { id: ... }`)
|
||||
- [ ] `null` is used for missing optional fields (not empty strings or empty objects)
|
||||
- [ ] Returns `{ input: { ... } }` — not a bare object
|
||||
- [ ] Every mapped payload field is backed by official docs or live-verified webhook payloads
|
||||
|
||||
### Idempotency
|
||||
- [ ] `extractIdempotencyId` returns a stable, unique key per delivery
|
||||
@@ -195,6 +210,7 @@ After fixing, confirm:
|
||||
1. `bun run type-check` passes
|
||||
2. Re-read all modified files to verify fixes are correct
|
||||
3. Provider handler tests pass (if they exist): `bun test {service}`
|
||||
4. Any remaining unknown webhook payload schemas were explicitly reported to the user instead of guessed
|
||||
|
||||
## Checklist Summary
|
||||
|
||||
|
||||
@@ -3554,7 +3554,7 @@ export function FireworksIcon(props: SVGProps<SVGSVGElement>) {
|
||||
>
|
||||
<path
|
||||
d='M314.333 110.167L255.98 251.729l-58.416-141.562h-37.459l64 154.75c5.23 12.854 17.771 21.312 31.646 21.312s26.417-8.437 31.646-21.27l64.396-154.792h-37.459zm24.917 215.666L446 216.583l-14.562-34.77-116.584 119.562c-9.708 9.958-12.541 24.833-7.146 37.646 5.292 12.73 17.792 21.083 31.584 21.083l.042.063L506 359.75l-14.562-34.77-152.146.853h-.042zM66 216.5l14.563-34.77 116.583 119.562a34.592 34.592 0 017.146 37.646C199 351.667 186.5 360.02 172.708 360.02l-166.666-.375-.042.042 14.563-34.771 152.145.875L66 216.5z'
|
||||
fill='currentColor'
|
||||
fill='#5019c5'
|
||||
/>
|
||||
</svg>
|
||||
)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user