From ee0157df4bd5d90f7543ba2d6f8ecd5f19cb5abf Mon Sep 17 00:00:00 2001 From: Waleed Date: Thu, 30 Jul 2026 17:21:17 -0700 Subject: [PATCH] feat(tables): add currency column type on a new column-type registry (#6106) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(tables): add currency column type on a new column-type registry Adds a `currency` column type, and consolidates the per-type knowledge it would otherwise have been scattered across. **Currency.** Stores a plain number and carries an ISO 4217 `currencyCode` as display metadata. That split is what keeps it cheap: filtering, sorting, uniqueness and CSV export all reuse the numeric paths unchanged, changing a column's currency rewrites no rows, and the public row output stays a number rather than a locale-formatted string consumers would have to reparse. Input accepts the shapes an amount actually arrives in — `$1,234.56`, `1 234,56 €`, `(12.00)` — so pastes, CSV imports and tool writes land as numbers instead of being nulled. **The registry.** Adding this type initially required edits in ~40 places: 32 switch arms under `lib/table`, ~26 UI branches, two hand-maintained icon maps, and a coercion implementation duplicated four times. Every one of those failed silently when missed — a missing `jsonbCastForType` arm compares numbers as text; a missing compatibility arm blocks all conversions. `lib/table/column-types/` now holds one file per type carrying its label, icon, badge colour, storage cast, filter operators, coercion, validation, compatibility and formatting. `Record` on both registries is the completeness gate: adding a type to the union is a compile error naming exactly the two files to fill in, and the interface then requires every field. The 32 switch arms are down to 3. Two duplicates collapse as a consequence: - The client no longer mirrors the server's select id-resolution. Those helpers lived in `validation.ts`, which imports drizzle, so anything reaching them became server-only and the grid hand-rolled its own copy. Extracting them to `select-options.ts` lets both sides share one implementation, so the optimistic cache can no longer disagree with what gets persisted. - The two icon maps become one registry read. It also fixes a live inconsistency it surfaced: currency got a numeric keypad in the grid's inline editor but a plain text field in the row modal. Behaviour-neutral by construction: all 1046 tests in the touched areas pass unchanged, with no test edits. * test(tables): guard the column-type registry's invariants Property tests for the registry itself rather than any one type: entries key by their own id, COLUMN_TYPES stays derived, an unknown type degrades to string instead of throwing, only opaque-id types restrict filter operators, only configuration-free types are CSV-inferable, and every type that can reject a draft has a message to show. Plus the metadata-ownership matrix, which pins the generic ownership check to the same answers the hardcoded per-type rules gave. These target the registry's silent-failure class — a wrong jsonbCast or a stray operator whitelist used to be invisible until a filter failed in SQL. Both are verified to fail under mutation. * fix(tables): read exponent-form amounts and reject bad currency PATCHes up front Two P1s from review. Scientific notation lost magnitude. `String()` emits exponent form past 1e21, so a stored amount round-trips through the editor as `1e+21` — and the sanitizer treated the `e` as decoration to strip, reading it back as 121. An untouched cell silently lost 19 orders of magnitude on its next edit. Exponent form is now taken at face value, but only when the string is wholly a numeric literal once symbols are removed, so `12 EUR` (whose `E` survives the strip) still parses through the separator path. A failed currency PATCH left a partial rename. `renameColumn` commits in its own transaction before the currency write, so a `currencyCode` the service would reject — an unsupported code, or any code on a non-currency column — errored only after the rename had stuck. Both are now caught before the first write, matching the guard the route already applies to unique-on-select for exactly this reason. * refactor(tables): finish the registry migration and drop the dead config Audit pass over every consumer, closing the gaps the first cut left. Functional gap: the copilot agent had no currency support at all — it could create a currency column with no code and could never re-denominate one. `add_column` and `update_column` now accept `currencyCode`, with the same up-front validation and the same code-only routing as the HTTP routes. Config that consumers were still restating, now read from the registry: - `supportsUnique` replaces the unique-on-select guard stated in three places (service, both column routes, the copilot tool). - `editor === 'toggle'` replaces seven `type === 'boolean'` checks in the grid and expanded popover, all of which meant the same thing. - `defaultMetadata` replaces the per-type stamping in `addTableColumn` and `updateColumnType`. - `sampleValue` replaces the per-type example values in the LLM prompt scaffolding. - `storesOpaqueIds` replaces the select filter in the find-row matcher. Dead config removed: `getTypeBadgeVariant` had zero callers (already dead on staging), and it was the only reader of `badgeVariant` — so the field, its union, and all seven values went with it. `inferFromCsv` was read by nothing but a comment; CSV inference is an ordered heuristic a boolean cannot express, so it is gone too and `InferredCsvColumnType` is no longer exported. Fixes a latent crash found on the way: unique-constraint checking normalized a cell keyed on its RUNTIME type but reconstructed it keyed on the column's DECLARED type, so a unique `date` column stored a bare `2024-01-01` and then threw `SyntaxError` parsing it back. Both directions now go through JSON unconditionally. Pre-existing, unrelated to currency. Adds the `/add-column-type` skill and a Tables section in CLAUDE.md/AGENTS.md pointing at it, so the next type is one file plus two registry entries. * fix(tables): run the column PATCH guards ahead of the rename, not after it Greptile was right and my previous reply was wrong. The guards were added in the right shape but the wrong place — below `renameColumn`, which is the first write and commits in its own transaction. A PATCH combining a rename with an invalid currency therefore still committed the rename and then returned 400, exactly the counterexample reported. Moved the column lookup and all three pre-flight guards above every write. This also closes the same latent hole for the pre-existing unique-on-select guard, which sat in the same position. Adds route tests that assert `renameColumn` was never called on each rejection path, and that a valid combined rename + currency change still targets the new name. Verified to fail against the previous ordering. * fix(tables): make the retype gate and the write path share one parser A simplify pass over the registry found two real defects and several places the abstraction was being worked around. Silent data loss on conversion. `isCompatibleWith` was hand-written per type and had already drifted from `coerce`, despite the interface promising they could not: `boolean` accepted '1'/'0'/0/1 in the gate but only 'true'/'false' in the write path, so converting a column holding "1" reported zero incompatible rows and then nulled every one of them. `date` drifted the other way. `isCompatibleWith` is now optional and defaults to `coerce(...).ok`, so the two are the same code; only `select` overrides, because its rules are about the column (cleared-vs-required, cardinality) not the value. `isColumnType` used `in`, which matches inherited keys — `isColumnType('toString')` was true and `columnTypeById('toString')` returned `Function.prototype.toString`, which the validator would then call `.validateDefinition()` on. Now `Object.hasOwn`. `defaultMetadata` only ran on the currency arm of a retype, so a future type would get its defaults on create but silently not on conversion. It now runs for every non-select target, carrying forward only metadata the TARGET type declares it owns — a currency→text conversion no longer strands a currencyCode. The index doc claimed the registry is kept out of the `@/lib/table` barrel so 44 server modules don't pull `@sim/emcn/icons`. That was false: `constants.ts` re-exported `COLUMN_TYPES` from the icon-carrying `registry.ts`, and the barrel re-exports `constants`. `COLUMN_TYPES` now lives in the icon-free `types.ts`; verified with an import tracer that both are icon-free again. Also: 5 no-op `validateDefinition`s and 4 duplicated formatters collapsed into registry defaults; `CURRENCY_OPTIONS` was an eager module-load IIFE costing ~8ms of ICU work on every table API route for a list only the config sidebar reads, now built on first call; and the skill's validation grep claimed 'should return nothing' when it returns 8 legitimate hits — it now explains how to tell a leak from a genuine special case. * fix(tables): reject a non-leading sign so dates don't parse as amounts Found by Cursor Bugbot. `parseCurrencyInput` dropped every `-` as decoration, so an ISO date's hyphens vanished and its digit groups joined: `2024-01-01` read as 20240101. With the gate now sharing the write path's parser, a date → currency conversion reported zero incompatible rows and silently turned every cell into a huge number. A sign is only meaningful at the front; an interior one means the string is not a single amount. Leading signs, accounting parentheses, symbols, ISO codes, grouping separators, and exponent form all still parse — covered by the existing cases plus new ones, verified to fail without the fix. * fix(tables): use getErrorMessage in the columns route test mock `check:utils` bans the inline `e instanceof Error ? e.message : fallback` form; the mock for `rootErrorMessage` used it. * fix(tables): rename the column last so a failed write leaves it untouched Greptile's remaining concern: the pre-flight guards read a schema snapshot, so a column-type change landing concurrently can still make a later write fail — and with the rename running first, that failure returned an error with the rename already committed. Guards cannot close that window; each write is its own locked transaction and only the write itself sees the authoritative state. Ordering can. The rename is the one write that is purely cosmetic, so it now runs last: a failed typed write leaves the column entirely untouched, and a failed rename leaves the typed change applied under the old name — the recoverable half. The typed writes target the column's current name, since no rename has happened yet. Tests cover both directions: a typed write rejected mid-flight must not rename, and a successful one must rename strictly after. Verified to fail under the previous ordering. * fix(tables): write back coerced values on every conversion Round 4 findings, all real. A conversion is allowed exactly when the target type's `coerce` accepts the value — and `coerce` frequently TRANSFORMS it. Only `select` and `currency` wrote the transformed value back, so a conversion to any other transforming type left the cell holding its old bytes under the new type. Converting a number column to `date` accepted epoch values, stored them unchanged, and then `(data->>'col')::timestamptz` failed on EVERY query against that column. I opened this myself by defaulting `isCompatibleWith` to `coerce(...).ok`. Fixed at the class rather than the instance: the compatibility scan now records whatever `coerce` produced whenever it differs from what is stored, and one generic write-back applies it. That subsumes the currency-specific migration entirely, so it and its helpers are gone. `select` keeps its own id↔name migrations, which are not coerce-expressible in the outbound direction. The post-conversion column definition is built once, before the scan, so the coercion reads the same metadata the stored value is later validated against. Exponent parsing was ambiguous when followed by text: `1e5 EUR` read as 15. An `e` with a digit on both sides is an exponent marker, so if the string is not a clean numeric literal it is refused rather than guessed — the digit on both sides is what keeps the `E` inside `12 EUR` parsing normally. A failed rename could still leave a typed change committed. The one rename failure a caller can cause — a name already taken — is now rejected up front, leaving only the concurrent-collision race, which no pre-flight check can close without spanning all writes in one transaction. * fix(tables): stop a blank cell blocking an optional type conversion Found by Cursor Bugbot. `''` is incompatible with every numeric type, and the compatibility scan counted it as a hard blocker regardless of whether the target was optional — so a text column with a single empty cell could not be converted to a number at all, and the error said 'to a required ...' either way. An unreadable-but-empty cell is not a conversion failure. The write path already turns an unreadable value into null on an optional column, so the conversion now does the same and records null for it. A required target still reports it, which the existing guard above already does with the message that actually fits. Also pins the two intentional divergences from the pre-registry behavior. A differential run of the registry against the pre-refactor implementations (55 values x 7 column shapes) found ZERO coercion differences and exactly two compatibility differences, both deliberate: boolean now rejects the '1'/'0' conversions the old gate accepted and then nulled, and date now accepts the epoch numbers its write path always accepted. Tests pin both so neither can be silently reverted or widened. * fix(tables): refuse conversions that would invent or destroy values Final adversarial scan found two data-corrupting conversions, both opened by defaulting the retype gate to the write path's parser. number → date destroyed every value. `date.coerce` reads a number as epoch milliseconds, which is right for one deliberate write and catastrophic applied to a whole column: 1, 5, 42 became three timestamps in January 1970, and a Unix-seconds column landed in 1970 rather than the year it meant. Irreversible. `date` now overrides the gate to reject numbers, restoring the pre-refactor behavior, and the contract states the rule the override obeys: a gate may be STRICTER than `coerce`, never looser. Stricter refuses a bulk conversion while single writes still work; looser is the direction that corrupts. string → currency invented values. The parser stripped every non-digit and joined what was left, so `01/02/2024` read as 1022024, `Room 101` as 101, and `0.1.2` as 12 — a column of SKUs or phone numbers converted with zero reported incompatibilities. What remains after removing symbols, spacing and an ISO code must now be only digits and separators, and grouping must be well-formed (a first group of 1-3 digits, the rest exactly 3). Every legitimate form still parses, including all the locale variants. Also generifies the last three metadata leaks: `buildConvertedColumn` strips and carries back by iterating the key list rather than naming keys (naming them meant a future type's metadata rode onto a target that rejects it, failing that column's validation on every later write), `normalizeColumn` forwards metadata through a shared `typeMetadataOf`, and `filterOperatorsFor` moved onto the definition — it was a per-type branch inside the registry's own accessor, the one thing the registry exists to forbid. Skill corrected: it claimed COLUMN_TYPES derives from the registry (backwards), promised exactly two compile errors (four once a type owns metadata), used a grep that missed half the real branches, and never mentioned `import.ts`'s second coercion path, whose silent default arm is the costliest miss available. Differential re-run vs the pre-refactor implementations: 0 coercion differences, 1 intentional compatibility difference (boolean no longer accepts the 0/1 conversions the old gate accepted and then nulled). * fix(tables): let the row modal accept formatted amounts again Found by Cursor Bugbot. I unified the row modal's input type with the grid's `inputMode` last round, but in the wrong direction: mapping `inputMode: 'decimal'` to `` made the modal reject $1,234.56, 1.234,56 and (12.00) — the exact formats `parseCurrencyInput` exists to accept, and which the grid's inline editor takes fine. A native number input and a numeric keypad are different things. Types whose parser accepts formatted text now say so, and get a text field with `inputMode='decimal'` — the shape the grid already uses. A plain number keeps the native input, its spinner, and its validation. * fix(tables): fold a rename into the write it accompanies Closes the last partial-update window, properly rather than by pre-checking around it. A rename is metadata-only — `renameColumn`'s own comment says so: rows, metadata, and workflow-group refs all key on the stable column id, so it is a pure schema write. Nothing forced it to be its own transaction. Running it separately is what created the window: whichever half committed first survived a failure in the other, and no pre-flight guard can close a concurrent collision because only the write itself sees authoritative state. The four column writes now accept an optional `newName` and apply it through one shared `applyPendingRename`, which validates the name shape and checks the collision against the very schema snapshot that write is landing in. A combined request rides the rename on whichever write runs last, so both halves commit together or neither does — a concurrent claim on the name now aborts the whole transaction instead of leaving the other change applied. The routes also address every write by the column's stable id rather than its name, so folding a rename into one write cannot break the next one's lookup. A rename with nothing to ride on still runs standalone. What remains partial is a type write followed by a failing constraints write — two independently locked transactions, pre-existing, and untouched by this PR. * fix(tables): migrate scalar cells when converting a column to select Found by Cursor Bugbot. `resolveSelectOptionId` stringifies a number or boolean before matching, so a `number` column whose values equal option NAMES passes the compatibility gate — but `migrateCellsToSelectIds` only rewrote JSONB `string` and `array` cells. Those cells stayed raw numbers inside a select column, where they render as nothing and fail option membership on the next write. `data->>key` yields the text form for every scalar, so the existing lookup already worked; the predicate was simply too narrow. Widened to cover `number` and `boolean`. The outbound migration is unchanged — cells leaving a select column are option ids, always strings or arrays. Pre-existing on staging (both the resolver's scalar handling and the migration SQL predate this branch), but it lives in a file this PR creates. Tests pin the resolver behavior the predicate depends on, so narrowing either one without the other now fails. * fix(tables): validate a retype's unique against the values it writes Validated the last partial-update seam with a focused investigation rather than assuming. The answer was split. `required` is already safe: `updateColumnType` runs the same `countEmptyCells` against the constraint the request is about to set, which is why that check exists. `unique` was not, and the reachable case commits the unrecoverable half. A text column holding "5" and "5.0", PATCHed with {type: number, unique: true}: the conversion succeeds and coerces both to 5, then the separate constraint write finds duplicates and 400s — with the column already numeric and "5.0" irreversibly rewritten. A pre-scan of the raw text finds nothing; the conversion is what manufactures the duplicate. The retype now carries `unique` and checks it after the write-back, against the values it just wrote. Constraint changes on a workflow-output column were the same shape — rejected by the constraint write, after a type change had committed. Now rejected in the route's pre-flight block, before any write. The duplicate scan is extracted and shared between both paths for the same reason `countEmptyCells` is: two copies of one rule is the drift that produced the original required-check bug. Deliberately NOT merging `updateColumnType` and `updateColumnConstraints`. They assert different lock levels (destructive vs schema-only) and only the retype needs the full row scan, so merging would either force a constraints-only toggle to materialize every row or reintroduce the branching it was meant to remove. With both reachable failures pre-validated, what remains at the seam is concurrent races no in-process check can close. * fix(tables): don't drop a rename when the write it rides on no-ops Found by Cursor Bugbot — a bug I introduced folding the rename in. `updateColumnCurrency` returns early when the code is unchanged, and that return sat ahead of the rename, so PATCH {name, currencyCode} with the column's current code answered 200 with the rename silently discarded. Both early returns now treat a pending rename as work: the currency path only no-ops when the code is unchanged AND no rename is riding along, and the retype path applies a rename-only write when the type is unchanged. `applyPendingRename` signals "nothing to do" by returning the same reference, which is what lets both detect it cleanly. Also extracts `persistColumns` — five sites were repeating the same schema-write-and-return. * fix(tables): make a combined column PATCH a single transaction Finishes the fold-in rather than pre-validating around the seam. A retype now APPLIES the constraints it already validates against — it checks empty cells for `required` and post-conversion duplicates for `unique`, so it was doing the work without persisting the result — and the route skips the separate constraint write when the type changed. A request combining a rename, a retype and constraint changes is now one locked transaction: no half of it can commit while another fails. The separate constraint write remains for requests that do not change type, which is the only case that still needs it. Deliberately still NOT merging the two service functions. They assert different lock levels (destructive vs schema-only) and only the retype needs the full row scan into memory, so a merged function would force a constraints-only toggle to materialize every row or reintroduce the branching it was meant to remove. Folding the payload in gets atomicity without either cost. * fix(tables): reject a flattened list as an amount; fold constraints into every typed write Two findings from round 11. Multi-select converted to nonsense amounts. `selectValueForConversion` flattens a multi cell to its comma-joined option names, and the parser read that as a formatted number: options 12 and 34 became 12.34, and 100 and 200 became 100200. No real amount puts whitespace after a separator, but a delimited list does — so a separator followed by whitespace is now refused. Every legitimate form still parses, including space-grouped locales. Combined options-or-currency + constraints could still commit partially. Those two writes now carry constraints the same way the retype does, through one shared `applyConstraints` that validates (workflow-output, empty cells for required, supportsUnique and duplicates for unique) and applies them. The separate constraint write now runs only when no typed write does. Three copies of those rules is the drift that produced the original required-check bug, so they live in one place. * fix(tables): validate constraints after the migrations that rewrite cells Self-caught while reviewing my own previous commit, which introduced both. `updateColumnOptions` ran the shared `applyConstraints` BEFORE its cell migrations. Those migrations rewrite stored values — a single<->multi toggle changes the shape, removing an option clears cells — so a `unique` scan read the pre-migration values, passed, and the rewrite could then produce the duplicates the scan was meant to prevent. Moved to after the migrations, which is where `updateColumnType` already had it. The same commit also left the options path running `required`'s empty-cell check twice: once in the shared helper and once in its original inline block, whose comment still described a separate constraint write that no longer runs. Removed the duplicate — one query, one rule, which is the whole point of the shared helper. Also routes the options path through `persistColumns` like the others. * fix(tables): stop inventing amounts from identifiers; fix the copilot retype Adversarial pass over the final state, seven real findings. Two destroyed data. The copilot `update_column` still used the two-transaction pattern the HTTP routes were fixed for: `unique` was never forwarded to the typed write, so a retype+unique committed the conversion and then failed the constraint — the same irrecoverable half. It now rides the typed write, and the separate constraint write only runs when no typed write did. And the parser's three-letter strip removed ANY three letters, not an ISO code: `SKU400` parsed as 400, `ABC1234` as 1234. Converting a column of part numbers to currency rewrote every cell with an invented value — while the comment two lines above claimed a SKU was exactly what it prevented. The rule is now that a letter touching a digit means identifier, not amount; a currency marker is always separated by a space or a symbol. That same change fixed a class the review surfaced: the pinned currencies could not parse their own conventional notation. `R$ 1.234,56`, `1 234,56 kr`, `1234,56 zł`, `CHF 1’234.56` and Indian lakh grouping (`₹12,34,567.89`) all work now — these are what Intl emits, so a paste from a spreadsheet was being rejected. `updateColumnConstraints` was a fourth copy of the constraint rules the shared helper exists to unify, and had already drifted: it hardcoded `type === 'select'` where the helper asks the registry, so a future type declaring `supportsUnique: false` would have been ignored on that path. It now uses the helper. `updateColumnType`'s unchanged-type early return silently discarded every field except the rename. Callers gate on the type changing, but from a read taken before the lock — so a concurrent change could land there with real work pending and answer success. It now throws. Also: `UpdateColumnCurrencyData` was missing `required`, which only compiled because the routes pass it through a spread; a missing column returns 404 instead of a 400 reading "of type undefined"; and the comments describing the old two-transaction architecture are gone. Verified NOT a bug: CSV export of a currency column writes the raw number, so export/import round-trips losslessly. * fix(tables): read the negative and RTL forms Intl actually emits An Intl sweep across 24 locales found two forms the parser rejected, both from an ordinary spreadsheet paste. `Intl` emits U+2212 MINUS SIGN rather than the ASCII hyphen for negatives in several locales, so `−12,50 kr` read as null instead of -12.5. And it wraps RTL-locale output in invisible bidi control marks, so `‏1,234.56 ‏₪` carried characters that are not part of the amount. Both are now normalized away. 24 locales x 6 amounts now round-trip, up from 99/100 when the sweep started — and the test generates them from `Intl` rather than listing them by hand, so a parser change cannot quietly regress a locale nobody remembered to write down. Locales that format with their own numeral systems (Arabic-Indic) are still rejected, and now say so in the docstring. That is a safe failure — null rather than a wrong value — and supporting them is a wider decision than this type, since it would also touch `number`, display, and sorting. --- .agents/skills/add-column-type/SKILL.md | 160 ++++ .claude/commands/add-column-type.md | 159 ++++ .claude/skills/add-column-type | 1 + .cursor/commands/add-column-type.md | 154 ++++ AGENTS.md | 6 + CLAUDE.md | 6 + .../content/docs/en/integrations/table.mdx | 2 +- apps/docs/content/docs/en/tables/index.mdx | 4 + .../api/table/[tableId]/columns/route.test.ts | 320 ++++++++ .../app/api/table/[tableId]/columns/route.ts | 140 +++- apps/sim/app/api/table/utils.ts | 6 +- .../api/v1/tables/[tableId]/columns/route.ts | 140 +++- .../column-config-sidebar.tsx | 45 ++ .../column-config-sidebar/column-types.ts | 27 +- .../components/row-modal/row-modal.tsx | 27 +- .../table-grid/cells/cell-render.tsx | 8 + .../cells/expanded-cell-popover.tsx | 22 +- .../table-grid/cells/inline-editors.tsx | 31 +- .../table-grid/headers/column-type-icon.tsx | 28 +- .../components/table-grid/headers/index.ts | 2 +- .../components/table-grid/table-grid.tsx | 69 +- .../workflow-sidebar/workflow-sidebar.tsx | 12 +- .../[workspaceId]/tables/[tableId]/table.tsx | 4 +- .../[workspaceId]/tables/[tableId]/utils.ts | 107 +-- apps/sim/hooks/use-table-undo.ts | 1 + apps/sim/lib/api/contracts/tables.ts | 34 +- .../copilot/tools/server/table/user-table.ts | 61 +- .../table/__tests__/column-conversion.test.ts | 145 +++- .../__tests__/column-type-registry.test.ts | 205 ++++++ apps/sim/lib/table/__tests__/currency.test.ts | 279 +++++++ .../lib/table/__tests__/validation.test.ts | 66 ++ apps/sim/lib/table/cell-format.ts | 3 +- apps/sim/lib/table/column-naming.ts | 16 +- apps/sim/lib/table/column-types/boolean.ts | 40 + apps/sim/lib/table/column-types/currency.ts | 68 ++ apps/sim/lib/table/column-types/date.ts | 63 ++ apps/sim/lib/table/column-types/index.ts | 21 + apps/sim/lib/table/column-types/json.ts | 38 + apps/sim/lib/table/column-types/number.ts | 45 ++ .../lib/table/column-types/registry.server.ts | 238 ++++++ apps/sim/lib/table/column-types/registry.ts | 117 +++ apps/sim/lib/table/column-types/select.ts | 143 ++++ apps/sim/lib/table/column-types/string.ts | 39 + .../lib/table/column-types/types.server.ts | 47 ++ apps/sim/lib/table/column-types/types.ts | 199 +++++ apps/sim/lib/table/columns/service.ts | 694 +++++++++++------- apps/sim/lib/table/constants.ts | 12 +- apps/sim/lib/table/currency.ts | 326 ++++++++ apps/sim/lib/table/export-format.ts | 9 +- apps/sim/lib/table/import.test.ts | 9 + apps/sim/lib/table/import.ts | 33 +- apps/sim/lib/table/index.ts | 1 + apps/sim/lib/table/llm/enrichment.ts | 5 +- apps/sim/lib/table/rows/service.ts | 3 +- apps/sim/lib/table/select-options.ts | 87 +++ apps/sim/lib/table/select-values.ts | 2 +- apps/sim/lib/table/sql.ts | 46 +- apps/sim/lib/table/types.ts | 62 +- apps/sim/lib/table/validation.ts | 280 ++----- apps/sim/stores/table/types.ts | 3 + packages/emcn/src/icons/index.ts | 1 + packages/emcn/src/icons/type-currency.tsx | 26 + 62 files changed, 4149 insertions(+), 798 deletions(-) create mode 100644 .agents/skills/add-column-type/SKILL.md create mode 100644 .claude/commands/add-column-type.md create mode 120000 .claude/skills/add-column-type create mode 100644 .cursor/commands/add-column-type.md create mode 100644 apps/sim/app/api/table/[tableId]/columns/route.test.ts create mode 100644 apps/sim/lib/table/__tests__/column-type-registry.test.ts create mode 100644 apps/sim/lib/table/__tests__/currency.test.ts create mode 100644 apps/sim/lib/table/column-types/boolean.ts create mode 100644 apps/sim/lib/table/column-types/currency.ts create mode 100644 apps/sim/lib/table/column-types/date.ts create mode 100644 apps/sim/lib/table/column-types/index.ts create mode 100644 apps/sim/lib/table/column-types/json.ts create mode 100644 apps/sim/lib/table/column-types/number.ts create mode 100644 apps/sim/lib/table/column-types/registry.server.ts create mode 100644 apps/sim/lib/table/column-types/registry.ts create mode 100644 apps/sim/lib/table/column-types/select.ts create mode 100644 apps/sim/lib/table/column-types/string.ts create mode 100644 apps/sim/lib/table/column-types/types.server.ts create mode 100644 apps/sim/lib/table/column-types/types.ts create mode 100644 apps/sim/lib/table/currency.ts create mode 100644 apps/sim/lib/table/select-options.ts create mode 100644 packages/emcn/src/icons/type-currency.tsx diff --git a/.agents/skills/add-column-type/SKILL.md b/.agents/skills/add-column-type/SKILL.md new file mode 100644 index 0000000000..05f9816aa5 --- /dev/null +++ b/.agents/skills/add-column-type/SKILL.md @@ -0,0 +1,160 @@ +--- +name: add-column-type +description: Add a new table column type to Sim — registry entry, icon, storage shape, coercion, and the behavioral hooks the grid and API read. Use when adding a value kind under `apps/sim/lib/table/column-types/`. +argument-hint: +--- + +# Adding a Table Column Type + +A column type is **one file** in `apps/sim/lib/table/column-types/` plus a registry entry. Everything that varies per type — label, icon, storage cast, coercion, validation, conversion compatibility, formatting, editor, filter operators — lives on that one object, so no consumer needs editing. + +This was not always true: adding `currency` originally took ~40 edits across 32 `switch` arms and 26 UI branches, each of which failed **silently** when missed. The registry exists to make that impossible, so the rule is absolute: **if you find yourself adding a `case 'yourtype':` anywhere outside `column-types/`, the registry is missing a field. Add the field instead.** + +## Hard Rule: the compiler tells you what to do + +Do **not** hunt for places to edit. Add your type to the `ColumnType` union first and let `tsc` produce the list: + +```bash +cd apps/sim && bunx tsc --noEmit -p tsconfig.json +``` + +You will get two errors, naming `column-types/registry.ts` and `column-types/registry.server.ts`. Register in both. + +If your type owns metadata, adding its key to `TYPE_SPECIFIC_COLUMN_KEYS` produces two more legitimate errors — `FOREIGN_METADATA_VERB` in `validation.ts` (a `Record` over those keys) and the key's absence from `ColumnDefinition`. Those are the gate working, not sites to "fix". + +Any error beyond those four is a site reading a hardcoded type list that should read the registry — fix that site, don't work around it. + +## Directory Structure + +``` +apps/sim/lib/table/column-types/ +├── types.ts # ColumnTypeDefinition — the contract you implement +├── types.server.ts # ColumnTypeServerDefinition — cell migrations only +├── registry.ts # Record ← client-safe, the gate +├── registry.server.ts # Record ← adds migrations (drizzle) +├── index.ts # barrel + accessors (columnTypeOf, columnTypeById, …) +└── {type}.ts # one file per type — what you write +``` + +## Step 1: Pick the storage shape + +Decide what a cell literally holds in `user_table_rows.data` (JSONB). This drives almost everything else: + +| Storage | `jsonbCast` | Notes | +|---------|-------------|-------| +| number | `'numeric'` | Filters/sorts compare numerically. `currency` does this. | +| ISO string | `'timestamptz'` | `date` does this. | +| string / bool / object | `null` | Text comparison is correct. | + +**Prefer an existing primitive over a new shape.** `currency` stores a plain number and keeps its ISO code as *display metadata* — which is why filtering, sorting, uniqueness, and CSV export all reuse the numeric paths untouched, and why re-denominating a column rewrites zero rows. + +## Step 2: Add the icon + +Create `packages/emcn/src/icons/type-{name}.tsx`, copying the geometry conventions of its siblings exactly: + +```tsx +import type { SVGProps } from 'react' + +/** + * Type {name} icon component - {what the glyph is} for {name} columns + * @param props - SVG properties including className, fill, etc. + */ +export function Type{Pascal}(props: SVGProps) { + return ( + + ) +} +``` + +- `viewBox='-1.75 -1.5 24 24'` is the **`type-*` family** value, not the set-wide default. Match the family. +- Center the glyph on the viewBox's optical center (**y = 10.5**, **x = 10.25**) — every sibling does, and a few tenths off is visible at `size-[14px]`. +- Export alphabetically **by component name** in `packages/emcn/src/icons/index.ts`. + +## Step 3: Write the type file + +`apps/sim/lib/table/column-types/{name}.ts`. Copy the closest existing type and change what differs. Every field is required by the interface, so the compiler enumerates them for you — read the TSDoc in `types.ts` rather than guessing. + +The three that are easy to get wrong: + +- **`coerce`** is the *single* write-path implementation. The server runs it before persisting **and** the grid runs it to fill the optimistic cache. Accept every shape the value legitimately arrives in (paste, CSV, tool write), because rejecting means the cell is nulled. +- **`isCompatibleWith`** gates type conversion and must read the value **exactly as `coerce` will**, or a conversion will pass its check and then null the cell. +- **`ownedMetadata`** lists the `ColumnDefinition` keys your type owns. Anything you add must also be added to `TYPE_SPECIFIC_COLUMN_KEYS` in `types.ts` and given a phrase in `FOREIGN_METADATA_VERB` in `validation.ts` — both are `Record`-typed, so the compiler will tell you. + +## Step 4: Register + +Add the entry to `COLUMN_TYPE_REGISTRY` in `registry.ts` **and** `COLUMN_TYPE_SERVER_REGISTRY` in `registry.server.ts`. + +`COLUMN_TYPES` is declared in `types.ts` (not derived from the registry — the registry is annotated `Record` against it, which is the gate). `constants.ts` re-exports it, so `columnTypeSchema = z.enum(COLUMN_TYPES)` picks your type up with no edit. **Type-specific metadata does not** — see the next step. + +## Step 5: Migrations (only if the stored bytes change) + +If converting an existing column **to** your type must rewrite cells, add `migrateCellsTo` in `registry.server.ts`; if converting **away** must rewrite them, add `migrateCellsFrom`. + +This is load-bearing, not cosmetic: filters and sorts apply `jsonbCast` to whatever is stored, so leaving a non-castable string behind makes **every query on that column fail** — not merely render oddly. + +Prefer set-based SQL. When the transform genuinely needs JS (`currency`'s separator disambiguation), compute the values during the compatibility scan and pass them through `resolved`, then apply them in one batched statement. + +## Naming Convention + +- Type id: lowercase, singular — `currency`, not `Currency` or `currencies` +- File: `column-types/{id}.ts`, export `const {id}ColumnType` +- Icon: `type-{kebab}.tsx`, export `Type{Pascal}` + +## Watch out + +- **Import cycles.** `column-types/select.ts` imports `select-values.ts`, so `select-values.ts` must **not** import the registry — that closes a cycle and fails at module init. Inside a type's own helper module the string literal is the implementation, not a config leak. +- **The client-safe boundary.** `registry.ts` and everything it imports must stay free of `@sim/db`, `drizzle-orm`, and `next/server` — the tables grid imports it directly. A React icon is fine (it's a component *reference*, never called server-side). Only `registry.server.ts` may touch drizzle. +- **Don't re-export the registry from `@/lib/table`.** 44 server modules import that barrel; routing this through it pulls `@sim/emcn/icons` into all of them. Deep-import `@/lib/table/column-types`. +- **`import.ts`'s `coerceValue` is a SECOND write path and is not opt-in.** Importing into a column of your type always hits it, and its `default` arm silently `String(value)`s — so a missing `case` stores text in a column whose `jsonbCast` is numeric, and then every filter and sort on that column errors in Postgres. Add a `case`, even though the switch compiles without one. (It is deliberately separate from the registry's `coerce`: an import wants an unparseable value to survive as its raw string so the row error can name it.) +- **CSV inference** is an ordered heuristic in `import.ts`, deliberately not registry-driven. A new type is not inferred from a CSV unless you extend `inferColumnType` — usually you should not, since inference cannot supply configuration (an option set, a currency code). + +## If your type owns metadata, read this + +Registering the *type* is compiler-enforced. Registering its *metadata* is not, and that is where the remaining manual work lives. A key like `precision` has to be added in each of these, none of which will fail to compile if you forget: + +| Where | What happens if you forget | +|---|---| +| `lib/table/types.ts` `ColumnDefinition` | (this one DOES fail — the ownership loop indexes it) | +| `column-types/types.ts` `TYPE_SPECIFIC_COLUMN_KEYS` | it is never stripped on conversion, and poisons the target type | +| `lib/api/contracts/tables.ts` — the schema slot in all three column schemas, plus `refineColumnOptions` | zod strips it at the boundary; silently never saved | +| `columns/service.ts` `addTableColumn` param type | callers cannot pass it | +| A metadata-only update path (`updateColumnCurrency` is the model) + a branch in both column routes + the copilot tool | changing it on an existing column is a silent 200 no-op | +| `column-config-sidebar.tsx` | no UI to set it | +| `table-grid.tsx` delete-column undo + `use-table-undo.ts` restore | undo silently resets it to the default | + +`normalizeColumn`, `buildConvertedColumn`, and the undo snapshot read `TYPE_SPECIFIC_COLUMN_KEYS` generically, so those three are already zero-edit. + +**Known gap:** the metadata-only update path is ~6 near-identical copies (service + 2 routes + copilot). A `metadataUpdate` descriptor on `ColumnTypeServerDefinition` would collapse them; until that exists, copy `currency`'s. + +## Checklist Before Finishing + +- [ ] Added to the `ColumnType` union in `column-types/types.ts` +- [ ] `column-types/{id}.ts` created, every interface field filled in +- [ ] Registered in **both** `registry.ts` and `registry.server.ts` +- [ ] Icon added, centered on the family's optical center, exported alphabetically +- [ ] `migrateCellsTo` / `migrateCellsFrom` added if the stored bytes change +- [ ] New metadata keys added to `TYPE_SPECIFIC_COLUMN_KEYS` + `FOREIGN_METADATA_VERB` +- [ ] Unit tests for `coerce` / `isCompatibleWith` round-trips, verified to fail without the code +- [ ] Docs row added to `apps/docs/content/docs/en/tables/index.mdx` + +## Final Validation (Required) + +1. **`cd apps/sim && bunx tsc --noEmit -p tsconfig.json`** — must be clean. If any file *outside* `column-types/` errors, that file has a hardcoded type list; fix it to read the registry. +2. **Grep for leaks** — `grep -rnE "(===|!==) '{id}'|case '{id}':" apps/sim --include='*.ts' --include='*.tsx' | grep -v column-types/`. (All three forms: a plain `!==` and a `case` are how half of `currency`'s real branches are written.) Hits are expected; judge each. A hit is fine when it mounts a specific React component or encodes a genuinely one-off behavior (`json`'s mono textarea, `date`'s timezone-aware parsing). A hit is a **leak** when it restates something the registry could answer — an icon, a label, a colour, an operator set, a cast, a coercion. Leaks get a registry field, not a new branch. +3. **Run the suite** — `bunx vitest run lib/table 'app/workspace/[workspaceId]/tables' lib/api app/api/table app/api/v1 lib/copilot/tools/server/table`. Existing tests must pass **unchanged**; needing to edit one means you changed behavior for the other types. +4. **`bun run lint:check`, `bun run check:api-validation`, `bun run check:client-boundary`** from the repo root. +5. **Exercise it in the running app** on a table with one column of every type: create, edit inline / in the expanded popover / in the row modal, paste from a spreadsheet, filter, sort, convert to and from other types, export CSV, undo a column delete. diff --git a/.claude/commands/add-column-type.md b/.claude/commands/add-column-type.md new file mode 100644 index 0000000000..b390ccc0b9 --- /dev/null +++ b/.claude/commands/add-column-type.md @@ -0,0 +1,159 @@ +--- +description: Add a new table column type to Sim — registry entry, icon, storage shape, coercion, and the behavioral hooks the grid and API read. Use when adding a value kind under `apps/sim/lib/table/column-types/`. +argument-hint: +--- + +# Adding a Table Column Type + +A column type is **one file** in `apps/sim/lib/table/column-types/` plus a registry entry. Everything that varies per type — label, icon, storage cast, coercion, validation, conversion compatibility, formatting, editor, filter operators — lives on that one object, so no consumer needs editing. + +This was not always true: adding `currency` originally took ~40 edits across 32 `switch` arms and 26 UI branches, each of which failed **silently** when missed. The registry exists to make that impossible, so the rule is absolute: **if you find yourself adding a `case 'yourtype':` anywhere outside `column-types/`, the registry is missing a field. Add the field instead.** + +## Hard Rule: the compiler tells you what to do + +Do **not** hunt for places to edit. Add your type to the `ColumnType` union first and let `tsc` produce the list: + +```bash +cd apps/sim && bunx tsc --noEmit -p tsconfig.json +``` + +You will get two errors, naming `column-types/registry.ts` and `column-types/registry.server.ts`. Register in both. + +If your type owns metadata, adding its key to `TYPE_SPECIFIC_COLUMN_KEYS` produces two more legitimate errors — `FOREIGN_METADATA_VERB` in `validation.ts` (a `Record` over those keys) and the key's absence from `ColumnDefinition`. Those are the gate working, not sites to "fix". + +Any error beyond those four is a site reading a hardcoded type list that should read the registry — fix that site, don't work around it. + +## Directory Structure + +``` +apps/sim/lib/table/column-types/ +├── types.ts # ColumnTypeDefinition — the contract you implement +├── types.server.ts # ColumnTypeServerDefinition — cell migrations only +├── registry.ts # Record ← client-safe, the gate +├── registry.server.ts # Record ← adds migrations (drizzle) +├── index.ts # barrel + accessors (columnTypeOf, columnTypeById, …) +└── {type}.ts # one file per type — what you write +``` + +## Step 1: Pick the storage shape + +Decide what a cell literally holds in `user_table_rows.data` (JSONB). This drives almost everything else: + +| Storage | `jsonbCast` | Notes | +|---------|-------------|-------| +| number | `'numeric'` | Filters/sorts compare numerically. `currency` does this. | +| ISO string | `'timestamptz'` | `date` does this. | +| string / bool / object | `null` | Text comparison is correct. | + +**Prefer an existing primitive over a new shape.** `currency` stores a plain number and keeps its ISO code as *display metadata* — which is why filtering, sorting, uniqueness, and CSV export all reuse the numeric paths untouched, and why re-denominating a column rewrites zero rows. + +## Step 2: Add the icon + +Create `packages/emcn/src/icons/type-{name}.tsx`, copying the geometry conventions of its siblings exactly: + +```tsx +import type { SVGProps } from 'react' + +/** + * Type {name} icon component - {what the glyph is} for {name} columns + * @param props - SVG properties including className, fill, etc. + */ +export function Type{Pascal}(props: SVGProps) { + return ( + + ) +} +``` + +- `viewBox='-1.75 -1.5 24 24'` is the **`type-*` family** value, not the set-wide default. Match the family. +- Center the glyph on the viewBox's optical center (**y = 10.5**, **x = 10.25**) — every sibling does, and a few tenths off is visible at `size-[14px]`. +- Export alphabetically **by component name** in `packages/emcn/src/icons/index.ts`. + +## Step 3: Write the type file + +`apps/sim/lib/table/column-types/{name}.ts`. Copy the closest existing type and change what differs. Every field is required by the interface, so the compiler enumerates them for you — read the TSDoc in `types.ts` rather than guessing. + +The three that are easy to get wrong: + +- **`coerce`** is the *single* write-path implementation. The server runs it before persisting **and** the grid runs it to fill the optimistic cache. Accept every shape the value legitimately arrives in (paste, CSV, tool write), because rejecting means the cell is nulled. +- **`isCompatibleWith`** gates type conversion and must read the value **exactly as `coerce` will**, or a conversion will pass its check and then null the cell. +- **`ownedMetadata`** lists the `ColumnDefinition` keys your type owns. Anything you add must also be added to `TYPE_SPECIFIC_COLUMN_KEYS` in `types.ts` and given a phrase in `FOREIGN_METADATA_VERB` in `validation.ts` — both are `Record`-typed, so the compiler will tell you. + +## Step 4: Register + +Add the entry to `COLUMN_TYPE_REGISTRY` in `registry.ts` **and** `COLUMN_TYPE_SERVER_REGISTRY` in `registry.server.ts`. + +`COLUMN_TYPES` is declared in `types.ts` (not derived from the registry — the registry is annotated `Record` against it, which is the gate). `constants.ts` re-exports it, so `columnTypeSchema = z.enum(COLUMN_TYPES)` picks your type up with no edit. **Type-specific metadata does not** — see the next step. + +## Step 5: Migrations (only if the stored bytes change) + +If converting an existing column **to** your type must rewrite cells, add `migrateCellsTo` in `registry.server.ts`; if converting **away** must rewrite them, add `migrateCellsFrom`. + +This is load-bearing, not cosmetic: filters and sorts apply `jsonbCast` to whatever is stored, so leaving a non-castable string behind makes **every query on that column fail** — not merely render oddly. + +Prefer set-based SQL. When the transform genuinely needs JS (`currency`'s separator disambiguation), compute the values during the compatibility scan and pass them through `resolved`, then apply them in one batched statement. + +## Naming Convention + +- Type id: lowercase, singular — `currency`, not `Currency` or `currencies` +- File: `column-types/{id}.ts`, export `const {id}ColumnType` +- Icon: `type-{kebab}.tsx`, export `Type{Pascal}` + +## Watch out + +- **Import cycles.** `column-types/select.ts` imports `select-values.ts`, so `select-values.ts` must **not** import the registry — that closes a cycle and fails at module init. Inside a type's own helper module the string literal is the implementation, not a config leak. +- **The client-safe boundary.** `registry.ts` and everything it imports must stay free of `@sim/db`, `drizzle-orm`, and `next/server` — the tables grid imports it directly. A React icon is fine (it's a component *reference*, never called server-side). Only `registry.server.ts` may touch drizzle. +- **Don't re-export the registry from `@/lib/table`.** 44 server modules import that barrel; routing this through it pulls `@sim/emcn/icons` into all of them. Deep-import `@/lib/table/column-types`. +- **`import.ts`'s `coerceValue` is a SECOND write path and is not opt-in.** Importing into a column of your type always hits it, and its `default` arm silently `String(value)`s — so a missing `case` stores text in a column whose `jsonbCast` is numeric, and then every filter and sort on that column errors in Postgres. Add a `case`, even though the switch compiles without one. (It is deliberately separate from the registry's `coerce`: an import wants an unparseable value to survive as its raw string so the row error can name it.) +- **CSV inference** is an ordered heuristic in `import.ts`, deliberately not registry-driven. A new type is not inferred from a CSV unless you extend `inferColumnType` — usually you should not, since inference cannot supply configuration (an option set, a currency code). + +## If your type owns metadata, read this + +Registering the *type* is compiler-enforced. Registering its *metadata* is not, and that is where the remaining manual work lives. A key like `precision` has to be added in each of these, none of which will fail to compile if you forget: + +| Where | What happens if you forget | +|---|---| +| `lib/table/types.ts` `ColumnDefinition` | (this one DOES fail — the ownership loop indexes it) | +| `column-types/types.ts` `TYPE_SPECIFIC_COLUMN_KEYS` | it is never stripped on conversion, and poisons the target type | +| `lib/api/contracts/tables.ts` — the schema slot in all three column schemas, plus `refineColumnOptions` | zod strips it at the boundary; silently never saved | +| `columns/service.ts` `addTableColumn` param type | callers cannot pass it | +| A metadata-only update path (`updateColumnCurrency` is the model) + a branch in both column routes + the copilot tool | changing it on an existing column is a silent 200 no-op | +| `column-config-sidebar.tsx` | no UI to set it | +| `table-grid.tsx` delete-column undo + `use-table-undo.ts` restore | undo silently resets it to the default | + +`normalizeColumn`, `buildConvertedColumn`, and the undo snapshot read `TYPE_SPECIFIC_COLUMN_KEYS` generically, so those three are already zero-edit. + +**Known gap:** the metadata-only update path is ~6 near-identical copies (service + 2 routes + copilot). A `metadataUpdate` descriptor on `ColumnTypeServerDefinition` would collapse them; until that exists, copy `currency`'s. + +## Checklist Before Finishing + +- [ ] Added to the `ColumnType` union in `column-types/types.ts` +- [ ] `column-types/{id}.ts` created, every interface field filled in +- [ ] Registered in **both** `registry.ts` and `registry.server.ts` +- [ ] Icon added, centered on the family's optical center, exported alphabetically +- [ ] `migrateCellsTo` / `migrateCellsFrom` added if the stored bytes change +- [ ] New metadata keys added to `TYPE_SPECIFIC_COLUMN_KEYS` + `FOREIGN_METADATA_VERB` +- [ ] Unit tests for `coerce` / `isCompatibleWith` round-trips, verified to fail without the code +- [ ] Docs row added to `apps/docs/content/docs/en/tables/index.mdx` + +## Final Validation (Required) + +1. **`cd apps/sim && bunx tsc --noEmit -p tsconfig.json`** — must be clean. If any file *outside* `column-types/` errors, that file has a hardcoded type list; fix it to read the registry. +2. **Grep for leaks** — `grep -rnE "(===|!==) '{id}'|case '{id}':" apps/sim --include='*.ts' --include='*.tsx' | grep -v column-types/`. (All three forms: a plain `!==` and a `case` are how half of `currency`'s real branches are written.) Hits are expected; judge each. A hit is fine when it mounts a specific React component or encodes a genuinely one-off behavior (`json`'s mono textarea, `date`'s timezone-aware parsing). A hit is a **leak** when it restates something the registry could answer — an icon, a label, a colour, an operator set, a cast, a coercion. Leaks get a registry field, not a new branch. +3. **Run the suite** — `bunx vitest run lib/table 'app/workspace/[workspaceId]/tables' lib/api app/api/table app/api/v1 lib/copilot/tools/server/table`. Existing tests must pass **unchanged**; needing to edit one means you changed behavior for the other types. +4. **`bun run lint:check`, `bun run check:api-validation`, `bun run check:client-boundary`** from the repo root. +5. **Exercise it in the running app** on a table with one column of every type: create, edit inline / in the expanded popover / in the row modal, paste from a spreadsheet, filter, sort, convert to and from other types, export CSV, undo a column delete. diff --git a/.claude/skills/add-column-type b/.claude/skills/add-column-type new file mode 120000 index 0000000000..9d7b29aa20 --- /dev/null +++ b/.claude/skills/add-column-type @@ -0,0 +1 @@ +../../.agents/skills/add-column-type \ No newline at end of file diff --git a/.cursor/commands/add-column-type.md b/.cursor/commands/add-column-type.md new file mode 100644 index 0000000000..f0be823ab6 --- /dev/null +++ b/.cursor/commands/add-column-type.md @@ -0,0 +1,154 @@ +# Adding a Table Column Type + +A column type is **one file** in `apps/sim/lib/table/column-types/` plus a registry entry. Everything that varies per type — label, icon, storage cast, coercion, validation, conversion compatibility, formatting, editor, filter operators — lives on that one object, so no consumer needs editing. + +This was not always true: adding `currency` originally took ~40 edits across 32 `switch` arms and 26 UI branches, each of which failed **silently** when missed. The registry exists to make that impossible, so the rule is absolute: **if you find yourself adding a `case 'yourtype':` anywhere outside `column-types/`, the registry is missing a field. Add the field instead.** + +## Hard Rule: the compiler tells you what to do + +Do **not** hunt for places to edit. Add your type to the `ColumnType` union first and let `tsc` produce the list: + +```bash +cd apps/sim && bunx tsc --noEmit -p tsconfig.json +``` + +You will get two errors, naming `column-types/registry.ts` and `column-types/registry.server.ts`. Register in both. + +If your type owns metadata, adding its key to `TYPE_SPECIFIC_COLUMN_KEYS` produces two more legitimate errors — `FOREIGN_METADATA_VERB` in `validation.ts` (a `Record` over those keys) and the key's absence from `ColumnDefinition`. Those are the gate working, not sites to "fix". + +Any error beyond those four is a site reading a hardcoded type list that should read the registry — fix that site, don't work around it. + +## Directory Structure + +``` +apps/sim/lib/table/column-types/ +├── types.ts # ColumnTypeDefinition — the contract you implement +├── types.server.ts # ColumnTypeServerDefinition — cell migrations only +├── registry.ts # Record ← client-safe, the gate +├── registry.server.ts # Record ← adds migrations (drizzle) +├── index.ts # barrel + accessors (columnTypeOf, columnTypeById, …) +└── {type}.ts # one file per type — what you write +``` + +## Step 1: Pick the storage shape + +Decide what a cell literally holds in `user_table_rows.data` (JSONB). This drives almost everything else: + +| Storage | `jsonbCast` | Notes | +|---------|-------------|-------| +| number | `'numeric'` | Filters/sorts compare numerically. `currency` does this. | +| ISO string | `'timestamptz'` | `date` does this. | +| string / bool / object | `null` | Text comparison is correct. | + +**Prefer an existing primitive over a new shape.** `currency` stores a plain number and keeps its ISO code as *display metadata* — which is why filtering, sorting, uniqueness, and CSV export all reuse the numeric paths untouched, and why re-denominating a column rewrites zero rows. + +## Step 2: Add the icon + +Create `packages/emcn/src/icons/type-{name}.tsx`, copying the geometry conventions of its siblings exactly: + +```tsx +import type { SVGProps } from 'react' + +/** + * Type {name} icon component - {what the glyph is} for {name} columns + * @param props - SVG properties including className, fill, etc. + */ +export function Type{Pascal}(props: SVGProps) { + return ( + + ) +} +``` + +- `viewBox='-1.75 -1.5 24 24'` is the **`type-*` family** value, not the set-wide default. Match the family. +- Center the glyph on the viewBox's optical center (**y = 10.5**, **x = 10.25**) — every sibling does, and a few tenths off is visible at `size-[14px]`. +- Export alphabetically **by component name** in `packages/emcn/src/icons/index.ts`. + +## Step 3: Write the type file + +`apps/sim/lib/table/column-types/{name}.ts`. Copy the closest existing type and change what differs. Every field is required by the interface, so the compiler enumerates them for you — read the TSDoc in `types.ts` rather than guessing. + +The three that are easy to get wrong: + +- **`coerce`** is the *single* write-path implementation. The server runs it before persisting **and** the grid runs it to fill the optimistic cache. Accept every shape the value legitimately arrives in (paste, CSV, tool write), because rejecting means the cell is nulled. +- **`isCompatibleWith`** gates type conversion and must read the value **exactly as `coerce` will**, or a conversion will pass its check and then null the cell. +- **`ownedMetadata`** lists the `ColumnDefinition` keys your type owns. Anything you add must also be added to `TYPE_SPECIFIC_COLUMN_KEYS` in `types.ts` and given a phrase in `FOREIGN_METADATA_VERB` in `validation.ts` — both are `Record`-typed, so the compiler will tell you. + +## Step 4: Register + +Add the entry to `COLUMN_TYPE_REGISTRY` in `registry.ts` **and** `COLUMN_TYPE_SERVER_REGISTRY` in `registry.server.ts`. + +`COLUMN_TYPES` is declared in `types.ts` (not derived from the registry — the registry is annotated `Record` against it, which is the gate). `constants.ts` re-exports it, so `columnTypeSchema = z.enum(COLUMN_TYPES)` picks your type up with no edit. **Type-specific metadata does not** — see the next step. + +## Step 5: Migrations (only if the stored bytes change) + +If converting an existing column **to** your type must rewrite cells, add `migrateCellsTo` in `registry.server.ts`; if converting **away** must rewrite them, add `migrateCellsFrom`. + +This is load-bearing, not cosmetic: filters and sorts apply `jsonbCast` to whatever is stored, so leaving a non-castable string behind makes **every query on that column fail** — not merely render oddly. + +Prefer set-based SQL. When the transform genuinely needs JS (`currency`'s separator disambiguation), compute the values during the compatibility scan and pass them through `resolved`, then apply them in one batched statement. + +## Naming Convention + +- Type id: lowercase, singular — `currency`, not `Currency` or `currencies` +- File: `column-types/{id}.ts`, export `const {id}ColumnType` +- Icon: `type-{kebab}.tsx`, export `Type{Pascal}` + +## Watch out + +- **Import cycles.** `column-types/select.ts` imports `select-values.ts`, so `select-values.ts` must **not** import the registry — that closes a cycle and fails at module init. Inside a type's own helper module the string literal is the implementation, not a config leak. +- **The client-safe boundary.** `registry.ts` and everything it imports must stay free of `@sim/db`, `drizzle-orm`, and `next/server` — the tables grid imports it directly. A React icon is fine (it's a component *reference*, never called server-side). Only `registry.server.ts` may touch drizzle. +- **Don't re-export the registry from `@/lib/table`.** 44 server modules import that barrel; routing this through it pulls `@sim/emcn/icons` into all of them. Deep-import `@/lib/table/column-types`. +- **`import.ts`'s `coerceValue` is a SECOND write path and is not opt-in.** Importing into a column of your type always hits it, and its `default` arm silently `String(value)`s — so a missing `case` stores text in a column whose `jsonbCast` is numeric, and then every filter and sort on that column errors in Postgres. Add a `case`, even though the switch compiles without one. (It is deliberately separate from the registry's `coerce`: an import wants an unparseable value to survive as its raw string so the row error can name it.) +- **CSV inference** is an ordered heuristic in `import.ts`, deliberately not registry-driven. A new type is not inferred from a CSV unless you extend `inferColumnType` — usually you should not, since inference cannot supply configuration (an option set, a currency code). + +## If your type owns metadata, read this + +Registering the *type* is compiler-enforced. Registering its *metadata* is not, and that is where the remaining manual work lives. A key like `precision` has to be added in each of these, none of which will fail to compile if you forget: + +| Where | What happens if you forget | +|---|---| +| `lib/table/types.ts` `ColumnDefinition` | (this one DOES fail — the ownership loop indexes it) | +| `column-types/types.ts` `TYPE_SPECIFIC_COLUMN_KEYS` | it is never stripped on conversion, and poisons the target type | +| `lib/api/contracts/tables.ts` — the schema slot in all three column schemas, plus `refineColumnOptions` | zod strips it at the boundary; silently never saved | +| `columns/service.ts` `addTableColumn` param type | callers cannot pass it | +| A metadata-only update path (`updateColumnCurrency` is the model) + a branch in both column routes + the copilot tool | changing it on an existing column is a silent 200 no-op | +| `column-config-sidebar.tsx` | no UI to set it | +| `table-grid.tsx` delete-column undo + `use-table-undo.ts` restore | undo silently resets it to the default | + +`normalizeColumn`, `buildConvertedColumn`, and the undo snapshot read `TYPE_SPECIFIC_COLUMN_KEYS` generically, so those three are already zero-edit. + +**Known gap:** the metadata-only update path is ~6 near-identical copies (service + 2 routes + copilot). A `metadataUpdate` descriptor on `ColumnTypeServerDefinition` would collapse them; until that exists, copy `currency`'s. + +## Checklist Before Finishing + +- [ ] Added to the `ColumnType` union in `column-types/types.ts` +- [ ] `column-types/{id}.ts` created, every interface field filled in +- [ ] Registered in **both** `registry.ts` and `registry.server.ts` +- [ ] Icon added, centered on the family's optical center, exported alphabetically +- [ ] `migrateCellsTo` / `migrateCellsFrom` added if the stored bytes change +- [ ] New metadata keys added to `TYPE_SPECIFIC_COLUMN_KEYS` + `FOREIGN_METADATA_VERB` +- [ ] Unit tests for `coerce` / `isCompatibleWith` round-trips, verified to fail without the code +- [ ] Docs row added to `apps/docs/content/docs/en/tables/index.mdx` + +## Final Validation (Required) + +1. **`cd apps/sim && bunx tsc --noEmit -p tsconfig.json`** — must be clean. If any file *outside* `column-types/` errors, that file has a hardcoded type list; fix it to read the registry. +2. **Grep for leaks** — `grep -rnE "(===|!==) '{id}'|case '{id}':" apps/sim --include='*.ts' --include='*.tsx' | grep -v column-types/`. (All three forms: a plain `!==` and a `case` are how half of `currency`'s real branches are written.) Hits are expected; judge each. A hit is fine when it mounts a specific React component or encodes a genuinely one-off behavior (`json`'s mono textarea, `date`'s timezone-aware parsing). A hit is a **leak** when it restates something the registry could answer — an icon, a label, a colour, an operator set, a cast, a coercion. Leaks get a registry field, not a new branch. +3. **Run the suite** — `bunx vitest run lib/table 'app/workspace/[workspaceId]/tables' lib/api app/api/table app/api/v1 lib/copilot/tools/server/table`. Existing tests must pass **unchanged**; needing to edit one means you changed behavior for the other types. +4. **`bun run lint:check`, `bun run check:api-validation`, `bun run check:client-boundary`** from the repo root. +5. **Exercise it in the running app** on a table with one column of every type: create, edit inline / in the expanded popover / in the row modal, paste from a spreadsheet, filter, sort, convert to and from other types, export CSV, undo a column delete. diff --git a/AGENTS.md b/AGENTS.md index 9ce16b909d..f36d633df6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -468,3 +468,9 @@ Two hard rules that the skills assume: For the full authoring instructions — SubBlock property tables, `condition`/`dependsOn`/`required`/`mode`/`canonicalParamId` syntax, required block metadata (`integrationType`, `tags`, `authMode`, `docsLink`, `{Service}BlockMeta`), file-input/`normalizeFileInput` patterns, and checklists — use the skills: `/add-integration` (end-to-end), `/add-tools`, `/add-block`, `/add-trigger`. +## Tables + +Table column types are registry entries in `apps/sim/lib/table/column-types/` — one file per type owning its label, icon, storage cast, coercion, validation, conversion compatibility, formatting, and editor. `Record` on `registry.ts` and `registry.server.ts` is a compile-time completeness gate: adding a type to the union errors until both entries exist. + +Never add a `case 'sometype':` outside `column-types/` — a missing arm fails silently (a wrong `jsonbCast` breaks every filter on the column). If a consumer needs per-type knowledge, add a registry field. Use `/add-column-type` for the full procedure. + diff --git a/CLAUDE.md b/CLAUDE.md index a0632e40ad..d2e376dbd9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -478,3 +478,9 @@ Two hard rules that the skills assume: For the full authoring instructions — SubBlock property tables, `condition`/`dependsOn`/`required`/`mode`/`canonicalParamId` syntax, required block metadata (`integrationType`, `tags`, `authMode`, `docsLink`, `{Service}BlockMeta`), file-input/`normalizeFileInput` patterns, and checklists — use the skills: `/add-integration` (end-to-end), `/add-tools`, `/add-block`, `/add-trigger`. +## Tables + +Table column types are registry entries in `apps/sim/lib/table/column-types/` — one file per type owning its label, icon, storage cast, coercion, validation, conversion compatibility, formatting, and editor. `Record` on `registry.ts` and `registry.server.ts` is a compile-time completeness gate: adding a type to the union errors until both entries exist. + +Never add a `case 'sometype':` outside `column-types/` — a missing arm fails silently (a wrong `jsonbCast` breaks every filter on the column). If a consumer needs per-type knowledge, add a registry field. Use `/add-column-type` for the full procedure. + diff --git a/apps/docs/content/docs/en/integrations/table.mdx b/apps/docs/content/docs/en/integrations/table.mdx index df92f381e0..4c26c6fa71 100644 --- a/apps/docs/content/docs/en/integrations/table.mdx +++ b/apps/docs/content/docs/en/integrations/table.mdx @@ -16,7 +16,7 @@ Tables allow you to create and manage custom data tables directly within Sim. St **Why Use Tables?** - **No external setup**: Create tables instantly without configuring external databases - **Workflow-native**: Data persists across workflow executions and is accessible from any workflow in your workspace -- **Flexible schema**: Define columns with types (string, number, boolean, date, json) and constraints (required, unique) +- **Flexible schema**: Define columns with types (string, number, currency, boolean, date, json, select) and constraints (required, unique) - **Powerful querying**: Filter, sort, and paginate data using MongoDB-style operators - **Agent-friendly**: Tables can be used as tools by AI agents for dynamic data storage and retrieval diff --git a/apps/docs/content/docs/en/tables/index.mdx b/apps/docs/content/docs/en/tables/index.mdx index ae70683af6..913711ecdf 100644 --- a/apps/docs/content/docs/en/tables/index.mdx +++ b/apps/docs/content/docs/en/tables/index.mdx @@ -22,12 +22,16 @@ Every column has a type, which decides how its values are stored and validated. | --- | --- | --- | | **Text** | A free-form string | `"Acme Corp"` | | **Number** | A numeric value | `42` | +| **Currency** | An amount in a currency you pick per column | `$1,234.56` | | **Boolean** | `true` or `false` | `true` | | **Date** | A date | `2026-03-16` | | **JSON** | An object or array | `{ "tier": "pro" }` | +| **Select** | One of a fixed set of options, or several | `Pro` | Types are enforced as you enter values, so a Number column only takes numbers. +A Currency column stores a plain number and renders it in the currency you choose for that column, so filters, sorts, and exports all see the amount itself. Changing a column's currency relabels it — it does not convert the amounts. + ## Editing a table Open the **Tables** section in the sidebar and click **New table** to create one. Add columns from the column header, type into a cell to edit it, and paste rows from a spreadsheet to bulk-load. Filter and sort from the toolbar without changing the underlying data. The editor has full keyboard support; see [keyboard shortcuts](/keyboard-shortcuts). diff --git a/apps/sim/app/api/table/[tableId]/columns/route.test.ts b/apps/sim/app/api/table/[tableId]/columns/route.test.ts new file mode 100644 index 0000000000..4ac282861c --- /dev/null +++ b/apps/sim/app/api/table/[tableId]/columns/route.test.ts @@ -0,0 +1,320 @@ +/** + * @vitest-environment node + * + * The PATCH handler performs several writes, each in its own locked + * transaction, so one that fails leaves the earlier ones committed. Two things + * keep that from producing a partial update the caller cannot see or undo: the + * guards reject the knowable cases before any write, and the rename — the only + * write that is purely cosmetic — goes LAST, so a failed typed write leaves the + * column entirely untouched. These pin both. + */ +import { hybridAuthMockFns } from '@sim/testing' +import { getErrorMessage } from '@sim/utils/errors' +import { NextRequest } from 'next/server' +import { beforeEach, describe, expect, it, vi } from 'vitest' + +const { + mockCheckAccess, + mockRenameColumn, + mockUpdateColumnType, + mockUpdateColumnCurrency, + mockUpdateColumnOptions, + mockUpdateColumnConstraints, + mockAddTableColumn, + mockDeleteColumn, +} = vi.hoisted(() => ({ + mockCheckAccess: vi.fn(), + mockRenameColumn: vi.fn(), + mockUpdateColumnType: vi.fn(), + mockUpdateColumnCurrency: vi.fn(), + mockUpdateColumnOptions: vi.fn(), + mockUpdateColumnConstraints: vi.fn(), + mockAddTableColumn: vi.fn(), + mockDeleteColumn: vi.fn(), +})) + +vi.mock('@/lib/table', () => ({ + addTableColumn: mockAddTableColumn, + deleteColumn: mockDeleteColumn, + renameColumn: mockRenameColumn, + updateColumnConstraints: mockUpdateColumnConstraints, + updateColumnCurrency: mockUpdateColumnCurrency, + updateColumnOptions: mockUpdateColumnOptions, + updateColumnType: mockUpdateColumnType, +})) +vi.mock('@/app/api/table/utils', () => ({ + accessError: () => new Response('denied', { status: 403 }), + checkAccess: mockCheckAccess, + normalizeColumn: (c: unknown) => c, + rootErrorMessage: (e: unknown) => getErrorMessage(e), + tableLockErrorResponse: () => null, +})) + +import { PATCH } from '@/app/api/table/[tableId]/columns/route' + +const WORKSPACE_ID = '11111111-1111-4111-8111-111111111111' + +function patch(updates: Record) { + return PATCH( + new NextRequest('http://localhost/api/table/t1/columns', { + method: 'PATCH', + body: JSON.stringify({ workspaceId: WORKSPACE_ID, columnName: 'amount', updates }), + headers: { 'content-type': 'application/json' }, + }), + { params: Promise.resolve({ tableId: 't1' }) } + ) +} + +describe('PATCH /api/table/[tableId]/columns — pre-flight guards', () => { + beforeEach(() => { + vi.clearAllMocks() + hybridAuthMockFns.mockCheckSessionOrInternalAuth.mockResolvedValue({ + success: true, + userId: 'user-1', + authType: 'session', + }) + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { columns: [{ id: 'col_a', name: 'amount', type: 'number' }] }, + }, + }) + mockRenameColumn.mockResolvedValue({ schema: { columns: [] } }) + }) + + it('rejects a currency code on a non-currency column without renaming first', async () => { + const response = await patch({ name: 'renamed', currencyCode: 'USD' }) + + expect(response.status).toBe(400) + expect(await response.json()).toMatchObject({ + error: expect.stringContaining('Cannot set currency'), + }) + // The whole point: the rename must not have been committed. + expect(mockRenameColumn).not.toHaveBeenCalled() + expect(mockUpdateColumnCurrency).not.toHaveBeenCalled() + }) + + it('rejects an unsupported currency code without renaming first', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { columns: [{ id: 'col_a', name: 'amount', type: 'currency' }] }, + }, + }) + + const response = await patch({ name: 'renamed', currencyCode: 'ZZZ' }) + + expect(response.status).toBe(400) + expect(await response.json()).toMatchObject({ + error: expect.stringContaining('Invalid currency code'), + }) + expect(mockRenameColumn).not.toHaveBeenCalled() + }) + + it('still applies a rename when the currency it rides on is unchanged', async () => { + // `updateColumnCurrency` no-ops on an unchanged code. The rename folded into + // the same request must not be dropped with it. + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { + columns: [{ id: 'col_a', name: 'amount', type: 'currency', currencyCode: 'USD' }], + }, + }, + }) + mockUpdateColumnCurrency.mockResolvedValue({ schema: { columns: [] } }) + + const response = await patch({ name: 'renamed', currencyCode: 'USD' }) + + expect(response.status).toBe(200) + expect(mockUpdateColumnCurrency).toHaveBeenCalledWith( + expect.objectContaining({ currencyCode: 'USD', newName: 'renamed' }), + expect.any(String) + ) + }) + + it('renames standalone when there is no other write to ride on', async () => { + mockRenameColumn.mockResolvedValue({ schema: { columns: [] } }) + + const response = await patch({ name: 'renamed' }) + + expect(response.status).toBe(200) + expect(mockRenameColumn).toHaveBeenCalledWith( + expect.objectContaining({ oldName: 'col_a', newName: 'renamed' }), + expect.any(String) + ) + }) + + it('leaves the column untouched when a typed write fails', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { columns: [{ id: 'col_a', name: 'amount', type: 'currency' }] }, + }, + }) + // Stands in for the race the guards cannot close: the column stopped being + // a currency between the snapshot the guards read and this write. + mockUpdateColumnCurrency.mockRejectedValue( + new Error('Cannot set currency on column "amount" of type "string"') + ) + + const response = await patch({ name: 'renamed', currencyCode: 'USD' }) + + expect(response.status).toBe(400) + expect(mockRenameColumn).not.toHaveBeenCalled() + }) + + it('rejects a name already taken before any write runs', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { + columns: [ + { id: 'col_a', name: 'amount', type: 'currency' }, + { id: 'col_b', name: 'taken', type: 'string' }, + ], + }, + }, + }) + + const response = await patch({ name: 'taken', currencyCode: 'EUR' }) + + expect(response.status).toBe(400) + expect(await response.json()).toMatchObject({ + error: expect.stringContaining('already exists'), + }) + // The typed write would otherwise have committed under a rename that fails. + expect(mockUpdateColumnCurrency).not.toHaveBeenCalled() + expect(mockRenameColumn).not.toHaveBeenCalled() + }) + + it('forwards unique to the retype so it validates post-conversion values', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { columns: [{ id: 'col_a', name: 'amount', type: 'string' }] }, + }, + }) + mockUpdateColumnType.mockResolvedValue({ schema: { columns: [] } }) + mockUpdateColumnConstraints.mockResolvedValue({ schema: { columns: [] } }) + + const response = await patch({ type: 'number', unique: true }) + + expect(response.status).toBe(200) + // The conversion itself can manufacture duplicates ("5" and "5.0" both + // coerce to 5), so the retype has to see `unique` — discovering it in the + // separate constraint write would report an error with the conversion + // already committed and the original text irrecoverably rewritten. + expect(mockUpdateColumnType).toHaveBeenCalledWith( + expect.objectContaining({ newType: 'number', unique: true }), + expect.any(String) + ) + }) + + it('applies a rename, retype and constraints in a single write', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { columns: [{ id: 'col_a', name: 'amount', type: 'string' }] }, + }, + }) + mockUpdateColumnType.mockResolvedValue({ schema: { columns: [] } }) + + const response = await patch({ name: 'total', type: 'number', required: true, unique: true }) + + expect(response.status).toBe(200) + // One transaction for the whole request: no separate rename, no separate + // constraint write, so no half of it can commit without the others. + expect(mockRenameColumn).not.toHaveBeenCalled() + expect(mockUpdateColumnConstraints).not.toHaveBeenCalled() + expect(mockUpdateColumnType).toHaveBeenCalledTimes(1) + expect(mockUpdateColumnType).toHaveBeenCalledWith( + expect.objectContaining({ + newType: 'number', + required: true, + unique: true, + newName: 'total', + }), + expect.any(String) + ) + }) + + it('still runs the constraint write when the type is unchanged', async () => { + mockUpdateColumnConstraints.mockResolvedValue({ schema: { columns: [] } }) + + const response = await patch({ required: true }) + + expect(response.status).toBe(200) + expect(mockUpdateColumnConstraints).toHaveBeenCalledTimes(1) + expect(mockUpdateColumnType).not.toHaveBeenCalled() + }) + + it('rejects constraint changes on a workflow-output column before any write', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { + columns: [{ id: 'col_a', name: 'amount', type: 'number', workflowGroupId: 'g1' }], + }, + }, + }) + + const response = await patch({ type: 'string', required: true }) + + expect(response.status).toBe(400) + expect(await response.json()).toMatchObject({ + error: expect.stringContaining('workflow-output column'), + }) + expect(mockUpdateColumnType).not.toHaveBeenCalled() + }) + + it('rejects unique on a type that cannot carry it without renaming first', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { + columns: [ + { id: 'col_a', name: 'amount', type: 'select', options: [{ id: 'o', name: 'O' }] }, + ], + }, + }, + }) + + const response = await patch({ name: 'renamed', unique: true }) + + expect(response.status).toBe(400) + expect(mockRenameColumn).not.toHaveBeenCalled() + }) + + it('folds a rename into the typed write instead of running it separately', async () => { + mockCheckAccess.mockResolvedValue({ + ok: true, + table: { + workspaceId: WORKSPACE_ID, + schema: { columns: [{ id: 'col_a', name: 'amount', type: 'currency' }] }, + }, + }) + mockUpdateColumnCurrency.mockResolvedValue({ schema: { columns: [] } }) + + const response = await patch({ name: 'renamed', currencyCode: 'eur' }) + + expect(response.status).toBe(200) + // One transaction, not two: the rename rides along with the currency write, + // so neither half can commit without the other. + expect(mockRenameColumn).not.toHaveBeenCalled() + expect(mockUpdateColumnCurrency).toHaveBeenCalledWith( + // Addressed by stable id; the contract upper-cases the code on the way in. + expect.objectContaining({ columnName: 'col_a', currencyCode: 'EUR', newName: 'renamed' }), + expect.any(String) + ) + }) +}) diff --git a/apps/sim/app/api/table/[tableId]/columns/route.ts b/apps/sim/app/api/table/[tableId]/columns/route.ts index 227c1422b0..19289a0198 100644 --- a/apps/sim/app/api/table/[tableId]/columns/route.ts +++ b/apps/sim/app/api/table/[tableId]/columns/route.ts @@ -15,10 +15,13 @@ import { deleteColumn, renameColumn, updateColumnConstraints, + updateColumnCurrency, updateColumnOptions, updateColumnType, } from '@/lib/table' -import { columnMatchesRef } from '@/lib/table/column-keys' +import { columnMatchesRef, getColumnId } from '@/lib/table/column-keys' +import { columnTypeById } from '@/lib/table/column-types' +import { isSupportedCurrencyCode } from '@/lib/table/currency' import { accessError, checkAccess, @@ -120,13 +123,6 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu const { updates } = validated let updatedTable = null - if (updates.name) { - updatedTable = await renameColumn( - { tableId, oldName: validated.columnName, newName: updates.name }, - requestId - ) - } - // A payload that repeats the current type must not go through // `updateColumnType` — it early-returns on an unchanged type and would drop // any `options` alongside it. Only a real type change routes there; an @@ -134,29 +130,121 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu const currentColumn = table.schema.columns.find((c) => columnMatchesRef(c, validated.columnName) ) + // Address every write below by the stable id, not the name: a rename folded + // into one of them must not break the next one's lookup. + const columnRef = currentColumn ? getColumnId(currentColumn) : validated.columnName + // The constraints write below is a separate, unconditional step, so it is + // the last one whenever it runs — that is the write the rename rides on. const typeChanging = updates.type !== undefined && updates.type !== currentColumn?.type + if (!currentColumn) { + return NextResponse.json( + { error: `Column "${validated.columnName}" not found` }, + { status: 404 } + ) + } - // Every write below is its own locked transaction, so any of them paired - // with a constraint write that is going to fail commits and then errors. + // A retype applies and validates the constraints itself, so the separate + // constraint write only runs when the type is unchanged. The rename rides + // whichever write actually runs last. + const typedWriteRuns = + typeChanging || + updates.currencyCode !== undefined || + updates.options !== undefined || + updates.multiple !== undefined + const constraintsWriteRuns = + !typedWriteRuns && (updates.required !== undefined || updates.unique !== undefined) + const renameWithTypedWrite = + updates.name && !constraintsWriteRuns ? { newName: updates.name } : {} + + // Every write below is its own locked transaction, so one that is going to + // fail leaves the earlier ones committed. These guards reject the knowable + // cases up front, before any write at all. // Gate on the type the column ENDS UP with, not on whether the type is // changing: an options-only update on an existing select column carries the // same hazard as a conversion does. const resultingType = updates.type ?? currentColumn?.type - if (updates.unique === true && resultingType === 'select') { - return NextResponse.json({ error: 'Cannot set a select column as unique' }, { status: 400 }) + if (updates.currencyCode !== undefined) { + if (resultingType !== 'currency') { + return NextResponse.json( + { + error: `Cannot set currency on column "${validated.columnName}" of type "${resultingType}"`, + }, + { status: 400 } + ) + } + if (!isSupportedCurrencyCode(updates.currencyCode)) { + return NextResponse.json( + { + error: `Invalid currency code "${updates.currencyCode}". Use an ISO 4217 code, e.g. USD`, + }, + { status: 400 } + ) + } + } + // The rename runs last (see below), so a name already taken would fail after + // the typed write committed. This is the only rename failure a caller can + // cause; catching it here leaves just the concurrent-collision race, which + // no pre-flight check can close. + if ( + updates.name && + table.schema.columns.some( + (c) => + c.name.toLowerCase() === updates.name?.toLowerCase() && + !columnMatchesRef(c, validated.columnName) + ) + ) { + return NextResponse.json( + { error: `Column "${updates.name}" already exists` }, + { status: 400 } + ) + } + if ( + currentColumn?.workflowGroupId && + (updates.required !== undefined || updates.unique !== undefined) + ) { + return NextResponse.json( + { + error: `Cannot change constraints on workflow-output column "${currentColumn.name}". Constraints aren't applicable to columns whose values come from workflow execution.`, + }, + { status: 400 } + ) + } + if (updates.unique === true && !columnTypeById(resultingType).supportsUnique) { + return NextResponse.json( + { error: `Cannot set a ${resultingType} column as unique` }, + { status: 400 } + ) } if (typeChanging) { updatedTable = await updateColumnType( { tableId, - columnName: updates.name ?? validated.columnName, + columnName: columnRef, newType: updates.type as NonNullable, ...(updates.options !== undefined ? { options: updates.options } : {}), ...(updates.multiple !== undefined ? { multiple: updates.multiple } : {}), + ...(updates.currencyCode !== undefined ? { currencyCode: updates.currencyCode } : {}), // Forwarded so the conversion validates against the constraint this // same request is about to set, not the column's current one. ...(updates.required !== undefined ? { required: updates.required } : {}), + ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...renameWithTypedWrite, + }, + requestId + ) + } else if (updates.currencyCode !== undefined) { + // Re-denominating an existing currency column: schema-only, no cell + // rewrite. Reached only when the type is unchanged — a conversion INTO + // currency carries the code through `updateColumnType` above. + updatedTable = await updateColumnCurrency( + { + tableId, + columnName: columnRef, + currencyCode: updates.currencyCode, + ...(updates.required !== undefined ? { required: updates.required } : {}), + ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...renameWithTypedWrite, }, requestId ) @@ -164,29 +252,46 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu updatedTable = await updateColumnOptions( { tableId, - columnName: updates.name ?? validated.columnName, + columnName: columnRef, options: updates.options ?? currentColumn?.options ?? [], ...(updates.multiple !== undefined ? { multiple: updates.multiple } : {}), // Forwarded so the removal guard validates against the constraint this // same request is about to set, not the column's current one. ...(updates.required !== undefined ? { required: updates.required } : {}), + ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...renameWithTypedWrite, }, requestId ) } - if (updates.required !== undefined || updates.unique !== undefined) { + // Skipped whenever a typed write ran: that write already applied and + // validated these, in one transaction with the change they accompany. + if (constraintsWriteRuns) { updatedTable = await updateColumnConstraints( { tableId, - columnName: updates.name ?? validated.columnName, + columnName: columnRef, ...(updates.required !== undefined ? { required: updates.required } : {}), ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...(updates.name ? { newName: updates.name } : {}), }, requestId ) } + // A rename rides along with the LAST write above, inside that write's + // transaction — a rename is metadata-only (rows key on the stable column + // id), so nothing forces it to be its own write, and folding it in is what + // stops a combined request from committing one half and then failing. Only + // a rename with nothing to ride on runs standalone. + if (updates.name && !updatedTable) { + updatedTable = await renameColumn( + { tableId, oldName: columnRef, newName: updates.name }, + requestId + ) + } + if (!updatedTable) { return NextResponse.json({ error: 'No updates specified' }, { status: 400 }) } @@ -217,7 +322,8 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu msg.includes('exceeds maximum') || msg.includes('incompatible') || msg.includes('duplicate') || - msg.includes('option') + msg.includes('option') || + msg.includes('currency') ) { return NextResponse.json({ error: msg }, { status: 400 }) } diff --git a/apps/sim/app/api/table/utils.ts b/apps/sim/app/api/table/utils.ts index 835866d829..ceb399556c 100644 --- a/apps/sim/app/api/table/utils.ts +++ b/apps/sim/app/api/table/utils.ts @@ -11,6 +11,7 @@ import { isFeatureEnabled } from '@/lib/core/config/feature-flags' import type { MultipartError } from '@/lib/core/utils/multipart' import type { ColumnDefinition, Filter, TableDefinition, TablePredicate } from '@/lib/table' import { buildFilterClause, getTableById, TableQueryValidationError } from '@/lib/table' +import { typeMetadataOf } from '@/lib/table/column-types' import { USER_TABLE_ROWS_SQL_NAME } from '@/lib/table/constants' import { TableLockedError } from '@/lib/table/mutation-locks' import { isTablePredicate } from '@/lib/table/query-builder/converters' @@ -339,7 +340,8 @@ export function normalizeColumn(col: ColumnDefinition): ColumnDefinition { required: col.required ?? false, unique: col.unique ?? false, ...(col.workflowGroupId ? { workflowGroupId: col.workflowGroupId } : {}), - ...(col.options ? { options: col.options } : {}), - ...(col.multiple ? { multiple: true } : {}), + // Type-specific metadata is forwarded generically: naming keys here meant a + // new type's metadata was stored server-side but silently never returned. + ...typeMetadataOf(col), } } diff --git a/apps/sim/app/api/v1/tables/[tableId]/columns/route.ts b/apps/sim/app/api/v1/tables/[tableId]/columns/route.ts index f1751ee212..56caed8353 100644 --- a/apps/sim/app/api/v1/tables/[tableId]/columns/route.ts +++ b/apps/sim/app/api/v1/tables/[tableId]/columns/route.ts @@ -14,10 +14,13 @@ import { deleteColumn, renameColumn, updateColumnConstraints, + updateColumnCurrency, updateColumnOptions, updateColumnType, } from '@/lib/table' -import { columnMatchesRef } from '@/lib/table/column-keys' +import { columnMatchesRef, getColumnId } from '@/lib/table/column-keys' +import { columnTypeById } from '@/lib/table/column-types' +import { isSupportedCurrencyCode } from '@/lib/table/currency' import { accessError, checkAccess, @@ -154,13 +157,6 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu const { updates } = validated let updatedTable = null - if (updates.name) { - updatedTable = await renameColumn( - { tableId, oldName: validated.columnName, newName: updates.name }, - requestId - ) - } - // A payload that repeats the current type must not go through // `updateColumnType` — it early-returns on an unchanged type and would drop // any `options` alongside it. Only a real type change routes there; an @@ -168,29 +164,121 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu const currentColumn = table.schema.columns.find((c) => columnMatchesRef(c, validated.columnName) ) + // Address every write below by the stable id, not the name: a rename folded + // into one of them must not break the next one's lookup. + const columnRef = currentColumn ? getColumnId(currentColumn) : validated.columnName + // The constraints write below is a separate, unconditional step, so it is + // the last one whenever it runs — that is the write the rename rides on. const typeChanging = updates.type !== undefined && updates.type !== currentColumn?.type + if (!currentColumn) { + return NextResponse.json( + { error: `Column "${validated.columnName}" not found` }, + { status: 404 } + ) + } - // Every write below is its own locked transaction, so any of them paired - // with a constraint write that is going to fail commits and then errors. + // A retype applies and validates the constraints itself, so the separate + // constraint write only runs when the type is unchanged. The rename rides + // whichever write actually runs last. + const typedWriteRuns = + typeChanging || + updates.currencyCode !== undefined || + updates.options !== undefined || + updates.multiple !== undefined + const constraintsWriteRuns = + !typedWriteRuns && (updates.required !== undefined || updates.unique !== undefined) + const renameWithTypedWrite = + updates.name && !constraintsWriteRuns ? { newName: updates.name } : {} + + // Every write below is its own locked transaction, so one that is going to + // fail leaves the earlier ones committed. These guards reject the knowable + // cases up front, before any write at all. // Gate on the type the column ENDS UP with, not on whether the type is // changing: an options-only update on an existing select column carries the // same hazard as a conversion does. const resultingType = updates.type ?? currentColumn?.type - if (updates.unique === true && resultingType === 'select') { - return NextResponse.json({ error: 'Cannot set a select column as unique' }, { status: 400 }) + if (updates.currencyCode !== undefined) { + if (resultingType !== 'currency') { + return NextResponse.json( + { + error: `Cannot set currency on column "${validated.columnName}" of type "${resultingType}"`, + }, + { status: 400 } + ) + } + if (!isSupportedCurrencyCode(updates.currencyCode)) { + return NextResponse.json( + { + error: `Invalid currency code "${updates.currencyCode}". Use an ISO 4217 code, e.g. USD`, + }, + { status: 400 } + ) + } + } + // The rename runs last (see below), so a name already taken would fail after + // the typed write committed. This is the only rename failure a caller can + // cause; catching it here leaves just the concurrent-collision race, which + // no pre-flight check can close. + if ( + updates.name && + table.schema.columns.some( + (c) => + c.name.toLowerCase() === updates.name?.toLowerCase() && + !columnMatchesRef(c, validated.columnName) + ) + ) { + return NextResponse.json( + { error: `Column "${updates.name}" already exists` }, + { status: 400 } + ) + } + if ( + currentColumn?.workflowGroupId && + (updates.required !== undefined || updates.unique !== undefined) + ) { + return NextResponse.json( + { + error: `Cannot change constraints on workflow-output column "${currentColumn.name}". Constraints aren't applicable to columns whose values come from workflow execution.`, + }, + { status: 400 } + ) + } + if (updates.unique === true && !columnTypeById(resultingType).supportsUnique) { + return NextResponse.json( + { error: `Cannot set a ${resultingType} column as unique` }, + { status: 400 } + ) } if (typeChanging) { updatedTable = await updateColumnType( { tableId, - columnName: updates.name ?? validated.columnName, + columnName: columnRef, newType: updates.type as NonNullable, ...(updates.options !== undefined ? { options: updates.options } : {}), ...(updates.multiple !== undefined ? { multiple: updates.multiple } : {}), + ...(updates.currencyCode !== undefined ? { currencyCode: updates.currencyCode } : {}), // Forwarded so the conversion validates against the constraint this // same request is about to set, not the column's current one. ...(updates.required !== undefined ? { required: updates.required } : {}), + ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...renameWithTypedWrite, + }, + requestId + ) + } else if (updates.currencyCode !== undefined) { + // Re-denominating an existing currency column: schema-only, no cell + // rewrite. Reached only when the type is unchanged — a conversion INTO + // currency carries the code through `updateColumnType` above. + updatedTable = await updateColumnCurrency( + { + tableId, + columnName: columnRef, + currencyCode: updates.currencyCode, + ...(updates.required !== undefined ? { required: updates.required } : {}), + ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...renameWithTypedWrite, }, requestId ) @@ -198,29 +286,46 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu updatedTable = await updateColumnOptions( { tableId, - columnName: updates.name ?? validated.columnName, + columnName: columnRef, options: updates.options ?? currentColumn?.options ?? [], ...(updates.multiple !== undefined ? { multiple: updates.multiple } : {}), // Forwarded so the removal guard validates against the constraint this // same request is about to set, not the column's current one. ...(updates.required !== undefined ? { required: updates.required } : {}), + ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...renameWithTypedWrite, }, requestId ) } - if (updates.required !== undefined || updates.unique !== undefined) { + // Skipped whenever a typed write ran: that write already applied and + // validated these, in one transaction with the change they accompany. + if (constraintsWriteRuns) { updatedTable = await updateColumnConstraints( { tableId, - columnName: updates.name ?? validated.columnName, + columnName: columnRef, ...(updates.required !== undefined ? { required: updates.required } : {}), ...(updates.unique !== undefined ? { unique: updates.unique } : {}), + ...(updates.name ? { newName: updates.name } : {}), }, requestId ) } + // A rename rides along with the LAST write above, inside that write's + // transaction — a rename is metadata-only (rows key on the stable column + // id), so nothing forces it to be its own write, and folding it in is what + // stops a combined request from committing one half and then failing. Only + // a rename with nothing to ride on runs standalone. + if (updates.name && !updatedTable) { + updatedTable = await renameColumn( + { tableId, oldName: columnRef, newName: updates.name }, + requestId + ) + } + if (!updatedTable) { return NextResponse.json({ error: 'No updates specified' }, { status: 400 }) } @@ -262,7 +367,8 @@ export const PATCH = withRouteHandler(async (request: NextRequest, context: Colu msg.includes('exceeds maximum') || msg.includes('incompatible') || msg.includes('duplicate') || - msg.includes('option') + msg.includes('option') || + msg.includes('currency') ) { return NextResponse.json({ error: msg }, { status: 400 }) } diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx index 2c0a9cffda..af2a0c797c 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-config-sidebar.tsx @@ -6,6 +6,11 @@ import { X } from '@sim/emcn/icons' import { toError } from '@sim/utils/errors' import { findValidationIssue, isValidationError } from '@/lib/api/client/errors' import type { ColumnDefinition, SelectOption } from '@/lib/table' +import { + DEFAULT_CURRENCY_CODE, + getCurrencyOptions, + resolveCurrencyCode, +} from '@/lib/table/currency' import { FieldError, RequiredLabel, @@ -19,6 +24,15 @@ function isSelectType(type: ColumnDefinition['type']): boolean { return type === 'select' } +/** + * Picker entries, built once at module load: the option list is derived from the + * runtime's currency data and never varies per column. + */ +const CURRENCY_COMBOBOX_OPTIONS = getCurrencyOptions().map((c) => ({ + value: c.code, + label: `${c.code} · ${c.name}`, +})) + function optionsEqual(a: SelectOption[], b: SelectOption[]): boolean { return JSON.stringify(a) === JSON.stringify(b) } @@ -110,6 +124,11 @@ function ColumnConfigBody({ const [multipleInput, setMultipleInput] = useState(() => config.mode === 'edit' ? !!existingColumn?.multiple : false ) + const [currencyInput, setCurrencyInput] = useState(() => + config.mode === 'edit' + ? resolveCurrencyCode(existingColumn?.currencyCode) + : DEFAULT_CURRENCY_CODE + ) const [showValidation, setShowValidation] = useState(false) const [nameError, setNameError] = useState(null) const [optionsError, setOptionsError] = useState(null) @@ -117,6 +136,7 @@ function ColumnConfigBody({ const saveDisabled = updateColumn.isPending || addColumn.isPending const trimmedName = nameInput.trim() const wantsOptions = isSelectType(typeInput) + const wantsCurrency = typeInput === 'currency' const trimmedOptions = optionsInput.map((o) => ({ ...o, name: o.name.trim() })) /** Client-side option validation mirroring the server rules; returns an error message or null. */ @@ -150,6 +170,7 @@ function ColumnConfigBody({ ...(!wantsOptions && uniqueInput ? { unique: true } : {}), ...(wantsOptions ? { options: trimmedOptions } : {}), ...(wantsOptions && multipleInput ? { multiple: true } : {}), + ...(wantsCurrency ? { currencyCode: currencyInput } : {}), }) toast.success(`Added "${trimmedName}"`) onClose() @@ -168,6 +189,8 @@ function ColumnConfigBody({ const optionsChanged = wantsOptions && !optionsEqual(existingColumn?.options ?? [], trimmedOptions) const multipleChanged = wantsOptions && !!existingColumn?.multiple !== multipleInput + const currencyChanged = + wantsCurrency && resolveCurrencyCode(existingColumn?.currencyCode) !== currencyInput const updates: { name?: string @@ -175,6 +198,7 @@ function ColumnConfigBody({ unique?: boolean options?: SelectOption[] multiple?: boolean + currencyCode?: string } = { ...(renamed ? { name: trimmedName } : {}), ...(typeChanged ? { type: typeInput } : {}), @@ -182,6 +206,9 @@ function ColumnConfigBody({ ...(uniqueCleared ? { unique: false } : {}), ...(wantsOptions && (typeChanged || optionsChanged) ? { options: trimmedOptions } : {}), ...(wantsOptions && (typeChanged || multipleChanged) ? { multiple: multipleInput } : {}), + ...(wantsCurrency && (typeChanged || currencyChanged) + ? { currencyCode: currencyInput } + : {}), } if (Object.keys(updates).length === 0) { onClose() @@ -261,6 +288,24 @@ function ColumnConfigBody({ )} + {wantsCurrency && ( + <> + +
+ Currency + +
+ + )} + {wantsOptions && ( <> diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-types.ts b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-types.ts index 6ca8023e80..2f235137f1 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-types.ts +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/column-config-sidebar/column-types.ts @@ -1,14 +1,7 @@ import type React from 'react' -import { - Calendar as CalendarIcon, - PlayOutline, - TagIcon, - TypeBoolean, - TypeJson, - TypeNumber, - TypeText, -} from '@sim/emcn/icons' +import { PlayOutline } from '@sim/emcn/icons' import type { ColumnDefinition } from '@/lib/table' +import { ALL_COLUMN_TYPES } from '@/lib/table/column-types' /** * UI-only column type. `'workflow'` is the virtual entry users pick from the @@ -23,13 +16,17 @@ export interface ColumnTypeOption { icon: React.ComponentType<{ className?: string }> } +/** + * Real column types come from the registry — adding one there makes it appear + * in every picker automatically. `workflow` is appended because it is a UI + * affordance, not a storable type. + */ export const COLUMN_TYPE_OPTIONS: ColumnTypeOption[] = [ - { type: 'string', label: 'Text', icon: TypeText }, - { type: 'number', label: 'Number', icon: TypeNumber }, - { type: 'boolean', label: 'Boolean', icon: TypeBoolean }, - { type: 'date', label: 'Date', icon: CalendarIcon }, - { type: 'json', label: 'JSON', icon: TypeJson }, - { type: 'select', label: 'Select', icon: TagIcon }, + ...ALL_COLUMN_TYPES.map((definition) => ({ + type: definition.id, + label: definition.label, + icon: definition.icon, + })), { type: 'workflow', label: 'Workflow', icon: PlayOutline }, ] diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx index efee32d04e..092e7046e7 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/row-modal/row-modal.tsx @@ -18,6 +18,8 @@ import { createLogger } from '@sim/logger' import { getErrorMessage } from '@sim/utils/errors' import { useParams } from 'next/navigation' import type { ColumnDefinition, TableInfo, TableRow } from '@/lib/table' +import { columnTypeOf } from '@/lib/table/column-types' +import { resolveCurrencyCode } from '@/lib/table/currency' import { useTimezone } from '@/hooks/queries/general-settings' import { useDeleteTableRow, useDeleteTableRows, useUpdateTableRow } from '@/hooks/queries/tables' import { @@ -209,9 +211,16 @@ function ColumnField({ column, value, onChange }: ColumnFieldProps) { )} ) - const hint = `Type: ${column.type}${column.required ? '' : ' (optional)'}` + // Currency names its code — the modal edits the bare amount, so without it + // there is nothing on screen saying which currency the number is in. + const typeLabel = + column.type === 'currency' + ? `currency (${resolveCurrencyCode(column.currencyCode)})` + : column.type + const hint = `Type: ${typeLabel}${column.required ? '' : ' (optional)'}` + const definition = columnTypeOf(column) - if (column.type === 'boolean') { + if (definition.editor === 'toggle') { return (
@@ -231,6 +240,9 @@ function ColumnField({ column, value, onChange }: ColumnFieldProps) { ) } + // The one type wanting a mono multi-line field; `editor: 'text'` covers both + // this and a plain input, so it stays explicit rather than inventing a field + // only one type would ever set. if (column.type === 'json') { return ( @@ -276,7 +288,7 @@ function ColumnField({ column, value, onChange }: ColumnFieldProps) { ) } - if (column.type === 'select') { + if (definition.editor === 'select') { return ( @@ -290,7 +302,12 @@ function ColumnField({ column, value, onChange }: ColumnFieldProps) { title={title} required={column.required} hint={hint} - inputType={column.type === 'number' ? 'number' : 'text'} + // A native number input rejects the formatted amounts this type's parser + // exists to accept, so those types take a text field — the same shape the + // grid's inline editor uses. + inputType={ + definition.inputMode === 'decimal' && !definition.acceptsFormattedInput ? 'number' : 'text' + } value={formatValueForInput(value, column.type)} onChange={onChange} placeholder={`Enter ${column.name}`} diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/cell-render.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/cell-render.tsx index 283dfa21d8..4e16d03912 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/cell-render.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/cell-render.tsx @@ -6,6 +6,7 @@ import { Badge, Checkbox, cn, Tooltip } from '@sim/emcn' import { parse } from 'tldts' import { faviconUrl } from '@/lib/core/utils/favicon' import type { RowExecutionMetadata, SelectOption } from '@/lib/table' +import { columnTypeOf } from '@/lib/table/column-types' import { StatusBadge } from '@/app/workspace/[workspaceId]/logs/utils' import { storageToDisplay } from '../../../utils' import { resolveSelectOptions, SelectPill } from '../../select-field' @@ -128,6 +129,13 @@ export function resolveCellRender({ return { kind: 'select', options: resolveSelectOptions(column, value) } } if (isNull) return { kind: 'empty' } + // Formatted here rather than in a render branch because the symbol and + // fraction digits come from the COLUMN's currency, which the render switch + // (keyed on kind alone) no longer has. Renders as plain text — a currency + // cell is a number cell with a symbol, so it stays left-aligned like one. + if (column.type === 'currency') { + return { kind: 'text', text: columnTypeOf(column).formatForDisplay(value, column) } + } if (column.type === 'json') return { kind: 'json', text: JSON.stringify(value) } if (column.type === 'date') return { kind: 'date', text: String(value) } if (column.type === 'string') { diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/expanded-cell-popover.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/expanded-cell-popover.tsx index 84a61ebab3..9d1d1208e1 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/expanded-cell-popover.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/expanded-cell-popover.tsx @@ -4,6 +4,7 @@ import type React from 'react' import { useEffect, useEffectEvent, useLayoutEffect, useMemo, useRef, useState } from 'react' import { Button } from '@sim/emcn' import type { TableRow as TableRowType } from '@/lib/table' +import { columnTypeOf } from '@/lib/table/column-types' import { useTimezone } from '@/hooks/queries/general-settings' import type { EditingCell, SaveReason } from '../../../types' import { @@ -58,11 +59,11 @@ export function ExpandedCellPopover({ return { row, column, colIndex, value: row.data[column.key] } }, [expandedCell, rows, columns]) - const isBooleanCell = target?.column.type === 'boolean' + const isTogglingCell = target ? columnTypeOf(target.column).editor === 'toggle' : false // Workflow-output cells are editable in the expanded view too — the user // can override the workflow's value. Booleans toggle inline; the expanded // popover only handles text-shaped inputs. - const isEditable = Boolean(target) && canEdit && !isBooleanCell + const isEditable = Boolean(target) && canEdit && !isTogglingCell const displayText = useMemo(() => { if (!target) return '' @@ -74,6 +75,9 @@ export function ExpandedCellPopover({ if (target.column.type === 'date' && typeof value === 'string') { return storageToDisplay(value, { seconds: true }) } + if (target.column.type === 'currency') { + return columnTypeOf(target.column).formatForDisplay(value, target.column) + } if (typeof value === 'string') return value return JSON.stringify(value, null, 2) }, [target]) @@ -231,14 +235,12 @@ function ExpandedCellEditor({ setParseError('Invalid JSON') return } - /** `cleanCellValue` nulls unparseable dates/numbers instead of throwing — reject rather than silently clear. */ - if ( - cleaned === null && - draftValue.trim() !== '' && - (column.type === 'date' || column.type === 'number') - ) { - setParseError(column.type === 'date' ? 'Invalid date' : 'Invalid number') - return + if (cleaned === null && draftValue.trim() !== '') { + const message = columnTypeOf(column).parseErrorMessage + if (message) { + setParseError(message) + return + } } onSave(rowId, column.key, cleaned, 'blur') onClose() diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx index e3caa259fe..b93cea863d 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/cells/inline-editors.tsx @@ -15,6 +15,7 @@ import { } from '@sim/emcn' import { Check } from '@sim/emcn/icons' import type { ColumnDefinition } from '@/lib/table' +import { columnTypeOf } from '@/lib/table/column-types' import { isCalendarDateString } from '@/lib/table/dates' import { useTimezone } from '@/hooks/queries/general-settings' import type { SaveReason } from '../../../types' @@ -242,7 +243,7 @@ function InlineDateEditor({ ) } -/** Inline editor for `string`/`number`/`json` columns — single-line text input. Number columns use `type="number"` so the browser rejects non-numeric input. */ +/** Inline editor for `string`/`number`/`currency`/`json` columns — single-line text input. Numeric columns get a decimal keypad and reject a draft that cannot be parsed. */ function InlineTextEditor({ value, column, @@ -291,8 +292,11 @@ function InlineTextEditor({ rejectDraft('Invalid JSON', reason) return } - if (column.type === 'number' && cleaned === null && draft.trim() !== '') { - rejectDraft('Invalid number', reason) + // `cleanCellValue` nulls an unparseable draft rather than throwing; types + // that declare a message reject it instead of silently clearing the cell. + const parseError = columnTypeOf(column).parseErrorMessage + if (cleaned === null && draft.trim() !== '' && parseError) { + rejectDraft(parseError, reason) return } doneRef.current = true @@ -313,13 +317,13 @@ function InlineTextEditor({ } } - const isNumber = column.type === 'number' + const inputMode = columnTypeOf(column).inputMode return ( { setDraft(e.target.value) @@ -425,13 +429,16 @@ function InlineSelectEditor({ value, column, onSave, onCancel }: InlineEditorPro ) } -/** Dispatches to the right editor variant based on the column type. */ +/** Dispatches to the editor variant the column type declares. */ export function InlineEditor(props: InlineEditorProps) { - if (props.column.type === 'date') { - return + switch (columnTypeOf(props.column).editor) { + case 'date': + return + case 'select': + return + // `toggle` types never open an editor — the grid flips them in place — so + // reaching here at all means a text draft is the sane fallback. + default: + return } - if (props.column.type === 'select') { - return - } - return } diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/column-type-icon.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/column-type-icon.tsx index dfaf662fbf..5331129a24 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/column-type-icon.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/column-type-icon.tsx @@ -2,25 +2,17 @@ import type React from 'react' import { Tooltip } from '@sim/emcn' -import { - Calendar as CalendarIcon, - PlayOutline, - TagIcon, - TypeBoolean, - TypeJson, - TypeNumber, - TypeText, - WorkflowX, -} from '@sim/emcn/icons' +import { PlayOutline, WorkflowX } from '@sim/emcn/icons' +import { columnTypeById } from '@/lib/table/column-types' import type { BlockIconInfo } from '../types' -export const COLUMN_TYPE_ICONS: Record = { - string: TypeText, - number: TypeNumber, - boolean: TypeBoolean, - date: CalendarIcon, - json: TypeJson, - select: TagIcon, +/** + * Icon for a column type. Reads the column-type registry rather than restating + * it — this and the type picker's list used to be two hand-maintained copies + * that had to be edited together. Unknown types fall back to the text icon. + */ +export function columnTypeIcon(type: string): React.ComponentType<{ className?: string }> { + return columnTypeById(type).icon } interface ColumnTypeIconProps { @@ -72,6 +64,6 @@ export function ColumnTypeIcon({ const Icon = blockIconInfo?.icon ?? PlayOutline return } - const Icon = COLUMN_TYPE_ICONS[type] ?? TypeText + const Icon = columnTypeIcon(type) return } diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/index.ts b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/index.ts index 8c8ef9f9dc..17ba6d7c7d 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/index.ts +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/headers/index.ts @@ -1,3 +1,3 @@ export { ColumnHeaderMenu } from './column-header-menu' -export { COLUMN_TYPE_ICONS, ColumnTypeIcon } from './column-type-icon' +export { ColumnTypeIcon, columnTypeIcon } from './column-type-icon' export { ColumnOptionsMenu, WorkflowGroupMetaCell } from './workflow-group-meta-cell' diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/table-grid.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/table-grid.tsx index d7bf866e72..e2b8ef12aa 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/table-grid.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/table-grid/table-grid.tsx @@ -20,6 +20,7 @@ import type { WorkflowGroup, } from '@/lib/table' import { getColumnId } from '@/lib/table/column-keys' +import { columnTypeOf } from '@/lib/table/column-types' import { TABLE_LIMITS } from '@/lib/table/constants' import { useUserPermissionsContext } from '@/app/workspace/[workspaceId]/providers/workspace-permissions-provider' import type { BlockedTableAction } from '@/app/workspace/[workspaceId]/tables/[tableId]/lock-copy' @@ -43,15 +44,10 @@ import { extractCreatedRowId, useTableUndo } from '@/hooks/use-table-undo' import type { DeletedRowSnapshot } from '@/stores/table/types' import { useContextMenu, useTable } from '../../hooks' import type { EditingCell, QueryOptions, SaveReason } from '../../types' -import { - cleanCellValue, - generateColumnName as sharedGenerateColumnName, - storageToDisplay, -} from '../../utils' +import { cleanCellValue, generateColumnName as sharedGenerateColumnName } from '../../utils' import type { ColumnConfig } from '../column-config-sidebar' import { ContextMenu } from '../context-menu' import { NewColumnDropdown } from '../new-column-dropdown' -import { resolveSelectOptions } from '../select-field' import type { WorkflowConfig } from '../workflow-sidebar' import { ExpandedCellPopover } from './cells' import { ADD_COL_WIDTH, COL_WIDTH, SELECTION_TINT_BG } from './constants' @@ -294,10 +290,10 @@ interface TableGridProps { */ function cellToText(value: unknown, column?: DisplayColumn): string { if (value === null || value === undefined) return '' - if (column?.type === 'select') { - return resolveSelectOptions(column, value) - .map((o) => o.name) - .join(', ') + // Types storing opaque ids copy their labels — the clipboard should hold what + // the user sees, and pasting it back round-trips through the same resolver. + if (column && columnTypeOf(column).storesOpaqueIds) { + return columnTypeOf(column).formatForDisplay(value, column) } return typeof value === 'object' ? JSON.stringify(value) : String(value) } @@ -1106,7 +1102,7 @@ export function TableGrid({ function handleContextMenuEditCell() { if (contextMenu.row && contextMenu.columnName) { const column = columnsRef.current.find((c) => getColumnId(c) === contextMenu.columnName) - if (column?.type === 'boolean') { + if (column && columnTypeOf(column).editor === 'toggle') { toggleBooleanCell( contextMenu.row.id, contextMenu.columnName, @@ -1576,7 +1572,8 @@ export function TableGrid({ if (colIndex === -1) return const column = cols[colIndex] - if (column.type === 'boolean') return + // A toggle renders a fixed-size control, so there is no text to fit to. + if (columnTypeOf(column).editor === 'toggle') return const host = containerRef.current ?? document.body const currentRows = rowsRef.current @@ -1595,27 +1592,18 @@ export function TableGrid({ for (const row of currentRows) { const val = row.data[column.key] if (val == null) continue + // Measure what the cell actually RENDERS, not the stored value — + // otherwise auto-fit sizes a select column to its opaque option ids and + // a currency column to a bare number without its symbol or separators. let text: string - if (column.type === 'json') { - if (typeof val === 'string') { - text = val - } else { - try { - text = JSON.stringify(val) - } catch { - text = String(val) - } + if (column.type === 'json' && typeof val !== 'string') { + try { + text = JSON.stringify(val) + } catch { + text = String(val) } - } else if (column.type === 'date') { - text = storageToDisplay(String(val), { seconds: true }) - } else if (column.type === 'select') { - // Cells store option ids; measure the rendered pill labels instead so - // auto-fit doesn't size the column to opaque ids. - text = resolveSelectOptions(column, val) - .map((o) => o.name) - .join(', ') } else { - text = String(val) + text = columnTypeOf(column).formatForDisplay(val, column) } measure.textContent = text maxWidth = Math.max(maxWidth, measure.getBoundingClientRect().width + 17) @@ -2178,7 +2166,7 @@ export function TableGrid({ const handleCellClick = useCallback( (rowId: string, columnName: string, options?: { toggleBoolean?: boolean }) => { const column = columnsRef.current.find((c) => c.key === columnName) - if (column?.type === 'boolean') { + if (column && columnTypeOf(column).editor === 'toggle') { if (!options?.toggleBoolean || !canEditCellRef.current) return const row = rowsRef.current.find((r) => r.id === rowId) if (row) { @@ -2198,7 +2186,7 @@ export function TableGrid({ const handleCellDoubleClick = useCallback( (rowId: string, columnName: string, columnKey: string) => { const column = columnsRef.current.find((c) => c.key === columnKey) - if (column?.type === 'boolean') return + if (column && columnTypeOf(column).editor === 'toggle') return // Double-click means "edit this cell". On an update-locked table, say so // rather than opening the expanded viewer — which looks like an editor @@ -2213,8 +2201,9 @@ export function TableGrid({ setSelectionFocus(null) setIsColumnSelection(false) - // Date/number: use inline editor (calendar picker / numeric input). - if ((column?.type === 'date' || column?.type === 'number') && canEditCellRef.current) { + // Types with a bounded value edit in place (calendar picker, numeric + // input); only free-form prose opens the big expanded popover. + if (column && !columnTypeOf(column).expandable && canEditCellRef.current) { setEditingCell({ rowId, columnName }) setInitialCharacter(null) return @@ -2470,7 +2459,7 @@ export function TableGrid({ const row = currentRows[anchor.rowIndex] if (!row) return - if (col.type === 'boolean') { + if (columnTypeOf(col).editor === 'toggle') { toggleBooleanCellRef.current(row.id, col.key, row.data[col.key]) return } @@ -2712,9 +2701,12 @@ export function TableGrid({ // Workflow-output cells are editable: the user can override the // workflow's value if they want. Booleans toggle on space/click — // typeahead doesn't apply to them. - if (!col || col.type === 'boolean') return - if (col.type === 'number' && !/[\d.-]/.test(e.key)) return - if (col.type === 'date' && !/[\d\-/]/.test(e.key)) return + if (!col || columnTypeOf(col).editor === 'toggle') return + // Types that parse their input only start an edit on a key they could + // actually accept, so a stray letter doesn't open an editor that can + // never save. + const typeahead = columnTypeOf(col).typeaheadPattern + if (typeahead && !typeahead.test(e.key)) return e.preventDefault() const row = currentRows[anchor.rowIndex] @@ -3472,6 +3464,7 @@ export function TableGrid({ // invalid with no options, and the saved cell data is option ids. ...(entry.def?.options ? { columnOptions: entry.def.options } : {}), ...(entry.def?.multiple ? { columnMultiple: true } : {}), + ...(entry.def?.currencyCode ? { columnCurrencyCode: entry.def.currencyCode } : {}), cellData, previousOrder: orderSnapshot, previousWidth, diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/workflow-sidebar/workflow-sidebar.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/workflow-sidebar/workflow-sidebar.tsx index b6702c721f..d5d67dc2a8 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/workflow-sidebar/workflow-sidebar.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/components/workflow-sidebar/workflow-sidebar.tsx @@ -43,6 +43,7 @@ import type { } from '@/lib/table' import { getColumnId } from '@/lib/table/column-keys' import { columnTypeForLeaf, deriveOutputColumnName } from '@/lib/table/column-naming' +import { columnTypeById } from '@/lib/table/column-types' import { type FlattenOutputsBlockInput, type FlattenOutputsEdgeInput, @@ -161,16 +162,7 @@ interface WorkflowStatePayload { } function tableColumnTypeToInputType(colType: ColumnDefinition['type'] | undefined): string { - switch (colType) { - case 'number': - return 'number' - case 'boolean': - return 'boolean' - case 'json': - return 'object' - default: - return 'string' - } + return columnTypeById(colType).workflowInputType } const TagIcon: React.FC<{ diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/table.tsx b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/table.tsx index 146c02c103..553d032101 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/table.tsx +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/table.tsx @@ -71,7 +71,7 @@ import { WorkflowSidebar, } from './components' import { COLUMN_SIDEBAR_WIDTH } from './components/table-grid/constants' -import { COLUMN_TYPE_ICONS } from './components/table-grid/headers' +import { columnTypeIcon } from './components/table-grid/headers' import { useTable, useTableEventStream } from './hooks' import { type BlockedTableAction, describeBlockedAction, lockedNouns } from './lock-copy' import { @@ -1047,7 +1047,7 @@ export function Table({ id: getColumnId(col), label: col.name, type: col.type, - icon: COLUMN_TYPE_ICONS[col.type], + icon: columnTypeIcon(col.type), })), [columns] ) diff --git a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts index cbeb8c122d..d4c3fc5b3c 100644 --- a/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts +++ b/apps/sim/app/workspace/[workspaceId]/tables/[tableId]/utils.ts @@ -1,12 +1,7 @@ -import type { ColumnDefinition } from '@/lib/table' -import { - formatDateCellDisplay, - getWallClockParts, - normalizeDateCellValue, - storedDateToEditable, -} from '@/lib/table/dates' - -type BadgeVariant = 'green' | 'blue' | 'purple' | 'orange' | 'teal' | 'gray' +import type { ColumnDefinition, JsonValue } from '@/lib/table' +import type { ColumnType } from '@/lib/table/column-types' +import { columnTypeById, columnTypeOf } from '@/lib/table/column-types' +import { formatDateCellDisplay, getWallClockParts, normalizeDateCellValue } from '@/lib/table/dates' /** * Pick a fresh "untitled[_N]" name not already taken by `columns`. Used by @@ -23,26 +18,6 @@ export function generateColumnName(columns: ReadonlyArray<{ name: string }>): st return name } -/** - * Returns the appropriate badge color variant for a column type - */ -export function getTypeBadgeVariant(type: string): BadgeVariant { - switch (type) { - case 'string': - return 'green' - case 'number': - return 'blue' - case 'boolean': - return 'purple' - case 'json': - return 'orange' - case 'date': - return 'teal' - default: - return 'gray' - } -} - /** * Coerce a raw input value to the appropriate type for a column. * Throws on invalid JSON. @@ -52,11 +27,9 @@ export function cleanCellValue( column: ColumnDefinition, timeZone?: string ): unknown { - if (column.type === 'number') { - if (value === '') return null - const num = Number(value) - return Number.isNaN(num) ? null : num - } + // These three read the browser's own context (the viewer's timezone, a JSON + // draft that must throw so the editor can show a parse error, a checkbox's + // truthiness) so they cannot come from the shared coercion. if (column.type === 'json') { if (typeof value === 'string') { if (value === '') return null @@ -64,56 +37,17 @@ export function cleanCellValue( } return value } - if (column.type === 'boolean') { - return Boolean(value) - } + if (column.type === 'boolean') return Boolean(value) if (column.type === 'date') { if (value === '' || value === null || value === undefined) return null return displayToStorage(String(value), timeZone) } - if (column.type === 'select') { - return cleanSelectValue(value, column) - } - return value || null -} + if (value === '' || value === null || value === undefined) return null -/** - * Client-side mirror of the server's `select` coercion: a cell stores option - * ids, but pasted or imported text carries names. Resolving here — not only on - * the server — is what keeps the optimistic cache holding ids, so the pasted - * cell renders its pill immediately instead of blanking until the refetch. - */ -function cleanSelectValue(value: unknown, column: ColumnDefinition): unknown { - const options = column.options ?? [] - const resolve = (raw: unknown): string | null => { - if (typeof raw !== 'string') return null - const match = - options.find((o) => o.id === raw) ?? - options.find((o) => o.name === raw) ?? - options.find((o) => o.name.toLowerCase() === raw.toLowerCase()) - return match ? match.id : null - } - - if (column.multiple) { - // Comma-delimited is the multi cell's own clipboard/CSV format, so a paste - // of one round-trips. Option names containing commas are a known ambiguity. - const raw = Array.isArray(value) - ? value - : typeof value === 'string' - ? value - .split(',') - .map((part) => part.trim()) - .filter((part) => part !== '') - : [] - const ids: string[] = [] - for (const entry of raw) { - const id = resolve(entry) - if (id !== null && !ids.includes(id)) ids.push(id) - } - return ids - } - - return resolve(Array.isArray(value) ? value[0] : value) + // Everything else runs the SAME coercion the server will run, so the + // optimistic cache holds exactly the value that gets persisted. + const coerced = columnTypeOf(column).coerce(value as JsonValue, column) + return coerced.ok ? coerced.value : null } /** @@ -125,14 +59,15 @@ function cleanSelectValue(value: unknown, column: ColumnDefinition): unknown { */ export function formatValueForInput(value: unknown, type: string): string { if (value === null || value === undefined) return '' - if (type === 'json') { - return typeof value === 'string' ? value : JSON.stringify(value) + const definition = columnTypeById(type) + // Shape-drift guard, kept ahead of the registry: a column whose declared type + // lags its actual data (a workflow column mid-remap, where the schema cache + // hasn't refetched but row data already holds the new mapping's value) would + // otherwise render `[object Object]` through a scalar type's formatter. + if (typeof value === 'object' && !definition.storesOpaqueIds && type !== 'json') { + return JSON.stringify(value) } - if (type === 'date' && value) { - return storedDateToEditable(String(value)) - } - if (typeof value === 'object') return JSON.stringify(value) - return String(value) + return definition.formatForInput(value, { name: '', type: type as ColumnType }) } /** A canonical date-cell value split into its wall-clock editing parts. */ diff --git a/apps/sim/hooks/use-table-undo.ts b/apps/sim/hooks/use-table-undo.ts index c1d4c72c9d..205e52b8b5 100644 --- a/apps/sim/hooks/use-table-undo.ts +++ b/apps/sim/hooks/use-table-undo.ts @@ -390,6 +390,7 @@ export function useTableUndo({ // cell data restored below is keyed by those option ids. ...(action.columnOptions ? { options: action.columnOptions } : {}), ...(action.columnMultiple ? { multiple: true } : {}), + ...(action.columnCurrencyCode ? { currencyCode: action.columnCurrencyCode } : {}), position: action.columnPosition, }, { diff --git a/apps/sim/lib/api/contracts/tables.ts b/apps/sim/lib/api/contracts/tables.ts index 5bccaa5206..075a1a8a19 100644 --- a/apps/sim/lib/api/contracts/tables.ts +++ b/apps/sim/lib/api/contracts/tables.ts @@ -55,19 +55,45 @@ export const selectOptionsSchema = z .array(selectOptionSchema) .max(MAX_SELECT_OPTIONS, `A select column cannot have more than ${MAX_SELECT_OPTIONS} options`) +/** + * ISO 4217 code for a `currency` column, normalized to upper case. + * + * Deliberately shape-only. Whether a code is one the runtime can actually + * format is checked server-side in `validateColumnDefinition`, because this + * same schema parses RESPONSES: pinning the boundary to the *client's* ICU + * table would make any divergence between the two runtimes' currency lists + * reject an entire table schema over one column's code. + */ +export const currencyCodeSchema = z + .string() + .regex(/^[A-Za-z]{3}$/, 'Must be a 3-letter ISO 4217 currency code, e.g. USD') + .transform((code) => code.toUpperCase()) + /** * Cross-field rule: a `select` column must declare a non-empty option set; - * other types must not carry options or `multiple`. Skipped when `type` is - * absent (an options-only update on an existing select column). + * other types must not carry options or `multiple`, and only a `currency` + * column may carry `currencyCode`. Skipped when `type` is absent (a + * metadata-only update on an existing column). */ function refineColumnOptions( data: { type?: (typeof COLUMN_TYPES)[number] options?: z.infer multiple?: boolean + currencyCode?: string }, ctx: z.RefinementCtx ): void { + // `currencyCode` on a non-currency column is inert until a later + // convert-to-currency inherits it, silently overriding the currency the user + // picked in that request. + if (data.type !== undefined && data.type !== 'currency' && data.currencyCode !== undefined) { + ctx.addIssue({ + code: 'custom', + path: ['currencyCode'], + message: 'currencyCode is only allowed on currency columns', + }) + } if (data.type === 'select') { if (!data.options || data.options.length === 0) { ctx.addIssue({ @@ -165,6 +191,8 @@ export const tableColumnSchema = z options: selectOptionsSchema.optional(), /** A `select` column that accepts multiple options per cell. */ multiple: z.boolean().optional(), + /** ISO 4217 code for a `currency` column. */ + currencyCode: currencyCodeSchema.optional(), }) .superRefine(refineColumnOptions) @@ -242,6 +270,7 @@ export const createTableColumnBodySchema = z.object({ position: z.number().int().min(0).optional(), options: selectOptionsSchema.optional(), multiple: z.boolean().optional(), + currencyCode: currencyCodeSchema.optional(), }) .superRefine(refineColumnOptions), }) @@ -257,6 +286,7 @@ export const updateTableColumnBodySchema = z.object({ unique: z.boolean().optional(), options: selectOptionsSchema.optional(), multiple: z.boolean().optional(), + currencyCode: currencyCodeSchema.optional(), }) .superRefine(refineColumnOptions), }) diff --git a/apps/sim/lib/copilot/tools/server/table/user-table.ts b/apps/sim/lib/copilot/tools/server/table/user-table.ts index 9f66adcb9d..036ceb2cd4 100644 --- a/apps/sim/lib/copilot/tools/server/table/user-table.ts +++ b/apps/sim/lib/copilot/tools/server/table/user-table.ts @@ -34,15 +34,18 @@ import { sortSpecNamesToIds, } from '@/lib/table/column-keys' import { columnTypeForLeaf, deriveOutputColumnName } from '@/lib/table/column-naming' +import { columnTypeById } from '@/lib/table/column-types' import { addTableColumn, deleteColumn, deleteColumns, renameColumn, updateColumnConstraints, + updateColumnCurrency, updateColumnOptions, updateColumnType, } from '@/lib/table/columns/service' +import { isSupportedCurrencyCode } from '@/lib/table/currency' import { markTableDeleteFailed, runTableDelete } from '@/lib/table/delete-runner' import { runTableImport, type TableImportPayload } from '@/lib/table/import-runner' import { markTableJobRunning, releaseJobClaim } from '@/lib/table/jobs/service' @@ -1540,6 +1543,7 @@ export const userTableServerTool: BaseServerTool position?: number options?: unknown multiple?: boolean + currencyCode?: string } | undefined if (!col?.name || !col?.type) { @@ -1554,6 +1558,12 @@ export const userTableServerTool: BaseServerTool } const requestId = generateId().slice(0, 8) assertNotAborted() + if (col.currencyCode !== undefined && !isSupportedCurrencyCode(col.currencyCode)) { + return { + success: false, + message: `Invalid currency code "${col.currencyCode}". Use an ISO 4217 code, e.g. USD`, + } + } // Agent authors select options by name; generate their stable ids here. const columnToAdd = col.type === 'select' @@ -1653,15 +1663,24 @@ export const userTableServerTool: BaseServerTool const uniqFlag = (args as Record).unique as boolean | undefined const rawOptions = (args as Record).options const multiple = (args as Record).multiple as boolean | undefined + const currencyCode = (args as Record).currencyCode as string | undefined if ( newType === undefined && uniqFlag === undefined && rawOptions === undefined && - multiple === undefined + multiple === undefined && + currencyCode === undefined ) { return { success: false, - message: 'At least one of newType, unique, options, or multiple must be provided', + message: + 'At least one of newType, unique, options, multiple, or currencyCode must be provided', + } + } + if (currencyCode !== undefined && !isSupportedCurrencyCode(currencyCode)) { + return { + success: false, + message: `Invalid currency code "${currencyCode}". Use an ISO 4217 code, e.g. USD`, } } const tableForUpdate = await getTableById(args.tableId) @@ -1693,10 +1712,10 @@ export const userTableServerTool: BaseServerTool // update on an existing select column carries the same hazard as a // conversion. Same guard the HTTP column routes apply. const resultingType = newType ?? currentColumn?.type - if (uniqFlag === true && resultingType === 'select') { + if (uniqFlag === true && !columnTypeById(resultingType).supportsUnique) { return { success: false, - message: `Cannot set column "${colName}" as unique: select columns cannot be unique.`, + message: `Cannot set column "${colName}" as unique: ${resultingType} columns cannot be unique.`, } } if (typeChanging) { @@ -1708,6 +1727,27 @@ export const userTableServerTool: BaseServerTool newType: newType as (typeof COLUMN_TYPES)[number], options, multiple, + ...(currencyCode !== undefined ? { currencyCode } : {}), + ...(uniqFlag !== undefined ? { unique: uniqFlag } : {}), + }, + requestId + ) + } else if (currencyCode !== undefined) { + // Re-denominating an existing currency column: schema-only, no cell + // rewrite. Mirrors the HTTP columns routes. + if (currentColumn?.type !== 'currency') { + return { + success: false, + message: `Column "${colName}" is not a currency column. Pass newType: "currency" with currencyCode to convert it.`, + } + } + assertNotAborted() + result = await updateColumnCurrency( + { + tableId: args.tableId, + columnName: colName, + currencyCode, + ...(uniqFlag !== undefined ? { unique: uniqFlag } : {}), }, requestId ) @@ -1725,11 +1765,20 @@ export const userTableServerTool: BaseServerTool } assertNotAborted() result = await updateColumnOptions( - { tableId: args.tableId, columnName: colName, options: nextOptions, multiple }, + { + tableId: args.tableId, + columnName: colName, + options: nextOptions, + multiple, + ...(uniqFlag !== undefined ? { unique: uniqFlag } : {}), + }, requestId ) } - if (uniqFlag !== undefined) { + // Skipped when a typed write ran: that write already applied and + // validated the constraint, in one transaction with the change it + // accompanies. Mirrors the HTTP columns routes. + if (uniqFlag !== undefined && result === undefined) { assertNotAborted() result = await updateColumnConstraints( { tableId: args.tableId, columnName: colName, unique: uniqFlag }, diff --git a/apps/sim/lib/table/__tests__/column-conversion.test.ts b/apps/sim/lib/table/__tests__/column-conversion.test.ts index 71726cf00e..3e7c1bf8a5 100644 --- a/apps/sim/lib/table/__tests__/column-conversion.test.ts +++ b/apps/sim/lib/table/__tests__/column-conversion.test.ts @@ -5,7 +5,13 @@ * coverage and every case below was a shipped defect caught in review. */ import { describe, expect, it } from 'vitest' -import { isValueCompatibleWithType, selectValueForConversion } from '@/lib/table/columns/service' +import { COLUMN_TYPE_REGISTRY } from '@/lib/table/column-types' +import { + applyPendingRename, + isValueCompatibleWithType, + selectValueForConversion, +} from '@/lib/table/columns/service' +import { resolveSelectOptionId } from '@/lib/table/select-options' import type { ColumnDefinition, SelectOption } from '@/lib/table/types' const OPTIONS: SelectOption[] = [ @@ -113,3 +119,140 @@ describe('isValueCompatibleWithType — string target', () => { expect(isValueCompatibleWithType(flattened, 'string')).toBe(true) }) }) + +describe('isValueCompatibleWithType — currency', () => { + it('accepts numbers and the formatted shapes a text column holds', () => { + expect(isValueCompatibleWithType(1234.56, 'currency')).toBe(true) + expect(isValueCompatibleWithType(0, 'currency')).toBe(true) + expect(isValueCompatibleWithType('$1,234.56', 'currency')).toBe(true) + expect(isValueCompatibleWithType('1.234,56 €', 'currency')).toBe(true) + expect(isValueCompatibleWithType('(12.00)', 'currency')).toBe(true) + }) + + it('rejects values that would leave un-castable text in a numeric column', () => { + expect(isValueCompatibleWithType('ask sales', 'currency')).toBe(false) + expect(isValueCompatibleWithType('', 'currency')).toBe(false) + expect(isValueCompatibleWithType(true, 'currency')).toBe(false) + expect(isValueCompatibleWithType({ amount: 1 }, 'currency')).toBe(false) + }) + + it('treats an absent cell as convertible, like every other target type', () => { + expect(isValueCompatibleWithType(null, 'currency')).toBe(true) + expect(isValueCompatibleWithType(undefined, 'currency')).toBe(true) + }) + + it('accepts a currency cell converting back to a plain number', () => { + expect(isValueCompatibleWithType(1234.56, 'number')).toBe(true) + }) +}) + +describe('blank cells during conversion', () => { + // A text column with a single empty cell could not be converted to a number + // at all: `''` is incompatible with every numeric type, and the scan counted + // it as a hard blocker even when the target was optional — reporting it with + // an error that said "to a required ..." regardless. + it('treats an empty cell as incompatible with the numeric types', () => { + for (const type of ['number', 'currency'] as const) { + expect(isValueCompatibleWithType('', type)).toBe(false) + } + }) + + it('still accepts an empty string for text, which can legitimately hold it', () => { + expect(isValueCompatibleWithType('', 'string')).toBe(true) + expect(isValueCompatibleWithType('', 'json')).toBe(true) + }) + + it('lets a cleared select cell through only when the target is optional', () => { + expect(isValueCompatibleWithType('', 'select', OPTIONS, false, false)).toBe(true) + expect(isValueCompatibleWithType('', 'select', OPTIONS, false, true)).toBe(false) + }) +}) + +describe('rename folded into another write', () => { + // A rename is metadata-only — rows key on the stable column id — so nothing + // forces it to be its own transaction. Folding it into whichever write a + // request already carries is what makes a combined PATCH all-or-nothing: the + // name collision is detected against the same schema snapshot the other + // change is being applied to, and both abort together. + it('rejects a collision against the schema the write is landing in', () => { + const columns: ColumnDefinition[] = [ + { id: 'col_a', name: 'amount', type: 'currency' }, + { id: 'col_b', name: 'taken', type: 'string' }, + ] + expect(() => applyPendingRename(columns, 0, 'taken')).toThrow(/already exists/) + expect(() => applyPendingRename(columns, 0, 'TAKEN')).toThrow(/already exists/) + }) + + it('signals a no-op by identity, which is how callers detect nothing to write', () => { + // The early returns in `updateColumnType` / `updateColumnCurrency` rely on + // this: same reference means there is genuinely nothing to persist. + const columns: ColumnDefinition[] = [{ id: 'col_a', name: 'amount', type: 'currency' }] + expect(applyPendingRename(columns, 0, undefined)).toBe(columns[0]) + expect(applyPendingRename(columns, 0, 'amount')).toBe(columns[0]) + expect(applyPendingRename(columns, 0, 'renamed')).not.toBe(columns[0]) + }) + + it('applies a valid rename and is a no-op without one', () => { + const columns: ColumnDefinition[] = [{ id: 'col_a', name: 'amount', type: 'currency' }] + expect(applyPendingRename(columns, 0, 'total').name).toBe('total') + expect(applyPendingRename(columns, 0, undefined)).toBe(columns[0]) + expect(applyPendingRename(columns, 0, 'amount')).toBe(columns[0]) + }) + + it('rejects a name the column-name rules forbid', () => { + const columns: ColumnDefinition[] = [{ id: 'col_a', name: 'amount', type: 'currency' }] + expect(() => applyPendingRename(columns, 0, '1bad')).toThrow(/must start with/) + expect(() => applyPendingRename(columns, 0, 'a'.repeat(200))).toThrow(/maximum length/) + }) +}) + +describe('select accepts scalar cells', () => { + // The resolver stringifies a scalar before matching, so a `number` or + // `boolean` column whose values equal option NAMES converts. The migration's + // JSONB predicate has to cover those types too — matching only `'string'` + // left the cells as raw numbers inside a select column, where they render as + // nothing and fail option membership on the next write. + const NUMERIC_OPTIONS: SelectOption[] = [ + { id: 'opt_1', name: '123' }, + { id: 'opt_t', name: 'true' }, + ] + + it('resolves a numeric or boolean cell to its option id', () => { + expect(resolveSelectOptionId(123, NUMERIC_OPTIONS)).toBe('opt_1') + expect(resolveSelectOptionId(true, NUMERIC_OPTIONS)).toBe('opt_t') + }) + + it('reports those cells as convertible, which is what obliges the migration', () => { + expect(isValueCompatibleWithType(123, 'select', NUMERIC_OPTIONS)).toBe(true) + expect(isValueCompatibleWithType(true, 'select', NUMERIC_OPTIONS)).toBe(true) + expect(isValueCompatibleWithType(999, 'select', NUMERIC_OPTIONS)).toBe(false) + }) + + it('leaves structured values unresolvable', () => { + expect(resolveSelectOptionId({ a: 1 } as never, NUMERIC_OPTIONS)).toBeNull() + }) +}) + +describe('constraint validation reads post-migration values', () => { + // The ordering that matters: `updateColumnOptions` rewrites stored cells (a + // single<->multi toggle changes the shape; removing an option clears cells), + // and `updateColumnType` rewrites them through the coercion write-back. A + // `unique` scan run BEFORE those would read values that no longer exist by + // the time the constraint is persisted, pass, and let the rewrite produce the + // duplicates it was supposed to prevent. + // + // The coercion case is the concrete one: two distinct strings can collapse to + // one number. + it('shows how a conversion manufactures duplicates the pre-scan cannot see', () => { + const column: ColumnDefinition = { name: 'sku', type: 'number' } + const before = ['5', '5.0'] + expect(new Set(before).size).toBe(2) + + const after = before.map((value) => { + const coerced = COLUMN_TYPE_REGISTRY.number.coerce(value, column) + return coerced.ok ? coerced.value : value + }) + expect(after).toEqual([5, 5]) + expect(new Set(after).size).toBe(1) + }) +}) diff --git a/apps/sim/lib/table/__tests__/column-type-registry.test.ts b/apps/sim/lib/table/__tests__/column-type-registry.test.ts new file mode 100644 index 0000000000..73c5ffc424 --- /dev/null +++ b/apps/sim/lib/table/__tests__/column-type-registry.test.ts @@ -0,0 +1,205 @@ +/** + * @vitest-environment node + * + * Guards for the column-type registry itself, rather than for any one type. + * + * The registry replaced ~40 hand-maintained `switch` arms whose failure mode + * was silence — a missing arm compared numbers as text or blocked every + * conversion, with nothing to notice. These assert the properties that used to + * be spread across those arms, so a new type either satisfies them or fails + * here. + */ +import { describe, expect, it } from 'vitest' +import type { ColumnType } from '@/lib/table/column-types' +import { + ALL_COLUMN_TYPES, + COLUMN_TYPE_REGISTRY, + COLUMN_TYPES, + columnTypeById, + isColumnType, + isValueCompatible, +} from '@/lib/table/column-types' +import type { ColumnDefinition } from '@/lib/table/types' +import { validateColumnDefinition } from '@/lib/table/validation' + +describe('registry shape', () => { + it('keys every entry by its own id', () => { + for (const [key, definition] of Object.entries(COLUMN_TYPE_REGISTRY)) { + expect(definition.id).toBe(key) + } + }) + + it('derives COLUMN_TYPES from the registry, with no drift', () => { + expect([...COLUMN_TYPES].sort()).toEqual(Object.keys(COLUMN_TYPE_REGISTRY).sort()) + expect(ALL_COLUMN_TYPES).toHaveLength(COLUMN_TYPES.length) + }) + + it('falls back to string for an unknown type instead of throwing', () => { + // A malformed or future schema must render as text, not crash mid-render. + expect(columnTypeById('percent').id).toBe('string') + expect(columnTypeById(undefined).id).toBe('string') + expect(isColumnType('percent')).toBe(false) + expect(isColumnType('currency')).toBe(true) + }) + + it('only casts to numeric/timestamptz for types whose storage is actually that', () => { + // A wrong cast makes every filter and sort on the column fail in SQL. + for (const definition of ALL_COLUMN_TYPES) { + if (definition.jsonbCast === null) continue + expect(['numeric', 'timestamptz']).toContain(definition.jsonbCast) + } + expect(COLUMN_TYPE_REGISTRY.currency.jsonbCast).toBe(COLUMN_TYPE_REGISTRY.number.jsonbCast) + }) + + it('restricts filter operators only for types storing opaque ids', () => { + // Restricting a comparable type would silently drop valid filters. + for (const definition of ALL_COLUMN_TYPES) { + const restricted = definition.filterOperatorsFor?.({ name: 'c', type: definition.id }) + if (restricted) expect(definition.storesOpaqueIds).toBe(true) + } + }) + + it('never lets a type with its own metadata be unique-constrained implicitly', () => { + // Uniqueness compares the stored value; for a type whose storage is an + // opaque id that caps each option at one row for the whole table. + for (const definition of ALL_COLUMN_TYPES) { + if (definition.storesOpaqueIds) expect(definition.supportsUnique).toBe(false) + } + }) + + it('never asks for a native number input on a type accepting formatted text', () => { + // `` rejects `$1,234.56` outright, so a type whose + // parser exists to accept that must get a text field with a numeric keypad. + for (const definition of ALL_COLUMN_TYPES) { + if (!definition.acceptsFormattedInput) continue + expect(definition.inputMode).toBe('decimal') + } + }) + + it('gives every type that can reject a draft a message to show', () => { + // Without one, `cleanCellValue` nulls the draft and the edit vanishes with + // no explanation. + for (const definition of ALL_COLUMN_TYPES) { + if (definition.typeaheadPattern) expect(definition.parseErrorMessage).toBeTruthy() + } + }) +}) + +describe('conversion write-back', () => { + // A retype is allowed exactly when the target's `coerce` accepts the value, + // and `coerce` often TRANSFORMS it. The conversion must therefore write the + // transformed value back — filters and sorts apply `jsonbCast` to whatever is + // stored, so a value left in its old shape breaks every query on the column. + it.each` + type | stored | expected + ${'date'} | ${1700000000000} | ${'2023-11-14T22:13:20.000Z'} + ${'date'} | ${'2024-01-01'} | ${'2024-01-01'} + ${'currency'} | ${'$1,234.56'} | ${1234.56} + ${'currency'} | ${'1.234,56'} | ${1234.56} + ${'number'} | ${'1999'} | ${1999} + `('$type coerces $stored to a value its jsonbCast can read', ({ type, stored, expected }) => { + const column = { name: 'c', type } as ColumnDefinition + const result = COLUMN_TYPE_REGISTRY[type as ColumnType].coerce(stored, column) + expect(result.ok && result.value).toEqual(expected) + }) + + it('never leaves a numeric-cast type holding something Postgres cannot cast', () => { + // The concrete failure this guards: an epoch number left in a `date` + // column makes `(data->>'col')::timestamptz` throw on every query. + for (const definition of ALL_COLUMN_TYPES) { + if (definition.jsonbCast !== 'timestamptz') continue + const coerced = definition.coerce(1700000000000, { name: 'c', type: definition.id }) + expect(coerced.ok).toBe(true) + expect(typeof (coerced as { value: unknown }).value).toBe('string') + } + }) +}) + +describe('intentional divergences from the pre-registry behavior', () => { + // A differential run of the registry against the pre-refactor implementations + // (55 values x 7 column shapes) found ZERO coercion differences and exactly + // these compatibility differences. Both are deliberate fixes; pinning them + // here so neither can be silently reverted or quietly widened. + + it('rejects boolean conversions the old gate accepted and then nulled', () => { + // The old gate accepted '1'/'0'/1/0 but the write path only ever accepted + // 'true'/'false' — so the conversion reported zero incompatible rows and + // then nulled every one of them. Rejecting is the honest answer. + const column: ColumnDefinition = { name: 'b', type: 'boolean' } + for (const value of ['0', '1', 0, 1]) { + expect(isValueCompatible(value, column)).toBe(false) + expect(COLUMN_TYPE_REGISTRY.boolean.coerce(value as never, column).ok).toBe(false) + } + for (const value of [true, false, 'true', 'false']) { + expect(isValueCompatible(value, column)).toBe(true) + } + }) + + it('refuses to bulk-convert a number column to date', () => { + // `date.coerce` accepts an epoch for a single deliberate write, but + // reinterpreting a whole numeric column as epoch milliseconds is + // destructive and irreversible — 1, 5, 42 would become three timestamps in + // January 1970. The gate may be stricter than `coerce`, never looser. + const column: ColumnDefinition = { name: 'd', type: 'date' } + for (const value of [0, 1, 42, 1700000000]) { + expect(isValueCompatible(value, column)).toBe(false) + // The write path still accepts it. + expect(COLUMN_TYPE_REGISTRY.date.coerce(value as never, column).ok).toBe(true) + } + expect(isValueCompatible('2024-01-01', column)).toBe(true) + }) + + it('never lets a gate be LOOSER than its write path', () => { + // The dangerous direction: a gate that accepts what `coerce` rejects + // reports zero incompatible rows and then nulls every one of them. + const samples: unknown[] = ['', '0', '1', 'abc', 0, 1, 42, true, '2024-01-01', '$1.50'] + for (const definition of ALL_COLUMN_TYPES) { + if (definition.id === 'select') continue + const column: ColumnDefinition = { name: 'c', type: definition.id } + for (const value of samples) { + if (!isValueCompatible(value, column)) continue + expect( + definition.coerce(value as never, column).ok, + `${definition.id} gate accepts ${JSON.stringify(value)} but coerce rejects it` + ).toBe(true) + } + } + }) +}) + +describe('metadata ownership', () => { + const column = (over: Partial): ColumnDefinition => + ({ name: 'c', type: 'string', ...over }) as ColumnDefinition + const options = [{ id: 'opt_a', name: 'A' }] + + it.each` + label | definition | valid | needle + ${'options on select'} | ${column({ type: 'select', options })} | ${true} | ${''} + ${'options on string'} | ${column({ type: 'string', options })} | ${false} | ${'cannot define options'} + ${'options on currency'} | ${column({ type: 'currency', options })} | ${false} | ${'cannot define options'} + ${'multiple on number'} | ${column({ type: 'number', multiple: true })} | ${false} | ${'cannot be multiple'} + ${'code on currency'} | ${column({ type: 'currency', currencyCode: 'USD' })} | ${true} | ${''} + ${'code on number'} | ${column({ type: 'number', currencyCode: 'USD' })} | ${false} | ${'cannot define a currency'} + ${'code on select'} | ${column({ type: 'select', currencyCode: 'USD', options })} | ${false} | ${'cannot define a currency'} + ${'unsupported code'} | ${column({ type: 'currency', currencyCode: 'ZZZ' })} | ${false} | ${'invalid currency code'} + ${'unique on select'} | ${column({ type: 'select', unique: true, options })} | ${false} | ${'cannot be unique'} + ${'unique on currency'} | ${column({ type: 'currency', unique: true })} | ${true} | ${''} + ${'select with no option'} | ${column({ type: 'select' })} | ${false} | ${'at least one option'} + ${'unknown type'} | ${column({ type: 'percent' as ColumnDefinition['type'] })} | ${false} | ${'invalid type'} + `( + 'rejects $label', + ({ + definition, + valid, + needle, + }: { + definition: ColumnDefinition + valid: boolean + needle: string + }) => { + const result = validateColumnDefinition(definition) + expect(result.valid, result.errors.join('; ')).toBe(valid) + if (!valid) expect(result.errors.join(' ').toLowerCase()).toContain(needle.toLowerCase()) + } + ) +}) diff --git a/apps/sim/lib/table/__tests__/currency.test.ts b/apps/sim/lib/table/__tests__/currency.test.ts new file mode 100644 index 0000000000..e7b7802552 --- /dev/null +++ b/apps/sim/lib/table/__tests__/currency.test.ts @@ -0,0 +1,279 @@ +/** + * @vitest-environment node + * + * Guards for the `currency` column type. The parser is the load-bearing piece: + * filters and sorts cast a currency cell to `numeric`, so anything it lets + * through unparsed would break every query against the column. + */ +import { describe, expect, it } from 'vitest' +import { + DEFAULT_CURRENCY_CODE, + formatCurrencyDisplay, + formatCurrencyForInput, + getCurrencyOptions, + isSupportedCurrencyCode, + parseCurrencyInput, + resolveCurrencyCode, +} from '@/lib/table/currency' + +describe('parseCurrencyInput', () => { + it('passes finite numbers through and rejects non-finite ones', () => { + expect(parseCurrencyInput(1234.56)).toBe(1234.56) + expect(parseCurrencyInput(0)).toBe(0) + expect(parseCurrencyInput(-5)).toBe(-5) + expect(parseCurrencyInput(Number.NaN)).toBeNull() + expect(parseCurrencyInput(Number.POSITIVE_INFINITY)).toBeNull() + }) + + it('strips symbols, ISO codes, and every flavor of space', () => { + expect(parseCurrencyInput('$1234.56')).toBe(1234.56) + expect(parseCurrencyInput('€ 12')).toBe(12) + expect(parseCurrencyInput('USD 12.50')).toBe(12.5) + expect(parseCurrencyInput('12,50 €')).toBe(12.5) + // Non-breaking and narrow-no-break spaces are what fr/ru locales group with. + expect(parseCurrencyInput('1 234,56 €')).toBe(1234.56) + expect(parseCurrencyInput('1 234.56')).toBe(1234.56) + }) + + it('reads accounting negatives', () => { + expect(parseCurrencyInput('(1,234.56)')).toBe(-1234.56) + expect(parseCurrencyInput('($12)')).toBe(-12) + expect(parseCurrencyInput('-$12.50')).toBe(-12.5) + }) + + it('treats the later separator as the decimal point when both appear', () => { + expect(parseCurrencyInput('1,234.56')).toBe(1234.56) + expect(parseCurrencyInput('1.234,56')).toBe(1234.56) + expect(parseCurrencyInput('1,234,567.89')).toBe(1234567.89) + expect(parseCurrencyInput('1.234.567,89')).toBe(1234567.89) + }) + + it('disambiguates a lone comma by the digits that follow it', () => { + // Exactly three trailing digits reads as grouping... + expect(parseCurrencyInput('1,500')).toBe(1500) + expect(parseCurrencyInput('100,000')).toBe(100000) + // ...anything else is a decimal comma. + expect(parseCurrencyInput('1,5')).toBe(1.5) + expect(parseCurrencyInput('0,25')).toBe(0.25) + expect(parseCurrencyInput('12,3456')).toBe(12.3456) + }) + + it('treats repeated dots as grouping and a lone dot as the decimal point', () => { + expect(parseCurrencyInput('1.234.567')).toBe(1234567) + expect(parseCurrencyInput('1.5')).toBe(1.5) + expect(parseCurrencyInput('1234.5')).toBe(1234.5) + }) + + it('reads exponent form at face value', () => { + // `String()` emits exponent form past 1e21, so a stored amount round-trips + // through the editor as `1e+21`. Treating the `e` as decoration to strip + // read that back as 121 — a silent 19-orders-of-magnitude loss on the next + // edit of an untouched cell. + expect(parseCurrencyInput('1e5')).toBe(100000) + expect(parseCurrencyInput('1e+21')).toBe(1e21) + expect(parseCurrencyInput('1.5e-3')).toBe(0.0015) + expect(parseCurrencyInput('-1e5')).toBe(-100000) + expect(parseCurrencyInput('(1e5)')).toBe(-100000) + expect(parseCurrencyInput('$1e5')).toBe(100000) + }) + + it('does not mistake an ISO code for an exponent', () => { + // `EUR` survives the symbol strip with its `E` intact; it must still parse + // through the ordinary separator path. + expect(parseCurrencyInput('12 EUR')).toBe(12) + expect(parseCurrencyInput('EUR 12,50')).toBe(12.5) + expect(parseCurrencyInput('USD 1,234.56')).toBe(1234.56) + }) + + it('round-trips a magnitude that stringifies to exponent form', () => { + expect(parseCurrencyInput(formatCurrencyForInput(1e21))).toBe(1e21) + }) + + it('rejects a sign that is not leading, so dates do not read as amounts', () => { + // A date column converting to currency previously turned `2024-01-01` into + // 20240101 — the hyphens were dropped as decoration and the digit groups + // joined. Every cell in the column would have been silently corrupted. + expect(parseCurrencyInput('2024-01-01')).toBeNull() + expect(parseCurrencyInput('2024-01-01T10:30:00Z')).toBeNull() + expect(parseCurrencyInput('1-2-3')).toBeNull() + expect(parseCurrencyInput('12--3')).toBeNull() + // A leading sign is still a sign. + expect(parseCurrencyInput('-12')).toBe(-12) + expect(parseCurrencyInput('+12')).toBe(12) + expect(parseCurrencyInput('-$12.50')).toBe(-12.5) + }) + + it('parses what Intl emits, across locales and signs', () => { + // The realistic input: a user pastes a cell from a spreadsheet. Generated + // rather than hand-listed so a parser change cannot quietly regress a + // locale nobody thought to write down. + const pairs: Array<[string, string]> = [ + ['en-US', 'USD'], + ['de-DE', 'EUR'], + ['fr-FR', 'EUR'], + ['pt-BR', 'BRL'], + ['en-IN', 'INR'], + ['ja-JP', 'JPY'], + ['en-GB', 'GBP'], + ['de-CH', 'CHF'], + ['sv-SE', 'SEK'], + ['da-DK', 'DKK'], + ['pl-PL', 'PLN'], + ['ru-RU', 'RUB'], + ['it-IT', 'EUR'], + ['nl-NL', 'EUR'], + ['tr-TR', 'TRY'], + ['ko-KR', 'KRW'], + ['zh-CN', 'CNY'], + ['en-CA', 'CAD'], + ['en-AU', 'AUD'], + ['he-IL', 'ILS'], + ] + const amounts = [0, 12, 1234.56, 1234567.89, -12.5, -1234.56] + + for (const [locale, currency] of pairs) { + for (const amount of amounts) { + const formatted = new Intl.NumberFormat(locale, { + style: 'currency', + currency, + }).format(amount) + const parsed = parseCurrencyInput(formatted) + expect(parsed, `${locale}/${currency} ${JSON.stringify(formatted)}`).not.toBeNull() + // Zero-decimal currencies round, so compare within one unit. + expect( + Math.abs((parsed as number) - amount), + `${locale}/${currency} ${JSON.stringify(formatted)} -> ${parsed}` + ).toBeLessThanOrEqual(1) + } + } + }) + + it('parses the locale formats of the currencies the picker pins', () => { + // These are exactly what `Intl.NumberFormat` emits, i.e. what a user pastes + // from a spreadsheet. Rejecting them would make the pinned currencies + // unusable in their own conventional notation. + expect(parseCurrencyInput('R$ 1.234,56')).toBe(1234.56) + expect(parseCurrencyInput('₹12,34,567.89')).toBe(1234567.89) + expect(parseCurrencyInput('1 234,56 kr')).toBe(1234.56) + expect(parseCurrencyInput('1.234,56 kr.')).toBe(1234.56) + expect(parseCurrencyInput('1234,56 zł')).toBe(1234.56) + expect(parseCurrencyInput('CHF 1’234.56')).toBe(1234.56) + }) + + it('rejects an identifier whose letters touch its digits', () => { + // The distinguishing rule: a currency marker is always separated from the + // number by a space or a symbol, so letters touching digits mean this is a + // part number, not an amount. Without it, converting a column of SKUs to + // currency rewrote every cell with an invented value. + expect(parseCurrencyInput('SKU400')).toBeNull() + expect(parseCurrencyInput('ABC1234')).toBeNull() + expect(parseCurrencyInput('A1B2')).toBeNull() + // A marker separated properly still parses. + expect(parseCurrencyInput('USD 400')).toBe(400) + expect(parseCurrencyInput('$400')).toBe(400) + }) + + it('rejects text that merely contains digits', () => { + // Scraping digits out of arbitrary text invents a value. A string column of + // SKUs, phone numbers, or US-format dates converting to currency would + // otherwise report zero incompatible rows and rewrite every cell. + expect(parseCurrencyInput('01/02/2024')).toBeNull() + expect(parseCurrencyInput('Room 101')).toBeNull() + expect(parseCurrencyInput('Invoice 2024')).toBeNull() + expect(parseCurrencyInput('1_000')).toBeNull() + }) + + it('rejects malformed separator runs and invalid grouping', () => { + // `0.1.2` and `1,000,00` would read as 12 and 100000 under a plain + // strip-the-separator rule. + expect(parseCurrencyInput('1..2')).toBeNull() + expect(parseCurrencyInput('1,,2')).toBeNull() + expect(parseCurrencyInput('0.1.2')).toBeNull() + expect(parseCurrencyInput('1,000,00')).toBeNull() + // Valid grouping still works, western and Indian. + expect(parseCurrencyInput('1.234.567')).toBe(1234567) + expect(parseCurrencyInput('1,234,567.89')).toBe(1234567.89) + expect(parseCurrencyInput('12,34,567')).toBe(1234567) + }) + + it('rejects values carrying no amount', () => { + expect(parseCurrencyInput('')).toBeNull() + expect(parseCurrencyInput(' ')).toBeNull() + expect(parseCurrencyInput('n/a')).toBeNull() + expect(parseCurrencyInput('$')).toBeNull() + expect(parseCurrencyInput(null)).toBeNull() + expect(parseCurrencyInput(undefined)).toBeNull() + expect(parseCurrencyInput(true)).toBeNull() + expect(parseCurrencyInput(['12'])).toBeNull() + expect(parseCurrencyInput({ amount: 12 })).toBeNull() + }) + + it('round-trips its own display output', () => { + for (const amount of [0, 12, -12.5, 1234.56, 1234567.89]) { + const rendered = formatCurrencyDisplay(amount, 'USD', 'en-US') + expect(parseCurrencyInput(rendered)).toBe(amount) + } + }) +}) + +describe('formatCurrencyDisplay', () => { + it('uses the currency’s own symbol and fraction digits', () => { + expect(formatCurrencyDisplay(1234.56, 'USD', 'en-US')).toBe('$1,234.56') + // JPY has no minor unit, so it rounds to whole yen. + expect(formatCurrencyDisplay(1234.56, 'JPY', 'en-US')).toBe('¥1,235') + }) + + it('falls back to the default currency when the column declares none', () => { + expect(formatCurrencyDisplay(12, undefined, 'en-US')).toBe( + formatCurrencyDisplay(12, DEFAULT_CURRENCY_CODE, 'en-US') + ) + }) + + it('renders an unreadable value verbatim rather than blanking it', () => { + // What a string → currency conversion can leave behind. + expect(formatCurrencyDisplay('pending', 'USD', 'en-US')).toBe('pending') + expect(formatCurrencyDisplay(null, 'USD', 'en-US')).toBe('') + }) + + it('formats a stored string as if it were the number it encodes', () => { + expect(formatCurrencyDisplay('1234.56', 'USD', 'en-US')).toBe('$1,234.56') + }) +}) + +describe('formatCurrencyForInput', () => { + it('renders the bare amount, with no symbol or grouping', () => { + expect(formatCurrencyForInput(1234.56)).toBe('1234.56') + expect(formatCurrencyForInput('$1,234.56')).toBe('1234.56') + expect(formatCurrencyForInput(null)).toBe('') + expect(formatCurrencyForInput(undefined)).toBe('') + }) + + it('keeps unparseable text so an edit does not silently erase it', () => { + expect(formatCurrencyForInput('pending')).toBe('pending') + }) +}) + +describe('currency codes', () => { + it('accepts ISO 4217 codes case-insensitively and rejects the rest', () => { + expect(isSupportedCurrencyCode('USD')).toBe(true) + expect(isSupportedCurrencyCode('eur')).toBe(true) + expect(isSupportedCurrencyCode('ZZZ')).toBe(false) + expect(isSupportedCurrencyCode('US')).toBe(false) + expect(isSupportedCurrencyCode('DOLLAR')).toBe(false) + expect(isSupportedCurrencyCode('')).toBe(false) + }) + + it('upper-cases and defaults', () => { + expect(resolveCurrencyCode('eur')).toBe('EUR') + expect(resolveCurrencyCode(undefined)).toBe(DEFAULT_CURRENCY_CODE) + expect(resolveCurrencyCode('')).toBe(DEFAULT_CURRENCY_CODE) + }) + + it('offers the pinned codes first and no duplicates', () => { + const options = getCurrencyOptions() + expect(options[0].code).toBe('USD') + const codes = options.map((c) => c.code) + expect(new Set(codes).size).toBe(codes.length) + expect(codes).toContain('JPY') + }) +}) diff --git a/apps/sim/lib/table/__tests__/validation.test.ts b/apps/sim/lib/table/__tests__/validation.test.ts index f6b8be7dc3..fc3b77ed57 100644 --- a/apps/sim/lib/table/__tests__/validation.test.ts +++ b/apps/sim/lib/table/__tests__/validation.test.ts @@ -131,6 +131,38 @@ describe('Validation', () => { }) }) + describe('validateColumnDefinition — currency', () => { + const base: ColumnDefinition = { name: 'price', type: 'currency' } + + it('accepts a currency column with no code (it defaults on write)', () => { + expect(validateColumnDefinition(base).valid).toBe(true) + }) + + it('accepts a supported ISO 4217 code', () => { + expect(validateColumnDefinition({ ...base, currencyCode: 'JPY' }).valid).toBe(true) + }) + + it('rejects a code no runtime can format', () => { + const result = validateColumnDefinition({ ...base, currencyCode: 'ZZZ' }) + expect(result.valid).toBe(false) + expect(result.errors.join(' ')).toContain('invalid currency code') + }) + + it('rejects a currency code stashed on a non-currency column', () => { + const result = validateColumnDefinition({ + name: 'price', + type: 'number', + currencyCode: 'USD', + }) + expect(result.valid).toBe(false) + expect(result.errors.join(' ')).toContain('cannot define a currency') + }) + + it('allows a unique constraint, unlike select', () => { + expect(validateColumnDefinition({ ...base, unique: true }).valid).toBe(true) + }) + }) + describe('validateTableSchema', () => { it('should accept valid schema', () => { const schema: TableSchema = { @@ -470,6 +502,40 @@ describe('Validation', () => { expect(patch.tags).toEqual(['opt_a', 'opt_b']) }) }) + + describe('currency coercion', () => { + const currencySchema: TableSchema = { + columns: [ + { id: 'price', name: 'price', type: 'currency', currencyCode: 'USD' }, + { id: 'cost', name: 'cost', type: 'currency', currencyCode: 'EUR', required: true }, + ], + } + + it('parses a formatted amount down to a bare number', () => { + const patch: Record = { price: '$1,234.56' } + coerceRowValues(patch as never, currencySchema) + expect(patch.price).toBe(1234.56) + }) + + it('leaves an already-numeric cell untouched', () => { + const patch: Record = { price: 42 } + coerceRowValues(patch as never, currencySchema) + expect(patch.price).toBe(42) + }) + + it('nulls an unreadable amount on an optional column', () => { + const patch: Record = { price: 'ask sales' } + coerceRowValues(patch as never, currencySchema) + expect(patch.price).toBeNull() + }) + + it('leaves an unreadable amount in place on a required column so validation reports it', () => { + const patch: Record = { cost: 'ask sales' } + coerceRowValues(patch as never, currencySchema) + expect(patch.cost).toBe('ask sales') + expect(validateRowAgainstSchema(patch as never, currencySchema).valid).toBe(false) + }) + }) }) describe('getUniqueColumns', () => { diff --git a/apps/sim/lib/table/cell-format.ts b/apps/sim/lib/table/cell-format.ts index 9fe544885e..2461cb286e 100644 --- a/apps/sim/lib/table/cell-format.ts +++ b/apps/sim/lib/table/cell-format.ts @@ -11,6 +11,7 @@ */ import { getColumnId } from '@/lib/table/column-keys' +import { columnTypeOf } from '@/lib/table/column-types' import { selectValueToNames } from '@/lib/table/select-values' import type { ColumnDefinition, JsonValue, RowData } from '@/lib/table/types' @@ -24,7 +25,7 @@ import type { ColumnDefinition, JsonValue, RowData } from '@/lib/table/types' * must reach consumers byte-identical to what is stored. */ export function formatCellValue(value: unknown, column: ColumnDefinition): JsonValue { - if (column.type === 'select') return selectValueToNames(column, value) + if (columnTypeOf(column).storesOpaqueIds) return selectValueToNames(column, value) return value as JsonValue } diff --git a/apps/sim/lib/table/column-naming.ts b/apps/sim/lib/table/column-naming.ts index 1124e8da2d..eba112ee0d 100644 --- a/apps/sim/lib/table/column-naming.ts +++ b/apps/sim/lib/table/column-naming.ts @@ -6,6 +6,7 @@ * get from the sidebar. */ +import { COLUMN_TYPE_REGISTRY, isColumnType } from '@/lib/table/column-types' import type { ColumnDefinition } from '@/lib/table/types' /** @@ -43,14 +44,11 @@ export function deriveOutputColumnName(path: string, taken: Set): string * union falls back to `json`, the most permissive shape that still validates. */ export function columnTypeForLeaf(leafType: string | undefined): ColumnDefinition['type'] { - switch (leafType) { - case 'string': - case 'number': - case 'boolean': - case 'date': - case 'json': - return leafType - default: - return 'json' + // A block output can only land on a type that carries no configuration of its + // own — a `select` needs an option set and a `currency` needs a code, neither + // of which a leaf type supplies. + if (isColumnType(leafType) && COLUMN_TYPE_REGISTRY[leafType].ownedMetadata.length === 0) { + return leafType } + return 'json' } diff --git a/apps/sim/lib/table/column-types/boolean.ts b/apps/sim/lib/table/column-types/boolean.ts new file mode 100644 index 0000000000..e3f5aacc62 --- /dev/null +++ b/apps/sim/lib/table/column-types/boolean.ts @@ -0,0 +1,40 @@ +import { TypeBoolean } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' + +export const booleanColumnType: ColumnTypeDefinition = { + id: 'boolean', + label: 'Boolean', + icon: TypeBoolean, + jsonbCast: null, + storesOpaqueIds: false, + supportsUnique: true, + sampleValue: true, + ownedMetadata: [], + workflowInputType: 'boolean', + // Toggled in place on click, Enter, and fill — never opens an editor, so it + // has no `typeaheadPattern` and the expanded popover skips it entirely. + editor: 'toggle', + expandable: false, + + coerce(value) { + if (typeof value === 'boolean') return { ok: true, value } + if (typeof value === 'string') { + const normalized = value.trim().toLowerCase() + if (normalized === 'true') return { ok: true, value: true } + if (normalized === 'false') return { ok: true, value: false } + } + return { ok: false } + }, + + validateCell(value, column) { + return typeof value === 'boolean' ? null : `${column.name} must be boolean` + }, + + formatForDisplay(value) { + return String(value) + }, + + formatForInput(value) { + return String(value) + }, +} diff --git a/apps/sim/lib/table/column-types/currency.ts b/apps/sim/lib/table/column-types/currency.ts new file mode 100644 index 0000000000..16ee515f83 --- /dev/null +++ b/apps/sim/lib/table/column-types/currency.ts @@ -0,0 +1,68 @@ +import { TypeCurrency } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' +import { + formatCurrencyDisplay, + formatCurrencyForInput, + isSupportedCurrencyCode, + parseCurrencyInput, + resolveCurrencyCode, +} from '@/lib/table/currency' + +export const currencyColumnType: ColumnTypeDefinition = { + id: 'currency', + label: 'Currency', + icon: TypeCurrency, + jsonbCast: 'numeric', + storesOpaqueIds: false, + supportsUnique: true, + sampleValue: 123, + ownedMetadata: ['currencyCode'], + workflowInputType: 'number', + editor: 'text', + expandable: false, + inputMode: 'decimal', + acceptsFormattedInput: true, + // Also accepts the grouping separator and the symbol the user is likely to + // type first — `parseCurrencyInput` strips both. + typeaheadPattern: /[\d.,\-\p{Sc}]/u, + parseErrorMessage: 'Invalid amount', + + coerce(value) { + // Stored as a bare number, but accepts the formatted shapes an amount + // arrives in — `$1,234.56`, `1 234,56 €`, `(12.00)` — so a paste, CSV + // import, or tool write lands as a number rather than being nulled. + const parsed = parseCurrencyInput(value) + return parsed === null ? { ok: false } : { ok: true, value: parsed } + }, + + validateCell(value, column) { + // A currency cell is a plain number; `currencyCode` is display metadata and + // never part of the stored value. + return typeof value === 'number' && !Number.isNaN(value) + ? null + : `${column.name} must be number` + }, + + validateDefinition(column) { + if (column.currencyCode !== undefined && !isSupportedCurrencyCode(column.currencyCode)) { + return [ + `Column "${column.name}" has invalid currency code "${column.currencyCode}". Use an ISO 4217 code, e.g. USD`, + ] + } + return [] + }, + + formatForDisplay(value, column) { + return formatCurrencyDisplay(value, column.currencyCode) + }, + + formatForInput(value) { + return formatCurrencyForInput(value) + }, + + defaultMetadata(column) { + // Always stamped, so the schema states the rendered currency explicitly + // rather than leaving readers to know the default. + return { currencyCode: resolveCurrencyCode(column.currencyCode) } + }, +} diff --git a/apps/sim/lib/table/column-types/date.ts b/apps/sim/lib/table/column-types/date.ts new file mode 100644 index 0000000000..11b980eeac --- /dev/null +++ b/apps/sim/lib/table/column-types/date.ts @@ -0,0 +1,63 @@ +import { Calendar as CalendarIcon } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' +import { + formatDateCellDisplay, + normalizeDateCellValue, + storedDateToEditable, +} from '@/lib/table/dates' +import type { JsonValue } from '@/lib/table/types' + +export const dateColumnType: ColumnTypeDefinition = { + id: 'date', + label: 'Date', + icon: CalendarIcon, + jsonbCast: 'timestamptz', + storesOpaqueIds: false, + supportsUnique: true, + sampleValue: '2024-01-31', + ownedMetadata: [], + workflowInputType: 'string', + editor: 'date', + expandable: false, + typeaheadPattern: /[\d\-/]/, + parseErrorMessage: 'Invalid date', + + coerce(value) { + if (typeof value === 'string') { + const normalized = normalizeDateCellValue(value) + return normalized === null ? { ok: false } : { ok: true, value: normalized } + } + // Date instances and epoch numbers may still be out of the representable + // range (>±8.64e15ms) — guard `toISOString()`, which throws RangeError on + // an Invalid Date, so an over-range value degrades to `{ ok: false }` + // rather than crashing the write. + const date = value instanceof Date ? value : typeof value === 'number' ? new Date(value) : null + if (date && !Number.isNaN(date.getTime())) return { ok: true, value: date.toISOString() } + return { ok: false } + }, + + isCompatibleWith(value) { + // Stricter than `coerce` on purpose. Writing a number into a date cell is a + // deliberate act — the caller means epoch milliseconds. Reinterpreting a + // whole NUMBER column as epochs is not: a column of 1, 5, 42 would become + // three timestamps in January 1970, irreversibly, and a Unix-seconds column + // would land in 1970 rather than the year it means. Refuse the bulk + // conversion; single writes still accept epochs. + if (typeof value === 'number') return false + return dateColumnType.coerce(value as JsonValue, { name: '', type: 'date' }).ok + }, + + validateCell(value, column) { + const valid = + value instanceof Date || (typeof value === 'string' && !Number.isNaN(Date.parse(value))) + return valid ? null : `${column.name} must be valid date` + }, + + formatForDisplay(value) { + return formatDateCellDisplay(String(value), { seconds: true }) + }, + + formatForInput(value) { + return storedDateToEditable(String(value)) + }, +} diff --git a/apps/sim/lib/table/column-types/index.ts b/apps/sim/lib/table/column-types/index.ts new file mode 100644 index 0000000000..9aebb2b100 --- /dev/null +++ b/apps/sim/lib/table/column-types/index.ts @@ -0,0 +1,21 @@ +/** + * Column-type registry barrel. + * + * Client-safe — importing this never pulls `@sim/db` or `drizzle-orm`. Server + * code that needs the retype cell migrations imports `registry.server` instead. + * + * Deliberately NOT re-exported from `@/lib/table`: 44 server modules import + * that barrel, and routing this through it would pull `@sim/emcn/icons` into + * every one of them. Import `@/lib/table/column-types` directly, the same way + * `column-keys`, `constants`, and `dates` are already imported. + */ + +export * from '@/lib/table/column-types/registry' +export type { + CoerceResult, + ColumnCellEditor, + ColumnType, + ColumnTypeDefinition, + TypeSpecificColumnKey, +} from '@/lib/table/column-types/types' +export { TYPE_SPECIFIC_COLUMN_KEYS } from '@/lib/table/column-types/types' diff --git a/apps/sim/lib/table/column-types/json.ts b/apps/sim/lib/table/column-types/json.ts new file mode 100644 index 0000000000..027f57cea6 --- /dev/null +++ b/apps/sim/lib/table/column-types/json.ts @@ -0,0 +1,38 @@ +import { TypeJson } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' + +export const jsonColumnType: ColumnTypeDefinition = { + id: 'json', + label: 'JSON', + icon: TypeJson, + jsonbCast: null, + storesOpaqueIds: false, + supportsUnique: true, + sampleValue: 'value', + ownedMetadata: [], + workflowInputType: 'object', + editor: 'text', + expandable: true, + + coerce(value) { + // Anything JSON-serializable is already valid — this is the widest type. + return { ok: true, value } + }, + + validateCell(value, column) { + try { + JSON.stringify(value) + return null + } catch { + return `${column.name} must be valid JSON` + } + }, + + formatForDisplay(value) { + return JSON.stringify(value) + }, + + formatForInput(value) { + return typeof value === 'string' ? value : JSON.stringify(value) + }, +} diff --git a/apps/sim/lib/table/column-types/number.ts b/apps/sim/lib/table/column-types/number.ts new file mode 100644 index 0000000000..92c5e321ba --- /dev/null +++ b/apps/sim/lib/table/column-types/number.ts @@ -0,0 +1,45 @@ +import { TypeNumber } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' + +export const numberColumnType: ColumnTypeDefinition = { + id: 'number', + label: 'Number', + icon: TypeNumber, + jsonbCast: 'numeric', + storesOpaqueIds: false, + supportsUnique: true, + sampleValue: 123, + ownedMetadata: [], + workflowInputType: 'number', + editor: 'text', + expandable: false, + inputMode: 'decimal', + typeaheadPattern: /[\d.-]/, + parseErrorMessage: 'Invalid number', + + coerce(value) { + if (typeof value === 'number') { + return Number.isFinite(value) ? { ok: true, value } : { ok: false } + } + if (typeof value === 'string' && value.trim() !== '') { + const parsed = Number(value) + return Number.isFinite(parsed) ? { ok: true, value: parsed } : { ok: false } + } + return { ok: false } + }, + + validateCell(value, column) { + return typeof value === 'number' && !Number.isNaN(value) + ? null + : `${column.name} must be number` + }, + + formatForDisplay(value) { + return String(value) + }, + + formatForInput(value) { + if (typeof value === 'object') return JSON.stringify(value) + return String(value) + }, +} diff --git a/apps/sim/lib/table/column-types/registry.server.ts b/apps/sim/lib/table/column-types/registry.server.ts new file mode 100644 index 0000000000..a87eda3604 --- /dev/null +++ b/apps/sim/lib/table/column-types/registry.server.ts @@ -0,0 +1,238 @@ +/** + * Server-only half of the column-type registry: the cell rewrites that run + * inside the retype transaction. + * + * Kept out of `registry.ts` because these need a drizzle transaction, and the + * tables grid imports that module. Mirrors `connectors/registry.server.ts`. + * + * A migration is keyed by *direction*: `migrateCellsTo` runs when a column is + * converted into the type, `migrateCellsFrom` when it is converted out of it. + * `select` needs both — its cells store opaque option ids that mean nothing + * under any other type. `currency` needs only the inbound one. + */ + +import { userTableRows } from '@sim/db/schema' +import { sql } from 'drizzle-orm' +import { COLUMN_TYPE_REGISTRY } from '@/lib/table/column-types/registry' +import type { ColumnType } from '@/lib/table/column-types/types' +import type { + ColumnCellMigration, + ColumnTypeServerEntry, +} from '@/lib/table/column-types/types.server' +import type { DbTransaction } from '@/lib/table/planner' +import type { JsonValue, SelectOption } from '@/lib/table/types' + +/** + * Rewrites a column's cells from stored option **ids** to option **names**, for + * a column that is ceasing to be a `select`. A multi cell joins comma-separated + * — the same shape it exports as. + * + * An id whose option no longer exists becomes null, matching + * {@link selectValueForConversion}, which is what the compatibility check ran + * on. Passing it through instead would leave an opaque `opt_…` in a typed cell + * the check had already accounted as empty. + * + * Set-based: one statement per stored shape, driven by a jsonb id→name map, so + * cost is independent of row count. + */ +async function migrateSelectCellsToNames( + trx: DbTransaction, + tableId: string, + columnKey: string, + options: SelectOption[] +): Promise { + const nameById = JSON.stringify(Object.fromEntries(options.map((o) => [o.id, o.name]))) + await trx.execute( + sql`UPDATE ${userTableRows} + SET data = jsonb_set(data, ARRAY[${columnKey}::text], + COALESCE(${nameById}::jsonb -> (data->>${columnKey}::text), 'null'::jsonb)) + WHERE table_id = ${tableId} + AND jsonb_typeof(data->${columnKey}::text) = 'string'` + ) + await trx.execute( + sql`UPDATE ${userTableRows} + SET data = jsonb_set(data, ARRAY[${columnKey}::text], COALESCE(to_jsonb(( + SELECT string_agg(${nameById}::jsonb ->> e.v, ', ' ORDER BY e.ord) + FROM jsonb_array_elements_text(data->${columnKey}::text) WITH ORDINALITY AS e(v, ord) + WHERE ${nameById}::jsonb ? e.v + )), 'null'::jsonb)) + WHERE table_id = ${tableId} + AND jsonb_typeof(data->${columnKey}::text) = 'array'` + ) +} + +/** Rows rewritten per statement, bounding the size of the jsonb map parameter. */ +const COERCED_WRITE_BACK_BATCH_SIZE = 5000 + +/** + * Writes back the values a conversion's coercion produced. + * + * A retype is allowed exactly when the target type's `coerce` accepts the + * value, and `coerce` frequently *transforms* it — an epoch number becomes an + * ISO date, `$1,234.56` becomes `1234.56`. Without this, the cell keeps its old + * bytes under the new type, and since filters and sorts apply the type's + * `jsonbCast` to whatever is stored, an epoch left in a `date` column makes + * `::timestamptz` fail on EVERY query against that column. + * + * The values arrive already computed (the compatibility scan derived them), so + * this is purely the write. It cannot be expressed set-based — the coercions + * are JS, and a naive SQL equivalent would mangle exactly the inputs they + * disambiguate — so the map is folded into one statement per batch. + */ +export async function writeBackCoercedCells( + trx: DbTransaction, + tableId: string, + columnKey: string, + valueByRowId: ReadonlyMap +): Promise { + if (valueByRowId.size === 0) return + + const entries = [...valueByRowId] + for (let start = 0; start < entries.length; start += COERCED_WRITE_BACK_BATCH_SIZE) { + const batch = JSON.stringify( + Object.fromEntries(entries.slice(start, start + COERCED_WRITE_BACK_BATCH_SIZE)) + ) + await trx.execute( + sql`UPDATE ${userTableRows} AS r + SET data = jsonb_set(r.data, ARRAY[${columnKey}::text], m.value) + FROM jsonb_each(${batch}::jsonb) AS m(key, value) + WHERE r.table_id = ${tableId} AND r.id = m.key` + ) + } +} + +/** + * Rewrites a column's cells into the canonical `select` storage shape: the + * option **id**, wrapped in an array when the column is `multiple`. + * + * Needed in both directions of a select change. Converting *to* select, cells + * hold option names (that is what made them compatible) but every reader — + * pills, filters, exports — resolves by id. Toggling single→multi, cells hold a + * scalar id while multi filters compile to array containment, which a scalar + * never matches. Either way the cell silently drops out until it is re-edited. + * + * Scalar cells count, not just strings: `resolveSelectOptionId` stringifies a + * number or boolean before matching, so a `number` column whose values equal + * option NAMES passes the compatibility gate. Matching only `jsonb_typeof = + * 'string'` left those cells as raw numbers inside a select column, where they + * render as nothing and fail option membership on the next write. `data->>key` + * yields the text form for every scalar, so one widened predicate covers them. + * + * The map keys ids, names, and lower-cased names — ids so re-running is a no-op, + * lower-cased names because `resolveSelectOptionId` accepts a case-mismatched + * name and a cell that passed that check must actually migrate. Duplicate option + * names are rejected case-insensitively at validation, so the folded key is + * unambiguous; lookups still try the exact form first to preserve its precedence. + */ +async function migrateCellsToSelectIds( + trx: DbTransaction, + tableId: string, + columnKey: string, + options: SelectOption[], + multiple: boolean +): Promise { + // Ids are written last so an id always outranks any option's name, matching + // `resolveSelectOptionId`'s id-before-name precedence. A single flat pass + // would let a later option whose *name* equals an earlier option's *id* + // overwrite that id entry and repoint its cells at the wrong option. + // A Map, not plain-object assignment: an option named `__proto__` would set + // the prototype instead of an own key and drop out of the serialized map. + const refs = new Map() + for (const o of options) { + refs.set(o.name.toLowerCase(), o.id) + refs.set(o.name, o.id) + } + for (const o of options) { + refs.set(o.id, o.id) + } + const idByRef = JSON.stringify(Object.fromEntries(refs)) + + if (multiple) { + // A string cell reaching a multi target is either one option name or the + // comma-joined form a multiselect converts to text as. Try the whole string + // first so an option whose own name contains a comma still wins, then split + // — mirroring `splitMultiSelectInput` on the write path, including its + // first-occurrence dedup. + await trx.execute( + sql`UPDATE ${userTableRows} + SET data = jsonb_set(data, ARRAY[${columnKey}::text], + CASE WHEN data->>${columnKey}::text = '' THEN '[]'::jsonb + WHEN COALESCE(${idByRef}::jsonb -> (data->>${columnKey}::text), ${idByRef}::jsonb -> lower(data->>${columnKey}::text)) IS NOT NULL + THEN jsonb_build_array(COALESCE(${idByRef}::jsonb -> (data->>${columnKey}::text), ${idByRef}::jsonb -> lower(data->>${columnKey}::text))) + ELSE COALESCE(( + SELECT jsonb_agg(v ORDER BY ord) FROM ( + SELECT COALESCE(${idByRef}::jsonb -> btrim(part), ${idByRef}::jsonb -> lower(btrim(part)), to_jsonb(btrim(part))) AS v, + min(o) AS ord + FROM unnest(string_to_array(data->>${columnKey}::text, ',')) WITH ORDINALITY AS u(part, o) + WHERE btrim(part) <> '' + GROUP BY 1 + ) d), '[]'::jsonb) + END) + WHERE table_id = ${tableId} + AND jsonb_typeof(data->${columnKey}::text) IN ('string', 'number', 'boolean')` + ) + await trx.execute( + sql`UPDATE ${userTableRows} + SET data = jsonb_set(data, ARRAY[${columnKey}::text], COALESCE(( + SELECT jsonb_agg(COALESCE(${idByRef}::jsonb -> e.v, ${idByRef}::jsonb -> lower(e.v), to_jsonb(e.v)) ORDER BY e.ord) + FROM jsonb_array_elements_text(data->${columnKey}::text) WITH ORDINALITY AS e(v, ord) + ), '[]'::jsonb)) + WHERE table_id = ${tableId} + AND jsonb_typeof(data->${columnKey}::text) = 'array'` + ) + return + } + + // A cleared cell is stored as '' — compatibility lets it through, so it has + // to land as null rather than an '' that fails option membership on the next + // write. + await trx.execute( + sql`UPDATE ${userTableRows} + SET data = jsonb_set(data, ARRAY[${columnKey}::text], + CASE WHEN data->>${columnKey}::text = '' THEN 'null'::jsonb + ELSE COALESCE(${idByRef}::jsonb -> (data->>${columnKey}::text), ${idByRef}::jsonb -> lower(data->>${columnKey}::text), data->${columnKey}::text) + END) + WHERE table_id = ${tableId} + AND jsonb_typeof(data->${columnKey}::text) IN ('string', 'number', 'boolean')` + ) + // Compatibility already rejected multi-valued cells for a single target, so + // any array here holds at most one option. + await trx.execute( + sql`UPDATE ${userTableRows} + SET data = jsonb_set(data, ARRAY[${columnKey}::text], + COALESCE(${idByRef}::jsonb -> (data->${columnKey}::text->>0), ${idByRef}::jsonb -> lower(data->${columnKey}::text->>0), data->${columnKey}::text->0, 'null'::jsonb)) + WHERE table_id = ${tableId} + AND jsonb_typeof(data->${columnKey}::text) = 'array'` + ) +} + +/** + * Every column type plus its migrations. The `Record` + * annotation is the same completeness gate the client-safe registry uses: a + * new type will not compile until it appears here too. + */ +export const COLUMN_TYPE_SERVER_REGISTRY: Record = { + string: COLUMN_TYPE_REGISTRY.string, + number: COLUMN_TYPE_REGISTRY.number, + boolean: COLUMN_TYPE_REGISTRY.boolean, + date: COLUMN_TYPE_REGISTRY.date, + json: COLUMN_TYPE_REGISTRY.json, + select: { + ...COLUMN_TYPE_REGISTRY.select, + migrateCellsTo: ({ trx, tableId, columnKey, target }) => + migrateCellsToSelectIds(trx, tableId, columnKey, target.options ?? [], !!target.multiple), + migrateCellsFrom: ({ trx, tableId, columnKey, previous }) => + migrateSelectCellsToNames(trx, tableId, columnKey, previous.options ?? []), + }, + currency: COLUMN_TYPE_REGISTRY.currency, +} + +/** The inbound migration for a target type, if it has one. */ +export function migrationTo(type: ColumnType): ColumnCellMigration | undefined { + return COLUMN_TYPE_SERVER_REGISTRY[type]?.migrateCellsTo +} + +/** The outbound migration for a source type, if it has one. */ +export function migrationFrom(type: ColumnType): ColumnCellMigration | undefined { + return COLUMN_TYPE_SERVER_REGISTRY[type]?.migrateCellsFrom +} diff --git a/apps/sim/lib/table/column-types/registry.ts b/apps/sim/lib/table/column-types/registry.ts new file mode 100644 index 0000000000..8bc336a1dc --- /dev/null +++ b/apps/sim/lib/table/column-types/registry.ts @@ -0,0 +1,117 @@ +/** + * The column-type registry — one entry per table column type. + * + * Client-safe: this module and everything it imports stay free of `@sim/db`, + * `drizzle-orm`, and `next/server`, so the tables grid can import it directly. + * The server-only half (retype cell migrations) lives in `registry.server.ts`. + * + * ## Adding a column type + * + * 1. Add its id to {@link ColumnType} in `types.ts`. + * 2. Write `column-types/.ts` exporting a `ColumnTypeDefinition`. + * 3. Add it to `COLUMN_TYPE_REGISTRY` below. + * + * Step 3 is not optional and cannot be forgotten: the `Record` + * annotation makes step 1 a **compile error** until the entry exists, and the + * `ColumnTypeDefinition` interface then makes it an error until every field is + * filled in. That is the whole point of this file — adding a type used to mean + * remembering ~40 scattered `switch` arms, each of which failed silently when + * missed. + */ + +import { booleanColumnType } from '@/lib/table/column-types/boolean' +import { currencyColumnType } from '@/lib/table/column-types/currency' +import { dateColumnType } from '@/lib/table/column-types/date' +import { jsonColumnType } from '@/lib/table/column-types/json' +import { numberColumnType } from '@/lib/table/column-types/number' +import { + MULTI_SELECT_OPERATORS, + SINGLE_SELECT_OPERATORS, + selectColumnType, +} from '@/lib/table/column-types/select' +import { stringColumnType } from '@/lib/table/column-types/string' +import type { ColumnType, ColumnTypeDefinition } from '@/lib/table/column-types/types' +import { COLUMN_TYPES, TYPE_SPECIFIC_COLUMN_KEYS } from '@/lib/table/column-types/types' +import type { ColumnDefinition, JsonValue } from '@/lib/table/types' + +export { COLUMN_TYPES } +export { MULTI_SELECT_OPERATORS, SINGLE_SELECT_OPERATORS } + +/** + * Every column type, keyed by id. The annotation is the completeness gate — + * see the module doc. + */ +export const COLUMN_TYPE_REGISTRY: Record = { + string: stringColumnType, + number: numberColumnType, + boolean: booleanColumnType, + date: dateColumnType, + json: jsonColumnType, + select: selectColumnType, + currency: currencyColumnType, +} + +/** Every definition, in the same order as {@link COLUMN_TYPES}. */ +export const ALL_COLUMN_TYPES: readonly ColumnTypeDefinition[] = COLUMN_TYPES.map( + (id) => COLUMN_TYPE_REGISTRY[id] +) + +/** Whether `value` is a known column type. */ +export function isColumnType(value: unknown): value is ColumnType { + // `in` would also match inherited keys, so `'toString'` would type-guard as a + // column type and then resolve to `Function.prototype.toString`. + return typeof value === 'string' && Object.hasOwn(COLUMN_TYPE_REGISTRY, value) +} + +/** + * The definition for a column's type. Falls back to `string` for a column whose + * declared type is not (or is no longer) a known one, so a malformed schema + * renders as text instead of throwing mid-render. + */ +export function columnTypeOf(column: Pick): ColumnTypeDefinition { + return COLUMN_TYPE_REGISTRY[column.type] ?? stringColumnType +} + +/** The definition for a type id, or `string`'s when the id is unknown. */ +export function columnTypeById(type: string | undefined): ColumnTypeDefinition { + return (isColumnType(type) && COLUMN_TYPE_REGISTRY[type]) || stringColumnType +} + +/** + * Whether an existing cell survives a conversion **to** `target`'s type. + * + * Falls back to "whatever the type's `coerce` accepts". Only `select` + * overrides, because its rules (a cleared `''` against `required`, and single + * vs multi cardinality) are about the column, not the value. + */ +export function isValueCompatible(value: unknown, target: ColumnDefinition): boolean { + const definition = columnTypeOf(target) + if (definition.isCompatibleWith) return definition.isCompatibleWith(value, target) + return definition.coerce(value as JsonValue, target).ok +} + +/** This type's own metadata errors; types carrying no metadata report none. */ +export function validateTypeMetadata(column: ColumnDefinition): string[] { + return columnTypeOf(column).validateDefinition?.(column) ?? [] +} + +/** + * A column's type-specific metadata, as a spreadable object. + * + * Callers that copy a column — the API response serializer, the undo snapshot — + * used to name `options`/`multiple`/`currencyCode` by hand, so a new type's + * metadata was stored but silently dropped on the way out. Reading the key list + * keeps them zero-edit. + */ +export function typeMetadataOf(column: ColumnDefinition): Partial { + const metadata: Partial = {} + for (const key of TYPE_SPECIFIC_COLUMN_KEYS) { + if (column[key] !== undefined) Object.assign(metadata, { [key]: column[key] }) + } + return metadata +} + +/** Wire operators a column accepts, or `null` for "all operators". */ +export function filterOperatorsFor(column: ColumnDefinition): ReadonlySet | null { + return columnTypeOf(column).filterOperatorsFor?.(column) ?? null +} diff --git a/apps/sim/lib/table/column-types/select.ts b/apps/sim/lib/table/column-types/select.ts new file mode 100644 index 0000000000..88c3f65ec0 --- /dev/null +++ b/apps/sim/lib/table/column-types/select.ts @@ -0,0 +1,143 @@ +import { TagIcon } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' +import { MAX_SELECT_OPTIONS } from '@/lib/table/constants' +import { + optionIds, + resolveSelectCellValue, + resolveSelectOptionId, + splitMultiSelectInput, +} from '@/lib/table/select-options' +import { selectValueToNames } from '@/lib/table/select-values' +import type { JsonValue } from '@/lib/table/types' + +/** + * Operators that make sense on a `select` column (whose values are opaque option + * ids), split by cardinality. A single-select cell holds one id, so it compares + * for equality; a multi-select cell holds an array of ids, so the question is + * membership — hence contains / does-not-contain. `$eq` against an array cell + * can never be true (`{"t":["a"]} @> {"t":"a"}` is false in Postgres), so + * allowing it would silently match nothing. + */ +export const SINGLE_SELECT_OPERATORS: ReadonlySet = new Set([ + '$eq', + '$ne', + '$in', + '$nin', + '$empty', +]) +export const MULTI_SELECT_OPERATORS: ReadonlySet = new Set([ + '$contains', + '$ncontains', + '$empty', +]) + +export const selectColumnType: ColumnTypeDefinition = { + id: 'select', + label: 'Select', + icon: TagIcon, + // Cells hold opaque option ids; comparison is by id, never by cast. + jsonbCast: null, + storesOpaqueIds: true, + supportsUnique: false, + sampleValue: 'Option', + ownedMetadata: ['options', 'multiple'], + workflowInputType: 'string', + editor: 'select', + expandable: false, + + filterOperatorsFor(column) { + return column.multiple ? MULTI_SELECT_OPERATORS : SINGLE_SELECT_OPERATORS + }, + + coerce(value, column) { + const resolved = resolveSelectCellValue(value, column) + // A multi target always resolves (to `[]` at worst); a single target that + // matches no option has nothing safe to store. + return resolved === null ? { ok: false } : { ok: true, value: resolved } + }, + + validateCell(value, column) { + const ids = optionIds(column) + if (column.multiple) { + if (!Array.isArray(value)) return `${column.name} must be a list of options` + if (!value.every((v) => typeof v === 'string' && ids.has(v))) { + return `${column.name} must only contain defined options` + } + if (column.required && value.length === 0) return `Missing required field: ${column.name}` + return null + } + return typeof value === 'string' && ids.has(value) + ? null + : `${column.name} must be one of the defined options` + }, + + validateDefinition(column) { + const errors: string[] = [] + const options = column.options + if (!Array.isArray(options) || options.length === 0) { + errors.push( + `Column "${column.name}" of type "${column.type}" must define at least one option` + ) + return errors + } + if (options.length > MAX_SELECT_OPTIONS) { + errors.push(`Column "${column.name}" cannot have more than ${MAX_SELECT_OPTIONS} options`) + } + + const ids = new Set() + const names = new Set() + for (const opt of options) { + if (!opt.id || typeof opt.id !== 'string') { + errors.push(`Column "${column.name}" has an option missing an id`) + } else if (ids.has(opt.id)) { + errors.push(`Column "${column.name}" has duplicate option id "${opt.id}"`) + } else { + ids.add(opt.id) + } + if (!opt.name || typeof opt.name !== 'string') { + errors.push(`Column "${column.name}" has an option missing a name`) + } else { + const key = opt.name.toLowerCase() + if (names.has(key)) { + errors.push(`Column "${column.name}" has duplicate option name "${opt.name}"`) + } else { + names.add(key) + } + } + } + return errors + }, + + isCompatibleWith(value, target) { + // A cleared select cell is written as '' — still convertible, unless the + // target is required. Required only rejects null/undefined on a write, so + // a required string column legitimately holds ''; the migration turns that + // into null (or [] for a multi), and every later update of that row would + // then fail its own required check. + if (value === '') return !target.required + // Read the value exactly as the write-path coercion will. A multi target + // splits a comma-delimited string, so a multiselect → text → multiselect + // round-trip (text holding this feature's own `Bug, Docs` export shape) + // stays convertible instead of being rejected as one unknown option. + const parts = target.multiple + ? splitMultiSelectInput(value as JsonValue) + : Array.isArray(value) + ? value + : [value] + // A single-select target can't hold several options. `updateColumnOptions` + // blocks the same transition; without this the next coerce would silently + // keep only the first id. + if (!target.multiple && parts.length > 1) return false + const options = target.options ?? [] + return parts.every((v) => resolveSelectOptionId(v as JsonValue, options) !== null) + }, + + formatForDisplay(value, column) { + const resolved = selectValueToNames(column, value) + return Array.isArray(resolved) ? resolved.join(', ') : (resolved ?? '') + }, + + formatForInput(value, column) { + return selectColumnType.formatForDisplay(value, column) + }, +} diff --git a/apps/sim/lib/table/column-types/string.ts b/apps/sim/lib/table/column-types/string.ts new file mode 100644 index 0000000000..4dc531c7b3 --- /dev/null +++ b/apps/sim/lib/table/column-types/string.ts @@ -0,0 +1,39 @@ +import { TypeText } from '@sim/emcn/icons' +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' + +export const stringColumnType: ColumnTypeDefinition = { + id: 'string', + label: 'Text', + icon: TypeText, + jsonbCast: null, + storesOpaqueIds: false, + supportsUnique: true, + sampleValue: 'example', + ownedMetadata: [], + workflowInputType: 'string', + editor: 'text', + expandable: true, + + coerce(value) { + if (typeof value === 'string') return { ok: true, value } + if (typeof value === 'number' || typeof value === 'boolean') { + return { ok: true, value: String(value) } + } + return { ok: false } + }, + + validateCell(value, column) { + return typeof value === 'string' ? null : `${column.name} must be string, got ${typeof value}` + }, + + formatForDisplay(value) { + if (typeof value === 'string') return value + if (value === null || value === undefined) return '' + return JSON.stringify(value) + }, + + formatForInput(value) { + if (typeof value === 'object') return JSON.stringify(value) + return String(value) + }, +} diff --git a/apps/sim/lib/table/column-types/types.server.ts b/apps/sim/lib/table/column-types/types.server.ts new file mode 100644 index 0000000000..c650c31c72 --- /dev/null +++ b/apps/sim/lib/table/column-types/types.server.ts @@ -0,0 +1,47 @@ +/** + * The server-only half of a column type: rewriting stored cells when a column + * is converted into or out of this type. + * + * Separate from `types.ts` so the client-safe definition never references a + * drizzle transaction type. Mirrors `connectors/`'s `ConnectorMeta` / + * `ConnectorConfig` split. + */ + +import type { ColumnTypeDefinition } from '@/lib/table/column-types/types' +import type { DbTransaction } from '@/lib/table/planner' +import type { ColumnDefinition, JsonValue } from '@/lib/table/types' + +export interface ColumnCellMigrationContext { + trx: DbTransaction + tableId: string + /** JSONB storage key for the column (its stable id). */ + columnKey: string + /** The column definition as it was before the conversion. */ + previous: ColumnDefinition + /** The column definition the table will end up with. */ + target: ColumnDefinition + /** + * Row id → replacement value, for types whose migration is computed in JS + * during the compatibility scan rather than expressed set-based in SQL. + */ + resolved: ReadonlyMap +} + +export type ColumnCellMigration = (context: ColumnCellMigrationContext) => Promise + +export interface ColumnTypeServerDefinition { + /** + * Rewrites cells into this type's canonical storage shape when a column is + * converted **to** it. Omitted when the stored bytes are already correct. + */ + readonly migrateCellsTo?: ColumnCellMigration + /** + * Rewrites cells out of this type's storage shape when a column is converted + * **away from** it. Needed when the stored value is meaningless under any + * other type — a `select`'s option ids, for instance. + */ + readonly migrateCellsFrom?: ColumnCellMigration +} + +/** A column type plus its server-only migrations. */ +export type ColumnTypeServerEntry = ColumnTypeDefinition & ColumnTypeServerDefinition diff --git a/apps/sim/lib/table/column-types/types.ts b/apps/sim/lib/table/column-types/types.ts new file mode 100644 index 0000000000..0de148ac1d --- /dev/null +++ b/apps/sim/lib/table/column-types/types.ts @@ -0,0 +1,199 @@ +/** + * The shape of a table column type. + * + * Everything that varies per column type — how it looks, how it stores, how it + * coerces, how it compares in SQL — lives on one of these, so adding a type is + * "write one file and register it" rather than finding ~40 `switch` arms. + * + * Split in two, on the axis that actually constrains us: + * + * - {@link ColumnTypeDefinition} is **client-safe**. It may carry a React icon + * (an icon is a component *reference*; server code never calls it, and + * `scripts/check-client-boundary-imports.ts` only forbids calling a + * `'use client'` export from a server surface). It must NOT reach `@sim/db`, + * `drizzle-orm`, or `next/server` — the tables grid imports it directly. + * - `ColumnTypeServerDefinition` (in `types.server.ts`) adds the one genuinely + * server-only concern: rewriting stored cells inside a transaction. + * + * This mirrors `connectors/types.ts`'s `ConnectorMeta` / `ConnectorConfig` + * split and its `registry.ts` / `registry.server.ts` pair. + */ + +import type React from 'react' +import type { ColumnDefinition, JsonValue } from '@/lib/table/types' + +/** + * Every column type id, in picker order. Declared here — not derived from the + * registry — so `constants.ts` can re-export it without dragging the registry's + * icon imports into the 44 server modules that read the `@/lib/table` barrel. + * + * The registry's `Record` annotation is what keeps the two in + * step: adding an id here fails compilation until both registries have an entry. + */ +export const COLUMN_TYPES = [ + 'string', + 'number', + 'currency', + 'boolean', + 'date', + 'json', + 'select', +] as const + +export type ColumnType = (typeof COLUMN_TYPES)[number] + +/** Which inline editor the grid mounts for a cell of this type. */ +export type ColumnCellEditor = + /** Single-line text input. Numeric types additionally set `inputMode`. */ + | 'text' + /** Calendar + time picker. */ + | 'date' + /** Option dropdown. */ + | 'select' + /** Not editable inline — the grid toggles it in place instead. */ + | 'toggle' + +/** + * Optional `ColumnDefinition` keys that belong to a specific column type rather + * than to every column. Each type declares which it owns; a key appearing on + * any other type is rejected generically, so adding metadata for a new type + * means extending this list and that type's `ownedMetadata` — not editing the + * validator. + */ +export const TYPE_SPECIFIC_COLUMN_KEYS = ['options', 'multiple', 'currencyCode'] as const + +export type TypeSpecificColumnKey = (typeof TYPE_SPECIFIC_COLUMN_KEYS)[number] + +/** Result of coercing a raw value toward a column's declared type. */ +export type CoerceResult = { ok: true; value: JsonValue } | { ok: false } + +export interface ColumnTypeDefinition { + readonly id: ColumnType + + /** Human label in the type picker, column header menu, and docs. */ + readonly label: string + /** Type icon. A component reference only — never invoked server-side. */ + readonly icon: React.ComponentType<{ className?: string }> + /** + * Postgres cast needed to compare this type's JSONB text, or `null` when text + * comparison is correct. Single source for both filter ranges and sort order. + */ + readonly jsonbCast: 'numeric' | 'timestamptz' | null + + /** + * Wire operators a column of this type accepts, or `null` for "all + * operators". Only types whose stored value is opaque (a `select`'s option + * id) need to restrict. Takes the column, so a type whose answer depends on + * its own configuration — select's single vs multi cardinality — owns that + * rule instead of the registry special-casing it. + */ + filterOperatorsFor?(column: ColumnDefinition): ReadonlySet | null + + /** + * True when the stored value is an opaque identifier that must be resolved to + * a display label for search, filtering, export, and clipboard. Only `select` + * sets this; it is why those paths special-case it. + */ + readonly storesOpaqueIds: boolean + + /** + * Whether a column of this type can carry a `unique` constraint. False for + * types whose stored value is opaque: uniqueness would compare the stored + * option id, capping each option at one row for the whole table. + */ + readonly supportsUnique: boolean + + /** + * A representative value, used to show an LLM what this column's cells look + * like. Keeps prompt examples from restating per-type knowledge. + */ + readonly sampleValue: JsonValue + + /** + * Optional `ColumnDefinition` keys this type owns. Any type-specific key + * present on a column of a *different* type is rejected, generically — a + * stored `multiple` or `currencyCode` on the wrong type is inert until a + * later conversion inherits it and silently overrides what that request + * asked for. Declaring ownership here is what lets a new type add metadata + * without touching the validator. + */ + readonly ownedMetadata: readonly TypeSpecificColumnKey[] + + /** Workflow/block param type a column of this type maps onto. */ + readonly workflowInputType: 'string' | 'number' | 'boolean' | 'object' + + /** Inline editor variant. */ + readonly editor: ColumnCellEditor + /** + * Whether double-clicking a cell opens the large expanded popover instead of + * the compact inline editor. True for free-form prose (`string`, `json`) + * where a cell can hold far more than one line; false for types with a + * bounded, structured value. + */ + readonly expandable: boolean + /** `inputMode` for the text editor, when the type wants a specific keypad. */ + readonly inputMode?: 'decimal' + /** + * Whether the editor must accept text an `` would reject. + * A currency cell legitimately takes `$1,234.56` or `1.234,56`, so it needs a + * text input with a numeric keypad; a plain number takes neither and keeps + * the native numeric input with its spinner and validation. + */ + readonly acceptsFormattedInput?: boolean + /** + * Keys that may start a type-ahead edit. Absent means any printable key + * starts one — only types that parse their input restrict it, so a stray + * letter can't open an editor whose draft could never save. + */ + readonly typeaheadPattern?: RegExp + /** + * Message shown when a draft cannot be parsed. Absent means any text is + * valid, so a draft always saves. + */ + readonly parseErrorMessage?: string + + /** + * Coerces a non-null raw value toward this type. The single write-path + * implementation — the server calls it before persisting and the grid calls + * it to fill the optimistic cache, so the two can no longer disagree. + */ + coerce(value: JsonValue, column: ColumnDefinition): CoerceResult + + /** Validates a stored cell's shape. Returns an error message, or null when valid. */ + validateCell(value: JsonValue, column: ColumnDefinition): string | null + + /** + * Validates this type's own column metadata (a `select`'s options, a + * `currency`'s code). Omitted by types that carry none. + */ + validateDefinition?(column: ColumnDefinition): string[] + + /** + * Whether an existing cell survives a conversion **to** this type. + * + * Defaults to "whatever {@link coerce} accepts", which is what makes the + * retype gate and the write path incapable of disagreeing — a gate that is + * more permissive than the write path reports zero incompatible rows and + * then rewrites every one of them. + * + * An override may only ever be **stricter** than `coerce`, never looser. + * Stricter is safe: the bulk conversion is refused while individual writes + * still work. Looser is the direction that corrupts data. Use it when a + * coercion that is reasonable for a single deliberate write would be + * destructive applied to a whole column at once. + */ + isCompatibleWith?(value: unknown, target: ColumnDefinition): boolean + + /** Stored value → display text (grid cell, CSV, clipboard, width measurement). */ + formatForDisplay(value: unknown, column: ColumnDefinition): string + + /** Stored value → the text an editor input starts with. */ + formatForInput(value: unknown, column: ColumnDefinition): string + + /** + * Metadata stamped onto a newly created column of this type, so the schema + * states the type's configuration explicitly instead of leaving readers to + * know the default. Returns nothing for types that carry no metadata. + */ + defaultMetadata?(column: ColumnDefinition): Partial +} diff --git a/apps/sim/lib/table/columns/service.ts b/apps/sim/lib/table/columns/service.ts index 4226dc935f..d8f627fc11 100644 --- a/apps/sim/lib/table/columns/service.ts +++ b/apps/sim/lib/table/columns/service.ts @@ -12,9 +12,22 @@ import { db } from '@sim/db' import { userTableDefinitions, userTableRows } from '@sim/db/schema' import { createLogger } from '@sim/logger' +import { omit } from '@sim/utils/object' import { and, count, eq, sql } from 'drizzle-orm' import { columnMatchesRef, generateColumnId, getColumnId } from '@/lib/table/column-keys' +import { + columnTypeById, + columnTypeOf, + isValueCompatible, + TYPE_SPECIFIC_COLUMN_KEYS, +} from '@/lib/table/column-types' +import { + migrationFrom, + migrationTo, + writeBackCoercedCells, +} from '@/lib/table/column-types/registry.server' import { COLUMN_TYPES, NAME_PATTERN, TABLE_LIMITS } from '@/lib/table/constants' +import { resolveCurrencyCode } from '@/lib/table/currency' import { assertColumnDestructive, assertSchemaMutable } from '@/lib/table/mutation-locks' import type { DbTransaction } from '@/lib/table/planner' import { stripGroupExecutions } from '@/lib/table/rows/executions' @@ -32,14 +45,11 @@ import type { TableMetadata, TableSchema, UpdateColumnConstraintsData, + UpdateColumnCurrencyData, UpdateColumnOptionsData, UpdateColumnTypeData, } from '@/lib/table/types' -import { - resolveSelectOptionId, - splitMultiSelectInput, - validateColumnDefinition, -} from '@/lib/table/validation' +import { validateColumnDefinition } from '@/lib/table/validation' import { assertValidSchema, stripGroupDeps } from '@/lib/table/workflow-columns' const logger = createLogger('TableColumnService') @@ -64,6 +74,7 @@ export async function addTableColumn( position?: number options?: SelectOption[] multiple?: boolean + currencyCode?: string }, requestId: string ): Promise { @@ -108,6 +119,7 @@ export async function addTableColumn( unique: column.unique ?? false, ...(column.options ? { options: column.options } : {}), ...(column.multiple ? { multiple: true } : {}), + ...columnTypeById(column.type).defaultMetadata?.(column as ColumnDefinition), } const columnValidation = validateColumnDefinition(newColumn) @@ -474,6 +486,181 @@ export async function deleteColumns( return def } +/** + * Validates a constraint change against the column's stored data, and returns + * the column with those constraints applied. + * + * Shared by every write that can carry constraints, for the reason the + * duplicate scan and {@link countEmptyCells} are shared: three copies of these + * rules is the drift that produced the original required-check bug. Applying + * them in the same write as the change they accompany is what stops a combined + * request from committing one half and then failing on the other. + */ +async function applyConstraints( + trx: DbTransaction, + tableId: string, + column: ColumnDefinition, + columnKey: string, + data: { required?: boolean; unique?: boolean } +): Promise { + if (data.required === undefined && data.unique === undefined) return column + + if (column.workflowGroupId) { + throw new Error( + `Cannot change constraints on workflow-output column "${column.name}". Constraints aren't applicable to columns whose values come from workflow execution.` + ) + } + if (data.required === true && !column.required) { + const emptyCount = await countEmptyCells(trx, tableId, columnKey) + if (emptyCount > 0) { + throw new Error( + `Cannot set column "${column.name}" as required: ${emptyCount} row(s) have null, missing, or empty values` + ) + } + } + if (data.unique === true && !column.unique) { + if (!columnTypeOf(column).supportsUnique) { + throw new Error( + `Cannot set column "${column.name}" as unique: ${column.type} columns compare stored values that would allow only one row per value.` + ) + } + if (await hasDuplicateValues(trx, tableId, columnKey)) { + throw new Error(`Cannot set column "${column.name}" as unique: duplicate values exist`) + } + } + return { + ...column, + ...(data.required !== undefined ? { required: data.required } : {}), + ...(data.unique !== undefined ? { unique: data.unique } : {}), + } +} + +/** Persists a column list as the table's schema and returns the updated definition. */ +async function persistColumns( + trx: DbTransaction, + table: TableDefinition, + columns: ColumnDefinition[] +): Promise { + const updatedSchema: TableSchema = { ...table.schema, columns } + const now = new Date() + await trx + .update(userTableDefinitions) + .set({ schema: updatedSchema, updatedAt: now }) + .where(eq(userTableDefinitions.id, table.id)) + return { ...table, schema: updatedSchema, updatedAt: now } +} + +/** + * Whether any two rows share a stored value in this column. + * + * Shared by the constraint write and the retype's pre-validation so the two + * cannot drift — the same reason {@link countEmptyCells} is shared. A retype + * that sets `unique` in the same request has to run this against the values the + * conversion is ABOUT to write, not the ones on disk: coercing `"5"` and `"5.0"` + * to a number manufactures a duplicate that no pre-scan of the raw text sees. + */ +async function hasDuplicateValues( + trx: DbTransaction, + tableId: string, + columnKey: string +): Promise { + const duplicates = (await trx.execute( + sql`SELECT ${userTableRows.data}->>${columnKey}::text AS val, count(*) AS cnt FROM ${userTableRows} WHERE table_id = ${tableId} AND ${userTableRows.data} ? ${columnKey} AND ${userTableRows.data}->>${columnKey}::text IS NOT NULL GROUP BY val HAVING count(*) > 1 LIMIT 1` + )) as { val: string; cnt: number }[] + return duplicates.length > 0 +} + +/** + * Validates a pending rename against the schema it will land in, and returns + * the renamed column. + * + * Exists so a rename can be folded into whatever OTHER column write a request + * carries, inside that write's transaction. Each write is its own locked + * transaction, so a standalone rename alongside one of them means either order + * can commit and then fail — and since a rename is metadata-only (rows key on + * the stable column id), there is nothing forcing it to be its own write. + * + * Returns the column unchanged when there is no rename to apply. Exported so + * the collision and name-shape rules are testable without a transaction. + */ +export function applyPendingRename( + columns: ColumnDefinition[], + columnIndex: number, + newName: string | undefined +): ColumnDefinition { + const column = columns[columnIndex] + if (newName === undefined || newName === column.name) return column + + if (!NAME_PATTERN.test(newName)) { + throw new Error( + `Invalid column name "${newName}". Column names must start with a letter or underscore, followed by alphanumeric characters or underscores.` + ) + } + if (newName.length > TABLE_LIMITS.MAX_COLUMN_NAME_LENGTH) { + throw new Error( + `Column name exceeds maximum length (${TABLE_LIMITS.MAX_COLUMN_NAME_LENGTH} characters)` + ) + } + if (columns.some((c, i) => i !== columnIndex && c.name.toLowerCase() === newName.toLowerCase())) { + throw new Error(`Column "${newName}" already exists`) + } + return { ...column, name: newName } +} + +/** + * The column definition a retype produces: prior per-type metadata dropped, + * then only what the TARGET type declares it owns carried forward, then that + * type's own defaults stamped on. + */ +function buildConvertedColumn( + column: ColumnDefinition, + data: UpdateColumnTypeData, + { isSelectType, targetMultiple }: { isSelectType: boolean; targetMultiple: boolean } +): ColumnDefinition { + // Strip EVERY type-specific key generically, so a future type's metadata + // cannot ride through `...rest` onto a target that does not own it — which + // `validateColumnDefinition` would then reject on every later write. + const rest = omit(column, [...TYPE_SPECIFIC_COLUMN_KEYS]) as ColumnDefinition + // Constraints arriving with the retype are APPLIED here, not left to a second + // transaction. `updateColumnType` already validates against them (empty cells + // for `required`, post-conversion duplicates for `unique`), so applying them + // in the same write is what makes a combined request all-or-nothing. + const withConstraints: ColumnDefinition = { + ...rest, + ...(data.required !== undefined ? { required: data.required } : {}), + ...(data.unique !== undefined ? { unique: data.unique } : {}), + } + + if (isSelectType) { + return { + ...withConstraints, + type: data.newType, + options: data.options ?? column.options, + ...(targetMultiple ? { multiple: true } : {}), + // Select columns carry no unique constraint: it would compare the stored + // option id, capping each option at one row table-wide, and the UI hides + // the toggle so it could never be cleared again. Dropped here rather than + // in each caller — the sidebar was the only one clearing it, leaving the + // v1 and agent paths to strand it. + unique: false, + } + } + + // Then carry back only the keys the TARGET type declares it owns, preferring + // the value this request supplied over the column's existing one. Iterating + // the key list rather than naming keys is what keeps this zero-edit for a + // future type. + const definition = columnTypeById(data.newType) + const owned = new Set(definition.ownedMetadata) + const carried: ColumnDefinition = { ...withConstraints, type: data.newType } + for (const key of TYPE_SPECIFIC_COLUMN_KEYS) { + if (!owned.has(key)) continue + const value = data[key] ?? column[key] + if (value !== undefined) Object.assign(carried, { [key]: value }) + } + return { ...carried, ...definition.defaultMetadata?.(carried) } +} + /** * Changes the type of a column. Validates that existing data is compatible. * @@ -514,7 +701,30 @@ export async function updateColumnType( const column = schema.columns[columnIndex] if (column.type === data.newType) { - return table + // Callers gate on the type actually changing, but they compute that from + // a schema read taken before this transaction took the lock — so a + // concurrent change can land us here with real work still to do. Only a + // rename can be honoured without a conversion; anything else would be + // silently discarded, and answering success for a change that never + // happened is the worst outcome available. + const carriesOtherWork = + data.required !== undefined || + data.unique !== undefined || + data.options !== undefined || + data.multiple !== undefined || + data.currencyCode !== undefined + if (carriesOtherWork) { + throw new Error( + `Column "${column.name}" is already type "${data.newType}"; re-issue the request without a type change.` + ) + } + const renamed = applyPendingRename(schema.columns, columnIndex, data.newName) + if (renamed === column) return table + return persistColumns( + trx, + table, + schema.columns.map((c, i) => (i === columnIndex ? renamed : c)) + ) } const columnKey = getColumnId(column) @@ -557,8 +767,34 @@ export async function updateColumnType( } } + /** + * The column definition the table ends up with. Built before the scan so + * the coercion below reads the same metadata (option set, currency) the + * stored value will be validated against afterwards. + */ + const convertedColumn = buildConvertedColumn(column, data, { + isSelectType, + targetMultiple: !!targetMultiple, + }) + let incompatibleCount = 0 let blankCount = 0 + /** + * Row id → the value the cell must END UP holding. + * + * Collected during the compatibility scan rather than re-derived later, so + * it reads the same `effective` value the check accepted — which for a + * `select` source is the option name, not the stored id. + * + * Load-bearing: a conversion is allowed exactly when the target type's + * `coerce` accepts the value, and `coerce` frequently *transforms* it (an + * epoch number becomes an ISO date, a formatted amount becomes a number). + * Without writing the transformed value back, the cell keeps its old bytes + * under the new type — and since filters and sorts apply the type's + * `jsonbCast` to whatever is stored, an epoch left in a `date` column makes + * `::timestamptz` fail on EVERY query against it. + */ + const coercedByRowId = new Map() for (const row of rows) { const rowData = row.data as RowData const value = rowData[columnKey] @@ -575,8 +811,28 @@ export async function updateColumnType( targetRequired ) ) { - if (effective === null || effective === '') blankCount++ - else incompatibleCount++ + // A cell the target cannot read but that is merely EMPTY is not a + // conversion failure — the write path already turns an unreadable value + // into null on an optional column, so the conversion does the same. Only + // a required target has a real problem with it, and the guard above has + // already reported those. Blocking here meant a text column with a + // single blank cell could not be converted to a number at all. + if (effective === null || effective === '') { + if (targetRequired) blankCount++ + else coercedByRowId.set(row.id, null) + } else { + incompatibleCount++ + } + continue + } + + // `select` keeps its own id↔name migrations; everything else writes back + // whatever `coerce` produced, when that differs from what is stored. + if (!isSelectType && effective !== null) { + const coerced = columnTypeById(data.newType).coerce(effective as JsonValue, convertedColumn) + if (coerced.ok && !Object.is(coerced.value, value)) { + coercedByRowId.set(row.id, coerced.value) + } } } @@ -592,25 +848,10 @@ export async function updateColumnType( ) } - const updatedColumns = schema.columns.map((c, i) => { - if (i !== columnIndex) return c - // Drop any prior select config, then re-add when the target type uses it. - const { options: _prevOptions, multiple: _prevMultiple, ...rest } = c - return isSelectType - ? { - ...rest, - type: data.newType, - options: data.options ?? c.options, - ...(targetMultiple ? { multiple: true } : {}), - // Select columns carry no unique constraint: it would compare the - // stored option id, capping each option at one row table-wide, and - // the UI hides the toggle so it could never be cleared again. Drop - // it here rather than in each caller — the sidebar was the only one - // clearing it, leaving the v1 and agent paths to strand it. - unique: false, - } - : { ...rest, type: data.newType } - }) + const renamedColumns = schema.columns.map((c, i) => (i === columnIndex ? convertedColumn : c)) + const updatedColumns = renamedColumns.map((c, i) => + i === columnIndex ? applyPendingRename(renamedColumns, columnIndex, data.newName) : c + ) const columnValidation = validateColumnDefinition(updatedColumns[columnIndex]) if (!columnValidation.valid) { @@ -620,10 +861,37 @@ export async function updateColumnType( const updatedSchema: TableSchema = { ...schema, columns: updatedColumns } const now = new Date() - if (convertingAwayFromSelect) { - await migrateSelectCellsToNames(trx, data.tableId, columnKey, column.options ?? []) - } else if (isSelectType) { - await migrateCellsToSelectIds(trx, data.tableId, columnKey, targetOptions, !!targetMultiple) + // Cell rewrites are owned by the column-type registry, keyed by direction. + // Outbound runs first: leaving `select` turns opaque option ids into names, + // which is the form the inbound migration (if any) then reads. + const migrationContext = { + trx, + tableId: data.tableId, + columnKey, + previous: column, + target: updatedColumns[columnIndex], + resolved: coercedByRowId, + } + await migrationFrom(column.type)?.(migrationContext) + if (isSelectType) { + await migrationTo(data.newType)?.(migrationContext) + } else { + await writeBackCoercedCells(trx, data.tableId, columnKey, coercedByRowId) + } + + // A `unique` arriving with this retype is validated HERE, against the values + // the conversion just wrote — not by the separate constraint write that + // follows. The conversion itself manufactures duplicates that no scan of the + // pre-conversion data can see (`"5"` and `"5.0"` both coerce to `5`), and + // that write runs in its own transaction, so discovering it there would + // report an error with the retype already committed and the original text + // irrecoverably rewritten. + if (data.unique === true && !column.unique) { + if (await hasDuplicateValues(trx, data.tableId, columnKey)) { + throw new Error( + `Cannot change column "${column.name}" to type "${data.newType}" and set it as unique: the converted values contain duplicates.` + ) + } } await trx @@ -672,44 +940,10 @@ export async function updateColumnConstraints( const column = schema.columns[columnIndex] const columnKey = getColumnId(column) - if (column.workflowGroupId) { - throw new Error( - `Cannot change constraints on workflow-output column "${column.name}". Constraints aren't applicable to columns whose values come from workflow execution.` - ) - } - if (data.required === true && !column.required) { - const emptyCount = await countEmptyCells(trx, data.tableId, columnKey) - if (emptyCount > 0) { - throw new Error( - `Cannot set column "${column.name}" as required: ${emptyCount} row(s) have null, missing, or empty values` - ) - } - } - - if (data.unique === true && column.type === 'select') { - throw new Error( - `Cannot set column "${column.name}" as unique: select columns compare stored option ids, which would allow only one row per option.` - ) - } - - if (data.unique === true && !column.unique) { - const duplicates = (await trx.execute( - sql`SELECT ${userTableRows.data}->>${columnKey}::text AS val, count(*) AS cnt FROM ${userTableRows} WHERE table_id = ${data.tableId} AND ${userTableRows.data} ? ${columnKey} AND ${userTableRows.data}->>${columnKey}::text IS NOT NULL GROUP BY val HAVING count(*) > 1 LIMIT 1` - )) as { val: string; cnt: number }[] - - if (duplicates.length > 0) { - throw new Error(`Cannot set column "${column.name}" as unique: duplicate values exist`) - } - } - - const updatedColumns = schema.columns.map((c, i) => - i === columnIndex - ? { - ...c, - ...(data.required !== undefined ? { required: data.required } : {}), - ...(data.unique !== undefined ? { unique: data.unique } : {}), - } - : c + const constrained = await applyConstraints(trx, data.tableId, column, columnKey, data) + const withConstraints = schema.columns.map((c, i) => (i === columnIndex ? constrained : c)) + const updatedColumns = withConstraints.map((c, i) => + i === columnIndex ? applyPendingRename(withConstraints, columnIndex, data.newName) : c ) const updatedSchema: TableSchema = { ...schema, columns: updatedColumns } const now = new Date() @@ -762,29 +996,16 @@ export async function updateColumnOptions( throw new Error(`Invalid column: ${columnValidation.errors.join('; ')}`) } - const updatedColumns = schema.columns.map((c, i) => (i === columnIndex ? updatedColumn : c)) - const updatedSchema: TableSchema = { ...schema, columns: updatedColumns } - const now = new Date() - const nextMultiple = !!(data.multiple ?? column.multiple) const wasMultiple = !!column.multiple const keptIds = new Set(data.options.map((o) => o.id)) const removedAny = (column.options ?? []).some((o) => !keptIds.has(o.id)) const togglingCardinality = nextMultiple !== wasMultiple + // The constraint the column ENDS UP with, which may be arriving in this same + // request. `applyConstraints` validates and applies it below, after the cell + // migrations; the checks in between need to read the target value. const targetRequired = !!(data.required ?? column.required) - // Newly imposing `required` in the same request: rows that are ALREADY empty - // would fail the separate constraint write after this one commits, so they - // have to be caught here, through the same predicate that write will use. - if (targetRequired && !column.required) { - const emptyCount = await countEmptyCells(trx, data.tableId, columnKey) - if (emptyCount > 0) { - throw new Error( - `Cannot make column "${column.name}" required: ${emptyCount} row(s) have null, missing, or empty values. Fill them first.` - ) - } - } - if (togglingCardinality || removedAny) { const timeoutMs = scaledStatementTimeoutMs(table.rowCount ?? 0, { baseMs: 60_000, @@ -859,159 +1080,122 @@ export async function updateColumnOptions( // scalar, so leaving cells un-normalized would silently drop every // pre-toggle row out of its own column's filters. if (togglingCardinality) { - await migrateCellsToSelectIds(trx, data.tableId, columnKey, data.options, nextMultiple) + // Same registry migration the retype path uses — `updatedColumn` already + // carries the post-toggle `options`/`multiple`, which is all it reads. + await migrationTo('select')?.({ + trx, + tableId: data.tableId, + columnKey, + previous: column, + target: updatedColumn, + resolved: new Map(), + }) } + // Constraints are validated and applied AFTER the migrations above, because + // those migrations rewrite stored values — a `unique` scan run before them + // would read the pre-migration shape and pass, and the migration could then + // produce the duplicates it was meant to prevent. + const constrainedColumn = await applyConstraints( + trx, + data.tableId, + updatedColumn, + columnKey, + data + ) + const withOptions = schema.columns.map((c, i) => (i === columnIndex ? constrainedColumn : c)) + const updatedColumns = withOptions.map((c, i) => + i === columnIndex ? applyPendingRename(withOptions, columnIndex, data.newName) : c + ) + + const updated = await persistColumns(trx, table, updatedColumns) + + logger.info( + `[${requestId}] Updated options for column "${column.name}" in table ${data.tableId}` + ) + + return updated + }) +} + +/** + * Changes the currency a `currency` column renders in. + * + * Deliberately the cheapest column mutation in this module: cells store a bare + * number, so re-denominating a column touches only the schema — no row rewrite, + * no compatibility scan, no scaled timeouts. It notably does **not** convert + * amounts between currencies; `1000` stays `1000`, now labelled in the new code. + * + * @param data - Column + target ISO 4217 code + * @param requestId - Request ID for logging + * @returns Updated table definition + * @throws Error if the table or column is missing, or the column is not a currency column + */ +export async function updateColumnCurrency( + data: UpdateColumnCurrencyData, + requestId: string +): Promise { + return withLockedTable(data.tableId, async (table, trx) => { + assertSchemaMutable(table) + + const schema = table.schema + const columnIndex = schema.columns.findIndex((c) => columnMatchesRef(c, data.columnName)) + if (columnIndex === -1) { + throw new Error(`Column "${data.columnName}" not found`) + } + + const column = schema.columns[columnIndex] + if (column.type !== 'currency') { + throw new Error(`Cannot set currency on column "${column.name}" of type "${column.type}"`) + } + + const updatedColumn: ColumnDefinition = { + ...column, + currencyCode: resolveCurrencyCode(data.currencyCode), + } + const columnValidation = validateColumnDefinition(updatedColumn) + if (!columnValidation.valid) { + throw new Error(`Invalid column: ${columnValidation.errors.join('; ')}`) + } + + const constrained = await applyConstraints( + trx, + data.tableId, + updatedColumn, + getColumnId(column), + data + ) + + // Only a no-op when nothing at all changed — currency, constraints, name. + const renamePending = data.newName !== undefined && data.newName !== column.name + if ( + constrained === updatedColumn && + updatedColumn.currencyCode === column.currencyCode && + !renamePending + ) { + return table + } + + const withCurrency = schema.columns.map((c, i) => (i === columnIndex ? constrained : c)) + const updatedColumns = withCurrency.map((c, i) => + i === columnIndex ? applyPendingRename(withCurrency, columnIndex, data.newName) : c + ) + const updatedSchema: TableSchema = { ...schema, columns: updatedColumns } + const now = new Date() + await trx .update(userTableDefinitions) .set({ schema: updatedSchema, updatedAt: now }) .where(eq(userTableDefinitions.id, data.tableId)) logger.info( - `[${requestId}] Updated options for column "${column.name}" in table ${data.tableId}` + `[${requestId}] Set currency for column "${column.name}" to "${updatedColumn.currencyCode}" in table ${data.tableId}` ) return { ...table, schema: updatedSchema, updatedAt: now } }) } -/** - * Rewrites a column's cells from stored option **ids** to option **names**, for - * a column that is ceasing to be a `select`. A multi cell joins comma-separated - * — the same shape it exports as. - * - * An id whose option no longer exists becomes null, matching - * {@link selectValueForConversion}, which is what the compatibility check ran - * on. Passing it through instead would leave an opaque `opt_…` in a typed cell - * the check had already accounted as empty. - * - * Set-based: one statement per stored shape, driven by a jsonb id→name map, so - * cost is independent of row count. - */ -async function migrateSelectCellsToNames( - trx: DbTransaction, - tableId: string, - columnKey: string, - options: SelectOption[] -): Promise { - const nameById = JSON.stringify(Object.fromEntries(options.map((o) => [o.id, o.name]))) - await trx.execute( - sql`UPDATE ${userTableRows} - SET data = jsonb_set(data, ARRAY[${columnKey}::text], - COALESCE(${nameById}::jsonb -> (data->>${columnKey}::text), 'null'::jsonb)) - WHERE table_id = ${tableId} - AND jsonb_typeof(data->${columnKey}::text) = 'string'` - ) - await trx.execute( - sql`UPDATE ${userTableRows} - SET data = jsonb_set(data, ARRAY[${columnKey}::text], COALESCE(to_jsonb(( - SELECT string_agg(${nameById}::jsonb ->> e.v, ', ' ORDER BY e.ord) - FROM jsonb_array_elements_text(data->${columnKey}::text) WITH ORDINALITY AS e(v, ord) - WHERE ${nameById}::jsonb ? e.v - )), 'null'::jsonb)) - WHERE table_id = ${tableId} - AND jsonb_typeof(data->${columnKey}::text) = 'array'` - ) -} - -/** - * Rewrites a column's cells into the canonical `select` storage shape: the - * option **id**, wrapped in an array when the column is `multiple`. - * - * Needed in both directions of a select change. Converting *to* select, cells - * hold option names (that is what made them compatible) but every reader — - * pills, filters, exports — resolves by id. Toggling single→multi, cells hold a - * scalar id while multi filters compile to array containment, which a scalar - * never matches. Either way the cell silently drops out until it is re-edited. - * - * The map keys ids, names, and lower-cased names — ids so re-running is a no-op, - * lower-cased names because `resolveSelectOptionId` accepts a case-mismatched - * name and a cell that passed that check must actually migrate. Duplicate option - * names are rejected case-insensitively at validation, so the folded key is - * unambiguous; lookups still try the exact form first to preserve its precedence. - */ -async function migrateCellsToSelectIds( - trx: DbTransaction, - tableId: string, - columnKey: string, - options: SelectOption[], - multiple: boolean -): Promise { - // Ids are written last so an id always outranks any option's name, matching - // `resolveSelectOptionId`'s id-before-name precedence. A single flat pass - // would let a later option whose *name* equals an earlier option's *id* - // overwrite that id entry and repoint its cells at the wrong option. - // A Map, not plain-object assignment: an option named `__proto__` would set - // the prototype instead of an own key and drop out of the serialized map. - const refs = new Map() - for (const o of options) { - refs.set(o.name.toLowerCase(), o.id) - refs.set(o.name, o.id) - } - for (const o of options) { - refs.set(o.id, o.id) - } - const idByRef = JSON.stringify(Object.fromEntries(refs)) - - if (multiple) { - // A string cell reaching a multi target is either one option name or the - // comma-joined form a multiselect converts to text as. Try the whole string - // first so an option whose own name contains a comma still wins, then split - // — mirroring `splitMultiSelectInput` on the write path, including its - // first-occurrence dedup. - await trx.execute( - sql`UPDATE ${userTableRows} - SET data = jsonb_set(data, ARRAY[${columnKey}::text], - CASE WHEN data->>${columnKey}::text = '' THEN '[]'::jsonb - WHEN COALESCE(${idByRef}::jsonb -> (data->>${columnKey}::text), ${idByRef}::jsonb -> lower(data->>${columnKey}::text)) IS NOT NULL - THEN jsonb_build_array(COALESCE(${idByRef}::jsonb -> (data->>${columnKey}::text), ${idByRef}::jsonb -> lower(data->>${columnKey}::text))) - ELSE COALESCE(( - SELECT jsonb_agg(v ORDER BY ord) FROM ( - SELECT COALESCE(${idByRef}::jsonb -> btrim(part), ${idByRef}::jsonb -> lower(btrim(part)), to_jsonb(btrim(part))) AS v, - min(o) AS ord - FROM unnest(string_to_array(data->>${columnKey}::text, ',')) WITH ORDINALITY AS u(part, o) - WHERE btrim(part) <> '' - GROUP BY 1 - ) d), '[]'::jsonb) - END) - WHERE table_id = ${tableId} - AND jsonb_typeof(data->${columnKey}::text) = 'string'` - ) - await trx.execute( - sql`UPDATE ${userTableRows} - SET data = jsonb_set(data, ARRAY[${columnKey}::text], COALESCE(( - SELECT jsonb_agg(COALESCE(${idByRef}::jsonb -> e.v, ${idByRef}::jsonb -> lower(e.v), to_jsonb(e.v)) ORDER BY e.ord) - FROM jsonb_array_elements_text(data->${columnKey}::text) WITH ORDINALITY AS e(v, ord) - ), '[]'::jsonb)) - WHERE table_id = ${tableId} - AND jsonb_typeof(data->${columnKey}::text) = 'array'` - ) - return - } - - // A cleared cell is stored as '' — compatibility lets it through, so it has - // to land as null rather than an '' that fails option membership on the next - // write. - await trx.execute( - sql`UPDATE ${userTableRows} - SET data = jsonb_set(data, ARRAY[${columnKey}::text], - CASE WHEN data->>${columnKey}::text = '' THEN 'null'::jsonb - ELSE COALESCE(${idByRef}::jsonb -> (data->>${columnKey}::text), ${idByRef}::jsonb -> lower(data->>${columnKey}::text), data->${columnKey}::text) - END) - WHERE table_id = ${tableId} - AND jsonb_typeof(data->${columnKey}::text) = 'string'` - ) - // Compatibility already rejected multi-valued cells for a single target, so - // any array here holds at most one option. - await trx.execute( - sql`UPDATE ${userTableRows} - SET data = jsonb_set(data, ARRAY[${columnKey}::text], - COALESCE(${idByRef}::jsonb -> (data->${columnKey}::text->>0), ${idByRef}::jsonb -> lower(data->${columnKey}::text->>0), data->${columnKey}::text->0, 'null'::jsonb)) - WHERE table_id = ${tableId} - AND jsonb_typeof(data->${columnKey}::text) = 'array'` - ) -} - /** * Rows whose cell counts as empty for a `required` constraint: the key is * missing, the value is JSON null, or it is an emptied multiselect `[]`. @@ -1164,58 +1348,12 @@ export function isValueCompatibleWithType( targetRequired = false ): boolean { if (value === null || value === undefined) return true - - switch (targetType) { - case 'string': - // Arrays and objects can't become text — the write-path coercion rejects - // them and would null the cell. Multi-select values are flattened before - // this check, so anything still structured here is genuinely lossy. - return typeof value !== 'object' - case 'select': { - // A cleared select cell is written as '' — still convertible, unless the - // target is required. Required only rejects null/undefined on a write, so - // a required string column legitimately holds ''; the migration turns that - // into null (or [] for a multi), and every later update of that row would - // then fail its own required check. - if (value === '') return !targetRequired - // Read the value exactly as the write-path coercion will. A multi target - // splits a comma-delimited string, so a multiselect → text → multiselect - // round-trip (text holding this feature's own `Bug, Docs` export shape) - // stays convertible instead of being rejected as one unknown option. - const parts = targetMultiple - ? splitMultiSelectInput(value as JsonValue) - : Array.isArray(value) - ? value - : [value] - // A single-select target can't hold several options. `updateColumnOptions` - // blocks the same transition; without this the next coerce would silently - // keep only the first id. - if (!targetMultiple && parts.length > 1) return false - return parts.every((v) => resolveSelectOptionId(v as JsonValue, targetOptions) !== null) - } - case 'number': { - if (typeof value === 'number') return Number.isFinite(value) - if (typeof value === 'string') { - const num = Number(value) - return Number.isFinite(num) && value.trim() !== '' - } - return false - } - case 'boolean': { - if (typeof value === 'boolean') return true - if (typeof value === 'string') - return ['true', 'false', '1', '0'].includes(value.toLowerCase()) - if (typeof value === 'number') return value === 0 || value === 1 - return false - } - case 'date': { - if (value instanceof Date) return !Number.isNaN(value.getTime()) - if (typeof value === 'string') return !Number.isNaN(Date.parse(value)) - return false - } - case 'json': - return true - default: - return false - } + // Each type reads only the metadata it owns. + return isValueCompatible(value, { + name: '', + type: targetType, + options: targetOptions, + multiple: targetMultiple, + required: targetRequired, + }) } diff --git a/apps/sim/lib/table/constants.ts b/apps/sim/lib/table/constants.ts index 1384d3dc18..216e1e826e 100644 --- a/apps/sim/lib/table/constants.ts +++ b/apps/sim/lib/table/constants.ts @@ -163,7 +163,17 @@ export function getTablePlanLimits(): TablePlanLimitsByPlan { } } -export const COLUMN_TYPES = ['string', 'number', 'boolean', 'date', 'json', 'select'] as const +/** + * Re-exported from the column-type module, which is the single source of truth. + * Kept here because this is where callers already import it from — restating + * the list would let the two drift, which is the class of bug the registry + * exists to remove. + * + * Points at `types` rather than `registry` on purpose: this module is re- + * exported by the `@/lib/table` barrel that 44 server modules import, and the + * registry pulls in `@sim/emcn/icons`. + */ +export { COLUMN_TYPES } from '@/lib/table/column-types/types' /** Maximum number of options a `select`/`multiselect` column may declare. */ export const MAX_SELECT_OPTIONS = 100 diff --git a/apps/sim/lib/table/currency.ts b/apps/sim/lib/table/currency.ts new file mode 100644 index 0000000000..c4aeb97e09 --- /dev/null +++ b/apps/sim/lib/table/currency.ts @@ -0,0 +1,326 @@ +/** + * Helpers for the `currency` column type. + * + * A currency cell stores a **plain JSON number** — the same storage shape as a + * `number` column — and the column carries a `currencyCode` (ISO 4217) as pure + * display metadata. That split is deliberate: filtering, sorting, uniqueness, + * and CSV export all reuse the numeric paths unchanged, changing a column's + * currency never rewrites a single cell, and the public row output stays a + * number rather than a locale-formatted string consumers would have to reparse. + */ + +/** Currency assumed when a column declares none. */ +export const DEFAULT_CURRENCY_CODE = 'USD' + +/** A bare numeric literal in exponent form, e.g. `1e+21` or `-1.5e-3`. */ +const EXPONENT_LITERAL = /^[+-]?\d+(?:\.\d+)?[eE][+-]?\d+$/ + +/** + * Invisible bidi control marks. `Intl` wraps RTL-locale output in them, so a + * pasted `‏1,234.56 ‏₪` carries characters that are not part of the amount. + */ +const BIDI_MARKS = /[\u200e\u200f\u061c\u202a-\u202e\u2066-\u2069]/g + +/** Minus-sign characters `Intl` emits in place of the ASCII hyphen. */ +const UNICODE_MINUS = /[\u2212\u2012\u2013\uFE63\uFF0D]/g + +/** A letter directly adjacent to a digit: an identifier, not an amount. */ +const LETTER_TOUCHING_DIGIT = /\p{L}\d|\d\p{L}/u + +/** + * A leading currency marker: up to three letters and/or a symbol, optionally + * behind a sign. The sign is captured so `-$12.50` keeps it. + */ +const CURRENCY_MARKER_PREFIX = + /^[\s\u00a0\u202f]*([+-]?)[\s\u00a0\u202f]*(?:\p{Sc}\p{L}{0,3}|\p{L}{1,3}\p{Sc}?)[\s\u00a0\u202f]*/u + +/** The same, trailing. */ +const CURRENCY_MARKER_SUFFIX = + /[\s\u00a0\u202f]*(?:\p{Sc}\p{L}{0,3}|\p{L}{1,3}\p{Sc}?)\.?[\s\u00a0\u202f]*$/u + +/** Only digits and separators, with at least one digit. */ +const AMOUNT_SHAPE = /^[+-]?[\d.,]*\d[\d.,]*$/ + +/** + * Whether `text` is validly grouped by `separator`: a first group of 1-3 digits + * and every later group exactly 3. Rejects `0.1.2` and `1,000,00`, which the + * plain "strip the separator" reading would silently turn into 12 and 100000. + */ +function hasValidGrouping(text: string, separator: string): boolean { + const groups = text.split(separator) + if (groups.length === 1) return true + if (!/^\d{1,3}$/.test(groups[0])) return false + const rest = groups.slice(1) + // Western: every later group is exactly three. + if (rest.every((group) => /^\d{3}$/.test(group))) return true + // Indian: the final group is three and the ones before it are two — + // `12,34,567`. Still rejects `1,000,00`, whose final group is two. + return ( + rest.length > 1 && + /^\d{3}$/.test(rest[rest.length - 1]) && + rest.slice(0, -1).every((group) => /^\d{2}$/.test(group)) + ) +} + +/** A decimal/grouping separator followed by whitespace — a list, not an amount. */ +const SEPARATOR_THEN_SPACE = /[.,]\s/ + +/** An `e` with a digit on both sides — an exponent marker, however spaced. */ +const INTERIOR_EXPONENT = /\d\s*[eE][+-]?\s*\d/ + +/** ISO 4217 alphabetic code: exactly three letters. */ +const CURRENCY_CODE_PATTERN = /^[A-Za-z]{3}$/ + +/** + * Codes offered first in the picker. The rest of ICU's set is still selectable + * (and any valid code is accepted over the API) — these are just the ones worth + * reaching without typing. + */ +const PINNED_CURRENCY_CODES = [ + 'USD', + 'EUR', + 'GBP', + 'JPY', + 'CAD', + 'AUD', + 'CHF', + 'CNY', + 'INR', + 'BRL', +] as const + +/** + * Every ISO 4217 code the runtime knows, or `null` when the runtime predates + * `Intl.supportedValuesOf` — in which case validation falls back to the shape + * check alone rather than rejecting codes it cannot enumerate. + */ +const supportedCurrencyCodes: ReadonlySet | null = (() => { + const supportedValuesOf = ( + Intl as typeof Intl & { + supportedValuesOf?: (key: string) => string[] + } + ).supportedValuesOf + if (typeof supportedValuesOf !== 'function') return null + try { + return new Set(supportedValuesOf('currency')) + } catch { + return null + } +})() + +/** Whether `code` is a well-formed ISO 4217 code this runtime can format. */ +export function isSupportedCurrencyCode(code: string): boolean { + if (!CURRENCY_CODE_PATTERN.test(code)) return false + const upper = code.toUpperCase() + return supportedCurrencyCodes === null || supportedCurrencyCodes.has(upper) +} + +/** A column's effective currency code, upper-cased, defaulting to {@link DEFAULT_CURRENCY_CODE}. */ +export function resolveCurrencyCode(currencyCode: string | undefined): string { + return currencyCode ? currencyCode.toUpperCase() : DEFAULT_CURRENCY_CODE +} + +export interface CurrencyOption { + code: string + /** Localized currency name, e.g. `US Dollar`. Falls back to the code. */ + name: string +} + +/** + * Codes for the column-config picker: the pinned set first, then every other + * code the runtime supports, alphabetically. + * + * Built on first call, not at module load: constructing `Intl.DisplayNames` and + * naming ~160 currencies costs several milliseconds of ICU work, and the only + * caller is the column-config sidebar — every table API route imports this + * module and would otherwise pay for a list it never reads. + */ +let currencyOptions: readonly CurrencyOption[] | null = null + +function currencyDisplayNames(): Intl.DisplayNames | null { + try { + return new Intl.DisplayNames(['en'], { type: 'currency' }) + } catch { + return null + } +} + +export function getCurrencyOptions(): readonly CurrencyOption[] { + if (currencyOptions) return currencyOptions + const pinned = new Set(PINNED_CURRENCY_CODES) + const rest = supportedCurrencyCodes + ? [...supportedCurrencyCodes].filter((code) => !pinned.has(code)).sort() + : [] + const displayNames = currencyDisplayNames() + currencyOptions = [...PINNED_CURRENCY_CODES, ...rest].map((code) => ({ + code, + name: displayNames?.of(code) ?? code, + })) + return currencyOptions +} + +/** + * Parses a user-entered or imported amount into a number, tolerating the shapes + * a currency value arrives in: symbols and ISO codes (`$1,234.56`, `1 234,56 €`, + * `USD 12`), grouping separators (including the non-breaking spaces several + * locales use), and accounting negatives (`(1,234.56)` → `-1234.56`). + * + * Separator disambiguation, when only commas are present: a single comma + * followed by exactly three digits is grouping (`1,500` → `1500`); anything + * else is a decimal comma (`1,50` → `1.5`). `1,500` meaning one-and-a-half is + * therefore read as fifteen hundred — a known ambiguity that resolves in favor + * of the far more common reading. + * + * Reads ASCII digits only. Locales that format with their own numeral systems + * (Arabic-Indic `١٢٣`, for instance) are rejected rather than misread — + * supporting them is a wider decision than this type, since it would also + * touch `number`, display, and sorting. + * + * Returns `null` when no amount can be read, so callers can distinguish + * "unparseable" from a legitimate `0`. + */ +export function parseCurrencyInput(raw: unknown): number | null { + if (typeof raw === 'number') return Number.isFinite(raw) ? raw : null + if (typeof raw !== 'string') return null + + // `Intl` emits U+2212 MINUS SIGN (and locale-specific dashes) rather than the + // ASCII hyphen for negatives in several locales, so a pasted `−12,50 kr` + // would otherwise fail to read as negative. + const trimmed = raw.trim().replace(BIDI_MARKS, '').replace(UNICODE_MINUS, '-') + if (trimmed === '') return null + + const parenthesized = /^\((.*)\)$/.exec(trimmed) + const body = parenthesized ? parenthesized[1] : trimmed + + // No real amount puts whitespace after a separator, but a delimited LIST + // does — and a multi-select column flattens to exactly that when it converts. + // Without this, `12, 34` reads as 12.34 and `100, 200` as 100200, so a + // multi-select column of numeric option names would convert to nonsense. + if (SEPARATOR_THEN_SPACE.test(body)) return null + + // Exponent form is taken at face value. `String()` emits it for any magnitude + // past 1e21, so a stored amount round-trips through the editor as `1e+21` — + // and stripping the `e` as decoration would read that back as 121, silently + // losing 19 orders of magnitude. + const exponentCandidate = body.replace(/[^\d.,\-+eE]/g, '') + if (EXPONENT_LITERAL.test(exponentCandidate)) { + const parsed = Number(exponentCandidate) + if (Number.isFinite(parsed)) return parenthesized ? -Math.abs(parsed) : parsed + } + // An `e` sitting between digits is an exponent marker, not decoration. If the + // string is not a clean literal we cannot read it unambiguously, and dropping + // the `e` would join the digit groups and change the magnitude (`1e5 EUR` + // would become 15) — so refuse instead of guessing. The digit on BOTH sides + // is what distinguishes this from the `E` inside an ISO code like `12 EUR`. + if (INTERIOR_EXPONENT.test(body)) return null + + // A letter touching a digit means this is an identifier, not an amount — + // `SKU400`, `ABC1234`. Currency markers are always separated from the number + // by a space or a symbol, so this distinguishes them without a symbol list. + if (LETTER_TOUCHING_DIGIT.test(body)) return null + + // Strip the currency marker: up to three letters (an ISO code, `kr`, `zł`) + // optionally joined to a symbol (`R$`, `CHF`), at either end. Bounded at + // three so prose does not qualify — `Revenue 5` keeps its letters and is + // rejected below. + const cleaned = body + .replace(CURRENCY_MARKER_PREFIX, '$1') + .replace(CURRENCY_MARKER_SUFFIX, '') + .replace(/[\s\u00a0\u202f\u2019']/gu, '') + // What remains must be ONLY digits and separators. Anything else — a + // US-format date, leftover prose — is not an amount. + if (!AMOUNT_SHAPE.test(cleaned)) return null + + const signed = /^[+-]/.test(cleaned) + const digitsAndSeps = signed ? cleaned.slice(1) : cleaned + const negative = parenthesized !== null || (signed && cleaned.startsWith('-')) + + const lastComma = digitsAndSeps.lastIndexOf(',') + const lastDot = digitsAndSeps.lastIndexOf('.') + let normalized: string + if (lastComma !== -1 && lastDot !== -1) { + // Both present: whichever comes last is the decimal separator. + const decimalSeparator = lastComma > lastDot ? ',' : '.' + const groupSeparator = decimalSeparator === ',' ? '.' : ',' + const [integerPart, ...decimalParts] = digitsAndSeps.split(decimalSeparator) + if (decimalParts.length > 1 || !hasValidGrouping(integerPart, groupSeparator)) return null + normalized = `${integerPart.split(groupSeparator).join('')}.${decimalParts[0]}` + } else if (lastComma !== -1) { + // A single comma followed by exactly three digits is grouping (`1,500`); + // anything else is a decimal comma (`1,50`). + const grouping = digitsAndSeps.indexOf(',') !== lastComma || /,\d{3}$/.test(digitsAndSeps) + if (grouping) { + if (!hasValidGrouping(digitsAndSeps, ',')) return null + normalized = digitsAndSeps.split(',').join('') + } else { + normalized = digitsAndSeps.replace(',', '.') + } + } else if (lastDot !== -1 && digitsAndSeps.indexOf('.') !== lastDot) { + // More than one dot can only be grouping: `1.234.567`. + if (!hasValidGrouping(digitsAndSeps, '.')) return null + normalized = digitsAndSeps.split('.').join('') + } else { + normalized = digitsAndSeps + } + + const parsed = Number(normalized) + if (!Number.isFinite(parsed)) return null + return negative ? -parsed : parsed +} + +/** + * Formatters are cached by locale + code: a grid paints thousands of currency + * cells per scroll, and constructing an `Intl.NumberFormat` per cell is orders + * of magnitude more expensive than the format call itself. + */ +const formatterCache = new Map() + +function currencyFormatter( + currencyCode: string, + locale: string | undefined +): Intl.NumberFormat | null { + const key = `${locale ?? ''}:${currencyCode}` + const cached = formatterCache.get(key) + if (cached !== undefined) return cached + let formatter: Intl.NumberFormat | null + try { + formatter = new Intl.NumberFormat(locale, { style: 'currency', currency: currencyCode }) + } catch { + formatter = null + } + formatterCache.set(key, formatter) + return formatter +} + +/** + * Formats a stored cell for display — symbol placement and fraction digits come + * from the currency itself, so `JPY` renders `¥1,235` while `USD` renders + * `$1,234.56`. Values that carry no readable amount (a string left behind by a + * `string` → `currency` conversion, say) render verbatim rather than blanking, + * and an unformattable code degrades to `CODE amount`. + * + * `locale` is left to the caller's runtime by default, which means the viewer's + * own grouping/decimal conventions in the browser. + */ +export function formatCurrencyDisplay( + value: unknown, + currencyCode: string | undefined, + locale?: string +): string { + const amount = parseCurrencyInput(value) + if (amount === null) return typeof value === 'string' ? value : '' + const code = resolveCurrencyCode(currencyCode) + const formatter = currencyFormatter(code, locale) + return formatter ? formatter.format(amount) : `${code} ${amount}` +} + +/** + * Renders a stored cell for a text input: the bare amount, with no symbol or + * grouping, so editing round-trips through {@link parseCurrencyInput} exactly. + */ +export function formatCurrencyForInput(value: unknown): string { + if (value === null || value === undefined) return '' + const amount = parseCurrencyInput(value) + if (amount !== null) return String(amount) + return typeof value === 'string' ? value : '' +} diff --git a/apps/sim/lib/table/export-format.ts b/apps/sim/lib/table/export-format.ts index c24c586393..60f3d79eae 100644 --- a/apps/sim/lib/table/export-format.ts +++ b/apps/sim/lib/table/export-format.ts @@ -4,6 +4,7 @@ * byte-identical files. */ +import { columnTypeOf } from '@/lib/table/column-types' import { selectValueToNames } from '@/lib/table/select-values' import type { ColumnDefinition } from '@/lib/table/types' @@ -43,10 +44,10 @@ export function formatCsvValue(value: unknown): string { * (comma-joined for multi) so the file shows the enum label, not the id. */ export function formatCsvCell(column: ColumnDefinition, value: unknown): string { - if (column.type === 'select') { - const resolved = resolveSelectExportValue(column, value) - const text = Array.isArray(resolved) ? resolved.join(', ') : (resolved ?? '') - return neutralizeCsvFormula(text) + // Every other type writes its stored value verbatim so the file re-imports + // byte-identically. + if (columnTypeOf(column).storesOpaqueIds) { + return neutralizeCsvFormula(columnTypeOf(column).formatForDisplay(value, column)) } return formatCsvValue(value) } diff --git a/apps/sim/lib/table/import.test.ts b/apps/sim/lib/table/import.test.ts index 91baff2b94..2943870e89 100644 --- a/apps/sim/lib/table/import.test.ts +++ b/apps/sim/lib/table/import.test.ts @@ -136,6 +136,15 @@ describe('import', () => { expect(coerceValue('not a number', 'number')).toBeNull() }) + it('coerces a formatted amount into a currency column', () => { + // Importing into an EXISTING currency column — inference never picks + // currency, so this is the only way the branch is reached. + expect(coerceValue('$1,234.56', 'currency')).toBe(1234.56) + expect(coerceValue('1.234,56', 'currency')).toBe(1234.56) + expect(coerceValue(12, 'currency')).toBe(12) + expect(coerceValue('ask sales', 'currency')).toBeNull() + }) + it('coerces booleans strictly', () => { expect(coerceValue('true', 'boolean')).toBe(true) expect(coerceValue('FALSE', 'boolean')).toBe(false) diff --git a/apps/sim/lib/table/import.ts b/apps/sim/lib/table/import.ts index 94b4dc2272..e01147c36f 100644 --- a/apps/sim/lib/table/import.ts +++ b/apps/sim/lib/table/import.ts @@ -13,6 +13,8 @@ import { type Options as CsvParseOptions, type Parser, parse as parseCsvStream } from 'csv-parse' import { getColumnId } from '@/lib/table/column-keys' +import type { ColumnType } from '@/lib/table/column-types' +import { parseCurrencyInput } from '@/lib/table/currency' import { type NormalizeDateCellOptions, normalizeDateCellValue } from '@/lib/table/dates' import type { ColumnDefinition, RowData, TableSchema } from '@/lib/table/types' @@ -193,8 +195,18 @@ export async function detectCsvDelimiter( return best?.delimiter ?? fallback } -/** Narrower type than `COLUMN_TYPES` used internally for coercion. */ -export type CsvColumnType = 'string' | 'number' | 'boolean' | 'date' | 'json' +/** + * Column types the CSV path coerces. Derived from the registry rather than + * restated, so a new type is covered automatically. + */ +export type CsvColumnType = ColumnType + +/** + * The subset {@link inferColumnType} can return. Every other type either needs + * configuration inference cannot supply (`select`'s options, `currency`'s + * code) or would swallow ordinary text (`json`). + */ +type InferredCsvColumnType = Extract /** Number of CSV rows sampled when inferring column types for a new table. */ export const CSV_SCHEMA_SAMPLE_SIZE = 100 @@ -275,9 +287,11 @@ export async function parseCsvBuffer( /** * Infers a column type from a sample of non-empty values. Order matters: we * prefer narrower types (number > boolean > ISO date) and fall back to string. - * JSON is never inferred automatically. + * JSON and currency are never inferred automatically — currency because the + * column would also have to guess an ISO code from a symbol, and guessing wrong + * mislabels every amount in the column. */ -export function inferColumnType(values: unknown[]): Exclude { +export function inferColumnType(values: unknown[]): InferredCsvColumnType { const nonEmpty = values.filter((v) => v !== null && v !== undefined && v !== '') if (nonEmpty.length === 0) return 'string' @@ -360,6 +374,13 @@ export function inferSchemaFromCsv( * empty inputs or values that cannot be parsed (numbers/booleans). Dates fall * back to the original string when unparseable so that schema validation can * reject it with context rather than silently inserting `null`. + * + * Deliberately NOT routed through the column-type registry's `coerce`, despite + * covering the same types. The registry's contract is "coerced or rejected", + * which the write path turns into `null`; an import instead wants an + * unparseable date or JSON blob to survive as its raw string so the row-level + * validation error names the offending value. Unifying the two would silently + * swap a descriptive import error for a blanked cell. */ export function coerceValue( value: unknown, @@ -372,6 +393,10 @@ export function coerceValue( const n = Number(value) return Number.isNaN(n) ? null : n } + // Importing into an existing currency column: the file carries the + // formatted amount (`$1,234.56`) but the cell stores a bare number. + case 'currency': + return parseCurrencyInput(value) case 'boolean': { const s = String(value).toLowerCase() if (s === 'true') return true diff --git a/apps/sim/lib/table/index.ts b/apps/sim/lib/table/index.ts index fad60f6537..72585f3c07 100644 --- a/apps/sim/lib/table/index.ts +++ b/apps/sim/lib/table/index.ts @@ -9,6 +9,7 @@ export * from '@/lib/table/billing' export * from '@/lib/table/column-keys' export * from '@/lib/table/columns/service' export * from '@/lib/table/constants' +export * from '@/lib/table/currency' export * from '@/lib/table/dates' export * from '@/lib/table/errors' export * from '@/lib/table/import' diff --git a/apps/sim/lib/table/llm/enrichment.ts b/apps/sim/lib/table/llm/enrichment.ts index 2d9cb7d5d6..3225fd8e5b 100644 --- a/apps/sim/lib/table/llm/enrichment.ts +++ b/apps/sim/lib/table/llm/enrichment.ts @@ -5,6 +5,7 @@ * with table-specific information so LLMs can construct proper queries. */ +import { columnTypeById } from '@/lib/table/column-types' import type { TableSummary } from '@/lib/table/types' /** @@ -91,7 +92,7 @@ ${filterExample}${sortExample}` const exampleCols = table.columns.slice(0, 3) const dataExample = exampleCols.reduce( (obj, col) => { - obj[col.name] = col.type === 'number' ? 123 : col.type === 'boolean' ? true : 'example' + obj[col.name] = columnTypeById(col.type).sampleValue return obj }, {} as Record @@ -168,7 +169,7 @@ export function enrichTableToolParameters( const exampleCols = table.columns.slice(0, 2) const exampleData = exampleCols.reduce( (obj: Record, col: { name: string; type: string }) => { - obj[col.name] = col.type === 'number' ? 123 : col.type === 'boolean' ? true : 'value' + obj[col.name] = columnTypeById(col.type).sampleValue return obj }, {} as Record diff --git a/apps/sim/lib/table/rows/service.ts b/apps/sim/lib/table/rows/service.ts index dfcc3a4c28..2cf9e357b6 100644 --- a/apps/sim/lib/table/rows/service.ts +++ b/apps/sim/lib/table/rows/service.ts @@ -24,6 +24,7 @@ import { wouldExceedRowLimit, } from '@/lib/table/billing' import { getColumnId } from '@/lib/table/column-keys' +import { columnTypeOf } from '@/lib/table/column-types' import { getMaxPageBytes, TABLE_LIMITS, USER_TABLE_ROWS_SQL_NAME } from '@/lib/table/constants' import { TableQueryValidationError } from '@/lib/table/errors' import { @@ -814,7 +815,7 @@ const FIND_MATCH_LIMIT = 1000 * are trusted schema data, escaped and embedded literally; the row alias is `o`. */ export function buildSelectFindNameExpr(columns: ColumnDefinition[]): string | null { - const selectColumns = columns.filter((c) => c.type === 'select') + const selectColumns = columns.filter((c) => columnTypeOf(c).storesOpaqueIds) if (selectColumns.length === 0) return null const esc = (s: string) => s.replace(/'/g, "''") const whens = selectColumns diff --git a/apps/sim/lib/table/select-options.ts b/apps/sim/lib/table/select-options.ts new file mode 100644 index 0000000000..aa62cfa0bf --- /dev/null +++ b/apps/sim/lib/table/select-options.ts @@ -0,0 +1,87 @@ +/** + * Pure `select`-option helpers, with no database dependency. + * + * These live in their own leaf module rather than in `validation.ts` on + * purpose. `validation.ts` imports `@sim/db`, `drizzle-orm`, and `next/server`, + * so anything importing it — `select-values.ts`, and transitively + * `cell-format.ts` and `export-format.ts` — becomes server-only. That taint is + * the sole reason the tables grid used to hand-roll its own copy of the + * id-resolution logic client-side, and why a select cell's stored ids could + * drift from the names the client resolved them to. + * + * Keeping the option primitives here lets both sides share one implementation. + */ + +import type { ColumnDefinition, JsonValue, SelectOption } from '@/lib/table/types' + +/** Set of valid option ids for a `select`/`multiselect` column. */ +export function optionIds(column: ColumnDefinition): Set { + return new Set((column.options ?? []).map((o) => o.id)) +} + +/** + * Resolves a raw cell value to a declared option id, accepting either the + * stable id or (tolerant for tool/import writes) the option's display name. + * Returns null when no option matches. + * + * Id wins over name, and an exact name wins over a case-folded one, so an + * option whose *name* equals another option's *id* can never repoint a cell. + */ +export function resolveSelectOptionId(value: JsonValue, options: SelectOption[]): string | null { + // The block builder serializes without schema access, so an option NAME that + // looks numeric or boolean ("123", "true") arrives scalar-coerced. Stringify + // scalars so the name still resolves; arrays/objects stay unresolvable. + const text = + typeof value === 'string' + ? value + : typeof value === 'number' || typeof value === 'boolean' + ? String(value) + : null + if (text === null) return null + const byId = options.find((o) => o.id === text) + if (byId) return byId.id + const byName = + options.find((o) => o.name === text) ?? + options.find((o) => o.name.toLowerCase() === text.toLowerCase()) + return byName ? byName.id : null +} + +/** + * Splits a raw value into the parts a multi-select cell should resolve. A cell + * may arrive as an array (canonical) or as a single comma-delimited string — + * the shape a multi cell exports, copies, and converts to text as — so both the + * write-path coercion and the column-conversion compatibility check read it + * through here rather than each deciding for itself. Option names that + * themselves contain commas are an accepted ambiguity. + */ +export function splitMultiSelectInput(value: JsonValue): JsonValue[] { + if (Array.isArray(value)) return value + if (typeof value !== 'string') return [value] + return value + .split(',') + .map((part) => part.trim()) + .filter((part) => part !== '') +} + +/** + * Resolves a raw value to the canonical stored shape for a select column: an + * array of option ids when `multiple`, otherwise a single id (or `null`). + * + * This is the one place the multi/single split is decided. Unresolvable parts + * are dropped and duplicates collapse to their first occurrence. + */ +export function resolveSelectCellValue(value: JsonValue, column: ColumnDefinition): JsonValue { + const options = column.options ?? [] + if (column.multiple) { + const ids: string[] = [] + for (const entry of splitMultiSelectInput(value)) { + const id = resolveSelectOptionId(entry, options) + if (id !== null && !ids.includes(id)) ids.push(id) + } + return ids + } + // Tolerate an array left behind by a multiple→single toggle by resolving its + // first element, rather than dropping the cell wholesale. + const single = Array.isArray(value) ? value[0] : value + return single === undefined ? null : resolveSelectOptionId(single, options) +} diff --git a/apps/sim/lib/table/select-values.ts b/apps/sim/lib/table/select-values.ts index c120a989ed..5c1c2ae0f6 100644 --- a/apps/sim/lib/table/select-values.ts +++ b/apps/sim/lib/table/select-values.ts @@ -10,6 +10,7 @@ */ import { buildIdByName, getColumnId, predicateNamesToIds } from '@/lib/table/column-keys' +import { resolveSelectOptionId } from '@/lib/table/select-options' import type { ColumnDefinition, ConditionOperators, @@ -21,7 +22,6 @@ import type { TablePredicate, TableSchema, } from '@/lib/table/types' -import { resolveSelectOptionId } from '@/lib/table/validation' /** * Resolves a `select` cell's stored option id(s) to their display name(s). A diff --git a/apps/sim/lib/table/sql.ts b/apps/sim/lib/table/sql.ts index 7ea5a6de3f..97dff495b3 100644 --- a/apps/sim/lib/table/sql.ts +++ b/apps/sim/lib/table/sql.ts @@ -9,6 +9,12 @@ import { isRecordLike } from '@sim/utils/object' import type { SQL } from 'drizzle-orm' import { sql } from 'drizzle-orm' import { getColumnId } from '@/lib/table/column-keys' +import { + columnTypeById, + filterOperatorsFor, + MULTI_SELECT_OPERATORS, + SINGLE_SELECT_OPERATORS, +} from '@/lib/table/column-types' import { NAME_PATTERN } from '@/lib/table/constants' import { TableQueryValidationError } from '@/lib/table/errors' import type { @@ -23,20 +29,16 @@ import type { TablePredicate, } from '@/lib/table/types' +/** + * Re-exported: the `$`-prefixed wire whitelists now live with the `select` type + * definition, but this module is where callers and tests already look for them. + */ +export { MULTI_SELECT_OPERATORS, SINGLE_SELECT_OPERATORS } + type ColumnType = ColumnDefinition['type'] type ColumnMap = ReadonlyMap /** - * Operators that make sense on a `select` column (whose values are opaque option - * ids), split by cardinality. A single-select cell holds one id, so it compares - * for equality; a multi-select cell holds an array of ids, so the question is - * membership — hence contains / does-not-contain. `$eq` against an array cell - * can never be true (`{"t":["a"]} @> {"t":"a"}` is false in Postgres), so - * allowing it would silently match nothing. - */ -export const SINGLE_SELECT_OPERATORS = new Set(['$eq', '$ne', '$in', '$nin', '$empty']) -export const MULTI_SELECT_OPERATORS = new Set(['$contains', '$ncontains', '$empty']) - /** * The same allowlists in the v2 bare-operator grammar, applied inside * `fieldPredicate` so both wire formats gate identically. Not derived from the @@ -71,14 +73,7 @@ const MULTI_SELECT_OPS = new Set([ * paths from drifting apart. */ function jsonbCastForType(type: ColumnType | undefined): 'numeric' | 'timestamptz' | null { - switch (type) { - case 'number': - return 'numeric' - case 'date': - return 'timestamptz' - default: - return null - } + return columnTypeById(type).jsonbCast } /** @@ -405,6 +400,9 @@ function buildFieldCondition( const columnType = column?.type const isSelect = columnType === 'select' const isMultiSelect = isSelect && column?.multiple === true + // Types whose stored value is opaque (a select's option ids) restrict which + // operators mean anything; `null` means the type accepts them all. + const allowedOperators = column ? filterOperatorsFor(column) : null const conditions: SQL[] = [] if (isRecordLike(condition)) { @@ -412,14 +410,10 @@ function buildFieldCondition( // Validate against the legacy `$`-whitelist, then normalize onto the shared // `FilterOp` so v1 and v2 emit byte-identical leaf SQL. validateOperator(op) - // Select values are opaque option ids — range/pattern operators are meaningless. - if (isSelect) { - const allowed = isMultiSelect ? MULTI_SELECT_OPERATORS : SINGLE_SELECT_OPERATORS - if (!allowed.has(op)) { - throw new TableQueryValidationError( - `Operator "${op}" is not supported on ${isMultiSelect ? 'multi-select' : 'select'} column "${field}". Allowed: ${Array.from(allowed).join(', ')}` - ) - } + if (allowedOperators && !allowedOperators.has(op)) { + throw new TableQueryValidationError( + `Operator "${op}" is not supported on ${isMultiSelect ? 'multi-select' : columnType} column "${field}". Allowed: ${Array.from(allowedOperators).join(', ')}` + ) } if (op === '$empty') { diff --git a/apps/sim/lib/table/types.ts b/apps/sim/lib/table/types.ts index 1c6b5e3991..40231c7790 100644 --- a/apps/sim/lib/table/types.ts +++ b/apps/sim/lib/table/types.ts @@ -61,6 +61,12 @@ export interface ColumnDefinition { options?: SelectOption[] /** When true, a `select` column accepts several options per cell (string[]). */ multiple?: boolean + /** + * ISO 4217 code for a `currency` column, e.g. `USD`. Display metadata only — + * cells store a plain number, so changing this reformats without touching a + * single row. Absent means {@link DEFAULT_CURRENCY_CODE}. + */ + currencyCode?: string } /** The column `type` discriminator, named so callers don't index into the interface. */ @@ -775,16 +781,27 @@ export interface RenameColumnData { export interface UpdateColumnTypeData { tableId: string columnName: string + /** + * A rename to apply in the SAME transaction as this write. Folding it in is + * what stops a combined request from committing one half and then failing. + */ + newName?: string newType: (typeof COLUMN_TYPES)[number] /** Options to set when changing to a `select` type. */ options?: SelectOption[] /** Whether the `select` column accepts multiple options per cell. */ multiple?: boolean + /** Currency to set when changing to the `currency` type. */ + currencyCode?: string /** - * The `required` value the same request is about to set, when it changes type - * and constraints together. Those are separate transactions, so the - * conversion has to validate against the constraint the column will END UP - * with — otherwise it commits and the constraint write then fails. + * The `unique` value the same request is about to set. Validated inside the + * retype against the post-conversion values, because the conversion is what + * can create the duplicates. + */ + unique?: boolean + /** + * The `required` value the same request is about to set. Applied by this + * write, in the same transaction as the change it accompanies. */ required?: boolean } @@ -792,21 +809,50 @@ export interface UpdateColumnTypeData { export interface UpdateColumnOptionsData { tableId: string columnName: string + /** + * A rename to apply in the SAME transaction as this write. Folding it in is + * what stops a combined request from committing one half and then failing. + */ + newName?: string + /** Constraints to apply in the SAME transaction as this write. */ + unique?: boolean options: SelectOption[] /** Toggle single/multi selection alongside the options update. */ multiple?: boolean /** - * The `required` value the same request is about to set. The constraint write - * is a separate transaction, so the options update has to validate against - * the constraint the column will END UP with — otherwise it clears cells and - * the constraint write then fails, leaving the removal committed. + * The `required` value the same request is about to set. Applied by this + * write, in the same transaction as the options change. */ required?: boolean } +/** + * Payload for `updateColumnCurrency`. Unlike an options update this rewrites no + * cells — a currency cell stores a plain number, and `currencyCode` only + * changes how it is rendered. + */ +export interface UpdateColumnCurrencyData { + tableId: string + columnName: string + /** + * A rename to apply in the SAME transaction as this write. Folding it in is + * what stops a combined request from committing one half and then failing. + */ + newName?: string + /** Constraints to apply in the SAME transaction as this write. */ + unique?: boolean + required?: boolean + currencyCode: string +} + export interface UpdateColumnConstraintsData { tableId: string columnName: string + /** + * A rename to apply in the SAME transaction as this write. Folding it in is + * what stops a combined request from committing one half and then failing. + */ + newName?: string required?: boolean unique?: boolean } diff --git a/apps/sim/lib/table/validation.ts b/apps/sim/lib/table/validation.ts index 14275ea74e..270ee5e3ee 100644 --- a/apps/sim/lib/table/validation.ts +++ b/apps/sim/lib/table/validation.ts @@ -7,28 +7,51 @@ import { userTableRows } from '@sim/db/schema' import { and, eq, or, type SQL, sql } from 'drizzle-orm' import { NextResponse } from 'next/server' import { getColumnId } from '@/lib/table/column-keys' +import type { CoerceResult, TypeSpecificColumnKey } from '@/lib/table/column-types' import { + COLUMN_TYPE_REGISTRY, COLUMN_TYPES, + columnTypeOf, + isColumnType, + TYPE_SPECIFIC_COLUMN_KEYS, + validateTypeMetadata, +} from '@/lib/table/column-types' +import { getMaxRowSizeBytes, - MAX_SELECT_OPTIONS, NAME_PATTERN, TABLE_LIMITS, USER_TABLE_ROWS_SQL_NAME, } from '@/lib/table/constants' -import { normalizeDateCellValue } from '@/lib/table/dates' import { withSeqscanOff } from '@/lib/table/planner' +import { resolveSelectOptionId, splitMultiSelectInput } from '@/lib/table/select-options' import { fieldPredicate } from '@/lib/table/sql' import type { ColumnDefinition, JsonValue, RowData, - SelectOption, TableSchema, ValidationResult, } from '@/lib/table/types' export type { ColumnDefinition, TableSchema, ValidationResult } +/** + * Re-exported so existing importers keep working; the implementations moved to + * `select-options.ts` to break this module's drizzle dependency for clients. + */ +export { resolveSelectOptionId, splitMultiSelectInput } + +/** + * How each type-specific key is named when it appears on a type that doesn't + * own it. `Record` keeps this exhaustive — a new + * key cannot be added without giving it a message. + */ +const FOREIGN_METADATA_VERB: Record = { + options: 'define options', + multiple: 'be multiple', + currencyCode: 'define a currency', +} + type ValidationSuccess = { valid: true } type ValidationFailure = { valid: false; response: NextResponse } @@ -234,174 +257,21 @@ export function validateRowAgainstSchema(data: RowData, schema: TableSchema): Va if (value === null || value === undefined) continue - switch (column.type) { - case 'string': - if (typeof value !== 'string') { - errors.push(`${column.name} must be string, got ${typeof value}`) - } - break - case 'number': - if (typeof value !== 'number' || Number.isNaN(value)) { - errors.push(`${column.name} must be number`) - } - break - case 'boolean': - if (typeof value !== 'boolean') { - errors.push(`${column.name} must be boolean`) - } - break - case 'date': - if ( - !(value instanceof Date) && - (typeof value !== 'string' || Number.isNaN(Date.parse(value))) - ) { - errors.push(`${column.name} must be valid date`) - } - break - case 'json': - try { - JSON.stringify(value) - } catch { - errors.push(`${column.name} must be valid JSON`) - } - break - case 'select': { - const ids = optionIds(column) - if (column.multiple) { - if (!Array.isArray(value)) { - errors.push(`${column.name} must be a list of options`) - } else if (!value.every((v) => typeof v === 'string' && ids.has(v))) { - errors.push(`${column.name} must only contain defined options`) - } else if (column.required && value.length === 0) { - errors.push(`Missing required field: ${column.name}`) - } - } else if (typeof value !== 'string' || !ids.has(value)) { - errors.push(`${column.name} must be one of the defined options`) - } - break - } - } + const error = columnTypeOf(column).validateCell(value, column) + if (error !== null) errors.push(error) } return { valid: errors.length === 0, errors } } -/** Set of valid option ids for a `select`/`multiselect` column. */ -function optionIds(column: ColumnDefinition): Set { - return new Set((column.options ?? []).map((o) => o.id)) -} - -/** - * Resolves a raw cell value to a declared option id, accepting either the - * stable id or (tolerant for tool/import writes) the option's display name. - * Returns null when no option matches. Exported so the column-type-conversion - * path can gate a `select`/`multiselect` change on whether existing values - * actually fit the target option set. - */ -export function resolveSelectOptionId(value: JsonValue, options: SelectOption[]): string | null { - // The block builder serializes without schema access, so an option NAME that - // looks numeric or boolean ("123", "true") arrives scalar-coerced. Stringify - // scalars so the name still resolves; arrays/objects stay unresolvable. - const text = - typeof value === 'string' - ? value - : typeof value === 'number' || typeof value === 'boolean' - ? String(value) - : null - if (text === null) return null - const byId = options.find((o) => o.id === text) - if (byId) return byId.id - const byName = - options.find((o) => o.name === text) ?? - options.find((o) => o.name.toLowerCase() === text.toLowerCase()) - return byName ? byName.id : null -} - -/** - * Splits a raw value into the parts a multi-select cell should resolve. A cell - * may arrive as an array (canonical) or as a single comma-delimited string — - * the shape a multi cell exports, copies, and converts to text as — so both the - * write-path coercion and the column-conversion compatibility check read it - * through here rather than each deciding for itself. Option names that - * themselves contain commas are an accepted ambiguity. - */ -export function splitMultiSelectInput(value: JsonValue): JsonValue[] { - if (Array.isArray(value)) return value - if (typeof value !== 'string') return [value] - return value - .split(',') - .map((part) => part.trim()) - .filter((part) => part !== '') -} - /** * Attempts to coerce a non-null value to a column's declared type. Returns the * coerced value when the value already matches or can be converted without * ambiguity (e.g. the string `"1999"` to the number `1999`), and `ok: false` * when no safe conversion exists. */ -function coerceValueToColumnType( - value: JsonValue, - column: ColumnDefinition -): { ok: true; value: JsonValue } | { ok: false } { - switch (column.type) { - case 'string': - if (typeof value === 'string') return { ok: true, value } - if (typeof value === 'number' || typeof value === 'boolean') { - return { ok: true, value: String(value) } - } - return { ok: false } - case 'number': - if (typeof value === 'number') { - return Number.isFinite(value) ? { ok: true, value } : { ok: false } - } - if (typeof value === 'string' && value.trim() !== '') { - const parsed = Number(value) - return Number.isFinite(parsed) ? { ok: true, value: parsed } : { ok: false } - } - return { ok: false } - case 'boolean': - if (typeof value === 'boolean') return { ok: true, value } - if (typeof value === 'string') { - const normalized = value.trim().toLowerCase() - if (normalized === 'true') return { ok: true, value: true } - if (normalized === 'false') return { ok: true, value: false } - } - return { ok: false } - case 'date': { - if (typeof value === 'string') { - const normalized = normalizeDateCellValue(value) - return normalized === null ? { ok: false } : { ok: true, value: normalized } - } - // Date instances and epoch numbers may still be out of the representable - // range (>±8.64e15ms) — guard `toISOString()`, which throws RangeError on - // an Invalid Date, so an over-range value degrades to `{ ok: false }` - // rather than crashing the write. - const date = - value instanceof Date ? value : typeof value === 'number' ? new Date(value) : null - if (date && !Number.isNaN(date.getTime())) return { ok: true, value: date.toISOString() } - return { ok: false } - } - case 'select': { - const options = column.options ?? [] - if (column.multiple) { - const raw = splitMultiSelectInput(value) - const ids: string[] = [] - for (const entry of raw) { - const id = resolveSelectOptionId(entry, options) - if (id !== null && !ids.includes(id)) ids.push(id) - } - return { ok: true, value: ids } - } - // Single: tolerate an array (e.g. right after a multiple→single toggle) by - // resolving its first element so the value isn't dropped wholesale. - const single = Array.isArray(value) ? value[0] : value - const id = single === undefined ? null : resolveSelectOptionId(single, options) - return id !== null ? { ok: true, value: id } : { ok: false } - } - default: - return { ok: true, value } - } +function coerceValueToColumnType(value: JsonValue, column: ColumnDefinition): CoerceResult { + return columnTypeOf(column).coerce(value, column) } /** @@ -628,7 +498,7 @@ export async function checkBatchUniqueConstraintsDb( const value = rowData[key] if (value === null || value === undefined) continue - const normalizedValue = typeof value === 'string' ? value : JSON.stringify(value) + const normalizedValue = JSON.stringify(value) // Check for duplicate within batch const columnValueMap = batchValueMap.get(key)! @@ -664,10 +534,13 @@ export async function checkBatchUniqueConstraintsDb( const valueArray = Array.from(values) const valueConditions = valueArray.map((normalizedValue) => { - // Reconstruct the original typed value from its normalized key: string - // columns store the raw string; others store JSON.stringify(value). - const originalValue: JsonValue = - column.type === 'string' ? normalizedValue : JSON.parse(normalizedValue) + // Reconstruct the original typed value from its normalized key. Both + // directions go through JSON unconditionally: keying the write on the + // value's RUNTIME type while keying the read on the column's DECLARED + // type made them disagree for any non-`string` type that stores a + // string — a unique `date` column normalized to a bare `2024-01-01` + // and then threw `SyntaxError` trying to parse it back. + const originalValue: JsonValue = JSON.parse(normalizedValue) // Same case-sensitive containment leaf as every other matcher. const clause = fieldPredicate( USER_TABLE_ROWS_SQL_NAME, @@ -759,67 +632,34 @@ export function validateColumnDefinition(column: ColumnDefinition): ValidationRe ) } - if (!COLUMN_TYPES.includes(column.type)) { + if (!isColumnType(column.type)) { errors.push( `Column "${column.name}" has invalid type "${column.type}". Valid types: ${COLUMN_TYPES.join(', ')}` ) + // Every check below reads the type's own rules; without a known type there + // are none to apply. + return { valid: false, errors } } - if (column.type === 'select') { - errors.push(...validateSelectOptions(column)) - // Uniqueness on a select compares the stored option id, so it caps each - // option at one row for the whole table — and the UI hides the toggle, so a - // constraint set through the API or an agent could never be cleared. - if (column.unique) { - errors.push(`Column "${column.name}" of type "select" cannot be unique`) - } - } else { - if (column.options !== undefined) { - errors.push(`Column "${column.name}" cannot define options for type "${column.type}"`) - } - // A stored `multiple` on a non-select column is inert until the column is - // converted, at which point `updateColumnType` inherits it — silently - // turning an intended single-select into a multiselect and rewriting every - // cell as an array. - if (column.multiple) { - errors.push(`Column "${column.name}" cannot be multiple for type "${column.type}"`) - } + const definition = COLUMN_TYPE_REGISTRY[column.type] + errors.push(...validateTypeMetadata(column)) + + // Uniqueness compares the stored value, which is meaningless for a type whose + // storage is an opaque id — it would cap each option at one row for the whole + // table, and the UI hides the toggle so it could never be cleared again. + if (column.unique && !definition.supportsUnique) { + errors.push(`Column "${column.name}" of type "${column.type}" cannot be unique`) + } + + // Type-specific metadata stored on the wrong type is inert until a later + // conversion inherits it — silently overriding what that request asked for. + const owned = new Set(definition.ownedMetadata) + for (const key of TYPE_SPECIFIC_COLUMN_KEYS) { + if (column[key] === undefined || owned.has(key)) continue + errors.push( + `Column "${column.name}" cannot ${FOREIGN_METADATA_VERB[key]} for type "${column.type}"` + ) } return { valid: errors.length === 0, errors } } - -/** Validates the option set declared on a `select` column. */ -function validateSelectOptions(column: ColumnDefinition): string[] { - const errors: string[] = [] - const options = column.options - if (!Array.isArray(options) || options.length === 0) { - errors.push(`Column "${column.name}" of type "${column.type}" must define at least one option`) - return errors - } - if (options.length > MAX_SELECT_OPTIONS) { - errors.push(`Column "${column.name}" cannot have more than ${MAX_SELECT_OPTIONS} options`) - } - const ids = new Set() - const names = new Set() - for (const opt of options) { - if (!opt.id || typeof opt.id !== 'string') { - errors.push(`Column "${column.name}" has an option missing an id`) - } else if (ids.has(opt.id)) { - errors.push(`Column "${column.name}" has duplicate option id "${opt.id}"`) - } else { - ids.add(opt.id) - } - if (!opt.name || typeof opt.name !== 'string') { - errors.push(`Column "${column.name}" has an option missing a name`) - } else { - const key = opt.name.toLowerCase() - if (names.has(key)) { - errors.push(`Column "${column.name}" has duplicate option name "${opt.name}"`) - } else { - names.add(key) - } - } - } - return errors -} diff --git a/apps/sim/stores/table/types.ts b/apps/sim/stores/table/types.ts index 030377b775..1da15ace21 100644 --- a/apps/sim/stores/table/types.ts +++ b/apps/sim/stores/table/types.ts @@ -60,6 +60,9 @@ export type TableUndoAction = // holds option ids, would have nothing to attach to. columnOptions?: ColumnDefinition['options'] columnMultiple?: boolean + // Likewise for a `currency` column: without its code the restore would + // silently re-denominate every cell to the default currency. + columnCurrencyCode?: string cellData: Array<{ rowId: string; value: unknown }> previousOrder: string[] | null previousWidth: number | null diff --git a/packages/emcn/src/icons/index.ts b/packages/emcn/src/icons/index.ts index 5bbd98df1b..1f871b6f68 100644 --- a/packages/emcn/src/icons/index.ts +++ b/packages/emcn/src/icons/index.ts @@ -103,6 +103,7 @@ export { TrashOutline } from './trash-outline' export { Trash2 } from './trash2' export { TriangleAlert } from './triangle-alert' export { TypeBoolean } from './type-boolean' +export { TypeCurrency } from './type-currency' export { TypeJson } from './type-json' export { TypeNumber } from './type-number' export { TypeText } from './type-text' diff --git a/packages/emcn/src/icons/type-currency.tsx b/packages/emcn/src/icons/type-currency.tsx new file mode 100644 index 0000000000..ca0bbd0d76 --- /dev/null +++ b/packages/emcn/src/icons/type-currency.tsx @@ -0,0 +1,26 @@ +import type { SVGProps } from 'react' + +/** + * Type currency icon component - dollar sign for currency columns + * @param props - SVG properties including className, fill, etc. + */ +export function TypeCurrency(props: SVGProps) { + return ( + + ) +}