fix(utils): drop the .js specifiers Turbopack cannot resolve (#6351)

* fix(utils): drop the .js specifiers Turbopack cannot resolve

Every dev server on staging is currently returning 500 from any route whose module
graph reaches the `@sim/utils` barrel:

  Module not found: Can't resolve './errors.js'
  > 1 | export { getErrorMessage, getPostgresErrorCode, toError } from './errors.js'

  Import trace:
    ./packages/utils/src/index.ts
    ./apps/sim/lib/embeddings/client.ts
    ./apps/sim/lib/knowledge/embeddings.ts
    ./apps/sim/app/api/knowledge/route.ts

`packages/utils/src/index.ts` addresses its siblings as `./errors.js` while the files
are `./errors.ts`. webpack rewrites that through `resolve.extensionAlias`; Turbopack has
no equivalent (vercel/next.js#82945). `next build` is webpack and `next dev` is
Turbopack, so this passes CI and breaks every local dev server — #6317 went green.

Nothing required the extensions: the repo is on `moduleResolution: "bundler"`, and no
other package barrel uses them.

Two changes, either of which fixes the symptom; both are here because they fail
differently:

- `packages/utils/src/index.ts` drops all 12 `.js` specifiers. Fixes the barrel for
  every current and future consumer.
- `apps/sim/lib/embeddings/client.ts` imports `chunkArray` from `@sim/utils/helpers`
  rather than the barrel. #6317 added the only bare-barrel `@sim/utils` import in the
  monorepo; the subpath form is the documented convention (CLAUDE.md, "Common
  Utilities") and resolves to one module instead of pulling twelve.

`scripts/check-import-specifiers.ts` fails the build on either shape and runs in CI.
Verified it goes red by restoring both halves of the bug. It scans only bundler-compiled
source — vitest and standalone `bun run` scripts resolve `.js` -> `.ts` themselves, so
flagging their specifiers would be noise.

Verified against a real dev server with production env: `/api/knowledge`,
`/api/tools/embeddings` and `/api/workflows/[id]/deploy` all go 500 -> 401, `/workspace`
renders, and the Turbopack log is free of resolution errors. `tsc --noEmit` clean,
`packages/utils` 147/147.

* refactor(scripts): resolve specifiers instead of pattern-matching one mistake

The first version banned `.js` specifiers by regex, which catches the bug that happened
and nothing adjacent to it. This runs the actual resolution algorithm with Turbopack's
rules — extensionAlias deliberately absent — and fails on anything that does not land on
a real file.

That covers the whole "Module not found" class rather than one shape of it: `.js`
specifiers, typo'd paths, files moved or deleted with a stale importer left behind, `@/`
aliases pointing nowhere, and `@sim/*` subpaths a package does not export. Verified
against three synthetic breakages the regex version passed clean:

    '@/lib/webhooks/providerz'  — '@/' alias matches a tsconfig path but nothing is there
    './does-not-exist'          — no file at that path
    '@sim/utils/chunking'       — @sim/utils does not export './chunking'

Getting to zero false positives on 37,307 specifiers needed three things the naive
version got wrong:

- tsconfig `paths` are per-workspace. `@/*` is `apps/sim/*` inside apps/sim but
  `apps/realtime/src/*` inside apps/realtime, and apps/sim maps `@sim/db/*` straight at
  the package directory, legitimately bypassing that package's exports map. One
  hardcoded alias produced ~30 false positives in apps/realtime alone.
- `exports` maps have wildcards. `@sim/emcn` publishes `"./*": "./src/*"`, so
  `@sim/emcn/components/code/code.css` is valid despite no literal entry.
- TSDoc contains example imports. `packages/db/triggers.ts` documents
  `import { ensureRowCountTriggers } from '@sim/db/triggers'` — a subpath the package
  deliberately does not export. Comments are now blanked in place, preserving byte
  offsets so reported line numbers stay exact.

* fix(scripts): close three coverage gaps in the specifier audit

Review round 1 on #6351. All three findings were real and all three let the exact
regression this guard exists for slip through.

- Reported line numbers were one early. `SPECIFIER_RE` opens with `(?:^|\n)`, so
  `m.index` is the newline ENDING the previous line, not the start of the statement.
  `./helpers.js` on line 13 was reported as line 12. Anchoring to the specifier's own
  offset is exact, and for a multi-line import it points at the `from '...'` line —
  where the reader needs to look anyway.

- `require()` was not scanned. This repo uses lazy requires deliberately to break import
  cycles: `tools/params.ts` reaches `@/blocks` that way and `blocks/blocks/agent.ts`
  reaches `@/blocks/registry`, 22 first-party call sites in total. Those edges resolve
  exactly like static ones, so a bad specifier in one fails identically. Verified by
  pointing `tools/params.ts` at a non-existent module and watching the audit catch it.

- `apps/docs` was not scanned, despite being a second Next.js app with its own
  `next.config.ts` — so it carries identical Turbopack exposure. Now covered, and clean.

Side-effect imports and dynamic `import()` were called out in the same round but are
already covered: the optional `from` group in `SPECIFIER_RE` matches bare `import '...'`,
and `DYNAMIC_RE` handles `import('...')`. That review ran against 1c6073e0, before the
resolver rewrite.

Coverage goes from 37,307 specifiers across 11,182 files to 37,438 across 11,243, still
with zero violations.

* chore(tools): regenerate the stale tool metadata

`bun run tool-metadata:check` has been failing on staging since #6317, so every PR
branched off it inherits a red CI regardless of its own contents. Reproduced against a
clean `origin/staging` to confirm it is not this branch's doing.

#6317 rewrote the embeddings tools' `apiKey` descriptions from provider-specific strings
to one generic string in `tools/embeddings/factory.ts`, but did not regenerate
`tools/generated/tool-metadata.ts`. The whole delta is 89 bytes of description text — the
tool set is unchanged at 4380 ids, none added, none removed:

    - "description":"Cohere Embeddings API key"
    + "description":"API key for the selected embedding provider"

The old strings no longer exist anywhere in source, so the generated file was the stale
side. `tool-metadata:check` passes after regenerating, and the generator's own resolver
cross-check agrees.

`mship:check` and `mship-tools:check` also fail locally, but neither is a CI gate and both
fail only because they read contracts from the sibling copilot repo, which is not checked
out here. Left alone.

* fix(scripts): substitute every wildcard in a resolved target

CodeQL js/incomplete-sanitization, two instances, both correct.

`String.replace('*', x)` fills only the first occurrence. Node's `exports`
resolver uses a global regex, so a target carrying more than one `*` — e.g.
`"./src/*/index-*.ts"` — gets every occurrence substituted. Replacing only the
first leaves a literal `*` in the path, so `probe()` finds nothing and the audit
reports a perfectly valid subpath as missing.

TypeScript `paths` allows at most one `*`, so the tsconfig branch was already
correct in practice; it changes for consistency and because nothing enforces that
assumption.

Not a suppression — the resolver now matches Node's behaviour. 37,438 specifiers
still resolve clean.

* fix(scripts): do not assert on generated output in the specifier audit

CI red on a fresh checkout, green locally — the tell that the audit was
depending on build state rather than on source.

apps/docs/lib/source.ts imports '@/.source/server'. apps/docs maps '@/.source/*'
at './.source/*', which fumadocs-mdx generates and apps/docs/.gitignore excludes.
It exists on any machine that has built the docs and is absent from CI's
checkout, so the audit reported a valid import as unresolvable.

A path landing in output the scanner itself refuses to read as source —
node_modules, a build directory, any dot-directory — is now treated as
unverifiable rather than missing. That is the consistent rule: if we do not scan
it as source, we cannot assert on its presence, and asserting anyway makes the
verdict depend on build order. Applied to all three resolution paths (relative,
tsconfig paths, exports map), with a GENERATED sentinel keeping 'matched but
generated' distinct from 'matched and genuinely missing'.

Only the repo-relative portion is inspected. Checking the absolute path would
match the '.claude/worktrees/...' a git worktree lives under and silently skip
every specifier in the repo.

Verified both directions: passes with apps/docs/.source moved away (CI's state),
and still catches a require('@/blocks/still-not-real') planted in tools/params.ts.

* refactor(scripts): trim the specifier audit's comments

The audit shipped at 24% comment lines — the header alone retold the whole
incident. Cut to 15% (452 -> 401 lines) by collapsing the narrative and keeping
only what the code cannot say: the webpack/Turbopack extensionAlias divergence,
why '.js' is a probed extension but not a fallback, why paths resolve
per-workspace, why targets substitute with replaceAll, why generated output is
unverifiable, and the '.claude/' worktree trap in the relative-path check.

No behaviour change: 37,437 specifiers still resolve clean.
This commit is contained in:
Waleed
2026-08-06 17:08:11 -07:00
committed by GitHub
parent 2b35a3c9c7
commit 10878fbde5
6 changed files with 420 additions and 14 deletions
+400
View File
@@ -0,0 +1,400 @@
/**
* Resolves every first-party import specifier the way Turbopack does, failing on any that
* does not land on a real file.
*
* `next build` runs webpack and `next dev` runs Turbopack, and they do not resolve the same
* specifiers. webpack rewrites `./errors.js` -> `./errors.ts` via `resolve.extensionAlias`;
* Turbopack has no equivalent (vercel/next.js#82945). So that shape builds green in CI and
* 500s on every developer's machine — CI never runs the Turbopack graph.
*
* Running real resolution rather than matching that one mistake covers the whole
* "Module not found" class: bad extensions, typo'd paths, stale importers of moved files,
* dead `@/` aliases, and `@sim/*` subpaths a package does not export.
*
* Skipped: bare npm specifiers (node_modules' business, and flaky on install state),
* type-only imports (erased before resolution), and tests plus `apps/*/scripts/**`,
* which run under vitest and bun — both of which do resolve `.js` -> `.ts`.
*
* Usage: `bun run scripts/check-import-specifiers.ts [--verbose]`
*/
import { readdirSync, readFileSync, statSync } from 'node:fs'
import { dirname, join, relative, resolve } from 'node:path'
import { fileURLToPath } from 'node:url'
const SCRIPT_DIR = dirname(fileURLToPath(import.meta.url))
const ROOT = resolve(SCRIPT_DIR, '..')
const SCAN_DIRS = ['apps/sim', 'apps/realtime', 'apps/docs', 'packages']
const SKIP_DIRS = new Set(['node_modules', '.next', 'dist', 'build', '.turbo'])
/**
* `.js` is listed because a real `foo.js` resolves fine. What does not happen is `./foo.js`
* falling back to `foo.ts` — that asymmetry is the bug this guard exists for.
*/
const EXTENSIONS = ['.ts', '.tsx', '.js', '.jsx', '.mjs', '.cjs', '.json']
/** Static value imports and re-exports. `import type` / `export type` are erased. */
const SPECIFIER_RE =
/(?:^|\n)\s*(?:import|export)\s+(?!type\s)(?:[\s\S]*?from\s*)?['"]([^'"]+)['"]/g
/** `import(...)` — resolved at call time, but the path still has to exist. */
const DYNAMIC_RE = /\bimport\s*\(\s*['"]([^'"]+)['"]\s*\)/g
/** Lazy `require()` is used here to break import cycles; those edges resolve like static ones. */
const REQUIRE_RE = /\brequire\s*\(\s*['"]([^'"]+)['"]\s*\)/g
/**
* Subpath-only packages. Opt-in: `@sim/emcn` and `@sim/desktop-bridge` are barrel-first by
* design, so flagging them would bury the one rule that matters.
*/
const SUBPATH_REQUIRED = new Set(['@sim/utils'])
/** Only source a bundler compiles — see the "Deliberately NOT checked" note above. */
function isCompiledSource(full: string, name: string): boolean {
if (!/\.(ts|tsx)$/.test(name) || name.endsWith('.d.ts')) return false
if (/\.(test|spec)\.tsx?$/.test(name)) return false
const rel = relative(ROOT, full)
return !rel.startsWith('apps/sim/scripts/') && !rel.startsWith('apps/realtime/scripts/')
}
function walk(dir: string, acc: string[] = []): string[] {
let entries
try {
entries = readdirSync(dir, { withFileTypes: true })
} catch {
return acc
}
for (const e of entries) {
if (e.name.startsWith('.') || SKIP_DIRS.has(e.name)) continue
const full = join(dir, e.name)
if (e.isDirectory()) walk(full, acc)
else if (isCompiledSource(full, e.name)) acc.push(full)
}
return acc
}
/**
* Build output the scanner will not read as source, so it cannot assert on its presence
* either — `apps/docs/.source` is generated by fumadocs-mdx and absent from a fresh checkout.
*
* Only the repo-relative portion is inspected: a git worktree lives under `.claude/`, which
* would otherwise make every specifier in the repo look generated.
*/
function isGeneratedPath(absolute: string): boolean {
const rel = relative(ROOT, absolute)
if (rel.startsWith('..')) return true
return rel.split('/').some((segment) => segment.startsWith('.') || SKIP_DIRS.has(segment))
}
function isFile(p: string): boolean {
try {
return statSync(p).isFile()
} catch {
return false
}
}
/** `<base>`, `<base><ext>`, or `<base>/index<ext>`. */
function probe(base: string): string | null {
if (isFile(base)) return base
for (const ext of EXTENSIONS) {
if (isFile(base + ext)) return base + ext
}
for (const ext of EXTENSIONS) {
const idx = join(base, `index${ext}`)
if (isFile(idx)) return idx
}
return null
}
/**
* `paths` from the workspace owning a file. Per-workspace, not global: `@/*` differs between
* apps/sim and apps/realtime, and apps/sim maps `@sim/db/*` straight at the package directory,
* bypassing its `exports` map.
*/
interface PathRule {
prefix: string
suffix: string
wildcard: boolean
/**
* Absolute targets, `*` substituted at match time with `replaceAll` — Node's `exports`
* resolver uses a global regex, so a target with two wildcards fills both.
*/
targets: string[]
}
interface Workspace {
dir: string
paths: PathRule[]
}
const workspaces: Workspace[] = []
for (const group of ['apps', 'packages']) {
let names: string[]
try {
names = readdirSync(join(ROOT, group))
} catch {
continue
}
for (const name of names) {
const dir = join(ROOT, group, name)
const tsconfig = join(dir, 'tsconfig.json')
if (!isFile(tsconfig)) continue
try {
const raw = readFileSync(tsconfig, 'utf8').replace(/^\s*\/\/.*$/gm, '')
const paths = JSON.parse(raw)?.compilerOptions?.paths ?? {}
const entries: PathRule[] = Object.entries<string[]>(paths).map(([pattern, targets]) => {
const [prefix, suffix = ''] = pattern.split('*')
return {
prefix,
suffix,
wildcard: pattern.includes('*'),
targets: targets.map((t) => resolve(dir, t)),
}
})
// Longest prefix wins, matching TypeScript's own precedence.
entries.sort((a, b) => b.prefix.length - a.prefix.length)
workspaces.push({ dir, paths: entries })
} catch {
/* unparseable tsconfig — skip rather than fail the whole run */
}
}
}
workspaces.sort((a, b) => b.dir.length - a.dir.length)
function workspaceFor(file: string): Workspace | undefined {
return workspaces.find((w) => file.startsWith(`${w.dir}/`))
}
/** Matched a tsconfig path, but every target is generated — distinct from missing (`null`). */
const GENERATED = Symbol('generated')
/** Resolve through the owning workspace's tsconfig `paths`. */
function resolveViaPaths(
spec: string,
importer: string
): string | null | undefined | typeof GENERATED {
const ws = workspaceFor(importer)
if (!ws) return undefined
for (const { prefix, suffix, wildcard, targets } of ws.paths) {
if (!spec.startsWith(prefix)) continue
if (!wildcard) {
if (spec !== prefix) continue
if (targets.every(isGeneratedPath)) return GENERATED
for (const t of targets) {
const hit = probe(t)
if (hit) return hit
}
return null
}
if (suffix && !spec.endsWith(suffix)) continue
const middle = spec.slice(prefix.length, suffix ? spec.length - suffix.length : undefined)
const filled = targets.map((t) => t.replaceAll('*', middle))
if (filled.every(isGeneratedPath)) return GENERATED
for (const t of filled) {
const hit = probe(t)
if (hit) return hit
}
return null
}
return undefined
}
/** Subpath -> target file, read from a workspace package's `exports` map. */
const pkgExportCache = new Map<string, Map<string, string> | null>()
function packageExports(pkg: string): Map<string, string> | null {
if (pkgExportCache.has(pkg)) return pkgExportCache.get(pkg) as Map<string, string> | null
const dir = join(ROOT, 'packages', pkg.replace('@sim/', ''))
let map: Map<string, string> | null = null
try {
const json = JSON.parse(readFileSync(join(dir, 'package.json'), 'utf8'))
if (json.name === pkg && json.exports) {
map = new Map()
for (const [key, val] of Object.entries<any>(json.exports)) {
const target = typeof val === 'string' ? val : (val?.default ?? val?.types)
if (typeof target === 'string') map.set(key, join(dir, target))
}
}
} catch {
/* not a workspace package, or unreadable */
}
pkgExportCache.set(pkg, map)
return map
}
type Outcome = { ok: true } | { ok: false; reason: string }
function resolveSpecifier(spec: string, importer: string): Outcome | null {
if (spec.startsWith('.')) {
const base = resolve(dirname(importer), spec)
if (isGeneratedPath(base)) return null
return probe(base) ? { ok: true } : { ok: false, reason: 'no file at that path' }
}
// tsconfig `paths` first — it legitimately overrides a package's exports map.
const viaPaths = resolveViaPaths(spec, importer)
if (viaPaths === GENERATED) return null
if (viaPaths) return { ok: true }
if (viaPaths === null) {
return {
ok: false,
reason: spec.startsWith('@/')
? "'@/' alias matches a tsconfig path but nothing is there"
: 'matches a tsconfig path but nothing is there',
}
}
if (spec.startsWith('@sim/')) {
const [, name, ...rest] = spec.split('/')
const pkg = `@sim/${name}`
const exports = packageExports(pkg)
if (!exports) return null // package not in packages/, or has no exports map
const key = rest.length ? `./${rest.join('/')}` : '.'
const exact = exports.get(key)
if (exact) {
if (isGeneratedPath(exact)) return null
return probe(exact) ? { ok: true } : { ok: false, reason: `${key} points at a missing file` }
}
// Wildcard subpaths, e.g. `"./*": "./src/*"` on @sim/emcn.
for (const [pattern, target] of exports) {
const star = pattern.indexOf('*')
if (star === -1) continue
const head = pattern.slice(0, star)
const tail = pattern.slice(star + 1)
if (!key.startsWith(head) || !key.endsWith(tail)) continue
const middle = key.slice(head.length, key.length - tail.length)
const filled = target.replaceAll('*', middle)
if (isGeneratedPath(filled)) return null
if (probe(filled)) return { ok: true }
return { ok: false, reason: `${pkg}'s '${pattern}' export has no file for '${key}'` }
}
return { ok: false, reason: `${pkg} does not export '${key}'` }
}
return null // bare npm specifier — not ours to verify
}
interface Violation {
file: string
line: number
specifier: string
kind: 'unresolved' | 'bare-barrel'
reason: string
}
const files = SCAN_DIRS.flatMap((d) => walk(join(ROOT, d)))
const violations: Violation[] = []
let checked = 0
/**
* Blank comments in place, preserving byte offsets so line numbers stay exact. TSDoc carries
* example imports that are not real edges — `packages/db/triggers.ts` documents a subpath the
* package deliberately does not export.
*/
function blankComments(src: string): string {
return src
.replace(/\/\*[\s\S]*?\*\//g, (m) => m.replace(/[^\n]/g, ' '))
.replace(/(^|[^:])\/\/[^\n]*/g, (m, lead) => lead + ' '.repeat(m.length - lead.length))
}
for (const file of files) {
const raw = readFileSync(file, 'utf8')
const src = blankComments(raw)
let lineStarts: number[] | null = null
const lineAt = (idx: number) => {
if (!lineStarts) {
lineStarts = [0]
for (let i = 0; i < src.length; i++) if (src[i] === '\n') lineStarts.push(i + 1)
}
let lo = 0
let hi = lineStarts.length - 1
while (lo < hi) {
const mid = (lo + hi + 1) >> 1
if (lineStarts[mid] <= idx) lo = mid
else hi = mid - 1
}
return lo + 1
}
for (const pattern of [SPECIFIER_RE, DYNAMIC_RE, REQUIRE_RE]) {
pattern.lastIndex = 0
let m = pattern.exec(src)
while (m !== null) {
const spec = m[1]
// `m.index` is the newline ending the previous line; the specifier's offset is exact.
const at = m.index + m[0].lastIndexOf(spec)
const outcome = resolveSpecifier(spec, file)
if (outcome) {
checked++
if (!outcome.ok) {
violations.push({
file: relative(ROOT, file),
line: lineAt(at),
specifier: spec,
kind: 'unresolved',
reason: outcome.reason,
})
}
}
if (pattern === SPECIFIER_RE && SUBPATH_REQUIRED.has(spec)) {
const subs = packageExports(spec)
const example = subs ? [...subs.keys()].find((k) => k !== '.') : undefined
violations.push({
file: relative(ROOT, file),
line: lineAt(at),
specifier: spec,
kind: 'bare-barrel',
reason: example
? `import from a subpath instead, e.g. '${spec}${example.slice(1)}'`
: 'import from a subpath instead',
})
}
m = pattern.exec(src)
}
}
}
const verbose = process.argv.includes('--verbose')
if (violations.length === 0) {
console.log(
`✓ check-import-specifiers: ${checked} first-party specifiers across ${files.length} files all resolve`
)
process.exit(0)
}
const unresolved = violations.filter((v) => v.kind === 'unresolved')
const barrels = violations.filter((v) => v.kind === 'bare-barrel')
if (unresolved.length) {
console.error(`\n✗ ${unresolved.length} specifier(s) do not resolve:\n`)
for (const v of unresolved) {
console.error(` ${v.file}:${v.line}`)
console.error(` '${v.specifier}' — ${v.reason}`)
if (/\.(js|jsx|mjs)$/.test(v.specifier)) {
console.error(` drop the extension: '${v.specifier.replace(/\.\w+$/, '')}'`)
}
}
console.error(
"\n These are 'Module not found' at dev time. A '.js' specifier pointing at a '.ts'\n" +
' file is the common case: webpack rewrites it via resolve.extensionAlias, Turbopack\n' +
' does not (vercel/next.js#82945). CI builds with webpack and every developer runs\n' +
" Turbopack, so this class of break is invisible to CI. moduleResolution is 'bundler'\n" +
' here — extensions are never required.\n'
)
}
if (barrels.length) {
console.error(`\n✗ ${barrels.length} bare barrel import(s) of a subpath-only package:\n`)
for (const v of barrels) {
console.error(` ${v.file}:${v.line} '${v.specifier}'`)
console.error(` ${v.reason}`)
}
console.error(
'\n A barrel import pulls every module the barrel re-exports, so one helper drags in\n' +
' the whole package — and one bad specifier anywhere inside it takes the importer down.\n'
)
}
if (verbose) console.error(`\nscanned ${files.length} files, ${checked} first-party specifiers`)
process.exit(1)