opensource/sim
3
0
Fork 0
mirror of https://github.com/simstudioai/sim.git synced 2026-10-30 19:32:46 +08:00
Code Issues Packages Projects Releases Wiki Activity

fix(schedule): reconcile interrupted schedule executions (#6780)

Browse Source 
* fix schedule execution recovery

* simplify schedule recovery provider handling

* fix(schedules): stop reconciliation rewriting healthy carriers and index its scan

Follow-up hardening on the schedule recovery reconciliation.

Reconciliation settled every carrier it examined, including ones that had
already reached a terminal status and recorded their own outcome. Because the
normal completion path never stamps the reconciled marker, each successful
run's carrier was picked up on the next tick and rewritten: `completedAt`
bumped, `error` nulled and `output` replaced with a recovery stub, all of which
`GET /api/jobs/[jobId]` surfaces. A completed carrier whose execution log had
aged out was additionally flipped to failed. Settle only carriers still in
flight; a terminal one is owed schedule accounting and the marker, nothing more.

The recovery scan matched no index. There was none on `async_jobs.updated_at`,
and the unreconciled-terminal branch tested a jsonb extraction, so the whole
OR fell back to a sequential scan and sort of `async_jobs` on every tick. Add
the partial index, and spell the branch's status list and metadata key as SQL
literals: Postgres cannot prove a parameterised predicate implies a literal
index predicate, so a bound key would have left the new index unused.

Irrecoverable carrier tombstones were exempted from retention with no secondary
expiry, so the one class of row that can never reconcile grew without bound.
Give them a longer bounded window instead.

`WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN` had become a per-status budget when the
stale-execution sweep gained its `redacting` pass, silently doubling the
per-run cap. Share the budget across both passes, and add the matching partial
indexes so the second pass keeps an index rather than seq-scanning.

Also consolidates the carrier metadata keys, their predicates and the jsonb
merge into one module -- the two routes were the writer and reader of the same
keys with no shared symbol, so a rename type-checked clean while silently
breaking retention.

* fix(schedules): rotate deferred carriers and stop the redacting sweep failing live runs

Addresses the review findings on the reconciliation hardening.

A carrier whose accounting was deferred got no write at all, so it kept its
`updatedAt` and stayed at the head of the `updatedAt`-ordered recovery batch on
every tick, starving every other claimed carrier and never becoming eligible for
retention. This was a regression from the previous commit: settling used to bump
the timestamp for every examined row, and skipping the settle for already-terminal
carriers removed the bump with it. A payload with no `scheduledFor` can never
reconcile, so such a row pinned a batch slot permanently. Bump `updatedAt`
unconditionally and keep only the reconciled marker conditional.

The stale-execution sweep terminalized `redacting` logs on the execution deadline.
That deadline bounds execution, while `redacting` covers payload masking after the
run already finished -- so a run that used most of its budget entered redaction
with the deadline due, and the sweep failed it five minutes later while the worker
was still masking. Schedule recovery then read the log as a failed occurrence and
counted a failure that never happened, even though the worker's terminal write
later restored `completed`. Sweep `redacting` on the generic stale window only.

`getScheduleNextRunAt` falls back to a daily cadence when a schedule has no cron
expression. Deployment cannot persist such a schedule, so the branch is
unreachable, but this change widened its use from failure recovery to every
outcome -- log a warning when it fires rather than silently guessing a cadence.

`executionDeadlineAt` was missing from the shared `workflowExecutionLogs` schema
mock, so it read as `undefined` and assertions comparing against that column were
trivially true. Add it.

* refactor(schedules): drop vestigial recovery code and close a builder gap

Follow-ups from a full re-read of the change. No behavior change except the
removed dead code paths.

The metadata merge stripped a `scheduleRecoveryBlocked` key on every write. That
key has never been written by any shipped code -- it appears nowhere in staging
and nowhere in history outside this branch -- so the strip guarded against a
state that cannot exist, at the cost of an extra jsonb operation and a bind
parameter on every reconciliation write.

`processScheduleItem` set `carrierObservedOrLookupUncertain` immediately before
returning on an ambiguous enqueue. The flag is only read from the surrounding
catch block, which a normal return skips, so the assignment was dead and read as
though it were load-bearing. Replaced with a comment stating why the occurrence
is preserved.

The stale-execution sweep carried two near-identical `jsonb_set` templates that
differed only in their error expression, kept flat because the test mock renders
nested SQL fragments as placeholders. The suite now has a recursive renderer, so
the error expression is a named per-status value and there is one `jsonb_set`.

Success was the only schedule outcome without a named update builder, which left
`executeScheduleJob` using two idioms for the same guarded write and left the
update shape untested. Add `buildScheduleSuccessUpdate` beside its cancellation
and failure siblings, use it from both call sites, and cover it the way
`buildScheduleCancellationUpdate` is covered -- a mutation of its `failedCount`
reset previously passed every suite.

---------

Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>
Co-authored-by: Waleed Latif <walif6@gmail.com>
This commit is contained in:
Bill Leoutsakos
2026-08-20 11:55:45 -07:00
committed by GitHub
co-authored by Bill Leoutsakos Waleed Latif
parent 85902eb7ff
commit 3b4d9e9149
14 changed files with 21782 additions and 416 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/sim/app/api/cron/cleanup-stale-executions/route.test.ts
+196 -20
View File
@@ -5,7 +5,16 @@ import { asyncJobs, tableJobs, workflowExecutionLogs } from '@sim/db/schema'
import { createLogger } from '@sim/logger'
import { createMockRequest, dbChainMockFns, queueTableRows, resetDbChainMock } from '@sim/testing'
import { beforeEach, describe, expect, it, vi } from 'vitest'
import { MAX_JOB_DURATION_SECONDS, MIN_JOB_DURATION_SECONDS } from '@/lib/core/async-jobs'
import {
JOB_RETENTION_HOURS,
MAX_JOB_DURATION_SECONDS,
MIN_JOB_DURATION_SECONDS,
} from '@/lib/core/async-jobs'
import {
SCHEDULE_CARRIER_IRRECOVERABLE_METADATA_KEY,
SCHEDULE_CARRIER_IRRECOVERABLE_RETENTION_HOURS,
SCHEDULE_CARRIER_RECONCILED_METADATA_KEY,
} from '@/lib/workflows/schedules/carrier-metadata'
const { mockDeleteFile, mockVerifyCronAuth } = vi.hoisted(() => ({
mockDeleteFile: vi.fn().mockResolvedValue(undefined),
@@ -27,6 +36,7 @@ interface MockCondition {
conditions?: unknown[]
left?: unknown
right?: unknown
column?: unknown
values?: unknown
toSQL?: () => { sql: string; params: unknown[] }
}
@@ -61,6 +71,33 @@ function createRequest() {
)
}
/** Recursively renders a mocked drizzle fragment, expanding nested fragments. */
function renderSql(fragment: unknown): string {
if (fragment === null || fragment === undefined) return ''
if (typeof fragment !== 'object') return String(fragment)
const candidate = fragment as { rawSql?: string; strings?: string[]; values?: unknown[] }
if (typeof candidate.rawSql === 'string') return candidate.rawSql
if (!candidate.strings) return ''
return candidate.strings
.map((part, index) =>
index < (candidate.values?.length ?? 0)
? `${part}${renderSql(candidate.values?.[index])}`
: part
)
.join('')
}
/** Recursively collects the non-fragment bind values of a mocked fragment. */
function collectSqlParams(fragment: unknown): unknown[] {
if (!fragment || typeof fragment !== 'object') return []
const candidate = fragment as { values?: unknown[] }
if (!candidate.values) return []
return candidate.values.flatMap((value) => {
const nested = collectSqlParams(value)
return nested.length > 0 ? nested : [value]
})
}
describe('stale execution cleanup deadline grace', () => {
beforeEach(() => {
vi.clearAllMocks()
@@ -103,25 +140,18 @@ describe('stale execution cleanup deadline grace', () => {
totalDurationMs: { toSQL: () => { sql: string; params: unknown[] } }
executionData: { toSQL: () => { sql: string; params: unknown[] } }
}
const errorExpression = update.executionData.toSQL()
const staleDurationExpression = errorExpression.params.find(
(value): value is { toSQL: () => { sql: string; params: unknown[] } } =>
typeof value === 'object' &&
value !== null &&
'toSQL' in value &&
value.toSQL().sql.includes('EXTRACT(EPOCH')
)
const totalDurationLeaves = flattenSqlParams(update.totalDurationMs.toSQL())
const renderedError = renderSql(update.executionData)
const errorLeaves = collectSqlParams(update.executionData)
expect(errorExpression.sql).toContain('CASE')
expect(errorExpression.sql).toContain('IS NOT NULL')
expect(errorExpression.params).toContain(workflowExecutionLogs.executionDeadlineAt)
expect(errorExpression.params).toContain('Execution timed out')
expect(errorExpression.params).toContain(
'Execution terminated: worker timeout or crash after '
)
expect(staleDurationExpression?.toSQL().sql).toContain('ROUND')
expect(staleDurationExpression?.toSQL().params).toContain(workflowExecutionLogs.startedAt)
expect(renderedError).toContain('CASE')
expect(renderedError).toContain('IS NOT NULL')
expect(renderedError).toContain('ROUND')
expect(renderedError).toContain('EXTRACT(EPOCH')
expect(errorLeaves).toContain(workflowExecutionLogs.executionDeadlineAt)
expect(errorLeaves).toContain('Execution timed out')
expect(errorLeaves).toContain('Execution terminated: worker timeout or crash after ')
expect(errorLeaves).toContain(workflowExecutionLogs.startedAt)
expect(totalDurationLeaves).toContain(2_147_483_647)
expect(totalDurationLeaves).toContain(workflowExecutionLogs.startedAt)
expect(totalDurationLeaves).toContainEqual(new Date('2026-08-03T12:10:00.000Z'))
@@ -131,6 +161,50 @@ describe('stale execution cleanup deadline grace', () => {
}
})
it('sweeps redacting logs on the generic window, never the execution deadline', async () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
const redactingPredicates = dbChainMockFns.where.mock.calls
.map(([condition]) => flattenConditions(condition))
.filter((conditions) =>
conditions.some(
(condition) =>
condition.type === 'eq' &&
condition.left === workflowExecutionLogs.status &&
condition.right === 'redacting'
)
)
expect(redactingPredicates.length).toBeGreaterThan(0)
for (const conditions of redactingPredicates) {
expect(
conditions.some((condition) => condition.left === workflowExecutionLogs.executionDeadlineAt)
).toBe(false)
expect(
conditions.some(
(condition) =>
condition.type === 'lt' && condition.left === workflowExecutionLogs.startedAt
)
).toBe(true)
}
})
it('terminalizes stale running and redacting execution logs', async () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
const statusPredicates = dbChainMockFns.where.mock.calls
.flatMap(([condition]) => flattenConditions(condition))
.filter(
(condition) => condition.type === 'eq' && condition.left === workflowExecutionLogs.status
)
expect(statusPredicates.map(({ right }) => right)).toEqual(
expect.arrayContaining(['running', 'redacting'])
)
})
it('reports a worker cleanup deadline while preserving the generic stale fallback', async () => {
queueTableRows(asyncJobs, [{ id: 'async-job-1' }])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'async-job-1' }])
@@ -175,6 +249,108 @@ describe('stale execution cleanup deadline grace', () => {
)
})
it('leaves pending and processing schedule jobs to schedule recovery', async () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
const activeAsyncPredicates = dbChainMockFns.where.mock.calls
.map(([condition]) => flattenConditions(condition))
.filter((conditions) =>
conditions.some(
(condition) =>
condition.type === 'ne' &&
condition.left === asyncJobs.type &&
condition.right === 'schedule-execution'
)
)
expect(
activeAsyncPredicates.some((conditions) =>
conditions.some(
(condition) =>
condition.type === 'eq' &&
condition.left === asyncJobs.status &&
condition.right === 'processing'
)
)
).toBe(true)
expect(
activeAsyncPredicates.some((conditions) =>
conditions.some(
(condition) =>
condition.type === 'eq' &&
condition.left === asyncJobs.status &&
condition.right === 'pending'
)
)
).toBe(true)
})
it('retains terminal schedule carriers until reconciliation is recorded', async () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
const retentionConditions = dbChainMockFns.where.mock.calls.flatMap(([condition]) =>
flattenConditions(condition)
)
const reconciliationMarker = retentionConditions.find((condition) =>
renderSql(condition).includes(SCHEDULE_CARRIER_RECONCILED_METADATA_KEY)
)
expect(collectSqlParams(reconciliationMarker)).toContain(asyncJobs.metadata)
expect(
retentionConditions.some(
(condition) =>
condition.type === 'ne' &&
condition.left === asyncJobs.type &&
condition.right === 'schedule-execution'
)
).toBe(true)
})
it('spells carrier metadata keys as SQL literals so the partial index matches', async () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
const reconciliationMarker = dbChainMockFns.where.mock.calls
.flatMap(([condition]) => flattenConditions(condition))
.find((condition) => renderSql(condition).includes(SCHEDULE_CARRIER_RECONCILED_METADATA_KEY))
expect(renderSql(reconciliationMarker)).toContain(
`'${SCHEDULE_CARRIER_RECONCILED_METADATA_KEY}'`
)
expect(collectSqlParams(reconciliationMarker)).not.toContain(
SCHEDULE_CARRIER_RECONCILED_METADATA_KEY
)
})
it('deletes irrecoverable schedule carrier tombstones once their longer window lapses', async () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
const retentionConditions = dbChainMockFns.where.mock.calls.flatMap(([condition]) =>
flattenConditions(condition)
)
const irrecoverableExclusion = retentionConditions.find((condition) =>
renderSql(condition).includes(SCHEDULE_CARRIER_IRRECOVERABLE_METADATA_KEY)
)
expect(renderSql(irrecoverableExclusion)).toContain("<> 'true'")
expect(collectSqlParams(irrecoverableExclusion)).toContain(asyncJobs.metadata)
const tombstoneWindow = retentionConditions.filter(
(condition) =>
condition.type === 'lt' &&
condition.left === asyncJobs.completedAt &&
condition.right instanceof Date
)
const oldest = Math.min(...tombstoneWindow.map(({ right }) => (right as Date).getTime()))
const newest = Math.max(...tombstoneWindow.map(({ right }) => (right as Date).getTime()))
expect(newest - oldest).toBe(
(SCHEDULE_CARRIER_IRRECOVERABLE_RETENTION_HOURS - JOB_RETENTION_HOURS) * 60 * 60 * 1000
)
})
it('keeps table-job heartbeat cleanup independent from workflow timeout policy', async () => {
vi.useFakeTimers()
vi.setSystemTime(new Date('2026-08-03T12:00:00.000Z'))
@@ -220,8 +396,8 @@ describe('stale execution cleanup deadline grace', () => {
const response = await GET(createRequest())
expect(response.status).toBe(200)
expect(dbChainMockFns.transaction).toHaveBeenCalledTimes(7)
expect(dbChainMockFns.for).toHaveBeenCalledTimes(7)
expect(dbChainMockFns.transaction).toHaveBeenCalledTimes(8)
expect(dbChainMockFns.for).toHaveBeenCalledTimes(8)
for (const [strength, options] of dbChainMockFns.for.mock.calls) {
expect(strength).toBe('update')
expect(options).toEqual({ skipLocked: true })
apps/sim/app/api/cron/cleanup-stale-executions/route.ts
+124 -63
View File
@@ -7,7 +7,7 @@ import {
} from '@sim/db/schema'
import { createLogger } from '@sim/logger'
import { toError } from '@sim/utils/errors'
import { and, eq, exists, gt, inArray, isNull, lt, or, sql } from 'drizzle-orm'
import { and, eq, exists, gt, inArray, isNull, lt, ne, or, sql } from 'drizzle-orm'
import { alias } from 'drizzle-orm/pg-core'
import { type NextRequest, NextResponse } from 'next/server'
import { verifyCronAuth } from '@/lib/auth/internal'
@@ -17,6 +17,7 @@ import {
JOB_STATUS,
MAX_JOB_DURATION_SECONDS,
MIN_JOB_DURATION_SECONDS,
TERMINAL_JOB_STATUSES,
} from '@/lib/core/async-jobs'
import {
getExecutionReservationTtlMs,
@@ -26,7 +27,17 @@ import {
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import type { DbTransaction } from '@/lib/db/types'
import { elapsedDurationMsSql } from '@/lib/logs/execution/duration'
import {
STALE_SWEEPABLE_EXECUTION_STATUSES,
type StaleSweepableExecutionStatus,
} from '@/lib/logs/types'
import { deleteFile } from '@/lib/uploads/core/storage-service'
import {
carrierNotIrrecoverableSql,
carrierReconciledSql,
SCHEDULE_CARRIER_IRRECOVERABLE_RETENTION_HOURS,
} from '@/lib/workflows/schedules/carrier-metadata'
import { SCHEDULE_EXECUTION_QUEUE_NAME } from '@/lib/workflows/schedules/execution-limits'
const logger = createLogger('CleanupStaleExecutions')
@@ -140,79 +151,109 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
let currentWorkflowBatchSize = 0
try {
const staleExecutionPredicate = and(
eq(workflowExecutionLogs.status, 'running'),
or(
lt(workflowExecutionLogs.executionDeadlineAt, staleDeadlineThreshold),
and(
isNull(workflowExecutionLogs.executionDeadlineAt),
lt(workflowExecutionLogs.startedAt, staleThreshold)
)
)
)
/**
* `running` is swept on its execution deadline plus the cleanup grace, or
* on the generic stale window when it has no deadline.
*
* `redacting` gets the generic window only. That status is set *after* the
* run finished, while a live worker masks the payload, so the execution
* deadline is already in the past by the time redaction starts — the
* deadline rule would fail a masking pass that is merely slow, and
* schedule recovery would read that as a failed occurrence even though
* the worker is about to persist `completed`. A redaction that genuinely
* crashed still clears within the generic window.
*/
const staleExecutionTimePredicate = (status: StaleSweepableExecutionStatus) =>
status === 'redacting'
? lt(workflowExecutionLogs.startedAt, staleThreshold)
: or(
lt(workflowExecutionLogs.executionDeadlineAt, staleDeadlineThreshold),
and(
isNull(workflowExecutionLogs.executionDeadlineAt),
lt(workflowExecutionLogs.startedAt, staleThreshold)
)
)
const cleanupTimestamp = sql.param(now, workflowExecutionLogs.startedAt)
const staleDurationMinutes = sql<number>`ROUND(
EXTRACT(EPOCH FROM (${cleanupTimestamp} - ${workflowExecutionLogs.startedAt})) / 60
)::integer`
const totalDurationMs = elapsedDurationMsSql(now)
const staleDurationError = sql`${'Execution terminated: worker timeout or crash after '}::text
|| ${staleDurationMinutes}::text
|| ' minutes'`
/**
* A `redacting` row is never swept by the deadline rule, so the deadline
* message cannot apply to it.
*/
const staleExecutionError = (status: StaleSweepableExecutionStatus) =>
status === 'redacting'
? staleDurationError
: sql`CASE
WHEN ${workflowExecutionLogs.executionDeadlineAt} IS NOT NULL
THEN ${EXECUTION_DEADLINE_ERROR}::text
ELSE ${staleDurationError}
END`
/**
* Swept one status at a time so each pass stays on its own partial index;
* `status IN (...)` would match neither. The row budget is shared across
* both passes so the per-run cap keeps meaning what its name says.
*/
let workflowRowsConsidered = 0
while (workflowRowsConsidered < WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN) {
const limit = Math.min(
WORKFLOW_EXECUTION_MUTATION_BATCH_SIZE,
WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN - workflowRowsConsidered
for (const executionStatus of STALE_SWEEPABLE_EXECUTION_STATUSES) {
const staleExecutionPredicate = and(
eq(workflowExecutionLogs.status, executionStatus),
staleExecutionTimePredicate(executionStatus)
)
currentWorkflowBatchSize = 0
const { candidates, updatedExecutions } = await db.transaction(async (tx) => {
const candidates = await tx
.select({ id: workflowExecutionLogs.id })
.from(workflowExecutionLogs)
.where(staleExecutionPredicate)
.limit(limit)
.for('update', { skipLocked: true })
currentWorkflowBatchSize = candidates.length
if (candidates.length === 0) return { candidates, updatedExecutions: [] }
while (workflowRowsConsidered < WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN) {
const limit = Math.min(
WORKFLOW_EXECUTION_MUTATION_BATCH_SIZE,
WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN - workflowRowsConsidered
)
currentWorkflowBatchSize = 0
const { candidates, updatedExecutions } = await db.transaction(async (tx) => {
const candidates = await tx
.select({ id: workflowExecutionLogs.id })
.from(workflowExecutionLogs)
.where(staleExecutionPredicate)
.limit(limit)
.for('update', { skipLocked: true })
currentWorkflowBatchSize = candidates.length
if (candidates.length === 0) return { candidates, updatedExecutions: [] }
const updatedExecutions = await tx
.update(workflowExecutionLogs)
.set({
status: 'failed',
endedAt: now,
executionDeadlineAt: null,
totalDurationMs,
executionData: sql`jsonb_set(
COALESCE(execution_data, '{}'::jsonb),
ARRAY['error'],
to_jsonb(
CASE
WHEN ${workflowExecutionLogs.executionDeadlineAt} IS NOT NULL
THEN ${EXECUTION_DEADLINE_ERROR}::text
ELSE ${'Execution terminated: worker timeout or crash after '}::text
|| ${staleDurationMinutes}::text
|| ' minutes'
END
)
)`,
})
.where(
and(
staleExecutionPredicate,
inArray(
workflowExecutionLogs.id,
candidates.map(({ id }) => id)
const updatedExecutions = await tx
.update(workflowExecutionLogs)
.set({
status: 'failed',
endedAt: now,
executionDeadlineAt: null,
totalDurationMs,
executionData: sql`jsonb_set(
COALESCE(execution_data, '{}'::jsonb),
ARRAY['error'],
to_jsonb(${staleExecutionError(executionStatus)})
)`,
})
.where(
and(
staleExecutionPredicate,
inArray(
workflowExecutionLogs.id,
candidates.map(({ id }) => id)
)
)
)
)
.returning({ id: workflowExecutionLogs.id })
.returning({ id: workflowExecutionLogs.id })
return { candidates, updatedExecutions }
})
currentWorkflowBatchSize = 0
staleExecutionsFound += candidates.length
if (candidates.length === 0) break
return { candidates, updatedExecutions }
})
currentWorkflowBatchSize = 0
staleExecutionsFound += candidates.length
if (candidates.length === 0) break
cleaned += updatedExecutions.length
workflowRowsConsidered += candidates.length
if (candidates.length < limit) break
cleaned += updatedExecutions.length
workflowRowsConsidered += candidates.length
if (candidates.length < limit) break
}
}
if (workflowRowsConsidered >= WORKFLOW_EXECUTION_MAX_ROWS_PER_RUN) {
@@ -253,6 +294,7 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
END`
const staleProcessingPredicate = and(
eq(asyncJobs.status, JOB_STATUS.PROCESSING),
ne(asyncJobs.type, SCHEDULE_EXECUTION_QUEUE_NAME),
staleProcessingDurationPredicate
)
const staleProcessingResult = await runBatchedMutation({
@@ -395,6 +437,7 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
try {
const stalePendingPredicate = and(
eq(asyncJobs.status, JOB_STATUS.PENDING),
ne(asyncJobs.type, SCHEDULE_EXECUTION_QUEUE_NAME),
lt(asyncJobs.createdAt, stalePendingThreshold)
)
const stalePendingResult = await runBatchedMutation({
@@ -436,11 +479,29 @@ export const GET = withRouteHandler(async (request: NextRequest) => {
}
const retentionThreshold = new Date(Date.now() - JOB_RETENTION_HOURS * 60 * 60 * 1000)
const irrecoverableCarrierRetentionThreshold = new Date(
Date.now() - SCHEDULE_CARRIER_IRRECOVERABLE_RETENTION_HOURS * 60 * 60 * 1000
)
let asyncJobsDeleted = 0
try {
const retainedJobPredicate = and(
inArray(asyncJobs.status, [JOB_STATUS.COMPLETED, JOB_STATUS.FAILED, JOB_STATUS.CANCELLED]),
inArray(asyncJobs.status, TERMINAL_JOB_STATUSES),
or(
ne(asyncJobs.type, SCHEDULE_EXECUTION_QUEUE_NAME),
/**
* Schedule recovery owns a carrier until it stamps the reconciled
* marker, so retention waits for it rather than deleting an
* occurrence that has not been accounted for yet.
*/
and(
carrierReconciledSql(asyncJobs.metadata),
or(
carrierNotIrrecoverableSql(asyncJobs.metadata),
lt(asyncJobs.completedAt, irrecoverableCarrierRetentionThreshold)
)
)
),
lt(asyncJobs.completedAt, retentionThreshold)
)
const retainedJobResult = await runBatchedMutation({
apps/sim/app/api/schedules/execute/route.test.ts
+628 -142
View File
@@ -7,6 +7,7 @@ import {
createMockSql,
dbChainMock,
dbChainMockFns,
queueTableRows,
requestUtilsMockFns,
resetDbChainMock,
resetEnvFlagsMock,
@@ -16,6 +17,7 @@ import {
} from '@sim/testing'
import { type NextRequest, NextResponse } from 'next/server'
import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
import { AsyncJobEnqueueError } from '@/lib/core/async-jobs/types'
const orderByLimitMock = vi.fn()
@@ -32,10 +34,15 @@ const {
mockShouldExecuteInline,
mockResolveSystemBillingAttribution,
mockAssertBillingAttributionSnapshot,
mockApplyScheduleSuccessUpdate,
mockApplyScheduleCancellationUpdate,
mockApplyScheduleFailureUpdate,
mockNotifyScheduleAutoDisabled,
mockRegisterManualExecutionAborter,
mockUnregisterManualExecutionAborter,
mockAsyncJobs,
mockWorkflowSchedule,
mockWorkflowExecutionLogs,
} = vi.hoisted(() => ({
mockVerifyCronAuth: vi.fn().mockReturnValue(null),
mockExecuteScheduleJob: vi.fn().mockResolvedValue(undefined),
@@ -49,10 +56,49 @@ const {
mockShouldExecuteInline: vi.fn().mockReturnValue(false),
mockResolveSystemBillingAttribution: vi.fn(),
mockAssertBillingAttributionSnapshot: vi.fn(),
mockApplyScheduleSuccessUpdate: vi.fn().mockResolvedValue(true),
mockApplyScheduleCancellationUpdate: vi.fn().mockResolvedValue(true),
mockApplyScheduleFailureUpdate: vi.fn().mockResolvedValue({ updated: true, disabled: false }),
mockNotifyScheduleAutoDisabled: vi.fn().mockResolvedValue(undefined),
mockRegisterManualExecutionAborter: vi.fn(),
mockUnregisterManualExecutionAborter: vi.fn(),
mockAsyncJobs: {
id: 'id',
type: 'type',
payload: 'payload',
status: 'status',
createdAt: 'createdAt',
runAt: 'runAt',
startedAt: 'startedAt',
completedAt: 'completedAt',
attempts: 'attempts',
maxAttempts: 'maxAttempts',
error: 'error',
output: 'output',
metadata: 'metadata',
updatedAt: 'updatedAt',
},
mockWorkflowSchedule: {
id: 'id',
workflowId: 'workflowId',
blockId: 'blockId',
cronExpression: 'cronExpression',
lastRanAt: 'lastRanAt',
failedCount: 'failedCount',
infraRetryCount: 'infraRetryCount',
status: 'status',
timezone: 'timezone',
nextRunAt: 'nextRunAt',
lastQueuedAt: 'lastQueuedAt',
archivedAt: 'archivedAt',
deploymentVersionId: 'deploymentVersionId',
sourceType: 'sourceType',
},
mockWorkflowExecutionLogs: {
executionId: 'executionId',
workflowId: 'workflowId',
status: 'status',
},
}))
vi.mock('@/lib/auth/internal', () => ({
@@ -67,6 +113,8 @@ vi.mock('@/lib/billing/core/billing-attribution', () => ({
vi.mock('@/background/schedule-execution', () => ({
executeScheduleJob: mockExecuteScheduleJob,
releaseScheduleLock: mockReleaseScheduleLock,
applyScheduleSuccessUpdate: mockApplyScheduleSuccessUpdate,
applyScheduleCancellationUpdate: mockApplyScheduleCancellationUpdate,
applyScheduleFailureUpdate: mockApplyScheduleFailureUpdate,
}))
@@ -95,6 +143,7 @@ vi.mock('@/lib/core/async-jobs', () => ({
vi.mock('drizzle-orm', () => ({
and: vi.fn((...conditions: unknown[]) => ({ type: 'and', conditions })),
eq: vi.fn((field: unknown, value: unknown) => ({ field, value, type: 'eq' })),
gt: vi.fn((field: unknown, value: unknown) => ({ field, value, type: 'gt' })),
ne: vi.fn((field: unknown, value: unknown) => ({ field, value, type: 'ne' })),
lte: vi.fn((field: unknown, value: unknown) => ({ field, value, type: 'lte' })),
lt: vi.fn((field: unknown, value: unknown) => ({ field, value, type: 'lt' })),
@@ -108,21 +157,7 @@ vi.mock('drizzle-orm', () => ({
vi.mock('@sim/db', () => ({
...dbChainMock,
workflowSchedule: {
id: 'id',
workflowId: 'workflowId',
blockId: 'blockId',
cronExpression: 'cronExpression',
lastRanAt: 'lastRanAt',
failedCount: 'failedCount',
infraRetryCount: 'infraRetryCount',
status: 'status',
timezone: 'timezone',
nextRunAt: 'nextRunAt',
lastQueuedAt: 'lastQueuedAt',
deploymentVersionId: 'deploymentVersionId',
sourceType: 'sourceType',
},
workflowSchedule: mockWorkflowSchedule,
workflowDeploymentVersion: {
id: 'id',
workflowId: 'workflowId',
@@ -133,20 +168,8 @@ vi.mock('@sim/db', () => ({
userId: 'userId',
workspaceId: 'workspaceId',
},
asyncJobs: {
id: 'id',
type: 'type',
payload: 'payload',
status: 'status',
createdAt: 'createdAt',
runAt: 'runAt',
startedAt: 'startedAt',
completedAt: 'completedAt',
attempts: 'attempts',
maxAttempts: 'maxAttempts',
error: 'error',
updatedAt: 'updatedAt',
},
asyncJobs: mockAsyncJobs,
workflowExecutionLogs: mockWorkflowExecutionLogs,
}))
vi.mock('@sim/utils/id', () => ({
@@ -311,6 +334,12 @@ describe('Scheduled Workflow Execution API Route', () => {
mockExecuteScheduleJob.mockResolvedValue(undefined)
mockReleaseScheduleLock.mockReset()
mockReleaseScheduleLock.mockResolvedValue(undefined)
mockApplyScheduleSuccessUpdate.mockReset()
mockApplyScheduleSuccessUpdate.mockResolvedValue(true)
mockApplyScheduleCancellationUpdate.mockReset()
mockApplyScheduleCancellationUpdate.mockResolvedValue(true)
mockApplyScheduleFailureUpdate.mockReset()
mockApplyScheduleFailureUpdate.mockResolvedValue({ updated: true, disabled: false })
mockAssertBillingAttributionSnapshot.mockReset()
mockAssertBillingAttributionSnapshot.mockImplementation((value: unknown) => {
if (!value || typeof value !== 'object') {
@@ -355,6 +384,18 @@ describe('Scheduled Workflow Execution API Route', () => {
expect(result.processedCount).toBe(0)
})
it('rotates deferred recovery carriers behind untouched work', async () => {
mockShouldExecuteInline.mockReturnValue(true)
dbChainMockFns.returning.mockReturnValueOnce([]).mockReturnValueOnce([])
await runScheduleTick('test-request-id')
expect(dbChainMockFns.orderBy).toHaveBeenCalledWith(
{ type: 'asc', field: mockAsyncJobs.updatedAt },
{ type: 'asc', field: mockAsyncJobs.id }
)
})
it('should execute multiple schedules in parallel', async () => {
dbChainMockFns.limit
.mockResolvedValueOnce([
@@ -441,6 +482,36 @@ describe('Scheduled Workflow Execution API Route', () => {
)
expect(mockUnregisterManualExecutionAborter).toHaveBeenCalledWith('schedule-execution-1')
expect(mockCompleteJob).toHaveBeenCalledWith('job-id-1', null)
const authoritativeStartCondition = dbChainMockFns.where.mock.calls
.map(([condition]) => condition)
.find(
(condition) =>
conditionContains(
condition,
(entry) => entry.type === 'eq' && entry.field === mockAsyncJobs.id
) &&
conditionContains(
condition,
(entry) =>
entry.type === 'eq' &&
entry.field === mockAsyncJobs.type &&
entry.value === 'schedule-execution'
) &&
conditionContains(
condition,
(entry) =>
entry.type === 'eq' &&
entry.field === mockAsyncJobs.status &&
entry.value === 'pending'
) &&
conditionContains(
condition,
(entry) =>
entry.type === 'eq' && entry.field === mockAsyncJobs.attempts && entry.value === 0
)
)
expect(authoritativeStartCondition).toBeDefined()
})
it('forwards database fallback cancellation into the schedule execution signal', async () => {
@@ -482,66 +553,375 @@ describe('Scheduled Workflow Execution API Route', () => {
expect(mockReleaseScheduleLock).not.toHaveBeenCalled()
})
it('recovers database fallback jobs after their admitted per-job deadline', async () => {
it.each([
{ persistedStatus: 'completed', accounting: 'success', carrierStatus: 'completed' },
{ persistedStatus: 'failed', accounting: 'failure', carrierStatus: 'completed' },
{ persistedStatus: 'cancelled', accounting: 'cancelled', carrierStatus: 'completed' },
{ persistedStatus: 'pending', accounting: 'success', carrierStatus: 'completed' },
{ persistedStatus: 'paused', accounting: 'success', carrierStatus: 'completed' },
])(
'reconciles a stale database job from a $persistedStatus execution log',
async ({ persistedStatus, accounting, carrierStatus }) => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const payload = {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
}
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{ id: 'claimed-job-id', payload, status: 'processing' },
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: persistedStatus },
])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'claimed-job-id' }])
await runScheduleTick('test-request-id')
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({
status: carrierStatus,
completedAt: expect.any(Date),
output: expect.objectContaining({ executionStatus: persistedStatus }),
})
)
if (accounting === 'failure') {
expect(mockApplyScheduleFailureUpdate).toHaveBeenCalledOnce()
} else if (accounting === 'cancelled') {
expect(mockApplyScheduleCancellationUpdate).toHaveBeenCalledOnce()
} else {
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledOnce()
}
}
)
it('preserves an already-cancelled database carrier while reconciling it', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const staleStartedAt = new Date(Date.now() - 6 * 60 * 1000)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
mockGetJob
.mockResolvedValueOnce({
id: 'job-id-1',
status: 'processing',
startedAt: staleStartedAt,
metadata: { maxDurationSeconds: 300 },
})
.mockResolvedValueOnce({
id: 'job-id-1',
status: 'pending',
orderByLimitMock.mockResolvedValueOnce([
{
id: 'cancelled-job-id',
status: 'cancelled',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
workspaceId: 'workspace-1',
billingAttribution: createBillingAttribution('workspace-1'),
now: '2025-01-01T00:00:00.000Z',
executionId: 'execution-1',
now: claimedAt.toISOString(),
scheduledFor: claimedAt.toISOString(),
},
})
orderByLimitMock
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
.mockResolvedValueOnce([
{
id: 'job-id-1',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
now: '2025-01-01T00:00:00.000Z',
},
attempts: 0,
maxAttempts: 3,
},
])
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([])
dbChainMockFns.returning
.mockReturnValueOnce([{ ...SINGLE_SCHEDULE[0], lastQueuedAt: new Date('2025-01-01') }])
.mockResolvedValueOnce([{ id: 'job-id-1' }])
},
])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'cancelled-job-id' }])
await runScheduleTick('test-request-id')
expect(mockExecuteScheduleJob).toHaveBeenCalledWith(
expect.objectContaining({ scheduleId: 'schedule-1' }),
expect.any(AbortSignal)
)
expect(mockCompleteJob).toHaveBeenCalledWith(
expect.stringMatching(/^schedule_[0-9a-f]{32}$/),
null
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ status: expect.anything() })
)
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({ metadata: expect.anything() })
)
expect(mockApplyScheduleCancellationUpdate).toHaveBeenCalledOnce()
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('preserves a cancelled carrier when its workflow log completed', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'cancelled-job-id',
status: 'cancelled',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
scheduledFor: claimedAt.toISOString(),
},
},
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: 'completed' },
])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'cancelled-job-id' }])
await runScheduleTick('test-request-id')
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ status: expect.anything() })
)
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledOnce()
expect(mockApplyScheduleCancellationUpdate).not.toHaveBeenCalled()
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('marks a terminal carrier reconciled when its occurrence already advanced', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const nextOccurrence = new Date('2025-01-02T00:00:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'completed-job-id',
status: 'completed',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
scheduledFor: claimedAt.toISOString(),
},
},
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: 'completed' },
])
queueTableRows(mockWorkflowSchedule, [
{
archivedAt: null,
lastQueuedAt: null,
nextRunAt: nextOccurrence,
status: 'active',
},
])
mockApplyScheduleSuccessUpdate.mockResolvedValueOnce(false)
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'completed-job-id' }])
await runScheduleTick('test-request-id')
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledOnce()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({ metadata: expect.anything(), updatedAt: expect.any(Date) })
)
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('restores a released claim before retrying terminal carrier accounting', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'completed-job-id',
status: 'completed',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
scheduledFor: claimedAt.toISOString(),
},
},
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: 'completed' },
])
queueTableRows(mockWorkflowSchedule, [
{
archivedAt: null,
lastQueuedAt: null,
nextRunAt: claimedAt,
status: 'active',
},
])
mockApplyScheduleSuccessUpdate.mockResolvedValueOnce(false).mockResolvedValueOnce(true)
dbChainMockFns.returning
.mockResolvedValueOnce([{ id: 'completed-job-id' }])
.mockResolvedValueOnce([{ id: 'schedule-1' }])
await runScheduleTick('test-request-id')
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledTimes(2)
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({ lastQueuedAt: claimedAt })
)
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('does not overwrite a newer non-null claim while reconciling an old carrier', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const newerClaim = new Date('2025-01-01T00:05:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'completed-job-id',
status: 'completed',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
scheduledFor: claimedAt.toISOString(),
},
},
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: 'completed' },
])
queueTableRows(mockWorkflowSchedule, [
{
archivedAt: null,
lastQueuedAt: newerClaim,
nextRunAt: claimedAt,
status: 'active',
},
])
mockApplyScheduleSuccessUpdate.mockResolvedValueOnce(false)
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'completed-job-id' }])
await runScheduleTick('test-request-id')
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledOnce()
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ lastQueuedAt: claimedAt })
)
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ metadata: expect.anything() })
)
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('rotates a deferred carrier to the back of the batch instead of starving it', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const newerClaim = new Date('2025-01-01T00:05:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'completed-job-id',
status: 'completed',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
scheduledFor: claimedAt.toISOString(),
},
},
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: 'completed' },
])
queueTableRows(mockWorkflowSchedule, [
{
archivedAt: null,
lastQueuedAt: newerClaim,
nextRunAt: claimedAt,
status: 'active',
},
])
mockApplyScheduleSuccessUpdate.mockResolvedValueOnce(false)
await runScheduleTick('test-request-id')
const carrierWrites = dbChainMockFns.set.mock.calls
.map(([values]) => values as Record<string, unknown>)
.filter((values) => values.updatedAt instanceof Date)
expect(carrierWrites).toContainEqual({ updatedAt: expect.any(Date) })
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ metadata: expect.anything() })
)
})
it('marks malformed terminal carriers irrecoverable so they leave the recovery batch', async () => {
mockShouldExecuteInline.mockReturnValue(true)
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'malformed-job-id',
status: 'completed',
payload: { workflowId: 'workflow-1' },
},
])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'malformed-job-id' }])
await runScheduleTick('test-request-id')
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ status: expect.anything() })
)
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({ metadata: expect.anything(), updatedAt: expect.any(Date) })
)
expect(mockApplyScheduleFailureUpdate).not.toHaveBeenCalled()
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('settles a malformed in-flight carrier before marking it irrecoverable', async () => {
mockShouldExecuteInline.mockReturnValue(true)
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'malformed-job-id',
status: 'processing',
payload: { workflowId: 'workflow-1' },
},
])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'malformed-job-id' }])
await runScheduleTick('test-request-id')
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({ status: 'failed', completedAt: expect.any(Date) })
)
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({ metadata: expect.anything(), updatedAt: expect.any(Date) })
)
expect(mockApplyScheduleFailureUpdate).not.toHaveBeenCalled()
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it.each([
{ name: 'a missing execution log', executionId: 'execution-1', persistedStatus: null },
{ name: 'a missing execution ID', executionId: undefined, persistedStatus: null },
{ name: 'a stale running log', executionId: 'execution-1', persistedStatus: 'running' },
{ name: 'a stale redacting log', executionId: 'execution-1', persistedStatus: 'redacting' },
])('fails a claimed database job with $name without rerunning it', async (testCase) => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'claimed-job-id',
status: 'processing',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: testCase.executionId,
now: claimedAt.toISOString(),
},
},
])
if (testCase.persistedStatus) {
queueTableRows(mockWorkflowExecutionLogs, [
{
executionId: 'execution-1',
workflowId: 'workflow-1',
status: testCase.persistedStatus,
},
])
}
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'claimed-job-id' }])
await runScheduleTick('test-request-id')
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({
status: 'pending',
startedAt: null,
error: expect.stringContaining('stale schedule execution processing lease'),
status: 'failed',
error: expect.stringContaining('Indeterminate schedule execution outcome'),
})
)
expect(mockApplyScheduleFailureUpdate).toHaveBeenCalledOnce()
})
it('resumes pending database fallback jobs without waiting for a stale schedule claim', async () => {
@@ -579,6 +959,14 @@ describe('Scheduled Workflow Execution API Route', () => {
expect.any(AbortSignal)
)
expect(mockCompleteJob).toHaveBeenCalledWith('pending-job-id', null)
expect(
dbChainMockFns.where.mock.calls.some(([condition]) =>
conditionContains(
condition,
(entry) => entry.type === 'eq' && entry.field === 'attempts' && entry.value === 0
)
)
).toBe(true)
})
it.each([
@@ -688,7 +1076,7 @@ describe('Scheduled Workflow Execution API Route', () => {
)
})
it('completes stale pending database fallback jobs whose schedule claim was already released', async () => {
it('cancels stale pending database fallback jobs whose schedule claim was already released', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
@@ -706,57 +1094,54 @@ describe('Scheduled Workflow Execution API Route', () => {
.mockResolvedValueOnce([{ lastQueuedAt: null }])
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockReturnValueOnce([]).mockReturnValueOnce([])
await runScheduleTick('test-request-id')
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
expect(mockCompleteJob).toHaveBeenCalledWith(
'stale-pending-job-id',
expect.objectContaining({
skipped: true,
})
)
})
it('fails exhausted stale database fallback jobs instead of retrying forever', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'exhausted-job-id',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
now: claimedAt.toISOString(),
},
attempts: 3,
maxAttempts: 3,
},
])
dbChainMockFns.limit
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'stale-pending-job-id' }])
await runScheduleTick('test-request-id')
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({
status: 'failed',
error: expect.stringContaining('exhausted retry attempts'),
})
)
expect(mockApplyScheduleFailureUpdate).toHaveBeenCalledWith(
expect.objectContaining({
scheduleId: 'schedule-1',
expectedLastQueuedAt: claimedAt,
executor: expect.anything(),
status: 'cancelled',
error: expect.stringContaining('claim was released'),
metadata: expect.anything(),
})
)
expect(mockCancelJob).not.toHaveBeenCalled()
expect(mockCompleteJob).not.toHaveBeenCalled()
})
it('defers schedule claims when retryable lookup infrastructure fails before enqueue', async () => {
it('reconciles pending database jobs with attempts instead of executing them', async () => {
mockShouldExecuteInline.mockReturnValue(true)
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
mockProcessingCounts(0, 0)
orderByLimitMock.mockResolvedValueOnce([
{
id: 'claimed-pending-job-id',
status: 'pending',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
},
},
])
queueTableRows(mockWorkflowExecutionLogs, [
{ executionId: 'execution-1', workflowId: 'workflow-1', status: 'completed' },
])
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'claimed-pending-job-id' }])
await runScheduleTick('test-request-id')
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({
status: 'completed',
})
)
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledOnce()
})
it('preserves the occurrence when carrier lookup is uncertain', async () => {
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const schedule = {
...SINGLE_SCHEDULE[0],
@@ -772,14 +1157,37 @@ describe('Scheduled Workflow Execution API Route', () => {
await runScheduleTick('test-request-id')
expect(mockEnqueue).not.toHaveBeenCalled()
expect(mockResolveSystemBillingAttribution).not.toHaveBeenCalled()
expect(mockReleaseScheduleLock).not.toHaveBeenCalled()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({
lastQueuedAt: null,
nextRunAt: expect.any(Date),
infraRetryCount: 1,
})
)
expect(mockApplyScheduleFailureUpdate).not.toHaveBeenCalled()
})
it('preserves the occurrence when enqueue acceptance is unknown', async () => {
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const schedule = { ...SINGLE_SCHEDULE[0], lastQueuedAt: claimedAt }
mockEnqueue.mockRejectedValueOnce(
new AsyncJobEnqueueError('response lost after enqueue', {
acceptance: 'unknown',
retryable: true,
})
)
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockReturnValueOnce([schedule]).mockReturnValueOnce([])
await runScheduleTick('test-request-id')
expect(mockEnqueue).toHaveBeenCalledOnce()
expect(mockApplyScheduleFailureUpdate).not.toHaveBeenCalled()
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ infraRetryCount: 1 })
)
})
it('marks schedules failed when non-retryable setup errors happen before enqueue', async () => {
@@ -788,7 +1196,7 @@ describe('Scheduled Workflow Execution API Route', () => {
...SINGLE_SCHEDULE[0],
lastQueuedAt: claimedAt,
}
mockGetJob.mockRejectedValueOnce(new Error('bad setup invariant'))
mockResolveSystemBillingAttribution.mockRejectedValueOnce(new Error('bad setup invariant'))
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([])
@@ -816,7 +1224,7 @@ describe('Scheduled Workflow Execution API Route', () => {
...SINGLE_SCHEDULE[0],
lastQueuedAt: claimedAt,
}
mockGetJob.mockRejectedValueOnce(new Error('bad setup invariant'))
mockResolveSystemBillingAttribution.mockRejectedValueOnce(new Error('bad setup invariant'))
mockApplyScheduleFailureUpdate.mockResolvedValueOnce({ updated: true, disabled: true })
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
@@ -836,7 +1244,7 @@ describe('Scheduled Workflow Execution API Route', () => {
...SINGLE_SCHEDULE[0],
lastQueuedAt: claimedAt,
}
mockGetJob.mockRejectedValueOnce(new Error('bad setup invariant'))
mockResolveSystemBillingAttribution.mockRejectedValueOnce(new Error('bad setup invariant'))
mockApplyScheduleFailureUpdate.mockResolvedValueOnce({ updated: true, disabled: false })
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
@@ -888,6 +1296,7 @@ describe('Scheduled Workflow Execution API Route', () => {
await runScheduleTick('test-request-id')
expect(mockEnqueue).not.toHaveBeenCalled()
expect(mockResolveSystemBillingAttribution).not.toHaveBeenCalled()
expect(mockReleaseScheduleLock).not.toHaveBeenCalled()
expect(dbChainMockFns.set).toHaveBeenCalledWith(
expect.objectContaining({
@@ -1008,9 +1417,9 @@ describe('Scheduled Workflow Execution API Route', () => {
})
it('cancels stale Trigger.dev runs instead of restoring an expired claim forever', async () => {
const originalClaim = new Date(Date.now() - 2 * 60 * 60 * 1000)
const startedAt = new Date(Date.now() - 11 * 60 * 1000)
const staleReclaim = new Date()
const originalClaim = staleReclaim
const schedule = {
...SINGLE_SCHEDULE[0],
lastQueuedAt: staleReclaim,
@@ -1023,25 +1432,26 @@ describe('Scheduled Workflow Execution API Route', () => {
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: originalClaim.toISOString(),
executionTimeoutMs: 5 * 60 * 1000,
},
})
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([{ workflowId: 'workflow-1', status: 'completed' }])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockReturnValueOnce([schedule]).mockReturnValueOnce([])
await runScheduleTick('test-request-id')
expect(mockCancelJob).toHaveBeenCalledWith('trigger-run-id')
expect(mockReleaseScheduleLock).toHaveBeenCalledWith(
'schedule-1',
'test-request-id',
expect.any(Date),
expect.stringContaining('cancelling stale queued schedule execution job'),
undefined,
{ expectedLastQueuedAt: staleReclaim }
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledWith(
expect.objectContaining({
scheduleId: 'schedule-1',
expectedLastQueuedAt: originalClaim,
})
)
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('bounds workflow schedule claims to the configured enqueue budget', async () => {
@@ -1077,7 +1487,39 @@ describe('Scheduled Workflow Execution API Route', () => {
expect(mockEnqueue).toHaveBeenCalledTimes(100)
})
it('guards route-side stale release updates with the claimed occurrence', async () => {
it('reconciles a terminal provider job with the claimed occurrence', async () => {
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const schedule = {
...SINGLE_SCHEDULE[0],
lastQueuedAt: claimedAt,
}
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([{ workflowId: 'workflow-1', status: 'completed' }])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockReturnValueOnce([schedule]).mockReturnValueOnce([])
mockGetJob.mockResolvedValueOnce({
id: 'job-id-1',
status: 'completed',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
},
})
await runScheduleTick('test-request-id')
expect(mockApplyScheduleSuccessUpdate).toHaveBeenCalledWith(
expect.objectContaining({
scheduleId: 'schedule-1',
expectedLastQueuedAt: claimedAt,
})
)
expect(mockExecuteScheduleJob).not.toHaveBeenCalled()
})
it('uses a terminal cancelled carrier when no execution log exists', async () => {
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const schedule = {
...SINGLE_SCHEDULE[0],
@@ -1086,17 +1528,61 @@ describe('Scheduled Workflow Execution API Route', () => {
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockReturnValueOnce([schedule]).mockReturnValueOnce([])
mockGetJob.mockResolvedValueOnce({ id: 'job-id-1', status: 'completed' })
mockGetJob.mockResolvedValueOnce({
id: 'job-id-1',
status: 'cancelled',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
},
})
await runScheduleTick('test-request-id')
expect(mockReleaseScheduleLock).toHaveBeenCalledWith(
'schedule-1',
'test-request-id',
expect.any(Date),
expect.stringContaining('finished job'),
null,
{ expectedLastQueuedAt: claimedAt }
expect(mockApplyScheduleCancellationUpdate).toHaveBeenCalledWith(
expect.objectContaining({
scheduleId: 'schedule-1',
expectedLastQueuedAt: claimedAt,
})
)
expect(mockApplyScheduleFailureUpdate).not.toHaveBeenCalled()
expect(mockResolveSystemBillingAttribution).not.toHaveBeenCalled()
})
it('does not change cadence when reconciliation fails after observing a carrier', async () => {
const claimedAt = new Date('2025-01-01T00:00:00.000Z')
const schedule = {
...SINGLE_SCHEDULE[0],
lastQueuedAt: claimedAt,
}
dbChainMockFns.limit
.mockResolvedValueOnce(SINGLE_CLAIMED_SCHEDULE_ROWS)
.mockResolvedValueOnce([{ workflowId: 'workflow-1', status: 'completed' }])
.mockResolvedValueOnce([])
dbChainMockFns.returning.mockReturnValueOnce([schedule]).mockReturnValueOnce([])
mockGetJob.mockResolvedValueOnce({
id: 'job-id-1',
status: 'completed',
payload: {
scheduleId: 'schedule-1',
workflowId: 'workflow-1',
executionId: 'execution-1',
now: claimedAt.toISOString(),
},
})
mockApplyScheduleSuccessUpdate.mockRejectedValueOnce(new Error('database unavailable'))
await runScheduleTick('test-request-id')
expect(mockEnqueue).not.toHaveBeenCalled()
expect(mockResolveSystemBillingAttribution).not.toHaveBeenCalled()
expect(mockApplyScheduleFailureUpdate).not.toHaveBeenCalled()
expect(dbChainMockFns.set).not.toHaveBeenCalledWith(
expect.objectContaining({ infraRetryCount: 1 })
)
})
apps/sim/app/api/schedules/execute/route.ts
+644 -183
View File
File diff suppressed because it is too large Load Diff
apps/sim/background/schedule-execution.test.ts
+17
View File
@@ -24,6 +24,7 @@ vi.mock('@sim/db', () => ({ ...databaseMock, ...schemaMock }))
import {
applyScheduleFailureUpdate,
buildScheduleCancellationUpdate,
buildScheduleSuccessUpdate,
classifyScheduleExecutionResult,
releaseScheduleLock,
} from '@/background/schedule-execution'
@@ -141,4 +142,20 @@ describe('schedule cancellation accounting', () => {
expect(buildScheduleCancellationUpdate(now, nextRunAt)).not.toHaveProperty('failedCount')
expect(buildScheduleCancellationUpdate(now, nextRunAt)).not.toHaveProperty('lastFailedAt')
})
it('advances cadence, releases the claim and clears failure accounting on success', () => {
const now = new Date('2026-08-03T12:00:00.000Z')
const nextRunAt = new Date('2026-08-03T13:00:00.000Z')
expect(buildScheduleSuccessUpdate(now, nextRunAt)).toEqual({
lastRanAt: now,
updatedAt: now,
nextRunAt,
failedCount: 0,
lastQueuedAt: null,
infraRetryCount: 0,
})
expect(buildScheduleSuccessUpdate(now, nextRunAt)).not.toHaveProperty('lastFailedAt')
expect(buildScheduleSuccessUpdate(now, nextRunAt)).not.toHaveProperty('status')
})
})
apps/sim/background/schedule-execution.ts
+64 -8
View File
@@ -158,6 +158,23 @@ export function classifyScheduleExecutionResult(
}
/** Advances cadence after user cancellation without mutating the failure counter. */
/**
* Successful-run accounting: stamps `lastRanAt`, clears the claim, advances to
* `nextRunAt`, and resets the consecutive-failure and infra-retry counters. The
* sibling of {@link buildScheduleCancellationUpdate} and
* {@link buildScheduleFailureUpdate}.
*/
export function buildScheduleSuccessUpdate(now: Date, nextRunAt: Date): WorkflowScheduleUpdate {
return {
lastRanAt: now,
updatedAt: now,
nextRunAt,
failedCount: 0,
lastQueuedAt: null,
...resetScheduleInfraRetryCount(),
}
}
export function buildScheduleCancellationUpdate(
now: Date,
nextRunAt: Date
@@ -269,6 +286,52 @@ export async function releaseScheduleLock(
return outcome.updated
}
/** Applies successful-run accounting only while the caller still owns the claim. */
export async function applyScheduleSuccessUpdate(params: {
scheduleId: string
now: Date
nextRunAt: Date
expectedLastQueuedAt: Date | null
requestId: string
context: string
executor?: DbOrTx
}): Promise<boolean> {
const { scheduleId, now, nextRunAt, expectedLastQueuedAt, requestId, context, executor } = params
const outcome = await applyScheduleUpdate(
scheduleId,
buildScheduleSuccessUpdate(now, nextRunAt),
requestId,
context,
{ expectedLastQueuedAt, executor }
)
return outcome.updated
}
/** Applies cancelled-run accounting only while the caller still owns the claim. */
export async function applyScheduleCancellationUpdate(params: {
scheduleId: string
now: Date
nextRunAt: Date
expectedLastQueuedAt: Date | null
requestId: string
context: string
executor?: DbOrTx
}): Promise<boolean> {
const { scheduleId, now, nextRunAt, expectedLastQueuedAt, requestId, context, executor } = params
const outcome = await applyScheduleUpdate(
scheduleId,
buildScheduleCancellationUpdate(now, nextRunAt),
requestId,
context,
{ expectedLastQueuedAt, executor }
)
return outcome.updated
}
/**
* Applies {@link buildScheduleFailureUpdate} through the same guarded write the
* trigger.dev path uses, and reports whether the row just transitioned to
@@ -1090,14 +1153,7 @@ export async function executeScheduleJob(
const nextRunAt = calculateNextRunTime(payload, executionResult.blocks)
await updateClaimedSchedule(
{
lastRanAt: now,
updatedAt: now,
nextRunAt,
failedCount: 0,
lastQueuedAt: null,
...resetScheduleInfraRetryCount(),
},
buildScheduleSuccessUpdate(now, nextRunAt),
`Error updating schedule ${payload.scheduleId} after success`
)
return
apps/sim/lib/core/async-jobs/index.ts
+1
View File
@@ -16,4 +16,5 @@ export {
JOB_STATUS,
MAX_JOB_DURATION_SECONDS,
MIN_JOB_DURATION_SECONDS,
TERMINAL_JOB_STATUSES,
} from './types'
apps/sim/lib/logs/types.ts
+15
View File
@@ -230,6 +230,21 @@ export const PERSISTED_WORKFLOW_EXECUTION_STATUSES = [
export type PersistedWorkflowExecutionStatus =
(typeof PERSISTED_WORKFLOW_EXECUTION_STATUSES)[number]
/**
* In-flight statuses a crashed worker can strand, which the stale-execution
* cron terminalizes. `pending` and `paused` are excluded: both are written as
* the resting state of a run waiting on a resume, so sweeping them would fail
* live work. Each status here needs its own partial index on
* `workflow_execution_logs` — the sweep runs one status per pass so it can use
* them.
*/
export const STALE_SWEEPABLE_EXECUTION_STATUSES = [
'running',
'redacting',
] as const satisfies readonly PersistedWorkflowExecutionStatus[]
export type StaleSweepableExecutionStatus = (typeof STALE_SWEEPABLE_EXECUTION_STATUSES)[number]
export interface CompletedWorkflowExecutionLog extends WorkflowExecutionLog {
persistedStatus: PersistedWorkflowExecutionStatus
}
apps/sim/lib/workflows/schedules/carrier-metadata.ts
+76
View File
@@ -0,0 +1,76 @@
import { type Column, type SQL, sql } from 'drizzle-orm'
/**
* Marks a schedule carrier job as reconciled against its persisted execution
* log. Schedule recovery owns these rows, so retention only deletes a carrier
* once this key is present.
*/
export const SCHEDULE_CARRIER_RECONCILED_METADATA_KEY = 'scheduleReconciled'
/**
* Marks a reconciled carrier whose payload could not be interpreted, so its
* schedule accounting can never be replayed. Retention holds these tombstones
* past normal job retention, bounded by
* {@link SCHEDULE_CARRIER_IRRECOVERABLE_RETENTION_HOURS}.
*/
export const SCHEDULE_CARRIER_IRRECOVERABLE_METADATA_KEY = 'scheduleRecoveryIrrecoverable'
/**
* How long an irrecoverable carrier tombstone is kept past normal job
* retention. These rows exist only for operator forensics — a carrier whose
* payload could not be read, so its schedule accounting can never be replayed
* — and they are the one class of carrier that can never clear on its own, so
* the window has to be bounded rather than infinite.
*/
export const SCHEDULE_CARRIER_IRRECOVERABLE_RETENTION_HOURS = 30 * 24
/**
* `COALESCE(metadata ->> '<key>', 'false')`, with the key spelled as a SQL
* *literal* rather than a bind parameter.
*
* This is load-bearing for the partial index
* `async_jobs_schedule_unreconciled_terminal_idx`, whose predicate is written
* with the same literal: Postgres cannot prove that a parameterised predicate
* implies a literal one, so a bound key silently drops the recovery scan to a
* sequential scan of `async_jobs`. The key is a fixed identifier defined above,
* so there is no injection surface.
*
* The metadata column is a parameter rather than an import so this module never
* pins a second copy of the schema.
*/
function carrierMetadataFlag(metadata: Column, key: string): SQL {
return sql`COALESCE(${metadata} ->> ${sql.raw(`'${key}'`)}, 'false')`
}
/** Carriers whose schedule accounting has not been replayed yet. */
export function carrierNotReconciledSql(metadata: Column): SQL {
return sql`${carrierMetadataFlag(metadata, SCHEDULE_CARRIER_RECONCILED_METADATA_KEY)} <> 'true'`
}
/** Carriers whose schedule accounting was replayed, successfully or not. */
export function carrierReconciledSql(metadata: Column): SQL {
return sql`${carrierMetadataFlag(metadata, SCHEDULE_CARRIER_RECONCILED_METADATA_KEY)} = 'true'`
}
/** Reconciled carriers whose payload could never be interpreted. */
export function carrierNotIrrecoverableSql(metadata: Column): SQL {
return sql`${carrierMetadataFlag(metadata, SCHEDULE_CARRIER_IRRECOVERABLE_METADATA_KEY)} <> 'true'`
}
/**
* Builds the jsonb merge that stamps reconciliation bookkeeping onto a carrier
* while preserving every other metadata key it already carries.
*/
export function buildCarrierReconciledMetadata(
metadata: Column,
options: { irrecoverable?: boolean } = {}
): SQL {
const patch: Record<string, boolean> = {
[SCHEDULE_CARRIER_RECONCILED_METADATA_KEY]: true,
}
if (options.irrecoverable) {
patch[SCHEDULE_CARRIER_IRRECOVERABLE_METADATA_KEY] = true
}
return sql`COALESCE(${metadata}, '{}'::jsonb) || ${JSON.stringify(patch)}::jsonb`
}
packages/db/migrations/0295_chilly_franklin_storm.sql
+35
View File
@@ -0,0 +1,35 @@
-- Schedule recovery scans async_jobs on every tick, and the stale-execution
-- cron sweeps workflow_execution_logs. Both tables take live writes, so the
-- supporting indexes are built without a table-wide write lock. The migration
-- runner starts a transaction per pending batch, so end it before CONCURRENTLY.
-- Everything below is replayable even when a failed concurrent build left an
-- INVALID same-named index behind.
COMMIT;--> statement-breakpoint
-- `lock_timeout = 0` for the concurrent builds, per packages/db/scripts/migrate.ts.
-- CREATE INDEX CONCURRENTLY waits on every concurrent write in the database, not
-- just this table, so the session's 5s DDL timeout would cancel it (55P03) and
-- strand an INVALID index that the IF NOT EXISTS below would skip forever.
SET lock_timeout = 0;--> statement-breakpoint
-- Carriers whose schedule accounting has not been replayed yet. The predicate
-- matches the recovery query's third branch verbatim -- both spell the status
-- list and the metadata key as literals, because Postgres cannot prove a
-- parameterised predicate implies a literal one and would seq-scan instead.
-- Rows leave the index the moment the reconciled marker is stamped, so it stays
-- small.
-- migration-safe: replay cleanup for the index introduced by this same unjournaled migration; CONCURRENTLY preserves async_jobs writes
DROP INDEX CONCURRENTLY IF EXISTS "async_jobs_schedule_unreconciled_terminal_idx";--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "async_jobs_schedule_unreconciled_terminal_idx" ON "async_jobs" USING btree ("updated_at","id") WHERE "async_jobs"."type" = 'schedule-execution' AND "async_jobs"."status" IN ('completed', 'failed', 'cancelled') AND COALESCE("async_jobs"."metadata" ->> 'scheduleReconciled', 'false') <> 'true';--> statement-breakpoint
-- Mirrors the existing `status = 'running'` pair so the stale-execution sweep
-- keeps an index for its second pass instead of falling back to a seq scan.
-- migration-safe: replay cleanup for the index introduced by this same unjournaled migration; CONCURRENTLY preserves workflow_execution_logs writes
DROP INDEX CONCURRENTLY IF EXISTS "workflow_execution_logs_redacting_started_at_idx";--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "workflow_execution_logs_redacting_started_at_idx" ON "workflow_execution_logs" USING btree ("started_at") WHERE status = 'redacting';--> statement-breakpoint
-- migration-safe: replay cleanup for the index introduced by this same unjournaled migration; CONCURRENTLY preserves workflow_execution_logs writes
DROP INDEX CONCURRENTLY IF EXISTS "workflow_execution_logs_redacting_deadline_idx";--> statement-breakpoint
CREATE INDEX CONCURRENTLY IF NOT EXISTS "workflow_execution_logs_redacting_deadline_idx" ON "workflow_execution_logs" USING btree ("execution_deadline_at") WHERE "workflow_execution_logs"."status" = 'redacting' AND "workflow_execution_logs"."execution_deadline_at" IS NOT NULL;--> statement-breakpoint
SET lock_timeout = '5s';
packages/db/migrations/meta/0295_snapshot.json
+19963
View File
File diff suppressed because it is too large Load Diff
packages/db/migrations/meta/_journal.json
+7
View File
@@ -2059,6 +2059,13 @@
"when": 1787178437209,
"tag": "0294_workable_jigsaw",
"breakpoints": true
},
{
"idx": 295,
"version": "7",
"when": 1787247269246,
"tag": "0295_chilly_franklin_storm",
"breakpoints": true
}
]
}
packages/db/schema.ts
+11
View File
@@ -513,6 +513,12 @@ export const workflowExecutionLogs = pgTable(
runningExecutionDeadlineIdx: index('workflow_execution_logs_running_deadline_idx')
.on(table.executionDeadlineAt)
.where(sql`${table.status} = 'running' AND ${table.executionDeadlineAt} IS NOT NULL`),
redactingStartedAtIdx: index('workflow_execution_logs_redacting_started_at_idx')
.on(table.startedAt)
.where(sql`status = 'redacting'`),
redactingExecutionDeadlineIdx: index('workflow_execution_logs_redacting_deadline_idx')
.on(table.executionDeadlineAt)
.where(sql`${table.status} = 'redacting' AND ${table.executionDeadlineAt} IS NOT NULL`),
completedEndedAtIdx: index('workflow_execution_logs_completed_ended_at_idx')
.on(table.endedAt, table.workspaceId, table.executionId)
.where(
@@ -4262,6 +4268,11 @@ export const asyncJobs = pgTable(
scheduleProcessingStartedAtIdx: index('async_jobs_schedule_processing_started_at_idx')
.on(table.startedAt, table.id)
.where(sql`${table.type} = 'schedule-execution' AND ${table.status} = 'processing'`),
scheduleUnreconciledTerminalIdx: index('async_jobs_schedule_unreconciled_terminal_idx')
.on(table.updatedAt, table.id)
.where(
sql`${table.type} = 'schedule-execution' AND ${table.status} IN ('completed', 'failed', 'cancelled') AND COALESCE(${table.metadata} ->> 'scheduleReconciled', 'false') <> 'true'`
),
})
)
packages/testing/src/mocks/schema.mock.ts
+1
View File
@@ -187,6 +187,7 @@ export const schemaMock = {
status: 'status',
trigger: 'trigger',
startedAt: 'startedAt',
executionDeadlineAt: 'executionDeadlineAt',
endedAt: 'endedAt',
totalDurationMs: 'totalDurationMs',
executionData: 'executionData',