mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-01 14:59:19 +08:00
feat(enterprise): add data drains for continuous export to S3 / webhook (#4440)
* feat(enterprise): add data drains for continuous export to S3 / webhook * chore(data-drains): regenerate migration on top of staging + bump route baseline Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(data-drains): clarify retention pairing is user-coupled, not enforced Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(data-drains): preserve explicit forcePathStyle=false + reserve x-sim-signature Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test(data-drains): drift guard ensures every webhook header is reserved Asserts that any header buildHeaders writes is rejected when reused as a custom signatureHeader. Adding a new metadata header without mirroring it into RESERVED_SIGNATURE_HEADER_NAMES now fails CI. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Waleed
@@ -0,0 +1,185 @@
|
||||
---
|
||||
title: Data Drains
|
||||
description: Continuously export workflow logs, audit logs, and Mothership data to your own S3 bucket or HTTPS endpoint on a schedule
|
||||
---
|
||||
|
||||
import { FAQ } from '@/components/ui/faq'
|
||||
|
||||
Data Drains let organization owners and admins on Enterprise plans continuously export Sim data to a destination they control — a customer-owned S3 bucket or an HTTPS webhook. A drain runs on a schedule, picks up only new rows since its last successful run, and writes them as NDJSON to the destination. Viewing drain configuration and run history is restricted to owners and admins as well, since destinations expose internal bucket names and webhook URLs.
|
||||
|
||||
Drains are independent of [Data Retention](/enterprise/data-retention) but designed to compose with it — see [Pairing with Data Retention](#pairing-with-data-retention) below.
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
Go to **Settings → Enterprise → Data Drains** in your workspace, then click **New drain**.
|
||||
|
||||
Each drain has four pieces:
|
||||
|
||||
1. A **source** — the category of data to export
|
||||
2. A **destination** — where the data goes
|
||||
3. A **schedule** — how often it runs
|
||||
4. A **name** — unique within your organization
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
A drain exports exactly one source. To export multiple sources, create multiple drains.
|
||||
|
||||
| Source | Description |
|
||||
|---|---|
|
||||
| **Workflow logs** | Workflow execution records (one row per execution, only after the run reaches a terminal state). |
|
||||
| **Job logs** | Background job records (deployed APIs, schedules, webhooks). Only terminal-state rows are exported. |
|
||||
| **Audit logs** | Organization- and workspace-scoped audit events — logins, permission changes, resource creation/deletion, drain configuration changes. |
|
||||
| **Copilot chats** | Mothership chat history. |
|
||||
| **Copilot runs** | Mothership run records (terminal state only). |
|
||||
|
||||
Each row is delivered as a single line of NDJSON. The shape of each row is part of the public schema and stable across versions; every row carries an `id` field that downstream consumers can use to dedupe.
|
||||
|
||||
Drains export each row exactly once based on its creation cursor. Mutable fields on **Copilot chats** (messages, title, `lastSeenAt`) are a point-in-time snapshot and won't be re-emitted if the chat is later updated. Treat the export as append-only and reconstitute current state from your own system of record if you need it.
|
||||
|
||||
---
|
||||
|
||||
## Destinations
|
||||
|
||||
### Amazon S3 (or any S3-compatible store)
|
||||
|
||||
Writes one NDJSON object per delivered chunk to your bucket.
|
||||
|
||||
- **Bucket** — the bucket name. Must already exist; Sim does not create buckets.
|
||||
- **Region** — AWS region (e.g. `us-east-1`).
|
||||
- **Prefix** *(optional)* — folder path inside the bucket. Trailing slash optional.
|
||||
- **Access key ID / Secret access key** — IAM credentials with `s3:PutObject` on the bucket. The "Test connection" button performs a real write probe to verify, then deletes it.
|
||||
- **Endpoint** *(optional)* — for non-AWS stores like MinIO, Cloudflare R2, or GCS S3-interop. Leave blank for AWS S3.
|
||||
- **Force path-style** *(optional)* — required for MinIO/Ceph, must be off for AWS S3 and R2.
|
||||
|
||||
Object keys are deterministic:
|
||||
|
||||
```
|
||||
{prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson
|
||||
```
|
||||
|
||||
Objects are written with `AES256` server-side encryption.
|
||||
|
||||
### HTTPS Webhook
|
||||
|
||||
POSTs each chunk as NDJSON to your endpoint.
|
||||
|
||||
- **URL** — must be HTTPS. Sim resolves the hostname and refuses to deliver to private, loopback, or cloud-metadata IPs. The resolved IP is pinned for the duration of a run to prevent DNS rebinding.
|
||||
- **Signing secret** — shared secret used for HMAC-SHA256 signing.
|
||||
- **Bearer token** *(optional)* — sent as `Authorization: Bearer <token>`.
|
||||
- **Signature header name** *(optional)* — defaults to `X-Sim-Signature`.
|
||||
|
||||
Each request includes:
|
||||
|
||||
```
|
||||
Content-Type: application/x-ndjson
|
||||
User-Agent: Sim-DataDrain/1.0
|
||||
X-Sim-Timestamp: <unix-seconds>
|
||||
X-Sim-Signature-Version: v1
|
||||
X-Sim-Signature: t=<unix-seconds>,v1=<hex(hmac-sha256)>
|
||||
X-Sim-Drain-Id: <drain id>
|
||||
X-Sim-Run-Id: <run id>
|
||||
X-Sim-Source: <source name>
|
||||
X-Sim-Sequence: <chunk index>
|
||||
X-Sim-Row-Count: <rows in this chunk>
|
||||
Idempotency-Key: <runId>-<sequence>
|
||||
```
|
||||
|
||||
The signature is computed as `HMAC-SHA256(secret, "${timestamp}.${body}")` and serialized as `t=<timestamp>,v1=<hex>`. Verify by recomputing over the same string and rejecting timestamps older than ~5 minutes — this defends against captured-request replay attacks.
|
||||
|
||||
Failed deliveries retry up to 3 times with exponential backoff (500ms, 1s, 2s with ±20% jitter), respecting `Retry-After` on 429/503. Non-retryable 4xx responses fail the run immediately.
|
||||
|
||||
---
|
||||
|
||||
## Schedule
|
||||
|
||||
| Cadence | Drain runs |
|
||||
|---|---|
|
||||
| **Hourly** | Once per hour. |
|
||||
| **Daily** | Once per day. |
|
||||
|
||||
You can also disable a drain with the **Enabled** toggle (it stops running but is preserved), or trigger an out-of-schedule run with **Run now** on any drain row.
|
||||
|
||||
---
|
||||
|
||||
## Delivery semantics
|
||||
|
||||
Drains use an **opaque cursor** that advances only on full success. If a delivery fails partway through a run, the cursor is unchanged and the next run replays from the last successful position.
|
||||
|
||||
This is **at-least-once delivery**. Combined with the `id` field on every row and the `Idempotency-Key` header on every webhook chunk, downstream systems can dedupe deterministically.
|
||||
|
||||
The **last 10 runs** for each drain are visible by expanding its row in the settings page, with status, row count, bytes written, destination locator (`s3://...` or webhook URL), and the error message if it failed.
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
- Destination credentials are encrypted at rest using the same key-rotation–aware encryption that protects OAuth tokens.
|
||||
- Credentials are **never** returned by the Sim API after creation. Updates accept new credentials; omitting them leaves the existing encrypted blob in place.
|
||||
- Webhook URLs are SSRF-validated: HTTPS-only, no private/loopback/metadata IPs, with the resolved IP pinned to defeat DNS rebinding.
|
||||
- Every create, update, delete, manual run, and test-connection call is recorded in the [Audit Log](/enterprise/audit-logs).
|
||||
|
||||
---
|
||||
|
||||
## Pairing with Data Retention
|
||||
|
||||
Drains and [Data Retention](/enterprise/data-retention) are independent modules. Sim does **not** gate retention on drain progress — if a drain is failing, retention will still purge data on its own schedule. This matches the model used by Datadog Archives and AWS CloudWatch + S3 Export: keep the two configurations orthogonal and let the customer pair them deliberately.
|
||||
|
||||
To safely use both together, set the drain cadence shorter than the retention period for the same data category:
|
||||
|
||||
| Drain source | Pairs with retention setting |
|
||||
|---|---|
|
||||
| Workflow logs, Job logs | **Log retention** |
|
||||
| Copilot chats, Copilot runs | **Task cleanup** |
|
||||
| Audit logs | *(no retention setting today — audit logs are kept indefinitely)* |
|
||||
|
||||
For example, with **Log retention** set to 30 days, set the workflow-logs drain to **Hourly** or **Daily** so every row is exported well before retention purges it from Sim. Monitor recent drain runs in the settings page; if a drain has been failing for longer than your retention window, you may lose rows that retention purges before they are exported.
|
||||
|
||||
After data lands in your bucket or webhook system, archive lifecycle (transitions to Glacier, expiration, GDPR right-to-erasure propagation) is governed by your own infrastructure — Sim has no further visibility into that data once delivery succeeds.
|
||||
|
||||
---
|
||||
|
||||
<FAQ items={[
|
||||
{
|
||||
question: "Who can configure data drains?",
|
||||
answer: "Only organization owners and admins can view, create, edit, run, or delete drains. On Sim Cloud, the organization must be on an Enterprise plan."
|
||||
},
|
||||
{
|
||||
question: "Will drained data be duplicated if a run fails?",
|
||||
answer: "The drain cursor only advances on overall success, so a failure replays the same chunks on the next run. Every row has a stable `id` field and every webhook chunk has an `Idempotency-Key` header so receivers can dedupe."
|
||||
},
|
||||
{
|
||||
question: "Can I export multiple sources to the same destination?",
|
||||
answer: "Yes — create one drain per source, all pointing at the same bucket or endpoint. S3 destinations namespace by source automatically; webhook receivers can branch on the `X-Sim-Source` header."
|
||||
},
|
||||
{
|
||||
question: "Does deleting a drain delete the data already exported?",
|
||||
answer: "No. Deletion only removes the drain's configuration and its run history from Sim. Data already written to your bucket or sent to your webhook is yours and is unaffected."
|
||||
},
|
||||
{
|
||||
question: "What happens if my credentials stop working mid-run?",
|
||||
answer: "The run fails, the drain cursor does not advance, and the failed run is recorded with the error. Once you fix the credentials with an Update or by re-creating the drain, the next run replays from where the last successful run left off."
|
||||
},
|
||||
{
|
||||
question: "What format is the data in?",
|
||||
answer: "NDJSON — newline-delimited JSON, one row per line. Each chunk is a single S3 object or a single POST body."
|
||||
}
|
||||
]} />
|
||||
|
||||
---
|
||||
|
||||
## Self-hosted setup
|
||||
|
||||
### Environment variables
|
||||
|
||||
```bash
|
||||
DATA_DRAINS_ENABLED=true
|
||||
NEXT_PUBLIC_DATA_DRAINS_ENABLED=true
|
||||
```
|
||||
|
||||
`NEXT_PUBLIC_DATA_DRAINS_ENABLED` shows the **Settings → Enterprise → Data Drains** page in the UI. `DATA_DRAINS_ENABLED` gates the server-side mutating endpoints and the cron dispatcher — when unset on a self-hosted deployment, drain create/update/delete/run requests return `404` and the dispatcher is a no-op. Both should be set to `true` together.
|
||||
|
||||
Data Drains otherwise rely on the standard Trigger.dev background job infrastructure used elsewhere in Sim — no additional setup is required. The cron dispatcher runs hourly and fans out due drains as background jobs.
|
||||
@@ -59,6 +59,12 @@ Configure how long execution logs, soft-deleted resources, and Mothership data a
|
||||
|
||||
---
|
||||
|
||||
## Data Drains
|
||||
|
||||
Continuously export workflow logs, audit logs, and Mothership data to a customer-owned S3 bucket or HTTPS webhook on a schedule. See the [data drains guide](/docs/enterprise/data-drains).
|
||||
|
||||
---
|
||||
|
||||
<FAQ items={[
|
||||
{ question: "Who can manage Enterprise features?", answer: "Workspace admins on an Enterprise-entitled workspace. Access Control, SSO, whitelabeling, audit logs, and data retention are all configured per workspace under Settings → Enterprise." },
|
||||
{ question: "Which SSO providers are supported?", answer: "Sim supports SAML 2.0 and OIDC, which works with virtually any enterprise identity provider including Okta, Azure AD (Entra ID), Google Workspace, ADFS, and OneLogin." },
|
||||
@@ -79,6 +85,7 @@ Self-hosted deployments enable enterprise features via environment variables ins
|
||||
| `WHITELABELING_ENABLED`, `NEXT_PUBLIC_WHITELABELING_ENABLED` | Custom branding |
|
||||
| `AUDIT_LOGS_ENABLED`, `NEXT_PUBLIC_AUDIT_LOGS_ENABLED` | Audit logging |
|
||||
| `NEXT_PUBLIC_DATA_RETENTION_ENABLED` | Data retention configuration |
|
||||
| `DATA_DRAINS_ENABLED`, `NEXT_PUBLIC_DATA_DRAINS_ENABLED` | Data drains |
|
||||
| `CREDENTIAL_SETS_ENABLED`, `NEXT_PUBLIC_CREDENTIAL_SETS_ENABLED` | Polling groups for email triggers |
|
||||
| `INBOX_ENABLED`, `NEXT_PUBLIC_INBOX_ENABLED` | Sim Mailer inbox |
|
||||
| `DISABLE_INVITATIONS`, `NEXT_PUBLIC_DISABLE_INVITATIONS` | Disable invitations; manage membership via Admin API |
|
||||
|
||||
@@ -1,5 +1,13 @@
|
||||
{
|
||||
"title": "Enterprise",
|
||||
"pages": ["index", "sso", "access-control", "whitelabeling", "audit-logs", "data-retention"],
|
||||
"pages": [
|
||||
"index",
|
||||
"sso",
|
||||
"access-control",
|
||||
"whitelabeling",
|
||||
"audit-logs",
|
||||
"data-retention",
|
||||
"data-drains"
|
||||
],
|
||||
"defaultOpen": false
|
||||
}
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { verifyCronAuth } from '@/lib/auth/internal'
|
||||
import { isBillingEnabled, isDataDrainsEnabled } from '@/lib/core/config/feature-flags'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { dispatchDueDrains } from '@/lib/data-drains/dispatcher'
|
||||
|
||||
const logger = createLogger('CronRunDataDrains')
|
||||
|
||||
export const GET = withRouteHandler(async (request: NextRequest) => {
|
||||
const authError = verifyCronAuth(request, 'Data drain dispatcher')
|
||||
if (authError) return authError
|
||||
|
||||
// Self-hosted opt-in: skip dispatch entirely when the deployment hasn't
|
||||
// enabled drains. Sim Cloud (billing enabled) gates per-org by enterprise
|
||||
// plan inside the dispatcher's join.
|
||||
if (!isBillingEnabled && !isDataDrainsEnabled) {
|
||||
return NextResponse.json({ success: true, dispatched: 0, skipped: 'disabled' })
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await dispatchDueDrains()
|
||||
logger.info('Data drain dispatcher run complete', result)
|
||||
return NextResponse.json({ success: true, ...result })
|
||||
} catch (error) {
|
||||
logger.error('Data drain dispatcher run failed', { error: toError(error).message })
|
||||
return NextResponse.json({ error: 'Internal server error' }, { status: 500 })
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,178 @@
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit'
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrains } from '@sim/db/schema'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { and, eq, ne } from 'drizzle-orm'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import {
|
||||
deleteDataDrainContract,
|
||||
getDataDrainContract,
|
||||
updateDataDrainContract,
|
||||
} from '@/lib/api/contracts/data-drains'
|
||||
import { parseRequest, validationErrorResponse } from '@/lib/api/server'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { authorizeDrainAccess, loadDrain } from '@/lib/data-drains/access'
|
||||
import { getDestination } from '@/lib/data-drains/destinations/registry'
|
||||
import { encryptCredentials } from '@/lib/data-drains/encryption'
|
||||
import { serializeDrain } from '@/lib/data-drains/serializers'
|
||||
|
||||
const logger = createLogger('DataDrainAPI')
|
||||
|
||||
type RouteContext = { params: Promise<{ id: string; drainId: string }> }
|
||||
|
||||
export const GET = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId, drainId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: false })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(getDataDrainContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const drain = await loadDrain(organizationId, drainId)
|
||||
if (!drain) {
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
return NextResponse.json({ drain: serializeDrain(drain) })
|
||||
})
|
||||
|
||||
export const PUT = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId, drainId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: true })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(updateDataDrainContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const body = parsed.data.body
|
||||
|
||||
const drain = await loadDrain(organizationId, drainId)
|
||||
if (!drain) {
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
if (body.name !== undefined && body.name !== drain.name) {
|
||||
const [conflict] = await db
|
||||
.select({ id: dataDrains.id })
|
||||
.from(dataDrains)
|
||||
.where(
|
||||
and(
|
||||
eq(dataDrains.organizationId, organizationId),
|
||||
eq(dataDrains.name, body.name),
|
||||
ne(dataDrains.id, drainId)
|
||||
)
|
||||
)
|
||||
.limit(1)
|
||||
if (conflict) {
|
||||
return NextResponse.json(
|
||||
{ error: 'A data drain with this name already exists in this organization' },
|
||||
{ status: 409 }
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
if (body.source !== undefined && body.source !== drain.source) {
|
||||
return NextResponse.json({ error: 'source cannot be changed after creation' }, { status: 400 })
|
||||
}
|
||||
|
||||
const updates: Partial<typeof dataDrains.$inferInsert> = { updatedAt: new Date() }
|
||||
if (body.name !== undefined) updates.name = body.name
|
||||
if (body.scheduleCadence !== undefined) updates.scheduleCadence = body.scheduleCadence
|
||||
if (body.enabled !== undefined) updates.enabled = body.enabled
|
||||
|
||||
if (body.destinationType !== undefined && body.destinationType !== drain.destinationType) {
|
||||
return NextResponse.json(
|
||||
{ error: 'destinationType cannot be changed after creation' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
if (body.destinationConfig !== undefined || body.destinationCredentials !== undefined) {
|
||||
const destination = getDestination(drain.destinationType)
|
||||
if (body.destinationConfig !== undefined) {
|
||||
const configResult = destination.configSchema.safeParse(body.destinationConfig)
|
||||
if (!configResult.success) return validationErrorResponse(configResult.error)
|
||||
updates.destinationConfig = configResult.data as Record<string, unknown>
|
||||
}
|
||||
if (body.destinationCredentials !== undefined) {
|
||||
const credentialsResult = destination.credentialsSchema.safeParse(body.destinationCredentials)
|
||||
if (!credentialsResult.success) return validationErrorResponse(credentialsResult.error)
|
||||
updates.destinationCredentials = await encryptCredentials(credentialsResult.data)
|
||||
}
|
||||
}
|
||||
|
||||
const [updated] = await db
|
||||
.update(dataDrains)
|
||||
.set(updates)
|
||||
.where(eq(dataDrains.id, drainId))
|
||||
.returning()
|
||||
|
||||
if (!updated) {
|
||||
// Concurrent DELETE landed between loadDrain() and this UPDATE.
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
logger.info('Data drain updated', { drainId, organizationId })
|
||||
|
||||
recordAudit({
|
||||
workspaceId: null,
|
||||
actorId: access.session.user.id,
|
||||
action: AuditAction.DATA_DRAIN_UPDATED,
|
||||
resourceType: AuditResourceType.DATA_DRAIN,
|
||||
resourceId: drainId,
|
||||
actorName: access.session.user.name ?? undefined,
|
||||
actorEmail: access.session.user.email ?? undefined,
|
||||
resourceName: updated.name,
|
||||
description: `Updated data drain '${updated.name}'`,
|
||||
metadata: {
|
||||
organizationId,
|
||||
changes: {
|
||||
name: body.name,
|
||||
source: body.source,
|
||||
scheduleCadence: body.scheduleCadence,
|
||||
enabled: body.enabled,
|
||||
destinationConfigChanged: body.destinationConfig !== undefined,
|
||||
destinationCredentialsChanged: body.destinationCredentials !== undefined,
|
||||
},
|
||||
},
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({ drain: serializeDrain(updated) })
|
||||
})
|
||||
|
||||
export const DELETE = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId, drainId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: true })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(deleteDataDrainContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const drain = await loadDrain(organizationId, drainId)
|
||||
if (!drain) {
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
await db.delete(dataDrains).where(eq(dataDrains.id, drainId))
|
||||
|
||||
logger.info('Data drain deleted', { drainId, organizationId })
|
||||
|
||||
recordAudit({
|
||||
workspaceId: null,
|
||||
actorId: access.session.user.id,
|
||||
action: AuditAction.DATA_DRAIN_DELETED,
|
||||
resourceType: AuditResourceType.DATA_DRAIN,
|
||||
resourceId: drainId,
|
||||
actorName: access.session.user.name ?? undefined,
|
||||
actorEmail: access.session.user.email ?? undefined,
|
||||
resourceName: drain.name,
|
||||
description: `Deleted data drain '${drain.name}'`,
|
||||
metadata: {
|
||||
organizationId,
|
||||
source: drain.source,
|
||||
destinationType: drain.destinationType,
|
||||
},
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({ success: true as const })
|
||||
})
|
||||
@@ -0,0 +1,76 @@
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit'
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrainRuns } from '@sim/db/schema'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { and, eq } from 'drizzle-orm'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { runDataDrainContract } from '@/lib/api/contracts/data-drains'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { getJobQueue } from '@/lib/core/async-jobs'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { authorizeDrainAccess, loadDrain } from '@/lib/data-drains/access'
|
||||
|
||||
const logger = createLogger('DataDrainRunAPI')
|
||||
|
||||
type RouteContext = { params: Promise<{ id: string; drainId: string }> }
|
||||
|
||||
export const POST = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId, drainId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: true })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(runDataDrainContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const drain = await loadDrain(organizationId, drainId)
|
||||
if (!drain) {
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
if (!drain.enabled) {
|
||||
return NextResponse.json(
|
||||
{ error: 'Cannot run a disabled drain. Enable it first.' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
|
||||
// Reject obvious double-fires up-front. The job-queue concurrencyKey is the
|
||||
// real serialization barrier (it covers the gap between enqueue and the
|
||||
// runner inserting the `running` row), but this gives the user immediate
|
||||
// feedback when a run is already in flight.
|
||||
const [inFlight] = await db
|
||||
.select({ id: dataDrainRuns.id })
|
||||
.from(dataDrainRuns)
|
||||
.where(and(eq(dataDrainRuns.drainId, drainId), eq(dataDrainRuns.status, 'running')))
|
||||
.limit(1)
|
||||
if (inFlight) {
|
||||
return NextResponse.json(
|
||||
{ error: 'A run is already in progress for this drain' },
|
||||
{ status: 409 }
|
||||
)
|
||||
}
|
||||
|
||||
const queue = await getJobQueue()
|
||||
const jobId = await queue.enqueue(
|
||||
'run-data-drain',
|
||||
{ drainId, trigger: 'manual' },
|
||||
{ concurrencyKey: `data-drain:${drainId}` }
|
||||
)
|
||||
|
||||
logger.info('Manually enqueued data drain run', { drainId, organizationId, jobId })
|
||||
|
||||
recordAudit({
|
||||
workspaceId: null,
|
||||
actorId: access.session.user.id,
|
||||
action: AuditAction.DATA_DRAIN_RAN,
|
||||
resourceType: AuditResourceType.DATA_DRAIN,
|
||||
resourceId: drainId,
|
||||
actorName: access.session.user.name ?? undefined,
|
||||
actorEmail: access.session.user.email ?? undefined,
|
||||
resourceName: drain.name,
|
||||
description: `Triggered manual run for data drain '${drain.name}'`,
|
||||
metadata: { organizationId, jobId, trigger: 'manual' },
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({ jobId })
|
||||
})
|
||||
@@ -0,0 +1,37 @@
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrainRuns } from '@sim/db/schema'
|
||||
import { desc, eq } from 'drizzle-orm'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { listDataDrainRunsContract } from '@/lib/api/contracts/data-drains'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { authorizeDrainAccess, loadDrain } from '@/lib/data-drains/access'
|
||||
import { serializeDrainRun } from '@/lib/data-drains/serializers'
|
||||
|
||||
const DEFAULT_LIMIT = 25
|
||||
|
||||
type RouteContext = { params: Promise<{ id: string; drainId: string }> }
|
||||
|
||||
export const GET = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId, drainId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: false })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(listDataDrainRunsContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const drain = await loadDrain(organizationId, drainId)
|
||||
if (!drain) {
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
const limit = parsed.data.query?.limit ?? DEFAULT_LIMIT
|
||||
const runs = await db
|
||||
.select()
|
||||
.from(dataDrainRuns)
|
||||
.where(eq(dataDrainRuns.drainId, drainId))
|
||||
.orderBy(desc(dataDrainRuns.startedAt))
|
||||
.limit(limit)
|
||||
|
||||
return NextResponse.json({ runs: runs.map(serializeDrainRun) })
|
||||
})
|
||||
@@ -0,0 +1,91 @@
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { testDataDrainContract } from '@/lib/api/contracts/data-drains'
|
||||
import { parseRequest } from '@/lib/api/server'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { authorizeDrainAccess, loadDrain } from '@/lib/data-drains/access'
|
||||
import { getDestination } from '@/lib/data-drains/destinations/registry'
|
||||
import { decryptCredentials } from '@/lib/data-drains/encryption'
|
||||
|
||||
const logger = createLogger('DataDrainTestAPI')
|
||||
|
||||
const TEST_TIMEOUT_MS = 10_000
|
||||
|
||||
type RouteContext = { params: Promise<{ id: string; drainId: string }> }
|
||||
|
||||
export const POST = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId, drainId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: true })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(testDataDrainContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const drain = await loadDrain(organizationId, drainId)
|
||||
if (!drain) {
|
||||
return NextResponse.json({ error: 'Data drain not found' }, { status: 404 })
|
||||
}
|
||||
|
||||
const destination = getDestination(drain.destinationType)
|
||||
if (!destination.test) {
|
||||
return NextResponse.json(
|
||||
{ error: `Destination '${drain.destinationType}' does not support connection testing` },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
|
||||
const config = destination.configSchema.parse(drain.destinationConfig)
|
||||
const credentials = destination.credentialsSchema.parse(
|
||||
await decryptCredentials(drain.destinationCredentials)
|
||||
)
|
||||
|
||||
const controller = new AbortController()
|
||||
const timeout = setTimeout(() => controller.abort(), TEST_TIMEOUT_MS)
|
||||
try {
|
||||
await destination.test({ config, credentials, signal: controller.signal })
|
||||
recordAudit({
|
||||
workspaceId: null,
|
||||
actorId: access.session.user.id,
|
||||
action: AuditAction.DATA_DRAIN_TESTED,
|
||||
resourceType: AuditResourceType.DATA_DRAIN,
|
||||
resourceId: drainId,
|
||||
actorName: access.session.user.name ?? undefined,
|
||||
actorEmail: access.session.user.email ?? undefined,
|
||||
resourceName: drain.name,
|
||||
description: `Tested connection for data drain '${drain.name}' (success)`,
|
||||
metadata: { organizationId, destinationType: drain.destinationType, outcome: 'success' },
|
||||
request,
|
||||
})
|
||||
return NextResponse.json({ ok: true as const })
|
||||
} catch (error) {
|
||||
const message = toError(error).message
|
||||
logger.warn('Data drain test connection failed', {
|
||||
drainId,
|
||||
destinationType: drain.destinationType,
|
||||
error: message,
|
||||
})
|
||||
recordAudit({
|
||||
workspaceId: null,
|
||||
actorId: access.session.user.id,
|
||||
action: AuditAction.DATA_DRAIN_TESTED,
|
||||
resourceType: AuditResourceType.DATA_DRAIN,
|
||||
resourceId: drainId,
|
||||
actorName: access.session.user.name ?? undefined,
|
||||
actorEmail: access.session.user.email ?? undefined,
|
||||
resourceName: drain.name,
|
||||
description: `Tested connection for data drain '${drain.name}' (failed)`,
|
||||
metadata: {
|
||||
organizationId,
|
||||
destinationType: drain.destinationType,
|
||||
outcome: 'failed',
|
||||
error: message,
|
||||
},
|
||||
request,
|
||||
})
|
||||
return NextResponse.json({ error: message }, { status: 400 })
|
||||
} finally {
|
||||
clearTimeout(timeout)
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,120 @@
|
||||
import { AuditAction, AuditResourceType, recordAudit } from '@sim/audit'
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrains } from '@sim/db/schema'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { generateId } from '@sim/utils/id'
|
||||
import { and, asc, eq } from 'drizzle-orm'
|
||||
import { type NextRequest, NextResponse } from 'next/server'
|
||||
import { createDataDrainContract, listDataDrainsContract } from '@/lib/api/contracts/data-drains'
|
||||
import { parseRequest, validationErrorResponse } from '@/lib/api/server'
|
||||
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
|
||||
import { authorizeDrainAccess } from '@/lib/data-drains/access'
|
||||
import { getDestination } from '@/lib/data-drains/destinations/registry'
|
||||
import { encryptCredentials } from '@/lib/data-drains/encryption'
|
||||
import { serializeDrain } from '@/lib/data-drains/serializers'
|
||||
|
||||
const logger = createLogger('DataDrainsAPI')
|
||||
|
||||
type RouteContext = { params: Promise<{ id: string }> }
|
||||
|
||||
export const GET = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: false })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(listDataDrainsContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(dataDrains)
|
||||
.where(eq(dataDrains.organizationId, organizationId))
|
||||
.orderBy(asc(dataDrains.createdAt))
|
||||
|
||||
return NextResponse.json({ drains: rows.map(serializeDrain) })
|
||||
})
|
||||
|
||||
export const POST = withRouteHandler(async (request: NextRequest, context: RouteContext) => {
|
||||
const { id: organizationId } = await context.params
|
||||
const access = await authorizeDrainAccess(organizationId, { requireMutating: true })
|
||||
if (!access.ok) return access.response
|
||||
|
||||
const parsed = await parseRequest(createDataDrainContract, request, context)
|
||||
if (!parsed.success) return parsed.response
|
||||
|
||||
const body = parsed.data.body
|
||||
|
||||
if (!body.destinationCredentials) {
|
||||
return NextResponse.json(
|
||||
{ error: 'destinationCredentials is required when creating a drain' },
|
||||
{ status: 400 }
|
||||
)
|
||||
}
|
||||
const destination = getDestination(body.destinationType)
|
||||
const configResult = destination.configSchema.safeParse(body.destinationConfig)
|
||||
if (!configResult.success) return validationErrorResponse(configResult.error)
|
||||
const credentialsResult = destination.credentialsSchema.safeParse(body.destinationCredentials)
|
||||
if (!credentialsResult.success) return validationErrorResponse(credentialsResult.error)
|
||||
const encryptedCredentials = await encryptCredentials(credentialsResult.data)
|
||||
|
||||
const [existing] = await db
|
||||
.select({ id: dataDrains.id })
|
||||
.from(dataDrains)
|
||||
.where(and(eq(dataDrains.organizationId, organizationId), eq(dataDrains.name, body.name)))
|
||||
.limit(1)
|
||||
if (existing) {
|
||||
return NextResponse.json(
|
||||
{ error: 'A data drain with this name already exists in this organization' },
|
||||
{ status: 409 }
|
||||
)
|
||||
}
|
||||
|
||||
const id = generateId()
|
||||
const now = new Date()
|
||||
const [inserted] = await db
|
||||
.insert(dataDrains)
|
||||
.values({
|
||||
id,
|
||||
organizationId,
|
||||
name: body.name,
|
||||
source: body.source,
|
||||
destinationType: body.destinationType,
|
||||
destinationConfig: configResult.data as Record<string, unknown>,
|
||||
destinationCredentials: encryptedCredentials,
|
||||
scheduleCadence: body.scheduleCadence,
|
||||
enabled: body.enabled ?? true,
|
||||
cursor: null,
|
||||
createdBy: access.session.user.id,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.returning()
|
||||
|
||||
logger.info('Data drain created', {
|
||||
drainId: id,
|
||||
organizationId,
|
||||
source: body.source,
|
||||
destinationType: body.destinationType,
|
||||
})
|
||||
|
||||
recordAudit({
|
||||
workspaceId: null,
|
||||
actorId: access.session.user.id,
|
||||
action: AuditAction.DATA_DRAIN_CREATED,
|
||||
resourceType: AuditResourceType.DATA_DRAIN,
|
||||
resourceId: id,
|
||||
actorName: access.session.user.name ?? undefined,
|
||||
actorEmail: access.session.user.email ?? undefined,
|
||||
resourceName: body.name,
|
||||
description: `Created data drain '${body.name}'`,
|
||||
metadata: {
|
||||
organizationId,
|
||||
source: body.source,
|
||||
destinationType: body.destinationType,
|
||||
scheduleCadence: body.scheduleCadence,
|
||||
},
|
||||
request,
|
||||
})
|
||||
|
||||
return NextResponse.json({ drain: serializeDrain(inserted) }, { status: 201 })
|
||||
})
|
||||
@@ -29,6 +29,7 @@ import {
|
||||
isCredentialSetsEnabled,
|
||||
} from '@/app/workspace/[workspaceId]/settings/navigation'
|
||||
import { AuditLogsSkeleton } from '@/ee/audit-logs/components/audit-logs-skeleton'
|
||||
import { DataDrainsSkeleton } from '@/ee/data-drains/components/data-drains-skeleton'
|
||||
import { DataRetentionSkeleton } from '@/ee/data-retention/components/data-retention-skeleton'
|
||||
|
||||
/**
|
||||
@@ -177,6 +178,11 @@ const DataRetentionSettings = dynamic(
|
||||
),
|
||||
{ loading: () => <DataRetentionSkeleton /> }
|
||||
)
|
||||
const DataDrainsSettings = dynamic(
|
||||
() =>
|
||||
import('@/ee/data-drains/components/data-drains-settings').then((m) => m.DataDrainsSettings),
|
||||
{ loading: () => <DataDrainsSkeleton /> }
|
||||
)
|
||||
const WhitelabelingSettings = dynamic(
|
||||
() =>
|
||||
import('@/ee/whitelabeling/components/whitelabeling-settings').then(
|
||||
@@ -235,6 +241,7 @@ export function SettingsPage({ section }: SettingsPageProps) {
|
||||
{isBillingEnabled && effectiveSection === 'organization' && <TeamManagement />}
|
||||
{effectiveSection === 'sso' && <SSO />}
|
||||
{effectiveSection === 'data-retention' && <DataRetentionSettings />}
|
||||
{effectiveSection === 'data-drains' && <DataDrainsSettings />}
|
||||
{effectiveSection === 'whitelabeling' && <WhitelabelingSettings />}
|
||||
{effectiveSection === 'byok' && <BYOK />}
|
||||
{effectiveSection === 'copilot' && <Copilot />}
|
||||
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
ShieldCheck,
|
||||
TerminalWindow,
|
||||
TrashOutline,
|
||||
Upload,
|
||||
Users,
|
||||
Wrench,
|
||||
} from '@/components/emcn'
|
||||
@@ -44,6 +45,7 @@ export type SettingsSection =
|
||||
| 'inbox'
|
||||
| 'admin'
|
||||
| 'data-retention'
|
||||
| 'data-drains'
|
||||
| 'mothership'
|
||||
| 'recently-deleted'
|
||||
|
||||
@@ -80,6 +82,7 @@ const isInboxEnabled = isTruthy(getEnv('NEXT_PUBLIC_INBOX_ENABLED'))
|
||||
const isWhitelabelingEnabled = isTruthy(getEnv('NEXT_PUBLIC_WHITELABELING_ENABLED'))
|
||||
const isAuditLogsEnabled = isTruthy(getEnv('NEXT_PUBLIC_AUDIT_LOGS_ENABLED'))
|
||||
const isDataRetentionEnabled = isTruthy(getEnv('NEXT_PUBLIC_DATA_RETENTION_ENABLED'))
|
||||
const isDataDrainsEnabled = isTruthy(getEnv('NEXT_PUBLIC_DATA_DRAINS_ENABLED'))
|
||||
|
||||
export const isBillingEnabled = isTruthy(getEnv('NEXT_PUBLIC_BILLING_ENABLED'))
|
||||
export { isCredentialSetsEnabled }
|
||||
@@ -190,6 +193,15 @@ export const allNavigationItems: NavigationItem[] = [
|
||||
requiresEnterprise: true,
|
||||
selfHostedOverride: isDataRetentionEnabled,
|
||||
},
|
||||
{
|
||||
id: 'data-drains',
|
||||
label: 'Data Drains',
|
||||
icon: Upload,
|
||||
section: 'enterprise',
|
||||
requiresHosted: true,
|
||||
requiresEnterprise: true,
|
||||
selfHostedOverride: isDataDrainsEnabled,
|
||||
},
|
||||
{
|
||||
id: 'whitelabeling',
|
||||
label: 'Whitelabeling',
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
import { task } from '@trigger.dev/sdk'
|
||||
import { runDrain } from '@/lib/data-drains/service'
|
||||
import type { RunTrigger } from '@/lib/data-drains/types'
|
||||
|
||||
interface RunDataDrainPayload {
|
||||
drainId: string
|
||||
trigger: RunTrigger
|
||||
}
|
||||
|
||||
export const runDataDrainTask = task({
|
||||
id: 'run-data-drain',
|
||||
run: async ({ drainId, trigger }: RunDataDrainPayload, { signal }) =>
|
||||
runDrain(drainId, trigger, { signal }),
|
||||
})
|
||||
@@ -0,0 +1,435 @@
|
||||
'use client'
|
||||
|
||||
import { useState } from 'react'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import {
|
||||
Badge,
|
||||
Button,
|
||||
Callout,
|
||||
Combobox,
|
||||
DropdownMenu,
|
||||
DropdownMenuContent,
|
||||
DropdownMenuItem,
|
||||
DropdownMenuTrigger,
|
||||
FormField,
|
||||
Input,
|
||||
Modal,
|
||||
ModalBody,
|
||||
ModalContent,
|
||||
ModalFooter,
|
||||
ModalHeader,
|
||||
ModalTitle,
|
||||
MoreHorizontal,
|
||||
Switch,
|
||||
Table,
|
||||
TableBody,
|
||||
TableCell,
|
||||
TableHead,
|
||||
TableHeader,
|
||||
TableRow,
|
||||
toast,
|
||||
} from '@/components/emcn'
|
||||
import type { CreateDataDrainBody, DataDrain, DataDrainRun } from '@/lib/api/contracts/data-drains'
|
||||
import { useSession } from '@/lib/auth/auth-client'
|
||||
import { cn } from '@/lib/core/utils/cn'
|
||||
import { CADENCE_TYPES, DESTINATION_TYPES, SOURCE_TYPES } from '@/lib/data-drains/types'
|
||||
import { getUserRole } from '@/lib/workspaces/organization/utils'
|
||||
import { DataDrainsSkeleton } from '@/ee/data-drains/components/data-drains-skeleton'
|
||||
import { DESTINATION_FORM_REGISTRY } from '@/ee/data-drains/destinations/registry'
|
||||
import {
|
||||
useCreateDataDrain,
|
||||
useDataDrainRuns,
|
||||
useDataDrains,
|
||||
useDeleteDataDrain,
|
||||
useRunDataDrainNow,
|
||||
useTestDataDrain,
|
||||
useUpdateDataDrain,
|
||||
} from '@/ee/data-drains/hooks/data-drains'
|
||||
import { useOrganizations } from '@/hooks/queries/organization'
|
||||
|
||||
const logger = createLogger('DataDrainsSettings')
|
||||
|
||||
const SOURCE_LABELS: Record<(typeof SOURCE_TYPES)[number], string> = {
|
||||
workflow_logs: 'Workflow logs',
|
||||
job_logs: 'Job logs',
|
||||
audit_logs: 'Audit logs',
|
||||
copilot_chats: 'Copilot chats',
|
||||
copilot_runs: 'Copilot runs',
|
||||
}
|
||||
|
||||
const DESTINATION_LABELS: Record<(typeof DESTINATION_TYPES)[number], string> = {
|
||||
s3: 'Amazon S3',
|
||||
webhook: 'HTTPS webhook',
|
||||
}
|
||||
|
||||
const CADENCE_LABELS: Record<(typeof CADENCE_TYPES)[number], string> = {
|
||||
hourly: 'Every hour',
|
||||
daily: 'Every day',
|
||||
}
|
||||
|
||||
const SOURCE_OPTIONS = SOURCE_TYPES.map((t) => ({ value: t, label: SOURCE_LABELS[t] }))
|
||||
const CADENCE_OPTIONS = CADENCE_TYPES.map((t) => ({ value: t, label: CADENCE_LABELS[t] }))
|
||||
const DESTINATION_OPTIONS = DESTINATION_TYPES.map((t) => ({
|
||||
value: t,
|
||||
label: DESTINATION_LABELS[t],
|
||||
}))
|
||||
|
||||
export function DataDrainsSettings() {
|
||||
const { data: session, isPending: sessionPending } = useSession()
|
||||
const { data: orgsData, isLoading: orgsLoading } = useOrganizations()
|
||||
const activeOrganization = orgsData?.activeOrganization
|
||||
const orgId = activeOrganization?.id
|
||||
|
||||
const userEmail = session?.user?.email
|
||||
const userRole = getUserRole(activeOrganization, userEmail)
|
||||
const canManage = userRole === 'owner' || userRole === 'admin'
|
||||
|
||||
const { data: drains, isLoading: drainsLoading, error: drainsError } = useDataDrains(orgId)
|
||||
|
||||
const [createOpen, setCreateOpen] = useState(false)
|
||||
const [expandedDrainId, setExpandedDrainId] = useState<string | null>(null)
|
||||
|
||||
if (sessionPending || orgsLoading || drainsLoading) {
|
||||
return <DataDrainsSkeleton />
|
||||
}
|
||||
|
||||
if (!orgId) {
|
||||
return (
|
||||
<div className='flex h-full items-center justify-center text-[var(--text-muted)] text-sm'>
|
||||
Data drains are configured per organization. Join or create one to continue.
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
if (!canManage) {
|
||||
return (
|
||||
<div className='flex h-full items-center justify-center text-[var(--text-muted)] text-sm'>
|
||||
Only organization owners and admins can configure data drains.
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
return (
|
||||
<div className='flex flex-col gap-6'>
|
||||
<Callout>
|
||||
Drains continuously export Sim data to your own storage on a schedule. Combine with Data
|
||||
Retention to satisfy long-term compliance archives.
|
||||
</Callout>
|
||||
|
||||
<div className='flex items-center justify-between'>
|
||||
<div className='text-[13px] text-[var(--text-muted)]'>
|
||||
{drains?.length ?? 0} drain{(drains?.length ?? 0) === 1 ? '' : 's'}
|
||||
</div>
|
||||
<Button variant='primary' onClick={() => setCreateOpen(true)}>
|
||||
New drain
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{drainsError ? (
|
||||
<Callout variant='destructive'>
|
||||
Failed to load data drains: {toError(drainsError).message}
|
||||
</Callout>
|
||||
) : drains && drains.length > 0 ? (
|
||||
<Table>
|
||||
<TableHeader>
|
||||
<TableRow>
|
||||
<TableHead>Name</TableHead>
|
||||
<TableHead>Source</TableHead>
|
||||
<TableHead>Destination</TableHead>
|
||||
<TableHead>Cadence</TableHead>
|
||||
<TableHead>Last run</TableHead>
|
||||
<TableHead>Enabled</TableHead>
|
||||
<TableHead className='w-[40px]' />
|
||||
</TableRow>
|
||||
</TableHeader>
|
||||
<TableBody>
|
||||
{drains.map((drain) => (
|
||||
<DrainRow
|
||||
key={drain.id}
|
||||
drain={drain}
|
||||
organizationId={orgId}
|
||||
expanded={expandedDrainId === drain.id}
|
||||
onToggleExpand={() =>
|
||||
setExpandedDrainId(expandedDrainId === drain.id ? null : drain.id)
|
||||
}
|
||||
/>
|
||||
))}
|
||||
</TableBody>
|
||||
</Table>
|
||||
) : (
|
||||
<div className='flex flex-col items-center justify-center gap-3 rounded-lg border border-[var(--border)] border-dashed py-12 text-center'>
|
||||
<div className='text-[14px] text-[var(--text-primary)]'>No drains yet</div>
|
||||
<div className='max-w-[400px] text-[13px] text-[var(--text-muted)]'>
|
||||
Create a drain to start exporting workflow logs, audit events, and copilot data to S3 or
|
||||
your own webhook.
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{createOpen && (
|
||||
<CreateDrainModal organizationId={orgId} onClose={() => setCreateOpen(false)} />
|
||||
)}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
interface DrainRowProps {
|
||||
drain: DataDrain
|
||||
organizationId: string
|
||||
expanded: boolean
|
||||
onToggleExpand: () => void
|
||||
}
|
||||
|
||||
function DrainRow({ drain, organizationId, expanded, onToggleExpand }: DrainRowProps) {
|
||||
const updateMutation = useUpdateDataDrain()
|
||||
const deleteMutation = useDeleteDataDrain()
|
||||
const runMutation = useRunDataDrainNow()
|
||||
const testMutation = useTestDataDrain()
|
||||
|
||||
async function handleToggleEnabled() {
|
||||
try {
|
||||
await updateMutation.mutateAsync({
|
||||
organizationId,
|
||||
drainId: drain.id,
|
||||
body: { enabled: !drain.enabled },
|
||||
})
|
||||
toast.success(drain.enabled ? 'Drain disabled' : 'Drain enabled')
|
||||
} catch (error) {
|
||||
toast.error(toError(error).message)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleRunNow() {
|
||||
try {
|
||||
await runMutation.mutateAsync({ organizationId, drainId: drain.id })
|
||||
toast.success('Drain run enqueued')
|
||||
} catch (error) {
|
||||
toast.error(toError(error).message)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleTest() {
|
||||
try {
|
||||
await testMutation.mutateAsync({ organizationId, drainId: drain.id })
|
||||
toast.success('Connection test succeeded')
|
||||
} catch (error) {
|
||||
toast.error(toError(error).message)
|
||||
}
|
||||
}
|
||||
|
||||
async function handleDelete() {
|
||||
if (!window.confirm(`Delete drain "${drain.name}"? This cannot be undone.`)) return
|
||||
try {
|
||||
await deleteMutation.mutateAsync({ organizationId, drainId: drain.id })
|
||||
toast.success('Drain deleted')
|
||||
} catch (error) {
|
||||
toast.error(toError(error).message)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<>
|
||||
<TableRow className='cursor-pointer' onClick={onToggleExpand}>
|
||||
<TableCell className='font-medium'>{drain.name}</TableCell>
|
||||
<TableCell>
|
||||
<Badge>{SOURCE_LABELS[drain.source]}</Badge>
|
||||
</TableCell>
|
||||
<TableCell>
|
||||
<Badge>{DESTINATION_LABELS[drain.destinationType]}</Badge>
|
||||
</TableCell>
|
||||
<TableCell>{CADENCE_LABELS[drain.scheduleCadence]}</TableCell>
|
||||
<TableCell className='text-[13px] text-[var(--text-muted)]'>
|
||||
{drain.lastRunAt ? new Date(drain.lastRunAt).toLocaleString() : 'Never'}
|
||||
</TableCell>
|
||||
<TableCell onClick={(e) => e.stopPropagation()}>
|
||||
<Switch
|
||||
checked={drain.enabled}
|
||||
onCheckedChange={handleToggleEnabled}
|
||||
disabled={updateMutation.isPending}
|
||||
/>
|
||||
</TableCell>
|
||||
<TableCell onClick={(e) => e.stopPropagation()}>
|
||||
<DropdownMenu>
|
||||
<DropdownMenuTrigger asChild>
|
||||
<Button variant='ghost' size='sm' aria-label='Drain actions'>
|
||||
<MoreHorizontal className='h-4 w-4' />
|
||||
</Button>
|
||||
</DropdownMenuTrigger>
|
||||
<DropdownMenuContent align='end'>
|
||||
<DropdownMenuItem onClick={handleRunNow} disabled={!drain.enabled}>
|
||||
Run now
|
||||
</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={handleTest}>Test connection</DropdownMenuItem>
|
||||
<DropdownMenuItem onClick={handleDelete} className='text-red-600'>
|
||||
Delete
|
||||
</DropdownMenuItem>
|
||||
</DropdownMenuContent>
|
||||
</DropdownMenu>
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
{expanded && (
|
||||
<TableRow>
|
||||
<TableCell colSpan={7} className='bg-[var(--surface-muted)] p-4'>
|
||||
<DrainRunsPanel organizationId={organizationId} drainId={drain.id} />
|
||||
</TableCell>
|
||||
</TableRow>
|
||||
)}
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
interface DrainRunsPanelProps {
|
||||
organizationId: string
|
||||
drainId: string
|
||||
}
|
||||
|
||||
function DrainRunsPanel({ organizationId, drainId }: DrainRunsPanelProps) {
|
||||
const { data: runs, isLoading } = useDataDrainRuns(organizationId, drainId, 10)
|
||||
|
||||
if (isLoading) {
|
||||
return <div className='text-[13px] text-[var(--text-muted)]'>Loading runs...</div>
|
||||
}
|
||||
if (!runs || runs.length === 0) {
|
||||
return <div className='text-[13px] text-[var(--text-muted)]'>No runs yet.</div>
|
||||
}
|
||||
|
||||
return (
|
||||
<div className='flex flex-col gap-2'>
|
||||
<div className='font-medium text-[13px] text-[var(--text-primary)]'>Recent runs</div>
|
||||
{runs.map((run) => (
|
||||
<RunRow key={run.id} run={run} />
|
||||
))}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function RunRow({ run }: { run: DataDrainRun }) {
|
||||
const statusColor =
|
||||
run.status === 'success'
|
||||
? 'text-green-600'
|
||||
: run.status === 'failed'
|
||||
? 'text-red-600'
|
||||
: 'text-[var(--text-muted)]'
|
||||
return (
|
||||
<div className='flex items-start justify-between gap-4 rounded border border-[var(--border)] px-3 py-2 text-[12px]'>
|
||||
<div className='flex flex-col gap-0.5'>
|
||||
<div className='flex items-center gap-2'>
|
||||
<span className={cn('font-medium', statusColor)}>{run.status}</span>
|
||||
<span className='text-[var(--text-muted)]'>{run.trigger}</span>
|
||||
<span className='text-[var(--text-muted)]'>
|
||||
{new Date(run.startedAt).toLocaleString()}
|
||||
</span>
|
||||
</div>
|
||||
{run.error && <div className='text-red-600'>{run.error}</div>}
|
||||
</div>
|
||||
<div className='text-right text-[var(--text-muted)]'>
|
||||
<div>{run.rowsExported.toLocaleString()} rows</div>
|
||||
<div>{(run.bytesWritten / 1024).toFixed(1)} KB</div>
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
interface CreateDrainModalProps {
|
||||
organizationId: string
|
||||
onClose: () => void
|
||||
}
|
||||
|
||||
function CreateDrainModal({ organizationId, onClose }: CreateDrainModalProps) {
|
||||
const createMutation = useCreateDataDrain()
|
||||
|
||||
const [name, setName] = useState('')
|
||||
const [source, setSource] = useState<(typeof SOURCE_TYPES)[number]>('workflow_logs')
|
||||
const [cadence, setCadence] = useState<(typeof CADENCE_TYPES)[number]>('daily')
|
||||
const [destinationType, setDestinationType] = useState<(typeof DESTINATION_TYPES)[number]>(
|
||||
DESTINATION_TYPES[0]
|
||||
)
|
||||
const [destState, setDestState] = useState<unknown>(
|
||||
() => DESTINATION_FORM_REGISTRY[DESTINATION_TYPES[0]].initialState
|
||||
)
|
||||
|
||||
const spec = DESTINATION_FORM_REGISTRY[destinationType]
|
||||
const canSubmit = name.trim().length > 0 && spec.isComplete(destState)
|
||||
|
||||
function handleDestinationChange(next: (typeof DESTINATION_TYPES)[number]) {
|
||||
setDestinationType(next)
|
||||
setDestState(DESTINATION_FORM_REGISTRY[next].initialState)
|
||||
}
|
||||
|
||||
async function handleSubmit() {
|
||||
if (!canSubmit) return
|
||||
try {
|
||||
const body = {
|
||||
name: name.trim(),
|
||||
source,
|
||||
scheduleCadence: cadence,
|
||||
...spec.toDestinationBranch(destState),
|
||||
} as CreateDataDrainBody
|
||||
await createMutation.mutateAsync({ organizationId, body })
|
||||
toast.success('Drain created')
|
||||
onClose()
|
||||
} catch (error) {
|
||||
const msg = toError(error).message
|
||||
logger.error('Failed to create data drain', { error: msg })
|
||||
toast.error(msg)
|
||||
}
|
||||
}
|
||||
|
||||
return (
|
||||
<Modal open onOpenChange={(open) => !open && onClose()}>
|
||||
<ModalContent className='max-w-[560px]'>
|
||||
<ModalHeader>
|
||||
<ModalTitle>New data drain</ModalTitle>
|
||||
</ModalHeader>
|
||||
<ModalBody className='flex flex-col gap-4'>
|
||||
<FormField label='Name'>
|
||||
<Input
|
||||
value={name}
|
||||
onChange={(e) => setName(e.target.value)}
|
||||
placeholder='Workflow logs to S3'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Source'>
|
||||
<Combobox
|
||||
value={source}
|
||||
onChange={(v) => setSource(v as (typeof SOURCE_TYPES)[number])}
|
||||
options={SOURCE_OPTIONS}
|
||||
dropdownWidth='trigger'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Cadence'>
|
||||
<Combobox
|
||||
value={cadence}
|
||||
onChange={(v) => setCadence(v as (typeof CADENCE_TYPES)[number])}
|
||||
options={CADENCE_OPTIONS}
|
||||
dropdownWidth='trigger'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Destination'>
|
||||
<Combobox
|
||||
value={destinationType}
|
||||
onChange={(v) => handleDestinationChange(v as (typeof DESTINATION_TYPES)[number])}
|
||||
options={DESTINATION_OPTIONS}
|
||||
dropdownWidth='trigger'
|
||||
/>
|
||||
</FormField>
|
||||
|
||||
<spec.FormFields state={destState} setState={setDestState} />
|
||||
</ModalBody>
|
||||
<ModalFooter>
|
||||
<Button variant='secondary' onClick={onClose}>
|
||||
Cancel
|
||||
</Button>
|
||||
<Button
|
||||
variant='primary'
|
||||
onClick={handleSubmit}
|
||||
disabled={!canSubmit || createMutation.isPending}
|
||||
>
|
||||
{createMutation.isPending ? 'Creating...' : 'Create drain'}
|
||||
</Button>
|
||||
</ModalFooter>
|
||||
</ModalContent>
|
||||
</Modal>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import { Skeleton } from '@/components/emcn'
|
||||
|
||||
export function DataDrainsSkeleton() {
|
||||
return (
|
||||
<div className='flex flex-col gap-8'>
|
||||
<div className='flex items-center justify-between'>
|
||||
<Skeleton className='h-[18px] w-[200px]' />
|
||||
<Skeleton className='h-[34px] w-[110px] rounded-lg' />
|
||||
</div>
|
||||
<div className='flex flex-col gap-3'>
|
||||
{Array.from({ length: 3 }).map((_, i) => (
|
||||
<Skeleton key={i} className='h-[64px] w-full rounded-lg' />
|
||||
))}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
'use client'
|
||||
|
||||
import type { ComponentType } from 'react'
|
||||
import { FormField, Input, SecretInput, Switch } from '@/components/emcn'
|
||||
import type { CreateDataDrainBody } from '@/lib/api/contracts/data-drains'
|
||||
import type { DestinationType } from '@/lib/data-drains/types'
|
||||
|
||||
type DestinationBranch = Pick<
|
||||
CreateDataDrainBody,
|
||||
'destinationType' | 'destinationConfig' | 'destinationCredentials'
|
||||
>
|
||||
|
||||
interface DestinationFormSpec<TState> {
|
||||
readonly displayName: string
|
||||
readonly initialState: TState
|
||||
readonly FormFields: ComponentType<{
|
||||
state: TState
|
||||
setState: (state: TState) => void
|
||||
}>
|
||||
readonly isComplete: (state: TState) => boolean
|
||||
readonly toDestinationBranch: (state: TState) => DestinationBranch
|
||||
}
|
||||
|
||||
interface S3State {
|
||||
bucket: string
|
||||
region: string
|
||||
prefix: string
|
||||
endpoint: string
|
||||
forcePathStyle: boolean
|
||||
accessKeyId: string
|
||||
secretAccessKey: string
|
||||
}
|
||||
|
||||
const s3FormSpec: DestinationFormSpec<S3State> = {
|
||||
displayName: 'Amazon S3',
|
||||
initialState: {
|
||||
bucket: '',
|
||||
region: 'us-east-1',
|
||||
prefix: '',
|
||||
endpoint: '',
|
||||
forcePathStyle: false,
|
||||
accessKeyId: '',
|
||||
secretAccessKey: '',
|
||||
},
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='Bucket'>
|
||||
<Input
|
||||
value={state.bucket}
|
||||
onChange={(e) => setState({ ...state, bucket: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Region'>
|
||||
<Input
|
||||
value={state.region}
|
||||
onChange={(e) => setState({ ...state, region: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Prefix (optional)'>
|
||||
<Input
|
||||
value={state.prefix}
|
||||
onChange={(e) => setState({ ...state, prefix: e.target.value })}
|
||||
placeholder='exports/sim'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Endpoint (optional, S3-compatible stores)'>
|
||||
<Input
|
||||
value={state.endpoint}
|
||||
onChange={(e) => setState({ ...state, endpoint: e.target.value })}
|
||||
placeholder='https://s3.example.com'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Force path style (MinIO, Ceph)'>
|
||||
<Switch
|
||||
checked={state.forcePathStyle}
|
||||
onCheckedChange={(v) => setState({ ...state, forcePathStyle: v })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Access key ID'>
|
||||
<SecretInput
|
||||
value={state.accessKeyId}
|
||||
onChange={(v) => setState({ ...state, accessKeyId: v })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Secret access key'>
|
||||
<SecretInput
|
||||
value={state.secretAccessKey}
|
||||
onChange={(v) => setState({ ...state, secretAccessKey: v })}
|
||||
/>
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) =>
|
||||
s.bucket.length > 0 &&
|
||||
s.region.length > 0 &&
|
||||
s.accessKeyId.length > 0 &&
|
||||
s.secretAccessKey.length > 0,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 's3',
|
||||
destinationConfig: {
|
||||
bucket: s.bucket,
|
||||
region: s.region,
|
||||
prefix: s.prefix || undefined,
|
||||
endpoint: s.endpoint || undefined,
|
||||
forcePathStyle: s.forcePathStyle,
|
||||
},
|
||||
destinationCredentials: {
|
||||
accessKeyId: s.accessKeyId,
|
||||
secretAccessKey: s.secretAccessKey,
|
||||
},
|
||||
}),
|
||||
}
|
||||
|
||||
interface WebhookState {
|
||||
url: string
|
||||
signatureHeader: string
|
||||
signingSecret: string
|
||||
bearerToken: string
|
||||
}
|
||||
|
||||
const webhookFormSpec: DestinationFormSpec<WebhookState> = {
|
||||
displayName: 'HTTPS webhook',
|
||||
initialState: { url: '', signatureHeader: '', signingSecret: '', bearerToken: '' },
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='URL'>
|
||||
<Input
|
||||
value={state.url}
|
||||
onChange={(e) => setState({ ...state, url: e.target.value })}
|
||||
placeholder='https://example.com/sim-drain'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Signature header (optional)'>
|
||||
<Input
|
||||
value={state.signatureHeader}
|
||||
onChange={(e) => setState({ ...state, signatureHeader: e.target.value })}
|
||||
placeholder='X-Sim-Signature'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Signing secret'>
|
||||
<SecretInput
|
||||
value={state.signingSecret}
|
||||
onChange={(v) => setState({ ...state, signingSecret: v })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Bearer token (optional)'>
|
||||
<SecretInput
|
||||
value={state.bearerToken}
|
||||
onChange={(v) => setState({ ...state, bearerToken: v })}
|
||||
/>
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) => s.url.length > 0 && s.signingSecret.length >= 8,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'webhook',
|
||||
destinationConfig: {
|
||||
url: s.url,
|
||||
signatureHeader: s.signatureHeader || undefined,
|
||||
},
|
||||
destinationCredentials: {
|
||||
signingSecret: s.signingSecret,
|
||||
bearerToken: s.bearerToken || undefined,
|
||||
},
|
||||
}),
|
||||
}
|
||||
|
||||
/**
|
||||
* Client-side mirror of `DESTINATION_REGISTRY`. The settings page selects a
|
||||
* spec by `destinationType` and never branches on the literal — adding a new
|
||||
* destination is one entry here plus one in the server-side registry.
|
||||
*/
|
||||
export const DESTINATION_FORM_REGISTRY: Record<DestinationType, DestinationFormSpec<unknown>> = {
|
||||
s3: s3FormSpec as DestinationFormSpec<unknown>,
|
||||
webhook: webhookFormSpec as DestinationFormSpec<unknown>,
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { keepPreviousData, useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
|
||||
import { requestJson } from '@/lib/api/client/request'
|
||||
import {
|
||||
type CreateDataDrainBody,
|
||||
createDataDrainContract,
|
||||
type DataDrain,
|
||||
type DataDrainRun,
|
||||
deleteDataDrainContract,
|
||||
listDataDrainRunsContract,
|
||||
listDataDrainsContract,
|
||||
runDataDrainContract,
|
||||
testDataDrainContract,
|
||||
type UpdateDataDrainBody,
|
||||
updateDataDrainContract,
|
||||
} from '@/lib/api/contracts/data-drains'
|
||||
|
||||
const logger = createLogger('DataDrainsQueries')
|
||||
|
||||
export const dataDrainKeys = {
|
||||
all: ['data-drains'] as const,
|
||||
lists: () => [...dataDrainKeys.all, 'list'] as const,
|
||||
list: (organizationId?: string) => [...dataDrainKeys.lists(), organizationId ?? ''] as const,
|
||||
runsAll: () => [...dataDrainKeys.all, 'runs'] as const,
|
||||
runs: (drainId?: string) => [...dataDrainKeys.runsAll(), drainId ?? ''] as const,
|
||||
runsList: (organizationId?: string, drainId?: string, limit?: number) =>
|
||||
[...dataDrainKeys.runs(drainId), organizationId ?? '', limit ?? 10] as const,
|
||||
}
|
||||
|
||||
async function fetchDataDrains(organizationId: string, signal?: AbortSignal): Promise<DataDrain[]> {
|
||||
const { drains } = await requestJson(listDataDrainsContract, {
|
||||
params: { id: organizationId },
|
||||
signal,
|
||||
})
|
||||
return drains
|
||||
}
|
||||
|
||||
async function fetchDataDrainRuns(
|
||||
organizationId: string,
|
||||
drainId: string,
|
||||
limit: number | undefined,
|
||||
signal?: AbortSignal
|
||||
): Promise<DataDrainRun[]> {
|
||||
const { runs } = await requestJson(listDataDrainRunsContract, {
|
||||
params: { id: organizationId, drainId },
|
||||
query: limit ? { limit } : undefined,
|
||||
signal,
|
||||
})
|
||||
return runs
|
||||
}
|
||||
|
||||
export function useDataDrains(organizationId?: string) {
|
||||
return useQuery<DataDrain[]>({
|
||||
queryKey: dataDrainKeys.list(organizationId),
|
||||
queryFn: ({ signal }) => fetchDataDrains(organizationId as string, signal),
|
||||
enabled: Boolean(organizationId),
|
||||
staleTime: 60 * 1000,
|
||||
})
|
||||
}
|
||||
|
||||
export function useDataDrainRuns(organizationId?: string, drainId?: string, limit = 10) {
|
||||
return useQuery<DataDrainRun[]>({
|
||||
queryKey: dataDrainKeys.runsList(organizationId, drainId, limit),
|
||||
queryFn: ({ signal }) =>
|
||||
fetchDataDrainRuns(organizationId as string, drainId as string, limit, signal),
|
||||
enabled: Boolean(organizationId && drainId),
|
||||
staleTime: 30 * 1000,
|
||||
placeholderData: keepPreviousData,
|
||||
})
|
||||
}
|
||||
|
||||
interface CreateDataDrainParams {
|
||||
organizationId: string
|
||||
body: CreateDataDrainBody
|
||||
}
|
||||
|
||||
export function useCreateDataDrain() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: async ({ organizationId, body }: CreateDataDrainParams) => {
|
||||
const { drain } = await requestJson(createDataDrainContract, {
|
||||
params: { id: organizationId },
|
||||
body,
|
||||
})
|
||||
logger.info('Created data drain', { drainId: drain.id, organizationId })
|
||||
return drain
|
||||
},
|
||||
onSuccess: (_drain, variables) => {
|
||||
queryClient.invalidateQueries({ queryKey: dataDrainKeys.list(variables.organizationId) })
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
interface UpdateDataDrainParams {
|
||||
organizationId: string
|
||||
drainId: string
|
||||
body: UpdateDataDrainBody
|
||||
}
|
||||
|
||||
export function useUpdateDataDrain() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: async ({ organizationId, drainId, body }: UpdateDataDrainParams) => {
|
||||
const { drain } = await requestJson(updateDataDrainContract, {
|
||||
params: { id: organizationId, drainId },
|
||||
body,
|
||||
})
|
||||
logger.info('Updated data drain', { drainId, organizationId })
|
||||
return drain
|
||||
},
|
||||
onSuccess: (_drain, variables) => {
|
||||
queryClient.invalidateQueries({ queryKey: dataDrainKeys.list(variables.organizationId) })
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
interface DeleteDataDrainParams {
|
||||
organizationId: string
|
||||
drainId: string
|
||||
}
|
||||
|
||||
export function useDeleteDataDrain() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: async ({ organizationId, drainId }: DeleteDataDrainParams) => {
|
||||
await requestJson(deleteDataDrainContract, {
|
||||
params: { id: organizationId, drainId },
|
||||
})
|
||||
logger.info('Deleted data drain', { drainId, organizationId })
|
||||
},
|
||||
onSuccess: (_data, variables) => {
|
||||
queryClient.invalidateQueries({ queryKey: dataDrainKeys.list(variables.organizationId) })
|
||||
queryClient.removeQueries({ queryKey: dataDrainKeys.runs(variables.drainId) })
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
interface RunDataDrainParams {
|
||||
organizationId: string
|
||||
drainId: string
|
||||
}
|
||||
|
||||
export function useRunDataDrainNow() {
|
||||
const queryClient = useQueryClient()
|
||||
return useMutation({
|
||||
mutationFn: async ({ organizationId, drainId }: RunDataDrainParams) => {
|
||||
const data = await requestJson(runDataDrainContract, {
|
||||
params: { id: organizationId, drainId },
|
||||
})
|
||||
logger.info('Enqueued data drain run', { drainId, jobId: data.jobId })
|
||||
return data
|
||||
},
|
||||
onSuccess: (_data, variables) => {
|
||||
queryClient.invalidateQueries({ queryKey: dataDrainKeys.runs(variables.drainId) })
|
||||
queryClient.invalidateQueries({ queryKey: dataDrainKeys.list(variables.organizationId) })
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
interface TestDataDrainParams {
|
||||
organizationId: string
|
||||
drainId: string
|
||||
}
|
||||
|
||||
export function useTestDataDrain() {
|
||||
return useMutation({
|
||||
mutationFn: async ({ organizationId, drainId }: TestDataDrainParams) => {
|
||||
return await requestJson(testDataDrainContract, {
|
||||
params: { id: organizationId, drainId },
|
||||
})
|
||||
},
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
import { z } from 'zod'
|
||||
import { defineRouteContract } from '@/lib/api/contracts/types'
|
||||
import { CADENCE_TYPES, DESTINATION_TYPES, SOURCE_TYPES } from '@/lib/data-drains/types'
|
||||
|
||||
export const dataDrainSourceSchema = z.enum(SOURCE_TYPES)
|
||||
export const dataDrainDestinationTypeSchema = z.enum(DESTINATION_TYPES)
|
||||
export const dataDrainCadenceSchema = z.enum(CADENCE_TYPES)
|
||||
export const dataDrainRunStatusSchema = z.enum(['running', 'success', 'failed'])
|
||||
export const dataDrainRunTriggerSchema = z.enum(['cron', 'manual'])
|
||||
|
||||
export const dataDrainOrgParamsSchema = z.object({
|
||||
id: z.string().min(1, 'organization id is required'),
|
||||
})
|
||||
|
||||
export const dataDrainParamsSchema = z.object({
|
||||
id: z.string().min(1, 'organization id is required'),
|
||||
drainId: z.string().min(1, 'drain id is required'),
|
||||
})
|
||||
|
||||
const drainNameSchema = z.string().trim().min(1, 'name is required').max(120)
|
||||
|
||||
const s3ConfigBodySchema = z.object({
|
||||
bucket: z.string().min(1, 'bucket is required').max(255),
|
||||
region: z.string().min(1, 'region is required').max(64),
|
||||
prefix: z.string().max(512).optional(),
|
||||
endpoint: z.string().url().optional(),
|
||||
forcePathStyle: z.boolean().optional(),
|
||||
})
|
||||
|
||||
const s3CredentialsBodySchema = z.object({
|
||||
accessKeyId: z.string().min(1, 'accessKeyId is required'),
|
||||
secretAccessKey: z.string().min(1, 'secretAccessKey is required'),
|
||||
})
|
||||
|
||||
const webhookConfigBodySchema = z.object({
|
||||
url: z.string().url('url must be a valid URL'),
|
||||
signatureHeader: z.string().min(1).max(128).optional(),
|
||||
})
|
||||
|
||||
const webhookCredentialsBodySchema = z.object({
|
||||
signingSecret: z.string().min(8, 'signingSecret must be at least 8 characters'),
|
||||
bearerToken: z.string().min(1).optional(),
|
||||
})
|
||||
|
||||
/**
|
||||
* Discriminated body shape used by both create and update. Each destination
|
||||
* variant carries its own typed `destinationConfig` and optional
|
||||
* `destinationCredentials`. On update, omitting `destinationCredentials`
|
||||
* leaves the encrypted blob in place.
|
||||
*/
|
||||
export const dataDrainDestinationBodySchema = z.discriminatedUnion('destinationType', [
|
||||
z.object({
|
||||
destinationType: z.literal('s3'),
|
||||
destinationConfig: s3ConfigBodySchema,
|
||||
destinationCredentials: s3CredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('webhook'),
|
||||
destinationConfig: webhookConfigBodySchema,
|
||||
destinationCredentials: webhookCredentialsBodySchema.optional(),
|
||||
}),
|
||||
])
|
||||
|
||||
const drainCommonBodyFieldsSchema = z.object({
|
||||
name: drainNameSchema,
|
||||
source: dataDrainSourceSchema,
|
||||
scheduleCadence: dataDrainCadenceSchema,
|
||||
enabled: z.boolean().optional(),
|
||||
})
|
||||
|
||||
export const createDataDrainBodySchema = z.intersection(
|
||||
drainCommonBodyFieldsSchema,
|
||||
dataDrainDestinationBodySchema
|
||||
)
|
||||
|
||||
/**
|
||||
* Update bodies are partial — every field is optional. We deliberately don't
|
||||
* use a discriminated union here: clients sending `{ enabled: false }` should
|
||||
* not be forced to also send `destinationType`. The route validates the
|
||||
* destination payloads against the typed `configSchema` / `credentialsSchema`
|
||||
* for the existing drain's destination type before persisting, so the
|
||||
* structural shape is still enforced — just at the route layer rather than at
|
||||
* the contract boundary.
|
||||
*/
|
||||
export const updateDataDrainBodySchema = drainCommonBodyFieldsSchema.partial().extend({
|
||||
destinationType: dataDrainDestinationTypeSchema.optional(),
|
||||
destinationConfig: z.record(z.string(), z.unknown()).optional(),
|
||||
destinationCredentials: z.record(z.string(), z.unknown()).optional(),
|
||||
})
|
||||
|
||||
const drainDestinationResponseSchema = z.discriminatedUnion('destinationType', [
|
||||
z.object({
|
||||
destinationType: z.literal('s3'),
|
||||
destinationConfig: s3ConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('webhook'),
|
||||
destinationConfig: webhookConfigBodySchema,
|
||||
}),
|
||||
])
|
||||
|
||||
const drainCommonResponseFieldsSchema = z.object({
|
||||
id: z.string(),
|
||||
organizationId: z.string(),
|
||||
name: z.string(),
|
||||
source: dataDrainSourceSchema,
|
||||
scheduleCadence: dataDrainCadenceSchema,
|
||||
enabled: z.boolean(),
|
||||
cursor: z.string().nullable(),
|
||||
lastRunAt: z.string().nullable(),
|
||||
lastSuccessAt: z.string().nullable(),
|
||||
createdBy: z.string(),
|
||||
createdAt: z.string(),
|
||||
updatedAt: z.string(),
|
||||
})
|
||||
|
||||
export const dataDrainSchema = z.intersection(
|
||||
drainCommonResponseFieldsSchema,
|
||||
drainDestinationResponseSchema
|
||||
)
|
||||
|
||||
export type DataDrain = z.output<typeof dataDrainSchema>
|
||||
export type CreateDataDrainBody = z.input<typeof createDataDrainBodySchema>
|
||||
export type UpdateDataDrainBody = z.input<typeof updateDataDrainBodySchema>
|
||||
|
||||
export const dataDrainListResponseSchema = z.object({
|
||||
drains: z.array(dataDrainSchema),
|
||||
})
|
||||
|
||||
export const dataDrainResponseSchema = z.object({
|
||||
drain: dataDrainSchema,
|
||||
})
|
||||
|
||||
export const dataDrainRunSchema = z.object({
|
||||
id: z.string(),
|
||||
drainId: z.string(),
|
||||
status: dataDrainRunStatusSchema,
|
||||
trigger: dataDrainRunTriggerSchema,
|
||||
startedAt: z.string(),
|
||||
finishedAt: z.string().nullable(),
|
||||
rowsExported: z.number().int(),
|
||||
bytesWritten: z.number().int(),
|
||||
cursorBefore: z.string().nullable(),
|
||||
cursorAfter: z.string().nullable(),
|
||||
error: z.string().nullable(),
|
||||
locators: z.array(z.string()),
|
||||
})
|
||||
|
||||
export type DataDrainRun = z.output<typeof dataDrainRunSchema>
|
||||
|
||||
export const dataDrainRunListResponseSchema = z.object({
|
||||
runs: z.array(dataDrainRunSchema),
|
||||
})
|
||||
|
||||
export const runDataDrainResponseSchema = z.object({
|
||||
jobId: z.string(),
|
||||
})
|
||||
|
||||
export const testDataDrainResponseSchema = z.object({
|
||||
ok: z.literal(true),
|
||||
})
|
||||
|
||||
export const listDataDrainsContract = defineRouteContract({
|
||||
method: 'GET',
|
||||
path: '/api/organizations/[id]/data-drains',
|
||||
params: dataDrainOrgParamsSchema,
|
||||
response: { mode: 'json', schema: dataDrainListResponseSchema },
|
||||
})
|
||||
|
||||
export const createDataDrainContract = defineRouteContract({
|
||||
method: 'POST',
|
||||
path: '/api/organizations/[id]/data-drains',
|
||||
params: dataDrainOrgParamsSchema,
|
||||
body: createDataDrainBodySchema,
|
||||
response: { mode: 'json', schema: dataDrainResponseSchema },
|
||||
})
|
||||
|
||||
export const getDataDrainContract = defineRouteContract({
|
||||
method: 'GET',
|
||||
path: '/api/organizations/[id]/data-drains/[drainId]',
|
||||
params: dataDrainParamsSchema,
|
||||
response: { mode: 'json', schema: dataDrainResponseSchema },
|
||||
})
|
||||
|
||||
export const updateDataDrainContract = defineRouteContract({
|
||||
method: 'PUT',
|
||||
path: '/api/organizations/[id]/data-drains/[drainId]',
|
||||
params: dataDrainParamsSchema,
|
||||
body: updateDataDrainBodySchema,
|
||||
response: { mode: 'json', schema: dataDrainResponseSchema },
|
||||
})
|
||||
|
||||
export const deleteDataDrainContract = defineRouteContract({
|
||||
method: 'DELETE',
|
||||
path: '/api/organizations/[id]/data-drains/[drainId]',
|
||||
params: dataDrainParamsSchema,
|
||||
response: { mode: 'json', schema: z.object({ success: z.literal(true) }) },
|
||||
})
|
||||
|
||||
export const runDataDrainContract = defineRouteContract({
|
||||
method: 'POST',
|
||||
path: '/api/organizations/[id]/data-drains/[drainId]/run',
|
||||
params: dataDrainParamsSchema,
|
||||
response: { mode: 'json', schema: runDataDrainResponseSchema },
|
||||
})
|
||||
|
||||
export const testDataDrainContract = defineRouteContract({
|
||||
method: 'POST',
|
||||
path: '/api/organizations/[id]/data-drains/[drainId]/test',
|
||||
params: dataDrainParamsSchema,
|
||||
response: { mode: 'json', schema: testDataDrainResponseSchema },
|
||||
})
|
||||
|
||||
export const listDataDrainRunsContract = defineRouteContract({
|
||||
method: 'GET',
|
||||
path: '/api/organizations/[id]/data-drains/[drainId]/runs',
|
||||
params: dataDrainParamsSchema,
|
||||
query: z
|
||||
.object({
|
||||
limit: z
|
||||
.preprocess(
|
||||
(v) => (typeof v === 'string' ? Number.parseInt(v, 10) : v),
|
||||
z.number().int().min(1).max(200)
|
||||
)
|
||||
.optional(),
|
||||
})
|
||||
.optional(),
|
||||
response: { mode: 'json', schema: dataDrainRunListResponseSchema },
|
||||
})
|
||||
@@ -24,6 +24,7 @@ const JOB_TYPE_TO_TASK_ID: Record<JobType, string> = {
|
||||
'cleanup-logs': 'cleanup-logs',
|
||||
'cleanup-soft-deletes': 'cleanup-soft-deletes',
|
||||
'cleanup-tasks': 'cleanup-tasks',
|
||||
'run-data-drain': 'run-data-drain',
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -29,6 +29,7 @@ export type JobType =
|
||||
| 'cleanup-logs'
|
||||
| 'cleanup-soft-deletes'
|
||||
| 'cleanup-tasks'
|
||||
| 'run-data-drain'
|
||||
|
||||
export type AsyncExecutionCorrelationSource = 'workflow' | 'schedule' | 'webhook'
|
||||
|
||||
|
||||
@@ -355,6 +355,7 @@ export const env = createEnv({
|
||||
WHITELABELING_ENABLED: z.boolean().optional(), // Enable whitelabeling on self-hosted (bypasses hosted requirements)
|
||||
AUDIT_LOGS_ENABLED: z.boolean().optional(), // Enable audit logs on self-hosted (bypasses hosted requirements)
|
||||
DATA_RETENTION_ENABLED: z.boolean().optional(), // Enable data retention settings on self-hosted (bypasses hosted requirements)
|
||||
DATA_DRAINS_ENABLED: z.boolean().optional(), // Enable data drains on self-hosted (bypasses hosted requirements)
|
||||
|
||||
// Organizations - for self-hosted deployments
|
||||
ORGANIZATIONS_ENABLED: z.boolean().optional(), // Enable organizations on self-hosted (bypasses plan requirements)
|
||||
@@ -451,6 +452,7 @@ export const env = createEnv({
|
||||
NEXT_PUBLIC_WHITELABELING_ENABLED: z.boolean().optional(), // Enable whitelabeling on self-hosted (bypasses hosted requirements)
|
||||
NEXT_PUBLIC_AUDIT_LOGS_ENABLED: z.boolean().optional(), // Enable audit logs on self-hosted (bypasses hosted requirements)
|
||||
NEXT_PUBLIC_DATA_RETENTION_ENABLED: z.boolean().optional(), // Enable data retention settings on self-hosted (bypasses hosted requirements)
|
||||
NEXT_PUBLIC_DATA_DRAINS_ENABLED: z.boolean().optional(), // Enable data drains on self-hosted (bypasses hosted requirements)
|
||||
NEXT_PUBLIC_ORGANIZATIONS_ENABLED: z.boolean().optional(), // Enable organizations on self-hosted (bypasses plan requirements)
|
||||
NEXT_PUBLIC_DISABLE_INVITATIONS: z.boolean().optional(), // Disable workspace invitations globally (for self-hosted deployments)
|
||||
NEXT_PUBLIC_DISABLE_PUBLIC_API: z.boolean().optional(), // Disable public API access UI toggle globally
|
||||
@@ -488,6 +490,7 @@ export const env = createEnv({
|
||||
NEXT_PUBLIC_WHITELABELING_ENABLED: process.env.NEXT_PUBLIC_WHITELABELING_ENABLED,
|
||||
NEXT_PUBLIC_AUDIT_LOGS_ENABLED: process.env.NEXT_PUBLIC_AUDIT_LOGS_ENABLED,
|
||||
NEXT_PUBLIC_DATA_RETENTION_ENABLED: process.env.NEXT_PUBLIC_DATA_RETENTION_ENABLED,
|
||||
NEXT_PUBLIC_DATA_DRAINS_ENABLED: process.env.NEXT_PUBLIC_DATA_DRAINS_ENABLED,
|
||||
NEXT_PUBLIC_ORGANIZATIONS_ENABLED: process.env.NEXT_PUBLIC_ORGANIZATIONS_ENABLED,
|
||||
NEXT_PUBLIC_DISABLE_INVITATIONS: process.env.NEXT_PUBLIC_DISABLE_INVITATIONS,
|
||||
NEXT_PUBLIC_DISABLE_PUBLIC_API: process.env.NEXT_PUBLIC_DISABLE_PUBLIC_API,
|
||||
|
||||
@@ -129,6 +129,18 @@ export const isWhitelabelingEnabled = isTruthy(env.WHITELABELING_ENABLED)
|
||||
*/
|
||||
export const isAuditLogsEnabled = isTruthy(env.AUDIT_LOGS_ENABLED)
|
||||
|
||||
/**
|
||||
* Is data retention enabled via env var override
|
||||
* This bypasses hosted requirements for self-hosted deployments
|
||||
*/
|
||||
export const isDataRetentionEnabled = isTruthy(env.DATA_RETENTION_ENABLED)
|
||||
|
||||
/**
|
||||
* Is data drains enabled via env var override
|
||||
* This bypasses hosted requirements for self-hosted deployments
|
||||
*/
|
||||
export const isDataDrainsEnabled = isTruthy(env.DATA_DRAINS_ENABLED)
|
||||
|
||||
/**
|
||||
* Is E2B enabled for remote code execution
|
||||
*/
|
||||
|
||||
@@ -192,6 +192,7 @@ export interface SecureFetchOptions {
|
||||
timeout?: number
|
||||
maxRedirects?: number
|
||||
maxResponseBytes?: number
|
||||
signal?: AbortSignal
|
||||
}
|
||||
|
||||
export class SecureFetchHeaders {
|
||||
@@ -310,7 +311,7 @@ export async function secureFetchWithPinnedIP(
|
||||
validateUrlWithDNS(redirectUrl, 'redirectUrl', { allowHttp: options.allowHttp })
|
||||
.then((validation) => {
|
||||
if (!validation.isValid) {
|
||||
reject(new Error(`Redirect blocked: ${validation.error}`))
|
||||
settledReject(new Error(`Redirect blocked: ${validation.error}`))
|
||||
return
|
||||
}
|
||||
return secureFetchWithPinnedIP(
|
||||
@@ -321,15 +322,15 @@ export async function secureFetchWithPinnedIP(
|
||||
)
|
||||
})
|
||||
.then((response) => {
|
||||
if (response) resolve(response)
|
||||
if (response) settledResolve(response)
|
||||
})
|
||||
.catch(reject)
|
||||
.catch(settledReject)
|
||||
return
|
||||
}
|
||||
|
||||
if (isRedirectStatus(statusCode) && location && redirectCount >= maxRedirects) {
|
||||
res.resume()
|
||||
reject(new Error(`Too many redirects (max: ${maxRedirects})`))
|
||||
settledReject(new Error(`Too many redirects (max: ${maxRedirects})`))
|
||||
return
|
||||
}
|
||||
|
||||
@@ -355,7 +356,7 @@ export async function secureFetchWithPinnedIP(
|
||||
})
|
||||
|
||||
res.on('error', (error) => {
|
||||
reject(error)
|
||||
settledReject(error)
|
||||
})
|
||||
|
||||
res.on('end', () => {
|
||||
@@ -371,7 +372,7 @@ export async function secureFetchWithPinnedIP(
|
||||
}
|
||||
}
|
||||
|
||||
resolve({
|
||||
settledResolve({
|
||||
ok: statusCode >= 200 && statusCode < 300,
|
||||
status: statusCode,
|
||||
statusText: res.statusMessage || '',
|
||||
@@ -387,15 +388,44 @@ export async function secureFetchWithPinnedIP(
|
||||
})
|
||||
})
|
||||
|
||||
let onAbort: (() => void) | null = null
|
||||
const cleanupAbort = () => {
|
||||
if (onAbort && options.signal) {
|
||||
options.signal.removeEventListener('abort', onAbort)
|
||||
onAbort = null
|
||||
}
|
||||
}
|
||||
const settledResolve: typeof resolve = (value) => {
|
||||
cleanupAbort()
|
||||
resolve(value)
|
||||
}
|
||||
const settledReject: typeof reject = (reason) => {
|
||||
cleanupAbort()
|
||||
reject(reason)
|
||||
}
|
||||
|
||||
req.on('error', (error) => {
|
||||
reject(error)
|
||||
settledReject(error)
|
||||
})
|
||||
|
||||
req.on('timeout', () => {
|
||||
req.destroy()
|
||||
reject(new Error(`Request timed out after ${requestOptions.timeout}ms`))
|
||||
settledReject(new Error(`Request timed out after ${requestOptions.timeout}ms`))
|
||||
})
|
||||
|
||||
if (options.signal) {
|
||||
if (options.signal.aborted) {
|
||||
req.destroy()
|
||||
settledReject(options.signal.reason ?? new Error('Aborted'))
|
||||
return
|
||||
}
|
||||
onAbort = () => {
|
||||
req.destroy()
|
||||
settledReject(options.signal?.reason ?? new Error('Aborted'))
|
||||
}
|
||||
options.signal.addEventListener('abort', onAbort, { once: true })
|
||||
}
|
||||
|
||||
if (options.body) {
|
||||
req.write(options.body)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrains, member } from '@sim/db/schema'
|
||||
import { and, eq } from 'drizzle-orm'
|
||||
import { NextResponse } from 'next/server'
|
||||
import { getSession } from '@/lib/auth'
|
||||
import { isOrganizationOnEnterprisePlan } from '@/lib/billing/core/subscription'
|
||||
import { isBillingEnabled, isDataDrainsEnabled } from '@/lib/core/config/feature-flags'
|
||||
|
||||
export interface DrainAccessSession {
|
||||
user: {
|
||||
id: string
|
||||
name?: string | null
|
||||
email?: string | null
|
||||
}
|
||||
membership: {
|
||||
role: string
|
||||
}
|
||||
}
|
||||
|
||||
export type DrainAccessResult =
|
||||
| { ok: true; session: DrainAccessSession }
|
||||
| { ok: false; response: NextResponse }
|
||||
|
||||
/**
|
||||
* Auth + membership + role + enterprise-plan gate shared by every data-drain
|
||||
* route. Owner/admin role is required for reads as well as writes since drain
|
||||
* configs expose customer bucket names and webhook URLs. On Sim Cloud the
|
||||
* gate is the Enterprise plan; on self-hosted it's `DATA_DRAINS_ENABLED`,
|
||||
* which 404s when unset so a newer image doesn't silently expose drains.
|
||||
*/
|
||||
export async function authorizeDrainAccess(
|
||||
organizationId: string,
|
||||
options: { requireMutating: boolean }
|
||||
): Promise<DrainAccessResult> {
|
||||
const session = await getSession()
|
||||
if (!session?.user?.id) {
|
||||
return { ok: false, response: NextResponse.json({ error: 'Unauthorized' }, { status: 401 }) }
|
||||
}
|
||||
|
||||
const [memberEntry] = await db
|
||||
.select({ role: member.role })
|
||||
.from(member)
|
||||
.where(and(eq(member.organizationId, organizationId), eq(member.userId, session.user.id)))
|
||||
.limit(1)
|
||||
|
||||
if (!memberEntry) {
|
||||
return {
|
||||
ok: false,
|
||||
response: NextResponse.json(
|
||||
{ error: 'Forbidden - Not a member of this organization' },
|
||||
{ status: 403 }
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
if (!isBillingEnabled && !isDataDrainsEnabled) {
|
||||
return {
|
||||
ok: false,
|
||||
response: NextResponse.json(
|
||||
{ error: 'Data Drains are not enabled on this deployment' },
|
||||
{ status: 404 }
|
||||
),
|
||||
}
|
||||
}
|
||||
if (isBillingEnabled) {
|
||||
const hasEnterprise = await isOrganizationOnEnterprisePlan(organizationId)
|
||||
if (!hasEnterprise) {
|
||||
return {
|
||||
ok: false,
|
||||
response: NextResponse.json(
|
||||
{ error: 'Data Drains are available on Enterprise plans only' },
|
||||
{ status: 403 }
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
if (memberEntry.role !== 'owner' && memberEntry.role !== 'admin') {
|
||||
return {
|
||||
ok: false,
|
||||
response: NextResponse.json(
|
||||
{
|
||||
error: options.requireMutating
|
||||
? 'Forbidden - Only organization owners and admins can manage data drains'
|
||||
: 'Forbidden - Only organization owners and admins can view data drains',
|
||||
},
|
||||
{ status: 403 }
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
session: {
|
||||
user: {
|
||||
id: session.user.id,
|
||||
name: session.user.name ?? null,
|
||||
email: session.user.email ?? null,
|
||||
},
|
||||
membership: { role: memberEntry.role },
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
export async function loadDrain(organizationId: string, drainId: string) {
|
||||
const [drain] = await db
|
||||
.select()
|
||||
.from(dataDrains)
|
||||
.where(and(eq(dataDrains.id, drainId), eq(dataDrains.organizationId, organizationId)))
|
||||
.limit(1)
|
||||
return drain ?? null
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
import { s3Destination } from '@/lib/data-drains/destinations/s3'
|
||||
import { webhookDestination } from '@/lib/data-drains/destinations/webhook'
|
||||
import type { DestinationType, DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
export const DESTINATION_REGISTRY = {
|
||||
s3: s3Destination,
|
||||
webhook: webhookDestination,
|
||||
} as const satisfies Record<DestinationType, DrainDestination>
|
||||
|
||||
export function getDestination(type: DestinationType): DrainDestination {
|
||||
return DESTINATION_REGISTRY[type]
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockSend, mockDestroy, S3ClientCtor, PutObjectCommandCtor, DeleteObjectCommandCtor } =
|
||||
vi.hoisted(() => {
|
||||
const mockSend = vi.fn(async () => ({}))
|
||||
const mockDestroy = vi.fn()
|
||||
return {
|
||||
mockSend,
|
||||
mockDestroy,
|
||||
S3ClientCtor: vi.fn(() => ({ send: mockSend, destroy: mockDestroy })),
|
||||
PutObjectCommandCtor: vi.fn((args: unknown) => ({ __cmd: 'put', args })),
|
||||
DeleteObjectCommandCtor: vi.fn((args: unknown) => ({ __cmd: 'delete', args })),
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('@aws-sdk/client-s3', () => ({
|
||||
S3Client: S3ClientCtor,
|
||||
PutObjectCommand: PutObjectCommandCtor,
|
||||
DeleteObjectCommand: DeleteObjectCommandCtor,
|
||||
}))
|
||||
|
||||
import { s3Destination } from '@/lib/data-drains/destinations/s3'
|
||||
|
||||
const config = {
|
||||
bucket: 'my-bucket',
|
||||
region: 'us-east-1',
|
||||
prefix: 'sim/',
|
||||
}
|
||||
const credentials = { accessKeyId: 'AKID', secretAccessKey: 'SECRET' }
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
})
|
||||
|
||||
describe('s3Destination openSession', () => {
|
||||
it('reuses one S3Client across multiple deliveries and destroys on close', async () => {
|
||||
const session = s3Destination.openSession({ config, credentials })
|
||||
expect(S3ClientCtor).toHaveBeenCalledTimes(1)
|
||||
|
||||
const body = Buffer.from('row\n', 'utf8')
|
||||
const meta = (sequence: number) => ({
|
||||
drainId: 'd1',
|
||||
runId: 'r1',
|
||||
source: 'workflow_logs' as const,
|
||||
sequence,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
})
|
||||
const signal = new AbortController().signal
|
||||
|
||||
const res1 = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal,
|
||||
})
|
||||
const res2 = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(1),
|
||||
signal,
|
||||
})
|
||||
|
||||
expect(S3ClientCtor).toHaveBeenCalledTimes(1)
|
||||
expect(mockSend).toHaveBeenCalledTimes(2)
|
||||
|
||||
expect(res1.locator).toMatch(
|
||||
/^s3:\/\/my-bucket\/sim\/workflow_logs\/d1\/\d{4}\/\d{2}\/\d{2}\/r1-00000\.ndjson$/
|
||||
)
|
||||
expect(res2.locator).toMatch(/r1-00001\.ndjson$/)
|
||||
|
||||
const putArgs = (PutObjectCommandCtor.mock.calls[0]?.[0] ?? {}) as Record<string, unknown>
|
||||
expect(putArgs.Bucket).toBe('my-bucket')
|
||||
expect(putArgs.Body).toBe(body)
|
||||
expect(putArgs.ContentType).toBe('application/x-ndjson')
|
||||
expect((putArgs.Metadata as Record<string, string>)['sim-drain-id']).toBe('d1')
|
||||
expect((putArgs.Metadata as Record<string, string>)['sim-sequence']).toBe('0')
|
||||
|
||||
await session.close()
|
||||
expect(mockDestroy).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('omits the prefix segment when prefix is empty', async () => {
|
||||
const session = s3Destination.openSession({
|
||||
config: { bucket: 'b', region: 'us-east-1' },
|
||||
credentials,
|
||||
})
|
||||
const result = await session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'audit_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(result.locator).toMatch(
|
||||
/^s3:\/\/b\/audit_logs\/d\/\d{4}\/\d{2}\/\d{2}\/r-00000\.ndjson$/
|
||||
)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('surfaces AWS error code in delivery errors', async () => {
|
||||
mockSend.mockRejectedValueOnce(
|
||||
Object.assign(new Error('Access Denied'), {
|
||||
name: 'AccessDenied',
|
||||
$metadata: { httpStatusCode: 403, requestId: 'req-1' },
|
||||
})
|
||||
)
|
||||
const session = s3Destination.openSession({ config, credentials })
|
||||
await expect(
|
||||
session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'audit_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/AccessDenied 403/)
|
||||
await session.close()
|
||||
})
|
||||
})
|
||||
|
||||
describe('s3Destination test()', () => {
|
||||
it('writes a probe object then attempts cleanup', async () => {
|
||||
await s3Destination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(PutObjectCommandCtor).toHaveBeenCalled()
|
||||
expect(DeleteObjectCommandCtor).toHaveBeenCalled()
|
||||
expect(mockDestroy).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('still returns success when cleanup delete fails', async () => {
|
||||
mockSend
|
||||
.mockResolvedValueOnce({}) // put probe
|
||||
.mockRejectedValueOnce(new Error('no delete perms')) // cleanup
|
||||
await expect(
|
||||
s3Destination.test!({ config, credentials, signal: new AbortController().signal })
|
||||
).resolves.toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,223 @@
|
||||
import {
|
||||
DeleteObjectCommand,
|
||||
PutObjectCommand,
|
||||
S3Client,
|
||||
type S3ServiceException,
|
||||
} from '@aws-sdk/client-s3'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { generateShortId } from '@sim/utils/id'
|
||||
import { z } from 'zod'
|
||||
import { validateExternalUrl } from '@/lib/core/security/input-validation'
|
||||
import { validateUrlWithDNS } from '@/lib/core/security/input-validation.server'
|
||||
import type { DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainS3Destination')
|
||||
|
||||
const s3ConfigSchema = z.object({
|
||||
bucket: z.string().min(1, 'bucket is required').max(255),
|
||||
region: z.string().min(1, 'region is required').max(64),
|
||||
/** Optional prefix; trailing slash is added automatically when assembling keys. */
|
||||
prefix: z.string().max(512).optional(),
|
||||
/**
|
||||
* Optional override for non-AWS S3-compatible providers (MinIO, R2, GCS interop, etc.).
|
||||
* SSRF-validated: HTTPS-only, must not resolve syntactically to a private,
|
||||
* loopback, or cloud-metadata address. The AWS SDK will issue requests to
|
||||
* this host, so we reject internal targets at the schema boundary.
|
||||
*/
|
||||
endpoint: z
|
||||
.string()
|
||||
.url()
|
||||
.refine((value) => validateExternalUrl(value, 'endpoint').isValid, {
|
||||
message: 'endpoint must be HTTPS and not point at a private, loopback, or metadata address',
|
||||
})
|
||||
.optional(),
|
||||
/**
|
||||
* Force path-style addressing. Set `true` for MinIO / Ceph RGW; defaults
|
||||
* to `false` for AWS S3 and Cloudflare R2.
|
||||
*/
|
||||
forcePathStyle: z.boolean().optional(),
|
||||
})
|
||||
|
||||
const s3CredentialsSchema = z.object({
|
||||
accessKeyId: z.string().min(1, 'accessKeyId is required'),
|
||||
secretAccessKey: z.string().min(1, 'secretAccessKey is required'),
|
||||
})
|
||||
|
||||
export type S3DestinationConfig = z.infer<typeof s3ConfigSchema>
|
||||
export type S3DestinationCredentials = z.infer<typeof s3CredentialsSchema>
|
||||
|
||||
function buildClient(config: S3DestinationConfig, credentials: S3DestinationCredentials): S3Client {
|
||||
return new S3Client({
|
||||
region: config.region,
|
||||
credentials: {
|
||||
accessKeyId: credentials.accessKeyId,
|
||||
secretAccessKey: credentials.secretAccessKey,
|
||||
},
|
||||
endpoint: config.endpoint,
|
||||
forcePathStyle: config.forcePathStyle ?? false,
|
||||
})
|
||||
}
|
||||
|
||||
function normalizePrefix(raw: string | undefined): string {
|
||||
if (!raw) return ''
|
||||
// S3 keys cannot start with `/` (creates an empty-name segment); also
|
||||
// collapse trailing slashes so the joiner produces a single boundary.
|
||||
const trimmed = raw.replace(/^\/+/, '').replace(/\/+$/, '')
|
||||
return trimmed.length === 0 ? '' : `${trimmed}/`
|
||||
}
|
||||
|
||||
function buildKey(
|
||||
config: S3DestinationConfig,
|
||||
metadata: {
|
||||
drainId: string
|
||||
runId: string
|
||||
source: string
|
||||
sequence: number
|
||||
runStartedAt: Date
|
||||
}
|
||||
): string {
|
||||
// Partition by the run's start time so all chunks from one run share a
|
||||
// single date prefix even if delivery crosses a midnight boundary.
|
||||
const partition = metadata.runStartedAt
|
||||
const yyyy = partition.getUTCFullYear().toString().padStart(4, '0')
|
||||
const mm = (partition.getUTCMonth() + 1).toString().padStart(2, '0')
|
||||
const dd = partition.getUTCDate().toString().padStart(2, '0')
|
||||
const seq = metadata.sequence.toString().padStart(5, '0')
|
||||
const prefix = normalizePrefix(config.prefix)
|
||||
return `${prefix}${metadata.source}/${metadata.drainId}/${yyyy}/${mm}/${dd}/${metadata.runId}-${seq}.ndjson`
|
||||
}
|
||||
|
||||
function isS3ServiceException(error: unknown): error is S3ServiceException {
|
||||
return (
|
||||
typeof error === 'object' &&
|
||||
error !== null &&
|
||||
'$metadata' in error &&
|
||||
typeof (error as { name?: unknown }).name === 'string'
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the optional custom endpoint and confirms it does not point at a
|
||||
* private, loopback, or cloud-metadata address. The schema-level
|
||||
* `validateExternalUrl` only catches IP literals, so a hostname like
|
||||
* `evil.example.com` resolving to `169.254.169.254` would slip past it; the
|
||||
* AWS SDK then resolves the host itself, bypassing the SSRF guard.
|
||||
*/
|
||||
async function assertEndpointIsPublic(endpoint: string | undefined): Promise<void> {
|
||||
if (!endpoint) return
|
||||
const result = await validateUrlWithDNS(endpoint, 'endpoint')
|
||||
if (!result.isValid) {
|
||||
throw new Error(result.error ?? 'S3 endpoint failed SSRF validation')
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Surfaces actionable S3 SDK error codes (`AccessDenied`, `NoSuchBucket`,
|
||||
* `InvalidAccessKeyId`, `SignatureDoesNotMatch`, ...) and preserves the
|
||||
* original error as `cause` so callers can still branch on `code`/`$metadata`.
|
||||
*/
|
||||
async function withS3ErrorContext<T>(action: string, fn: () => Promise<T>): Promise<T> {
|
||||
try {
|
||||
return await fn()
|
||||
} catch (error) {
|
||||
if (isS3ServiceException(error)) {
|
||||
const code = error.name
|
||||
const status = error.$metadata?.httpStatusCode
|
||||
const requestId = error.$metadata?.requestId
|
||||
logger.warn('S3 operation failed', { action, code, status, requestId })
|
||||
// Preserve the original SDK error as `cause` so callers can still
|
||||
// branch on `code` / `$metadata` while getting an actionable message.
|
||||
throw new Error(
|
||||
`S3 ${action} failed (${code}${status ? ` ${status}` : ''}): ${error.message}`,
|
||||
{ cause: error }
|
||||
)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
export const s3Destination: DrainDestination<S3DestinationConfig, S3DestinationCredentials> = {
|
||||
type: 's3',
|
||||
displayName: 'Amazon S3',
|
||||
configSchema: s3ConfigSchema,
|
||||
credentialsSchema: s3CredentialsSchema,
|
||||
|
||||
async test({ config, credentials, signal }) {
|
||||
await assertEndpointIsPublic(config.endpoint)
|
||||
const client = buildClient(config, credentials)
|
||||
// Probe with a real write so read-only creds and write-only IAM policies
|
||||
// surface here instead of at the first scheduled run.
|
||||
const probeKey = `${normalizePrefix(config.prefix)}.sim-drain-write-probe/${generateShortId(12)}`
|
||||
try {
|
||||
await withS3ErrorContext('test-put', () =>
|
||||
client.send(
|
||||
new PutObjectCommand({
|
||||
Bucket: config.bucket,
|
||||
Key: probeKey,
|
||||
Body: Buffer.alloc(0),
|
||||
ContentType: 'application/octet-stream',
|
||||
ServerSideEncryption: 'AES256',
|
||||
}),
|
||||
{ abortSignal: signal }
|
||||
)
|
||||
)
|
||||
// Best-effort cleanup; ignore failures so a missing s3:DeleteObject
|
||||
// doesn't fail the test (write was already proven).
|
||||
try {
|
||||
await client.send(new DeleteObjectCommand({ Bucket: config.bucket, Key: probeKey }), {
|
||||
abortSignal: signal,
|
||||
})
|
||||
} catch (cleanupError) {
|
||||
logger.debug('S3 test write probe cleanup failed (non-fatal)', {
|
||||
bucket: config.bucket,
|
||||
key: probeKey,
|
||||
error: cleanupError,
|
||||
})
|
||||
}
|
||||
} finally {
|
||||
client.destroy()
|
||||
}
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
const client = buildClient(config, credentials)
|
||||
// Cache the DNS-aware endpoint check across all chunks in a run so we
|
||||
// pay the lookup once. The SDK creates its own connections, so we can't
|
||||
// pin the IP — but doing the check before any S3 call still rejects
|
||||
// hostnames that resolve to internal targets at the start of the run.
|
||||
// Lazy-init avoids an unhandled rejection if the source yields no chunks
|
||||
// and `deliver` never runs (e.g., a drain with nothing new to export).
|
||||
let endpointCheck: Promise<void> | null = null
|
||||
return {
|
||||
async deliver({ body, contentType, metadata, signal }) {
|
||||
if (endpointCheck === null) endpointCheck = assertEndpointIsPublic(config.endpoint)
|
||||
await endpointCheck
|
||||
const key = buildKey(config, metadata)
|
||||
await withS3ErrorContext('put-object', () =>
|
||||
client.send(
|
||||
new PutObjectCommand({
|
||||
Bucket: config.bucket,
|
||||
Key: key,
|
||||
Body: body,
|
||||
ContentType: contentType,
|
||||
ServerSideEncryption: 'AES256',
|
||||
Metadata: {
|
||||
'sim-drain-id': metadata.drainId,
|
||||
'sim-run-id': metadata.runId,
|
||||
'sim-source': metadata.source,
|
||||
'sim-sequence': metadata.sequence.toString(),
|
||||
'sim-row-count': metadata.rowCount.toString(),
|
||||
},
|
||||
}),
|
||||
{ abortSignal: signal }
|
||||
)
|
||||
)
|
||||
logger.debug('S3 chunk delivered', { bucket: config.bucket, key, bytes: body.byteLength })
|
||||
return { locator: `s3://${config.bucket}/${key}` }
|
||||
},
|
||||
async close() {
|
||||
client.destroy()
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { createHmac } from 'node:crypto'
|
||||
import { inputValidationMock, inputValidationMockFns } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
vi.mock('@/lib/core/security/input-validation.server', () => inputValidationMock)
|
||||
|
||||
import { webhookDestination } from '@/lib/data-drains/destinations/webhook'
|
||||
|
||||
const config = { url: 'https://example.com/hook' }
|
||||
const credentials = { signingSecret: 'super-secret-key' }
|
||||
const metadata = {
|
||||
drainId: 'd1',
|
||||
runId: 'r1',
|
||||
source: 'workflow_logs' as const,
|
||||
sequence: 3,
|
||||
rowCount: 5,
|
||||
}
|
||||
|
||||
function mockPinnedFetchOnce(response: { ok: boolean; status: number; headers?: Headers }) {
|
||||
inputValidationMockFns.mockSecureFetchWithPinnedIP.mockResolvedValueOnce({
|
||||
ok: response.ok,
|
||||
status: response.status,
|
||||
statusText: '',
|
||||
headers: response.headers ?? new Headers(),
|
||||
text: async () => '',
|
||||
json: async () => ({}),
|
||||
arrayBuffer: async () => new ArrayBuffer(0),
|
||||
})
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
inputValidationMockFns.mockValidateUrlWithDNS.mockResolvedValue({
|
||||
isValid: true,
|
||||
resolvedIP: '93.184.216.34',
|
||||
originalHostname: 'example.com',
|
||||
})
|
||||
})
|
||||
|
||||
describe('webhookDestination openSession', () => {
|
||||
it('signs the body with HMAC-SHA256 over `<ts>.<body>`', async () => {
|
||||
mockPinnedFetchOnce({ ok: true, status: 200 })
|
||||
const session = webhookDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from('{"id":1}\n', 'utf8')
|
||||
|
||||
await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
const call = inputValidationMockFns.mockSecureFetchWithPinnedIP.mock.calls[0]
|
||||
const [calledUrl, pinnedIP, init] = call
|
||||
expect(calledUrl).toBe('https://example.com/hook')
|
||||
expect(pinnedIP).toBe('93.184.216.34')
|
||||
const headers = init.headers as Record<string, string>
|
||||
expect(headers['Content-Type']).toBe('application/x-ndjson')
|
||||
expect(headers['X-Sim-Drain-Id']).toBe('d1')
|
||||
expect(headers['X-Sim-Run-Id']).toBe('r1')
|
||||
expect(headers['X-Sim-Sequence']).toBe('3')
|
||||
expect(headers['Idempotency-Key']).toBe('r1-3')
|
||||
|
||||
const sig = headers['X-Sim-Signature']
|
||||
const tsPart = sig.match(/t=(\d+)/)![1]
|
||||
const v1Part = sig.match(/v1=([0-9a-f]+)/)![1]
|
||||
const expected = createHmac('sha256', credentials.signingSecret)
|
||||
.update(`${tsPart}.`)
|
||||
.update(body)
|
||||
.digest('hex')
|
||||
expect(v1Part).toBe(expected)
|
||||
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('retries on 5xx and succeeds', async () => {
|
||||
mockPinnedFetchOnce({ ok: false, status: 503 })
|
||||
mockPinnedFetchOnce({ ok: true, status: 200 })
|
||||
vi.spyOn(global, 'setTimeout').mockImplementation(((fn: () => void) => {
|
||||
fn()
|
||||
return 0 as unknown as NodeJS.Timeout
|
||||
}) as never)
|
||||
|
||||
const session = webhookDestination.openSession({ config, credentials })
|
||||
const result = await session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(result.locator).toContain('https://example.com/hook')
|
||||
expect(inputValidationMockFns.mockSecureFetchWithPinnedIP).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('does not retry on 4xx (other than 408/429)', async () => {
|
||||
mockPinnedFetchOnce({ ok: false, status: 401 })
|
||||
const session = webhookDestination.openSession({ config, credentials })
|
||||
await expect(
|
||||
session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/HTTP 401/)
|
||||
expect(inputValidationMockFns.mockSecureFetchWithPinnedIP).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('rejects when DNS resolves to a blocked IP', async () => {
|
||||
inputValidationMockFns.mockValidateUrlWithDNS.mockResolvedValueOnce({
|
||||
isValid: false,
|
||||
error: 'url resolves to a blocked IP address',
|
||||
})
|
||||
const session = webhookDestination.openSession({ config, credentials })
|
||||
await expect(
|
||||
session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/blocked IP/)
|
||||
expect(inputValidationMockFns.mockSecureFetchWithPinnedIP).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reuses the same pinned IP across deliveries (no DNS rebinding window)', async () => {
|
||||
mockPinnedFetchOnce({ ok: true, status: 200 })
|
||||
mockPinnedFetchOnce({ ok: true, status: 200 })
|
||||
const session = webhookDestination.openSession({ config, credentials })
|
||||
const signal = new AbortController().signal
|
||||
await session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata,
|
||||
signal,
|
||||
})
|
||||
await session.deliver({
|
||||
body: Buffer.from('y'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: { ...metadata, sequence: 4 },
|
||||
signal,
|
||||
})
|
||||
expect(inputValidationMockFns.mockValidateUrlWithDNS).toHaveBeenCalledTimes(1)
|
||||
const calls = inputValidationMockFns.mockSecureFetchWithPinnedIP.mock.calls
|
||||
expect(calls[0][1]).toBe('93.184.216.34')
|
||||
expect(calls[1][1]).toBe('93.184.216.34')
|
||||
})
|
||||
|
||||
it('rejects every header buildHeaders writes when reused as signatureHeader (drift guard)', async () => {
|
||||
mockPinnedFetchOnce({ ok: true, status: 200 })
|
||||
const session = webhookDestination.openSession({ config, credentials })
|
||||
await session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
const init = inputValidationMockFns.mockSecureFetchWithPinnedIP.mock.calls[0][2]
|
||||
const writtenHeaders = Object.keys(init.headers as Record<string, string>)
|
||||
|
||||
for (const name of writtenHeaders) {
|
||||
const result = webhookDestination.configSchema.safeParse({
|
||||
url: 'https://example.com/hook',
|
||||
signatureHeader: name,
|
||||
})
|
||||
expect(
|
||||
result.success,
|
||||
`expected signatureHeader="${name}" to be rejected (it is written by buildHeaders)`
|
||||
).toBe(false)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,283 @@
|
||||
import { createHmac } from 'node:crypto'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { z } from 'zod'
|
||||
import { validateExternalUrl } from '@/lib/core/security/input-validation'
|
||||
import {
|
||||
secureFetchWithPinnedIP,
|
||||
validateUrlWithDNS,
|
||||
} from '@/lib/core/security/input-validation.server'
|
||||
import type { DeliveryMetadata, DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainWebhookDestination')
|
||||
|
||||
/** Initial attempt + 3 retries — matches the documented 500ms/1s/2s backoff sequence. */
|
||||
const MAX_ATTEMPTS = 4
|
||||
const BASE_BACKOFF_MS = 500
|
||||
const MAX_BACKOFF_MS = 30_000
|
||||
const PER_ATTEMPT_TIMEOUT_MS = 30_000
|
||||
const SIGNATURE_VERSION = 'v1'
|
||||
const USER_AGENT = 'Sim-DataDrain/1.0'
|
||||
|
||||
/** Reserved header names that callers cannot reuse as the signature header. */
|
||||
const RESERVED_SIGNATURE_HEADER_NAMES = new Set([
|
||||
'authorization',
|
||||
'content-type',
|
||||
'user-agent',
|
||||
'idempotency-key',
|
||||
'x-sim-timestamp',
|
||||
'x-sim-signature-version',
|
||||
'x-sim-drain-id',
|
||||
'x-sim-run-id',
|
||||
'x-sim-source',
|
||||
'x-sim-sequence',
|
||||
'x-sim-row-count',
|
||||
'x-sim-probe',
|
||||
'x-sim-signature',
|
||||
])
|
||||
|
||||
/**
|
||||
* Resolves the URL's hostname and returns the validated public IP. Uses
|
||||
* `ipaddr.js` so all non-`unicast` ranges (RFC1918, loopback, CGNAT, multicast,
|
||||
* broadcast, IPv4-mapped IPv6, link-local, cloud metadata) are blocked
|
||||
* uniformly. The returned IP is then pinned to the underlying socket via
|
||||
* `secureFetchWithPinnedIP` to defeat DNS rebinding (TOCTOU) between the
|
||||
* validation lookup and the actual delivery.
|
||||
*/
|
||||
async function resolvePublicTarget(url: string): Promise<string> {
|
||||
const result = await validateUrlWithDNS(url, 'url')
|
||||
if (!result.isValid || !result.resolvedIP) {
|
||||
throw new Error(result.error ?? 'Webhook URL failed SSRF validation')
|
||||
}
|
||||
return result.resolvedIP
|
||||
}
|
||||
|
||||
const webhookConfigSchema = z.object({
|
||||
url: z
|
||||
.string()
|
||||
.url('url must be a valid URL')
|
||||
.refine((value) => validateExternalUrl(value, 'url').isValid, {
|
||||
message: 'url must be HTTPS and not point at a private, loopback, or metadata address',
|
||||
}),
|
||||
/** Optional custom header name for the signature (default: X-Sim-Signature). */
|
||||
signatureHeader: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(128)
|
||||
.refine((value) => !RESERVED_SIGNATURE_HEADER_NAMES.has(value.toLowerCase()), {
|
||||
message: 'signatureHeader cannot reuse a reserved Sim header name',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const webhookCredentialsSchema = z.object({
|
||||
/** Shared secret used for HMAC-SHA256 signing of the request body. */
|
||||
signingSecret: z.string().min(8, 'signingSecret must be at least 8 characters'),
|
||||
/** Optional bearer token sent as Authorization header. */
|
||||
bearerToken: z.string().min(1).optional(),
|
||||
})
|
||||
|
||||
export type WebhookDestinationConfig = z.infer<typeof webhookConfigSchema>
|
||||
export type WebhookDestinationCredentials = z.infer<typeof webhookCredentialsSchema>
|
||||
|
||||
/**
|
||||
* Stripe-style replay-resistant signature: signs `${unixSeconds}.${body}` and
|
||||
* emits `t=<unixSeconds>,v1=<hex(hmac)>`. Verifiers should reject signatures
|
||||
* older than ~5 minutes after also recomputing the HMAC over the same
|
||||
* concatenation, defending against captured-request replay attacks.
|
||||
*/
|
||||
function sign(body: Buffer, secret: string, timestamp: number): string {
|
||||
const hmac = createHmac('sha256', secret).update(`${timestamp}.`).update(body).digest('hex')
|
||||
return `t=${timestamp},${SIGNATURE_VERSION}=${hmac}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves after `ms` or as soon as `signal` aborts, whichever happens first.
|
||||
* The caller checks `signal.aborted` at the top of the next iteration to
|
||||
* surface the abort — keeping resolution side-effect-free here.
|
||||
*/
|
||||
function sleepUntilAborted(ms: number, signal: AbortSignal): Promise<void> {
|
||||
if (signal.aborted) return Promise.resolve()
|
||||
return new Promise((resolve) => {
|
||||
let timeoutId: ReturnType<typeof setTimeout>
|
||||
const onAbort = () => {
|
||||
clearTimeout(timeoutId)
|
||||
resolve()
|
||||
}
|
||||
timeoutId = setTimeout(() => {
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
resolve()
|
||||
}, ms)
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
})
|
||||
}
|
||||
|
||||
function backoffWithJitter(attempt: number, retryAfterMs?: number): number {
|
||||
if (retryAfterMs !== undefined) {
|
||||
// Floor at 500ms so a misbehaving server returning Retry-After: 0 cannot
|
||||
// pin us in a tight retry loop.
|
||||
return Math.min(Math.max(retryAfterMs, BASE_BACKOFF_MS), MAX_BACKOFF_MS)
|
||||
}
|
||||
const exponential = Math.min(BASE_BACKOFF_MS * 2 ** (attempt - 1), MAX_BACKOFF_MS)
|
||||
// ±20% jitter avoids thundering-herd alignment across drains.
|
||||
return exponential * (0.8 + Math.random() * 0.4)
|
||||
}
|
||||
|
||||
function parseRetryAfter(header: string | null): number | undefined {
|
||||
if (!header) return undefined
|
||||
const seconds = Number.parseInt(header, 10)
|
||||
if (!Number.isNaN(seconds) && seconds >= 0) return seconds * 1000
|
||||
const dateMs = Date.parse(header)
|
||||
if (!Number.isNaN(dateMs)) {
|
||||
const delta = dateMs - Date.now()
|
||||
return delta > 0 ? delta : 0
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function isRetryableStatus(status: number): boolean {
|
||||
return status === 408 || status === 429 || status >= 500
|
||||
}
|
||||
|
||||
function buildHeaders(input: {
|
||||
config: WebhookDestinationConfig
|
||||
credentials: WebhookDestinationCredentials
|
||||
body: Buffer
|
||||
contentType: string
|
||||
metadata?: DeliveryMetadata
|
||||
isProbe?: boolean
|
||||
}): Record<string, string> {
|
||||
const timestamp = Math.floor(Date.now() / 1000)
|
||||
const headers: Record<string, string> = {
|
||||
'Content-Type': input.contentType,
|
||||
'User-Agent': USER_AGENT,
|
||||
'X-Sim-Timestamp': timestamp.toString(),
|
||||
'X-Sim-Signature-Version': SIGNATURE_VERSION,
|
||||
[input.config.signatureHeader ?? 'X-Sim-Signature']: sign(
|
||||
input.body,
|
||||
input.credentials.signingSecret,
|
||||
timestamp
|
||||
),
|
||||
}
|
||||
if (input.metadata) {
|
||||
headers['X-Sim-Drain-Id'] = input.metadata.drainId
|
||||
headers['X-Sim-Run-Id'] = input.metadata.runId
|
||||
headers['X-Sim-Source'] = input.metadata.source
|
||||
headers['X-Sim-Sequence'] = input.metadata.sequence.toString()
|
||||
headers['X-Sim-Row-Count'] = input.metadata.rowCount.toString()
|
||||
// Lets idempotent receivers dedupe retried chunks server-side.
|
||||
headers['Idempotency-Key'] = `${input.metadata.runId}-${input.metadata.sequence}`
|
||||
}
|
||||
if (input.isProbe) {
|
||||
headers['X-Sim-Probe'] = '1'
|
||||
}
|
||||
if (input.credentials.bearerToken) {
|
||||
headers.Authorization = `Bearer ${input.credentials.bearerToken}`
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
export const webhookDestination: DrainDestination<
|
||||
WebhookDestinationConfig,
|
||||
WebhookDestinationCredentials
|
||||
> = {
|
||||
type: 'webhook',
|
||||
displayName: 'HTTPS Webhook',
|
||||
configSchema: webhookConfigSchema,
|
||||
credentialsSchema: webhookCredentialsSchema,
|
||||
|
||||
async test({ config, credentials, signal }) {
|
||||
const resolvedIP = await resolvePublicTarget(config.url)
|
||||
const probe = Buffer.from('{"sim":"connection-test"}\n', 'utf8')
|
||||
const headers = buildHeaders({
|
||||
config,
|
||||
credentials,
|
||||
body: probe,
|
||||
contentType: 'application/x-ndjson',
|
||||
isProbe: true,
|
||||
})
|
||||
const response = await secureFetchWithPinnedIP(config.url, resolvedIP, {
|
||||
method: 'POST',
|
||||
body: new Uint8Array(probe),
|
||||
headers,
|
||||
signal,
|
||||
timeout: PER_ATTEMPT_TIMEOUT_MS,
|
||||
})
|
||||
if (!response.ok) {
|
||||
throw new Error(`Webhook probe failed: HTTP ${response.status}`)
|
||||
}
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
let resolvedIP: string | null = null
|
||||
return {
|
||||
async deliver({ body, contentType, metadata, signal }) {
|
||||
// Resolve once per session — within a run we trust the result rather
|
||||
// than paying DNS on every chunk. Done lazily so a session that's
|
||||
// opened-and-immediately-closed pays no cost. The pinned IP is reused
|
||||
// across retries to defeat DNS rebinding (TOCTOU) attacks.
|
||||
if (resolvedIP === null) {
|
||||
resolvedIP = await resolvePublicTarget(config.url)
|
||||
}
|
||||
let lastError: unknown
|
||||
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
|
||||
if (signal.aborted) throw signal.reason ?? new Error('Aborted')
|
||||
// Re-build headers per attempt so the timestamp + signature are
|
||||
// fresh (otherwise long backoffs would push us outside the
|
||||
// verifier's skew window).
|
||||
const headers = buildHeaders({ config, credentials, body, contentType, metadata })
|
||||
let retryAfterMs: number | undefined
|
||||
let response: Awaited<ReturnType<typeof secureFetchWithPinnedIP>> | undefined
|
||||
try {
|
||||
response = await secureFetchWithPinnedIP(config.url, resolvedIP, {
|
||||
method: 'POST',
|
||||
body: new Uint8Array(body),
|
||||
headers,
|
||||
signal,
|
||||
timeout: PER_ATTEMPT_TIMEOUT_MS,
|
||||
})
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
logger.debug('Webhook delivery attempt failed', {
|
||||
url: config.url,
|
||||
attempt,
|
||||
error: toError(error).message,
|
||||
})
|
||||
}
|
||||
if (response) {
|
||||
if (response.ok) {
|
||||
const requestId =
|
||||
response.headers.get('x-request-id') ??
|
||||
response.headers.get('x-amzn-trace-id') ??
|
||||
null
|
||||
logger.debug('Webhook chunk delivered', {
|
||||
url: config.url,
|
||||
attempt,
|
||||
status: response.status,
|
||||
bytes: body.byteLength,
|
||||
})
|
||||
return {
|
||||
locator: requestId
|
||||
? `${config.url}#${metadata.runId}-${metadata.sequence}@${requestId}`
|
||||
: `${config.url}#${metadata.runId}-${metadata.sequence}`,
|
||||
}
|
||||
}
|
||||
if (!isRetryableStatus(response.status)) {
|
||||
// Non-retryable HTTP error: surface immediately without retrying.
|
||||
throw new Error(`Webhook responded with HTTP ${response.status}`)
|
||||
}
|
||||
lastError = new Error(`Webhook responded with HTTP ${response.status}`)
|
||||
retryAfterMs = parseRetryAfter(response.headers.get('retry-after'))
|
||||
}
|
||||
if (attempt < MAX_ATTEMPTS) {
|
||||
await sleepUntilAborted(backoffWithJitter(attempt, retryAfterMs), signal)
|
||||
}
|
||||
}
|
||||
throw lastError instanceof Error
|
||||
? lastError
|
||||
: new Error('Webhook delivery failed after retries')
|
||||
},
|
||||
async close() {},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { dbChainMock, dbChainMockFns, resetDbChainMock } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
vi.mock('@sim/db', () => dbChainMock)
|
||||
|
||||
const { mockIsEnterprise, mockEnqueue, mockGetJobQueue } = vi.hoisted(() => {
|
||||
const mockEnqueue = vi.fn(async () => 'job-id')
|
||||
return {
|
||||
mockIsEnterprise: vi.fn(),
|
||||
mockEnqueue,
|
||||
mockGetJobQueue: vi.fn(async () => ({ enqueue: mockEnqueue })),
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('@/lib/billing/core/subscription', () => ({
|
||||
isOrganizationOnEnterprisePlan: mockIsEnterprise,
|
||||
}))
|
||||
vi.mock('@/lib/core/async-jobs', () => ({ getJobQueue: mockGetJobQueue }))
|
||||
vi.mock('@/lib/core/config/feature-flags', () => ({ isBillingEnabled: true }))
|
||||
|
||||
import { dispatchDueDrains, reapOrphanedRuns } from '@/lib/data-drains/dispatcher'
|
||||
|
||||
function mockCandidates(rows: Array<{ id: string; organizationId: string }>) {
|
||||
// db.select().from().where() — override `from` so awaiting `.where(pred)`
|
||||
// resolves with the candidate rows.
|
||||
dbChainMockFns.from.mockReturnValueOnce({
|
||||
where: vi.fn().mockResolvedValueOnce(rows),
|
||||
} as never)
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
resetDbChainMock()
|
||||
})
|
||||
|
||||
describe('reapOrphanedRuns', () => {
|
||||
it('returns the count of rows updated to failed', async () => {
|
||||
dbChainMockFns.returning.mockResolvedValueOnce([{ id: 'run-1' }, { id: 'run-2' }])
|
||||
const result = await reapOrphanedRuns(new Date('2026-01-01T12:00:00.000Z'))
|
||||
expect(result).toEqual({ reaped: 2 })
|
||||
expect(dbChainMockFns.set).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ status: 'failed', error: expect.stringContaining('Orphaned') })
|
||||
)
|
||||
})
|
||||
|
||||
it('returns 0 when nothing is stuck', async () => {
|
||||
dbChainMockFns.returning.mockResolvedValueOnce([])
|
||||
expect(await reapOrphanedRuns()).toEqual({ reaped: 0 })
|
||||
})
|
||||
})
|
||||
|
||||
describe('dispatchDueDrains', () => {
|
||||
it('returns early when no candidates are due', async () => {
|
||||
dbChainMockFns.returning.mockResolvedValueOnce([]) // reaper
|
||||
mockCandidates([])
|
||||
|
||||
const result = await dispatchDueDrains()
|
||||
expect(result).toEqual({ candidates: 0, dispatched: 0, skipped: 0, reaped: 0 })
|
||||
expect(mockGetJobQueue).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('skips drains for orgs not on enterprise plan', async () => {
|
||||
dbChainMockFns.returning.mockResolvedValueOnce([]) // reaper
|
||||
mockCandidates([{ id: 'd1', organizationId: 'org-a' }])
|
||||
mockIsEnterprise.mockResolvedValueOnce(false)
|
||||
|
||||
const result = await dispatchDueDrains()
|
||||
expect(result).toMatchObject({ candidates: 1, dispatched: 0, skipped: 1 })
|
||||
expect(mockEnqueue).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('claims and enqueues a job per due drain', async () => {
|
||||
dbChainMockFns.returning
|
||||
.mockResolvedValueOnce([]) // reaper
|
||||
.mockResolvedValueOnce([{ id: 'd1' }]) // claim succeeds
|
||||
mockCandidates([{ id: 'd1', organizationId: 'org-a' }])
|
||||
mockIsEnterprise.mockResolvedValueOnce(true)
|
||||
|
||||
const result = await dispatchDueDrains()
|
||||
expect(result).toMatchObject({ candidates: 1, dispatched: 1, skipped: 0 })
|
||||
expect(mockEnqueue).toHaveBeenCalledWith(
|
||||
'run-data-drain',
|
||||
{ drainId: 'd1', trigger: 'cron' },
|
||||
{ concurrencyKey: 'data-drain:d1' }
|
||||
)
|
||||
})
|
||||
|
||||
it('does not enqueue when claim loses the race', async () => {
|
||||
dbChainMockFns.returning
|
||||
.mockResolvedValueOnce([]) // reaper
|
||||
.mockResolvedValueOnce([]) // claim returns nothing — lost the race
|
||||
mockCandidates([{ id: 'd1', organizationId: 'org-a' }])
|
||||
mockIsEnterprise.mockResolvedValueOnce(true)
|
||||
|
||||
const result = await dispatchDueDrains()
|
||||
expect(result.dispatched).toBe(0)
|
||||
expect(mockEnqueue).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('caches enterprise check across drains in the same org', async () => {
|
||||
dbChainMockFns.returning
|
||||
.mockResolvedValueOnce([]) // reaper
|
||||
.mockResolvedValueOnce([{ id: 'd1' }])
|
||||
.mockResolvedValueOnce([{ id: 'd2' }])
|
||||
mockCandidates([
|
||||
{ id: 'd1', organizationId: 'org-a' },
|
||||
{ id: 'd2', organizationId: 'org-a' },
|
||||
])
|
||||
mockIsEnterprise.mockResolvedValue(true)
|
||||
|
||||
await dispatchDueDrains()
|
||||
expect(mockIsEnterprise).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,186 @@
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrainRuns, dataDrains } from '@sim/db/schema'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { and, eq, isNull, lt, or } from 'drizzle-orm'
|
||||
import { isOrganizationOnEnterprisePlan } from '@/lib/billing/core/subscription'
|
||||
import { getJobQueue } from '@/lib/core/async-jobs'
|
||||
import { isBillingEnabled } from '@/lib/core/config/feature-flags'
|
||||
|
||||
const logger = createLogger('DataDrainsDispatcher')
|
||||
|
||||
const HOUR_MS = 60 * 60 * 1000
|
||||
const DAY_MS = 24 * HOUR_MS
|
||||
|
||||
/**
|
||||
* Cron fires hourly. Without a buffer, a drain that finishes a few minutes
|
||||
* after the tick (lastRunAt = 10:05) won't satisfy `lastRunAt < now - cadence`
|
||||
* at the next tick (10:05 < 10:00 is false), so an "hourly" drain effectively
|
||||
* runs every two hours. Subtracting a small buffer from the cadence absorbs
|
||||
* normal run duration plus cron jitter without allowing back-to-back runs
|
||||
* within the same tick.
|
||||
*/
|
||||
const CADENCE_BUFFER_MS = 5 * 60 * 1000
|
||||
|
||||
/**
|
||||
* Maximum wall-clock duration any single drain run is allowed before its
|
||||
* `data_drain_runs` row is considered orphaned. Runs that exceed this are
|
||||
* almost certainly the result of a Trigger.dev worker crash mid-run — there
|
||||
* is no live process still updating them.
|
||||
*/
|
||||
const ORPHAN_THRESHOLD_MS = 60 * 60 * 1000
|
||||
|
||||
/**
|
||||
* Marks `running` rows older than the orphan threshold as `failed`. Without
|
||||
* this, a worker crash leaves run history permanently misleading and (worse)
|
||||
* the drain row's `lastRunAt` reflects a successful claim that never finished
|
||||
* — but the drain `cursor` never advanced, so re-running is safe.
|
||||
*/
|
||||
export async function reapOrphanedRuns(now: Date = new Date()): Promise<{ reaped: number }> {
|
||||
const cutoff = new Date(now.getTime() - ORPHAN_THRESHOLD_MS)
|
||||
const reaped = await db
|
||||
.update(dataDrainRuns)
|
||||
.set({
|
||||
status: 'failed',
|
||||
finishedAt: now,
|
||||
error: `Orphaned run reaped after exceeding ${ORPHAN_THRESHOLD_MS / 60_000}m without completion`,
|
||||
})
|
||||
.where(and(eq(dataDrainRuns.status, 'running'), lt(dataDrainRuns.startedAt, cutoff)))
|
||||
.returning({ id: dataDrainRuns.id })
|
||||
if (reaped.length > 0) {
|
||||
logger.warn('Reaped orphaned data drain runs', { count: reaped.length })
|
||||
}
|
||||
return { reaped: reaped.length }
|
||||
}
|
||||
|
||||
/**
|
||||
* Selects every enabled drain whose schedule is due (or has never run) and
|
||||
* fans out one `run-data-drain` job per drain. Each drain is atomically
|
||||
* claimed via a conditional UPDATE before being enqueued — two concurrent
|
||||
* dispatcher invocations cannot both win the same row, and a manual run that
|
||||
* lands between the SELECT and the UPDATE will lose the race cleanly. Drains
|
||||
* belonging to orgs that have lapsed off the enterprise plan are skipped.
|
||||
*/
|
||||
export async function dispatchDueDrains(now: Date = new Date()): Promise<{
|
||||
candidates: number
|
||||
dispatched: number
|
||||
skipped: number
|
||||
reaped: number
|
||||
}> {
|
||||
const { reaped } = await reapOrphanedRuns(now)
|
||||
|
||||
const hourlyCutoff = new Date(now.getTime() - HOUR_MS + CADENCE_BUFFER_MS)
|
||||
const dailyCutoff = new Date(now.getTime() - DAY_MS + CADENCE_BUFFER_MS)
|
||||
|
||||
const duePredicate = and(
|
||||
eq(dataDrains.enabled, true),
|
||||
or(
|
||||
isNull(dataDrains.lastRunAt),
|
||||
and(eq(dataDrains.scheduleCadence, 'hourly'), lt(dataDrains.lastRunAt, hourlyCutoff)),
|
||||
and(eq(dataDrains.scheduleCadence, 'daily'), lt(dataDrains.lastRunAt, dailyCutoff))
|
||||
)
|
||||
)
|
||||
|
||||
const candidates = await db
|
||||
.select({
|
||||
id: dataDrains.id,
|
||||
organizationId: dataDrains.organizationId,
|
||||
lastRunAt: dataDrains.lastRunAt,
|
||||
})
|
||||
.from(dataDrains)
|
||||
.where(duePredicate)
|
||||
|
||||
if (candidates.length === 0) {
|
||||
return { candidates: 0, dispatched: 0, skipped: 0, reaped }
|
||||
}
|
||||
|
||||
// Self-hosted deployments have no subscription infra; `DATA_DRAINS_ENABLED`
|
||||
// is the global on/off there. Cache per-org so a multi-drain org pays one
|
||||
// billing lookup.
|
||||
const enterpriseCache = new Map<string, boolean>()
|
||||
const isEnterprise = async (orgId: string): Promise<boolean> => {
|
||||
if (!isBillingEnabled) return true
|
||||
const cached = enterpriseCache.get(orgId)
|
||||
if (cached !== undefined) return cached
|
||||
const result = await isOrganizationOnEnterprisePlan(orgId)
|
||||
enterpriseCache.set(orgId, result)
|
||||
return result
|
||||
}
|
||||
|
||||
const queue = await getJobQueue()
|
||||
let dispatched = 0
|
||||
let skipped = 0
|
||||
|
||||
for (const candidate of candidates) {
|
||||
let enterprise: boolean
|
||||
try {
|
||||
enterprise = await isEnterprise(candidate.organizationId)
|
||||
} catch (error) {
|
||||
// A billing-API failure for one org must not abort the whole batch —
|
||||
// skip this drain and let the next cron tick retry it.
|
||||
logger.warn('Enterprise check failed; skipping drain', {
|
||||
drainId: candidate.id,
|
||||
organizationId: candidate.organizationId,
|
||||
error,
|
||||
})
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
if (!enterprise) {
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
|
||||
// Conditional claim — re-asserts the due predicate to lose to any other
|
||||
// dispatcher or manual-run path that's already moved this drain forward.
|
||||
const claimed = await db
|
||||
.update(dataDrains)
|
||||
.set({ lastRunAt: now, updatedAt: now })
|
||||
.where(and(eq(dataDrains.id, candidate.id), duePredicate))
|
||||
.returning({ id: dataDrains.id })
|
||||
|
||||
if (claimed.length === 0) continue
|
||||
|
||||
try {
|
||||
// concurrencyKey serializes runs of the same drain on the job queue, so
|
||||
// a manual run-now racing a cron claim can never execute in parallel.
|
||||
await queue.enqueue(
|
||||
'run-data-drain',
|
||||
{ drainId: candidate.id, trigger: 'cron' },
|
||||
{ concurrencyKey: `data-drain:${candidate.id}` }
|
||||
)
|
||||
dispatched++
|
||||
} catch (error) {
|
||||
// Roll back the claim so a transient queue outage doesn't delay this
|
||||
// drain by a full cadence. Scoped to our own claim timestamp so it
|
||||
// can't trample a concurrent advance. The rollback itself is guarded
|
||||
// so a DB error here doesn't abort the rest of the batch.
|
||||
try {
|
||||
await db
|
||||
.update(dataDrains)
|
||||
.set({ lastRunAt: candidate.lastRunAt, updatedAt: now })
|
||||
.where(and(eq(dataDrains.id, candidate.id), eq(dataDrains.lastRunAt, now)))
|
||||
} catch (rollbackError) {
|
||||
logger.error('Failed to roll back data-drain claim after enqueue failure', {
|
||||
drainId: candidate.id,
|
||||
enqueueError: toError(error).message,
|
||||
rollbackError: toError(rollbackError).message,
|
||||
})
|
||||
continue
|
||||
}
|
||||
logger.error('Failed to enqueue data-drain job; rolled back claim', {
|
||||
drainId: candidate.id,
|
||||
error,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
logger.info('Data drain dispatch complete', {
|
||||
candidates: candidates.length,
|
||||
dispatched,
|
||||
skipped,
|
||||
reaped,
|
||||
})
|
||||
|
||||
return { candidates: candidates.length, dispatched, skipped, reaped }
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import { decryptSecret, encryptSecret } from '@/lib/core/security/encryption'
|
||||
|
||||
/**
|
||||
* Encrypts an arbitrary JSON-serializable credentials object into a single
|
||||
* `iv:ciphertext:authTag` string suitable for storage in
|
||||
* `data_drains.destination_credentials`. Wraps the shared AES-256-GCM helper.
|
||||
*/
|
||||
export async function encryptCredentials<T>(plaintext: T): Promise<string> {
|
||||
const { encrypted } = await encryptSecret(JSON.stringify(plaintext))
|
||||
return encrypted
|
||||
}
|
||||
|
||||
/**
|
||||
* Decrypts the inverse of `encryptCredentials`. The caller is expected to run
|
||||
* the destination's `credentialsSchema` on the result to defend against
|
||||
* encryption-format drift.
|
||||
*/
|
||||
export async function decryptCredentials<T>(ciphertext: string): Promise<T> {
|
||||
const { decrypted } = await decryptSecret(ciphertext)
|
||||
return JSON.parse(decrypted) as T
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import type { dataDrainRuns, dataDrains } from '@sim/db/schema'
|
||||
import { type DataDrain, type DataDrainRun, dataDrainSchema } from '@/lib/api/contracts/data-drains'
|
||||
import { getDestination } from '@/lib/data-drains/destinations/registry'
|
||||
|
||||
type DataDrainRow = typeof dataDrains.$inferSelect
|
||||
type DataDrainRunRow = typeof dataDrainRuns.$inferSelect
|
||||
|
||||
/**
|
||||
* Projects a DB row into the public `DataDrain` wire shape. Strips the
|
||||
* encrypted credentials column and normalizes timestamps to ISO strings so
|
||||
* clients receive a stable, schema-validated payload.
|
||||
*
|
||||
* The stored `destinationConfig` is JSONB and is re-validated against the
|
||||
* destination's typed config schema before serialization so unexpected shapes
|
||||
* surface as errors instead of leaking through the response.
|
||||
*/
|
||||
export function serializeDrain(row: DataDrainRow): DataDrain {
|
||||
const destinationConfig = getDestination(row.destinationType).configSchema.parse(
|
||||
row.destinationConfig
|
||||
)
|
||||
return dataDrainSchema.parse({
|
||||
id: row.id,
|
||||
organizationId: row.organizationId,
|
||||
name: row.name,
|
||||
source: row.source,
|
||||
scheduleCadence: row.scheduleCadence,
|
||||
enabled: row.enabled,
|
||||
cursor: row.cursor,
|
||||
lastRunAt: row.lastRunAt ? row.lastRunAt.toISOString() : null,
|
||||
lastSuccessAt: row.lastSuccessAt ? row.lastSuccessAt.toISOString() : null,
|
||||
createdBy: row.createdBy,
|
||||
createdAt: row.createdAt.toISOString(),
|
||||
updatedAt: row.updatedAt.toISOString(),
|
||||
destinationType: row.destinationType,
|
||||
destinationConfig,
|
||||
})
|
||||
}
|
||||
|
||||
export function serializeDrainRun(row: DataDrainRunRow): DataDrainRun {
|
||||
return {
|
||||
id: row.id,
|
||||
drainId: row.drainId,
|
||||
status: row.status,
|
||||
trigger: row.trigger,
|
||||
startedAt: row.startedAt.toISOString(),
|
||||
finishedAt: row.finishedAt ? row.finishedAt.toISOString() : null,
|
||||
rowsExported: row.rowsExported,
|
||||
bytesWritten: row.bytesWritten,
|
||||
cursorBefore: row.cursorBefore,
|
||||
cursorAfter: row.cursorAfter,
|
||||
error: row.error,
|
||||
locators: row.locators ?? [],
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { dbChainMock, dbChainMockFns, resetDbChainMock } from '@sim/testing'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
vi.mock('@sim/db', () => dbChainMock)
|
||||
|
||||
const { mockGetSource, mockGetDestination, mockDecryptCredentials } = vi.hoisted(() => ({
|
||||
mockGetSource: vi.fn(),
|
||||
mockGetDestination: vi.fn(),
|
||||
mockDecryptCredentials: vi.fn(),
|
||||
}))
|
||||
|
||||
vi.mock('@/lib/data-drains/sources/registry', () => ({ getSource: mockGetSource }))
|
||||
vi.mock('@/lib/data-drains/destinations/registry', () => ({ getDestination: mockGetDestination }))
|
||||
vi.mock('@/lib/data-drains/encryption', () => ({ decryptCredentials: mockDecryptCredentials }))
|
||||
|
||||
import { runDrain } from '@/lib/data-drains/service'
|
||||
|
||||
type Row = { id: string; ts: string }
|
||||
|
||||
function makeSource(pages: Row[][]) {
|
||||
return {
|
||||
type: 'workflow_logs' as const,
|
||||
displayName: 'Test',
|
||||
pages: vi.fn(async function* () {
|
||||
for (const page of pages) yield page
|
||||
}),
|
||||
serialize: vi.fn((row: Row) => row),
|
||||
cursorAfter: vi.fn((row: Row) => JSON.stringify({ ts: row.ts, id: row.id })),
|
||||
}
|
||||
}
|
||||
|
||||
function makeDestination(
|
||||
opts: { deliver?: ReturnType<typeof vi.fn>; close?: ReturnType<typeof vi.fn> } = {}
|
||||
) {
|
||||
const deliver =
|
||||
opts.deliver ??
|
||||
vi.fn(async ({ metadata }: { metadata: { sequence: number } }) => ({
|
||||
locator: `loc-${metadata.sequence}`,
|
||||
}))
|
||||
const close = opts.close ?? vi.fn(async () => {})
|
||||
return {
|
||||
type: 's3' as const,
|
||||
displayName: 'Test',
|
||||
configSchema: { parse: (v: unknown) => v },
|
||||
credentialsSchema: { parse: (v: unknown) => v },
|
||||
openSession: vi.fn(() => ({ deliver, close })),
|
||||
_deliver: deliver,
|
||||
_close: close,
|
||||
}
|
||||
}
|
||||
|
||||
const baseDrain = {
|
||||
id: 'drain-1',
|
||||
organizationId: 'org-1',
|
||||
enabled: true,
|
||||
source: 'workflow_logs',
|
||||
destinationType: 's3',
|
||||
destinationConfig: {},
|
||||
destinationCredentials: 'enc:blob',
|
||||
cursor: null,
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
resetDbChainMock()
|
||||
mockDecryptCredentials.mockResolvedValue({})
|
||||
})
|
||||
|
||||
describe('runDrain', () => {
|
||||
it('returns skipped when drain is disabled', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([{ ...baseDrain, enabled: false }])
|
||||
const result = await runDrain('drain-1', 'manual')
|
||||
expect(result.status).toBe('skipped')
|
||||
expect(result.rowsExported).toBe(0)
|
||||
expect(mockGetSource).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('throws when drain does not exist', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([])
|
||||
await expect(runDrain('drain-1', 'manual')).rejects.toThrow(/not found/)
|
||||
})
|
||||
|
||||
it('delivers each page and advances cursor on success', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([baseDrain])
|
||||
const source = makeSource([
|
||||
[
|
||||
{ id: 'r1', ts: '2026-01-01T00:00:00.000Z' },
|
||||
{ id: 'r2', ts: '2026-01-01T00:00:01.000Z' },
|
||||
],
|
||||
[{ id: 'r3', ts: '2026-01-01T00:00:02.000Z' }],
|
||||
])
|
||||
const destination = makeDestination()
|
||||
mockGetSource.mockReturnValue(source)
|
||||
mockGetDestination.mockReturnValue(destination)
|
||||
|
||||
const result = await runDrain('drain-1', 'cron')
|
||||
|
||||
expect(result.status).toBe('success')
|
||||
expect(result.rowsExported).toBe(3)
|
||||
expect(destination._deliver).toHaveBeenCalledTimes(2)
|
||||
expect(destination._close).toHaveBeenCalledTimes(1)
|
||||
expect(result.cursorAfter).toBe(JSON.stringify({ ts: '2026-01-01T00:00:02.000Z', id: 'r3' }))
|
||||
expect(result.locators).toEqual(['loc-0', 'loc-1'])
|
||||
|
||||
// Drain row updated with new cursor; transaction was used.
|
||||
expect(dbChainMockFns.transaction).toHaveBeenCalled()
|
||||
const drainUpdate = dbChainMockFns.set.mock.calls.find(
|
||||
(call) => (call[0] as { cursor?: unknown }).cursor !== undefined
|
||||
)
|
||||
expect(drainUpdate?.[0]).toMatchObject({ cursor: result.cursorAfter })
|
||||
})
|
||||
|
||||
it('does not advance drain cursor when delivery fails', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([{ ...baseDrain, cursor: 'prior' }])
|
||||
const source = makeSource([[{ id: 'r1', ts: '2026-01-01T00:00:00.000Z' }]])
|
||||
const destination = makeDestination({
|
||||
deliver: vi.fn(async () => {
|
||||
throw new Error('boom')
|
||||
}),
|
||||
})
|
||||
mockGetSource.mockReturnValue(source)
|
||||
mockGetDestination.mockReturnValue(destination)
|
||||
|
||||
await expect(runDrain('drain-1', 'cron')).rejects.toThrow('boom')
|
||||
|
||||
// Run row updated with status=failed and cursorAfter equal to prior cursor.
|
||||
const failedUpdate = dbChainMockFns.set.mock.calls.find(
|
||||
(call) => (call[0] as { status?: unknown }).status === 'failed'
|
||||
)
|
||||
expect(failedUpdate?.[0]).toMatchObject({ status: 'failed', cursorAfter: 'prior' })
|
||||
|
||||
// No drain-row update with a new cursor field.
|
||||
const cursorAdvanced = dbChainMockFns.set.mock.calls.some(
|
||||
(call) => 'cursor' in (call[0] as object)
|
||||
)
|
||||
expect(cursorAdvanced).toBe(false)
|
||||
|
||||
expect(destination._close).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('closes session even if close throws', async () => {
|
||||
dbChainMockFns.limit.mockResolvedValueOnce([baseDrain])
|
||||
const source = makeSource([])
|
||||
const destination = makeDestination({
|
||||
close: vi.fn(async () => {
|
||||
throw new Error('close-failed')
|
||||
}),
|
||||
})
|
||||
mockGetSource.mockReturnValue(source)
|
||||
mockGetDestination.mockReturnValue(destination)
|
||||
|
||||
const result = await runDrain('drain-1', 'manual')
|
||||
expect(result.status).toBe('success')
|
||||
expect(destination._close).toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,227 @@
|
||||
import { db } from '@sim/db'
|
||||
import { dataDrainRuns, dataDrains } from '@sim/db/schema'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { generateId } from '@sim/utils/id'
|
||||
import { eq } from 'drizzle-orm'
|
||||
import { getDestination } from '@/lib/data-drains/destinations/registry'
|
||||
import { decryptCredentials } from '@/lib/data-drains/encryption'
|
||||
import { getSource } from '@/lib/data-drains/sources/registry'
|
||||
import type { Cursor, RunTrigger } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainsService')
|
||||
|
||||
const CHUNK_SIZE = 1000
|
||||
|
||||
export interface RunDrainResult {
|
||||
drainId: string
|
||||
runId: string
|
||||
status: 'success' | 'failed' | 'skipped'
|
||||
rowsExported: number
|
||||
bytesWritten: number
|
||||
cursorBefore: Cursor
|
||||
cursorAfter: Cursor
|
||||
locators: string[]
|
||||
error?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Orchestrates one drain export. Source-/destination-agnostic — talks only to
|
||||
* the registry interfaces. The drain's cursor is advanced only when the entire
|
||||
* run completes successfully so consumers see at-least-once delivery and can
|
||||
* dedupe on the per-row `id` field.
|
||||
*/
|
||||
export async function runDrain(
|
||||
drainId: string,
|
||||
trigger: RunTrigger,
|
||||
options: { signal?: AbortSignal } = {}
|
||||
): Promise<RunDrainResult> {
|
||||
const signal = options.signal ?? new AbortController().signal
|
||||
const [drain] = await db.select().from(dataDrains).where(eq(dataDrains.id, drainId)).limit(1)
|
||||
if (!drain) {
|
||||
throw new Error(`Data drain not found: ${drainId}`)
|
||||
}
|
||||
if (!drain.enabled) {
|
||||
return {
|
||||
drainId,
|
||||
runId: '',
|
||||
status: 'skipped',
|
||||
rowsExported: 0,
|
||||
bytesWritten: 0,
|
||||
cursorBefore: drain.cursor,
|
||||
cursorAfter: drain.cursor,
|
||||
locators: [],
|
||||
}
|
||||
}
|
||||
|
||||
const source = getSource(drain.source)
|
||||
const destination = getDestination(drain.destinationType)
|
||||
|
||||
const runId = generateId()
|
||||
const startedAt = new Date()
|
||||
await db.insert(dataDrainRuns).values({
|
||||
id: runId,
|
||||
drainId,
|
||||
status: 'running',
|
||||
trigger,
|
||||
startedAt,
|
||||
cursorBefore: drain.cursor,
|
||||
})
|
||||
|
||||
const cursorBefore = drain.cursor
|
||||
let cursor: Cursor = drain.cursor
|
||||
let rowsExported = 0
|
||||
let bytesWritten = 0
|
||||
let sequence = 0
|
||||
const locators: string[] = []
|
||||
|
||||
/**
|
||||
* Schema-parse and decrypt happen *after* the run row is created so failures
|
||||
* in either (e.g. encryption-key rotation, schema drift across versions)
|
||||
* surface as a `failed` run row in the UI rather than vanishing into the
|
||||
* background-job logs while `lastRunAt` quietly advances.
|
||||
*/
|
||||
let session: ReturnType<typeof destination.openSession> | null = null
|
||||
|
||||
try {
|
||||
const config = destination.configSchema.parse(drain.destinationConfig)
|
||||
const credentials = destination.credentialsSchema.parse(
|
||||
await decryptCredentials(drain.destinationCredentials)
|
||||
)
|
||||
session = destination.openSession({ config, credentials })
|
||||
|
||||
for await (const chunk of source.pages({
|
||||
organizationId: drain.organizationId,
|
||||
cursor,
|
||||
chunkSize: CHUNK_SIZE,
|
||||
signal,
|
||||
})) {
|
||||
const ndjson = `${chunk.map((row) => JSON.stringify(source.serialize(row))).join('\n')}\n`
|
||||
const body = Buffer.from(ndjson, 'utf8')
|
||||
|
||||
const result = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId,
|
||||
runId,
|
||||
source: drain.source,
|
||||
sequence,
|
||||
rowCount: chunk.length,
|
||||
runStartedAt: startedAt,
|
||||
},
|
||||
signal,
|
||||
})
|
||||
|
||||
locators.push(result.locator)
|
||||
rowsExported += chunk.length
|
||||
bytesWritten += body.byteLength
|
||||
cursor = source.cursorAfter(chunk[chunk.length - 1])
|
||||
sequence++
|
||||
}
|
||||
|
||||
if (signal.aborted) {
|
||||
throw new Error('Data drain run cancelled')
|
||||
}
|
||||
|
||||
const finishedAt = new Date()
|
||||
await db.transaction(async (tx) => {
|
||||
await tx
|
||||
.update(dataDrains)
|
||||
.set({
|
||||
cursor,
|
||||
lastRunAt: finishedAt,
|
||||
lastSuccessAt: finishedAt,
|
||||
updatedAt: finishedAt,
|
||||
})
|
||||
.where(eq(dataDrains.id, drainId))
|
||||
await tx
|
||||
.update(dataDrainRuns)
|
||||
.set({
|
||||
status: 'success',
|
||||
finishedAt,
|
||||
rowsExported,
|
||||
bytesWritten,
|
||||
cursorAfter: cursor,
|
||||
locators,
|
||||
error: null,
|
||||
})
|
||||
.where(eq(dataDrainRuns.id, runId))
|
||||
})
|
||||
|
||||
logger.info('Data drain run succeeded', {
|
||||
drainId,
|
||||
runId,
|
||||
source: drain.source,
|
||||
destinationType: drain.destinationType,
|
||||
rowsExported,
|
||||
bytesWritten,
|
||||
chunks: sequence,
|
||||
})
|
||||
|
||||
return {
|
||||
drainId,
|
||||
runId,
|
||||
status: 'success',
|
||||
rowsExported,
|
||||
bytesWritten,
|
||||
cursorBefore,
|
||||
cursorAfter: cursor,
|
||||
locators,
|
||||
}
|
||||
} catch (error) {
|
||||
const finishedAt = new Date()
|
||||
const message = toError(error).message
|
||||
try {
|
||||
await db.transaction(async (tx) => {
|
||||
await tx
|
||||
.update(dataDrains)
|
||||
.set({ lastRunAt: finishedAt, updatedAt: finishedAt })
|
||||
.where(eq(dataDrains.id, drainId))
|
||||
await tx
|
||||
.update(dataDrainRuns)
|
||||
.set({
|
||||
status: 'failed',
|
||||
finishedAt,
|
||||
rowsExported,
|
||||
bytesWritten,
|
||||
cursorAfter: cursorBefore,
|
||||
locators,
|
||||
error: message.slice(0, 4000),
|
||||
})
|
||||
.where(eq(dataDrainRuns.id, runId))
|
||||
})
|
||||
} catch (statusError) {
|
||||
// Best-effort status write — the reaper repairs stuck rows. Log so DB
|
||||
// outages don't hide behind the original delivery error.
|
||||
logger.error('Failed to record data drain failure status', {
|
||||
drainId,
|
||||
runId,
|
||||
deliveryError: message,
|
||||
statusError: toError(statusError).message,
|
||||
})
|
||||
}
|
||||
|
||||
logger.error('Data drain run failed', {
|
||||
drainId,
|
||||
runId,
|
||||
source: drain.source,
|
||||
destinationType: drain.destinationType,
|
||||
error: message,
|
||||
})
|
||||
|
||||
throw error
|
||||
} finally {
|
||||
if (session) {
|
||||
try {
|
||||
await session.close()
|
||||
} catch (closeError) {
|
||||
logger.warn('Data drain session close failed', {
|
||||
drainId,
|
||||
runId,
|
||||
error: toError(closeError).message,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
import { db } from '@sim/db'
|
||||
import { auditLog } from '@sim/db/schema'
|
||||
import { and, inArray, isNull, or, sql } from 'drizzle-orm'
|
||||
import {
|
||||
decodeTimeCursor,
|
||||
encodeTimeCursor,
|
||||
timeCursorOrderBy,
|
||||
timeCursorPredicate,
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
|
||||
type AuditLogRow = typeof auditLog.$inferSelect
|
||||
|
||||
/**
|
||||
* Drains audit events scoped to the organization: rows from any of the org's
|
||||
* workspaces, plus org-level rows (`workspace_id IS NULL`) where
|
||||
* `metadata->>'organizationId'` matches. Audit-log writers consistently set
|
||||
* `metadata.organizationId` for org-scoped actions even though the table has
|
||||
* no dedicated FK column.
|
||||
*/
|
||||
async function* pages(input: SourcePageInput): AsyncIterable<AuditLogRow[]> {
|
||||
const workspaceIds = await getOrganizationWorkspaceIds(input.organizationId)
|
||||
|
||||
const orgScopedClause = and(
|
||||
isNull(auditLog.workspaceId),
|
||||
sql`${auditLog.metadata}->>'organizationId' = ${input.organizationId}`
|
||||
)
|
||||
const scopeClause =
|
||||
workspaceIds.length === 0
|
||||
? orgScopedClause
|
||||
: or(inArray(auditLog.workspaceId, workspaceIds), orgScopedClause)
|
||||
|
||||
let cursor = decodeTimeCursor(input.cursor)
|
||||
while (!input.signal.aborted) {
|
||||
const cursorClause = timeCursorPredicate(auditLog.createdAt, auditLog.id, cursor)
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(auditLog)
|
||||
.where(and(scopeClause, cursorClause))
|
||||
.orderBy(...timeCursorOrderBy(auditLog.createdAt, auditLog.id))
|
||||
.limit(input.chunkSize)
|
||||
|
||||
if (rows.length === 0) return
|
||||
yield rows
|
||||
const last = rows[rows.length - 1]
|
||||
cursor = { ts: last.createdAt.toISOString(), id: last.id }
|
||||
if (rows.length < input.chunkSize) return
|
||||
}
|
||||
}
|
||||
|
||||
export const auditLogsSource: DrainSource<AuditLogRow> = {
|
||||
type: 'audit_logs',
|
||||
displayName: 'Audit logs',
|
||||
pages,
|
||||
serialize(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
workspaceId: row.workspaceId,
|
||||
actorId: row.actorId,
|
||||
actorName: row.actorName,
|
||||
actorEmail: row.actorEmail,
|
||||
action: row.action,
|
||||
resourceType: row.resourceType,
|
||||
resourceId: row.resourceId,
|
||||
resourceName: row.resourceName,
|
||||
description: row.description,
|
||||
metadata: row.metadata,
|
||||
ipAddress: row.ipAddress,
|
||||
userAgent: row.userAgent,
|
||||
createdAt: row.createdAt.toISOString(),
|
||||
}
|
||||
},
|
||||
cursorAfter(row): Cursor {
|
||||
return encodeTimeCursor({ ts: row.createdAt.toISOString(), id: row.id })
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
import { db } from '@sim/db'
|
||||
import { copilotChats } from '@sim/db/schema'
|
||||
import { and, inArray } from 'drizzle-orm'
|
||||
import {
|
||||
decodeTimeCursor,
|
||||
encodeTimeCursor,
|
||||
timeCursorOrderBy,
|
||||
timeCursorPredicate,
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
|
||||
type CopilotChatRow = typeof copilotChats.$inferSelect
|
||||
|
||||
/**
|
||||
* Cursor is `createdAt` (immutable) but rows themselves are mutable —
|
||||
* `messages`, `title`, `lastSeenAt`, etc. are updated in-place over the chat's
|
||||
* lifetime. This means a chat exported once will not be re-exported when its
|
||||
* messages change. Consumers who need the latest state should periodically
|
||||
* full-refresh from a separate snapshot job; drains are append-mostly by
|
||||
* design and `data-drains` is not a CDC pipeline.
|
||||
*/
|
||||
async function* pages(input: SourcePageInput): AsyncIterable<CopilotChatRow[]> {
|
||||
const workspaceIds = await getOrganizationWorkspaceIds(input.organizationId)
|
||||
if (workspaceIds.length === 0) return
|
||||
|
||||
let cursor = decodeTimeCursor(input.cursor)
|
||||
while (!input.signal.aborted) {
|
||||
const cursorClause = timeCursorPredicate(copilotChats.createdAt, copilotChats.id, cursor)
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(copilotChats)
|
||||
.where(and(inArray(copilotChats.workspaceId, workspaceIds), cursorClause))
|
||||
.orderBy(...timeCursorOrderBy(copilotChats.createdAt, copilotChats.id))
|
||||
.limit(input.chunkSize)
|
||||
|
||||
if (rows.length === 0) return
|
||||
yield rows
|
||||
const last = rows[rows.length - 1]
|
||||
cursor = { ts: last.createdAt.toISOString(), id: last.id }
|
||||
if (rows.length < input.chunkSize) return
|
||||
}
|
||||
}
|
||||
|
||||
export const copilotChatsSource: DrainSource<CopilotChatRow> = {
|
||||
type: 'copilot_chats',
|
||||
displayName: 'Copilot chats',
|
||||
pages,
|
||||
serialize(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
userId: row.userId,
|
||||
workflowId: row.workflowId,
|
||||
workspaceId: row.workspaceId,
|
||||
type: row.type,
|
||||
title: row.title,
|
||||
messages: row.messages,
|
||||
model: row.model,
|
||||
conversationId: row.conversationId,
|
||||
previewYaml: row.previewYaml,
|
||||
planArtifact: row.planArtifact,
|
||||
config: row.config,
|
||||
resources: row.resources,
|
||||
lastSeenAt: row.lastSeenAt ? row.lastSeenAt.toISOString() : null,
|
||||
createdAt: row.createdAt.toISOString(),
|
||||
updatedAt: row.updatedAt.toISOString(),
|
||||
}
|
||||
},
|
||||
cursorAfter(row): Cursor {
|
||||
return encodeTimeCursor({ ts: row.createdAt.toISOString(), id: row.id })
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
import { db } from '@sim/db'
|
||||
import { copilotRuns } from '@sim/db/schema'
|
||||
import { and, inArray, isNotNull } from 'drizzle-orm'
|
||||
import {
|
||||
decodeTimeCursor,
|
||||
encodeTimeCursor,
|
||||
timeCursorOrderBy,
|
||||
timeCursorPredicate,
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
|
||||
type CopilotRunRow = typeof copilotRuns.$inferSelect
|
||||
|
||||
/**
|
||||
* Cursors on terminal `completedAt` so in-flight runs (mutable `status`,
|
||||
* `error`, `completedAt`) are not exported until they reach a terminal state.
|
||||
*/
|
||||
async function* pages(input: SourcePageInput): AsyncIterable<CopilotRunRow[]> {
|
||||
const workspaceIds = await getOrganizationWorkspaceIds(input.organizationId)
|
||||
if (workspaceIds.length === 0) return
|
||||
|
||||
let cursor = decodeTimeCursor(input.cursor)
|
||||
while (!input.signal.aborted) {
|
||||
const cursorClause = timeCursorPredicate(copilotRuns.completedAt, copilotRuns.id, cursor)
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(copilotRuns)
|
||||
.where(
|
||||
and(
|
||||
inArray(copilotRuns.workspaceId, workspaceIds),
|
||||
isNotNull(copilotRuns.completedAt),
|
||||
cursorClause
|
||||
)
|
||||
)
|
||||
.orderBy(...timeCursorOrderBy(copilotRuns.completedAt, copilotRuns.id))
|
||||
.limit(input.chunkSize)
|
||||
|
||||
if (rows.length === 0) return
|
||||
yield rows
|
||||
const last = rows[rows.length - 1]
|
||||
cursor = { ts: last.completedAt!.toISOString(), id: last.id }
|
||||
if (rows.length < input.chunkSize) return
|
||||
}
|
||||
}
|
||||
|
||||
export const copilotRunsSource: DrainSource<CopilotRunRow> = {
|
||||
type: 'copilot_runs',
|
||||
displayName: 'Copilot runs',
|
||||
pages,
|
||||
serialize(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
executionId: row.executionId,
|
||||
parentRunId: row.parentRunId,
|
||||
chatId: row.chatId,
|
||||
userId: row.userId,
|
||||
workflowId: row.workflowId,
|
||||
workspaceId: row.workspaceId,
|
||||
streamId: row.streamId,
|
||||
agent: row.agent,
|
||||
model: row.model,
|
||||
provider: row.provider,
|
||||
status: row.status,
|
||||
requestContext: row.requestContext,
|
||||
startedAt: row.startedAt.toISOString(),
|
||||
completedAt: row.completedAt ? row.completedAt.toISOString() : null,
|
||||
createdAt: row.createdAt.toISOString(),
|
||||
updatedAt: row.updatedAt.toISOString(),
|
||||
error: row.error,
|
||||
}
|
||||
},
|
||||
cursorAfter(row): Cursor {
|
||||
return encodeTimeCursor({ ts: row.completedAt!.toISOString(), id: row.id })
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { decodeTimeCursor, encodeTimeCursor } from '@/lib/data-drains/sources/cursor'
|
||||
|
||||
describe('time cursor encoding', () => {
|
||||
it('round-trips a valid cursor', () => {
|
||||
const value = { ts: '2026-01-01T00:00:00.000Z', id: 'row-1' }
|
||||
expect(decodeTimeCursor(encodeTimeCursor(value))).toEqual(value)
|
||||
})
|
||||
|
||||
it('returns null for null input', () => {
|
||||
expect(decodeTimeCursor(null)).toBeNull()
|
||||
})
|
||||
|
||||
it('returns null for malformed JSON', () => {
|
||||
expect(decodeTimeCursor('not-json')).toBeNull()
|
||||
})
|
||||
|
||||
it('returns null when shape is wrong', () => {
|
||||
expect(decodeTimeCursor(JSON.stringify({ ts: 1, id: 'x' }))).toBeNull()
|
||||
expect(decodeTimeCursor(JSON.stringify({ ts: '2026', id: 5 }))).toBeNull()
|
||||
expect(decodeTimeCursor(JSON.stringify({}))).toBeNull()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,56 @@
|
||||
import { type SQL, sql } from 'drizzle-orm'
|
||||
import type { PgColumn } from 'drizzle-orm/pg-core'
|
||||
import type { Cursor } from '@/lib/data-drains/types'
|
||||
|
||||
/**
|
||||
* Composite cursor for time-ordered tables. Pairs a timestamp with the row's id
|
||||
* so chunks split across rows that share a timestamp pick up cleanly without
|
||||
* skipping or duplicating.
|
||||
*/
|
||||
export interface TimeCursor {
|
||||
ts: string
|
||||
id: string
|
||||
}
|
||||
|
||||
export function encodeTimeCursor(value: TimeCursor): Cursor {
|
||||
return JSON.stringify(value)
|
||||
}
|
||||
|
||||
export function decodeTimeCursor(cursor: Cursor): TimeCursor | null {
|
||||
if (!cursor) return null
|
||||
try {
|
||||
const parsed = JSON.parse(cursor) as TimeCursor
|
||||
if (typeof parsed?.ts !== 'string' || typeof parsed?.id !== 'string') return null
|
||||
return parsed
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a strict-greater-than predicate over a `(timestampCol, idCol)` pair.
|
||||
*
|
||||
* Postgres `timestamp` columns store microsecond precision but JS `Date`
|
||||
* round-trips at millisecond precision, so the cursor only ever captures
|
||||
* millisecond-truncated timestamps. We compare in millisecond buckets via
|
||||
* `date_trunc('milliseconds', col)` so the predicate's notion of order matches
|
||||
* `timeCursorOrderBy` exactly. If ORDER BY used raw microseconds while the
|
||||
* predicate used millisecond buckets, a row sorted later by µs but with a
|
||||
* lexicographically earlier id than the cursor row would be skipped forever.
|
||||
*/
|
||||
export function timeCursorPredicate(
|
||||
timestampCol: PgColumn,
|
||||
idCol: PgColumn,
|
||||
cursor: TimeCursor | null
|
||||
): SQL | undefined {
|
||||
if (!cursor) return undefined
|
||||
return sql`(date_trunc('milliseconds', ${timestampCol}), ${idCol}) > (${new Date(cursor.ts)}, ${cursor.id})`
|
||||
}
|
||||
|
||||
/**
|
||||
* ORDER BY fragments paired with `timeCursorPredicate`. Both must agree on
|
||||
* millisecond bucketing so cursor advancement never skips rows.
|
||||
*/
|
||||
export function timeCursorOrderBy(timestampCol: PgColumn, idCol: PgColumn): [SQL, SQL] {
|
||||
return [sql`date_trunc('milliseconds', ${timestampCol}) asc`, sql`${idCol} asc`]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
import { db } from '@sim/db'
|
||||
import { workspace } from '@sim/db/schema'
|
||||
import { eq } from 'drizzle-orm'
|
||||
|
||||
/**
|
||||
* Returns the IDs of all workspaces belonging to the organization. Used by
|
||||
* sources whose underlying tables are workspace-scoped rather than org-scoped.
|
||||
*/
|
||||
export async function getOrganizationWorkspaceIds(organizationId: string): Promise<string[]> {
|
||||
const rows = await db
|
||||
.select({ id: workspace.id })
|
||||
.from(workspace)
|
||||
.where(eq(workspace.organizationId, organizationId))
|
||||
return rows.map((row) => row.id)
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
import { db } from '@sim/db'
|
||||
import { jobExecutionLogs } from '@sim/db/schema'
|
||||
import { and, inArray, isNotNull } from 'drizzle-orm'
|
||||
import {
|
||||
decodeTimeCursor,
|
||||
encodeTimeCursor,
|
||||
timeCursorOrderBy,
|
||||
timeCursorPredicate,
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
|
||||
type JobLogRow = typeof jobExecutionLogs.$inferSelect
|
||||
|
||||
/**
|
||||
* Cursors on terminal `endedAt` so in-flight rows (mutable `status`, `endedAt`,
|
||||
* `totalDurationMs`, `executionData`) are not exported until finalized.
|
||||
*/
|
||||
async function* pages(input: SourcePageInput): AsyncIterable<JobLogRow[]> {
|
||||
const workspaceIds = await getOrganizationWorkspaceIds(input.organizationId)
|
||||
if (workspaceIds.length === 0) return
|
||||
|
||||
let cursor = decodeTimeCursor(input.cursor)
|
||||
while (!input.signal.aborted) {
|
||||
const cursorClause = timeCursorPredicate(jobExecutionLogs.endedAt, jobExecutionLogs.id, cursor)
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(jobExecutionLogs)
|
||||
.where(
|
||||
and(
|
||||
inArray(jobExecutionLogs.workspaceId, workspaceIds),
|
||||
isNotNull(jobExecutionLogs.endedAt),
|
||||
cursorClause
|
||||
)
|
||||
)
|
||||
.orderBy(...timeCursorOrderBy(jobExecutionLogs.endedAt, jobExecutionLogs.id))
|
||||
.limit(input.chunkSize)
|
||||
|
||||
if (rows.length === 0) return
|
||||
yield rows
|
||||
const last = rows[rows.length - 1]
|
||||
cursor = { ts: last.endedAt!.toISOString(), id: last.id }
|
||||
if (rows.length < input.chunkSize) return
|
||||
}
|
||||
}
|
||||
|
||||
export const jobLogsSource: DrainSource<JobLogRow> = {
|
||||
type: 'job_logs',
|
||||
displayName: 'Job execution logs',
|
||||
pages,
|
||||
serialize(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
executionId: row.executionId,
|
||||
scheduleId: row.scheduleId,
|
||||
workspaceId: row.workspaceId,
|
||||
level: row.level,
|
||||
status: row.status,
|
||||
trigger: row.trigger,
|
||||
startedAt: row.startedAt.toISOString(),
|
||||
endedAt: row.endedAt ? row.endedAt.toISOString() : null,
|
||||
totalDurationMs: row.totalDurationMs,
|
||||
executionData: row.executionData,
|
||||
cost: row.cost,
|
||||
createdAt: row.createdAt.toISOString(),
|
||||
}
|
||||
},
|
||||
cursorAfter(row): Cursor {
|
||||
return encodeTimeCursor({ ts: row.endedAt!.toISOString(), id: row.id })
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import { auditLogsSource } from '@/lib/data-drains/sources/audit-logs'
|
||||
import { copilotChatsSource } from '@/lib/data-drains/sources/copilot-chats'
|
||||
import { copilotRunsSource } from '@/lib/data-drains/sources/copilot-runs'
|
||||
import { jobLogsSource } from '@/lib/data-drains/sources/job-logs'
|
||||
import { workflowLogsSource } from '@/lib/data-drains/sources/workflow-logs'
|
||||
import type { DrainSource, SourceType } from '@/lib/data-drains/types'
|
||||
|
||||
export const SOURCE_REGISTRY = {
|
||||
workflow_logs: workflowLogsSource,
|
||||
job_logs: jobLogsSource,
|
||||
audit_logs: auditLogsSource,
|
||||
copilot_chats: copilotChatsSource,
|
||||
copilot_runs: copilotRunsSource,
|
||||
} as const satisfies Record<SourceType, DrainSource>
|
||||
|
||||
export function getSource(type: SourceType): DrainSource {
|
||||
return SOURCE_REGISTRY[type]
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
import { db } from '@sim/db'
|
||||
import { workflowExecutionLogs } from '@sim/db/schema'
|
||||
import { and, inArray, isNotNull } from 'drizzle-orm'
|
||||
import {
|
||||
decodeTimeCursor,
|
||||
encodeTimeCursor,
|
||||
timeCursorOrderBy,
|
||||
timeCursorPredicate,
|
||||
} from '@/lib/data-drains/sources/cursor'
|
||||
import { getOrganizationWorkspaceIds } from '@/lib/data-drains/sources/helpers'
|
||||
import type { Cursor, DrainSource, SourcePageInput } from '@/lib/data-drains/types'
|
||||
|
||||
type WorkflowLogRow = typeof workflowExecutionLogs.$inferSelect
|
||||
|
||||
/**
|
||||
* Cursors on `endedAt` (terminal timestamp) rather than `startedAt`. A running
|
||||
* row's mutable fields (`endedAt`, `status`, `totalDurationMs`, `executionData`)
|
||||
* would otherwise be exported mid-flight and never re-emitted with their final
|
||||
* values. Filtering on `endedAt IS NOT NULL` guarantees rows are immutable
|
||||
* once visible to the drain.
|
||||
*/
|
||||
async function* pages(input: SourcePageInput): AsyncIterable<WorkflowLogRow[]> {
|
||||
const workspaceIds = await getOrganizationWorkspaceIds(input.organizationId)
|
||||
if (workspaceIds.length === 0) return
|
||||
|
||||
let cursor = decodeTimeCursor(input.cursor)
|
||||
while (!input.signal.aborted) {
|
||||
const cursorClause = timeCursorPredicate(
|
||||
workflowExecutionLogs.endedAt,
|
||||
workflowExecutionLogs.id,
|
||||
cursor
|
||||
)
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(workflowExecutionLogs)
|
||||
.where(
|
||||
and(
|
||||
inArray(workflowExecutionLogs.workspaceId, workspaceIds),
|
||||
isNotNull(workflowExecutionLogs.endedAt),
|
||||
cursorClause
|
||||
)
|
||||
)
|
||||
.orderBy(...timeCursorOrderBy(workflowExecutionLogs.endedAt, workflowExecutionLogs.id))
|
||||
.limit(input.chunkSize)
|
||||
|
||||
if (rows.length === 0) return
|
||||
yield rows
|
||||
const last = rows[rows.length - 1]
|
||||
cursor = { ts: last.endedAt!.toISOString(), id: last.id }
|
||||
if (rows.length < input.chunkSize) return
|
||||
}
|
||||
}
|
||||
|
||||
export const workflowLogsSource: DrainSource<WorkflowLogRow> = {
|
||||
type: 'workflow_logs',
|
||||
displayName: 'Workflow execution logs',
|
||||
pages,
|
||||
serialize(row) {
|
||||
return {
|
||||
id: row.id,
|
||||
executionId: row.executionId,
|
||||
workflowId: row.workflowId,
|
||||
workspaceId: row.workspaceId,
|
||||
stateSnapshotId: row.stateSnapshotId,
|
||||
deploymentVersionId: row.deploymentVersionId,
|
||||
level: row.level,
|
||||
status: row.status,
|
||||
trigger: row.trigger,
|
||||
startedAt: row.startedAt.toISOString(),
|
||||
endedAt: row.endedAt ? row.endedAt.toISOString() : null,
|
||||
totalDurationMs: row.totalDurationMs,
|
||||
executionData: row.executionData,
|
||||
cost: row.cost,
|
||||
files: row.files,
|
||||
createdAt: row.createdAt.toISOString(),
|
||||
}
|
||||
},
|
||||
cursorAfter(row): Cursor {
|
||||
return encodeTimeCursor({ ts: row.endedAt!.toISOString(), id: row.id })
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
import type { z } from 'zod'
|
||||
|
||||
export const SOURCE_TYPES = [
|
||||
'workflow_logs',
|
||||
'job_logs',
|
||||
'audit_logs',
|
||||
'copilot_chats',
|
||||
'copilot_runs',
|
||||
] as const
|
||||
|
||||
export type SourceType = (typeof SOURCE_TYPES)[number]
|
||||
|
||||
export const DESTINATION_TYPES = ['s3', 'webhook'] as const
|
||||
|
||||
export type DestinationType = (typeof DESTINATION_TYPES)[number]
|
||||
|
||||
export const CADENCE_TYPES = ['hourly', 'daily'] as const
|
||||
|
||||
export type CadenceType = (typeof CADENCE_TYPES)[number]
|
||||
|
||||
export const RUN_TRIGGERS = ['cron', 'manual'] as const
|
||||
|
||||
export type RunTrigger = (typeof RUN_TRIGGERS)[number]
|
||||
|
||||
/**
|
||||
* Opaque, source-defined cursor. Stored as text in `data_drains.cursor` and
|
||||
* round-tripped untouched. Sources may encode timestamps, ULIDs, or composite
|
||||
* keys — the runner never inspects it.
|
||||
*/
|
||||
export type Cursor = string | null
|
||||
|
||||
export interface SourcePageInput {
|
||||
organizationId: string
|
||||
cursor: Cursor
|
||||
chunkSize: number
|
||||
signal: AbortSignal
|
||||
}
|
||||
|
||||
export interface DrainSource<TRow = unknown> {
|
||||
readonly type: SourceType
|
||||
readonly displayName: string
|
||||
/**
|
||||
* Pages rows strictly newer than `cursor` in cursor-ascending order.
|
||||
* An empty iterator means no new rows.
|
||||
*/
|
||||
pages(input: SourcePageInput): AsyncIterable<TRow[]>
|
||||
/** Stable JSON-safe shape sent to destinations. Public NDJSON contract. */
|
||||
serialize(row: TRow): Record<string, unknown>
|
||||
/** Returns the cursor that, when passed back, excludes `row` and everything before it. */
|
||||
cursorAfter(row: TRow): Cursor
|
||||
}
|
||||
|
||||
export interface DeliveryMetadata {
|
||||
drainId: string
|
||||
runId: string
|
||||
source: SourceType
|
||||
/** 0-based chunk index within the run. */
|
||||
sequence: number
|
||||
rowCount: number
|
||||
/**
|
||||
* Wall-clock start of the run. Destinations that partition by date (e.g. S3
|
||||
* `YYYY/MM/DD` keys) should derive the partition from this so a single run
|
||||
* lands under one prefix even when delivery crosses a midnight boundary.
|
||||
*/
|
||||
runStartedAt: Date
|
||||
}
|
||||
|
||||
export interface DeliveryResult {
|
||||
/** Stable identifier for the written object: e.g. `s3://bucket/key` or `https://host/path`. */
|
||||
locator: string
|
||||
}
|
||||
|
||||
export interface DrainDeliverySession {
|
||||
deliver(input: {
|
||||
body: Buffer
|
||||
contentType: 'application/x-ndjson'
|
||||
metadata: DeliveryMetadata
|
||||
signal: AbortSignal
|
||||
}): Promise<DeliveryResult>
|
||||
close(): Promise<void>
|
||||
}
|
||||
|
||||
export interface DrainDestination<TConfig = unknown, TCredentials = unknown> {
|
||||
readonly type: DestinationType
|
||||
readonly displayName: string
|
||||
/** Validates non-secret config (bucket, region, prefix, url, ...) at the API boundary. */
|
||||
readonly configSchema: z.ZodType<TConfig>
|
||||
/** Validates secret payload separately so it can live in an encrypted column. */
|
||||
readonly credentialsSchema: z.ZodType<TCredentials>
|
||||
/** Optional reachability probe used by the "Test connection" UI button. */
|
||||
test?(input: { config: TConfig; credentials: TCredentials; signal: AbortSignal }): Promise<void>
|
||||
/**
|
||||
* Opens a delivery session for one drain run. Lets destinations amortize
|
||||
* expensive resources (S3Client, keep-alive connections) across all chunks
|
||||
* in a run instead of rebuilding per chunk. Caller must `close()` when done.
|
||||
*/
|
||||
openSession(input: { config: TConfig; credentials: TCredentials }): DrainDeliverySession
|
||||
}
|
||||
@@ -262,6 +262,8 @@ app:
|
||||
NEXT_PUBLIC_WHITELABELING_ENABLED: "" # Show whitelabeling settings page ("true" to enable)
|
||||
AUDIT_LOGS_ENABLED: "" # Enable audit logs on self-hosted ("true" to enable)
|
||||
NEXT_PUBLIC_AUDIT_LOGS_ENABLED: "" # Show audit logs settings page ("true" to enable)
|
||||
DATA_DRAINS_ENABLED: "" # Enable data drains on self-hosted ("true" to enable)
|
||||
NEXT_PUBLIC_DATA_DRAINS_ENABLED: "" # Show data drains settings page ("true" to enable)
|
||||
|
||||
# AWS Bedrock Credential Mode
|
||||
# Set to "true" when the deployment uses AWS default credential chain (IAM roles, instance
|
||||
@@ -1014,6 +1016,15 @@ cronjobs:
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 1
|
||||
|
||||
runDataDrains:
|
||||
enabled: true
|
||||
name: run-data-drains
|
||||
schedule: "0 * * * *"
|
||||
path: "/api/cron/run-data-drains"
|
||||
concurrencyPolicy: Forbid
|
||||
successfulJobsHistoryLimit: 3
|
||||
failedJobsHistoryLimit: 1
|
||||
|
||||
# Global CronJob settings
|
||||
image:
|
||||
repository: curlimages/curl
|
||||
|
||||
@@ -24,6 +24,13 @@ export const AuditAction = {
|
||||
CUSTOM_TOOL_UPDATED: 'custom_tool.updated',
|
||||
CUSTOM_TOOL_DELETED: 'custom_tool.deleted',
|
||||
|
||||
// Data Drains
|
||||
DATA_DRAIN_CREATED: 'data_drain.created',
|
||||
DATA_DRAIN_UPDATED: 'data_drain.updated',
|
||||
DATA_DRAIN_DELETED: 'data_drain.deleted',
|
||||
DATA_DRAIN_RAN: 'data_drain.ran',
|
||||
DATA_DRAIN_TESTED: 'data_drain.tested',
|
||||
|
||||
// Billing
|
||||
CREDIT_PURCHASED: 'credit.purchased',
|
||||
|
||||
@@ -194,6 +201,7 @@ export const AuditResourceType = {
|
||||
CREDENTIAL: 'credential',
|
||||
CREDENTIAL_SET: 'credential_set',
|
||||
CUSTOM_TOOL: 'custom_tool',
|
||||
DATA_DRAIN: 'data_drain',
|
||||
DOCUMENT: 'document',
|
||||
ENVIRONMENT: 'environment',
|
||||
FILE: 'file',
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
CREATE TYPE "public"."data_drain_cadence" AS ENUM('hourly', 'daily');--> statement-breakpoint
|
||||
CREATE TYPE "public"."data_drain_destination" AS ENUM('s3', 'webhook');--> statement-breakpoint
|
||||
CREATE TYPE "public"."data_drain_run_status" AS ENUM('running', 'success', 'failed');--> statement-breakpoint
|
||||
CREATE TYPE "public"."data_drain_run_trigger" AS ENUM('cron', 'manual');--> statement-breakpoint
|
||||
CREATE TYPE "public"."data_drain_source" AS ENUM('workflow_logs', 'job_logs', 'audit_logs', 'copilot_chats', 'copilot_runs');--> statement-breakpoint
|
||||
CREATE TABLE "data_drain_runs" (
|
||||
"id" text PRIMARY KEY NOT NULL,
|
||||
"drain_id" text NOT NULL,
|
||||
"status" "data_drain_run_status" NOT NULL,
|
||||
"trigger" "data_drain_run_trigger" NOT NULL,
|
||||
"started_at" timestamp DEFAULT now() NOT NULL,
|
||||
"finished_at" timestamp,
|
||||
"rows_exported" integer DEFAULT 0 NOT NULL,
|
||||
"bytes_written" bigint DEFAULT 0 NOT NULL,
|
||||
"cursor_before" text,
|
||||
"cursor_after" text,
|
||||
"error" text,
|
||||
"locators" jsonb DEFAULT '[]'::jsonb NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE TABLE "data_drains" (
|
||||
"id" text PRIMARY KEY NOT NULL,
|
||||
"organization_id" text NOT NULL,
|
||||
"name" text NOT NULL,
|
||||
"source" "data_drain_source" NOT NULL,
|
||||
"destination_type" "data_drain_destination" NOT NULL,
|
||||
"destination_config" jsonb NOT NULL,
|
||||
"destination_credentials" text NOT NULL,
|
||||
"schedule_cadence" "data_drain_cadence" NOT NULL,
|
||||
"enabled" boolean DEFAULT true NOT NULL,
|
||||
"cursor" text,
|
||||
"last_run_at" timestamp,
|
||||
"last_success_at" timestamp,
|
||||
"created_by" text NOT NULL,
|
||||
"created_at" timestamp DEFAULT now() NOT NULL,
|
||||
"updated_at" timestamp DEFAULT now() NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
ALTER TABLE "data_drain_runs" ADD CONSTRAINT "data_drain_runs_drain_id_data_drains_id_fk" FOREIGN KEY ("drain_id") REFERENCES "public"."data_drains"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "data_drains" ADD CONSTRAINT "data_drains_organization_id_organization_id_fk" FOREIGN KEY ("organization_id") REFERENCES "public"."organization"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint
|
||||
ALTER TABLE "data_drains" ADD CONSTRAINT "data_drains_created_by_user_id_fk" FOREIGN KEY ("created_by") REFERENCES "public"."user"("id") ON DELETE no action ON UPDATE no action;--> statement-breakpoint
|
||||
CREATE INDEX "data_drain_runs_drain_started_idx" ON "data_drain_runs" USING btree ("drain_id","started_at");--> statement-breakpoint
|
||||
CREATE INDEX "data_drains_org_idx" ON "data_drains" USING btree ("organization_id");--> statement-breakpoint
|
||||
CREATE INDEX "data_drains_due_idx" ON "data_drains" USING btree ("enabled","last_run_at");--> statement-breakpoint
|
||||
CREATE UNIQUE INDEX "data_drains_org_name_unique" ON "data_drains" USING btree ("organization_id","name");--> statement-breakpoint
|
||||
CREATE INDEX "audit_log_workspace_created_at_id_idx" ON "audit_log" USING btree ("workspace_id",date_trunc('milliseconds', "created_at"),"id");--> statement-breakpoint
|
||||
CREATE INDEX "copilot_chats_workspace_created_at_id_idx" ON "copilot_chats" USING btree ("workspace_id",date_trunc('milliseconds', "created_at"),"id");--> statement-breakpoint
|
||||
CREATE INDEX "copilot_runs_workspace_completed_at_id_idx" ON "copilot_runs" USING btree ("workspace_id",date_trunc('milliseconds', "completed_at"),"id");--> statement-breakpoint
|
||||
CREATE INDEX "job_execution_logs_workspace_ended_at_id_idx" ON "job_execution_logs" USING btree ("workspace_id",date_trunc('milliseconds', "ended_at"),"id");--> statement-breakpoint
|
||||
CREATE INDEX "workflow_execution_logs_workspace_ended_at_id_idx" ON "workflow_execution_logs" USING btree ("workspace_id",date_trunc('milliseconds', "ended_at"),"id");
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1422,6 +1422,13 @@
|
||||
"when": 1778022453620,
|
||||
"tag": "0203_curvy_quasimodo",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 204,
|
||||
"version": "7",
|
||||
"when": 1778024401275,
|
||||
"tag": "0204_powerful_medusa",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -354,6 +354,11 @@ export const workflowExecutionLogs = pgTable(
|
||||
table.workspaceId,
|
||||
table.startedAt
|
||||
),
|
||||
workspaceEndedAtIdIdx: index('workflow_execution_logs_workspace_ended_at_id_idx').on(
|
||||
table.workspaceId,
|
||||
sql`date_trunc('milliseconds', ${table.endedAt})`,
|
||||
table.id
|
||||
),
|
||||
runningStartedAtIdx: index('workflow_execution_logs_running_started_at_idx')
|
||||
.on(table.startedAt)
|
||||
.where(sql`status = 'running'`),
|
||||
@@ -588,6 +593,11 @@ export const jobExecutionLogs = pgTable(
|
||||
table.workspaceId,
|
||||
table.startedAt
|
||||
),
|
||||
workspaceEndedAtIdIdx: index('job_execution_logs_workspace_ended_at_id_idx').on(
|
||||
table.workspaceId,
|
||||
sql`date_trunc('milliseconds', ${table.endedAt})`,
|
||||
table.id
|
||||
),
|
||||
executionIdUnique: uniqueIndex('job_execution_logs_execution_id_unique').on(table.executionId),
|
||||
triggerIdx: index('job_execution_logs_trigger_idx').on(table.trigger),
|
||||
})
|
||||
@@ -1713,6 +1723,11 @@ export const copilotChats = pgTable(
|
||||
// Ordering indexes
|
||||
createdAtIdx: index('copilot_chats_created_at_idx').on(table.createdAt),
|
||||
updatedAtIdx: index('copilot_chats_updated_at_idx').on(table.updatedAt),
|
||||
workspaceCreatedAtIdIdx: index('copilot_chats_workspace_created_at_id_idx').on(
|
||||
table.workspaceId,
|
||||
sql`date_trunc('milliseconds', ${table.createdAt})`,
|
||||
table.id
|
||||
),
|
||||
})
|
||||
)
|
||||
|
||||
@@ -1844,6 +1859,11 @@ export const copilotRuns = pgTable(
|
||||
table.executionId,
|
||||
table.startedAt
|
||||
),
|
||||
workspaceCompletedAtIdIdx: index('copilot_runs_workspace_completed_at_id_idx').on(
|
||||
table.workspaceId,
|
||||
sql`date_trunc('milliseconds', ${table.completedAt})`,
|
||||
table.id
|
||||
),
|
||||
streamIdUnique: uniqueIndex('copilot_runs_stream_id_unique').on(table.streamId),
|
||||
})
|
||||
)
|
||||
@@ -2422,6 +2442,11 @@ export const auditLog = pgTable(
|
||||
table.workspaceId,
|
||||
table.createdAt
|
||||
),
|
||||
workspaceCreatedIdIdx: index('audit_log_workspace_created_at_id_idx').on(
|
||||
table.workspaceId,
|
||||
sql`date_trunc('milliseconds', ${table.createdAt})`,
|
||||
table.id
|
||||
),
|
||||
actorCreatedIdx: index('audit_log_actor_created_idx').on(table.actorId, table.createdAt),
|
||||
resourceIdx: index('audit_log_resource_idx').on(table.resourceType, table.resourceId),
|
||||
actionIdx: index('audit_log_action_idx').on(table.action),
|
||||
@@ -3094,3 +3119,90 @@ export const academyCertificate = pgTable(
|
||||
statusIdx: index('academy_certificate_status_idx').on(table.status),
|
||||
})
|
||||
)
|
||||
|
||||
export const dataDrainSourceEnum = pgEnum('data_drain_source', [
|
||||
'workflow_logs',
|
||||
'job_logs',
|
||||
'audit_logs',
|
||||
'copilot_chats',
|
||||
'copilot_runs',
|
||||
])
|
||||
|
||||
export type DataDrainSource = (typeof dataDrainSourceEnum.enumValues)[number]
|
||||
|
||||
export const dataDrainDestinationEnum = pgEnum('data_drain_destination', ['s3', 'webhook'])
|
||||
|
||||
export type DataDrainDestination = (typeof dataDrainDestinationEnum.enumValues)[number]
|
||||
|
||||
export const dataDrainCadenceEnum = pgEnum('data_drain_cadence', ['hourly', 'daily'])
|
||||
|
||||
export type DataDrainCadence = (typeof dataDrainCadenceEnum.enumValues)[number]
|
||||
|
||||
export const dataDrainRunStatusEnum = pgEnum('data_drain_run_status', [
|
||||
'running',
|
||||
'success',
|
||||
'failed',
|
||||
])
|
||||
|
||||
export type DataDrainRunStatus = (typeof dataDrainRunStatusEnum.enumValues)[number]
|
||||
|
||||
export const dataDrainRunTriggerEnum = pgEnum('data_drain_run_trigger', ['cron', 'manual'])
|
||||
|
||||
export type DataDrainRunTrigger = (typeof dataDrainRunTriggerEnum.enumValues)[number]
|
||||
|
||||
export const dataDrains = pgTable(
|
||||
'data_drains',
|
||||
{
|
||||
id: text('id').primaryKey(),
|
||||
organizationId: text('organization_id')
|
||||
.notNull()
|
||||
.references(() => organization.id, { onDelete: 'cascade' }),
|
||||
name: text('name').notNull(),
|
||||
source: dataDrainSourceEnum('source').notNull(),
|
||||
destinationType: dataDrainDestinationEnum('destination_type').notNull(),
|
||||
/** Non-secret destination config (bucket, region, prefix, url, ...). Validated by destination registry. */
|
||||
destinationConfig: jsonb('destination_config').$type<Record<string, unknown>>().notNull(),
|
||||
/** Encrypted JSON blob containing destination credentials. Never returned to clients. */
|
||||
destinationCredentials: text('destination_credentials').notNull(),
|
||||
scheduleCadence: dataDrainCadenceEnum('schedule_cadence').notNull(),
|
||||
enabled: boolean('enabled').notNull().default(true),
|
||||
/** Opaque cursor — JSON-encoded, source-defined. Advances only on overall run success. */
|
||||
cursor: text('cursor'),
|
||||
lastRunAt: timestamp('last_run_at'),
|
||||
lastSuccessAt: timestamp('last_success_at'),
|
||||
createdBy: text('created_by')
|
||||
.notNull()
|
||||
.references(() => user.id),
|
||||
createdAt: timestamp('created_at').notNull().defaultNow(),
|
||||
updatedAt: timestamp('updated_at').notNull().defaultNow(),
|
||||
},
|
||||
(table) => ({
|
||||
orgIdx: index('data_drains_org_idx').on(table.organizationId),
|
||||
dueIdx: index('data_drains_due_idx').on(table.enabled, table.lastRunAt),
|
||||
orgNameUnique: uniqueIndex('data_drains_org_name_unique').on(table.organizationId, table.name),
|
||||
})
|
||||
)
|
||||
|
||||
export const dataDrainRuns = pgTable(
|
||||
'data_drain_runs',
|
||||
{
|
||||
id: text('id').primaryKey(),
|
||||
drainId: text('drain_id')
|
||||
.notNull()
|
||||
.references(() => dataDrains.id, { onDelete: 'cascade' }),
|
||||
status: dataDrainRunStatusEnum('status').notNull(),
|
||||
trigger: dataDrainRunTriggerEnum('trigger').notNull(),
|
||||
startedAt: timestamp('started_at').notNull().defaultNow(),
|
||||
finishedAt: timestamp('finished_at'),
|
||||
rowsExported: integer('rows_exported').notNull().default(0),
|
||||
bytesWritten: bigint('bytes_written', { mode: 'number' }).notNull().default(0),
|
||||
cursorBefore: text('cursor_before'),
|
||||
cursorAfter: text('cursor_after'),
|
||||
error: text('error'),
|
||||
/** Destination-specific delivery locators for this run (e.g. S3 keys, webhook response ids). */
|
||||
locators: jsonb('locators').$type<string[]>().notNull().default(sql`'[]'::jsonb`),
|
||||
},
|
||||
(table) => ({
|
||||
drainStartedIdx: index('data_drain_runs_drain_started_idx').on(table.drainId, table.startedAt),
|
||||
})
|
||||
)
|
||||
|
||||
@@ -56,6 +56,11 @@ export const auditMock = {
|
||||
CUSTOM_TOOL_CREATED: 'custom_tool.created',
|
||||
CUSTOM_TOOL_UPDATED: 'custom_tool.updated',
|
||||
CUSTOM_TOOL_DELETED: 'custom_tool.deleted',
|
||||
DATA_DRAIN_CREATED: 'data_drain.created',
|
||||
DATA_DRAIN_UPDATED: 'data_drain.updated',
|
||||
DATA_DRAIN_DELETED: 'data_drain.deleted',
|
||||
DATA_DRAIN_RAN: 'data_drain.ran',
|
||||
DATA_DRAIN_TESTED: 'data_drain.tested',
|
||||
CONNECTOR_DOCUMENT_RESTORED: 'connector_document.restored',
|
||||
CONNECTOR_DOCUMENT_EXCLUDED: 'connector_document.excluded',
|
||||
DOCUMENT_UPLOADED: 'document.uploaded',
|
||||
@@ -156,6 +161,7 @@ export const auditMock = {
|
||||
CREDENTIAL: 'credential',
|
||||
CREDENTIAL_SET: 'credential_set',
|
||||
CUSTOM_TOOL: 'custom_tool',
|
||||
DATA_DRAIN: 'data_drain',
|
||||
DOCUMENT: 'document',
|
||||
ENVIRONMENT: 'environment',
|
||||
FILE: 'file',
|
||||
|
||||
@@ -1223,6 +1223,37 @@ export const schemaMock = {
|
||||
metadata: 'metadata',
|
||||
createdAt: 'createdAt',
|
||||
},
|
||||
dataDrains: {
|
||||
id: 'id',
|
||||
organizationId: 'organizationId',
|
||||
name: 'name',
|
||||
source: 'source',
|
||||
destinationType: 'destinationType',
|
||||
destinationConfig: 'destinationConfig',
|
||||
destinationCredentials: 'destinationCredentials',
|
||||
scheduleCadence: 'scheduleCadence',
|
||||
enabled: 'enabled',
|
||||
cursor: 'cursor',
|
||||
lastRunAt: 'lastRunAt',
|
||||
lastSuccessAt: 'lastSuccessAt',
|
||||
createdBy: 'createdBy',
|
||||
createdAt: 'createdAt',
|
||||
updatedAt: 'updatedAt',
|
||||
},
|
||||
dataDrainRuns: {
|
||||
id: 'id',
|
||||
drainId: 'drainId',
|
||||
status: 'status',
|
||||
trigger: 'trigger',
|
||||
startedAt: 'startedAt',
|
||||
finishedAt: 'finishedAt',
|
||||
rowsExported: 'rowsExported',
|
||||
bytesWritten: 'bytesWritten',
|
||||
cursorBefore: 'cursorBefore',
|
||||
cursorAfter: 'cursorAfter',
|
||||
error: 'error',
|
||||
locators: 'locators',
|
||||
},
|
||||
/** Custom type export for tsvector */
|
||||
tsvector: 'tsvector',
|
||||
}
|
||||
|
||||
@@ -9,8 +9,8 @@ const QUERY_HOOKS_DIR = path.join(ROOT, 'apps/sim/hooks/queries')
|
||||
const SELECTOR_HOOKS_DIR = path.join(ROOT, 'apps/sim/hooks/selectors')
|
||||
|
||||
const BASELINE = {
|
||||
totalRoutes: 727,
|
||||
zodRoutes: 727,
|
||||
totalRoutes: 733,
|
||||
zodRoutes: 733,
|
||||
nonZodRoutes: 0,
|
||||
} as const
|
||||
|
||||
@@ -70,6 +70,7 @@ const INDIRECT_ZOD_ROUTES = new Set([
|
||||
'apps/sim/app/api/cron/cleanup-soft-deletes/route.ts',
|
||||
'apps/sim/app/api/cron/cleanup-stale-executions/route.ts',
|
||||
'apps/sim/app/api/cron/renew-subscriptions/route.ts',
|
||||
'apps/sim/app/api/cron/run-data-drains/route.ts',
|
||||
'apps/sim/app/api/logs/cleanup/route.ts',
|
||||
'apps/sim/app/api/knowledge/connectors/sync/route.ts',
|
||||
'apps/sim/app/api/webhooks/outbox/process/route.ts',
|
||||
|
||||
Reference in New Issue
Block a user