mirror of
https://github.com/simstudioai/sim.git
synced 2026-09-01 14:59:19 +08:00
feat(data-drains): add GCS, Azure Blob, BigQuery, Snowflake, and Datadog destinations (#4552)
* feat(data-drains): add GCS, Azure Blob, BigQuery, Snowflake, and Datadog destinations
* fix(data-drains): address PR review comments
* fix(data-drains): extract sleepUntilAborted, honor abort across all destinations
* fix(data-drains): widen BigQuery projectId max and dedupe parseServiceAccount
* fix(data-drains): tighten GCS bucket contract and expose Azure endpointSuffix
* improvement(data-drains): extract normalizePrefix and buildObjectKey to shared utils
* fix(data-drains): retry BigQuery network errors; tighten Azure accountKey contract
- BigQuery insertAll now wraps the fetch in try/catch inside the retry loop so DNS failures, socket resets, and timeouts are retried with backoff instead of propagating immediately.
- Align azureBlobCredentialsBodySchema with the runtime schema (min 64 / max 120 / base64 regex) so obviously invalid keys are rejected at the API boundary rather than at drain-run time.
* improvement(data-drains): consolidate parseRetryAfter; add Datadog NDJSON line context
- Extract a single parseRetryAfter helper (capped at 30s, returns number | null) into lib/data-drains/destinations/utils.ts and remove the five local copies in bigquery, datadog, gcs, snowflake, and webhook.
- Datadog parseNdjson now wraps JSON.parse in try/catch and surfaces the failing line index, matching BigQuery's parser.
* fix(data-drains): correct Datadog size guard and Snowflake VARIANT limit
- Datadog payload guard now checks the uncompressed size against the 5 MB limit and the wire size against the 6 MB compressed limit, so gzip cannot smuggle an oversized body past the client-side check.
- Snowflake VARIANT limit is 16 MiB (16,777,216 bytes), not 16,000,000 bytes — small payloads between 16 MB and 16 MiB were being rejected unnecessarily.
- Drop the unused apiKey field on Datadog PostInput; the key is already embedded in the prepared request headers.
* improvement(data-drains): consolidate backoffWithJitter into shared utils
Datadog, GCS, and webhook each had byte-identical backoff helpers (BASE 500ms, MAX 30s, jitter ±20%, Retry-After floor). Lift the helper into lib/data-drains/destinations/utils.ts alongside parseRetryAfter and sleepUntilAborted, and drop the per-file copies and their BASE_BACKOFF_MS/MAX_BACKOFF_MS constants.
* fix(data-drains): align destinations with live provider specs
Audited every destination against live AWS/GCS/Azure/BigQuery/Snowflake/
Datadog/webhook docs and applied spec-correctness fixes:
- S3: reserved bucket prefix amzn-s3-demo-, suffixes --x-s3/--table-s3;
metadata byte formula excludes x-amz-meta- prefix per AWS spec
- GCS: reject -./.- adjacency; UTF-8 prefix cap; forbid .well-known/
acme-challenge/ prefix; ASCII-only x-goog-meta-* enforcement
- BigQuery: insertId is 128 chars (not bytes); split DATASET_RE (ASCII)
and TABLE_RE (Unicode L/M/N + connectors); UTF-8 byte cap on tableId
- Snowflake: disambiguate org-account vs legacy locator account formats;
requestId+retry=true for idempotent retries; server-side timeout=600;
default column DATA uppercase to match unquoted canonical form
- Azure: endpoint suffix allowlist (4 sovereign clouds); accountKey
length(88) base64
- Webhook: url max(2048); CRLF/NUL rejection on bearer/secret/sig header
* fix(data-drains): address PR review on snowflake poll + shared NDJSON parsing
- snowflake pollStatement: per-attempt timeout via AbortSignal.any, retry on 429/5xx with Retry-After + jitter
- bigquery parseNdjson error messages now 1-indexed
- consolidate parseNdjson variants into shared parseNdjsonLines/parseNdjsonObjects in utils
* fix(data-drains): per-attempt fetch timeouts in gcs/bigquery, snowflake poll double-sleep
- gcs.fetchWithRetry + bigquery.postInsertAll now use AbortSignal.any with a per-attempt timeout so a hung TCP connection cannot stall the drain
- snowflake.pollStatement skips the next interval sleep when it just slept for retry backoff
* fix(data-drains): bigquery probe timeout + jittered retries, align Snowflake column default UI/docs
- bigquery test() probe now uses AbortSignal.any + per-attempt timeout
- bigquery insertAll retry switches to backoffWithJitter for thundering-herd avoidance
- Snowflake column placeholder + docs say DATA (uppercase) to match the code default
* fix(data-drains): mirror webhook signingSecret min length in form gate
isComplete now requires signingSecret >= 32 to match the contract/runtime
schema so the Save button can't enable on a value that will fail server-side.
* fix(data-drains): validate JSON client-side for Snowflake before binding
Switch Snowflake to parseNdjsonObjects so malformed rows are caught locally
with 1-indexed line numbers instead of failing the whole INSERT server-side.
Re-stringify each parsed object before binding to PARSE_JSON(?).
Drop the now-unused parseNdjsonLines helper.
* fix(data-drains): cross-cutting audit pass against live provider docs
- Azure: bound retryOptions on BlobServiceClient (SDK default tryTimeoutInMs is per-try unbounded; cap at 30s x 5 tries)
- Webhook contract: mirror runtime — signingSecret.max(512), bearerToken.max(4096) + CRLF/NUL refine, signatureHeader charset + CRLF/NUL refine
- S3 (lib + contract): reject bucket names with dash adjacent to dot; require https:// endpoint at the schema layer
- Snowflake: bind original NDJSON line bytes (re-stringifying a JSON.parse'd value loses bigint precision beyond 2^53-1); check pollStatement 200 body for the SQL error envelope (sqlState/code)
- Datadog: entry builder writes defaults first then user attrs then forced ddtags/message so user rows can't clobber routing fields; validate config.tags as comma-separated key:value pairs
- registry.tsx: tighten isComplete predicates to mirror contract minimums (GCS bucket >= 3, Azure containerName >= 3 / accountKey === 88, BigQuery projectId >= 6, Snowflake account >= 3)
* fix(data-drains): force ddsource/service overrides on Datadog entries
Previous fix placed ddsource/service before ...attrs, leaving them clobberable
by a user row field. Per Datadog docs, service + ddsource pick the processing
pipeline, so a drain's routing config must not be overridable per-row. Spread
attrs first, then force all four reserved fields (ddsource, service, ddtags,
message).
* fix(data-drains): preserve row-distinguishing index when BigQuery insertId overflows
Truncating from the left dropped the index suffix, so any overflow would
collapse all rows in a chunk to the same insertId and BigQuery would silently
dedupe them. Path is unreachable today (UUIDs keep raw ~85 chars), but the
overflow branch is now correct: hash the prefix, keep the index intact.
* fix(data-drains): refresh GCS token per retry, tighten Azure key regex
- gcs: rebuild Authorization header per attempt via buildHeaders so token
refresh from google-auth-library kicks in if a 5xx retry crosses the
hour-long token lifetime
- azure_blob: pin account-key regex to {0,2} trailing '=' (base64 of 64
bytes = exactly 88 chars with up to two '=' pad chars)
* fix(data-drains): address bugbot review of 6336948f6
- gcs: allow 1-char dot-separated bucket components (e.g. "a.bucket")
to match GCS naming rules — overall name is 3-63 (or up to 222 with
dots), but per-component minimum is 1 per Google's spec
- bigquery: drain the 401 response body before re-issuing the request
with a refreshed token so undici can return the socket to the
keep-alive pool
- snowflake: hoist getJwt() above the perAttempt timer in
executeStatement so JWT signing doesn't eat the network budget
(matches the order already used in pollStatement)
* fix(data-drains): allow org-account Snowflake identifier with region suffix
The account validation rejected `<orgname>-<acctname>.<region>.<cloud>`
because `ACCOUNT_LOCATOR_RE`'s first segment forbade hyphens, while
`ACCOUNT_ORG_RE` forbade dots. `normalizeAccountForJwt` already handles
this composite form. Widen the first segment of `ACCOUNT_LOCATOR_RE` to
allow hyphens so the boundary contract and the runtime schema accept
what the JWT layer was already designed to process.
* fix(data-drains): drain retryable response bodies in datadog/gcs loops
Mirrors the bigquery 401 fix. Without consuming the body before
sleeping, undici can't return the socket to the keep-alive pool, so
each retry leaks a TCP connection instead of reusing it.
* fix(data-drains): drain snowflake poll bodies on 202 and retryable status
Mirrors the bigquery/datadog/gcs drains. Long async statements can poll
many times against the same connection; without consuming the body
undici can't return the socket to the keep-alive pool, so each iteration
leaks a connection until GC.
* fix(data-drains): consume success bodies; check Snowflake sqlState on 200
- gcs: drain the body on success paths so undici can return the socket
to the keep-alive pool
- snowflake: drain the body on synchronous 200 OK and run the same
sqlState envelope check pollStatement already does — otherwise a
statement-level failure that completes synchronously would silently
return success
* fix(data-drains): drain datadog and bigquery probe success bodies
Same undici keep-alive issue as the prior fixes: postWithRetries
returned the Response on success without draining (callers only read
headers); the BigQuery `test()` probe returned without consuming the
body. Both now drain before returning.
* chore(data-drains): regenerate enum migration as 0206 after staging rebase
* fix(data-drains): cap snowflake poll retries; tighten datadog tags min length
This commit is contained in:
Waleed
@@ -1,11 +1,11 @@
|
||||
---
|
||||
title: Data Drains
|
||||
description: Continuously export workflow logs, audit logs, and Mothership data to your own S3 bucket or HTTPS endpoint on a schedule
|
||||
description: Continuously export workflow logs, audit logs, and Mothership data to your own object store, data warehouse, observability platform, or HTTPS endpoint on a schedule
|
||||
---
|
||||
|
||||
import { FAQ } from '@/components/ui/faq'
|
||||
|
||||
Data Drains let organization owners and admins on Enterprise plans continuously export Sim data to a destination they control — a customer-owned S3 bucket or an HTTPS webhook. A drain runs on a schedule, picks up only new rows since its last successful run, and writes them as NDJSON to the destination. Viewing drain configuration and run history is restricted to owners and admins as well, since destinations expose internal bucket names and webhook URLs.
|
||||
Data Drains let organization owners and admins on Enterprise plans continuously export Sim data to a destination they control — a customer-owned S3 bucket, Google Cloud Storage bucket, Azure Blob container, BigQuery table, Snowflake table, Datadog logs intake, or an HTTPS webhook. A drain runs on a schedule, picks up only new rows since its last successful run, and writes them to the destination. Viewing drain configuration and run history is restricted to owners and admins as well, since destinations expose internal bucket names, table identifiers, and webhook URLs.
|
||||
|
||||
Drains are independent of [Data Retention](/enterprise/data-retention) but designed to compose with it — see [Pairing with Data Retention](#pairing-with-data-retention) below.
|
||||
|
||||
@@ -67,6 +67,62 @@ Object keys are deterministic:
|
||||
|
||||
Objects are written with `AES256` server-side encryption.
|
||||
|
||||
### Google Cloud Storage
|
||||
|
||||
Writes one NDJSON object per delivered chunk to your GCS bucket.
|
||||
|
||||
- **Bucket** — the bucket name. Must already exist; Sim does not create buckets.
|
||||
- **Prefix** *(optional)* — folder path inside the bucket. Trailing slash optional.
|
||||
- **Service account JSON key** — paste the full JSON key for a service account with `storage.objects.create` (and `storage.objects.delete` if you want "Test connection" to clean up its probe). Sim authenticates via OAuth2 service-account JWT and uploads through the GCS JSON API.
|
||||
|
||||
Object names follow the same `{prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson` layout as S3. Object metadata mirrors the S3 destination's `sim-*` keys via `x-goog-meta-*` headers.
|
||||
|
||||
### Azure Blob Storage
|
||||
|
||||
Writes one NDJSON block blob per delivered chunk to your container.
|
||||
|
||||
- **Account name** — your storage account (3–24 lowercase chars).
|
||||
- **Container** — must already exist; Sim does not create containers.
|
||||
- **Prefix** *(optional)* — folder path inside the container.
|
||||
- **Account key** — a storage account access key with write access to the container.
|
||||
|
||||
Blob names follow the same `{prefix}/{source}/{drainId}/{yyyy}/{mm}/{dd}/{runId}-{seq}.ndjson` layout. The `sim-*` metadata is exposed as Azure blob metadata (collapsed to lowercase per Azure's identifier rules).
|
||||
|
||||
For sovereign clouds, set **Endpoint suffix** to `blob.core.usgovcloudapi.net` (US Gov), `blob.core.chinacloudapi.cn` (China), or `blob.core.cloudapi.de` (Germany).
|
||||
|
||||
### Google BigQuery
|
||||
|
||||
Streams each row into a target table via the `tabledata.insertAll` API, with per-row insertId dedup.
|
||||
|
||||
- **Project ID** — your GCP project (supports domain-scoped IDs like `example.com:my-project`).
|
||||
- **Dataset ID / Table ID** — must already exist; Sim does not create tables. The table schema must accommodate the source's row shape (one column per top-level field, or a single `JSON`/`STRING` column with the rest as `ignoreUnknownValues`).
|
||||
- **Service account JSON key** — needs `roles/bigquery.dataEditor` (insert) and `roles/bigquery.metadataViewer` (for the `tables.get` probe used by "Test connection"). Sim authenticates via OAuth2 service-account JWT.
|
||||
|
||||
Each row is sent with an `insertId` of `{drainId}-{runId}-{sequence}-{index}`. BigQuery dedupes inserts with the same `insertId` for ~60 seconds, so retries inside that window won't duplicate. If a chunk reports partial failure (`insertErrors`), the run fails with the offending row indices and an outer-driver retry may duplicate rows that already succeeded — the dispatcher's bounded retry minimizes this risk. Enforced per-request limits: 10 MB body, 50,000 rows.
|
||||
|
||||
### Snowflake
|
||||
|
||||
Inserts each row into a target VARIANT column via the Snowflake SQL API v2 with key-pair JWT auth.
|
||||
|
||||
- **Account** — the Snowflake account identifier. Preferred form is `<orgname>-<acctname>` (no dots). Legacy `<locator>.<region>.<cloud>` is also accepted.
|
||||
- **User / Warehouse / Database / Schema / Table** — must already exist. The user needs `INSERT` privilege on the table and `USAGE` on the warehouse, database, and schema.
|
||||
- **Column** *(optional)* — target VARIANT column name. Defaults to `DATA` (matches Snowflake's unquoted identifier folding).
|
||||
- **Role** *(optional)* — Snowflake role to assume.
|
||||
- **Private key (PEM)** — PKCS8-encoded RSA private key. Register the matching public key on the Snowflake user via `ALTER USER ... SET RSA_PUBLIC_KEY = '...'`.
|
||||
|
||||
Each chunk becomes a single `INSERT INTO "DB"."SCHEMA"."TABLE" ("col") VALUES (PARSE_JSON(?)), ...` with one TEXT binding per row. Identifiers are quoted to preserve case. The destination handles Snowflake's async 202-then-poll pattern transparently. Per-row JSON payloads are capped at 16 MB to match Snowflake's VARIANT limit.
|
||||
|
||||
### Datadog Logs
|
||||
|
||||
POSTs each row as a log entry to Datadog's v2 logs intake.
|
||||
|
||||
- **Site** — your Datadog site: `us1`, `us3`, `us5`, `eu1`, `ap1`, `ap2`, or `gov`.
|
||||
- **Service** *(optional)* — value for the reserved `service` field. Defaults to `sim`.
|
||||
- **Tags** *(optional)* — comma-separated `ddtags` appended to every entry alongside auto-injected `sim_drain_id:`, `sim_run_id:`, and `sim_source:` tags.
|
||||
- **API key** — a Datadog API key (not an Application key) with logs-write permission.
|
||||
|
||||
Top-level row fields are auto-indexed as Datadog log attributes. The reserved fields `ddsource`, `service`, `ddtags`, and `message` are always set by Sim and override anything in the row. Payloads above 1 KB are gzip-compressed. Enforced limits match Datadog's intake: 5 MB per request (post-compression), 1000 entries per request, 1 MB per entry.
|
||||
|
||||
### HTTPS Webhook
|
||||
|
||||
POSTs each chunk as NDJSON to your endpoint.
|
||||
|
||||
@@ -6849,3 +6849,26 @@ export function HexIcon(props: SVGProps<SVGSVGElement>) {
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
|
||||
export function BigQueryIcon(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg viewBox='0 0 24 24' xmlns='http://www.w3.org/2000/svg' {...props}>
|
||||
<path
|
||||
fill='#4386FA'
|
||||
d='M12 2.5a9.5 9.5 0 1 0 5.81 17.02l3.4 3.4a1 1 0 0 0 1.41-1.42l-3.4-3.4A9.5 9.5 0 0 0 12 2.5Zm0 2a7.5 7.5 0 1 1 0 15 7.5 7.5 0 0 1 0-15Z'
|
||||
/>
|
||||
<path fill='#4386FA' d='M8 11h1.6v4H8v-4Zm3 -2h1.6v6H11V9Zm3 1.5h1.6V15H14v-3.5Z' />
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
|
||||
export function SnowflakeIcon(props: SVGProps<SVGSVGElement>) {
|
||||
return (
|
||||
<svg viewBox='0 0 24 24' xmlns='http://www.w3.org/2000/svg' {...props}>
|
||||
<path
|
||||
fill='#29B5E8'
|
||||
d='M12 2a1 1 0 0 1 1 1v3.59l2.3-2.3a1 1 0 1 1 1.4 1.42L13 9.41V12h2.6l3.7-3.7a1 1 0 0 1 1.4 1.4L18.42 12H22a1 1 0 1 1 0 2h-3.59l2.3 2.3a1 1 0 0 1-1.4 1.4L15.58 14H13v2.59l3.7 3.7a1 1 0 1 1-1.4 1.4L13 19.42V23a1 1 0 1 1-2 0v-3.58l-2.3 2.3a1 1 0 1 1-1.4-1.4l3.7-3.71V14H8.4l-3.7 3.7a1 1 0 0 1-1.4-1.4L5.58 14H2a1 1 0 0 1 0-2h3.59l-2.3-2.3a1 1 0 0 1 1.4-1.4L8.42 12H11V9.41L7.3 5.71a1 1 0 1 1 1.4-1.42l2.3 2.3V3a1 1 0 0 1 1-1Z'
|
||||
/>
|
||||
</svg>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -30,7 +30,14 @@ import {
|
||||
TableRow,
|
||||
toast,
|
||||
} from '@/components/emcn'
|
||||
import { S3Icon } from '@/components/icons'
|
||||
import {
|
||||
AzureIcon,
|
||||
BigQueryIcon,
|
||||
DatadogIcon,
|
||||
GoogleIcon,
|
||||
S3Icon,
|
||||
SnowflakeIcon,
|
||||
} from '@/components/icons'
|
||||
import { Input as BaseInput } from '@/components/ui'
|
||||
import type { CreateDataDrainBody, DataDrain, DataDrainRun } from '@/lib/api/contracts/data-drains'
|
||||
import { useSession } from '@/lib/auth/auth-client'
|
||||
@@ -62,6 +69,11 @@ const SOURCE_LABELS: Record<(typeof SOURCE_TYPES)[number], string> = {
|
||||
|
||||
const DESTINATION_LABELS: Record<(typeof DESTINATION_TYPES)[number], string> = {
|
||||
s3: 'Amazon S3',
|
||||
gcs: 'Google Cloud Storage',
|
||||
azure_blob: 'Azure Blob Storage',
|
||||
datadog: 'Datadog',
|
||||
bigquery: 'Google BigQuery',
|
||||
snowflake: 'Snowflake',
|
||||
webhook: 'HTTPS webhook',
|
||||
}
|
||||
|
||||
@@ -73,8 +85,22 @@ const CADENCE_LABELS: Record<(typeof CADENCE_TYPES)[number], string> = {
|
||||
const SOURCE_OPTIONS = SOURCE_TYPES.map((t) => ({ value: t, label: SOURCE_LABELS[t] }))
|
||||
const CADENCE_OPTIONS = CADENCE_TYPES.map((t) => ({ value: t, label: CADENCE_LABELS[t] }))
|
||||
function getDestinationIcon(type: (typeof DESTINATION_TYPES)[number]) {
|
||||
if (type !== 's3') return null
|
||||
return <S3Icon className='size-[14px] flex-shrink-0 text-[#1B660F]' />
|
||||
switch (type) {
|
||||
case 's3':
|
||||
return <S3Icon className='size-[14px] flex-shrink-0 text-[#1B660F]' />
|
||||
case 'gcs':
|
||||
return <GoogleIcon className='size-[14px] flex-shrink-0' />
|
||||
case 'azure_blob':
|
||||
return <AzureIcon className='size-[14px] flex-shrink-0' />
|
||||
case 'datadog':
|
||||
return <DatadogIcon className='size-[14px] flex-shrink-0' />
|
||||
case 'bigquery':
|
||||
return <BigQueryIcon className='size-[14px] flex-shrink-0' />
|
||||
case 'snowflake':
|
||||
return <SnowflakeIcon className='size-[14px] flex-shrink-0' />
|
||||
default:
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
const DESTINATION_OPTIONS = DESTINATION_TYPES.map((t) => ({
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use client'
|
||||
|
||||
import type { ComponentType } from 'react'
|
||||
import { FormField, Input, SecretInput, Switch } from '@/components/emcn'
|
||||
import { Combobox, FormField, Input, SecretInput, Switch, Textarea } from '@/components/emcn'
|
||||
import type { CreateDataDrainBody } from '@/lib/api/contracts/data-drains'
|
||||
import type { DestinationType } from '@/lib/data-drains/types'
|
||||
|
||||
@@ -111,6 +111,340 @@ const s3FormSpec: DestinationFormSpec<S3State> = {
|
||||
}),
|
||||
}
|
||||
|
||||
interface GCSState {
|
||||
bucket: string
|
||||
prefix: string
|
||||
serviceAccountJson: string
|
||||
}
|
||||
|
||||
const gcsFormSpec: DestinationFormSpec<GCSState> = {
|
||||
displayName: 'Google Cloud Storage',
|
||||
initialState: { bucket: '', prefix: '', serviceAccountJson: '' },
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='Bucket'>
|
||||
<Input
|
||||
value={state.bucket}
|
||||
onChange={(e) => setState({ ...state, bucket: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Prefix (optional)'>
|
||||
<Input
|
||||
value={state.prefix}
|
||||
onChange={(e) => setState({ ...state, prefix: e.target.value })}
|
||||
placeholder='exports/sim'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Service account JSON key'>
|
||||
<Textarea
|
||||
value={state.serviceAccountJson}
|
||||
onChange={(e) => setState({ ...state, serviceAccountJson: e.target.value })}
|
||||
placeholder='{ "type": "service_account", ... }'
|
||||
rows={6}
|
||||
/>
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) => s.bucket.length >= 3 && s.serviceAccountJson.length > 0,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'gcs',
|
||||
destinationConfig: { bucket: s.bucket, prefix: s.prefix || undefined },
|
||||
destinationCredentials: { serviceAccountJson: s.serviceAccountJson },
|
||||
}),
|
||||
}
|
||||
|
||||
interface AzureBlobState {
|
||||
accountName: string
|
||||
containerName: string
|
||||
prefix: string
|
||||
endpointSuffix: string
|
||||
accountKey: string
|
||||
}
|
||||
|
||||
const azureBlobFormSpec: DestinationFormSpec<AzureBlobState> = {
|
||||
displayName: 'Azure Blob Storage',
|
||||
initialState: {
|
||||
accountName: '',
|
||||
containerName: '',
|
||||
prefix: '',
|
||||
endpointSuffix: '',
|
||||
accountKey: '',
|
||||
},
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='Account name'>
|
||||
<Input
|
||||
value={state.accountName}
|
||||
onChange={(e) => setState({ ...state, accountName: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Container'>
|
||||
<Input
|
||||
value={state.containerName}
|
||||
onChange={(e) => setState({ ...state, containerName: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Prefix (optional)'>
|
||||
<Input
|
||||
value={state.prefix}
|
||||
onChange={(e) => setState({ ...state, prefix: e.target.value })}
|
||||
placeholder='exports/sim'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Endpoint suffix (optional)'>
|
||||
<Input
|
||||
value={state.endpointSuffix}
|
||||
onChange={(e) => setState({ ...state, endpointSuffix: e.target.value })}
|
||||
placeholder='blob.core.windows.net'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Account key'>
|
||||
<SecretInput
|
||||
value={state.accountKey}
|
||||
onChange={(v) => setState({ ...state, accountKey: v })}
|
||||
/>
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) =>
|
||||
s.accountName.length >= 3 && s.containerName.length >= 3 && s.accountKey.length === 88,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'azure_blob',
|
||||
destinationConfig: {
|
||||
accountName: s.accountName,
|
||||
containerName: s.containerName,
|
||||
prefix: s.prefix || undefined,
|
||||
endpointSuffix: s.endpointSuffix || undefined,
|
||||
},
|
||||
destinationCredentials: { accountKey: s.accountKey },
|
||||
}),
|
||||
}
|
||||
|
||||
const DATADOG_SITE_OPTIONS = [
|
||||
{ value: 'us1', label: 'US1 (datadoghq.com)' },
|
||||
{ value: 'us3', label: 'US3 (us3.datadoghq.com)' },
|
||||
{ value: 'us5', label: 'US5 (us5.datadoghq.com)' },
|
||||
{ value: 'eu1', label: 'EU1 (datadoghq.eu)' },
|
||||
{ value: 'ap1', label: 'AP1 (ap1.datadoghq.com)' },
|
||||
{ value: 'ap2', label: 'AP2 (ap2.datadoghq.com)' },
|
||||
{ value: 'gov', label: 'Gov (ddog-gov.com)' },
|
||||
]
|
||||
|
||||
interface DatadogState {
|
||||
site: 'us1' | 'us3' | 'us5' | 'eu1' | 'ap1' | 'ap2' | 'gov'
|
||||
service: string
|
||||
tags: string
|
||||
apiKey: string
|
||||
}
|
||||
|
||||
const datadogFormSpec: DestinationFormSpec<DatadogState> = {
|
||||
displayName: 'Datadog',
|
||||
initialState: { site: 'us1', service: '', tags: '', apiKey: '' },
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='Site'>
|
||||
<Combobox
|
||||
value={state.site}
|
||||
onChange={(v) => setState({ ...state, site: v as DatadogState['site'] })}
|
||||
options={DATADOG_SITE_OPTIONS}
|
||||
dropdownWidth='trigger'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Service (optional)'>
|
||||
<Input
|
||||
value={state.service}
|
||||
onChange={(e) => setState({ ...state, service: e.target.value })}
|
||||
placeholder='sim'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Tags (optional, comma-separated)'>
|
||||
<Input
|
||||
value={state.tags}
|
||||
onChange={(e) => setState({ ...state, tags: e.target.value })}
|
||||
placeholder='env:prod,team:platform'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='API key'>
|
||||
<SecretInput value={state.apiKey} onChange={(v) => setState({ ...state, apiKey: v })} />
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) => s.apiKey.length > 0,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'datadog',
|
||||
destinationConfig: {
|
||||
site: s.site,
|
||||
service: s.service || undefined,
|
||||
tags: s.tags || undefined,
|
||||
},
|
||||
destinationCredentials: { apiKey: s.apiKey },
|
||||
}),
|
||||
}
|
||||
|
||||
interface BigQueryState {
|
||||
projectId: string
|
||||
datasetId: string
|
||||
tableId: string
|
||||
serviceAccountJson: string
|
||||
}
|
||||
|
||||
const bigqueryFormSpec: DestinationFormSpec<BigQueryState> = {
|
||||
displayName: 'Google BigQuery',
|
||||
initialState: { projectId: '', datasetId: '', tableId: '', serviceAccountJson: '' },
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='Project ID'>
|
||||
<Input
|
||||
value={state.projectId}
|
||||
onChange={(e) => setState({ ...state, projectId: e.target.value })}
|
||||
placeholder='my-gcp-project'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Dataset'>
|
||||
<Input
|
||||
value={state.datasetId}
|
||||
onChange={(e) => setState({ ...state, datasetId: e.target.value })}
|
||||
placeholder='sim_drains'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Table'>
|
||||
<Input
|
||||
value={state.tableId}
|
||||
onChange={(e) => setState({ ...state, tableId: e.target.value })}
|
||||
placeholder='workflow_logs'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Service account JSON key'>
|
||||
<Textarea
|
||||
value={state.serviceAccountJson}
|
||||
onChange={(e) => setState({ ...state, serviceAccountJson: e.target.value })}
|
||||
placeholder='{ "type": "service_account", ... }'
|
||||
rows={6}
|
||||
/>
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) =>
|
||||
s.projectId.length >= 6 &&
|
||||
s.datasetId.length > 0 &&
|
||||
s.tableId.length > 0 &&
|
||||
s.serviceAccountJson.length > 0,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'bigquery',
|
||||
destinationConfig: { projectId: s.projectId, datasetId: s.datasetId, tableId: s.tableId },
|
||||
destinationCredentials: { serviceAccountJson: s.serviceAccountJson },
|
||||
}),
|
||||
}
|
||||
|
||||
interface SnowflakeState {
|
||||
account: string
|
||||
user: string
|
||||
warehouse: string
|
||||
database: string
|
||||
schema: string
|
||||
table: string
|
||||
column: string
|
||||
role: string
|
||||
privateKey: string
|
||||
}
|
||||
|
||||
const snowflakeFormSpec: DestinationFormSpec<SnowflakeState> = {
|
||||
displayName: 'Snowflake',
|
||||
initialState: {
|
||||
account: '',
|
||||
user: '',
|
||||
warehouse: '',
|
||||
database: '',
|
||||
schema: '',
|
||||
table: '',
|
||||
column: '',
|
||||
role: '',
|
||||
privateKey: '',
|
||||
},
|
||||
FormFields: ({ state, setState }) => (
|
||||
<>
|
||||
<FormField label='Account identifier'>
|
||||
<Input
|
||||
value={state.account}
|
||||
onChange={(e) => setState({ ...state, account: e.target.value })}
|
||||
placeholder='orgname-accountname'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='User'>
|
||||
<Input
|
||||
value={state.user}
|
||||
onChange={(e) => setState({ ...state, user: e.target.value })}
|
||||
placeholder='SIM_DRAIN_USER'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Warehouse'>
|
||||
<Input
|
||||
value={state.warehouse}
|
||||
onChange={(e) => setState({ ...state, warehouse: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Database'>
|
||||
<Input
|
||||
value={state.database}
|
||||
onChange={(e) => setState({ ...state, database: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Schema'>
|
||||
<Input
|
||||
value={state.schema}
|
||||
onChange={(e) => setState({ ...state, schema: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Table'>
|
||||
<Input
|
||||
value={state.table}
|
||||
onChange={(e) => setState({ ...state, table: e.target.value })}
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Column (optional, defaults to "DATA")'>
|
||||
<Input
|
||||
value={state.column}
|
||||
onChange={(e) => setState({ ...state, column: e.target.value })}
|
||||
placeholder='DATA'
|
||||
/>
|
||||
</FormField>
|
||||
<FormField label='Role (optional)'>
|
||||
<Input value={state.role} onChange={(e) => setState({ ...state, role: e.target.value })} />
|
||||
</FormField>
|
||||
<FormField label='Private key (PEM, PKCS8)'>
|
||||
<Textarea
|
||||
value={state.privateKey}
|
||||
onChange={(e) => setState({ ...state, privateKey: e.target.value })}
|
||||
placeholder='-----BEGIN PRIVATE KEY-----'
|
||||
rows={6}
|
||||
/>
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) =>
|
||||
s.account.length >= 3 &&
|
||||
s.user.length > 0 &&
|
||||
s.warehouse.length > 0 &&
|
||||
s.database.length > 0 &&
|
||||
s.schema.length > 0 &&
|
||||
s.table.length > 0 &&
|
||||
s.privateKey.length > 0,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'snowflake',
|
||||
destinationConfig: {
|
||||
account: s.account,
|
||||
user: s.user,
|
||||
warehouse: s.warehouse,
|
||||
database: s.database,
|
||||
schema: s.schema,
|
||||
table: s.table,
|
||||
column: s.column || undefined,
|
||||
role: s.role || undefined,
|
||||
},
|
||||
destinationCredentials: { privateKey: s.privateKey },
|
||||
}),
|
||||
}
|
||||
|
||||
interface WebhookState {
|
||||
url: string
|
||||
signatureHeader: string
|
||||
@@ -151,7 +485,7 @@ const webhookFormSpec: DestinationFormSpec<WebhookState> = {
|
||||
</FormField>
|
||||
</>
|
||||
),
|
||||
isComplete: (s) => s.url.length > 0 && s.signingSecret.length >= 8,
|
||||
isComplete: (s) => s.url.length > 0 && s.signingSecret.length >= 32,
|
||||
toDestinationBranch: (s) => ({
|
||||
destinationType: 'webhook',
|
||||
destinationConfig: {
|
||||
@@ -172,5 +506,10 @@ const webhookFormSpec: DestinationFormSpec<WebhookState> = {
|
||||
*/
|
||||
export const DESTINATION_FORM_REGISTRY: Record<DestinationType, DestinationFormSpec<unknown>> = {
|
||||
s3: s3FormSpec as DestinationFormSpec<unknown>,
|
||||
gcs: gcsFormSpec as DestinationFormSpec<unknown>,
|
||||
azure_blob: azureBlobFormSpec as DestinationFormSpec<unknown>,
|
||||
datadog: datadogFormSpec as DestinationFormSpec<unknown>,
|
||||
bigquery: bigqueryFormSpec as DestinationFormSpec<unknown>,
|
||||
snowflake: snowflakeFormSpec as DestinationFormSpec<unknown>,
|
||||
webhook: webhookFormSpec as DestinationFormSpec<unknown>,
|
||||
}
|
||||
|
||||
@@ -1,7 +1,67 @@
|
||||
import { z } from 'zod'
|
||||
import { defineRouteContract } from '@/lib/api/contracts/types'
|
||||
import { validateExternalUrl } from '@/lib/core/security/input-validation'
|
||||
import { CADENCE_TYPES, DESTINATION_TYPES, SOURCE_TYPES } from '@/lib/data-drains/types'
|
||||
|
||||
/** AWS S3 bucket: 3-63 chars, lowercase alnum + . / -, see s3.ts for full rules. */
|
||||
const S3_BUCKET_NAME_RE = /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/
|
||||
const S3_IPV4_LIKE_RE = /^(\d{1,3}\.){3}\d{1,3}$/
|
||||
const AWS_REGION_RE = /^[a-z]{2,}(-[a-z]+)+-\d+$/
|
||||
/** GCS bucket component: lowercase alnum + _ / -, start/end alnum. Mirrors gcs.ts. */
|
||||
const GCS_BUCKET_COMPONENT_RE = /^[a-z0-9]([a-z0-9_-]*[a-z0-9])?$/
|
||||
const GOOGLE_RESERVED_PREFIX_RE = /^(goog|google|g00gle)/i
|
||||
const GOOGLE_CONTAINS_RE = /(google|g00gle)/i
|
||||
function validateGcsBucketComponents(v: string): string | null {
|
||||
if (v.length < 3 || v.length > 222) return 'bucket must be 3-222 characters'
|
||||
const components = v.split('.')
|
||||
for (const c of components) {
|
||||
if (c.length < 1 || c.length > 63) {
|
||||
return 'each dot-separated component must be 1-63 characters'
|
||||
}
|
||||
if (!GCS_BUCKET_COMPONENT_RE.test(c)) {
|
||||
return 'each component must be lowercase, start/end alphanumeric, letters/digits/_/- only'
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
/** Azure storage account: 3-24 lowercase alnum. */
|
||||
const AZURE_ACCOUNT_NAME_RE = /^[a-z0-9]{3,24}$/
|
||||
/** Azure container: 3-63 chars, lowercase alnum + single hyphens. */
|
||||
const AZURE_CONTAINER_NAME_RE = /^[a-z0-9]([a-z0-9]|-(?!-))+[a-z0-9]$/
|
||||
/** Azure Blob Storage endpoint suffixes (Public, US Gov, China, Germany). */
|
||||
const AZURE_ENDPOINT_SUFFIXES = [
|
||||
'blob.core.windows.net',
|
||||
'blob.core.usgovcloudapi.net',
|
||||
'blob.core.chinacloudapi.cn',
|
||||
'blob.core.cloudapi.de',
|
||||
] as const
|
||||
/** BigQuery project / dataset / table identifiers. */
|
||||
const BQ_PROJECT_ID_RE = /^([a-z][a-z0-9.-]{0,61}[a-z0-9]:)?[a-z][a-z0-9-]{4,28}[a-z0-9]$/
|
||||
const BQ_DATASET_RE = /^[A-Za-z0-9_]{1,1024}$/
|
||||
const BQ_TABLE_RE = /^[\p{L}\p{M}\p{N}\p{Pc}\p{Pd} ]{1,1024}$/u
|
||||
/** Snowflake account + identifier shapes — mirrored from snowflake.ts. */
|
||||
const SNOWFLAKE_ACCOUNT_ORG_RE = /^[A-Za-z0-9][A-Za-z0-9_]*(?:-[A-Za-z0-9_]+)+$/
|
||||
/** First segment allows hyphens so org-account identifiers carrying a region/cloud suffix match. Mirrors snowflake.ts. */
|
||||
const SNOWFLAKE_ACCOUNT_LOCATOR_RE =
|
||||
/^[A-Za-z0-9][A-Za-z0-9_-]*(?:\.[A-Za-z0-9][A-Za-z0-9_-]*){0,2}$/
|
||||
const SNOWFLAKE_IDENTIFIER_RE = /^[A-Za-z_][A-Za-z0-9_$]{0,254}$/
|
||||
/** Reserved Sim-namespaced header names that cannot be reused as the webhook signature header. */
|
||||
const RESERVED_WEBHOOK_SIGNATURE_HEADER_NAMES = new Set([
|
||||
'authorization',
|
||||
'content-type',
|
||||
'user-agent',
|
||||
'idempotency-key',
|
||||
'x-sim-timestamp',
|
||||
'x-sim-signature-version',
|
||||
'x-sim-drain-id',
|
||||
'x-sim-run-id',
|
||||
'x-sim-source',
|
||||
'x-sim-sequence',
|
||||
'x-sim-row-count',
|
||||
'x-sim-probe',
|
||||
'x-sim-signature',
|
||||
])
|
||||
|
||||
export const dataDrainSourceSchema = z.enum(SOURCE_TYPES)
|
||||
export const dataDrainDestinationTypeSchema = z.enum(DESTINATION_TYPES)
|
||||
export const dataDrainCadenceSchema = z.enum(CADENCE_TYPES)
|
||||
@@ -20,10 +80,57 @@ export const dataDrainParamsSchema = z.object({
|
||||
const drainNameSchema = z.string().trim().min(1, 'name is required').max(120)
|
||||
|
||||
const s3ConfigBodySchema = z.object({
|
||||
bucket: z.string().min(1, 'bucket is required').max(255),
|
||||
region: z.string().min(1, 'region is required').max(64),
|
||||
prefix: z.string().max(512).optional(),
|
||||
endpoint: z.string().url().optional(),
|
||||
bucket: z
|
||||
.string()
|
||||
.min(3, 'bucket must be 3-63 characters')
|
||||
.max(63, 'bucket must be 3-63 characters')
|
||||
.refine((v) => S3_BUCKET_NAME_RE.test(v), {
|
||||
message: 'bucket must be lowercase, 3-63 chars, start/end alphanumeric',
|
||||
})
|
||||
.refine((v) => !v.includes('..'), { message: 'bucket must not contain consecutive dots' })
|
||||
.refine((v) => !v.includes('-.') && !v.includes('.-'), {
|
||||
message: 'bucket must not contain a dash adjacent to a dot',
|
||||
})
|
||||
.refine((v) => !S3_IPV4_LIKE_RE.test(v), { message: 'bucket must not look like an IP address' })
|
||||
.refine((v) => !v.startsWith('xn--'), { message: 'bucket must not start with "xn--"' })
|
||||
.refine((v) => !v.startsWith('sthree-'), { message: 'bucket must not start with "sthree-"' })
|
||||
.refine((v) => !v.startsWith('amzn-s3-demo-'), {
|
||||
message: 'bucket must not start with "amzn-s3-demo-" (reserved by AWS)',
|
||||
})
|
||||
.refine(
|
||||
(v) =>
|
||||
!v.endsWith('-s3alias') &&
|
||||
!v.endsWith('--ol-s3') &&
|
||||
!v.endsWith('.mrap') &&
|
||||
!v.endsWith('--x-s3') &&
|
||||
!v.endsWith('--table-s3'),
|
||||
{
|
||||
message:
|
||||
'bucket must not end with reserved suffix (-s3alias, --ol-s3, .mrap, --x-s3, --table-s3)',
|
||||
}
|
||||
),
|
||||
region: z
|
||||
.string()
|
||||
.min(1, 'region is required')
|
||||
.max(32, 'region is too long')
|
||||
.refine((v) => AWS_REGION_RE.test(v), {
|
||||
message: 'region must look like an AWS region code, e.g. us-east-1',
|
||||
}),
|
||||
prefix: z
|
||||
.string()
|
||||
.max(512)
|
||||
.refine((v) => Buffer.byteLength(v, 'utf8') <= 512, {
|
||||
message: 'prefix must be at most 512 bytes (UTF-8)',
|
||||
})
|
||||
.optional(),
|
||||
endpoint: z
|
||||
.string()
|
||||
.url()
|
||||
.refine((v) => v.startsWith('https://'), { message: 'endpoint must use https://' })
|
||||
.refine((value) => validateExternalUrl(value, 'endpoint').isValid, {
|
||||
message: 'endpoint must be HTTPS and not point at a private, loopback, or metadata address',
|
||||
})
|
||||
.optional(),
|
||||
forcePathStyle: z.boolean().optional(),
|
||||
})
|
||||
|
||||
@@ -32,14 +139,200 @@ const s3CredentialsBodySchema = z.object({
|
||||
secretAccessKey: z.string().min(1, 'secretAccessKey is required'),
|
||||
})
|
||||
|
||||
const gcsConfigBodySchema = z.object({
|
||||
bucket: z
|
||||
.string()
|
||||
.min(3, 'bucket must be 3-222 characters')
|
||||
.max(222, 'bucket must be 3-222 characters')
|
||||
.superRefine((v, ctx) => {
|
||||
const err = validateGcsBucketComponents(v)
|
||||
if (err) ctx.addIssue({ code: z.ZodIssueCode.custom, message: err })
|
||||
})
|
||||
.refine((v) => !S3_IPV4_LIKE_RE.test(v), { message: 'bucket must not look like an IP address' })
|
||||
.refine((v) => !v.includes('..'), { message: 'bucket must not contain consecutive dots' })
|
||||
.refine((v) => !v.includes('-.') && !v.includes('.-'), {
|
||||
message: 'bucket must not contain "-." or ".-"',
|
||||
})
|
||||
.refine((v) => !GOOGLE_RESERVED_PREFIX_RE.test(v) && !GOOGLE_CONTAINS_RE.test(v), {
|
||||
message: 'bucket name cannot begin with "goog" or contain "google" / close misspellings',
|
||||
}),
|
||||
prefix: z
|
||||
.string()
|
||||
.max(512)
|
||||
.refine((v) => Buffer.byteLength(v, 'utf8') <= 512, {
|
||||
message: 'prefix must be at most 512 bytes (UTF-8)',
|
||||
})
|
||||
.refine((v) => !v.startsWith('.well-known/acme-challenge/'), {
|
||||
message: 'prefix must not start with ".well-known/acme-challenge/" (reserved by GCS)',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const gcsCredentialsBodySchema = z.object({
|
||||
serviceAccountJson: z.string().min(1, 'serviceAccountJson is required'),
|
||||
})
|
||||
|
||||
const azureBlobConfigBodySchema = z.object({
|
||||
accountName: z
|
||||
.string()
|
||||
.min(1, 'accountName is required')
|
||||
.refine((v) => AZURE_ACCOUNT_NAME_RE.test(v), {
|
||||
message: 'accountName must be 3-24 lowercase letters or digits',
|
||||
}),
|
||||
containerName: z
|
||||
.string()
|
||||
.min(3, 'containerName must be 3-63 characters')
|
||||
.max(63)
|
||||
.refine((v) => AZURE_CONTAINER_NAME_RE.test(v), {
|
||||
message: 'containerName must use lowercase letters, digits, or single hyphens',
|
||||
}),
|
||||
prefix: z.string().max(512).optional(),
|
||||
endpointSuffix: z
|
||||
.string()
|
||||
.refine((v) => (AZURE_ENDPOINT_SUFFIXES as readonly string[]).includes(v), {
|
||||
message: `endpointSuffix must be one of: ${AZURE_ENDPOINT_SUFFIXES.join(', ')}`,
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const azureBlobCredentialsBodySchema = z.object({
|
||||
accountKey: z
|
||||
.string()
|
||||
.length(88, 'accountKey must be 88 base64 characters (64-byte Azure storage key)')
|
||||
.regex(/^[A-Za-z0-9+/]+={0,2}$/, {
|
||||
message: 'accountKey must be a base64-encoded Azure storage account key',
|
||||
}),
|
||||
})
|
||||
|
||||
const DATADOG_TAG_PAIR_RE = /^[A-Za-z][A-Za-z0-9_./-]*:[^,\s][^,]*$/
|
||||
|
||||
const datadogConfigBodySchema = z.object({
|
||||
site: z.enum(['us1', 'us3', 'us5', 'eu1', 'ap1', 'ap2', 'gov']),
|
||||
service: z.string().min(1).max(100).optional(),
|
||||
tags: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(1024)
|
||||
.refine(
|
||||
(v) =>
|
||||
v
|
||||
.split(',')
|
||||
.map((t) => t.trim())
|
||||
.filter((t) => t.length > 0)
|
||||
.every((t) => DATADOG_TAG_PAIR_RE.test(t)),
|
||||
{ message: 'tags must be comma-separated key:value pairs' }
|
||||
)
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const datadogCredentialsBodySchema = z.object({
|
||||
apiKey: z.string().min(1, 'apiKey is required'),
|
||||
})
|
||||
|
||||
const bigqueryConfigBodySchema = z.object({
|
||||
projectId: z
|
||||
.string()
|
||||
.min(6, 'projectId is required')
|
||||
.max(94)
|
||||
.refine((v) => BQ_PROJECT_ID_RE.test(v), {
|
||||
message: 'projectId must match Google Cloud project ID rules',
|
||||
}),
|
||||
datasetId: z
|
||||
.string()
|
||||
.min(1, 'datasetId is required')
|
||||
.refine((v) => BQ_DATASET_RE.test(v), {
|
||||
message: 'datasetId may only contain letters, digits, and underscores (max 1024)',
|
||||
}),
|
||||
tableId: z
|
||||
.string()
|
||||
.min(1, 'tableId is required')
|
||||
.refine((v) => BQ_TABLE_RE.test(v), {
|
||||
message:
|
||||
'tableId may contain Unicode letters, marks, numbers, connectors, dashes, and spaces (max 1024)',
|
||||
})
|
||||
.refine((v) => Buffer.byteLength(v, 'utf8') <= 1024, {
|
||||
message: 'tableId must be at most 1024 bytes (UTF-8)',
|
||||
}),
|
||||
})
|
||||
|
||||
const bigqueryCredentialsBodySchema = z.object({
|
||||
serviceAccountJson: z.string().min(1, 'serviceAccountJson is required'),
|
||||
})
|
||||
|
||||
const snowflakeConfigBodySchema = z.object({
|
||||
account: z
|
||||
.string()
|
||||
.min(3, 'account is required')
|
||||
.max(256)
|
||||
.refine((v) => SNOWFLAKE_ACCOUNT_ORG_RE.test(v) || SNOWFLAKE_ACCOUNT_LOCATOR_RE.test(v), {
|
||||
message:
|
||||
'account must be a Snowflake org-account identifier (orgname-accountname) or legacy locator (locator[.region[.cloud]])',
|
||||
}),
|
||||
user: z.string().min(1, 'user is required').regex(SNOWFLAKE_IDENTIFIER_RE, {
|
||||
message: 'user must be a valid Snowflake identifier',
|
||||
}),
|
||||
warehouse: z.string().min(1).regex(SNOWFLAKE_IDENTIFIER_RE, {
|
||||
message: 'warehouse must be a valid Snowflake identifier',
|
||||
}),
|
||||
database: z.string().min(1).regex(SNOWFLAKE_IDENTIFIER_RE, {
|
||||
message: 'database must be a valid Snowflake identifier',
|
||||
}),
|
||||
schema: z.string().min(1).regex(SNOWFLAKE_IDENTIFIER_RE, {
|
||||
message: 'schema must be a valid Snowflake identifier',
|
||||
}),
|
||||
table: z.string().min(1).regex(SNOWFLAKE_IDENTIFIER_RE, {
|
||||
message: 'table must be a valid Snowflake identifier',
|
||||
}),
|
||||
column: z
|
||||
.string()
|
||||
.min(1)
|
||||
.regex(SNOWFLAKE_IDENTIFIER_RE, { message: 'column must be a valid Snowflake identifier' })
|
||||
.optional(),
|
||||
role: z
|
||||
.string()
|
||||
.min(1)
|
||||
.regex(SNOWFLAKE_IDENTIFIER_RE, { message: 'role must be a valid Snowflake identifier' })
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const snowflakeCredentialsBodySchema = z.object({
|
||||
privateKey: z.string().min(1, 'privateKey is required'),
|
||||
})
|
||||
|
||||
const webhookConfigBodySchema = z.object({
|
||||
url: z.string().url('url must be a valid URL'),
|
||||
signatureHeader: z.string().min(1).max(128).optional(),
|
||||
url: z
|
||||
.string()
|
||||
.url('url must be a valid URL')
|
||||
.max(2048, 'url must be at most 2048 characters')
|
||||
.refine((value) => validateExternalUrl(value, 'url').isValid, {
|
||||
message: 'url must be HTTPS and not point at a private, loopback, or metadata address',
|
||||
}),
|
||||
signatureHeader: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(128)
|
||||
.refine((value) => !RESERVED_WEBHOOK_SIGNATURE_HEADER_NAMES.has(value.toLowerCase()), {
|
||||
message: 'signatureHeader cannot reuse a reserved Sim header name',
|
||||
})
|
||||
.refine((value) => /^[A-Za-z0-9\-_]+$/.test(value) && !/[\r\n\0]/.test(value), {
|
||||
message: 'signatureHeader must contain only letters, digits, hyphens, and underscores',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const webhookCredentialsBodySchema = z.object({
|
||||
signingSecret: z.string().min(8, 'signingSecret must be at least 8 characters'),
|
||||
bearerToken: z.string().min(1).optional(),
|
||||
signingSecret: z
|
||||
.string()
|
||||
.min(32, 'signingSecret must be at least 32 characters')
|
||||
.max(512, 'signingSecret must be at most 512 characters'),
|
||||
bearerToken: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(4096, 'bearerToken must be at most 4096 characters')
|
||||
.refine((value) => !/[\r\n\0]/.test(value), {
|
||||
message: 'bearerToken cannot contain CR, LF, or NUL characters',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
/**
|
||||
@@ -54,6 +347,31 @@ export const dataDrainDestinationBodySchema = z.discriminatedUnion('destinationT
|
||||
destinationConfig: s3ConfigBodySchema,
|
||||
destinationCredentials: s3CredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('gcs'),
|
||||
destinationConfig: gcsConfigBodySchema,
|
||||
destinationCredentials: gcsCredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('azure_blob'),
|
||||
destinationConfig: azureBlobConfigBodySchema,
|
||||
destinationCredentials: azureBlobCredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('datadog'),
|
||||
destinationConfig: datadogConfigBodySchema,
|
||||
destinationCredentials: datadogCredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('bigquery'),
|
||||
destinationConfig: bigqueryConfigBodySchema,
|
||||
destinationCredentials: bigqueryCredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('snowflake'),
|
||||
destinationConfig: snowflakeConfigBodySchema,
|
||||
destinationCredentials: snowflakeCredentialsBodySchema.optional(),
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('webhook'),
|
||||
destinationConfig: webhookConfigBodySchema,
|
||||
@@ -93,6 +411,26 @@ const drainDestinationResponseSchema = z.discriminatedUnion('destinationType', [
|
||||
destinationType: z.literal('s3'),
|
||||
destinationConfig: s3ConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('gcs'),
|
||||
destinationConfig: gcsConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('azure_blob'),
|
||||
destinationConfig: azureBlobConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('datadog'),
|
||||
destinationConfig: datadogConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('bigquery'),
|
||||
destinationConfig: bigqueryConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('snowflake'),
|
||||
destinationConfig: snowflakeConfigBodySchema,
|
||||
}),
|
||||
z.object({
|
||||
destinationType: z.literal('webhook'),
|
||||
destinationConfig: webhookConfigBodySchema,
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockUpload, mockDeleteIfExists, BlobServiceClientCtor, StorageSharedKeyCredentialCtor } =
|
||||
vi.hoisted(() => {
|
||||
const mockUpload = vi.fn(async () => ({}))
|
||||
const mockDeleteIfExists = vi.fn(async () => ({ succeeded: true }))
|
||||
const blockBlobClient = { upload: mockUpload, deleteIfExists: mockDeleteIfExists }
|
||||
const containerClient = { getBlockBlobClient: vi.fn(() => blockBlobClient) }
|
||||
return {
|
||||
mockUpload,
|
||||
mockDeleteIfExists,
|
||||
BlobServiceClientCtor: vi.fn(() => ({ getContainerClient: vi.fn(() => containerClient) })),
|
||||
StorageSharedKeyCredentialCtor: vi.fn(),
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('@azure/storage-blob', () => ({
|
||||
BlobServiceClient: BlobServiceClientCtor,
|
||||
StorageSharedKeyCredential: StorageSharedKeyCredentialCtor,
|
||||
}))
|
||||
|
||||
import { azureBlobDestination } from '@/lib/data-drains/destinations/azure_blob'
|
||||
|
||||
const config = { accountName: 'simstore', containerName: 'drains', prefix: 'sim/' }
|
||||
// Realistic 88-char base64 (64-byte) Azure storage key shape.
|
||||
const credentials = {
|
||||
accountKey:
|
||||
'YWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWE=',
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
})
|
||||
|
||||
describe('azureBlobDestination openSession', () => {
|
||||
it('uploads via BlockBlobClient and returns an azure:// locator', async () => {
|
||||
const session = azureBlobDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from('row\n', 'utf8')
|
||||
const result = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd1',
|
||||
runId: 'r1',
|
||||
source: 'workflow_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
expect(result.locator).toMatch(
|
||||
/^azure:\/\/simstore\/drains\/sim\/workflow_logs\/d1\/\d{4}\/\d{2}\/\d{2}\/r1-00000\.ndjson$/
|
||||
)
|
||||
expect(mockUpload).toHaveBeenCalledTimes(1)
|
||||
const [calledBody, calledLength, opts] = mockUpload.mock.calls[0] as [
|
||||
Buffer,
|
||||
number,
|
||||
{ metadata?: Record<string, string> },
|
||||
]
|
||||
expect(calledBody).toBe(body)
|
||||
expect(calledLength).toBe(body.byteLength)
|
||||
expect(opts.metadata?.simdrainid).toBe('d1')
|
||||
expect(opts.metadata?.simsequence).toBe('0')
|
||||
|
||||
const fullOpts = opts as {
|
||||
blobHTTPHeaders?: { blobContentType?: string }
|
||||
abortSignal?: AbortSignal
|
||||
}
|
||||
expect(fullOpts.blobHTTPHeaders?.blobContentType).toBe('application/x-ndjson')
|
||||
expect(fullOpts.abortSignal).toBeDefined()
|
||||
expect(StorageSharedKeyCredentialCtor).toHaveBeenCalledWith('simstore', credentials.accountKey)
|
||||
expect(BlobServiceClientCtor).toHaveBeenCalledWith(
|
||||
'https://simstore.blob.core.windows.net',
|
||||
expect.anything(),
|
||||
expect.objectContaining({ retryOptions: expect.any(Object) })
|
||||
)
|
||||
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('routes to a sovereign-cloud endpoint suffix when configured', async () => {
|
||||
const session = azureBlobDestination.openSession({
|
||||
config: { ...config, endpointSuffix: 'blob.core.usgovcloudapi.net' },
|
||||
credentials,
|
||||
})
|
||||
await session.deliver({
|
||||
body: Buffer.from('row\n', 'utf8'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'workflow_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(BlobServiceClientCtor).toHaveBeenCalledWith(
|
||||
'https://simstore.blob.core.usgovcloudapi.net',
|
||||
expect.anything(),
|
||||
expect.objectContaining({ retryOptions: expect.any(Object) })
|
||||
)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('surfaces Azure REST errors', async () => {
|
||||
mockUpload.mockRejectedValueOnce(
|
||||
Object.assign(new Error('Forbidden'), {
|
||||
statusCode: 403,
|
||||
code: 'AuthenticationFailed',
|
||||
})
|
||||
)
|
||||
const session = azureBlobDestination.openSession({ config, credentials })
|
||||
await expect(
|
||||
session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'audit_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/AuthenticationFailed 403/)
|
||||
await session.close()
|
||||
})
|
||||
})
|
||||
|
||||
describe('azureBlobDestination test()', () => {
|
||||
it('writes a probe blob then attempts cleanup', async () => {
|
||||
await azureBlobDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(mockUpload).toHaveBeenCalled()
|
||||
expect(mockDeleteIfExists).toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('azureBlobDestination config schema', () => {
|
||||
it('rejects invalid account names', () => {
|
||||
const result = azureBlobDestination.configSchema.safeParse({
|
||||
accountName: 'BAD-NAME',
|
||||
containerName: 'drains',
|
||||
})
|
||||
expect(result.success).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects invalid container names', () => {
|
||||
const result = azureBlobDestination.configSchema.safeParse({
|
||||
accountName: 'simstore',
|
||||
containerName: '--bad--',
|
||||
})
|
||||
expect(result.success).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('azureBlobDestination credentials schema', () => {
|
||||
it('rejects non-base64 account keys', () => {
|
||||
const padded = 'a'.repeat(70)
|
||||
const result = azureBlobDestination.credentialsSchema.safeParse({
|
||||
accountKey: `${padded}!@#$`,
|
||||
})
|
||||
expect(result.success).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects keys that are too short', () => {
|
||||
const result = azureBlobDestination.credentialsSchema.safeParse({ accountKey: 'YQ==' })
|
||||
expect(result.success).toBe(false)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,203 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { generateShortId } from '@sim/utils/id'
|
||||
import { z } from 'zod'
|
||||
import { buildObjectKey, normalizePrefix } from '@/lib/data-drains/destinations/utils'
|
||||
import type { DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainAzureBlobDestination')
|
||||
|
||||
/**
|
||||
* Azure storage account names: 3-24 chars, lowercase letters and digits only.
|
||||
* https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview#storage-account-name
|
||||
*/
|
||||
const ACCOUNT_NAME_RE = /^[a-z0-9]{3,24}$/
|
||||
|
||||
/**
|
||||
* Azure container names: 3-63 chars, lowercase letters, digits, single hyphens
|
||||
* (no leading/trailing/double hyphens).
|
||||
* https://learn.microsoft.com/en-us/rest/api/storageservices/naming-and-referencing-containers--blobs--and-metadata
|
||||
*/
|
||||
const CONTAINER_NAME_RE = /^[a-z0-9]([a-z0-9]|-(?!-))+[a-z0-9]$/
|
||||
|
||||
/** Azure storage account keys are 64 raw bytes => exactly 88 base64 chars (0-2 trailing `=`). */
|
||||
const ACCOUNT_KEY_RE = /^[A-Za-z0-9+/]+={0,2}$/
|
||||
|
||||
/** Public cloud default; sovereign clouds (Gov/China/legacy DE) are validated via allowlist. */
|
||||
const DEFAULT_ENDPOINT_SUFFIX = 'blob.core.windows.net'
|
||||
|
||||
/**
|
||||
* Allowlist of Azure Storage endpoint suffixes. URL host must end with one of these
|
||||
* (after the account name + dot). Reject anything else to prevent SSRF via attacker-controlled
|
||||
* endpoint suffix.
|
||||
*/
|
||||
const ALLOWED_ENDPOINT_SUFFIXES = [
|
||||
'blob.core.windows.net',
|
||||
'blob.core.usgovcloudapi.net',
|
||||
'blob.core.chinacloudapi.cn',
|
||||
'blob.core.cloudapi.de',
|
||||
] as const
|
||||
|
||||
const azureBlobConfigSchema = z.object({
|
||||
accountName: z
|
||||
.string()
|
||||
.min(1, 'accountName is required')
|
||||
.refine((value) => ACCOUNT_NAME_RE.test(value), {
|
||||
message: 'accountName must be 3-24 lowercase letters or digits',
|
||||
}),
|
||||
containerName: z
|
||||
.string()
|
||||
.min(3, 'containerName must be 3-63 characters')
|
||||
.max(63)
|
||||
.refine((value) => CONTAINER_NAME_RE.test(value), {
|
||||
message: 'containerName must use lowercase letters, digits, or single hyphens',
|
||||
}),
|
||||
/** Optional prefix; trailing slash is added automatically when assembling blob names. */
|
||||
prefix: z.string().max(512).optional(),
|
||||
/** Storage endpoint suffix. Must be one of the known Azure Storage suffixes (public/Gov/China/DE). */
|
||||
endpointSuffix: z
|
||||
.string()
|
||||
.refine((v) => (ALLOWED_ENDPOINT_SUFFIXES as readonly string[]).includes(v), {
|
||||
message: `endpointSuffix must be one of: ${ALLOWED_ENDPOINT_SUFFIXES.join(', ')}`,
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const azureBlobCredentialsSchema = z.object({
|
||||
accountKey: z
|
||||
.string()
|
||||
.length(88, 'accountKey must be exactly 88 base64 characters (64-byte Azure storage key)')
|
||||
.refine((v) => ACCOUNT_KEY_RE.test(v), {
|
||||
message: 'accountKey must be a base64-encoded Azure storage account key',
|
||||
}),
|
||||
})
|
||||
|
||||
export type AzureBlobDestinationConfig = z.infer<typeof azureBlobConfigSchema>
|
||||
export type AzureBlobDestinationCredentials = z.infer<typeof azureBlobCredentialsSchema>
|
||||
|
||||
interface BlobClients {
|
||||
containerClient: import('@azure/storage-blob').ContainerClient
|
||||
}
|
||||
|
||||
async function buildClients(
|
||||
config: AzureBlobDestinationConfig,
|
||||
credentials: AzureBlobDestinationCredentials
|
||||
): Promise<BlobClients> {
|
||||
const { BlobServiceClient, StorageSharedKeyCredential } = await import('@azure/storage-blob')
|
||||
const sharedKeyCredential = new StorageSharedKeyCredential(
|
||||
config.accountName,
|
||||
credentials.accountKey
|
||||
)
|
||||
const suffix = config.endpointSuffix ?? DEFAULT_ENDPOINT_SUFFIX
|
||||
/**
|
||||
* Bound per-attempt timeout. SDK default `tryTimeoutInMs` is "infinite" / OS
|
||||
* connection-idle limits, so a hung receiver could pin a delivery indefinitely.
|
||||
* 30s per try + 5 tries (~ exponential 0.5s → 30s) caps the worst-case wall time.
|
||||
*/
|
||||
const blobServiceClient = new BlobServiceClient(
|
||||
`https://${config.accountName}.${suffix}`,
|
||||
sharedKeyCredential,
|
||||
{
|
||||
retryOptions: {
|
||||
tryTimeoutInMs: 30_000,
|
||||
maxTries: 5,
|
||||
retryDelayInMs: 500,
|
||||
maxRetryDelayInMs: 30_000,
|
||||
},
|
||||
}
|
||||
)
|
||||
return { containerClient: blobServiceClient.getContainerClient(config.containerName) }
|
||||
}
|
||||
|
||||
interface AzureRestErrorLike {
|
||||
statusCode?: number
|
||||
code?: string
|
||||
message?: string
|
||||
}
|
||||
|
||||
function isAzureRestError(error: unknown): error is AzureRestErrorLike {
|
||||
return typeof error === 'object' && error !== null && ('statusCode' in error || 'code' in error)
|
||||
}
|
||||
|
||||
async function withAzureErrorContext<T>(action: string, fn: () => Promise<T>): Promise<T> {
|
||||
try {
|
||||
return await fn()
|
||||
} catch (error) {
|
||||
if (isAzureRestError(error)) {
|
||||
const status = error.statusCode
|
||||
const code = error.code
|
||||
logger.warn('Azure Blob operation failed', { action, code, status })
|
||||
throw new Error(
|
||||
`Azure Blob ${action} failed (${code ?? 'Error'}${status ? ` ${status}` : ''}): ${error.message ?? ''}`,
|
||||
{ cause: error }
|
||||
)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
export const azureBlobDestination: DrainDestination<
|
||||
AzureBlobDestinationConfig,
|
||||
AzureBlobDestinationCredentials
|
||||
> = {
|
||||
type: 'azure_blob',
|
||||
displayName: 'Azure Blob Storage',
|
||||
configSchema: azureBlobConfigSchema,
|
||||
credentialsSchema: azureBlobCredentialsSchema,
|
||||
|
||||
async test({ config, credentials, signal }) {
|
||||
const { containerClient } = await buildClients(config, credentials)
|
||||
const probeName = `${normalizePrefix(config.prefix)}.sim-drain-write-probe/${generateShortId(12)}`
|
||||
const blockBlobClient = containerClient.getBlockBlobClient(probeName)
|
||||
await withAzureErrorContext('test-put', () =>
|
||||
blockBlobClient.upload(Buffer.alloc(0), 0, {
|
||||
blobHTTPHeaders: { blobContentType: 'application/octet-stream' },
|
||||
abortSignal: signal,
|
||||
})
|
||||
)
|
||||
try {
|
||||
await blockBlobClient.deleteIfExists({ abortSignal: signal })
|
||||
} catch (cleanupError) {
|
||||
logger.debug('Azure Blob test write probe cleanup failed (non-fatal)', {
|
||||
accountName: config.accountName,
|
||||
containerName: config.containerName,
|
||||
blobName: probeName,
|
||||
error: cleanupError,
|
||||
})
|
||||
}
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
let clientsPromise: Promise<BlobClients> | null = null
|
||||
return {
|
||||
async deliver({ body, contentType, metadata, signal }) {
|
||||
if (clientsPromise === null) clientsPromise = buildClients(config, credentials)
|
||||
const { containerClient } = await clientsPromise
|
||||
const blobName = buildObjectKey(config.prefix, metadata)
|
||||
const blockBlobClient = containerClient.getBlockBlobClient(blobName)
|
||||
await withAzureErrorContext('put-object', () =>
|
||||
blockBlobClient.upload(body, body.byteLength, {
|
||||
blobHTTPHeaders: { blobContentType: contentType },
|
||||
metadata: {
|
||||
simdrainid: metadata.drainId,
|
||||
simrunid: metadata.runId,
|
||||
simsource: metadata.source,
|
||||
simsequence: metadata.sequence.toString(),
|
||||
simrowcount: metadata.rowCount.toString(),
|
||||
},
|
||||
abortSignal: signal,
|
||||
})
|
||||
)
|
||||
logger.debug('Azure Blob chunk delivered', {
|
||||
accountName: config.accountName,
|
||||
containerName: config.containerName,
|
||||
blobName,
|
||||
bytes: body.byteLength,
|
||||
})
|
||||
return {
|
||||
locator: `azure://${config.accountName}/${config.containerName}/${blobName}`,
|
||||
}
|
||||
},
|
||||
async close() {},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,272 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockGetAccessToken, JWTCtor, loggerInstance } = vi.hoisted(() => {
|
||||
const mockGetAccessToken = vi.fn(async () => ({ token: 'bq-token' }))
|
||||
const loggerInstance = {
|
||||
info: vi.fn(),
|
||||
warn: vi.fn(),
|
||||
error: vi.fn(),
|
||||
debug: vi.fn(),
|
||||
trace: vi.fn(),
|
||||
fatal: vi.fn(),
|
||||
child: vi.fn(),
|
||||
withMetadata: vi.fn(),
|
||||
}
|
||||
return {
|
||||
mockGetAccessToken,
|
||||
JWTCtor: vi.fn(() => ({ getAccessToken: mockGetAccessToken })),
|
||||
loggerInstance,
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('google-auth-library', () => ({ JWT: JWTCtor }))
|
||||
vi.mock('@sim/logger', () => ({
|
||||
createLogger: () => loggerInstance,
|
||||
logger: loggerInstance,
|
||||
runWithRequestContext: <T>(_ctx: unknown, fn: () => T): T => fn(),
|
||||
getRequestContext: () => undefined,
|
||||
}))
|
||||
vi.mock('@sim/utils/helpers', () => ({
|
||||
sleep: vi.fn(async () => {}),
|
||||
}))
|
||||
|
||||
const fetchMock = vi.fn(
|
||||
async () =>
|
||||
new Response(JSON.stringify({}), {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
})
|
||||
)
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
import { bigqueryDestination } from '@/lib/data-drains/destinations/bigquery'
|
||||
|
||||
const config = { projectId: 'my-proj', datasetId: 'logs', tableId: 'workflow' }
|
||||
const credentials = {
|
||||
serviceAccountJson: JSON.stringify({
|
||||
client_email: 'sa@p.iam.gserviceaccount.com',
|
||||
private_key: '-----BEGIN PRIVATE KEY-----\nfake\n-----END PRIVATE KEY-----\n',
|
||||
}),
|
||||
}
|
||||
|
||||
const meta = {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'workflow_logs' as const,
|
||||
sequence: 0,
|
||||
rowCount: 2,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
fetchMock.mockResolvedValue(
|
||||
new Response(JSON.stringify({}), {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
})
|
||||
)
|
||||
mockGetAccessToken.mockResolvedValue({ token: 'bq-token' })
|
||||
})
|
||||
|
||||
describe('bigqueryDestination', () => {
|
||||
it('posts rows with stable insertIds for dedup', async () => {
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ id: 'a' })}\n${JSON.stringify({ id: 'b' })}\n`,
|
||||
'utf8'
|
||||
)
|
||||
const result = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toContain('/projects/my-proj/datasets/logs/tables/workflow/insertAll')
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.rows).toHaveLength(2)
|
||||
expect(payload.rows[0].insertId).toBe('d-r-0-0')
|
||||
expect(payload.rows[0].json).toEqual({ id: 'a' })
|
||||
expect(payload.rows[1].insertId).toBe('d-r-0-1')
|
||||
expect(result.locator).toBe('bigquery://my-proj/logs/workflow#r-0')
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('throws with row indices and warns on partial-failure insertErrors', async () => {
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
new Response(
|
||||
JSON.stringify({
|
||||
insertErrors: [
|
||||
{ index: 1, errors: [{ message: 'invalid', reason: 'invalid' }] },
|
||||
{ index: 2, errors: [{ message: 'bad', reason: 'invalid' }] },
|
||||
],
|
||||
}),
|
||||
{ status: 200, headers: { 'content-type': 'application/json' } }
|
||||
)
|
||||
)
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ x: 1 })}\n${JSON.stringify({ x: 2 })}\n${JSON.stringify({ x: 3 })}\n`
|
||||
)
|
||||
await expect(
|
||||
session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/partial failure.*1,2.*dedup-keyed by insertId/s)
|
||||
expect(loggerInstance.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('partial failure'),
|
||||
expect.objectContaining({
|
||||
partialFailure: true,
|
||||
succeededRows: 1,
|
||||
failedRows: 2,
|
||||
})
|
||||
)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('test() probes table existence with a GET', async () => {
|
||||
await bigqueryDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toContain('?fields=id')
|
||||
expect(init.method).toBeUndefined()
|
||||
})
|
||||
|
||||
it('throws a clear error when an NDJSON line is malformed', async () => {
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(`${JSON.stringify({ id: 'a' })}\n{not json}\n`, 'utf8')
|
||||
await expect(
|
||||
session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/NDJSON parse failed at line 2/)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('throws when an NDJSON row is not a JSON object', async () => {
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(`${JSON.stringify({ id: 'a' })}\n42\n`, 'utf8')
|
||||
await expect(
|
||||
session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/NDJSON row at line 2 is not an object/)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('parses NDJSON with CRLF line endings', async () => {
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ id: 'a' })}\r\n${JSON.stringify({ id: 'b' })}\r\n`,
|
||||
'utf8'
|
||||
)
|
||||
await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const [, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.rows).toHaveLength(2)
|
||||
expect(payload.rows[0].json).toEqual({ id: 'a' })
|
||||
expect(payload.rows[1].json).toEqual({ id: 'b' })
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('insertId includes drainId prefix to avoid cross-drain collisions', async () => {
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(`${JSON.stringify({ id: 'a' })}\n`, 'utf8')
|
||||
await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: { ...meta, drainId: 'drain-xyz', runId: 'run-1', sequence: 7 },
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const [, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.rows[0].insertId).toBe('drain-xyz-run-1-7-0')
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('retries 5xx responses with backoff, then succeeds', async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response('boom', { status: 503 })).mockResolvedValueOnce(
|
||||
new Response(JSON.stringify({}), {
|
||||
status: 200,
|
||||
headers: { 'content-type': 'application/json' },
|
||||
})
|
||||
)
|
||||
const session = bigqueryDestination.openSession({ config, credentials })
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ id: 'a' })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
expect(loggerInstance.warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('transient error'),
|
||||
expect.objectContaining({ status: 503, attempt: 1 })
|
||||
)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('accepts domain-scoped project IDs', () => {
|
||||
const result = bigqueryDestination.configSchema.safeParse({
|
||||
projectId: 'example.com:my-project',
|
||||
datasetId: 'logs',
|
||||
tableId: 'workflow',
|
||||
})
|
||||
expect(result.success).toBe(true)
|
||||
const standard = bigqueryDestination.configSchema.safeParse({
|
||||
projectId: 'my-proj',
|
||||
datasetId: 'logs',
|
||||
tableId: 'workflow',
|
||||
})
|
||||
expect(standard.success).toBe(true)
|
||||
})
|
||||
|
||||
it('test() throws when serviceAccountJson is missing required fields', async () => {
|
||||
await expect(
|
||||
bigqueryDestination.test!({
|
||||
config,
|
||||
credentials: {
|
||||
serviceAccountJson: JSON.stringify({
|
||||
client_email: 'sa@p.iam.gserviceaccount.com',
|
||||
}),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/missing private_key/)
|
||||
await expect(
|
||||
bigqueryDestination.test!({
|
||||
config,
|
||||
credentials: {
|
||||
serviceAccountJson: JSON.stringify({
|
||||
private_key: '-----BEGIN PRIVATE KEY-----\nfake\n-----END PRIVATE KEY-----\n',
|
||||
}),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/missing client_email/)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,334 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { JWT } from 'google-auth-library'
|
||||
import { z } from 'zod'
|
||||
import {
|
||||
backoffWithJitter,
|
||||
type ParsedServiceAccount,
|
||||
parseNdjsonObjects,
|
||||
parseRetryAfter,
|
||||
parseServiceAccount,
|
||||
refineServiceAccountJson,
|
||||
sleepUntilAborted,
|
||||
} from '@/lib/data-drains/destinations/utils'
|
||||
import type { DeliveryMetadata, DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainBigQueryDestination')
|
||||
|
||||
/**
|
||||
* Uses the legacy `tabledata.insertAll` streaming endpoint. The Storage Write
|
||||
* API offers exactly-once semantics and lower pricing but requires gRPC; we
|
||||
* stay on insertAll for simplicity and direct HTTP support.
|
||||
*/
|
||||
|
||||
/** `insertdata` for streaming inserts; `readonly` for the `tables.get` probe in `test()`. */
|
||||
const SCOPES = [
|
||||
'https://www.googleapis.com/auth/bigquery.insertdata',
|
||||
'https://www.googleapis.com/auth/bigquery.readonly',
|
||||
]
|
||||
|
||||
/** Standard project IDs are 6-30 chars; the optional `domain.tld:` prefix supports legacy domain-scoped projects. */
|
||||
const PROJECT_ID_RE = /^([a-z][a-z0-9.-]{0,61}[a-z0-9]:)?[a-z][a-z0-9-]{4,28}[a-z0-9]$/
|
||||
const DATASET_RE = /^[A-Za-z0-9_]{1,1024}$/
|
||||
const TABLE_RE = /^[\p{L}\p{M}\p{N}\p{Pc}\p{Pd} ]{1,1024}$/u
|
||||
|
||||
const USER_AGENT = 'sim-data-drain/1.0'
|
||||
/** Per-request streaming limits: 10 MB body, 50,000 rows, 1 MB per row. */
|
||||
const MAX_REQUEST_BYTES = 10 * 1024 * 1024
|
||||
const MAX_ROWS_PER_REQUEST = 50_000
|
||||
const MAX_ROW_BYTES = 1024 * 1024
|
||||
/** `insertId` is capped at 128 characters (encoded length). */
|
||||
const MAX_INSERT_ID_LENGTH = 128
|
||||
const PER_ATTEMPT_TIMEOUT_MS = 60_000
|
||||
|
||||
const bigqueryConfigSchema = z.object({
|
||||
projectId: z
|
||||
.string()
|
||||
.min(6, 'projectId is required')
|
||||
.refine((v) => PROJECT_ID_RE.test(v), {
|
||||
message: 'projectId must match Google Cloud project ID rules',
|
||||
}),
|
||||
datasetId: z
|
||||
.string()
|
||||
.min(1, 'datasetId is required')
|
||||
.refine((v) => DATASET_RE.test(v), {
|
||||
message: 'datasetId may only contain ASCII letters, digits, and underscores (max 1024 chars)',
|
||||
}),
|
||||
tableId: z
|
||||
.string()
|
||||
.min(1, 'tableId is required')
|
||||
.refine((v) => TABLE_RE.test(v), {
|
||||
message:
|
||||
'tableId may only contain Unicode letters/marks/numbers, connectors, dashes, and spaces (max 1024 chars)',
|
||||
})
|
||||
.refine((v) => Buffer.byteLength(v, 'utf8') <= 1024, {
|
||||
message: 'tableId must be at most 1024 bytes when UTF-8 encoded',
|
||||
}),
|
||||
})
|
||||
|
||||
const bigqueryCredentialsSchema = z
|
||||
.object({
|
||||
serviceAccountJson: z.string().min(1, 'serviceAccountJson is required'),
|
||||
})
|
||||
.superRefine(refineServiceAccountJson)
|
||||
|
||||
export type BigQueryDestinationConfig = z.infer<typeof bigqueryConfigSchema>
|
||||
export type BigQueryDestinationCredentials = z.infer<typeof bigqueryCredentialsSchema>
|
||||
|
||||
function buildJwt(account: ParsedServiceAccount): JWT {
|
||||
return new JWT({ email: account.clientEmail, key: account.privateKey, scopes: SCOPES })
|
||||
}
|
||||
|
||||
async function getAccessToken(jwt: JWT, forceRefresh = false): Promise<string> {
|
||||
if (forceRefresh) {
|
||||
/** Clearing `credentials` forces `getAccessToken` to mint a new token instead of returning the cached one. */
|
||||
jwt.credentials = {}
|
||||
}
|
||||
const { token } = await jwt.getAccessToken()
|
||||
if (!token) throw new Error('Failed to obtain BigQuery access token')
|
||||
return token
|
||||
}
|
||||
|
||||
interface InsertAllInput {
|
||||
config: BigQueryDestinationConfig
|
||||
rows: Record<string, unknown>[]
|
||||
metadata: DeliveryMetadata
|
||||
jwt: JWT
|
||||
signal: AbortSignal
|
||||
}
|
||||
|
||||
interface InsertAllError {
|
||||
index: number
|
||||
errors: Array<{ reason?: string; message?: string; location?: string }>
|
||||
}
|
||||
|
||||
async function postInsertAll(
|
||||
input: InsertAllInput,
|
||||
url: string,
|
||||
body: string,
|
||||
forceRefresh = false
|
||||
): Promise<Response> {
|
||||
const token = await getAccessToken(input.jwt, forceRefresh)
|
||||
const perAttempt = AbortSignal.any([input.signal, AbortSignal.timeout(PER_ATTEMPT_TIMEOUT_MS)])
|
||||
try {
|
||||
return await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${token}`,
|
||||
'Content-Type': 'application/json',
|
||||
'User-Agent': USER_AGENT,
|
||||
},
|
||||
body,
|
||||
signal: perAttempt,
|
||||
})
|
||||
} catch (error) {
|
||||
logger.warn('BigQuery request failed', {
|
||||
table: `${input.config.projectId}.${input.config.datasetId}.${input.config.tableId}`,
|
||||
error: toError(error).message,
|
||||
})
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a stable `insertId` for best-effort dedup (~60s window). Prefixed
|
||||
* with `drainId` so (runId, sequence) collisions across drains do not
|
||||
* accidentally dedupe each other's rows. With UUID drain/run IDs the raw
|
||||
* form fits well under 128 chars; if anything pushes it over (e.g. a future
|
||||
* non-UUID id), hash the prefix and keep the row-distinguishing `index`
|
||||
* suffix intact so BigQuery does not silently dedupe distinct rows.
|
||||
*/
|
||||
function buildInsertId(metadata: DeliveryMetadata, index: number): string {
|
||||
const raw = `${metadata.drainId}-${metadata.runId}-${metadata.sequence}-${index}`
|
||||
if (raw.length <= MAX_INSERT_ID_LENGTH) return raw
|
||||
const prefixHash = createHash('sha1')
|
||||
.update(`${metadata.drainId}-${metadata.runId}-${metadata.sequence}`)
|
||||
.digest('hex')
|
||||
return `${prefixHash}-${index}`
|
||||
}
|
||||
|
||||
const RETRYABLE_STATUSES = new Set([408, 429, 500, 502, 503, 504])
|
||||
const MAX_RETRY_ATTEMPTS = 3
|
||||
const BASE_RETRY_DELAY_MS = 250
|
||||
|
||||
/**
|
||||
* Streams a chunk of rows to `tabledata.insertAll`.
|
||||
*
|
||||
* Partial-success caveat: BigQuery may return HTTP 200 with a non-empty
|
||||
* `insertErrors` array. Rows not listed there are inserted and dedup-keyed by
|
||||
* `insertId` for ~60s. We throw on any `insertErrors`; retries within the
|
||||
* dedup window are safe, but retries after it may duplicate succeeded rows.
|
||||
*/
|
||||
async function insertAll(input: InsertAllInput): Promise<void> {
|
||||
if (input.rows.length > MAX_ROWS_PER_REQUEST) {
|
||||
throw new Error(
|
||||
`BigQuery insertAll chunk has ${input.rows.length} rows, exceeds the ${MAX_ROWS_PER_REQUEST} per-request limit`
|
||||
)
|
||||
}
|
||||
const url = `https://bigquery.googleapis.com/bigquery/v2/projects/${encodeURIComponent(input.config.projectId)}/datasets/${encodeURIComponent(input.config.datasetId)}/tables/${encodeURIComponent(input.config.tableId)}/insertAll`
|
||||
/**
|
||||
* `skipInvalidRows: false` and `ignoreUnknownValues: false` surface schema
|
||||
* mismatches as `insertErrors` instead of silently dropping data — drains
|
||||
* should fail loudly so operators notice the schema drift.
|
||||
*/
|
||||
const payload = {
|
||||
skipInvalidRows: false,
|
||||
ignoreUnknownValues: false,
|
||||
rows: input.rows.map((row, index) => {
|
||||
const rowBytes = Buffer.byteLength(JSON.stringify(row), 'utf8')
|
||||
if (rowBytes > MAX_ROW_BYTES) {
|
||||
throw new Error(
|
||||
`BigQuery row at index ${index} is ${rowBytes} bytes, exceeds the ${MAX_ROW_BYTES}-byte per-row limit`
|
||||
)
|
||||
}
|
||||
return {
|
||||
insertId: buildInsertId(input.metadata, index),
|
||||
json: row,
|
||||
}
|
||||
}),
|
||||
}
|
||||
const body = JSON.stringify(payload)
|
||||
const byteLength = Buffer.byteLength(body, 'utf8')
|
||||
if (byteLength > MAX_REQUEST_BYTES) {
|
||||
throw new Error(
|
||||
`BigQuery insertAll body is ${byteLength} bytes, exceeds the ${MAX_REQUEST_BYTES}-byte per-request limit`
|
||||
)
|
||||
}
|
||||
let attempt = 0
|
||||
let response: Response | undefined
|
||||
let refreshedOnce = false
|
||||
while (true) {
|
||||
attempt++
|
||||
try {
|
||||
response = await postInsertAll(input, url, body)
|
||||
/** A 401 retry doesn't count against the 5xx/429 budget — token refresh is a one-shot recovery. */
|
||||
if (response.status === 401 && !refreshedOnce) {
|
||||
refreshedOnce = true
|
||||
logger.debug('BigQuery returned 401; refreshing access token and retrying once')
|
||||
/** Drain the 401 body before discarding so undici can return the socket to the keep-alive pool. */
|
||||
await response.text().catch(() => '')
|
||||
response = await postInsertAll(input, url, body, true)
|
||||
}
|
||||
if (!RETRYABLE_STATUSES.has(response.status)) break
|
||||
if (attempt >= MAX_RETRY_ATTEMPTS) break
|
||||
const retryAfterHeaderMs = parseRetryAfter(response.headers.get('retry-after'))
|
||||
const retryAfterMs = backoffWithJitter(attempt, retryAfterHeaderMs, {
|
||||
baseMs: BASE_RETRY_DELAY_MS,
|
||||
})
|
||||
logger.warn('BigQuery insertAll transient error; retrying', {
|
||||
status: response.status,
|
||||
attempt,
|
||||
retryAfterMs,
|
||||
})
|
||||
/** Drain the body so the keep-alive connection can be reused. */
|
||||
await response.text().catch(() => '')
|
||||
await sleepUntilAborted(retryAfterMs, input.signal)
|
||||
if (input.signal.aborted) throw input.signal.reason ?? new Error('Aborted')
|
||||
} catch (error) {
|
||||
/**
|
||||
* Connection-level failures (DNS, socket reset, timeout) never produce
|
||||
* a Response — treat them like 5xx and retry with backoff. Re-throw
|
||||
* aborts unwrapped so callers see the cancellation reason.
|
||||
*/
|
||||
if (input.signal.aborted) throw input.signal.reason ?? error
|
||||
if (attempt >= MAX_RETRY_ATTEMPTS) throw error
|
||||
const retryAfterMs = backoffWithJitter(attempt, null, { baseMs: BASE_RETRY_DELAY_MS })
|
||||
logger.warn('BigQuery insertAll network error; retrying', {
|
||||
attempt,
|
||||
retryAfterMs,
|
||||
error: toError(error).message,
|
||||
})
|
||||
await sleepUntilAborted(retryAfterMs, input.signal)
|
||||
if (input.signal.aborted) throw input.signal.reason ?? new Error('Aborted')
|
||||
}
|
||||
}
|
||||
if (!response) throw new Error('BigQuery insertAll failed: no response')
|
||||
if (!response.ok) {
|
||||
const text = await response.text().catch(() => '')
|
||||
throw new Error(`BigQuery insertAll failed (HTTP ${response.status}): ${text}`)
|
||||
}
|
||||
const result = (await response.json().catch(() => ({}))) as {
|
||||
insertErrors?: InsertAllError[]
|
||||
}
|
||||
if (result.insertErrors && result.insertErrors.length > 0) {
|
||||
const failedIndices = result.insertErrors.map((e) => e.index)
|
||||
const total = input.rows.length
|
||||
const failed = result.insertErrors.length
|
||||
const succeeded = total - failed
|
||||
logger.warn('BigQuery insertAll returned partial failure', {
|
||||
partialFailure: true,
|
||||
table: `${input.config.projectId}.${input.config.datasetId}.${input.config.tableId}`,
|
||||
succeededRows: succeeded,
|
||||
failedRows: failed,
|
||||
failedIndices: failedIndices.slice(0, 20),
|
||||
})
|
||||
const summary = result.insertErrors
|
||||
.slice(0, 3)
|
||||
.map(
|
||||
(e) =>
|
||||
`row ${e.index}: ${e.errors.map((er) => er.message ?? er.reason ?? 'unknown').join('; ')}`
|
||||
)
|
||||
.join(' | ')
|
||||
throw new Error(
|
||||
`BigQuery insertAll partial failure: ${failed} of ${total} rows failed (indices: ${failedIndices.slice(0, 20).join(',')}${failedIndices.length > 20 ? ',...' : ''}); ${succeeded} rows were inserted and are dedup-keyed by insertId for ~60s — retries within that window are safe, but retries after the window may duplicate the succeeded rows. First errors: ${summary}`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export const bigqueryDestination: DrainDestination<
|
||||
BigQueryDestinationConfig,
|
||||
BigQueryDestinationCredentials
|
||||
> = {
|
||||
type: 'bigquery',
|
||||
displayName: 'Google BigQuery',
|
||||
configSchema: bigqueryConfigSchema,
|
||||
credentialsSchema: bigqueryCredentialsSchema,
|
||||
|
||||
/**
|
||||
* Probes table existence, IAM access, and credential validity in a single
|
||||
* `tables.get` call. `fields=id` minimises response size — we only care
|
||||
* whether the call succeeds, not the payload.
|
||||
*/
|
||||
async test({ config, credentials, signal }) {
|
||||
const account = parseServiceAccount(credentials.serviceAccountJson)
|
||||
const jwt = buildJwt(account)
|
||||
const token = await getAccessToken(jwt)
|
||||
const url = `https://bigquery.googleapis.com/bigquery/v2/projects/${encodeURIComponent(config.projectId)}/datasets/${encodeURIComponent(config.datasetId)}/tables/${encodeURIComponent(config.tableId)}?fields=id`
|
||||
const perAttempt = AbortSignal.any([signal, AbortSignal.timeout(PER_ATTEMPT_TIMEOUT_MS)])
|
||||
const response = await fetch(url, {
|
||||
headers: { Authorization: `Bearer ${token}` },
|
||||
signal: perAttempt,
|
||||
})
|
||||
if (!response.ok) {
|
||||
const text = await response.text().catch(() => '')
|
||||
throw new Error(`BigQuery probe failed (HTTP ${response.status}): ${text}`)
|
||||
}
|
||||
/** Drain the success body so undici can return the socket to the keep-alive pool. */
|
||||
await response.text().catch(() => '')
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
const account = parseServiceAccount(credentials.serviceAccountJson)
|
||||
const jwt = buildJwt(account)
|
||||
return {
|
||||
async deliver({ body, metadata, signal }) {
|
||||
const rows = parseNdjsonObjects(body, { requireObject: true }) as Record<string, unknown>[]
|
||||
if (rows.length === 0) {
|
||||
return {
|
||||
locator: `bigquery://${config.projectId}/${config.datasetId}/${config.tableId}#${metadata.runId}-${metadata.sequence}`,
|
||||
}
|
||||
}
|
||||
await insertAll({ config, rows, metadata, jwt, signal })
|
||||
logger.debug('BigQuery chunk delivered', {
|
||||
table: `${config.projectId}.${config.datasetId}.${config.tableId}`,
|
||||
rows: rows.length,
|
||||
})
|
||||
return {
|
||||
locator: `bigquery://${config.projectId}/${config.datasetId}/${config.tableId}#${metadata.runId}-${metadata.sequence}`,
|
||||
}
|
||||
},
|
||||
async close() {},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { gunzipSync } from 'node:zlib'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const fetchMock = vi.fn(async () => new Response(null, { status: 202 }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
import { datadogDestination } from '@/lib/data-drains/destinations/datadog'
|
||||
|
||||
const config = { site: 'us1' as const, service: 'sim', tags: 'env:prod' }
|
||||
const credentials = { apiKey: 'dd-key' }
|
||||
|
||||
const meta = (sequence: number) => ({
|
||||
drainId: 'd1',
|
||||
runId: 'r1',
|
||||
source: 'workflow_logs' as const,
|
||||
sequence,
|
||||
rowCount: 2,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
})
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
fetchMock.mockResolvedValue(new Response(null, { status: 202 }))
|
||||
})
|
||||
|
||||
describe('datadogDestination', () => {
|
||||
it('parses NDJSON and POSTs a JSON array of log entries', async () => {
|
||||
const session = datadogDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ id: 'a', name: 'one' })}\n${JSON.stringify({ id: 'b', name: 'two' })}\n`,
|
||||
'utf8'
|
||||
)
|
||||
const result = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toBe('https://http-intake.logs.datadoghq.com/api/v2/logs')
|
||||
const headers = init.headers as Record<string, string>
|
||||
expect(headers['DD-API-KEY']).toBe('dd-key')
|
||||
expect(headers['Content-Type']).toBe('application/json')
|
||||
expect(headers.Accept).toBe('application/json')
|
||||
expect(headers['User-Agent']).toBe('sim-data-drain/1.0')
|
||||
expect(headers['Content-Encoding']).toBeUndefined()
|
||||
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload).toHaveLength(2)
|
||||
expect(payload[0].ddsource).toBe('sim')
|
||||
expect(payload[0].service).toBe('sim')
|
||||
expect(payload[0].ddtags).toContain('sim_drain_id:d1')
|
||||
expect(payload[0].ddtags).toContain('env:prod')
|
||||
expect(payload[0].id).toBe('a')
|
||||
expect(payload[0].name).toBe('one')
|
||||
expect(payload[0].attributes).toBeUndefined()
|
||||
|
||||
expect(result.locator).toMatch(/^datadog:\/\/us1#r1-0/)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('retries 5xx responses then surfaces the final error', async () => {
|
||||
vi.useFakeTimers()
|
||||
try {
|
||||
fetchMock.mockResolvedValue(new Response('boom', { status: 503 }))
|
||||
const session = datadogDestination.openSession({ config, credentials })
|
||||
const promise = session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
// Attach a handler so Node doesn't flag the in-flight rejection while
|
||||
// we advance fake timers; we still assert via the original promise below.
|
||||
const settled = promise.catch((e) => e)
|
||||
await vi.runAllTimersAsync()
|
||||
await expect(settled).resolves.toMatchObject({ message: expect.stringMatching(/HTTP 503/) })
|
||||
expect(fetchMock).toHaveBeenCalledTimes(4)
|
||||
await session.close()
|
||||
} finally {
|
||||
vi.useRealTimers()
|
||||
}
|
||||
})
|
||||
|
||||
it('does not retry on non-retryable 4xx (e.g. invalid API key)', async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response('forbidden', { status: 403 }))
|
||||
const session = datadogDestination.openSession({ config, credentials })
|
||||
await expect(
|
||||
session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/HTTP 403/)
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('routes to the EU site host', async () => {
|
||||
const session = datadogDestination.openSession({
|
||||
config: { ...config, site: 'eu1' },
|
||||
credentials,
|
||||
})
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock.mock.calls[0]?.[0]).toBe('https://http-intake.logs.datadoghq.eu/api/v2/logs')
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('routes to the AP2 site host', async () => {
|
||||
const session = datadogDestination.openSession({
|
||||
config: { ...config, site: 'ap2' },
|
||||
credentials,
|
||||
})
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock.mock.calls[0]?.[0]).toBe(
|
||||
'https://http-intake.logs.ap2.datadoghq.com/api/v2/logs'
|
||||
)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('throws with the entry index when a single entry exceeds 1 MB', async () => {
|
||||
const session = datadogDestination.openSession({ config, credentials })
|
||||
// Two entries; the second exceeds the 1 MB per-entry limit.
|
||||
const huge = 'x'.repeat(1024 * 1024 + 10)
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ id: 'small' })}\n${JSON.stringify({ blob: huge })}\n`,
|
||||
'utf8'
|
||||
)
|
||||
await expect(
|
||||
session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/entry at index 1 is .* exceeds the 1048576-byte per-entry limit/)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('gzips payloads larger than 1KB and sets Content-Encoding: gzip', async () => {
|
||||
const session = datadogDestination.openSession({ config, credentials })
|
||||
// Build > 1KB raw payload; padding string is JSON-safe.
|
||||
const padding = 'a'.repeat(2048)
|
||||
const body = Buffer.from(`${JSON.stringify({ id: 'a', big: padding })}\n`, 'utf8')
|
||||
await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(0),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const init = fetchMock.mock.calls[0]?.[1] as RequestInit
|
||||
const headers = init.headers as Record<string, string>
|
||||
expect(headers['Content-Encoding']).toBe('gzip')
|
||||
expect(init.body).toBeInstanceOf(Uint8Array)
|
||||
expect(typeof init.body).not.toBe('string')
|
||||
const decoded = JSON.parse(gunzipSync(init.body as Uint8Array).toString('utf8'))
|
||||
expect(decoded).toHaveLength(1)
|
||||
expect(decoded[0].id).toBe('a')
|
||||
expect(decoded[0].big).toBe(padding)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('locator includes the dd-request-id header when present', async () => {
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
new Response(null, { status: 202, headers: { 'dd-request-id': 'req-abc-123' } })
|
||||
)
|
||||
const session = datadogDestination.openSession({ config, credentials })
|
||||
const result = await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta(7),
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(result.locator).toBe('datadog://us1#r1-7@req-abc-123')
|
||||
await session.close()
|
||||
})
|
||||
})
|
||||
|
||||
describe('datadogDestination test()', () => {
|
||||
it('sends a single probe entry', async () => {
|
||||
await datadogDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const init = fetchMock.mock.calls[0]?.[1] as RequestInit
|
||||
const headers = init.headers as Record<string, string>
|
||||
expect(headers.Accept).toBe('application/json')
|
||||
expect(headers['User-Agent']).toBe('sim-data-drain/1.0')
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload).toHaveLength(1)
|
||||
expect(payload[0].message).toContain('connection test')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,274 @@
|
||||
import { gzipSync } from 'node:zlib'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { z } from 'zod'
|
||||
import {
|
||||
backoffWithJitter,
|
||||
parseNdjsonObjects,
|
||||
parseRetryAfter,
|
||||
sleepUntilAborted,
|
||||
} from '@/lib/data-drains/destinations/utils'
|
||||
import type { DeliveryMetadata, DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainDatadogDestination')
|
||||
|
||||
const DATADOG_SITES = ['us1', 'us3', 'us5', 'eu1', 'ap1', 'ap2', 'gov'] as const
|
||||
|
||||
type DatadogSite = (typeof DATADOG_SITES)[number]
|
||||
|
||||
const SITE_HOSTS: Record<DatadogSite, string> = {
|
||||
us1: 'datadoghq.com',
|
||||
us3: 'us3.datadoghq.com',
|
||||
us5: 'us5.datadoghq.com',
|
||||
eu1: 'datadoghq.eu',
|
||||
ap1: 'ap1.datadoghq.com',
|
||||
ap2: 'ap2.datadoghq.com',
|
||||
gov: 'ddog-gov.com',
|
||||
}
|
||||
|
||||
const MAX_ATTEMPTS = 4
|
||||
const PER_ATTEMPT_TIMEOUT_MS = 30_000
|
||||
const MAX_UNCOMPRESSED_BYTES = 5 * 1024 * 1024
|
||||
const MAX_WIRE_BYTES = 6 * 1024 * 1024
|
||||
const MAX_ENTRY_BYTES = 1024 * 1024
|
||||
const MAX_ENTRIES_PER_REQUEST = 1000
|
||||
const GZIP_THRESHOLD_BYTES = 1024
|
||||
|
||||
/**
|
||||
* Datadog tag format: comma-separated `key:value` pairs. Each key must start
|
||||
* with a letter and contain only [A-Za-z0-9_:./-]. Validating here so the
|
||||
* `ddtags` header we emit can't be mangled by user-supplied free-form input.
|
||||
*/
|
||||
const DATADOG_TAG_PAIR_RE = /^[A-Za-z][A-Za-z0-9_./-]*:[^,\s][^,]*$/
|
||||
|
||||
const datadogConfigSchema = z.object({
|
||||
site: z.enum(DATADOG_SITES),
|
||||
service: z.string().min(1).max(100).optional(),
|
||||
tags: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(1024)
|
||||
.refine(
|
||||
(v) =>
|
||||
v
|
||||
.split(',')
|
||||
.map((t) => t.trim())
|
||||
.filter((t) => t.length > 0)
|
||||
.every((t) => DATADOG_TAG_PAIR_RE.test(t)),
|
||||
{ message: 'tags must be comma-separated key:value pairs' }
|
||||
)
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const datadogCredentialsSchema = z.object({
|
||||
apiKey: z.string().min(1, 'apiKey is required'),
|
||||
})
|
||||
|
||||
export type DatadogDestinationConfig = z.infer<typeof datadogConfigSchema>
|
||||
export type DatadogDestinationCredentials = z.infer<typeof datadogCredentialsSchema>
|
||||
|
||||
interface DatadogLogEntry {
|
||||
ddsource: string
|
||||
service: string
|
||||
ddtags: string
|
||||
message: string
|
||||
[attribute: string]: unknown
|
||||
}
|
||||
|
||||
function buildEndpoint(site: DatadogSite): string {
|
||||
return `https://http-intake.logs.${SITE_HOSTS[site]}/api/v2/logs`
|
||||
}
|
||||
|
||||
function buildEntries(
|
||||
rows: unknown[],
|
||||
config: DatadogDestinationConfig,
|
||||
metadata: DeliveryMetadata
|
||||
): DatadogLogEntry[] {
|
||||
const ddtags = [
|
||||
`sim_drain_id:${metadata.drainId}`,
|
||||
`sim_run_id:${metadata.runId}`,
|
||||
`sim_source:${metadata.source}`,
|
||||
...(config.tags ? [config.tags] : []),
|
||||
].join(',')
|
||||
const service = config.service ?? 'sim'
|
||||
return rows.map((row) => {
|
||||
const attrs = typeof row === 'object' && row !== null ? (row as Record<string, unknown>) : {}
|
||||
let message: string
|
||||
if (typeof row === 'string') {
|
||||
message = row
|
||||
} else if (typeof attrs.message === 'string') {
|
||||
message = attrs.message
|
||||
} else {
|
||||
message = JSON.stringify(row)
|
||||
}
|
||||
/**
|
||||
* Spread user attributes first, then force all four reserved fields the
|
||||
* drain owns: `ddsource`, `service`, `ddtags`, and `message`. Per
|
||||
* https://docs.datadoghq.com/logs/log_configuration/pipelines/#service-and-source,
|
||||
* Datadog uses `service` + `ddsource` to pick the processing pipeline, so
|
||||
* letting a row field clobber them would silently re-route a drain.
|
||||
*/
|
||||
return {
|
||||
...attrs,
|
||||
ddsource: 'sim',
|
||||
service,
|
||||
ddtags,
|
||||
message,
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
function isRetryableStatus(status: number): boolean {
|
||||
return status === 408 || status === 429 || status >= 500
|
||||
}
|
||||
|
||||
interface PreparedBody {
|
||||
body: Uint8Array | string
|
||||
headers: Record<string, string>
|
||||
wireBytes: number
|
||||
rawBytes: number
|
||||
}
|
||||
|
||||
interface PostInput {
|
||||
url: string
|
||||
prepared: PreparedBody
|
||||
signal: AbortSignal
|
||||
}
|
||||
|
||||
function buildRequestBody(payload: string, apiKey: string): PreparedBody {
|
||||
const headers: Record<string, string> = {
|
||||
'Content-Type': 'application/json',
|
||||
'DD-API-KEY': apiKey,
|
||||
Accept: 'application/json',
|
||||
'User-Agent': 'sim-data-drain/1.0',
|
||||
}
|
||||
const rawBytes = Buffer.byteLength(payload, 'utf8')
|
||||
if (rawBytes > GZIP_THRESHOLD_BYTES) {
|
||||
const compressed = gzipSync(payload)
|
||||
headers['Content-Encoding'] = 'gzip'
|
||||
const view = new Uint8Array(compressed.buffer, compressed.byteOffset, compressed.byteLength)
|
||||
return { body: view, headers, wireBytes: view.byteLength, rawBytes }
|
||||
}
|
||||
return { body: payload, headers, wireBytes: rawBytes, rawBytes }
|
||||
}
|
||||
|
||||
async function postWithRetries(input: PostInput): Promise<Response> {
|
||||
const { body, headers } = input.prepared
|
||||
let lastError: unknown
|
||||
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
|
||||
if (input.signal.aborted) throw input.signal.reason ?? new Error('Aborted')
|
||||
const perAttempt = AbortSignal.any([input.signal, AbortSignal.timeout(PER_ATTEMPT_TIMEOUT_MS)])
|
||||
let retryAfterMs: number | null = null
|
||||
let response: Response | undefined
|
||||
try {
|
||||
response = await fetch(input.url, {
|
||||
method: 'POST',
|
||||
// double-cast-allowed: Uint8Array is a valid runtime BodyInit but the DOM lib types only enumerate Blob/FormData/string/etc.
|
||||
body: body as unknown as BodyInit,
|
||||
headers,
|
||||
signal: perAttempt,
|
||||
})
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
logger.debug('Datadog request failed', { attempt, error: toError(error).message })
|
||||
}
|
||||
if (response) {
|
||||
if (response.ok) {
|
||||
/** Drain the success body so undici can return the socket to the keep-alive pool. Headers remain readable after consumption. */
|
||||
await response.text().catch(() => '')
|
||||
return response
|
||||
}
|
||||
if (!isRetryableStatus(response.status)) {
|
||||
const text = await response.text().catch(() => '')
|
||||
throw new Error(`Datadog responded with HTTP ${response.status}: ${text}`)
|
||||
}
|
||||
lastError = new Error(`Datadog responded with HTTP ${response.status}`)
|
||||
retryAfterMs = parseRetryAfter(response.headers.get('retry-after'))
|
||||
/** Drain the retryable response body so undici can return the socket to the keep-alive pool. */
|
||||
await response.text().catch(() => '')
|
||||
}
|
||||
if (attempt < MAX_ATTEMPTS) {
|
||||
await sleepUntilAborted(backoffWithJitter(attempt, retryAfterMs), input.signal)
|
||||
}
|
||||
}
|
||||
throw lastError instanceof Error ? lastError : new Error('Datadog delivery failed after retries')
|
||||
}
|
||||
|
||||
export const datadogDestination: DrainDestination<
|
||||
DatadogDestinationConfig,
|
||||
DatadogDestinationCredentials
|
||||
> = {
|
||||
type: 'datadog',
|
||||
displayName: 'Datadog',
|
||||
configSchema: datadogConfigSchema,
|
||||
credentialsSchema: datadogCredentialsSchema,
|
||||
|
||||
async test({ config, credentials, signal }) {
|
||||
const probe = [
|
||||
{
|
||||
ddsource: 'sim',
|
||||
service: config.service ?? 'sim',
|
||||
ddtags: `sim_probe:1${config.tags ? `,${config.tags}` : ''}`,
|
||||
message: 'sim-data-drain connection test',
|
||||
},
|
||||
]
|
||||
await postWithRetries({
|
||||
url: buildEndpoint(config.site),
|
||||
prepared: buildRequestBody(JSON.stringify(probe), credentials.apiKey),
|
||||
signal,
|
||||
})
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
const url = buildEndpoint(config.site)
|
||||
return {
|
||||
async deliver({ body, metadata, signal }) {
|
||||
const rows = parseNdjsonObjects(body)
|
||||
const entries = buildEntries(rows, config, metadata)
|
||||
if (entries.length > MAX_ENTRIES_PER_REQUEST) {
|
||||
throw new Error(
|
||||
`Datadog chunk has ${entries.length} entries, exceeds the ${MAX_ENTRIES_PER_REQUEST} per-request limit`
|
||||
)
|
||||
}
|
||||
for (let i = 0; i < entries.length; i++) {
|
||||
const entryBytes = Buffer.byteLength(JSON.stringify(entries[i]), 'utf8')
|
||||
if (entryBytes > MAX_ENTRY_BYTES) {
|
||||
throw new Error(
|
||||
`Datadog entry at index ${i} is ${entryBytes} bytes, exceeds the ${MAX_ENTRY_BYTES}-byte per-entry limit`
|
||||
)
|
||||
}
|
||||
}
|
||||
const payload = JSON.stringify(entries)
|
||||
const prepared = buildRequestBody(payload, credentials.apiKey)
|
||||
if (prepared.rawBytes > MAX_UNCOMPRESSED_BYTES) {
|
||||
throw new Error(
|
||||
`Datadog payload is ${prepared.rawBytes} bytes uncompressed, exceeds the ${MAX_UNCOMPRESSED_BYTES}-byte per-request limit`
|
||||
)
|
||||
}
|
||||
if (prepared.wireBytes > MAX_WIRE_BYTES) {
|
||||
throw new Error(
|
||||
`Datadog payload is ${prepared.wireBytes} bytes on the wire, exceeds the ${MAX_WIRE_BYTES}-byte defensive wire-size cap`
|
||||
)
|
||||
}
|
||||
const response = await postWithRetries({
|
||||
url,
|
||||
prepared,
|
||||
signal,
|
||||
})
|
||||
const requestId = response.headers.get('dd-request-id') ?? null
|
||||
logger.debug('Datadog chunk delivered', {
|
||||
site: config.site,
|
||||
rows: entries.length,
|
||||
rawBytes: prepared.rawBytes,
|
||||
wireBytes: prepared.wireBytes,
|
||||
})
|
||||
return {
|
||||
locator: requestId
|
||||
? `datadog://${config.site}#${metadata.runId}-${metadata.sequence}@${requestId}`
|
||||
: `datadog://${config.site}#${metadata.runId}-${metadata.sequence}`,
|
||||
}
|
||||
},
|
||||
async close() {},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const { mockGetAccessToken, JWTCtor } = vi.hoisted(() => {
|
||||
const mockGetAccessToken = vi.fn(async () => ({ token: 'fake-access-token' }))
|
||||
return {
|
||||
mockGetAccessToken,
|
||||
JWTCtor: vi.fn(() => ({ getAccessToken: mockGetAccessToken })),
|
||||
}
|
||||
})
|
||||
|
||||
vi.mock('google-auth-library', () => ({ JWT: JWTCtor }))
|
||||
|
||||
const fetchMock = vi.fn(async () => new Response(null, { status: 200 }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
import { gcsDestination } from '@/lib/data-drains/destinations/gcs'
|
||||
|
||||
const config = { bucket: 'my-bucket', prefix: 'sim/' }
|
||||
const credentials = {
|
||||
serviceAccountJson: JSON.stringify({
|
||||
client_email: 'sa@project.iam.gserviceaccount.com',
|
||||
private_key: '-----BEGIN PRIVATE KEY-----\nfake\n-----END PRIVATE KEY-----\n',
|
||||
}),
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
fetchMock.mockResolvedValue(new Response(null, { status: 200 }))
|
||||
})
|
||||
|
||||
describe('gcsDestination openSession', () => {
|
||||
it('uploads via the JSON API and returns a gs:// locator', async () => {
|
||||
const session = gcsDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from('row\n', 'utf8')
|
||||
const result = await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd1',
|
||||
runId: 'r1',
|
||||
source: 'workflow_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
expect(result.locator).toMatch(
|
||||
/^gs:\/\/my-bucket\/sim\/workflow_logs\/d1\/\d{4}\/\d{2}\/\d{2}\/r1-00000\.ndjson$/
|
||||
)
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toContain('/upload/storage/v1/b/my-bucket/o')
|
||||
expect(url).toContain('uploadType=media')
|
||||
const headers = init.headers as Record<string, string>
|
||||
expect(headers.Authorization).toBe('Bearer fake-access-token')
|
||||
expect(headers['Content-Type']).toBe('application/x-ndjson')
|
||||
expect(headers['x-goog-meta-sim-drain-id']).toBe('d1')
|
||||
expect(headers['x-goog-meta-sim-sequence']).toBe('0')
|
||||
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('surfaces non-2xx responses as errors', async () => {
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
new Response('Permission denied', { status: 403, statusText: 'Forbidden' })
|
||||
)
|
||||
const session = gcsDestination.openSession({ config, credentials })
|
||||
await expect(
|
||||
session.deliver({
|
||||
body: Buffer.from('x'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'audit_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/HTTP 403/)
|
||||
await session.close()
|
||||
})
|
||||
})
|
||||
|
||||
describe('gcsDestination test()', () => {
|
||||
it('writes a probe object then attempts cleanup', async () => {
|
||||
await gcsDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
const [, deleteCall] = fetchMock.mock.calls
|
||||
expect((deleteCall[1] as RequestInit).method).toBe('DELETE')
|
||||
})
|
||||
})
|
||||
|
||||
describe('gcsDestination credentials schema', () => {
|
||||
it('rejects invalid JSON', () => {
|
||||
const result = gcsDestination.credentialsSchema.safeParse({ serviceAccountJson: 'not-json' })
|
||||
expect(result.success).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects JSON missing client_email', () => {
|
||||
const result = gcsDestination.credentialsSchema.safeParse({
|
||||
serviceAccountJson: JSON.stringify({ private_key: 'k' }),
|
||||
})
|
||||
expect(result.success).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('gcsDestination config schema', () => {
|
||||
it('accepts a 3-character bucket name', () => {
|
||||
const result = gcsDestination.configSchema.safeParse({ bucket: 'abc' })
|
||||
expect(result.success).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects bucket names beginning with goog or containing google', () => {
|
||||
expect(gcsDestination.configSchema.safeParse({ bucket: 'goog-prefixed' }).success).toBe(false)
|
||||
expect(gcsDestination.configSchema.safeParse({ bucket: 'my-google-bucket' }).success).toBe(
|
||||
false
|
||||
)
|
||||
expect(gcsDestination.configSchema.safeParse({ bucket: 'g00gle-bucket' }).success).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('gcsDestination upload headers', () => {
|
||||
it('does not set a Content-Length header on uploads', async () => {
|
||||
const session = gcsDestination.openSession({ config, credentials })
|
||||
await session.deliver({
|
||||
body: Buffer.from('row\n', 'utf8'),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: {
|
||||
drainId: 'd1',
|
||||
runId: 'r1',
|
||||
source: 'workflow_logs',
|
||||
sequence: 0,
|
||||
rowCount: 1,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
},
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const [, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
const headers = init.headers as Record<string, string>
|
||||
const headerKeys = Object.keys(headers).map((k) => k.toLowerCase())
|
||||
expect(headerKeys).not.toContain('content-length')
|
||||
expect(headers['User-Agent']).toBe('sim-data-drain/1.0')
|
||||
await session.close()
|
||||
})
|
||||
})
|
||||
|
||||
describe('gcsDestination deleteObject behavior', () => {
|
||||
it('treats 404 as success on delete during test() cleanup', async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response(null, { status: 200 })) // upload probe
|
||||
fetchMock.mockResolvedValueOnce(new Response(null, { status: 404 })) // delete probe
|
||||
await expect(
|
||||
gcsDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).resolves.toBeUndefined()
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('retries DELETE on 503 then succeeds on 200', async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response(null, { status: 200 })) // upload probe
|
||||
fetchMock.mockResolvedValueOnce(new Response('busy', { status: 503 })) // first delete attempt
|
||||
fetchMock.mockResolvedValueOnce(new Response(null, { status: 200 })) // retry succeeds
|
||||
await gcsDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock).toHaveBeenCalledTimes(3)
|
||||
const [, deleteCall1] = fetchMock.mock.calls[1] as [string, RequestInit]
|
||||
const [, deleteCall2] = fetchMock.mock.calls[2] as [string, RequestInit]
|
||||
expect(deleteCall1.method).toBe('DELETE')
|
||||
expect(deleteCall2.method).toBe('DELETE')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,321 @@
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { generateShortId } from '@sim/utils/id'
|
||||
import { JWT } from 'google-auth-library'
|
||||
import { z } from 'zod'
|
||||
import {
|
||||
backoffWithJitter,
|
||||
buildObjectKey,
|
||||
normalizePrefix,
|
||||
type ParsedServiceAccount,
|
||||
parseRetryAfter,
|
||||
parseServiceAccount,
|
||||
refineServiceAccountJson,
|
||||
sleepUntilAborted,
|
||||
} from '@/lib/data-drains/destinations/utils'
|
||||
import type { DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainGCSDestination')
|
||||
|
||||
const SCOPE = 'https://www.googleapis.com/auth/devstorage.read_write'
|
||||
const GCS_HOST = 'https://storage.googleapis.com'
|
||||
const USER_AGENT = 'sim-data-drain/1.0'
|
||||
const MAX_ATTEMPTS = 4
|
||||
const PER_ATTEMPT_TIMEOUT_MS = 60_000
|
||||
/** GCS caps total custom metadata at 8 KiB per object (sum of key + value bytes). */
|
||||
const MAX_CUSTOM_METADATA_BYTES = 8 * 1024
|
||||
/** GCS object names are at most 1024 bytes when UTF-8 encoded (flat-namespace buckets). */
|
||||
const MAX_OBJECT_NAME_BYTES = 1024
|
||||
|
||||
const GCS_BUCKET_COMPONENT_RE = /^[a-z0-9]([a-z0-9_-]*[a-z0-9])?$/
|
||||
const IPV4_LIKE_RE = /^(\d{1,3}\.){3}\d{1,3}$/
|
||||
const GOOGLE_RESERVED_RE = /^(goog|google|g00gle)/i
|
||||
const GOOGLE_CONTAINS_RE = /(google|g00gle)/i
|
||||
|
||||
function validateGcsBucketComponents(v: string): string | null {
|
||||
if (v.length < 3 || v.length > 222) return 'bucket must be 3-222 characters'
|
||||
const components = v.split('.')
|
||||
for (const c of components) {
|
||||
if (c.length < 1 || c.length > 63) {
|
||||
return 'each dot-separated component must be 1-63 characters'
|
||||
}
|
||||
if (!GCS_BUCKET_COMPONENT_RE.test(c)) {
|
||||
return 'each component must be lowercase, start/end alphanumeric, letters/digits/_/- only'
|
||||
}
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
const gcsConfigSchema = z.object({
|
||||
bucket: z
|
||||
.string()
|
||||
.min(3, 'bucket must be 3-222 characters')
|
||||
.max(222, 'bucket must be 3-222 characters')
|
||||
.superRefine((v, ctx) => {
|
||||
const err = validateGcsBucketComponents(v)
|
||||
if (err) ctx.addIssue({ code: z.ZodIssueCode.custom, message: err })
|
||||
})
|
||||
.refine((v) => !IPV4_LIKE_RE.test(v), {
|
||||
message: 'bucket must not look like an IP address',
|
||||
})
|
||||
.refine((v) => !v.includes('..'), { message: 'bucket must not contain consecutive dots' })
|
||||
.refine((v) => !v.includes('-.') && !v.includes('.-'), {
|
||||
message: 'bucket must not contain a dash adjacent to a dot',
|
||||
})
|
||||
.refine((v) => !GOOGLE_RESERVED_RE.test(v) && !GOOGLE_CONTAINS_RE.test(v), {
|
||||
message: 'bucket name cannot begin with "goog" or contain "google" / close misspellings',
|
||||
}),
|
||||
prefix: z
|
||||
.string()
|
||||
.max(512)
|
||||
.refine((v) => Buffer.byteLength(v, 'utf8') <= 512, {
|
||||
message: 'prefix must be at most 512 bytes (UTF-8)',
|
||||
})
|
||||
.refine((v) => !v.startsWith('.well-known/acme-challenge/'), {
|
||||
message: 'prefix must not start with ".well-known/acme-challenge/" (reserved by GCS)',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const gcsCredentialsSchema = z
|
||||
.object({
|
||||
serviceAccountJson: z.string().min(1, 'serviceAccountJson is required'),
|
||||
})
|
||||
.superRefine(refineServiceAccountJson)
|
||||
|
||||
export type GCSDestinationConfig = z.infer<typeof gcsConfigSchema>
|
||||
export type GCSDestinationCredentials = z.infer<typeof gcsCredentialsSchema>
|
||||
|
||||
function buildJwt(account: ParsedServiceAccount): JWT {
|
||||
return new JWT({ email: account.clientEmail, key: account.privateKey, scopes: [SCOPE] })
|
||||
}
|
||||
|
||||
async function getAccessToken(jwt: JWT): Promise<string> {
|
||||
const { token } = await jwt.getAccessToken()
|
||||
if (!token) throw new Error('Failed to obtain GCS access token')
|
||||
return token
|
||||
}
|
||||
|
||||
interface UploadInput {
|
||||
bucket: string
|
||||
objectName: string
|
||||
body: Buffer
|
||||
contentType: string
|
||||
metadata: Record<string, string>
|
||||
signal: AbortSignal
|
||||
jwt: JWT
|
||||
}
|
||||
|
||||
function isRetryableStatus(status: number): boolean {
|
||||
return (
|
||||
status === 408 ||
|
||||
status === 429 ||
|
||||
status === 500 ||
|
||||
status === 502 ||
|
||||
status === 503 ||
|
||||
status === 504
|
||||
)
|
||||
}
|
||||
|
||||
interface RetryRequestInput {
|
||||
action: string
|
||||
bucket: string
|
||||
url: string
|
||||
method: string
|
||||
/**
|
||||
* Built per attempt so the OAuth access token is refreshed if it expired
|
||||
* between retries (google-auth-library caches and refreshes on demand).
|
||||
*/
|
||||
buildHeaders: () => Promise<Record<string, string>>
|
||||
body?: BodyInit | Buffer
|
||||
signal: AbortSignal
|
||||
/** HTTP statuses to treat as success in addition to 2xx. */
|
||||
successStatuses?: number[]
|
||||
}
|
||||
|
||||
async function fetchWithRetry(input: RetryRequestInput): Promise<void> {
|
||||
let lastError: unknown
|
||||
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
|
||||
if (input.signal.aborted) throw input.signal.reason ?? new Error('Aborted')
|
||||
const perAttempt = AbortSignal.any([input.signal, AbortSignal.timeout(PER_ATTEMPT_TIMEOUT_MS)])
|
||||
let response: Response
|
||||
try {
|
||||
const headers = await input.buildHeaders()
|
||||
response = await fetch(input.url, {
|
||||
method: input.method,
|
||||
body: input.body as BodyInit | undefined,
|
||||
headers,
|
||||
signal: perAttempt,
|
||||
})
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
logger.debug('GCS request failed', {
|
||||
action: input.action,
|
||||
attempt,
|
||||
bucket: input.bucket,
|
||||
error: toError(error).message,
|
||||
})
|
||||
if (attempt < MAX_ATTEMPTS) {
|
||||
await sleepUntilAborted(backoffWithJitter(attempt, null), input.signal)
|
||||
continue
|
||||
}
|
||||
throw error
|
||||
}
|
||||
if (response.ok || input.successStatuses?.includes(response.status)) {
|
||||
/** Drain the success body so undici can return the socket to the keep-alive pool. */
|
||||
await response.text().catch(() => '')
|
||||
return
|
||||
}
|
||||
if (!isRetryableStatus(response.status) || attempt === MAX_ATTEMPTS) {
|
||||
const text = await response.text().catch(() => '')
|
||||
logger.warn('GCS operation failed', {
|
||||
action: input.action,
|
||||
bucket: input.bucket,
|
||||
status: response.status,
|
||||
})
|
||||
throw new Error(
|
||||
`GCS ${input.action} failed (HTTP ${response.status}): ${text || response.statusText}`
|
||||
)
|
||||
}
|
||||
lastError = new Error(`GCS ${input.action} responded with HTTP ${response.status}`)
|
||||
const retryAfterMs = parseRetryAfter(response.headers.get('retry-after'))
|
||||
/** Drain the retryable response body so undici can return the socket to the keep-alive pool. */
|
||||
await response.text().catch(() => '')
|
||||
await sleepUntilAborted(backoffWithJitter(attempt, retryAfterMs), input.signal)
|
||||
}
|
||||
throw lastError instanceof Error
|
||||
? lastError
|
||||
: new Error(`GCS ${input.action} failed after retries`)
|
||||
}
|
||||
|
||||
/** GCS uses HTTP headers (x-goog-meta-*) to carry custom metadata; the spec forbids non-ASCII. */
|
||||
const ASCII_ONLY_RE = /^[\x20-\x7e]*$/
|
||||
|
||||
async function uploadObject(action: string, input: UploadInput): Promise<void> {
|
||||
const objectNameBytes = Buffer.byteLength(input.objectName, 'utf8')
|
||||
if (objectNameBytes < 1 || objectNameBytes > MAX_OBJECT_NAME_BYTES) {
|
||||
throw new Error(
|
||||
`GCS object name is ${objectNameBytes} bytes, must be 1-${MAX_OBJECT_NAME_BYTES} bytes (UTF-8)`
|
||||
)
|
||||
}
|
||||
let metadataBytes = 0
|
||||
for (const [key, value] of Object.entries(input.metadata)) {
|
||||
if (!ASCII_ONLY_RE.test(key) || !ASCII_ONLY_RE.test(value)) {
|
||||
throw new Error(`GCS custom metadata key/value must be ASCII printable: ${key}`)
|
||||
}
|
||||
metadataBytes += Buffer.byteLength(key, 'utf8') + Buffer.byteLength(value, 'utf8')
|
||||
}
|
||||
if (metadataBytes > MAX_CUSTOM_METADATA_BYTES) {
|
||||
throw new Error(
|
||||
`GCS custom metadata is ${metadataBytes} bytes, exceeds the ${MAX_CUSTOM_METADATA_BYTES}-byte per-object limit`
|
||||
)
|
||||
}
|
||||
const url = `${GCS_HOST}/upload/storage/v1/b/${encodeURIComponent(input.bucket)}/o?uploadType=media&name=${encodeURIComponent(input.objectName)}`
|
||||
await fetchWithRetry({
|
||||
action,
|
||||
bucket: input.bucket,
|
||||
url,
|
||||
method: 'POST',
|
||||
buildHeaders: async () => {
|
||||
const token = await getAccessToken(input.jwt)
|
||||
const headers: Record<string, string> = {
|
||||
Authorization: `Bearer ${token}`,
|
||||
'Content-Type': input.contentType,
|
||||
'User-Agent': USER_AGENT,
|
||||
}
|
||||
for (const [key, value] of Object.entries(input.metadata)) {
|
||||
headers[`x-goog-meta-${key}`] = value
|
||||
}
|
||||
return headers
|
||||
},
|
||||
body: input.body,
|
||||
signal: input.signal,
|
||||
})
|
||||
}
|
||||
|
||||
async function deleteObject(input: {
|
||||
bucket: string
|
||||
objectName: string
|
||||
jwt: JWT
|
||||
signal: AbortSignal
|
||||
}): Promise<void> {
|
||||
const url = `${GCS_HOST}/storage/v1/b/${encodeURIComponent(input.bucket)}/o/${encodeURIComponent(input.objectName)}`
|
||||
await fetchWithRetry({
|
||||
action: 'delete-object',
|
||||
bucket: input.bucket,
|
||||
url,
|
||||
method: 'DELETE',
|
||||
buildHeaders: async () => {
|
||||
const token = await getAccessToken(input.jwt)
|
||||
return {
|
||||
Authorization: `Bearer ${token}`,
|
||||
'User-Agent': USER_AGENT,
|
||||
}
|
||||
},
|
||||
signal: input.signal,
|
||||
successStatuses: [404],
|
||||
})
|
||||
}
|
||||
|
||||
export const gcsDestination: DrainDestination<GCSDestinationConfig, GCSDestinationCredentials> = {
|
||||
type: 'gcs',
|
||||
displayName: 'Google Cloud Storage',
|
||||
configSchema: gcsConfigSchema,
|
||||
credentialsSchema: gcsCredentialsSchema,
|
||||
|
||||
async test({ config, credentials, signal }) {
|
||||
const account = parseServiceAccount(credentials.serviceAccountJson)
|
||||
const jwt = buildJwt(account)
|
||||
const probeName = `${normalizePrefix(config.prefix)}.sim-drain-write-probe/${generateShortId(12)}`
|
||||
await uploadObject('test-put', {
|
||||
bucket: config.bucket,
|
||||
objectName: probeName,
|
||||
body: Buffer.alloc(0),
|
||||
contentType: 'application/octet-stream',
|
||||
metadata: {},
|
||||
signal,
|
||||
jwt,
|
||||
})
|
||||
try {
|
||||
await deleteObject({ bucket: config.bucket, objectName: probeName, jwt, signal })
|
||||
} catch (cleanupError) {
|
||||
logger.debug('GCS test write probe cleanup failed (non-fatal)', {
|
||||
bucket: config.bucket,
|
||||
objectName: probeName,
|
||||
error: cleanupError,
|
||||
})
|
||||
}
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
const account = parseServiceAccount(credentials.serviceAccountJson)
|
||||
const jwt = buildJwt(account)
|
||||
return {
|
||||
async deliver({ body, contentType, metadata, signal }) {
|
||||
const objectName = buildObjectKey(config.prefix, metadata)
|
||||
await uploadObject('put-object', {
|
||||
bucket: config.bucket,
|
||||
objectName,
|
||||
body,
|
||||
contentType,
|
||||
metadata: {
|
||||
'sim-drain-id': metadata.drainId,
|
||||
'sim-run-id': metadata.runId,
|
||||
'sim-source': metadata.source,
|
||||
'sim-sequence': metadata.sequence.toString(),
|
||||
'sim-row-count': metadata.rowCount.toString(),
|
||||
},
|
||||
signal,
|
||||
jwt,
|
||||
})
|
||||
logger.debug('GCS chunk delivered', {
|
||||
bucket: config.bucket,
|
||||
objectName,
|
||||
bytes: body.byteLength,
|
||||
})
|
||||
return { locator: `gs://${config.bucket}/${objectName}` }
|
||||
},
|
||||
async close() {},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -1,9 +1,19 @@
|
||||
import { azureBlobDestination } from '@/lib/data-drains/destinations/azure_blob'
|
||||
import { bigqueryDestination } from '@/lib/data-drains/destinations/bigquery'
|
||||
import { datadogDestination } from '@/lib/data-drains/destinations/datadog'
|
||||
import { gcsDestination } from '@/lib/data-drains/destinations/gcs'
|
||||
import { s3Destination } from '@/lib/data-drains/destinations/s3'
|
||||
import { snowflakeDestination } from '@/lib/data-drains/destinations/snowflake'
|
||||
import { webhookDestination } from '@/lib/data-drains/destinations/webhook'
|
||||
import type { DestinationType, DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
export const DESTINATION_REGISTRY = {
|
||||
s3: s3Destination,
|
||||
gcs: gcsDestination,
|
||||
azure_blob: azureBlobDestination,
|
||||
datadog: datadogDestination,
|
||||
bigquery: bigqueryDestination,
|
||||
snowflake: snowflakeDestination,
|
||||
webhook: webhookDestination,
|
||||
} as const satisfies Record<DestinationType, DrainDestination>
|
||||
|
||||
|
||||
@@ -9,15 +9,72 @@ import { generateShortId } from '@sim/utils/id'
|
||||
import { z } from 'zod'
|
||||
import { validateExternalUrl } from '@/lib/core/security/input-validation'
|
||||
import { validateUrlWithDNS } from '@/lib/core/security/input-validation.server'
|
||||
import { buildObjectKey, normalizePrefix } from '@/lib/data-drains/destinations/utils'
|
||||
import type { DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainS3Destination')
|
||||
|
||||
/** https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html */
|
||||
const S3_BUCKET_NAME_RE = /^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$/
|
||||
const S3_IPV4_LIKE_RE = /^(\d{1,3}\.){3}\d{1,3}$/
|
||||
/** Matches standard and 4-segment ISO partition codes (e.g. `us-iso-east-1`). */
|
||||
const AWS_REGION_RE = /^[a-z]{2,}(-[a-z]+)+-\d+$/
|
||||
/** Cap is over key + value bytes only (no `x-amz-meta-` prefix). */
|
||||
const MAX_S3_METADATA_BYTES = 2 * 1024
|
||||
const MAX_S3_KEY_BYTES = 1024
|
||||
|
||||
const s3BucketSchema = z
|
||||
.string()
|
||||
.min(3, 'bucket must be 3-63 characters')
|
||||
.max(63, 'bucket must be 3-63 characters')
|
||||
.refine((v) => S3_BUCKET_NAME_RE.test(v), {
|
||||
message:
|
||||
'bucket must be lowercase, 3-63 chars, start/end alphanumeric, only letters/digits/./-',
|
||||
})
|
||||
.refine((v) => !v.includes('..'), { message: 'bucket must not contain consecutive dots' })
|
||||
.refine((v) => !v.includes('-.') && !v.includes('.-'), {
|
||||
message: 'bucket must not contain a dash adjacent to a dot',
|
||||
})
|
||||
.refine((v) => !S3_IPV4_LIKE_RE.test(v), { message: 'bucket must not look like an IP address' })
|
||||
.refine((v) => !v.startsWith('xn--'), { message: 'bucket must not start with "xn--"' })
|
||||
.refine((v) => !v.startsWith('sthree-'), { message: 'bucket must not start with "sthree-"' })
|
||||
.refine((v) => !v.startsWith('amzn-s3-demo-'), {
|
||||
message: 'bucket must not start with "amzn-s3-demo-" (reserved by AWS)',
|
||||
})
|
||||
.refine((v) => !v.endsWith('-s3alias') && !v.endsWith('--ol-s3') && !v.endsWith('.mrap'), {
|
||||
message: 'bucket must not end with reserved suffix (-s3alias, --ol-s3, .mrap)',
|
||||
})
|
||||
.refine((v) => !v.endsWith('--x-s3'), {
|
||||
message:
|
||||
'bucket must not end with "--x-s3" (reserved for S3 Express One Zone directory buckets)',
|
||||
})
|
||||
.refine((v) => !v.endsWith('--table-s3'), {
|
||||
message: 'bucket must not end with "--table-s3" (reserved for S3 Tables)',
|
||||
})
|
||||
|
||||
const s3RegionSchema = z
|
||||
.string()
|
||||
.min(1, 'region is required')
|
||||
.max(32, 'region is too long')
|
||||
.refine((v) => AWS_REGION_RE.test(v), {
|
||||
message: 'region must look like an AWS region code, e.g. us-east-1',
|
||||
})
|
||||
|
||||
const s3ConfigSchema = z.object({
|
||||
bucket: z.string().min(1, 'bucket is required').max(255),
|
||||
region: z.string().min(1, 'region is required').max(64),
|
||||
/** Optional prefix; trailing slash is added automatically when assembling keys. */
|
||||
prefix: z.string().max(512).optional(),
|
||||
bucket: s3BucketSchema,
|
||||
region: s3RegionSchema,
|
||||
/**
|
||||
* Optional prefix; trailing slash is added automatically when assembling keys.
|
||||
* Bounded by UTF-8 byte length (not code units) so non-ASCII prefixes can't
|
||||
* push assembled keys past S3's 1024-byte object key limit.
|
||||
*/
|
||||
prefix: z
|
||||
.string()
|
||||
.max(512)
|
||||
.refine((v) => Buffer.byteLength(v, 'utf8') <= 512, {
|
||||
message: 'prefix must be at most 512 bytes (UTF-8)',
|
||||
})
|
||||
.optional(),
|
||||
/**
|
||||
* Optional override for non-AWS S3-compatible providers (MinIO, R2, GCS interop, etc.).
|
||||
* SSRF-validated: HTTPS-only, must not resolve syntactically to a private,
|
||||
@@ -27,6 +84,7 @@ const s3ConfigSchema = z.object({
|
||||
endpoint: z
|
||||
.string()
|
||||
.url()
|
||||
.refine((v) => v.startsWith('https://'), { message: 'endpoint must use https://' })
|
||||
.refine((value) => validateExternalUrl(value, 'endpoint').isValid, {
|
||||
message: 'endpoint must be HTTPS and not point at a private, loopback, or metadata address',
|
||||
})
|
||||
@@ -58,35 +116,6 @@ function buildClient(config: S3DestinationConfig, credentials: S3DestinationCred
|
||||
})
|
||||
}
|
||||
|
||||
function normalizePrefix(raw: string | undefined): string {
|
||||
if (!raw) return ''
|
||||
// S3 keys cannot start with `/` (creates an empty-name segment); also
|
||||
// collapse trailing slashes so the joiner produces a single boundary.
|
||||
const trimmed = raw.replace(/^\/+/, '').replace(/\/+$/, '')
|
||||
return trimmed.length === 0 ? '' : `${trimmed}/`
|
||||
}
|
||||
|
||||
function buildKey(
|
||||
config: S3DestinationConfig,
|
||||
metadata: {
|
||||
drainId: string
|
||||
runId: string
|
||||
source: string
|
||||
sequence: number
|
||||
runStartedAt: Date
|
||||
}
|
||||
): string {
|
||||
// Partition by the run's start time so all chunks from one run share a
|
||||
// single date prefix even if delivery crosses a midnight boundary.
|
||||
const partition = metadata.runStartedAt
|
||||
const yyyy = partition.getUTCFullYear().toString().padStart(4, '0')
|
||||
const mm = (partition.getUTCMonth() + 1).toString().padStart(2, '0')
|
||||
const dd = partition.getUTCDate().toString().padStart(2, '0')
|
||||
const seq = metadata.sequence.toString().padStart(5, '0')
|
||||
const prefix = normalizePrefix(config.prefix)
|
||||
return `${prefix}${metadata.source}/${metadata.drainId}/${yyyy}/${mm}/${dd}/${metadata.runId}-${seq}.ndjson`
|
||||
}
|
||||
|
||||
function isS3ServiceException(error: unknown): error is S3ServiceException {
|
||||
return (
|
||||
typeof error === 'object' &&
|
||||
@@ -96,13 +125,7 @@ function isS3ServiceException(error: unknown): error is S3ServiceException {
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the optional custom endpoint and confirms it does not point at a
|
||||
* private, loopback, or cloud-metadata address. The schema-level
|
||||
* `validateExternalUrl` only catches IP literals, so a hostname like
|
||||
* `evil.example.com` resolving to `169.254.169.254` would slip past it; the
|
||||
* AWS SDK then resolves the host itself, bypassing the SSRF guard.
|
||||
*/
|
||||
/** DNS-aware SSRF check: catches hostnames that resolve to internal IPs (the schema check only catches IP literals). */
|
||||
async function assertEndpointIsPublic(endpoint: string | undefined): Promise<void> {
|
||||
if (!endpoint) return
|
||||
const result = await validateUrlWithDNS(endpoint, 'endpoint')
|
||||
@@ -125,8 +148,7 @@ async function withS3ErrorContext<T>(action: string, fn: () => Promise<T>): Prom
|
||||
const status = error.$metadata?.httpStatusCode
|
||||
const requestId = error.$metadata?.requestId
|
||||
logger.warn('S3 operation failed', { action, code, status, requestId })
|
||||
// Preserve the original SDK error as `cause` so callers can still
|
||||
// branch on `code` / `$metadata` while getting an actionable message.
|
||||
/** Preserve SDK error as `cause` so callers can still branch on `code` / `$metadata`. */
|
||||
throw new Error(
|
||||
`S3 ${action} failed (${code}${status ? ` ${status}` : ''}): ${error.message}`,
|
||||
{ cause: error }
|
||||
@@ -145,8 +167,7 @@ export const s3Destination: DrainDestination<S3DestinationConfig, S3DestinationC
|
||||
async test({ config, credentials, signal }) {
|
||||
await assertEndpointIsPublic(config.endpoint)
|
||||
const client = buildClient(config, credentials)
|
||||
// Probe with a real write so read-only creds and write-only IAM policies
|
||||
// surface here instead of at the first scheduled run.
|
||||
/** Real write probe so write-only IAM policies surface here, not at first run. */
|
||||
const probeKey = `${normalizePrefix(config.prefix)}.sim-drain-write-probe/${generateShortId(12)}`
|
||||
try {
|
||||
await withS3ErrorContext('test-put', () =>
|
||||
@@ -161,8 +182,7 @@ export const s3Destination: DrainDestination<S3DestinationConfig, S3DestinationC
|
||||
{ abortSignal: signal }
|
||||
)
|
||||
)
|
||||
// Best-effort cleanup; ignore failures so a missing s3:DeleteObject
|
||||
// doesn't fail the test (write was already proven).
|
||||
/** Best-effort cleanup: write was already proven, so a missing s3:DeleteObject must not fail the test. */
|
||||
try {
|
||||
await client.send(new DeleteObjectCommand({ Bucket: config.bucket, Key: probeKey }), {
|
||||
abortSignal: signal,
|
||||
@@ -181,18 +201,40 @@ export const s3Destination: DrainDestination<S3DestinationConfig, S3DestinationC
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
const client = buildClient(config, credentials)
|
||||
// Cache the DNS-aware endpoint check across all chunks in a run so we
|
||||
// pay the lookup once. The SDK creates its own connections, so we can't
|
||||
// pin the IP — but doing the check before any S3 call still rejects
|
||||
// hostnames that resolve to internal targets at the start of the run.
|
||||
// Lazy-init avoids an unhandled rejection if the source yields no chunks
|
||||
// and `deliver` never runs (e.g., a drain with nothing new to export).
|
||||
/**
|
||||
* Lazy + cached DNS-aware endpoint check. SDK manages its own connections
|
||||
* so we can't pin the IP, but failing the first deliver still rejects
|
||||
* hostnames that resolve to internal targets. Lazy init avoids an
|
||||
* unhandled rejection when the source yields no chunks.
|
||||
*/
|
||||
let endpointCheck: Promise<void> | null = null
|
||||
return {
|
||||
async deliver({ body, contentType, metadata, signal }) {
|
||||
if (endpointCheck === null) endpointCheck = assertEndpointIsPublic(config.endpoint)
|
||||
await endpointCheck
|
||||
const key = buildKey(config, metadata)
|
||||
const key = buildObjectKey(config.prefix, metadata)
|
||||
const keyBytes = Buffer.byteLength(key, 'utf8')
|
||||
if (keyBytes > MAX_S3_KEY_BYTES) {
|
||||
throw new Error(
|
||||
`S3 object key is ${keyBytes} bytes, exceeds the ${MAX_S3_KEY_BYTES}-byte limit`
|
||||
)
|
||||
}
|
||||
const userMetadata: Record<string, string> = {
|
||||
'sim-drain-id': metadata.drainId,
|
||||
'sim-run-id': metadata.runId,
|
||||
'sim-source': metadata.source,
|
||||
'sim-sequence': metadata.sequence.toString(),
|
||||
'sim-row-count': metadata.rowCount.toString(),
|
||||
}
|
||||
let metadataBytes = 0
|
||||
for (const [k, v] of Object.entries(userMetadata)) {
|
||||
metadataBytes += Buffer.byteLength(k, 'utf8') + Buffer.byteLength(v, 'utf8')
|
||||
}
|
||||
if (metadataBytes > MAX_S3_METADATA_BYTES) {
|
||||
throw new Error(
|
||||
`S3 user metadata is ${metadataBytes} bytes, exceeds the ${MAX_S3_METADATA_BYTES}-byte per-object limit`
|
||||
)
|
||||
}
|
||||
await withS3ErrorContext('put-object', () =>
|
||||
client.send(
|
||||
new PutObjectCommand({
|
||||
@@ -201,13 +243,7 @@ export const s3Destination: DrainDestination<S3DestinationConfig, S3DestinationC
|
||||
Body: body,
|
||||
ContentType: contentType,
|
||||
ServerSideEncryption: 'AES256',
|
||||
Metadata: {
|
||||
'sim-drain-id': metadata.drainId,
|
||||
'sim-run-id': metadata.runId,
|
||||
'sim-source': metadata.source,
|
||||
'sim-sequence': metadata.sequence.toString(),
|
||||
'sim-row-count': metadata.rowCount.toString(),
|
||||
},
|
||||
Metadata: userMetadata,
|
||||
}),
|
||||
{ abortSignal: signal }
|
||||
)
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
/**
|
||||
* @vitest-environment node
|
||||
*/
|
||||
import { generateKeyPairSync } from 'node:crypto'
|
||||
import { decodeJwt } from 'jose'
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const fetchMock = vi.fn(async () => new Response(JSON.stringify({}), { status: 200 }))
|
||||
vi.stubGlobal('fetch', fetchMock)
|
||||
|
||||
import { snowflakeDestination } from '@/lib/data-drains/destinations/snowflake'
|
||||
|
||||
const { privateKey } = generateKeyPairSync('rsa', {
|
||||
modulusLength: 2048,
|
||||
privateKeyEncoding: { type: 'pkcs8', format: 'pem' },
|
||||
publicKeyEncoding: { type: 'spki', format: 'pem' },
|
||||
})
|
||||
|
||||
const config = {
|
||||
account: 'orgname-acct',
|
||||
user: 'sim_user',
|
||||
warehouse: 'WH',
|
||||
database: 'DB',
|
||||
schema: 'PUBLIC',
|
||||
table: 'DRAINS',
|
||||
}
|
||||
const credentials = { privateKey }
|
||||
|
||||
const meta = {
|
||||
drainId: 'd',
|
||||
runId: 'r',
|
||||
source: 'workflow_logs' as const,
|
||||
sequence: 0,
|
||||
rowCount: 2,
|
||||
runStartedAt: new Date('2025-06-15T12:00:00Z'),
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks()
|
||||
fetchMock.mockResolvedValue(new Response(JSON.stringify({}), { status: 200 }))
|
||||
})
|
||||
|
||||
describe('snowflakeDestination', () => {
|
||||
it('posts a multi-row INSERT with TEXT bindings and a Bearer JWT', async () => {
|
||||
const session = snowflakeDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ id: 'a' })}\n${JSON.stringify({ id: 'b' })}\n`,
|
||||
'utf8'
|
||||
)
|
||||
await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
|
||||
expect(fetchMock).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = fetchMock.mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toMatch(
|
||||
/^https:\/\/orgname-acct\.snowflakecomputing\.com\/api\/v2\/statements\?requestId=[0-9a-f-]+$/
|
||||
)
|
||||
const headers = init.headers as Record<string, string>
|
||||
expect(headers.Authorization).toMatch(/^Bearer ey/)
|
||||
expect(headers['X-Snowflake-Authorization-Token-Type']).toBe('KEYPAIR_JWT')
|
||||
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.statement).toContain('INSERT INTO "DB"."PUBLIC"."DRAINS"')
|
||||
expect(payload.statement).toContain('VALUES (PARSE_JSON(?)), (PARSE_JSON(?))')
|
||||
expect(payload.statement.match(/PARSE_JSON\(\?\)/g)).toHaveLength(2)
|
||||
expect(payload.bindings['1']).toEqual({ type: 'TEXT', value: JSON.stringify({ id: 'a' }) })
|
||||
expect(payload.bindings['2']).toEqual({ type: 'TEXT', value: JSON.stringify({ id: 'b' }) })
|
||||
expect(payload.warehouse).toBe('WH')
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('uses the configured column when provided', async () => {
|
||||
const session = snowflakeDestination.openSession({
|
||||
config: { ...config, column: 'payload' },
|
||||
credentials,
|
||||
})
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const init = fetchMock.mock.calls[0]?.[1] as RequestInit
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.statement).toContain('("payload")')
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('strips region/cloud suffix from the JWT iss/sub', async () => {
|
||||
const session = snowflakeDestination.openSession({
|
||||
config: { ...config, account: 'orgname-acct.us-east-1.aws' },
|
||||
credentials,
|
||||
})
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const init = fetchMock.mock.calls[0]?.[1] as RequestInit
|
||||
const headers = init.headers as Record<string, string>
|
||||
const token = headers.Authorization.replace(/^Bearer /, '')
|
||||
const claims = decodeJwt(token)
|
||||
expect(claims.sub).toBe('ORGNAME-ACCT.SIM_USER')
|
||||
expect(claims.iss).toMatch(/^ORGNAME-ACCT\.SIM_USER\.SHA256:/)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('polls /statements/{handle} when Snowflake returns 202', async () => {
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
new Response(JSON.stringify({ statementHandle: 'h-1' }), { status: 202 })
|
||||
)
|
||||
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({}), { status: 202 }))
|
||||
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({}), { status: 200 }))
|
||||
const session = snowflakeDestination.openSession({ config, credentials })
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock).toHaveBeenCalledTimes(3)
|
||||
const pollUrl = fetchMock.mock.calls[1]?.[0]
|
||||
expect(pollUrl).toBe('https://orgname-acct.snowflakecomputing.com/api/v2/statements/h-1')
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('parses CRLF NDJSON bodies correctly', async () => {
|
||||
const session = snowflakeDestination.openSession({ config, credentials })
|
||||
const body = Buffer.from(
|
||||
`${JSON.stringify({ id: 'a' })}\r\n${JSON.stringify({ id: 'b' })}\r\n`,
|
||||
'utf8'
|
||||
)
|
||||
await session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const init = fetchMock.mock.calls[0]?.[1] as RequestInit
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.bindings['1']).toEqual({ type: 'TEXT', value: JSON.stringify({ id: 'a' }) })
|
||||
expect(payload.bindings['2']).toEqual({ type: 'TEXT', value: JSON.stringify({ id: 'b' }) })
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('retries the POST on 5xx and succeeds on the next attempt', async () => {
|
||||
fetchMock.mockResolvedValueOnce(new Response('boom', { status: 503 }))
|
||||
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({}), { status: 200 }))
|
||||
const session = snowflakeDestination.openSession({ config, credentials })
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('honors Retry-After (delta seconds) on 429 before retrying', async () => {
|
||||
fetchMock.mockResolvedValueOnce(
|
||||
new Response('slow down', { status: 429, headers: { 'Retry-After': '1' } })
|
||||
)
|
||||
fetchMock.mockResolvedValueOnce(new Response(JSON.stringify({}), { status: 200 }))
|
||||
const session = snowflakeDestination.openSession({ config, credentials })
|
||||
const start = Date.now()
|
||||
await session.deliver({
|
||||
body: Buffer.from(`${JSON.stringify({ x: 1 })}\n`),
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const elapsed = Date.now() - start
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2)
|
||||
expect(elapsed).toBeGreaterThanOrEqual(900)
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('throws a clear error when a binding exceeds the 16 MiB VARIANT limit', async () => {
|
||||
const session = snowflakeDestination.openSession({ config, credentials })
|
||||
const huge = `"${'a'.repeat(16 * 1024 * 1024 + 1)}"`
|
||||
const body = Buffer.from(`${huge}\n`, 'utf8')
|
||||
await expect(
|
||||
session.deliver({
|
||||
body,
|
||||
contentType: 'application/x-ndjson',
|
||||
metadata: meta,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
).rejects.toThrow(/16 MB/)
|
||||
expect(fetchMock).not.toHaveBeenCalled()
|
||||
await session.close()
|
||||
})
|
||||
|
||||
it('test() runs SELECT 1', async () => {
|
||||
await snowflakeDestination.test!({
|
||||
config,
|
||||
credentials,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
const init = fetchMock.mock.calls[0]?.[1] as RequestInit
|
||||
const payload = JSON.parse(init.body as string)
|
||||
expect(payload.statement).toBe('SELECT 1')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,518 @@
|
||||
import { createHash, createPublicKey } from 'node:crypto'
|
||||
import { createLogger } from '@sim/logger'
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { generateId } from '@sim/utils/id'
|
||||
import { importPKCS8, SignJWT } from 'jose'
|
||||
import { z } from 'zod'
|
||||
import {
|
||||
backoffWithJitter,
|
||||
parseRetryAfter,
|
||||
sleepUntilAborted,
|
||||
} from '@/lib/data-drains/destinations/utils'
|
||||
import type { DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainSnowflakeDestination')
|
||||
|
||||
/**
|
||||
* Snowflake account identifier formats (https://docs.snowflake.com/en/user-guide/admin-account-identifier):
|
||||
* - Org-account: `<orgname>-<acctname>` — alphanumerics/underscore, hyphen-separated, no dots.
|
||||
* - Legacy account locator: `<locator>` or `<locator>.<region>[.<cloud>]` — dots allowed.
|
||||
*/
|
||||
const ACCOUNT_ORG_RE = /^[A-Za-z0-9][A-Za-z0-9_]*(?:-[A-Za-z0-9_]+)+$/
|
||||
/**
|
||||
* First segment allows hyphens so org-account identifiers (`<orgname>-<acctname>`)
|
||||
* carrying a legacy region/cloud suffix (e.g. `myorg-acct.us-east-1.aws`) match.
|
||||
* `normalizeAccountForJwt` strips the dotted suffix for JWT `iss`/`sub`.
|
||||
*/
|
||||
const ACCOUNT_LOCATOR_RE = /^[A-Za-z0-9][A-Za-z0-9_-]*(?:\.[A-Za-z0-9][A-Za-z0-9_-]*){0,2}$/
|
||||
function isValidAccount(v: string): boolean {
|
||||
return ACCOUNT_ORG_RE.test(v) || ACCOUNT_LOCATOR_RE.test(v)
|
||||
}
|
||||
const IDENTIFIER_RE = /^[A-Za-z_][A-Za-z0-9_$]{0,254}$/
|
||||
/** JWT lifetime; Snowflake caps server-side enforcement at 60 minutes regardless of `exp`. */
|
||||
const JWT_LIFETIME_SECONDS = 55 * 60
|
||||
/** Safety margin (in seconds) subtracted from the JWT exp when caching. */
|
||||
const JWT_CACHE_SAFETY_MARGIN_SECONDS = 300
|
||||
const PER_ATTEMPT_TIMEOUT_MS = 60_000
|
||||
const POLL_INITIAL_INTERVAL_MS = 500
|
||||
const POLL_MAX_INTERVAL_MS = 5_000
|
||||
const POLL_DEADLINE_MS = 10 * 60_000
|
||||
/**
|
||||
* Cap on consecutive failed poll attempts (network errors or retryable HTTP
|
||||
* statuses). Independent of the 10-minute wall-clock deadline so that
|
||||
* persistent failures surface in seconds, not minutes — matches the
|
||||
* MAX_ATTEMPTS shape used by `executeStatement`. Reset to 0 on a successful
|
||||
* 202 (still-executing) response.
|
||||
*/
|
||||
const POLL_MAX_CONSECUTIVE_RETRIES = 8
|
||||
/** Maximum number of attempts (including the initial attempt) for retryable POST failures. */
|
||||
const EXECUTE_MAX_ATTEMPTS = 3
|
||||
const EXECUTE_RETRY_BASE_DELAY_MS = 500
|
||||
const EXECUTE_RETRY_MAX_DELAY_MS = 5_000
|
||||
/** Conservative pre-2025_03 BCR VARIANT max size (16 MiB) so the same value works on every account. */
|
||||
const VARIANT_MAX_BYTES = 16 * 1024 * 1024
|
||||
/** Server-side statement execution timeout (seconds) sent in the SQL API request body. */
|
||||
const SQL_API_TIMEOUT_SECONDS = 600
|
||||
|
||||
/**
|
||||
* Snowflake JWT `iss`/`sub` require the bare account identifier without any
|
||||
* region/cloud suffix. For account-locator format `xy12345.us-east-1.aws`,
|
||||
* only `XY12345` is valid; for org-account format `myorg-acct.us-east-1`,
|
||||
* only `MYORG-ACCT` is valid. Strip everything after the first dot.
|
||||
*/
|
||||
function normalizeAccountForJwt(account: string): string {
|
||||
const dot = account.indexOf('.')
|
||||
return (dot === -1 ? account : account.slice(0, dot)).toUpperCase()
|
||||
}
|
||||
|
||||
const snowflakeConfigSchema = z.object({
|
||||
/**
|
||||
* Snowflake account identifier. Accepted formats:
|
||||
* - Org-account (preferred): `<orgname>-<acctname>` (no dots), e.g. `myorg-acct`
|
||||
* - Account locator: `<locator>` (no dots), e.g. `xy12345`
|
||||
* - Legacy regional locator: `<locator>.<region>.<cloud>`, e.g. `xy12345.us-east-1.aws`
|
||||
*
|
||||
* Do not include the `.snowflakecomputing.com` suffix. Modern org-account
|
||||
* identifiers must not contain dots; only legacy locator URLs use dots.
|
||||
*/
|
||||
account: z.string().min(3, 'account is required').refine(isValidAccount, {
|
||||
message: 'account must be the Snowflake account identifier (e.g. orgname-accountname)',
|
||||
}),
|
||||
user: z
|
||||
.string()
|
||||
.min(1, 'user is required')
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'user must be a valid Snowflake identifier',
|
||||
}),
|
||||
warehouse: z
|
||||
.string()
|
||||
.min(1)
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'warehouse must be a valid Snowflake identifier',
|
||||
}),
|
||||
database: z
|
||||
.string()
|
||||
.min(1)
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'database must be a valid Snowflake identifier',
|
||||
}),
|
||||
schema: z
|
||||
.string()
|
||||
.min(1)
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'schema must be a valid Snowflake identifier',
|
||||
}),
|
||||
table: z
|
||||
.string()
|
||||
.min(1)
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'table must be a valid Snowflake identifier',
|
||||
}),
|
||||
/** Target VARIANT column. Defaults to `DATA` (uppercase, matching Snowflake's unquoted identifier folding). */
|
||||
column: z
|
||||
.string()
|
||||
.min(1)
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'column must be a valid Snowflake identifier',
|
||||
})
|
||||
.optional(),
|
||||
/** Optional Snowflake role to assume for the insert. */
|
||||
role: z
|
||||
.string()
|
||||
.min(1)
|
||||
.refine((v) => IDENTIFIER_RE.test(v), {
|
||||
message: 'role must be a valid Snowflake identifier',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const snowflakeCredentialsSchema = z.object({
|
||||
/** PKCS8-encoded RSA private key (PEM). The matching public key must be registered on the user. */
|
||||
privateKey: z.string().min(1, 'privateKey is required'),
|
||||
})
|
||||
|
||||
export type SnowflakeDestinationConfig = z.infer<typeof snowflakeConfigSchema>
|
||||
export type SnowflakeDestinationCredentials = z.infer<typeof snowflakeCredentialsSchema>
|
||||
|
||||
/**
|
||||
* Computes the SHA256:<base64> fingerprint of the public key derived from the
|
||||
* given private key. Snowflake encodes this in the JWT issuer claim so the
|
||||
* server can match the signature against the registered public key.
|
||||
* Reference: https://docs.snowflake.com/en/developer-guide/sql-api/authenticating
|
||||
*/
|
||||
function computePublicKeyFingerprint(privateKeyPem: string): string {
|
||||
const publicKey = createPublicKey({ key: privateKeyPem, format: 'pem' })
|
||||
const spkiDer = publicKey.export({ type: 'spki', format: 'der' })
|
||||
return `SHA256:${createHash('sha256').update(spkiDer).digest('base64')}`
|
||||
}
|
||||
|
||||
interface JwtCacheEntry {
|
||||
token: string
|
||||
expiresAt: number
|
||||
}
|
||||
|
||||
async function buildJwt(
|
||||
account: string,
|
||||
user: string,
|
||||
privateKeyPem: string
|
||||
): Promise<JwtCacheEntry> {
|
||||
const fingerprint = computePublicKeyFingerprint(privateKeyPem)
|
||||
const accountForJwt = normalizeAccountForJwt(account)
|
||||
const userUpper = user.toUpperCase()
|
||||
const issuer = `${accountForJwt}.${userUpper}.${fingerprint}`
|
||||
const subject = `${accountForJwt}.${userUpper}`
|
||||
const now = Math.floor(Date.now() / 1000)
|
||||
const exp = now + JWT_LIFETIME_SECONDS
|
||||
let privateKey: Awaited<ReturnType<typeof importPKCS8>>
|
||||
try {
|
||||
privateKey = await importPKCS8(privateKeyPem, 'RS256')
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`privateKey must be an unencrypted PKCS#8 PEM (-----BEGIN PRIVATE KEY-----). ` +
|
||||
`Convert PKCS#1 with: openssl pkcs8 -topk8 -nocrypt -in rsa.pem -out pkcs8.pem. ` +
|
||||
`Decrypt encrypted PEMs first. Underlying error: ${toError(error).message}`
|
||||
)
|
||||
}
|
||||
const token = await new SignJWT({})
|
||||
.setProtectedHeader({ alg: 'RS256', typ: 'JWT' })
|
||||
.setIssuer(issuer)
|
||||
.setSubject(subject)
|
||||
.setIssuedAt(now)
|
||||
.setExpirationTime(exp)
|
||||
.sign(privateKey)
|
||||
return { token, expiresAt: exp - JWT_CACHE_SAFETY_MARGIN_SECONDS }
|
||||
}
|
||||
|
||||
/**
|
||||
* Quotes a Snowflake identifier so that whatever case the user typed is
|
||||
* preserved exactly. Without quoting, Snowflake folds unquoted identifiers
|
||||
* to uppercase, which silently breaks any table whose canonical name was
|
||||
* created with quoted mixed-case. Embedded `"` is escaped as `""`.
|
||||
*/
|
||||
function quoteIdentifier(name: string): string {
|
||||
return `"${name.replace(/"/g, '""')}"`
|
||||
}
|
||||
|
||||
function buildStatement(config: SnowflakeDestinationConfig, rowCount: number): string {
|
||||
const column = quoteIdentifier(config.column ?? 'DATA')
|
||||
const target = `${quoteIdentifier(config.database)}.${quoteIdentifier(config.schema)}.${quoteIdentifier(config.table)}`
|
||||
const placeholders = Array.from({ length: rowCount }, () => '(PARSE_JSON(?))').join(', ')
|
||||
return `INSERT INTO ${target} (${column}) VALUES ${placeholders}`
|
||||
}
|
||||
|
||||
function isRetryableStatus(status: number): boolean {
|
||||
return status === 408 || status === 429 || (status >= 500 && status <= 599)
|
||||
}
|
||||
|
||||
interface ExecuteInput {
|
||||
config: SnowflakeDestinationConfig
|
||||
getJwt: () => Promise<string>
|
||||
statement: string
|
||||
bindings: string[]
|
||||
signal: AbortSignal
|
||||
}
|
||||
|
||||
async function executeStatement(input: ExecuteInput): Promise<void> {
|
||||
for (const value of input.bindings) {
|
||||
const bytes = Buffer.byteLength(value, 'utf8')
|
||||
if (bytes > VARIANT_MAX_BYTES) {
|
||||
throw new Error(
|
||||
`Snowflake VARIANT value exceeds 16 MB limit (got ${bytes} bytes); split the row before delivery`
|
||||
)
|
||||
}
|
||||
}
|
||||
const baseUrl = `https://${input.config.account}.snowflakecomputing.com/api/v2/statements`
|
||||
const bindings: Record<string, { type: 'TEXT'; value: string }> = {}
|
||||
input.bindings.forEach((value, index) => {
|
||||
bindings[(index + 1).toString()] = { type: 'TEXT', value }
|
||||
})
|
||||
const body = {
|
||||
statement: input.statement,
|
||||
timeout: SQL_API_TIMEOUT_SECONDS,
|
||||
warehouse: input.config.warehouse,
|
||||
role: input.config.role,
|
||||
bindings,
|
||||
}
|
||||
const serializedBody = JSON.stringify(body)
|
||||
/** Stable per-request UUID enables idempotent retries via `retry=true` on subsequent attempts. */
|
||||
const requestId = generateId()
|
||||
|
||||
let lastError: unknown
|
||||
for (let attempt = 1; attempt <= EXECUTE_MAX_ATTEMPTS; attempt++) {
|
||||
if (input.signal.aborted) throw input.signal.reason ?? new Error('Aborted')
|
||||
/** Acquire JWT before starting the per-attempt timer so token signing doesn't eat the network budget (mirrors pollStatement). */
|
||||
const jwt = await input.getJwt()
|
||||
const perAttempt = AbortSignal.any([input.signal, AbortSignal.timeout(PER_ATTEMPT_TIMEOUT_MS)])
|
||||
const params = new URLSearchParams({ requestId })
|
||||
if (attempt > 1) params.set('retry', 'true')
|
||||
const url = `${baseUrl}?${params.toString()}`
|
||||
let response: Response
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${jwt}`,
|
||||
'Content-Type': 'application/json',
|
||||
Accept: 'application/json',
|
||||
'X-Snowflake-Authorization-Token-Type': 'KEYPAIR_JWT',
|
||||
'User-Agent': 'sim-data-drain/1.0',
|
||||
},
|
||||
body: serializedBody,
|
||||
signal: perAttempt,
|
||||
})
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
logger.warn('Snowflake request failed', {
|
||||
attempt,
|
||||
error: toError(error).message,
|
||||
})
|
||||
if (input.signal.aborted || attempt === EXECUTE_MAX_ATTEMPTS) throw error
|
||||
await sleepUntilAborted(
|
||||
backoffWithJitter(attempt, null, {
|
||||
baseMs: EXECUTE_RETRY_BASE_DELAY_MS,
|
||||
maxMs: EXECUTE_RETRY_MAX_DELAY_MS,
|
||||
}),
|
||||
input.signal
|
||||
)
|
||||
continue
|
||||
}
|
||||
if (response.status === 202) {
|
||||
const json = (await response.json().catch(() => ({}))) as { statementHandle?: string }
|
||||
if (!json.statementHandle) {
|
||||
throw new Error('Snowflake returned 202 without a statementHandle')
|
||||
}
|
||||
await pollStatement({
|
||||
account: input.config.account,
|
||||
getJwt: input.getJwt,
|
||||
handle: json.statementHandle,
|
||||
signal: input.signal,
|
||||
})
|
||||
return
|
||||
}
|
||||
if (response.ok) {
|
||||
/**
|
||||
* Synchronous completions return 200 — same statement-level error envelope as
|
||||
* the polled 200 path, so check `sqlState` here too instead of silently passing
|
||||
* failures. Consuming the body also lets undici reuse the socket.
|
||||
*/
|
||||
const completion = (await response.json().catch(() => ({}))) as {
|
||||
code?: string
|
||||
sqlState?: string
|
||||
message?: string
|
||||
}
|
||||
if (completion.sqlState && completion.sqlState !== '00000') {
|
||||
throw new Error(
|
||||
`Snowflake statement failed (sqlState ${completion.sqlState}${completion.code ? `, code ${completion.code}` : ''}): ${completion.message ?? ''}`
|
||||
)
|
||||
}
|
||||
return
|
||||
}
|
||||
const text = await response.text().catch(() => '')
|
||||
const error = new Error(`Snowflake responded with HTTP ${response.status}: ${text}`)
|
||||
if (!isRetryableStatus(response.status) || attempt === EXECUTE_MAX_ATTEMPTS) throw error
|
||||
lastError = error
|
||||
const retryAfterMs = parseRetryAfter(response.headers.get('Retry-After'))
|
||||
const delay = backoffWithJitter(attempt, retryAfterMs, {
|
||||
baseMs: EXECUTE_RETRY_BASE_DELAY_MS,
|
||||
maxMs: EXECUTE_RETRY_MAX_DELAY_MS,
|
||||
})
|
||||
logger.warn('Snowflake request retrying after retryable status', {
|
||||
attempt,
|
||||
status: response.status,
|
||||
delayMs: delay,
|
||||
})
|
||||
await sleepUntilAborted(delay, input.signal)
|
||||
}
|
||||
throw lastError ?? new Error('Snowflake request failed after retries')
|
||||
}
|
||||
|
||||
interface PollInput {
|
||||
account: string
|
||||
/** Thunk so long polls (past 55min) refresh the JWT instead of dying with 401. */
|
||||
getJwt: () => Promise<string>
|
||||
handle: string
|
||||
signal: AbortSignal
|
||||
}
|
||||
|
||||
/** Snowflake returns 202 while still executing and 200 on completion (async statement-handle semantics). */
|
||||
async function pollStatement(input: PollInput): Promise<void> {
|
||||
const url = `https://${input.account}.snowflakecomputing.com/api/v2/statements/${encodeURIComponent(input.handle)}`
|
||||
const deadline = Date.now() + POLL_DEADLINE_MS
|
||||
let interval = POLL_INITIAL_INTERVAL_MS
|
||||
let skipIntervalSleep = true
|
||||
let retryAttempt = 0
|
||||
while (Date.now() < deadline) {
|
||||
if (input.signal.aborted) throw input.signal.reason ?? new Error('Aborted')
|
||||
if (!skipIntervalSleep) {
|
||||
await sleepUntilAborted(interval, input.signal)
|
||||
}
|
||||
skipIntervalSleep = false
|
||||
const jwt = await input.getJwt()
|
||||
const perAttempt = AbortSignal.any([input.signal, AbortSignal.timeout(PER_ATTEMPT_TIMEOUT_MS)])
|
||||
let response: Response
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${jwt}`,
|
||||
Accept: 'application/json',
|
||||
'X-Snowflake-Authorization-Token-Type': 'KEYPAIR_JWT',
|
||||
},
|
||||
signal: perAttempt,
|
||||
})
|
||||
} catch (error) {
|
||||
if (input.signal.aborted) throw error
|
||||
retryAttempt++
|
||||
if (retryAttempt > POLL_MAX_CONSECUTIVE_RETRIES) throw error
|
||||
const delay = backoffWithJitter(retryAttempt, null, {
|
||||
baseMs: EXECUTE_RETRY_BASE_DELAY_MS,
|
||||
maxMs: EXECUTE_RETRY_MAX_DELAY_MS,
|
||||
})
|
||||
logger.warn('Snowflake poll request failed, retrying', {
|
||||
attempt: retryAttempt,
|
||||
delayMs: delay,
|
||||
error: toError(error).message,
|
||||
})
|
||||
await sleepUntilAborted(delay, input.signal)
|
||||
skipIntervalSleep = true
|
||||
continue
|
||||
}
|
||||
if (response.status === 202) {
|
||||
/** Drain the body so undici can return the socket to the keep-alive pool between polls. */
|
||||
await response.text().catch(() => '')
|
||||
retryAttempt = 0
|
||||
interval = Math.min(interval * 2, POLL_MAX_INTERVAL_MS)
|
||||
continue
|
||||
}
|
||||
if (isRetryableStatus(response.status)) {
|
||||
retryAttempt++
|
||||
if (retryAttempt > POLL_MAX_CONSECUTIVE_RETRIES) {
|
||||
/** Drain the body so undici can return the socket to the keep-alive pool. */
|
||||
const text = await response.text().catch(() => '')
|
||||
throw new Error(
|
||||
`Snowflake poll failed after ${POLL_MAX_CONSECUTIVE_RETRIES} consecutive retries (HTTP ${response.status}): ${text}`
|
||||
)
|
||||
}
|
||||
const retryAfterMs = parseRetryAfter(response.headers.get('Retry-After'))
|
||||
const delay = backoffWithJitter(retryAttempt, retryAfterMs, {
|
||||
baseMs: EXECUTE_RETRY_BASE_DELAY_MS,
|
||||
maxMs: EXECUTE_RETRY_MAX_DELAY_MS,
|
||||
})
|
||||
logger.warn('Snowflake poll retrying after retryable status', {
|
||||
attempt: retryAttempt,
|
||||
status: response.status,
|
||||
delayMs: delay,
|
||||
})
|
||||
/** Drain the body so undici can return the socket to the keep-alive pool between retries. */
|
||||
await response.text().catch(() => '')
|
||||
await sleepUntilAborted(delay, input.signal)
|
||||
skipIntervalSleep = true
|
||||
continue
|
||||
}
|
||||
if (!response.ok) {
|
||||
const text = await response.text().catch(() => '')
|
||||
throw new Error(`Snowflake poll failed (HTTP ${response.status}): ${text}`)
|
||||
}
|
||||
/**
|
||||
* Snowflake SQL API can return 200 with a statement-level error envelope
|
||||
* (`code` / `sqlState` / `message`). Successful completions return
|
||||
* `code === "090001"` ("statement executed successfully") or omit `code`,
|
||||
* while statement errors come back with codes like `"002032"` and
|
||||
* a populated `sqlState`. Treat anything with a `sqlState` as a failure.
|
||||
*/
|
||||
const completion = (await response.json().catch(() => ({}))) as {
|
||||
code?: string
|
||||
sqlState?: string
|
||||
message?: string
|
||||
}
|
||||
if (completion.sqlState && completion.sqlState !== '00000') {
|
||||
throw new Error(
|
||||
`Snowflake statement failed (sqlState ${completion.sqlState}${completion.code ? `, code ${completion.code}` : ''}): ${completion.message ?? ''}`
|
||||
)
|
||||
}
|
||||
return
|
||||
}
|
||||
throw new Error('Snowflake statement did not complete within the polling deadline')
|
||||
}
|
||||
|
||||
export const snowflakeDestination: DrainDestination<
|
||||
SnowflakeDestinationConfig,
|
||||
SnowflakeDestinationCredentials
|
||||
> = {
|
||||
type: 'snowflake',
|
||||
displayName: 'Snowflake',
|
||||
configSchema: snowflakeConfigSchema,
|
||||
credentialsSchema: snowflakeCredentialsSchema,
|
||||
|
||||
async test({ config, credentials, signal }) {
|
||||
let cached: JwtCacheEntry | null = null
|
||||
async function getJwt(): Promise<string> {
|
||||
const now = Math.floor(Date.now() / 1000)
|
||||
if (cached && cached.expiresAt > now) return cached.token
|
||||
cached = await buildJwt(config.account, config.user, credentials.privateKey)
|
||||
return cached.token
|
||||
}
|
||||
await executeStatement({
|
||||
config,
|
||||
getJwt,
|
||||
statement: 'SELECT 1',
|
||||
bindings: [],
|
||||
signal,
|
||||
})
|
||||
},
|
||||
|
||||
openSession({ config, credentials }) {
|
||||
let cached: JwtCacheEntry | null = null
|
||||
async function getJwt(): Promise<string> {
|
||||
const now = Math.floor(Date.now() / 1000)
|
||||
if (cached && cached.expiresAt > now) return cached.token
|
||||
cached = await buildJwt(config.account, config.user, credentials.privateKey)
|
||||
return cached.token
|
||||
}
|
||||
return {
|
||||
async deliver({ body, metadata, signal }) {
|
||||
/**
|
||||
* Bind the original line bytes — not `JSON.stringify(JSON.parse(line))` —
|
||||
* so JSON numbers outside the JS safe-integer range (e.g. Snowflake
|
||||
* NUMBER columns past 2^53-1) survive into VARIANT intact. We still
|
||||
* parse each line so a malformed payload fails fast at the runner.
|
||||
*/
|
||||
const text = body.toString('utf8')
|
||||
const rows: string[] = []
|
||||
const lines = text.split(/\r?\n/)
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i]
|
||||
if (line.length === 0) continue
|
||||
try {
|
||||
JSON.parse(line)
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Snowflake NDJSON parse failed at line ${i + 1}: ${toError(error).message}`
|
||||
)
|
||||
}
|
||||
rows.push(line)
|
||||
}
|
||||
if (rows.length === 0) {
|
||||
return {
|
||||
locator: `snowflake://${config.account}/${config.database}.${config.schema}.${config.table}#${metadata.runId}-${metadata.sequence}`,
|
||||
}
|
||||
}
|
||||
await executeStatement({
|
||||
config,
|
||||
getJwt,
|
||||
statement: buildStatement(config, rows.length),
|
||||
bindings: rows,
|
||||
signal,
|
||||
})
|
||||
logger.debug('Snowflake chunk delivered', {
|
||||
account: config.account,
|
||||
table: `${config.database}.${config.schema}.${config.table}`,
|
||||
rows: rows.length,
|
||||
})
|
||||
return {
|
||||
locator: `snowflake://${config.account}/${config.database}.${config.schema}.${config.table}#${metadata.runId}-${metadata.sequence}`,
|
||||
}
|
||||
},
|
||||
async close() {},
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
import { toError } from '@sim/utils/errors'
|
||||
import { z } from 'zod'
|
||||
|
||||
/**
|
||||
* Sleep for `ms` milliseconds, resolving early if `signal` aborts. Used by
|
||||
* destination retry/poll loops so cancelled drain runs do not hang waiting on
|
||||
* a `setTimeout` that ignores the abort signal.
|
||||
*/
|
||||
export function sleepUntilAborted(ms: number, signal: AbortSignal): Promise<void> {
|
||||
if (signal.aborted) return Promise.resolve()
|
||||
return new Promise((resolve) => {
|
||||
const onAbort = () => {
|
||||
clearTimeout(timeoutId)
|
||||
resolve()
|
||||
}
|
||||
const timeoutId = setTimeout(() => {
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
resolve()
|
||||
}, ms)
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Strips leading and trailing slashes from a path prefix and re-appends a
|
||||
* single trailing slash. Object stores reject keys that begin with `/`
|
||||
* (it produces an empty-name segment), and we want exactly one boundary
|
||||
* between prefix and the rest of the key.
|
||||
*/
|
||||
/**
|
||||
* Default retry pacing shared by destination backoff loops: 500 ms floor,
|
||||
* 30 s ceiling, ±20% jitter.
|
||||
*/
|
||||
const DEFAULT_BACKOFF_BASE_MS = 500
|
||||
const DEFAULT_BACKOFF_MAX_MS = 30_000
|
||||
|
||||
export interface BackoffOptions {
|
||||
baseMs?: number
|
||||
maxMs?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Computes the next delay for a retry loop. When the server returned a
|
||||
* `Retry-After` (`retryAfterMs` is non-null), the value is clamped to
|
||||
* `[baseMs, maxMs]` so a malformed `Retry-After: 0` cannot pin the loop into a
|
||||
* tight retry. Otherwise returns exponential backoff with ±20% jitter to avoid
|
||||
* thundering-herd alignment across concurrent drains. Attempt is 1-indexed.
|
||||
*/
|
||||
export function backoffWithJitter(
|
||||
attempt: number,
|
||||
retryAfterMs: number | null,
|
||||
options: BackoffOptions = {}
|
||||
): number {
|
||||
const baseMs = options.baseMs ?? DEFAULT_BACKOFF_BASE_MS
|
||||
const maxMs = options.maxMs ?? DEFAULT_BACKOFF_MAX_MS
|
||||
if (retryAfterMs !== null) {
|
||||
return Math.min(Math.max(retryAfterMs, baseMs), maxMs)
|
||||
}
|
||||
const exponential = Math.min(baseMs * 2 ** (attempt - 1), maxMs)
|
||||
return exponential * (0.8 + Math.random() * 0.4)
|
||||
}
|
||||
|
||||
/**
|
||||
* Maximum HTTP Retry-After value we honor. A server requesting >30s is treated
|
||||
* as a 30s delay so a misconfigured upstream can't stall a drain run.
|
||||
*/
|
||||
const RETRY_AFTER_MAX_MS = 30_000
|
||||
|
||||
/**
|
||||
* Parses an HTTP `Retry-After` header (either delta-seconds or HTTP-date) into
|
||||
* a millisecond delay, capped at 30s. Returns `null` when the header is
|
||||
* absent or unparseable so callers can fall back to their own backoff.
|
||||
*/
|
||||
export function parseRetryAfter(header: string | null): number | null {
|
||||
if (!header) return null
|
||||
const trimmed = header.trim()
|
||||
if (trimmed.length === 0) return null
|
||||
const seconds = Number(trimmed)
|
||||
if (Number.isFinite(seconds) && seconds >= 0) {
|
||||
return Math.min(Math.floor(seconds * 1000), RETRY_AFTER_MAX_MS)
|
||||
}
|
||||
const dateMs = Date.parse(trimmed)
|
||||
if (!Number.isNaN(dateMs)) {
|
||||
const delta = dateMs - Date.now()
|
||||
if (delta <= 0) return 0
|
||||
return Math.min(delta, RETRY_AFTER_MAX_MS)
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
export function normalizePrefix(raw: string | undefined): string {
|
||||
if (!raw) return ''
|
||||
const trimmed = raw.replace(/^\/+/, '').replace(/\/+$/, '')
|
||||
return trimmed.length === 0 ? '' : `${trimmed}/`
|
||||
}
|
||||
|
||||
export interface ObjectKeyMetadata {
|
||||
drainId: string
|
||||
runId: string
|
||||
source: string
|
||||
sequence: number
|
||||
runStartedAt: Date
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a date-partitioned NDJSON object key for blob-store destinations.
|
||||
* Layout: `<prefix><source>/<drainId>/<YYYY>/<MM>/<DD>/<runId>-<seq>.ndjson`.
|
||||
* Partition uses the run's start time so all chunks from a run share one
|
||||
* date prefix even if delivery crosses a UTC midnight boundary.
|
||||
*/
|
||||
export function buildObjectKey(prefix: string | undefined, metadata: ObjectKeyMetadata): string {
|
||||
const partition = metadata.runStartedAt
|
||||
const yyyy = partition.getUTCFullYear().toString().padStart(4, '0')
|
||||
const mm = (partition.getUTCMonth() + 1).toString().padStart(2, '0')
|
||||
const dd = partition.getUTCDate().toString().padStart(2, '0')
|
||||
const seq = metadata.sequence.toString().padStart(5, '0')
|
||||
return `${normalizePrefix(prefix)}${metadata.source}/${metadata.drainId}/${yyyy}/${mm}/${dd}/${metadata.runId}-${seq}.ndjson`
|
||||
}
|
||||
|
||||
export interface ParsedServiceAccount {
|
||||
clientEmail: string
|
||||
privateKey: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a Google service-account JSON key, returning the only two fields
|
||||
* that destinations need (client email + private key). Shared by GCS and
|
||||
* BigQuery so a fix in one place applies to both.
|
||||
*/
|
||||
export function parseServiceAccount(json: string): ParsedServiceAccount {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(json)
|
||||
} catch (error) {
|
||||
throw new Error(`serviceAccountJson is not valid JSON: ${toError(error).message}`)
|
||||
}
|
||||
if (typeof parsed !== 'object' || parsed === null) {
|
||||
throw new Error('serviceAccountJson must be a JSON object')
|
||||
}
|
||||
const obj = parsed as Record<string, unknown>
|
||||
const clientEmail = obj.client_email
|
||||
const privateKey = obj.private_key
|
||||
if (typeof clientEmail !== 'string' || clientEmail.length === 0) {
|
||||
throw new Error('serviceAccountJson is missing client_email')
|
||||
}
|
||||
if (typeof privateKey !== 'string' || privateKey.length === 0) {
|
||||
throw new Error('serviceAccountJson is missing private_key')
|
||||
}
|
||||
return { clientEmail, privateKey }
|
||||
}
|
||||
|
||||
/**
|
||||
* Zod `superRefine` helper that validates a service-account JSON key string
|
||||
* is parseable and carries `client_email` + `private_key`. Used by both
|
||||
* `gcsCredentialsSchema` and `bigqueryCredentialsSchema`.
|
||||
*/
|
||||
export interface ParseNdjsonObjectsOptions {
|
||||
/** When true, throw if a parsed value is not a plain object. */
|
||||
requireObject?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses an NDJSON buffer into per-row JSON values. Error messages use
|
||||
* 1-indexed line numbers so they line up with how editors and `Content-Range`
|
||||
* headers reference NDJSON payloads.
|
||||
*/
|
||||
export function parseNdjsonObjects(
|
||||
body: Buffer,
|
||||
options: ParseNdjsonObjectsOptions = {}
|
||||
): unknown[] {
|
||||
const text = body.toString('utf8')
|
||||
const rows: unknown[] = []
|
||||
const lines = text.split(/\r?\n/)
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i]
|
||||
if (line.length === 0) continue
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(line)
|
||||
} catch (error) {
|
||||
throw new Error(`NDJSON parse failed at line ${i + 1}: ${toError(error).message}`)
|
||||
}
|
||||
if (options.requireObject) {
|
||||
if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) {
|
||||
throw new Error(`NDJSON row at line ${i + 1} is not an object`)
|
||||
}
|
||||
}
|
||||
rows.push(parsed)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
export function refineServiceAccountJson(
|
||||
value: { serviceAccountJson: string },
|
||||
ctx: z.RefinementCtx
|
||||
): void {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(value.serviceAccountJson)
|
||||
} catch {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ['serviceAccountJson'],
|
||||
message: 'serviceAccountJson must be valid JSON',
|
||||
})
|
||||
return
|
||||
}
|
||||
if (typeof parsed !== 'object' || parsed === null) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ['serviceAccountJson'],
|
||||
message: 'serviceAccountJson must be a JSON object',
|
||||
})
|
||||
return
|
||||
}
|
||||
const obj = parsed as Record<string, unknown>
|
||||
if (typeof obj.client_email !== 'string' || obj.client_email.length === 0) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ['serviceAccountJson'],
|
||||
message: 'serviceAccountJson is missing client_email',
|
||||
})
|
||||
}
|
||||
if (typeof obj.private_key !== 'string' || obj.private_key.length === 0) {
|
||||
ctx.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ['serviceAccountJson'],
|
||||
message: 'serviceAccountJson is missing private_key',
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -7,19 +7,28 @@ import {
|
||||
secureFetchWithPinnedIP,
|
||||
validateUrlWithDNS,
|
||||
} from '@/lib/core/security/input-validation.server'
|
||||
import {
|
||||
backoffWithJitter,
|
||||
parseRetryAfter,
|
||||
sleepUntilAborted,
|
||||
} from '@/lib/data-drains/destinations/utils'
|
||||
import type { DeliveryMetadata, DrainDestination } from '@/lib/data-drains/types'
|
||||
|
||||
const logger = createLogger('DataDrainWebhookDestination')
|
||||
|
||||
/** Initial attempt + 3 retries — matches the documented 500ms/1s/2s backoff sequence. */
|
||||
/** Initial attempt + 3 retries — 500ms/1s/2s backoff sequence. */
|
||||
const MAX_ATTEMPTS = 4
|
||||
const BASE_BACKOFF_MS = 500
|
||||
const MAX_BACKOFF_MS = 30_000
|
||||
const PER_ATTEMPT_TIMEOUT_MS = 30_000
|
||||
/** Cap responder reply so a misbehaving receiver can't OOM the runner. */
|
||||
const MAX_RESPONSE_BYTES = 256 * 1024
|
||||
const SIGNATURE_VERSION = 'v1'
|
||||
const USER_AGENT = 'Sim-DataDrain/1.0'
|
||||
|
||||
/** Reserved header names that callers cannot reuse as the signature header. */
|
||||
/**
|
||||
* Headers `buildHeaders` emits. Callers cannot override these via
|
||||
* `signatureHeader`. Keep in sync with `buildHeaders` — the drift-guard test
|
||||
* enforces this by parsing the schema against every key the function writes.
|
||||
*/
|
||||
const RESERVED_SIGNATURE_HEADER_NAMES = new Set([
|
||||
'authorization',
|
||||
'content-type',
|
||||
@@ -36,14 +45,9 @@ const RESERVED_SIGNATURE_HEADER_NAMES = new Set([
|
||||
'x-sim-signature',
|
||||
])
|
||||
|
||||
/**
|
||||
* Resolves the URL's hostname and returns the validated public IP. Uses
|
||||
* `ipaddr.js` so all non-`unicast` ranges (RFC1918, loopback, CGNAT, multicast,
|
||||
* broadcast, IPv4-mapped IPv6, link-local, cloud metadata) are blocked
|
||||
* uniformly. The returned IP is then pinned to the underlying socket via
|
||||
* `secureFetchWithPinnedIP` to defeat DNS rebinding (TOCTOU) between the
|
||||
* validation lookup and the actual delivery.
|
||||
*/
|
||||
/** CR/LF/NUL would let a bearer token smuggle additional response headers. */
|
||||
const HEADER_INJECTION_PATTERN = /[\r\n\0]/
|
||||
|
||||
async function resolvePublicTarget(url: string): Promise<string> {
|
||||
const result = await validateUrlWithDNS(url, 'url')
|
||||
if (!result.isValid || !result.resolvedIP) {
|
||||
@@ -56,10 +60,10 @@ const webhookConfigSchema = z.object({
|
||||
url: z
|
||||
.string()
|
||||
.url('url must be a valid URL')
|
||||
.max(2048, 'url must be at most 2048 characters')
|
||||
.refine((value) => validateExternalUrl(value, 'url').isValid, {
|
||||
message: 'url must be HTTPS and not point at a private, loopback, or metadata address',
|
||||
}),
|
||||
/** Optional custom header name for the signature (default: X-Sim-Signature). */
|
||||
signatureHeader: z
|
||||
.string()
|
||||
.min(1)
|
||||
@@ -67,73 +71,41 @@ const webhookConfigSchema = z.object({
|
||||
.refine((value) => !RESERVED_SIGNATURE_HEADER_NAMES.has(value.toLowerCase()), {
|
||||
message: 'signatureHeader cannot reuse a reserved Sim header name',
|
||||
})
|
||||
.refine((value) => !HEADER_INJECTION_PATTERN.test(value) && /^[A-Za-z0-9\-_]+$/.test(value), {
|
||||
message: 'signatureHeader must contain only letters, digits, hyphens, and underscores',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
const webhookCredentialsSchema = z.object({
|
||||
/** Shared secret used for HMAC-SHA256 signing of the request body. */
|
||||
signingSecret: z.string().min(8, 'signingSecret must be at least 8 characters'),
|
||||
/** Optional bearer token sent as Authorization header. */
|
||||
bearerToken: z.string().min(1).optional(),
|
||||
signingSecret: z
|
||||
.string()
|
||||
.min(32, 'signingSecret must be at least 32 characters')
|
||||
.max(512, 'signingSecret must be at most 512 characters'),
|
||||
bearerToken: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(4096, 'bearerToken must be at most 4096 characters')
|
||||
.refine((value) => !HEADER_INJECTION_PATTERN.test(value), {
|
||||
message: 'bearerToken cannot contain CR, LF, or NUL characters',
|
||||
})
|
||||
.optional(),
|
||||
})
|
||||
|
||||
export type WebhookDestinationConfig = z.infer<typeof webhookConfigSchema>
|
||||
export type WebhookDestinationCredentials = z.infer<typeof webhookCredentialsSchema>
|
||||
|
||||
/**
|
||||
* Stripe-style replay-resistant signature: signs `${unixSeconds}.${body}` and
|
||||
* emits `t=<unixSeconds>,v1=<hex(hmac)>`. Verifiers should reject signatures
|
||||
* older than ~5 minutes after also recomputing the HMAC over the same
|
||||
* concatenation, defending against captured-request replay attacks.
|
||||
* Stripe-style signature: HMAC-SHA256 over `${unixSeconds}.${body}` rendered as
|
||||
* `t=<unixSeconds>,v1=<hex>`. Verifiers reject stale timestamps (~5 min skew)
|
||||
* to block replay; we re-sign per attempt so long backoffs don't fall outside
|
||||
* that window.
|
||||
*/
|
||||
function sign(body: Buffer, secret: string, timestamp: number): string {
|
||||
const hmac = createHmac('sha256', secret).update(`${timestamp}.`).update(body).digest('hex')
|
||||
return `t=${timestamp},${SIGNATURE_VERSION}=${hmac}`
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves after `ms` or as soon as `signal` aborts, whichever happens first.
|
||||
* The caller checks `signal.aborted` at the top of the next iteration to
|
||||
* surface the abort — keeping resolution side-effect-free here.
|
||||
*/
|
||||
function sleepUntilAborted(ms: number, signal: AbortSignal): Promise<void> {
|
||||
if (signal.aborted) return Promise.resolve()
|
||||
return new Promise((resolve) => {
|
||||
const onAbort = () => {
|
||||
clearTimeout(timeoutId)
|
||||
resolve()
|
||||
}
|
||||
const timeoutId = setTimeout(() => {
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
resolve()
|
||||
}, ms)
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
})
|
||||
}
|
||||
|
||||
function backoffWithJitter(attempt: number, retryAfterMs?: number): number {
|
||||
if (retryAfterMs !== undefined) {
|
||||
// Floor at 500ms so a misbehaving server returning Retry-After: 0 cannot
|
||||
// pin us in a tight retry loop.
|
||||
return Math.min(Math.max(retryAfterMs, BASE_BACKOFF_MS), MAX_BACKOFF_MS)
|
||||
}
|
||||
const exponential = Math.min(BASE_BACKOFF_MS * 2 ** (attempt - 1), MAX_BACKOFF_MS)
|
||||
// ±20% jitter avoids thundering-herd alignment across drains.
|
||||
return exponential * (0.8 + Math.random() * 0.4)
|
||||
}
|
||||
|
||||
function parseRetryAfter(header: string | null): number | undefined {
|
||||
if (!header) return undefined
|
||||
const seconds = Number.parseInt(header, 10)
|
||||
if (!Number.isNaN(seconds) && seconds >= 0) return seconds * 1000
|
||||
const dateMs = Date.parse(header)
|
||||
if (!Number.isNaN(dateMs)) {
|
||||
const delta = dateMs - Date.now()
|
||||
return delta > 0 ? delta : 0
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function isRetryableStatus(status: number): boolean {
|
||||
return status === 408 || status === 429 || status >= 500
|
||||
}
|
||||
@@ -164,7 +136,7 @@ function buildHeaders(input: {
|
||||
headers['X-Sim-Source'] = input.metadata.source
|
||||
headers['X-Sim-Sequence'] = input.metadata.sequence.toString()
|
||||
headers['X-Sim-Row-Count'] = input.metadata.rowCount.toString()
|
||||
// Lets idempotent receivers dedupe retried chunks server-side.
|
||||
// Stable across retries of the same chunk so receivers can dedupe.
|
||||
headers['Idempotency-Key'] = `${input.metadata.runId}-${input.metadata.sequence}`
|
||||
}
|
||||
if (input.isProbe) {
|
||||
@@ -201,6 +173,7 @@ export const webhookDestination: DrainDestination<
|
||||
headers,
|
||||
signal,
|
||||
timeout: PER_ATTEMPT_TIMEOUT_MS,
|
||||
maxResponseBytes: MAX_RESPONSE_BYTES,
|
||||
})
|
||||
if (!response.ok) {
|
||||
throw new Error(`Webhook probe failed: HTTP ${response.status}`)
|
||||
@@ -211,21 +184,15 @@ export const webhookDestination: DrainDestination<
|
||||
let resolvedIP: string | null = null
|
||||
return {
|
||||
async deliver({ body, contentType, metadata, signal }) {
|
||||
// Resolve once per session — within a run we trust the result rather
|
||||
// than paying DNS on every chunk. Done lazily so a session that's
|
||||
// opened-and-immediately-closed pays no cost. The pinned IP is reused
|
||||
// across retries to defeat DNS rebinding (TOCTOU) attacks.
|
||||
// Resolve once per session and pin across retries to defeat DNS rebinding (TOCTOU).
|
||||
if (resolvedIP === null) {
|
||||
resolvedIP = await resolvePublicTarget(config.url)
|
||||
}
|
||||
let lastError: unknown
|
||||
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
|
||||
if (signal.aborted) throw signal.reason ?? new Error('Aborted')
|
||||
// Re-build headers per attempt so the timestamp + signature are
|
||||
// fresh (otherwise long backoffs would push us outside the
|
||||
// verifier's skew window).
|
||||
const headers = buildHeaders({ config, credentials, body, contentType, metadata })
|
||||
let retryAfterMs: number | undefined
|
||||
let retryAfterMs: number | null = null
|
||||
let response: Awaited<ReturnType<typeof secureFetchWithPinnedIP>> | undefined
|
||||
try {
|
||||
response = await secureFetchWithPinnedIP(config.url, resolvedIP, {
|
||||
@@ -234,6 +201,7 @@ export const webhookDestination: DrainDestination<
|
||||
headers,
|
||||
signal,
|
||||
timeout: PER_ATTEMPT_TIMEOUT_MS,
|
||||
maxResponseBytes: MAX_RESPONSE_BYTES,
|
||||
})
|
||||
} catch (error) {
|
||||
lastError = error
|
||||
@@ -262,7 +230,6 @@ export const webhookDestination: DrainDestination<
|
||||
}
|
||||
}
|
||||
if (!isRetryableStatus(response.status)) {
|
||||
// Non-retryable HTTP error: surface immediately without retrying.
|
||||
throw new Error(`Webhook responded with HTTP ${response.status}`)
|
||||
}
|
||||
lastError = new Error(`Webhook responded with HTTP ${response.status}`)
|
||||
|
||||
@@ -10,7 +10,15 @@ export const SOURCE_TYPES = [
|
||||
|
||||
export type SourceType = (typeof SOURCE_TYPES)[number]
|
||||
|
||||
export const DESTINATION_TYPES = ['s3', 'webhook'] as const
|
||||
export const DESTINATION_TYPES = [
|
||||
's3',
|
||||
'gcs',
|
||||
'azure_blob',
|
||||
'datadog',
|
||||
'bigquery',
|
||||
'snowflake',
|
||||
'webhook',
|
||||
] as const
|
||||
|
||||
export type DestinationType = (typeof DESTINATION_TYPES)[number]
|
||||
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
ALTER TYPE "public"."data_drain_destination" ADD VALUE 'gcs' BEFORE 'webhook';--> statement-breakpoint
|
||||
ALTER TYPE "public"."data_drain_destination" ADD VALUE 'azure_blob' BEFORE 'webhook';--> statement-breakpoint
|
||||
ALTER TYPE "public"."data_drain_destination" ADD VALUE 'datadog' BEFORE 'webhook';--> statement-breakpoint
|
||||
ALTER TYPE "public"."data_drain_destination" ADD VALUE 'bigquery' BEFORE 'webhook';--> statement-breakpoint
|
||||
ALTER TYPE "public"."data_drain_destination" ADD VALUE 'snowflake' BEFORE 'webhook';
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1436,6 +1436,13 @@
|
||||
"when": 1778540313360,
|
||||
"tag": "0205_smooth_sentinel",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 206,
|
||||
"version": "7",
|
||||
"when": 1778545079615,
|
||||
"tag": "0206_aromatic_veda",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -3148,7 +3148,15 @@ export const dataDrainSourceEnum = pgEnum('data_drain_source', [
|
||||
|
||||
export type DataDrainSource = (typeof dataDrainSourceEnum.enumValues)[number]
|
||||
|
||||
export const dataDrainDestinationEnum = pgEnum('data_drain_destination', ['s3', 'webhook'])
|
||||
export const dataDrainDestinationEnum = pgEnum('data_drain_destination', [
|
||||
's3',
|
||||
'gcs',
|
||||
'azure_blob',
|
||||
'datadog',
|
||||
'bigquery',
|
||||
'snowflake',
|
||||
'webhook',
|
||||
])
|
||||
|
||||
export type DataDrainDestination = (typeof dataDrainDestinationEnum.enumValues)[number]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user