mirror of
https://github.com/saltbo/zpan.git
synced 2026-08-28 15:51:29 +08:00
fix(analytics): enforce trustworthy offline statistics (#510)
* fix(analytics): enforce trustworthy offline statistics Separate immutable counters from point-in-time snapshots, expose incomplete coverage instead of synthetic zeroes, and keep browser analytics result-only. Restore finite Free quota baselines, fail closed for invalid storage quota, reconcile traffic reports fairly, and add production-safe backfill and data-quality diagnostics. * fix(analytics): preserve global backfill totals Group generated hourly backfill rows by their projected values so SQLite cannot resolve output aliases to source organization columns and overwrite cross-organization totals.
This commit is contained in:
@@ -13,7 +13,7 @@ The dashboard is read-only and does not support report or CSV export.
|
||||
|
||||
## Fact writes
|
||||
|
||||
Metrics derived from activity facts use a validated producer contract. New transfer and sharing facts fail fast when required fields such as `bytes`, `source`, `trafficEventId`, or `shareId` are missing. Historical rows that predate this contract remain diagnosable through quality metrics.
|
||||
Metrics derived from activity facts use a validated producer contract. New transfer and sharing facts fail fast when required fields such as `bytes`, `source`, `trafficEventId`, or `shareId` are missing. User signup, share creation, background-job completion, and remote-download completion each write a deterministic immutable stats fact in the same transaction as the business change. Deleting the mutable user/share/job/task row therefore does not rewrite history. Historical rows that predate this contract remain diagnosable through quality metrics.
|
||||
|
||||
A deduplicated share view updates `shares.views` and inserts its `share_view` fact in one database transaction. Download flows record an issued fact only after quota/metering and presigning succeed; a failed fact records the stable reason and traffic event id. If the issued-fact write fails, the flow refunds the counters it already reserved.
|
||||
|
||||
@@ -22,17 +22,19 @@ A deduplicated share view updates `shares.views` and inserts its `share_view` fa
|
||||
Metric and dimension combinations are declared in `server/domain/admin-stats-metrics.ts`:
|
||||
|
||||
- Counters are additive event totals for a closed hour.
|
||||
- Gauges are point-in-time snapshots written for that hour's close.
|
||||
- Gauges are point-in-time snapshots captured during the current open hour and continuously replaced in that same bucket.
|
||||
- High-volume source rows are grouped by the database before the Worker builds result rows; the scheduler does not load a whole hour of raw events into memory.
|
||||
|
||||
The ten-minute scheduler always rebuilds the latest closed UTC hour with counters and snapshots. It also checks the previous 48 hours and repairs at most three missing counter buckets per run. Historical repair does not invent old snapshots from current state.
|
||||
The ten-minute scheduler finalizes counters for the latest closed UTC hour, repairs at most three missing counter buckets from the previous 48 hours, and then captures a fresh gauge snapshot into the current open UTC hour. When the next hour starts, finalization keeps the snapshot captured in its original hour, preserves its exact `snapshotObservedAt`, and upgrades that bucket's completion marker. Historical repair has no API for generating snapshots and cannot copy current state into an old bucket. The dashboard reports the actual snapshot observation time instead of presenting the bucket end as the observation time.
|
||||
|
||||
Each row and `stats.rollup_run` marker carries:
|
||||
|
||||
- the rollup schema version;
|
||||
- `scope=full` for counters plus snapshots, or `scope=counters` for a historical counter repair;
|
||||
- `scope=snapshots` for an open-hour snapshot, `scope=full` for a closed bucket with both counters and a previously captured snapshot, or `scope=counters` for a closed bucket whose historical snapshot is unavailable;
|
||||
- `quality=exact` or `quality=lower_bound`.
|
||||
|
||||
Full markers carry separate `counterQuality` and `snapshotQuality` values so an incomplete historical transfer does not incorrectly downgrade an exact inventory snapshot.
|
||||
|
||||
A transaction deletes and reinserts the affected result scope, including its marker, so retries are idempotent. Readers reject old-version rows, rows without valid metadata, orphan rows without a compatible marker, and gauges backed only by a counter-repair marker.
|
||||
|
||||
## Query model
|
||||
@@ -50,7 +52,7 @@ Responses with period-over-period deltas also include `comparisonCoverage`. A de
|
||||
fully reconcilable when both the requested range and its comparison range are complete; the UI
|
||||
shows either side's missing buckets instead of presenting a partial comparison as authoritative.
|
||||
|
||||
Event-only traffic coverage accepts `counters` and `full` markers. Views that depend on snapshots require `full` markers.
|
||||
Event-only traffic coverage accepts `counters` and `full` markers. Closed-hour views that depend on snapshots require `full` markers; open `snapshots` markers are never queryable.
|
||||
|
||||
Request-time work is bounded and result-only. Charts and totals read only the aggregate dimensions they need; high-cardinality share dimensions are not loaded when querying totals. Top-share and top-space queries aggregate the result table and return at most eight ids; only those ids are hydrated from the operational tables for display names. No dashboard endpoint scans users, files, activities, shares, jobs, tasks, reports, or downloaders.
|
||||
|
||||
@@ -59,16 +61,23 @@ Request-time work is bounded and result-only. Charts and totals read only the ag
|
||||
- Upload bytes mean successfully confirmed upload bytes.
|
||||
- Download bytes and counts mean issued downloads, not client-completed transfers.
|
||||
- Cloud traffic report status is a closed-hour snapshot of the current metering queue and belongs to Operations, not download success. Mutable report rows are never modeled as additive counters.
|
||||
- Remote-download outcomes count terminal run transitions. A failed run followed by a successful retry contributes one failure and one completion; deleting the task does not erase either outcome.
|
||||
- User, storage, share lifecycle, active job/task, downloader, report, and webhook state comes from closed-hour snapshots, not live request-time reads.
|
||||
- Storage inventory includes normal files and image-hosting objects.
|
||||
- Storage inventory includes non-trashed normal files and image-hosting objects.
|
||||
- Quota usage includes files retained in trash until purge. `storage.trash_snapshot` exposes that file and byte total separately, so active-file inventory is not expected to equal the quota waterline by itself.
|
||||
- `stats.data_quality_snapshot/storage_usage_drift` compares the quota usage ledger with all currently billable database objects. A non-zero space or byte drift is shown as an error instead of silently presenting the waterline as authoritative.
|
||||
- Workspace storage quota is the active paid plan, or the active Free baseline when no paid plan is valid, plus active non-plan grants. Missing or zero effective storage entitlement is invalid and always fails closed; it never means unlimited space.
|
||||
- The ten-minute maintenance cycle idempotently restores missing or accidentally revoked Free baseline entitlements before capturing quota snapshots. It preserves each existing baseline size and uses the configured finite defaults only when a baseline row is absent.
|
||||
- Files older than 90 days are an age cohort, not proven cold data; the UI labels this explicitly.
|
||||
- Sharing views, downloads, and saves are independent event totals. Downloads/saves per 100 views are intensity ratios, not user-level conversion funnels.
|
||||
|
||||
Rows with transfer events but no recoverable byte count carry `quality=lower_bound`. `stats.quality_missing_bytes` exposes affected event counts by direction and source.
|
||||
|
||||
Legacy share counters can exceed the number of timestamped activity facts. The snapshot metric `stats.data_quality_snapshot` records the currently provable gap for views and downloads. The sharing dashboard labels its range values as located-event lower bounds and suppresses period comparisons and per-view ratios while either gap is non-zero; it never assigns those legacy counts to invented dates.
|
||||
|
||||
## Backfill and validation
|
||||
|
||||
Run `pnpm stats:backfill -- --apply ...` after the migration in each environment. The script repairs recoverable historical fact metadata, removes incompatible result versions, and rebuilds historical counters idempotently. It writes a continuous `scope=counters` marker for every closed UTC hour from the first available fact through the latest closed hour, including hours with zero activity, and rejects missing or open-hour markers during validation. Existing `scope=full` markers and snapshot rows remain intact.
|
||||
Run `pnpm stats:backfill -- --apply ...` after the migration in each environment. The script repairs recoverable historical fact metadata, creates deterministic lower-bound facts for currently recoverable signup/share/job/task history, removes incompatible result versions, and rebuilds historical counters idempotently. It writes a continuous `scope=counters` marker for every closed UTC hour from the first available immutable fact through the latest closed hour, including hours with zero activity, and rejects missing or open-hour markers during validation. Incompatible older snapshots are removed rather than relabeled as current-version facts.
|
||||
|
||||
The script does not fabricate historical inventory or active-user snapshots. Snapshot-backed views therefore report partial or empty coverage before full rollups began, while event-backed views can report complete historical coverage. Facts whose original byte size can no longer be recovered remain visible as `quality=lower_bound` instead of being guessed.
|
||||
|
||||
|
||||
@@ -89,7 +89,8 @@ Ownership:
|
||||
- **Cloud is merchant of record in v2.6.** Cloud owns Stripe Checkout, Stripe subscriptions, Stripe paid webhooks, credit grants, gift-card redemption, usage debit, order state, and webhook retries.
|
||||
- **The ZPan operator defines the package catalog per instance.** Package name, description, bytes, amount, currency, active state, and sort order are configured in ZPan admin.
|
||||
- **Terminal buyers do not need Cloud accounts.** ZPan creates checkout requests through the existing paid-tier Cloud binding and sends the user to Cloud only for checkout handling.
|
||||
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** ZPan keeps admin base quota separate from Cloud-delivered storage and traffic entitlements, then exposes effective quota as `base + entitlement`. Editing the admin base quota changes only the local base quota. The existing `0` base quota sentinel means unlimited only when no active Cloud entitlement exists for that resource.
|
||||
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** Every organization keeps a Free baseline plan. An active paid plan overrides that baseline while grants add to it; expiry or revocation therefore falls back to Free automatically. Missing or zero effective storage entitlement fails closed and never means unlimited storage.
|
||||
- A scheduled, idempotent reconciler restores missing or accidentally revoked Free baselines before quota snapshots while preserving existing baseline sizes.
|
||||
- **Monthly storage package subscriptions stay Cloud-owned.** Cloud owns Stripe subscriptions and sends authenticated `order.quota_changed` deliveries only for entitlement-changing subscription states: initial entitlement, positive create/update deltas, renewals that replace the current entitlement period, and terminal reversals. ZPan records deliveries idempotently under a stable subscription entitlement id such as `stripe_subscription:<subscriptionId>:<orgId>` and recomputes effective quota from active entitlements.
|
||||
- **Metered traffic billing is credit-backed and usage-event based.** Credits are unitless internal ZPan Cloud units, not USD and not Stripe Customer Balance. ZPan increments local monthly traffic usage before issuing a presigned download URL, reports `traffic_egress` usage to Cloud with an idempotency key, and denies the request if Cloud returns an insufficient-credit decision. ZPan does not price or debit credits.
|
||||
- **Subscriptions and credits are separate.** Stripe subscriptions remain normal USD subscriptions and may include credits per billing period. Free or fully couponed subscriptions still grant included credits when the Cloud subscription or invoice state qualifies. Credits can also come from top-up purchases, gift-card redemption, and admin grants, but credits only pay usage-based charges. They do not pay subscriptions or fixed package purchases.
|
||||
|
||||
@@ -43,7 +43,8 @@ The quota store lets a Pro instance operator sell monthly storage packages and m
|
||||
- **Cloud is merchant of record in v2.6.** Stripe Checkout, Stripe subscriptions, Stripe webhooks, Cloud order state, credit grants, gift-card redemption, and usage debits run through ZPan Cloud.
|
||||
- **The ZPan operator defines the package catalog per instance.** Admins create packages in ZPan with name, description, bytes, price, currency, active state, and sort order.
|
||||
- **Terminal buyers do not need Cloud accounts.** ZPan creates binding-authenticated checkout requests through the Pro Cloud binding; Cloud uses those requests to create orders.
|
||||
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** ZPan keeps admin base quota separate from Cloud-delivered storage and traffic entitlements, then exposes the effective quota as `base + entitlement`. Editing the admin base quota changes only the local base quota. The existing `0` base quota sentinel means unlimited only when no active Cloud entitlement exists for that resource.
|
||||
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** Every organization keeps a Free baseline plan. An active paid plan overrides that baseline while grants add to it; expiry or revocation therefore falls back to Free automatically. Missing or zero effective storage entitlement fails closed and never means unlimited storage.
|
||||
- The scheduled quota reconciler restores missing or previously revoked Free baseline rows before analytics snapshots, so installations affected by the old paid-plan replacement logic heal automatically after upgrade.
|
||||
- **Monthly storage package subscriptions are Cloud-owned.** ZPan does not run subscription billing locally. Cloud owns Stripe subscriptions and sends authenticated `order.quota_changed` deliveries only for entitlement-changing subscription states: initial entitlement, positive create/update deltas, renewals that replace the current entitlement period, and terminal reversals. ZPan records deliveries idempotently under a stable subscription entitlement id such as `stripe_subscription:<subscriptionId>:<orgId>` and recomputes the effective quota from active entitlements.
|
||||
- **Metered traffic billing is credit-backed and usage-event based.** Credits are unitless internal ZPan Cloud units, not USD and not Stripe Customer Balance. ZPan increments local monthly traffic usage before issuing a presigned download URL, reports `traffic_egress` usage to Cloud with an idempotency key, and denies the request if Cloud returns an insufficient-credit decision. ZPan does not price or debit credits.
|
||||
- **Subscriptions and credits are separate.** Stripe subscriptions remain normal USD subscriptions and may include credits per billing period. Free or fully couponed subscriptions still grant included credits when the Cloud subscription or invoice state qualifies. Credits can also come from top-up purchases, gift-card redemption, and admin grants, but credits only pay usage-based charges. They do not pay subscriptions or fixed package purchases.
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
DROP INDEX `org_quota_entitlements_active_plan_uniq`;--> statement-breakpoint
|
||||
CREATE UNIQUE INDEX `org_quota_entitlements_active_plan_uniq` ON `org_quota_entitlements` (`org_id`,`resource_type`,`entitlement_type`) WHERE status = 'active' AND entitlement_type = 'plan' AND source <> 'free_plan';--> statement-breakpoint
|
||||
ALTER TABLE `cloud_traffic_reports` ADD `attempt_count` integer DEFAULT 0 NOT NULL;--> statement-breakpoint
|
||||
ALTER TABLE `cloud_traffic_reports` ADD `next_retry_at` integer;--> statement-breakpoint
|
||||
CREATE INDEX `cloud_traffic_reports_retry_idx` ON `cloud_traffic_reports` (`status`,`next_retry_at`,`created_at`);
|
||||
@@ -0,0 +1 @@
|
||||
CREATE UNIQUE INDEX `org_quotas_org_uniq` ON `org_quotas` (`org_id`);
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -386,6 +386,20 @@
|
||||
"when": 1783697537563,
|
||||
"tag": "0055_hourly-admin-stats",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 56,
|
||||
"version": "6",
|
||||
"when": 1784524502907,
|
||||
"tag": "0056_stats-correctness-round-1",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 57,
|
||||
"version": "6",
|
||||
"when": 1784526402754,
|
||||
"tag": "0057_stats-quota-uniqueness",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
+299
-71
@@ -143,6 +143,7 @@ SET metadata = json_set(
|
||||
ELSE ''
|
||||
END
|
||||
)
|
||||
AND ctr.status <> 'blocked'
|
||||
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
|
||||
ORDER BY ctr.created_at, ctr.event_id
|
||||
LIMIT 1
|
||||
@@ -159,6 +160,7 @@ SET metadata = json_set(
|
||||
ELSE ''
|
||||
END
|
||||
)
|
||||
AND ctr.status <> 'blocked'
|
||||
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
|
||||
ORDER BY ctr.created_at, ctr.event_id
|
||||
LIMIT 1
|
||||
@@ -176,6 +178,7 @@ SET metadata = json_set(
|
||||
ELSE ''
|
||||
END
|
||||
)
|
||||
AND ctr.status <> 'blocked'
|
||||
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
|
||||
ORDER BY ctr.created_at, ctr.event_id
|
||||
LIMIT 1
|
||||
@@ -196,6 +199,7 @@ WHERE action IN ('share_download', 'object_download', 'image_hosting_download',
|
||||
ELSE ''
|
||||
END
|
||||
)
|
||||
AND ctr.status <> 'blocked'
|
||||
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
|
||||
);
|
||||
|
||||
@@ -237,6 +241,7 @@ WHERE ctr.source IN ('direct_share', 'landing_share', 'image_hosting', 'object_d
|
||||
OR (
|
||||
ae.target_id = ctr.source_id
|
||||
AND ae.action = CASE
|
||||
WHEN ctr.status = 'blocked' THEN 'download_failed'
|
||||
WHEN ctr.source IN ('direct_share', 'landing_share') THEN 'share_download'
|
||||
WHEN ctr.source = 'image_hosting' THEN 'image_hosting_download'
|
||||
WHEN ctr.source = 'object_download' THEN 'object_download'
|
||||
@@ -246,6 +251,8 @@ WHERE ctr.source IN ('direct_share', 'landing_share', 'image_hosting', 'object_d
|
||||
)
|
||||
);
|
||||
|
||||
${buildImmutableFactBackfillSql()}
|
||||
|
||||
${purgeLegacyRollupsSql()}
|
||||
${purgeOpenRollupsSql(now)}
|
||||
${purgeCounterRollupsSql()}
|
||||
@@ -254,6 +261,127 @@ ${buildHourlyBackfillSql(now)}
|
||||
`
|
||||
}
|
||||
|
||||
function buildImmutableFactBackfillSql(): string {
|
||||
return `INSERT OR IGNORE INTO activity_events (
|
||||
id, org_id, user_id, actor_type, actor_ref, action, target_type,
|
||||
target_id, target_name, metadata, created_at
|
||||
)
|
||||
SELECT
|
||||
'stats:stats_share_created:' || ae.target_id,
|
||||
ae.org_id, NULL, 'system', 'stats-backfill', 'stats_share_created', 'share',
|
||||
ae.target_id, ae.target_id,
|
||||
json_object(
|
||||
'kind', COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END, 'unknown'),
|
||||
'statsQuality', 'lower_bound'
|
||||
),
|
||||
ae.created_at
|
||||
FROM activity_events ae
|
||||
WHERE ae.action = 'share_create' AND ae.target_id IS NOT NULL;
|
||||
|
||||
INSERT OR IGNORE INTO activity_events (
|
||||
id, org_id, user_id, actor_type, actor_ref, action, target_type,
|
||||
target_id, target_name, metadata, created_at
|
||||
)
|
||||
SELECT
|
||||
'stats:stats_share_created:' || s.id,
|
||||
s.org_id, NULL, 'system', 'stats-backfill', 'stats_share_created', 'share',
|
||||
s.id, s.id, json_object('kind', s.kind, 'statsQuality', 'lower_bound'), s.created_at
|
||||
FROM shares s;
|
||||
|
||||
INSERT OR IGNORE INTO activity_events (
|
||||
id, org_id, user_id, actor_type, actor_ref, action, target_type,
|
||||
target_id, target_name, metadata, created_at
|
||||
)
|
||||
SELECT
|
||||
'stats:stats_user_signup:' || u.id,
|
||||
COALESCE((
|
||||
SELECT m.organization_id
|
||||
FROM member m
|
||||
JOIN organization o ON o.id = m.organization_id
|
||||
WHERE m.user_id = u.id AND o.metadata LIKE '%"type":"personal"%'
|
||||
ORDER BY m.created_at, m.id
|
||||
LIMIT 1
|
||||
), ''),
|
||||
NULL, 'system', 'stats-backfill', 'stats_user_signup', 'user',
|
||||
u.id, u.id,
|
||||
json_object(
|
||||
'provider', COALESCE((
|
||||
SELECT a.provider_id FROM account a WHERE a.user_id = u.id ORDER BY a.created_at, a.id LIMIT 1
|
||||
), 'direct'),
|
||||
'statsQuality', 'lower_bound'
|
||||
),
|
||||
CAST(u.created_at / 1000 AS INTEGER)
|
||||
FROM user u;
|
||||
|
||||
INSERT OR IGNORE INTO activity_events (
|
||||
id, org_id, user_id, actor_type, actor_ref, action, target_type,
|
||||
target_id, target_name, metadata, created_at
|
||||
)
|
||||
SELECT
|
||||
'stats:stats_background_job_finished:' || bj.id,
|
||||
bj.org_id, NULL, 'system', 'stats-backfill', 'stats_background_job_finished', 'background_job',
|
||||
bj.id, bj.id,
|
||||
json_object('jobType', bj.type, 'outcome', bj.status, 'statsQuality', 'lower_bound'),
|
||||
CAST(bj.finished_at / 1000 AS INTEGER)
|
||||
FROM background_jobs bj
|
||||
WHERE bj.finished_at IS NOT NULL;
|
||||
|
||||
INSERT OR IGNORE INTO activity_events (
|
||||
id, org_id, user_id, actor_type, actor_ref, action, target_type,
|
||||
target_id, target_name, metadata, created_at
|
||||
)
|
||||
SELECT
|
||||
'stats:stats_remote_download_finished:event:' || ae.id,
|
||||
ae.org_id, NULL, 'system', 'stats-backfill', 'stats_remote_download_finished', 'remote_download',
|
||||
ae.target_id, ae.target_id,
|
||||
json_object(
|
||||
'category', COALESCE(dt.category, 'uncategorized'),
|
||||
'downloaderId', dt.assigned_downloader_id,
|
||||
'outcome', CASE ae.action
|
||||
WHEN 'download_task_completed' THEN 'completed'
|
||||
WHEN 'download_task_failed' THEN 'failed'
|
||||
ELSE 'canceled'
|
||||
END,
|
||||
'bytes', CASE WHEN ae.action = 'download_task_completed' THEN COALESCE(dt.billing_charged_bytes, 0) ELSE 0 END,
|
||||
'statsQuality', 'lower_bound'
|
||||
),
|
||||
ae.created_at
|
||||
FROM activity_events ae
|
||||
LEFT JOIN download_tasks dt ON dt.id = ae.target_id
|
||||
WHERE ae.action IN ('download_task_completed', 'download_task_failed', 'download_task_canceled')
|
||||
AND ae.target_id IS NOT NULL;
|
||||
|
||||
INSERT OR IGNORE INTO activity_events (
|
||||
id, org_id, user_id, actor_type, actor_ref, action, target_type,
|
||||
target_id, target_name, metadata, created_at
|
||||
)
|
||||
SELECT
|
||||
'stats:stats_remote_download_finished:' || dt.id,
|
||||
dt.org_id, NULL, 'system', 'stats-backfill', 'stats_remote_download_finished', 'remote_download',
|
||||
dt.id, dt.id,
|
||||
json_object(
|
||||
'category', COALESCE(dt.category, 'uncategorized'),
|
||||
'downloaderId', dt.assigned_downloader_id,
|
||||
'outcome', dt.status,
|
||||
'bytes', dt.billing_charged_bytes,
|
||||
'statsQuality', 'lower_bound'
|
||||
),
|
||||
CAST(dt.finished_at / 1000 AS INTEGER)
|
||||
FROM download_tasks dt
|
||||
WHERE dt.finished_at IS NOT NULL
|
||||
AND dt.status IN ('completed', 'failed', 'canceled')
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM activity_events ae
|
||||
WHERE ae.target_id = dt.id
|
||||
AND ae.action = CASE dt.status
|
||||
WHEN 'completed' THEN 'download_task_completed'
|
||||
WHEN 'failed' THEN 'download_task_failed'
|
||||
WHEN 'canceled' THEN 'download_task_canceled'
|
||||
ELSE ''
|
||||
END
|
||||
);`
|
||||
}
|
||||
|
||||
type HourlySource = {
|
||||
metric: string
|
||||
source: string
|
||||
@@ -270,6 +398,22 @@ type HourlySource = {
|
||||
|
||||
function buildHourlyBackfillSql(now: Date): string {
|
||||
const currentHour = Math.floor(now.getTime() / 3_600_000) * 3_600_000
|
||||
const missingShareViewsQuality = `CASE WHEN EXISTS (
|
||||
SELECT 1 FROM shares s
|
||||
WHERE s.views > (
|
||||
SELECT COUNT(*) FROM activity_events history
|
||||
WHERE history.action = 'share_view'
|
||||
AND COALESCE(CASE WHEN json_valid(history.metadata) = 1 THEN json_extract(history.metadata, '$.shareId') END, history.target_id) = s.id
|
||||
)
|
||||
) THEN 'lower_bound' ELSE 'exact' END`
|
||||
const missingShareDownloadsQuality = `CASE WHEN EXISTS (
|
||||
SELECT 1 FROM shares s
|
||||
WHERE s.downloads > (
|
||||
SELECT COUNT(*) FROM activity_events history
|
||||
WHERE history.action = 'share_download'
|
||||
AND COALESCE(CASE WHEN json_valid(history.metadata) = 1 THEN json_extract(history.metadata, '$.shareId') END, history.target_id) = s.id
|
||||
)
|
||||
) THEN 'lower_bound' ELSE 'exact' END`
|
||||
const sources: HourlySource[] = [
|
||||
{
|
||||
metric: 'transfer.upload',
|
||||
@@ -293,7 +437,10 @@ function buildHourlyBackfillSql(now: Date): string {
|
||||
org: 'ae.org_id',
|
||||
where: "ae.action IN ('share_download','object_download','image_hosting_download','webdav_download')",
|
||||
bytes: "SUM(CASE WHEN json_valid(ae.metadata) = 1 THEN COALESCE(json_extract(ae.metadata, '$.bytes'), 0) ELSE 0 END)",
|
||||
quality: "CASE WHEN SUM(CASE WHEN ae.metadata IS NULL OR json_valid(ae.metadata) = 0 OR json_type(ae.metadata, '$.bytes') IS NULL THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
|
||||
quality: `CASE WHEN
|
||||
SUM(CASE WHEN ae.metadata IS NULL OR json_valid(ae.metadata) = 0 OR json_type(ae.metadata, '$.bytes') IS NULL THEN 1 ELSE 0 END) > 0
|
||||
OR (${missingShareDownloadsQuality}) = 'lower_bound'
|
||||
THEN 'lower_bound' ELSE 'exact' END`,
|
||||
dimensions: {
|
||||
source: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.source') END, CASE ae.action WHEN 'object_download' THEN 'object_download' WHEN 'image_hosting_download' THEN 'image_hosting' WHEN 'webdav_download' THEN 'webdav_download' ELSE 'landing_share' END)",
|
||||
actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)",
|
||||
@@ -314,31 +461,45 @@ function buildHourlyBackfillSql(now: Date): string {
|
||||
},
|
||||
{
|
||||
metric: 'share.created',
|
||||
source: 'shares s',
|
||||
timestampMs: 's.created_at * 1000',
|
||||
org: 's.org_id',
|
||||
dimensions: { kind: 's.kind' },
|
||||
source: 'activity_events ae',
|
||||
timestampMs: 'ae.created_at * 1000',
|
||||
org: 'ae.org_id',
|
||||
where: "ae.action = 'stats_share_created'",
|
||||
quality:
|
||||
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
|
||||
dimensions: { kind: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END, 'unknown')" },
|
||||
},
|
||||
activitySource('share.view', "ae.action = 'share_view'", { share_id: 'ae.target_id', actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }),
|
||||
activitySource('share.download_issued', "ae.action = 'share_download'", { share_id: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.shareId') END, ae.target_id)", kind: "CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END", source: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.source') END, 'landing_share')", actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, true),
|
||||
activitySource('share.view', "ae.action = 'share_view'", { share_id: 'ae.target_id', actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, false, missingShareViewsQuality),
|
||||
activitySource('share.download_issued', "ae.action = 'share_download'", { share_id: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.shareId') END, ae.target_id)", kind: "CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END", source: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.source') END, 'landing_share')", actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, true, missingShareDownloadsQuality),
|
||||
activitySource('share.saved', "ae.action = 'save_from_share'", { share_id: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.shareId') END, ae.target_id)", actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, true),
|
||||
activitySource('share.password_passed', "ae.action = 'share_password_passed'", { share_id: 'ae.target_id' }),
|
||||
{
|
||||
metric: 'remote_download.task_finished',
|
||||
source: 'download_tasks dt',
|
||||
timestampMs: 'dt.finished_at',
|
||||
org: 'dt.org_id',
|
||||
where: 'dt.finished_at IS NOT NULL',
|
||||
bytes: 'SUM(dt.billing_charged_bytes)',
|
||||
dimensions: { category: "COALESCE(dt.category, 'uncategorized')", downloader_id: 'dt.assigned_downloader_id', outcome: 'dt.status' },
|
||||
source: 'activity_events ae',
|
||||
timestampMs: 'ae.created_at * 1000',
|
||||
org: 'ae.org_id',
|
||||
where: "ae.action = 'stats_remote_download_finished'",
|
||||
bytes: "SUM(CASE WHEN json_valid(ae.metadata) = 1 THEN COALESCE(json_extract(ae.metadata, '$.bytes'), 0) ELSE 0 END)",
|
||||
quality:
|
||||
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
|
||||
dimensions: {
|
||||
category: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.category') END, 'uncategorized')",
|
||||
downloader_id: "CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.downloaderId') END",
|
||||
outcome: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.outcome') END, 'unknown')",
|
||||
},
|
||||
},
|
||||
{
|
||||
metric: 'background_job.finished',
|
||||
source: 'background_jobs bj',
|
||||
timestampMs: 'bj.finished_at',
|
||||
org: 'bj.org_id',
|
||||
where: 'bj.finished_at IS NOT NULL',
|
||||
dimensions: { job_type: 'bj.type', outcome: 'bj.status' },
|
||||
source: 'activity_events ae',
|
||||
timestampMs: 'ae.created_at * 1000',
|
||||
org: 'ae.org_id',
|
||||
where: "ae.action = 'stats_background_job_finished'",
|
||||
quality:
|
||||
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
|
||||
dimensions: {
|
||||
job_type: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.jobType') END, 'unknown')",
|
||||
outcome: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.outcome') END, 'unknown')",
|
||||
},
|
||||
},
|
||||
{
|
||||
metric: 'stats.quality_missing_bytes',
|
||||
@@ -361,8 +522,8 @@ function buildHourlyBackfillSql(now: Date): string {
|
||||
function purgeLegacyRollupsSql(): string {
|
||||
return `DELETE FROM stats_rollups_hourly
|
||||
WHERE CASE WHEN json_valid(metadata) = 1 THEN
|
||||
json_extract(metadata, '$.version') = 2
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
json_extract(metadata, '$.version') = 3
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
|
||||
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
|
||||
ELSE 0 END = 0;`
|
||||
}
|
||||
@@ -386,7 +547,13 @@ WHERE metric_key IN (
|
||||
);`
|
||||
}
|
||||
|
||||
function activitySource(metric: string, where: string, dimensions: Record<string, string>, bytes = false): HourlySource {
|
||||
function activitySource(
|
||||
metric: string,
|
||||
where: string,
|
||||
dimensions: Record<string, string>,
|
||||
bytes = false,
|
||||
quality?: string,
|
||||
): HourlySource {
|
||||
return {
|
||||
metric,
|
||||
source: 'activity_events ae',
|
||||
@@ -394,6 +561,7 @@ function activitySource(metric: string, where: string, dimensions: Record<string
|
||||
org: 'ae.org_id',
|
||||
where,
|
||||
bytes: bytes ? "SUM(CASE WHEN json_valid(ae.metadata) = 1 THEN COALESCE(json_extract(ae.metadata, '$.bytes'), 0) ELSE 0 END)" : undefined,
|
||||
quality,
|
||||
dimensions,
|
||||
}
|
||||
}
|
||||
@@ -428,14 +596,14 @@ SELECT
|
||||
CAST(bucket_start AS TEXT) || ':' || COALESCE(NULLIF(org_id, ''), 'global') || ':${source.metric}:${dimensionKey || 'all'}:' || hex(dimension_value),
|
||||
bucket_start, org_id, '${source.metric}', '${dimensionKey}', dimension_value,
|
||||
count_value, bytes_value, unique_value,
|
||||
json_object('version', 2, 'scope', '${source.scope ?? 'counters'}', 'quality', quality_value),
|
||||
json_object('version', 3, 'scope', '${source.scope ?? 'counters'}', 'quality', quality_value),
|
||||
bucket_start + 3600000
|
||||
FROM (
|
||||
SELECT ${bucket} AS bucket_start, ${source.org} AS org_id, ${dimension} AS dimension_value,
|
||||
${count} AS count_value, ${bytes} AS bytes_value, ${uniqueCount} AS unique_value, ${quality} AS quality_value
|
||||
FROM ${source.source}
|
||||
${where ? `WHERE ${where}` : ''}
|
||||
GROUP BY bucket_start, org_id${dimensionKey ? ', dimension_value' : ''}
|
||||
GROUP BY 1, 2${dimensionKey ? ', 3' : ''}
|
||||
) rollup
|
||||
WHERE true
|
||||
ON CONFLICT(bucket_start, org_id, metric_key, dimension_key, dimension_value)
|
||||
@@ -447,12 +615,15 @@ WHERE count <> excluded.count OR bytes <> excluded.bytes OR unique_count <> excl
|
||||
function userSignupBackfillSql(before: number): string {
|
||||
return hourlyStatements({
|
||||
metric: 'user.signup',
|
||||
source: `user u LEFT JOIN account a ON a.id = (
|
||||
SELECT a2.id FROM account a2 WHERE a2.user_id = u.id ORDER BY a2.created_at, a2.id LIMIT 1
|
||||
)`,
|
||||
timestampMs: 'u.created_at',
|
||||
source: 'activity_events ae',
|
||||
timestampMs: 'ae.created_at * 1000',
|
||||
org: "''",
|
||||
dimensions: { provider: "COALESCE(a.provider_id, 'direct')" },
|
||||
where: "ae.action = 'stats_user_signup'",
|
||||
quality:
|
||||
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
|
||||
dimensions: {
|
||||
provider: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.provider') END, 'unknown')",
|
||||
},
|
||||
}, before).join('\n\n')
|
||||
}
|
||||
|
||||
@@ -477,50 +648,100 @@ buckets AS (
|
||||
FROM bounds
|
||||
JOIN numbers ON start_at + numbers.n * 3600000 <= end_at
|
||||
WHERE start_at IS NOT NULL AND start_at <= end_at
|
||||
),
|
||||
snapshot_markers AS MATERIALIZED (
|
||||
SELECT
|
||||
bucket_start,
|
||||
COALESCE(json_extract(metadata, '$.snapshotQuality'), json_extract(metadata, '$.quality'), 'exact') AS quality,
|
||||
COALESCE(json_extract(metadata, '$.snapshotObservedAt'), json_extract(metadata, '$.observedAt')) AS observed_at
|
||||
FROM stats_rollups_hourly
|
||||
WHERE metric_key = 'stats.rollup_run' AND org_id = '' AND dimension_key = '' AND dimension_value = ''
|
||||
AND json_valid(metadata) = 1
|
||||
AND json_extract(metadata, '$.version') = 3
|
||||
AND json_extract(metadata, '$.scope') IN ('snapshots', 'full')
|
||||
AND COALESCE(json_extract(metadata, '$.snapshotObservedAt'), json_extract(metadata, '$.observedAt')) IS NOT NULL
|
||||
)
|
||||
INSERT INTO stats_rollups_hourly (
|
||||
id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
|
||||
count, bytes, unique_count, metadata, updated_at
|
||||
)
|
||||
SELECT
|
||||
CAST(bucket_start AS TEXT) || ':global:stats.rollup_run:all:all',
|
||||
bucket_start, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
json_object('version', 2, 'scope', 'counters', 'quality', 'exact'),
|
||||
bucket_start + 3600000
|
||||
CAST(buckets.bucket_start AS TEXT) || ':global:stats.rollup_run:all:all',
|
||||
buckets.bucket_start, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
CASE WHEN snapshot_markers.bucket_start IS NULL THEN
|
||||
json_object(
|
||||
'version', 3,
|
||||
'scope', 'counters',
|
||||
'quality', CASE WHEN EXISTS (
|
||||
SELECT 1 FROM stats_rollups_hourly result
|
||||
WHERE result.bucket_start = buckets.bucket_start
|
||||
AND result.metric_key <> 'stats.rollup_run'
|
||||
AND json_valid(result.metadata) = 1
|
||||
AND json_extract(result.metadata, '$.version') = 3
|
||||
AND json_extract(result.metadata, '$.scope') = 'counters'
|
||||
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
|
||||
) THEN 'lower_bound' ELSE 'exact' END,
|
||||
'counterQuality', CASE WHEN EXISTS (
|
||||
SELECT 1 FROM stats_rollups_hourly result
|
||||
WHERE result.bucket_start = buckets.bucket_start
|
||||
AND result.metric_key <> 'stats.rollup_run'
|
||||
AND json_valid(result.metadata) = 1
|
||||
AND json_extract(result.metadata, '$.version') = 3
|
||||
AND json_extract(result.metadata, '$.scope') = 'counters'
|
||||
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
|
||||
) THEN 'lower_bound' ELSE 'exact' END
|
||||
)
|
||||
ELSE
|
||||
json_object(
|
||||
'version', 3,
|
||||
'scope', 'full',
|
||||
'quality', CASE WHEN snapshot_markers.quality = 'lower_bound' OR EXISTS (
|
||||
SELECT 1 FROM stats_rollups_hourly result
|
||||
WHERE result.bucket_start = buckets.bucket_start
|
||||
AND result.metric_key <> 'stats.rollup_run'
|
||||
AND json_valid(result.metadata) = 1
|
||||
AND json_extract(result.metadata, '$.version') = 3
|
||||
AND json_extract(result.metadata, '$.scope') = 'counters'
|
||||
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
|
||||
) THEN 'lower_bound' ELSE 'exact' END,
|
||||
'counterQuality', CASE WHEN EXISTS (
|
||||
SELECT 1 FROM stats_rollups_hourly result
|
||||
WHERE result.bucket_start = buckets.bucket_start
|
||||
AND result.metric_key <> 'stats.rollup_run'
|
||||
AND json_valid(result.metadata) = 1
|
||||
AND json_extract(result.metadata, '$.version') = 3
|
||||
AND json_extract(result.metadata, '$.scope') = 'counters'
|
||||
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
|
||||
) THEN 'lower_bound' ELSE 'exact' END,
|
||||
'snapshotQuality', snapshot_markers.quality,
|
||||
'snapshotObservedAt', snapshot_markers.observed_at
|
||||
)
|
||||
END,
|
||||
buckets.bucket_start + 3600000
|
||||
FROM buckets
|
||||
LEFT JOIN snapshot_markers ON snapshot_markers.bucket_start = buckets.bucket_start
|
||||
WHERE true
|
||||
ON CONFLICT(bucket_start, org_id, metric_key, dimension_key, dimension_value)
|
||||
DO UPDATE SET count = excluded.count, bytes = excluded.bytes, unique_count = excluded.unique_count,
|
||||
metadata = excluded.metadata, updated_at = excluded.updated_at
|
||||
WHERE CASE WHEN json_valid(stats_rollups_hourly.metadata) = 1 THEN
|
||||
json_extract(stats_rollups_hourly.metadata, '$.version') = 2
|
||||
AND json_extract(stats_rollups_hourly.metadata, '$.scope') = 'full'
|
||||
AND json_extract(stats_rollups_hourly.metadata, '$.quality') IN ('exact', 'lower_bound')
|
||||
ELSE 0 END = 0
|
||||
AND (
|
||||
count <> excluded.count OR bytes <> excluded.bytes OR unique_count <> excluded.unique_count
|
||||
OR metadata <> excluded.metadata OR updated_at <> excluded.updated_at
|
||||
);`
|
||||
WHERE count <> excluded.count OR bytes <> excluded.bytes OR unique_count <> excluded.unique_count
|
||||
OR metadata <> excluded.metadata OR updated_at <> excluded.updated_at;`
|
||||
}
|
||||
|
||||
function statsHistoryStartSql(): string {
|
||||
const missing = '9223372036854775807'
|
||||
return `NULLIF(MIN(
|
||||
COALESCE((SELECT CAST(MIN(u.created_at) / 3600000 AS INTEGER) * 3600000
|
||||
FROM user u WHERE u.created_at >= ${MIN_VALID_TIMESTAMP_MS}), ${missing}),
|
||||
COALESCE((SELECT CAST((MIN(s.created_at) * 1000) / 3600000 AS INTEGER) * 3600000
|
||||
FROM shares s WHERE s.created_at >= ${MIN_VALID_TIMESTAMP_SECONDS}), ${missing}),
|
||||
COALESCE((SELECT CAST((MIN(ae.created_at) * 1000) / 3600000 AS INTEGER) * 3600000
|
||||
FROM activity_events ae WHERE ae.created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND ae.action IN (
|
||||
return `NULLIF(COALESCE((
|
||||
SELECT CAST((MIN(ae.created_at) * 1000) / 3600000 AS INTEGER) * 3600000
|
||||
FROM activity_events ae
|
||||
WHERE ae.created_at >= ${MIN_VALID_TIMESTAMP_SECONDS}
|
||||
AND ae.action IN (
|
||||
'upload_confirm', 'upload_cancel', 'upload_failed',
|
||||
'share_download', 'object_download', 'image_hosting_download', 'webdav_download', 'download_failed',
|
||||
'share_view', 'save_from_share', 'share_password_passed'
|
||||
)), ${missing}),
|
||||
COALESCE((SELECT CAST(MIN(dt.finished_at) / 3600000 AS INTEGER) * 3600000
|
||||
FROM download_tasks dt WHERE dt.finished_at IS NOT NULL AND dt.finished_at >= ${MIN_VALID_TIMESTAMP_MS}), ${missing}),
|
||||
COALESCE((SELECT CAST(MIN(bj.finished_at) / 3600000 AS INTEGER) * 3600000
|
||||
FROM background_jobs bj WHERE bj.finished_at IS NOT NULL AND bj.finished_at >= ${MIN_VALID_TIMESTAMP_MS}), ${missing})
|
||||
), ${missing})`
|
||||
'share_view', 'save_from_share', 'share_password_passed',
|
||||
'stats_user_signup', 'stats_share_created',
|
||||
'stats_remote_download_finished', 'stats_background_job_finished'
|
||||
)
|
||||
), ${missing}), ${missing})`
|
||||
}
|
||||
|
||||
export function buildValidationSql(now = new Date()): string {
|
||||
@@ -589,12 +810,14 @@ export function buildValidationSql(now = new Date()): string {
|
||||
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
),
|
||||
'rawUserSignups', (
|
||||
SELECT COUNT(*) FROM user
|
||||
WHERE created_at >= ${MIN_VALID_TIMESTAMP_MS} AND created_at < ${currentHour}
|
||||
SELECT COUNT(*) FROM activity_events
|
||||
WHERE action = 'stats_user_signup'
|
||||
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
),
|
||||
'rawSharesCreated', (
|
||||
SELECT COUNT(*) FROM shares
|
||||
WHERE created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
SELECT COUNT(*) FROM activity_events
|
||||
WHERE action = 'stats_share_created'
|
||||
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
),
|
||||
'rawShareDownloads', (
|
||||
SELECT COUNT(*) FROM activity_events
|
||||
@@ -617,12 +840,14 @@ export function buildValidationSql(now = new Date()): string {
|
||||
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
),
|
||||
'rawFinishedDownloadTasks', (
|
||||
SELECT COUNT(*) FROM download_tasks
|
||||
WHERE finished_at >= ${MIN_VALID_TIMESTAMP_MS} AND finished_at < ${currentHour}
|
||||
SELECT COUNT(*) FROM activity_events
|
||||
WHERE action = 'stats_remote_download_finished'
|
||||
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
),
|
||||
'rawFinishedBackgroundJobs', (
|
||||
SELECT COUNT(*) FROM background_jobs
|
||||
WHERE finished_at >= ${MIN_VALID_TIMESTAMP_MS} AND finished_at < ${currentHour}
|
||||
SELECT COUNT(*) FROM activity_events
|
||||
WHERE action = 'stats_background_job_finished'
|
||||
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
|
||||
),
|
||||
'rawMissingByteEvents', (
|
||||
SELECT COUNT(*) FROM activity_events
|
||||
@@ -636,8 +861,8 @@ export function buildValidationSql(now = new Date()): string {
|
||||
SELECT *
|
||||
FROM stats_rollups_hourly
|
||||
WHERE CASE WHEN json_valid(metadata) = 1 THEN
|
||||
json_extract(metadata, '$.version') = 2
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
json_extract(metadata, '$.version') = 3
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
|
||||
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
|
||||
ELSE 0 END = 1
|
||||
),
|
||||
@@ -686,8 +911,8 @@ SELECT json_object(
|
||||
'legacyRollupRows', (
|
||||
SELECT COUNT(*) FROM stats_rollups_hourly
|
||||
WHERE CASE WHEN json_valid(metadata) = 1 THEN
|
||||
json_extract(metadata, '$.version') = 2
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
json_extract(metadata, '$.version') = 3
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
|
||||
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
|
||||
ELSE 0 END = 0
|
||||
)
|
||||
@@ -697,14 +922,15 @@ SELECT json_object(
|
||||
SELECT *
|
||||
FROM stats_rollups_hourly
|
||||
WHERE CASE WHEN json_valid(metadata) = 1 THEN
|
||||
json_extract(metadata, '$.version') = 2
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
json_extract(metadata, '$.version') = 3
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
|
||||
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
|
||||
ELSE 0 END = 1
|
||||
),
|
||||
counter_markers AS MATERIALIZED (
|
||||
SELECT bucket_start FROM valid_rollups
|
||||
WHERE metric_key = 'stats.rollup_run' AND org_id = '' AND dimension_key = '' AND dimension_value = ''
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
),
|
||||
counter_rows AS MATERIALIZED (
|
||||
SELECT result.* FROM valid_rollups result
|
||||
@@ -759,10 +985,11 @@ SELECT json_object(
|
||||
FROM stats_rollups_hourly
|
||||
WHERE metric_key = 'stats.rollup_run' AND org_id = '' AND dimension_key = '' AND dimension_value = ''
|
||||
AND CASE WHEN json_valid(metadata) = 1 THEN
|
||||
json_extract(metadata, '$.version') = 2
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
json_extract(metadata, '$.version') = 3
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
|
||||
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
|
||||
ELSE 0 END = 1
|
||||
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
|
||||
),
|
||||
coverage AS MATERIALIZED (
|
||||
SELECT ${statsHistoryStartSql()} AS start_at, ${latestClosedHour} AS end_at
|
||||
@@ -828,6 +1055,7 @@ SELECT json_object(
|
||||
OR (
|
||||
ae.target_id = ctr.source_id
|
||||
AND ae.action = CASE
|
||||
WHEN ctr.status = 'blocked' THEN 'download_failed'
|
||||
WHEN ctr.source IN ('direct_share', 'landing_share') THEN 'share_download'
|
||||
WHEN ctr.source = 'image_hosting' THEN 'image_hosting_download'
|
||||
WHEN ctr.source = 'object_download' THEN 'object_download'
|
||||
|
||||
@@ -1,10 +1,11 @@
|
||||
import { and, count, desc, eq, gte, lte } from 'drizzle-orm'
|
||||
import { and, count, desc, eq, gte, lte, notInArray } from 'drizzle-orm'
|
||||
import { nanoid } from 'nanoid'
|
||||
import { organization, user } from '../../db/auth-schema'
|
||||
import { activityEvents } from '../../db/schema'
|
||||
import { assertAdminStatsEvent } from '../../domain/admin-stats-events'
|
||||
import type { Database } from '../../platform/interface'
|
||||
import type { ActivityActorType, ActivityRepo, RecordActivityInput } from '../../usecases/ports'
|
||||
import { ADMIN_STATS_FACT_ACTIONS } from './admin-stats-fact'
|
||||
|
||||
export function activityEventValues(event: RecordActivityInput): typeof activityEvents.$inferInsert {
|
||||
assertAdminStatsEvent(event.action, event.metadata)
|
||||
@@ -47,7 +48,11 @@ export function createActivityRepo(db: Database): ActivityRepo {
|
||||
const pageSize = opts.pageSize ?? 20
|
||||
const offset = (page - 1) * pageSize
|
||||
|
||||
const countRows = await db.select({ count: count() }).from(activityEvents).where(eq(activityEvents.orgId, orgId))
|
||||
const visible = and(
|
||||
eq(activityEvents.orgId, orgId),
|
||||
notInArray(activityEvents.action, [...ADMIN_STATS_FACT_ACTIONS]),
|
||||
)
|
||||
const countRows = await db.select({ count: count() }).from(activityEvents).where(visible)
|
||||
const total = countRows[0]?.count ?? 0
|
||||
|
||||
const rows = await db
|
||||
@@ -68,7 +73,7 @@ export function createActivityRepo(db: Database): ActivityRepo {
|
||||
})
|
||||
.from(activityEvents)
|
||||
.leftJoin(user, eq(activityEvents.userId, user.id))
|
||||
.where(eq(activityEvents.orgId, orgId))
|
||||
.where(visible)
|
||||
.orderBy(desc(activityEvents.createdAt))
|
||||
.limit(pageSize)
|
||||
.offset(offset)
|
||||
@@ -104,6 +109,7 @@ export function createActivityRepo(db: Database): ActivityRepo {
|
||||
const offset = (page - 1) * pageSize
|
||||
|
||||
const filters = [
|
||||
notInArray(activityEvents.action, [...ADMIN_STATS_FACT_ACTIONS]),
|
||||
opts.orgId ? eq(activityEvents.orgId, opts.orgId) : undefined,
|
||||
opts.userId ? eq(activityEvents.userId, opts.userId) : undefined,
|
||||
opts.action ? eq(activityEvents.action, opts.action) : undefined,
|
||||
@@ -176,6 +182,7 @@ export function createActivityRepo(db: Database): ActivityRepo {
|
||||
eq(activityEvents.orgId, opts.orgId),
|
||||
eq(activityEvents.targetType, opts.targetType),
|
||||
eq(activityEvents.targetId, opts.targetId),
|
||||
notInArray(activityEvents.action, [...ADMIN_STATS_FACT_ACTIONS]),
|
||||
)
|
||||
|
||||
const [countRows, rows] = await Promise.all([
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
import type { activityEvents } from '../../db/schema'
|
||||
|
||||
export const ADMIN_STATS_FACT_ACTIONS = [
|
||||
'stats_user_signup',
|
||||
'stats_share_created',
|
||||
'stats_background_job_finished',
|
||||
'stats_remote_download_finished',
|
||||
] as const
|
||||
|
||||
export type AdminStatsFactAction = (typeof ADMIN_STATS_FACT_ACTIONS)[number]
|
||||
|
||||
export function adminStatsFactValues(input: {
|
||||
action: AdminStatsFactAction
|
||||
sourceId: string
|
||||
targetId?: string
|
||||
orgId: string
|
||||
targetType: string
|
||||
occurredAt: Date
|
||||
metadata: Record<string, unknown>
|
||||
}): typeof activityEvents.$inferInsert {
|
||||
const targetId = input.targetId ?? input.sourceId
|
||||
return {
|
||||
id: `stats:${input.action}:${input.sourceId}`,
|
||||
orgId: input.orgId,
|
||||
userId: null,
|
||||
actorType: 'system',
|
||||
actorRef: 'admin-stats',
|
||||
action: input.action,
|
||||
targetType: input.targetType,
|
||||
targetId,
|
||||
targetName: targetId,
|
||||
metadata: JSON.stringify({ ...input.metadata, statsQuality: 'exact' }),
|
||||
createdAt: input.occurredAt,
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import {
|
||||
ADMIN_STATS_METRICS,
|
||||
type AdminStatsDimension,
|
||||
type AdminStatsMetric,
|
||||
type AdminStatsRollupMetadata,
|
||||
type AdminStatsRollupScope,
|
||||
assertMetricDimension,
|
||||
metricDefinition,
|
||||
@@ -33,7 +34,7 @@ export class AdminStatsHourlyReader {
|
||||
private readonly queryFrom: Date
|
||||
private readonly queryTo: Date
|
||||
private readonly metricRows = new Map<string, Promise<HourlyMetricRow[]>>()
|
||||
private readonly markerBucketsPromises = new Map<AdminStatsRollupScope, Promise<Set<number>>>()
|
||||
private readonly markerRowsPromises = new Map<AdminStatsRollupScope, Promise<CompatibleMarkerRow[]>>()
|
||||
|
||||
constructor(
|
||||
private readonly db: Database,
|
||||
@@ -121,19 +122,19 @@ export class AdminStatsHourlyReader {
|
||||
AND bucket_start >= ${this.queryFrom.getTime()}
|
||||
AND bucket_start < ${this.queryTo.getTime()}
|
||||
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.version') END = ${ROLLUP_VERSION}
|
||||
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.scope') END = 'full'
|
||||
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.scope') END IN ('snapshots', 'full')
|
||||
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.quality') END IN ('exact', 'lower_bound')
|
||||
)
|
||||
SELECT used.org_id AS orgId, used.bytes AS usedBytes, COALESCE(quota.bytes, 0) AS quotaBytes
|
||||
SELECT used.org_id AS orgId, used.bytes AS usedBytes, quota.bytes AS quotaBytes
|
||||
FROM stats_rollups_hourly used
|
||||
LEFT JOIN stats_rollups_hourly quota
|
||||
INNER JOIN stats_rollups_hourly quota
|
||||
ON quota.bucket_start = used.bucket_start
|
||||
AND quota.org_id = used.org_id
|
||||
AND quota.metric_key = ${ADMIN_STATS_METRICS.storageQuota}
|
||||
AND quota.dimension_key = ''
|
||||
AND quota.dimension_value = ''
|
||||
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.version') END = ${ROLLUP_VERSION}
|
||||
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.scope') END = 'full'
|
||||
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.scope') END IN ('snapshots', 'full')
|
||||
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.quality') END IN ('exact', 'lower_bound')
|
||||
WHERE used.bucket_start = (SELECT bucketStart FROM latest)
|
||||
AND used.org_id <> ''
|
||||
@@ -141,7 +142,7 @@ export class AdminStatsHourlyReader {
|
||||
AND used.dimension_key = ''
|
||||
AND used.dimension_value = ''
|
||||
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.version') END = ${ROLLUP_VERSION}
|
||||
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.scope') END = 'full'
|
||||
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.scope') END IN ('snapshots', 'full')
|
||||
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.quality') END IN ('exact', 'lower_bound')
|
||||
ORDER BY used.bytes DESC, used.org_id
|
||||
LIMIT ${DASHBOARD_RANKING_LIMIT}
|
||||
@@ -155,14 +156,22 @@ export class AdminStatsHourlyReader {
|
||||
|
||||
async coverage(requiredScope: AdminStatsRollupScope = 'full'): Promise<AdminStatsCoverage> {
|
||||
const expectedBuckets = Math.max(0, Math.floor((this.queryTo.getTime() - this.queryFrom.getTime()) / HOUR_MS))
|
||||
const markerBuckets = await this.markerBuckets(requiredScope)
|
||||
const completedBuckets = markerBuckets.size
|
||||
const latest = Math.max(...markerBuckets, Number.NEGATIVE_INFINITY)
|
||||
const markerRows = await this.markerRows(requiredScope)
|
||||
const completedBuckets = markerRows.length
|
||||
const lowerBoundBuckets = markerRows.filter(
|
||||
(row) => qualityForScope(row.metadata, requiredScope) === 'lower_bound',
|
||||
).length
|
||||
const latest = markerRows.reduce<CompatibleMarkerRow | null>(
|
||||
(value, row) => (!value || row.bucketStart.getTime() > value.bucketStart.getTime() ? row : value),
|
||||
null,
|
||||
)
|
||||
return {
|
||||
status: completedBuckets === 0 ? 'empty' : completedBuckets === expectedBuckets ? 'complete' : 'partial',
|
||||
expectedBuckets,
|
||||
completedBuckets,
|
||||
dataThrough: Number.isFinite(latest) ? new Date(latest + HOUR_MS).toISOString() : null,
|
||||
lowerBoundBuckets,
|
||||
quality: lowerBoundBuckets > 0 ? 'lower_bound' : 'exact',
|
||||
dataThrough: latest ? dataThroughForScope(latest, requiredScope) : null,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -179,7 +188,7 @@ export class AdminStatsHourlyReader {
|
||||
dimensionKeys: readonly (AdminStatsDimension | '')[],
|
||||
): Promise<HourlyMetricRow[]> {
|
||||
if (this.queryFrom >= this.queryTo) return []
|
||||
const requiredScope = metricDefinition(metric).kind === 'gauge' ? 'full' : 'counters'
|
||||
const requiredScope = metricDefinition(metric).kind === 'gauge' ? 'snapshots' : 'counters'
|
||||
const markerBuckets = await this.markerBuckets(requiredScope)
|
||||
if (markerBuckets.size === 0) return []
|
||||
const rows = await this.db
|
||||
@@ -201,8 +210,8 @@ export class AdminStatsHourlyReader {
|
||||
gte(statsRollupsHourly.bucketStart, this.queryFrom),
|
||||
lt(statsRollupsHourly.bucketStart, this.queryTo),
|
||||
sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.version') END = ${ROLLUP_VERSION}`,
|
||||
requiredScope === 'full'
|
||||
? sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.scope') END = 'full'`
|
||||
requiredScope === 'snapshots'
|
||||
? sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.scope') END IN ('snapshots', 'full')`
|
||||
: sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.scope') END IN ('counters', 'full')`,
|
||||
sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.quality') END IN ('exact', 'lower_bound')`,
|
||||
),
|
||||
@@ -225,15 +234,19 @@ export class AdminStatsHourlyReader {
|
||||
}
|
||||
|
||||
private markerBuckets(requiredScope: AdminStatsRollupScope): Promise<Set<number>> {
|
||||
const cached = this.markerBucketsPromises.get(requiredScope)
|
||||
if (cached) return cached
|
||||
const buckets = this.loadMarkerBuckets(requiredScope)
|
||||
this.markerBucketsPromises.set(requiredScope, buckets)
|
||||
return buckets
|
||||
return this.markerRows(requiredScope).then((rows) => new Set(rows.map((row) => row.bucketStart.getTime())))
|
||||
}
|
||||
|
||||
private async loadMarkerBuckets(requiredScope: AdminStatsRollupScope): Promise<Set<number>> {
|
||||
if (this.queryFrom >= this.queryTo) return new Set()
|
||||
private markerRows(requiredScope: AdminStatsRollupScope): Promise<CompatibleMarkerRow[]> {
|
||||
const cached = this.markerRowsPromises.get(requiredScope)
|
||||
if (cached) return cached
|
||||
const rows = this.loadMarkerRows(requiredScope)
|
||||
this.markerRowsPromises.set(requiredScope, rows)
|
||||
return rows
|
||||
}
|
||||
|
||||
private async loadMarkerRows(requiredScope: AdminStatsRollupScope): Promise<CompatibleMarkerRow[]> {
|
||||
if (this.queryFrom >= this.queryTo) return []
|
||||
const rows = await this.db
|
||||
.select({ bucketStart: statsRollupsHourly.bucketStart, metadata: statsRollupsHourly.metadata })
|
||||
.from(statsRollupsHourly)
|
||||
@@ -246,19 +259,37 @@ export class AdminStatsHourlyReader {
|
||||
lt(statsRollupsHourly.bucketStart, this.queryTo),
|
||||
),
|
||||
)
|
||||
return new Set(
|
||||
rows
|
||||
.filter((row) => {
|
||||
const metadata = parseAdminStatsRollupMetadata(row.metadata)
|
||||
return supportsScope(metadata?.scope, requiredScope)
|
||||
})
|
||||
.map((row) => row.bucketStart.getTime()),
|
||||
)
|
||||
return rows.flatMap((row) => {
|
||||
const metadata = parseAdminStatsRollupMetadata(row.metadata)
|
||||
return metadata && supportsScope(metadata.scope, requiredScope)
|
||||
? [{ bucketStart: row.bucketStart, metadata }]
|
||||
: []
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type CompatibleMarkerRow = { bucketStart: Date; metadata: AdminStatsRollupMetadata }
|
||||
|
||||
function qualityForScope(
|
||||
metadata: AdminStatsRollupMetadata,
|
||||
requiredScope: AdminStatsRollupScope,
|
||||
): 'exact' | 'lower_bound' {
|
||||
if (requiredScope === 'counters') return metadata.counterQuality ?? metadata.quality
|
||||
if (requiredScope === 'snapshots') return metadata.snapshotQuality ?? metadata.quality
|
||||
return metadata.quality
|
||||
}
|
||||
|
||||
function dataThroughForScope(row: CompatibleMarkerRow, requiredScope: AdminStatsRollupScope): string {
|
||||
if (requiredScope === 'snapshots' && row.metadata.snapshotObservedAt) return row.metadata.snapshotObservedAt
|
||||
return new Date(row.bucketStart.getTime() + HOUR_MS).toISOString()
|
||||
}
|
||||
|
||||
function supportsScope(scope: AdminStatsRollupScope | undefined, requiredScope: AdminStatsRollupScope): boolean {
|
||||
return scope === 'full' || (requiredScope === 'counters' && scope === 'counters')
|
||||
return (
|
||||
scope === 'full' ||
|
||||
(requiredScope === 'counters' && scope === 'counters') ||
|
||||
(requiredScope === 'snapshots' && scope === 'snapshots')
|
||||
)
|
||||
}
|
||||
|
||||
function floorHour(date: Date): Date {
|
||||
|
||||
@@ -3,7 +3,11 @@ import { describe, expect, it } from 'vitest'
|
||||
import { assertMetricDimension, ADMIN_STATS_METRICS as M, metricDefinition } from '../../domain/admin-stats-metrics'
|
||||
import { adminHeaders, createTestApp } from '../../test/setup.js'
|
||||
import { AdminStatsHourlyReader } from './admin-stats-hourly'
|
||||
import { ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE, rebuildAdminStatsHour } from './admin-stats-rollup'
|
||||
import {
|
||||
ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE,
|
||||
captureAdminStatsSnapshot,
|
||||
rebuildAdminStatsHour,
|
||||
} from './admin-stats-rollup'
|
||||
|
||||
type RollupRow = {
|
||||
orgId: string
|
||||
@@ -48,9 +52,18 @@ describe('admin hourly stats rollup', () => {
|
||||
`)
|
||||
await db.run(sql`
|
||||
UPDATE org_quotas
|
||||
SET used = 500, quota = 1000, traffic_used = 300, traffic_quota = 2000
|
||||
SET used = 1300, quota = 1000, traffic_used = 300, traffic_quota = 2000
|
||||
WHERE org_id = ${orgId}
|
||||
`)
|
||||
await db.run(sql`
|
||||
UPDATE org_quota_entitlements
|
||||
SET bytes = 1000, starts_at = ${atMs}
|
||||
WHERE org_id = ${orgId} AND resource_type = 'storage' AND source = 'free_plan' AND status = 'active'
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES ('rollup-orphan-quota', 'deleted-org', 999999, 999999, 0, 0, '2026-07')
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO storages
|
||||
(id, bucket, endpoint, region, access_key, secret_key, file_path, custom_host, capacity, used, status, created_at, updated_at)
|
||||
@@ -60,11 +73,11 @@ describe('admin hourly stats rollup', () => {
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO matters
|
||||
(id, org_id, alias, name, type, size, dirtype, parent, object, storage_id, status, created_at, updated_at)
|
||||
(id, org_id, alias, name, type, size, dirtype, parent, object, storage_id, status, trashed_at, created_at, updated_at)
|
||||
VALUES
|
||||
('rollup-file', ${orgId}, 'rollup-file', 'video.mp4', 'video/mp4', 200, 0, '', 'video.mp4', 'rollup-storage', 'active', ${atSec}, ${atSec}),
|
||||
('rollup-dir', ${orgId}, 'rollup-dir', 'folder', '', 0, 1, '', '', 'rollup-storage', 'active', ${atSec}, ${atSec}),
|
||||
('rollup-trashed', ${orgId}, 'rollup-trashed', 'old.bin', 'application/octet-stream', 800, 0, '', 'old.bin', 'rollup-storage', 'trashed', ${atSec}, ${atSec})
|
||||
('rollup-file', ${orgId}, 'rollup-file', 'video.mp4', 'video/mp4', 200, 0, '', 'video.mp4', 'rollup-storage', 'active', NULL, ${atSec}, ${atSec}),
|
||||
('rollup-dir', ${orgId}, 'rollup-dir', 'folder', '', 0, 1, '', '', 'rollup-storage', 'active', NULL, ${atSec}, ${atSec}),
|
||||
('rollup-trashed', ${orgId}, 'rollup-trashed', 'old.bin', 'application/octet-stream', 800, 0, '', 'old.bin', 'rollup-storage', 'trashed', ${atMs}, ${atSec}, ${atSec})
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO image_hostings
|
||||
@@ -114,7 +127,23 @@ describe('admin hourly stats rollup', () => {
|
||||
('rollup-team-join', ${orgId}, ${userId}, 'user', 'team_member_join', 'organization', 'rollup-team', 'team', NULL, ${atSec}),
|
||||
('rollup-team-remove', ${orgId}, ${userId}, 'user', 'team_member_remove', 'organization', 'rollup-team', 'team', NULL, ${atSec}),
|
||||
('rollup-license', ${orgId}, NULL, 'system', 'license_refresh', 'license', NULL, 'license',
|
||||
'{"status":"failed"}', ${atSec})
|
||||
'{"status":"failed"}', ${atSec}),
|
||||
('rollup-signup-credential', ${orgId}, NULL, 'system', 'stats_user_signup', 'user', ${userId}, ${userId},
|
||||
'{"provider":"credential","statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-signup-direct', '', NULL, 'system', 'stats_user_signup', 'user', 'rollup-direct-user', 'rollup-direct-user',
|
||||
'{"provider":"direct","statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-share-created-usable', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-usable', 'rollup-share-usable',
|
||||
'{"kind":"landing","statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-share-created-revoked', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-revoked', 'rollup-share-revoked',
|
||||
'{"kind":"direct","statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-share-created-expired', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-expired', 'rollup-share-expired',
|
||||
'{"kind":"landing","statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-share-created-limited', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-limited', 'rollup-share-limited',
|
||||
'{"kind":"direct","statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-task-finished-fact', ${orgId}, NULL, 'system', 'stats_remote_download_finished', 'remote_download', 'rollup-task-finished', 'rollup-task-finished',
|
||||
'{"category":"uncategorized","downloaderId":"rollup-downloader","outcome":"completed","bytes":60,"statsQuality":"exact"}', ${atSec}),
|
||||
('rollup-job-finished-fact', ${orgId}, NULL, 'system', 'stats_background_job_finished', 'background_job', 'rollup-job-finished', 'rollup-job-finished',
|
||||
'{"jobType":"archive","outcome":"failed","statsQuality":"exact"}', ${atSec})
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO cloud_traffic_reports
|
||||
@@ -160,7 +189,8 @@ describe('admin hourly stats rollup', () => {
|
||||
VALUES ('rollup-webhook', 'cloud', 'webhook-event', 'order.quota_changed', 'hash', '{}', 'processed', ${atMs}, ${atMs})
|
||||
`)
|
||||
|
||||
const first = await rebuildAdminStatsHour(db, bucketStart, generatedAt, true)
|
||||
await captureAdminStatsSnapshot(db, bucketStart, generatedAt)
|
||||
const first = await rebuildAdminStatsHour(db, bucketStart, generatedAt)
|
||||
const rows = await db.all<RollupRow>(sql`
|
||||
SELECT org_id AS orgId, metric_key AS metric, dimension_key AS dimensionKey,
|
||||
dimension_value AS dimensionValue, count, bytes, unique_count AS uniqueCount, metadata
|
||||
@@ -179,9 +209,10 @@ describe('admin hourly stats rollup', () => {
|
||||
expect(first).toMatchObject({
|
||||
bucketStart,
|
||||
bucketEnd: new Date('2026-07-10T13:00:00.000Z'),
|
||||
rows: rows.length,
|
||||
rows: expect.any(Number),
|
||||
lowerBoundRows: expect.any(Number),
|
||||
})
|
||||
expect(rows.length).toBeGreaterThan(first.rows)
|
||||
expect(first.lowerBoundRows).toBeGreaterThan(0)
|
||||
expect(row(M.transferUpload)).toMatchObject({ count: 4, bytes: 100 })
|
||||
expect(row(M.transferDownloadIssued)).toMatchObject({ count: 4, bytes: 90 })
|
||||
@@ -194,12 +225,19 @@ describe('admin hourly stats rollup', () => {
|
||||
expect(row(M.remoteDownloadTaskFinished)).toMatchObject({ count: 1, bytes: 60 })
|
||||
expect(row(M.backgroundJobFinished)).toMatchObject({ count: 1 })
|
||||
expect(row(M.storageInventory)).toMatchObject({ count: 2, bytes: 500 })
|
||||
expect(row(M.storageUsed)).toMatchObject({ bytes: 500 })
|
||||
expect(row(M.storageUsed)).toMatchObject({ bytes: 1300 })
|
||||
expect(row(M.storageQuota)).toMatchObject({ bytes: 1000 })
|
||||
expect(row(M.storageQuota, 'status', 'over', '')).toMatchObject({ count: 1 })
|
||||
expect(row(M.storageQuota, 'status', 'invalid', '')).toMatchObject({ count: 1 })
|
||||
expect(row(M.storageTrashSnapshot)).toMatchObject({ count: 1, bytes: 800 })
|
||||
expect(row(M.statsDataQualitySnapshot, 'kind', 'storage_usage_drift', '')).toMatchObject({ count: 0, bytes: 0 })
|
||||
expect(row(M.shareInventory, 'lifecycle', 'usable')).toMatchObject({ count: 1 })
|
||||
expect(row(M.shareInventory, 'lifecycle', 'revoked')).toMatchObject({ count: 1 })
|
||||
expect(row(M.shareInventory, 'lifecycle', 'expired')).toMatchObject({ count: 1 })
|
||||
expect(row(M.shareInventory, 'lifecycle', 'download_limit_reached')).toMatchObject({ count: 1 })
|
||||
expect(row(M.statsDataQualitySnapshot, '', '', '')).toMatchObject({ count: 1 })
|
||||
expect(row(M.statsDataQualitySnapshot, 'kind', 'share_downloads', '')).toMatchObject({ count: 1 })
|
||||
expect(row(M.statsDataQualitySnapshot, 'kind', 'share_views', '')).toMatchObject({ count: 0 })
|
||||
expect(row(M.backgroundJobSnapshot)).toMatchObject({ count: 1 })
|
||||
expect(row(M.remoteDownloadTaskSnapshot)).toMatchObject({ count: 1 })
|
||||
expect(row(M.downloaderSnapshot, '', '', '')).toMatchObject({ count: 2 })
|
||||
@@ -208,21 +246,40 @@ describe('admin hourly stats rollup', () => {
|
||||
expect(row(M.trafficReportSnapshot, '', '', '')).toMatchObject({ count: 2, bytes: 300 })
|
||||
expect(row(M.webhookSnapshot, 'status', 'processed', '')).toMatchObject({ count: 1 })
|
||||
expect(row(M.statsRollupRun, '', '', '')).toMatchObject({ count: 1 })
|
||||
expect(JSON.parse(row(M.transferUpload)?.metadata ?? '{}')).toMatchObject({ version: 2, quality: 'lower_bound' })
|
||||
expect(JSON.parse(row(M.transferUpload)?.metadata ?? '{}')).toMatchObject({ version: 3, quality: 'lower_bound' })
|
||||
expect(JSON.parse(row(M.statsRollupRun, '', '', '')?.metadata ?? '{}')).toMatchObject({
|
||||
version: 3,
|
||||
scope: 'full',
|
||||
quality: 'lower_bound',
|
||||
counterQuality: 'lower_bound',
|
||||
snapshotQuality: 'exact',
|
||||
snapshotObservedAt: generatedAt.toISOString(),
|
||||
})
|
||||
|
||||
const second = await rebuildAdminStatsHour(db, bucketStart, generatedAt, true)
|
||||
const reader = new AdminStatsHourlyReader(
|
||||
db,
|
||||
{ from: bucketStart, to: new Date(bucketStart.getTime() + 3_600_000 - 1), timeZone: 'UTC' },
|
||||
new Date(bucketStart.getTime() + 2 * 3_600_000),
|
||||
)
|
||||
await expect(reader.coverage('counters')).resolves.toMatchObject({ quality: 'lower_bound' })
|
||||
await expect(reader.coverage('snapshots')).resolves.toMatchObject({
|
||||
quality: 'exact',
|
||||
dataThrough: generatedAt.toISOString(),
|
||||
})
|
||||
|
||||
const second = await rebuildAdminStatsHour(db, bucketStart, generatedAt)
|
||||
const [{ count: storedRows }] = await db.all<{ count: number }>(sql`
|
||||
SELECT COUNT(*) AS count FROM stats_rollups_hourly WHERE bucket_start = ${bucketStart.getTime()}
|
||||
`)
|
||||
expect(second.rows).toBe(first.rows)
|
||||
expect(storedRows).toBe(first.rows)
|
||||
expect(storedRows).toBeGreaterThan(first.rows)
|
||||
})
|
||||
|
||||
it('rejects buckets that are not aligned to a UTC hour', async () => {
|
||||
const { db } = await createTestApp()
|
||||
|
||||
await expect(
|
||||
rebuildAdminStatsHour(db, new Date('2026-07-10T12:00:00.001Z'), new Date('2026-07-10T12:30:00Z'), false),
|
||||
rebuildAdminStatsHour(db, new Date('2026-07-10T12:00:00.001Z'), new Date('2026-07-10T12:30:00Z')),
|
||||
).rejects.toThrow('stats_bucket_must_align_to_utc_hour')
|
||||
})
|
||||
|
||||
@@ -231,8 +288,8 @@ describe('admin hourly stats rollup', () => {
|
||||
await db.run(sql`DROP TABLE org_quotas`)
|
||||
|
||||
await expect(
|
||||
rebuildAdminStatsHour(db, new Date('2026-07-10T12:00:00Z'), new Date('2026-07-10T12:30:00Z'), true),
|
||||
).rejects.toThrow('stats_rollup_query_failed:quota')
|
||||
captureAdminStatsSnapshot(db, new Date('2026-07-10T12:00:00Z'), new Date('2026-07-10T12:30:00Z')),
|
||||
).rejects.toThrow(/stats_rollup_query_failed:(quota|data-quality)/)
|
||||
})
|
||||
|
||||
it('reads only current-version result rows with a compatible completion scope', async () => {
|
||||
@@ -244,15 +301,15 @@ describe('admin hourly stats rollup', () => {
|
||||
count, bytes, unique_count, metadata, updated_at)
|
||||
VALUES
|
||||
('quality-marker', ${bucketStart}, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
('quality-null', ${bucketStart}, 'org-null', 'transfer.upload', '', '', 1, 1, 0, NULL, ${bucketStart}),
|
||||
('quality-array', ${bucketStart}, 'org-array', 'transfer.upload', '', '', 1, 2, 0, '[]', ${bucketStart}),
|
||||
('quality-v1', ${bucketStart}, 'org-v1', 'transfer.upload', '', '', 1, 3, 0,
|
||||
'{"version":1,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
('quality-exact', ${bucketStart}, 'org-exact', 'transfer.upload', '', '', 1, 4, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
('quality-lower', ${bucketStart}, 'org-lower', 'transfer.upload', '', '', 1, 5, 0,
|
||||
'{"version":2,"scope":"full","quality":"lower_bound"}', ${bucketStart})
|
||||
'{"version":3,"scope":"full","quality":"lower_bound"}', ${bucketStart})
|
||||
`)
|
||||
const reader = new AdminStatsHourlyReader(
|
||||
db,
|
||||
@@ -278,7 +335,7 @@ describe('admin hourly stats rollup', () => {
|
||||
it('keeps counter-only repairs separate from full snapshot results', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const bucketStart = Date.parse('2026-07-10T09:00:00.000Z')
|
||||
const metadata = '{"version":2,"scope":"counters","quality":"exact"}'
|
||||
const metadata = '{"version":3,"scope":"counters","quality":"exact"}'
|
||||
await db.run(sql`
|
||||
INSERT INTO stats_rollups_hourly
|
||||
(id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
|
||||
@@ -287,7 +344,7 @@ describe('admin hourly stats rollup', () => {
|
||||
('counter-marker', ${bucketStart}, '', 'stats.rollup_run', '', '', 1, 0, 0, ${metadata}, ${bucketStart}),
|
||||
('counter-result', ${bucketStart}, '', 'transfer.upload', '', '', 1, 42, 0, ${metadata}, ${bucketStart}),
|
||||
('orphan-gauge', ${bucketStart}, '', 'storage.used', '', '', 0, 99, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart})
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart})
|
||||
`)
|
||||
const reader = new AdminStatsHourlyReader(
|
||||
db,
|
||||
@@ -314,9 +371,9 @@ describe('admin hourly stats rollup', () => {
|
||||
count, bytes, unique_count, metadata, updated_at)
|
||||
VALUES
|
||||
('current-hour-marker', ${bucketStart}, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart}),
|
||||
('current-hour-rollup', ${bucketStart}, '', 'transfer.upload', '', '', 1, 42, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart})
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart})
|
||||
`)
|
||||
const reader = new AdminStatsHourlyReader(
|
||||
db,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { and, eq, gte, inArray, lt, sql } from 'drizzle-orm'
|
||||
import { and, eq, inArray, isNull, sql } from 'drizzle-orm'
|
||||
import { organization } from '../../db/auth-schema'
|
||||
import {
|
||||
backgroundJobs,
|
||||
cloudTrafficReports,
|
||||
@@ -17,9 +18,11 @@ import {
|
||||
type AdminStatsMetric,
|
||||
assertMetricDimension,
|
||||
ADMIN_STATS_METRICS as M,
|
||||
parseAdminStatsRollupMetadata,
|
||||
ROLLUP_VERSION,
|
||||
} from '../../domain/admin-stats-metrics'
|
||||
import type { Database } from '../../platform/interface'
|
||||
import { getEffectiveQuotasByOrg } from './quota'
|
||||
|
||||
const HOUR_MS = 3_600_000
|
||||
const D1_MAX_BOUND_PARAMS = 100
|
||||
@@ -41,6 +44,21 @@ const COUNTER_METRICS: AdminStatsMetric[] = [
|
||||
M.transferUpload,
|
||||
M.userSignup,
|
||||
]
|
||||
const GAUGE_METRICS: AdminStatsMetric[] = [
|
||||
M.backgroundJobSnapshot,
|
||||
M.downloaderSnapshot,
|
||||
M.remoteDownloadTaskSnapshot,
|
||||
M.shareInventory,
|
||||
M.statsDataQualitySnapshot,
|
||||
M.storageInventory,
|
||||
M.storageQuota,
|
||||
M.storageTrashSnapshot,
|
||||
M.storageUsed,
|
||||
M.trafficReportSnapshot,
|
||||
M.userActiveSnapshot,
|
||||
M.userInventory,
|
||||
M.webhookSnapshot,
|
||||
]
|
||||
|
||||
type RollupValue = {
|
||||
metric: AdminStatsMetric
|
||||
@@ -66,21 +84,21 @@ export async function rebuildAdminStatsHour(
|
||||
db: Database,
|
||||
bucketStartInput: Date,
|
||||
generatedAt: Date,
|
||||
includeSnapshots: boolean,
|
||||
): Promise<AdminStatsHourlyRollupResult> {
|
||||
const bucketStart = startOfHour(bucketStartInput)
|
||||
if (bucketStart.getTime() !== bucketStartInput.getTime()) throw new Error('stats_bucket_must_align_to_utc_hour')
|
||||
const bucketEnd = new Date(bucketStart.getTime() + HOUR_MS)
|
||||
const rollups = new RollupAccumulator()
|
||||
const capturedSnapshot = await compatibleSnapshotMarker(db, bucketStart)
|
||||
|
||||
await addEventMetrics(db, rollups, bucketStart, bucketEnd)
|
||||
await addUserMetrics(db, rollups, bucketStart, bucketEnd)
|
||||
await addOperationalMetrics(db, rollups, bucketStart, bucketEnd)
|
||||
if (includeSnapshots) await addSnapshotMetrics(db, rollups, generatedAt)
|
||||
|
||||
const values = rollups.values()
|
||||
const lowerBoundRows = values.filter((row) => row.lowerBound).length
|
||||
rollups.add(M.statsRollupRun, '', 1, 0, { outcome: 'success' })
|
||||
rollups.add(M.statsRollupRun, '', 1, 0, { outcome: 'success' }, lowerBoundRows > 0)
|
||||
const completionScope = capturedSnapshot ? 'full' : 'counters'
|
||||
const counterQuality = lowerBoundRows > 0 ? 'lower_bound' : 'exact'
|
||||
|
||||
const updatedAt = generatedAt
|
||||
const rows = rollups.values().map((row) => ({
|
||||
@@ -93,19 +111,39 @@ export async function rebuildAdminStatsHour(
|
||||
count: row.count,
|
||||
bytes: row.bytes,
|
||||
uniqueCount: row.uniqueCount,
|
||||
metadata: JSON.stringify({
|
||||
version: ROLLUP_VERSION,
|
||||
scope: includeSnapshots ? 'full' : 'counters',
|
||||
quality: row.lowerBound ? 'lower_bound' : 'exact',
|
||||
generatedAt: generatedAt.toISOString(),
|
||||
}),
|
||||
metadata: JSON.stringify(
|
||||
row.metric === M.statsRollupRun
|
||||
? {
|
||||
version: ROLLUP_VERSION,
|
||||
scope: completionScope,
|
||||
quality:
|
||||
counterQuality === 'lower_bound' || capturedSnapshot?.quality === 'lower_bound' ? 'lower_bound' : 'exact',
|
||||
counterQuality,
|
||||
...(capturedSnapshot
|
||||
? {
|
||||
snapshotQuality: capturedSnapshot.quality,
|
||||
snapshotObservedAt: capturedSnapshot.observedAt,
|
||||
}
|
||||
: {}),
|
||||
generatedAt: generatedAt.toISOString(),
|
||||
}
|
||||
: {
|
||||
version: ROLLUP_VERSION,
|
||||
scope: 'counters',
|
||||
quality: row.lowerBound ? 'lower_bound' : 'exact',
|
||||
generatedAt: generatedAt.toISOString(),
|
||||
},
|
||||
),
|
||||
updatedAt,
|
||||
}))
|
||||
|
||||
const deleteWhere = includeSnapshots
|
||||
? eq(statsRollupsHourly.bucketStart, bucketStart)
|
||||
: and(eq(statsRollupsHourly.bucketStart, bucketStart), inArray(statsRollupsHourly.metricKey, COUNTER_METRICS))
|
||||
const writes: AtomicQuery[] = [db.delete(statsRollupsHourly).where(deleteWhere)]
|
||||
const writes: AtomicQuery[] = [
|
||||
db
|
||||
.delete(statsRollupsHourly)
|
||||
.where(
|
||||
and(eq(statsRollupsHourly.bucketStart, bucketStart), inArray(statsRollupsHourly.metricKey, COUNTER_METRICS)),
|
||||
),
|
||||
]
|
||||
for (let index = 0; index < rows.length; index += ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE) {
|
||||
writes.push(db.insert(statsRollupsHourly).values(rows.slice(index, index + ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE)))
|
||||
}
|
||||
@@ -113,6 +151,79 @@ export async function rebuildAdminStatsHour(
|
||||
return { bucketStart, bucketEnd, rows: rows.length, lowerBoundRows }
|
||||
}
|
||||
|
||||
export async function captureAdminStatsSnapshot(
|
||||
db: Database,
|
||||
bucketStartInput: Date,
|
||||
observedAt: Date,
|
||||
): Promise<AdminStatsHourlyRollupResult> {
|
||||
const bucketStart = startOfHour(bucketStartInput)
|
||||
if (bucketStart.getTime() !== bucketStartInput.getTime()) throw new Error('stats_bucket_must_align_to_utc_hour')
|
||||
if (startOfHour(observedAt).getTime() !== bucketStart.getTime()) {
|
||||
throw new Error('stats_snapshot_must_be_captured_in_bucket')
|
||||
}
|
||||
const bucketEnd = new Date(bucketStart.getTime() + HOUR_MS)
|
||||
const rollups = new RollupAccumulator()
|
||||
await addSnapshotMetrics(db, rollups, observedAt)
|
||||
rollups.add(M.statsRollupRun, '', 1, 0, { outcome: 'success' })
|
||||
const rows = rollups.values().map((row) => ({
|
||||
id: rollupId(bucketStart, row),
|
||||
bucketStart,
|
||||
orgId: row.orgId,
|
||||
metricKey: row.metric,
|
||||
dimensionKey: row.dimensionKey,
|
||||
dimensionValue: row.dimensionValue,
|
||||
count: row.count,
|
||||
bytes: row.bytes,
|
||||
uniqueCount: row.uniqueCount,
|
||||
metadata: JSON.stringify({
|
||||
version: ROLLUP_VERSION,
|
||||
scope: 'snapshots',
|
||||
quality: 'exact',
|
||||
...(row.metric === M.statsRollupRun
|
||||
? { snapshotQuality: 'exact', snapshotObservedAt: observedAt.toISOString() }
|
||||
: { observedAt: observedAt.toISOString() }),
|
||||
}),
|
||||
updatedAt: observedAt,
|
||||
}))
|
||||
const writes: AtomicQuery[] = [
|
||||
db
|
||||
.delete(statsRollupsHourly)
|
||||
.where(
|
||||
and(
|
||||
eq(statsRollupsHourly.bucketStart, bucketStart),
|
||||
inArray(statsRollupsHourly.metricKey, [...GAUGE_METRICS, M.statsRollupRun]),
|
||||
),
|
||||
),
|
||||
]
|
||||
for (let index = 0; index < rows.length; index += ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE) {
|
||||
writes.push(db.insert(statsRollupsHourly).values(rows.slice(index, index + ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE)))
|
||||
}
|
||||
await executeWriteTransaction(db, writes)
|
||||
return { bucketStart, bucketEnd, rows: rows.length, lowerBoundRows: 0 }
|
||||
}
|
||||
|
||||
async function compatibleSnapshotMarker(
|
||||
db: Database,
|
||||
bucketStart: Date,
|
||||
): Promise<{ quality: 'exact' | 'lower_bound'; observedAt: string } | null> {
|
||||
const rows = await db
|
||||
.select({ metadata: statsRollupsHourly.metadata })
|
||||
.from(statsRollupsHourly)
|
||||
.where(
|
||||
and(
|
||||
eq(statsRollupsHourly.bucketStart, bucketStart),
|
||||
eq(statsRollupsHourly.orgId, ''),
|
||||
eq(statsRollupsHourly.metricKey, M.statsRollupRun),
|
||||
eq(statsRollupsHourly.dimensionKey, ''),
|
||||
eq(statsRollupsHourly.dimensionValue, ''),
|
||||
),
|
||||
)
|
||||
.limit(1)
|
||||
const metadata = parseAdminStatsRollupMetadata(rows[0]?.metadata ?? null)
|
||||
if ((metadata?.scope !== 'snapshots' && metadata?.scope !== 'full') || !metadata.snapshotObservedAt) return null
|
||||
return { quality: metadata.snapshotQuality ?? metadata.quality, observedAt: metadata.snapshotObservedAt }
|
||||
}
|
||||
|
||||
async function addEventMetrics(db: Database, rollups: RollupAccumulator, from: Date, to: Date): Promise<void> {
|
||||
const rows = await db.all<EventMetricGroup>(sql`
|
||||
WITH facts AS (
|
||||
@@ -241,70 +352,90 @@ type EventMetricGroup = {
|
||||
}
|
||||
|
||||
async function addUserMetrics(db: Database, rollups: RollupAccumulator, from: Date, to: Date): Promise<void> {
|
||||
const [signupRows, shareRows] = await Promise.all([
|
||||
db.all<{ provider: string; count: number }>(sql`
|
||||
SELECT provider, COUNT(*) AS count
|
||||
FROM (
|
||||
SELECT COALESCE((
|
||||
SELECT a.provider_id FROM account a
|
||||
WHERE a.user_id = u.id
|
||||
ORDER BY a.created_at, a.id
|
||||
LIMIT 1
|
||||
), 'direct') AS provider
|
||||
FROM "user" u
|
||||
WHERE u.created_at >= ${from.getTime()} AND u.created_at < ${to.getTime()}
|
||||
) signups
|
||||
GROUP BY provider
|
||||
`),
|
||||
db
|
||||
.select({ orgId: shares.orgId, kind: shares.kind, count: sql<number>`COUNT(*)` })
|
||||
.from(shares)
|
||||
.where(and(gte(shares.createdAt, from), lt(shares.createdAt, to)))
|
||||
.groupBy(shares.orgId, shares.kind),
|
||||
])
|
||||
const rows = await db.all<{
|
||||
action: string
|
||||
orgId: string
|
||||
provider: string | null
|
||||
kind: string | null
|
||||
count: number
|
||||
lowerBound: number
|
||||
}>(sql`
|
||||
SELECT
|
||||
action,
|
||||
org_id AS orgId,
|
||||
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.provider') END AS provider,
|
||||
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.kind') END AS kind,
|
||||
COUNT(*) AS count,
|
||||
SUM(CASE WHEN json_valid(metadata) = 1 AND json_extract(metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) AS lowerBound
|
||||
FROM activity_events
|
||||
WHERE created_at >= ${Math.floor(from.getTime() / 1000)}
|
||||
AND created_at < ${Math.floor(to.getTime() / 1000)}
|
||||
AND action IN ('stats_user_signup', 'stats_share_created')
|
||||
GROUP BY action, org_id, provider, kind
|
||||
`)
|
||||
|
||||
for (const row of signupRows) rollups.add(M.userSignup, '', Number(row.count), 0, { provider: row.provider })
|
||||
for (const row of shareRows) rollups.add(M.shareCreated, row.orgId, Number(row.count), 0, { kind: row.kind })
|
||||
for (const row of rows) {
|
||||
if (row.action === 'stats_user_signup') {
|
||||
rollups.add(M.userSignup, '', Number(row.count), 0, { provider: row.provider ?? 'unknown' }, row.lowerBound > 0)
|
||||
} else {
|
||||
rollups.add(M.shareCreated, row.orgId, Number(row.count), 0, { kind: row.kind ?? 'unknown' }, row.lowerBound > 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function addOperationalMetrics(db: Database, rollups: RollupAccumulator, from: Date, to: Date): Promise<void> {
|
||||
const [taskFinishedRows, jobRows] = await Promise.all([
|
||||
db
|
||||
.select({
|
||||
orgId: downloadTasks.orgId,
|
||||
category: downloadTasks.category,
|
||||
downloaderId: downloadTasks.assignedDownloaderId,
|
||||
status: downloadTasks.status,
|
||||
count: sql<number>`COUNT(*)`,
|
||||
bytes: sql<number>`COALESCE(SUM(${downloadTasks.billingChargedBytes}), 0)`,
|
||||
})
|
||||
.from(downloadTasks)
|
||||
.where(and(gte(downloadTasks.finishedAt, from), lt(downloadTasks.finishedAt, to)))
|
||||
.groupBy(downloadTasks.orgId, downloadTasks.category, downloadTasks.assignedDownloaderId, downloadTasks.status),
|
||||
db
|
||||
.select({
|
||||
orgId: backgroundJobs.orgId,
|
||||
type: backgroundJobs.type,
|
||||
status: backgroundJobs.status,
|
||||
count: sql<number>`COUNT(*)`,
|
||||
})
|
||||
.from(backgroundJobs)
|
||||
.where(and(gte(backgroundJobs.finishedAt, from), lt(backgroundJobs.finishedAt, to)))
|
||||
.groupBy(backgroundJobs.orgId, backgroundJobs.type, backgroundJobs.status),
|
||||
])
|
||||
const rows = await db.all<{
|
||||
action: string
|
||||
orgId: string
|
||||
category: string | null
|
||||
downloaderId: string | null
|
||||
jobType: string | null
|
||||
outcome: string | null
|
||||
count: number
|
||||
bytes: number
|
||||
lowerBound: number
|
||||
}>(sql`
|
||||
SELECT
|
||||
action,
|
||||
org_id AS orgId,
|
||||
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.category') END AS category,
|
||||
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.downloaderId') END AS downloaderId,
|
||||
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.jobType') END AS jobType,
|
||||
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.outcome') END AS outcome,
|
||||
COUNT(*) AS count,
|
||||
SUM(CASE WHEN json_valid(metadata) = 1 THEN COALESCE(json_extract(metadata, '$.bytes'), 0) ELSE 0 END) AS bytes,
|
||||
SUM(CASE WHEN json_valid(metadata) = 1 AND json_extract(metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) AS lowerBound
|
||||
FROM activity_events
|
||||
WHERE created_at >= ${Math.floor(from.getTime() / 1000)}
|
||||
AND created_at < ${Math.floor(to.getTime() / 1000)}
|
||||
AND action IN ('stats_background_job_finished', 'stats_remote_download_finished')
|
||||
GROUP BY action, org_id, category, downloaderId, jobType, outcome
|
||||
`)
|
||||
|
||||
for (const row of taskFinishedRows) {
|
||||
rollups.add(M.remoteDownloadTaskFinished, row.orgId, Number(row.count), Number(row.bytes), {
|
||||
category: row.category ?? 'uncategorized',
|
||||
downloader_id: row.downloaderId,
|
||||
outcome: row.status,
|
||||
})
|
||||
}
|
||||
for (const row of jobRows) {
|
||||
rollups.add(M.backgroundJobFinished, row.orgId, Number(row.count), 0, {
|
||||
job_type: row.type,
|
||||
outcome: row.status,
|
||||
})
|
||||
for (const row of rows) {
|
||||
if (row.action === 'stats_remote_download_finished') {
|
||||
rollups.add(
|
||||
M.remoteDownloadTaskFinished,
|
||||
row.orgId,
|
||||
Number(row.count),
|
||||
Number(row.bytes),
|
||||
{
|
||||
category: row.category ?? 'uncategorized',
|
||||
downloader_id: row.downloaderId,
|
||||
outcome: row.outcome ?? 'unknown',
|
||||
},
|
||||
row.lowerBound > 0,
|
||||
)
|
||||
} else {
|
||||
rollups.add(
|
||||
M.backgroundJobFinished,
|
||||
row.orgId,
|
||||
Number(row.count),
|
||||
0,
|
||||
{ job_type: row.jobType ?? 'unknown', outcome: row.outcome ?? 'unknown' },
|
||||
row.lowerBound > 0,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -312,12 +443,14 @@ async function addSnapshotMetrics(db: Database, rollups: RollupAccumulator, now:
|
||||
const [
|
||||
userRows,
|
||||
activeUserRows,
|
||||
quotaRows,
|
||||
quotaBaseRows,
|
||||
inventoryRows,
|
||||
inventoryByType,
|
||||
inventoryBySize,
|
||||
inventoryByAge,
|
||||
trashRows,
|
||||
shareRows,
|
||||
dataQualityRows,
|
||||
jobRows,
|
||||
taskRows,
|
||||
downloaderRows,
|
||||
@@ -330,17 +463,20 @@ async function addSnapshotMetrics(db: Database, rollups: RollupAccumulator, now:
|
||||
'quota',
|
||||
db
|
||||
.select({
|
||||
orgId: orgQuotas.orgId,
|
||||
used: orgQuotas.used,
|
||||
quota: orgQuotas.quota,
|
||||
orgId: organization.id,
|
||||
quotaId: orgQuotas.id,
|
||||
used: sql<number>`COALESCE(${orgQuotas.used}, 0)`,
|
||||
})
|
||||
.from(orgQuotas),
|
||||
.from(organization)
|
||||
.leftJoin(orgQuotas, eq(orgQuotas.orgId, organization.id)),
|
||||
),
|
||||
queryStage('inventory-base', inventoryGroups(db, now, 'base')),
|
||||
queryStage('inventory-type', inventoryGroups(db, now, 'file_type_group')),
|
||||
queryStage('inventory-size', inventoryGroups(db, now, 'size_bucket')),
|
||||
queryStage('inventory-age', inventoryGroups(db, now, 'age_bucket')),
|
||||
queryStage('trash-inventory', trashInventoryGroups(db)),
|
||||
queryStage('shares', shareSnapshotGroups(db, now)),
|
||||
queryStage('data-quality', dataQualitySnapshot(db)),
|
||||
queryStage(
|
||||
'jobs',
|
||||
db
|
||||
@@ -394,30 +530,72 @@ async function addSnapshotMetrics(db: Database, rollups: RollupAccumulator, now:
|
||||
),
|
||||
])
|
||||
|
||||
const effectiveQuotas = await queryStage(
|
||||
'effective-quota',
|
||||
getEffectiveQuotasByOrg(
|
||||
db,
|
||||
quotaBaseRows.map((row) => row.orgId),
|
||||
now,
|
||||
),
|
||||
)
|
||||
|
||||
rollups.setGauge(M.userInventory, '', userRows.total, 0)
|
||||
rollups.setGauge(M.userInventory, '', userRows.normal, 0, 'status', 'normal')
|
||||
rollups.setGauge(M.userInventory, '', userRows.verified, 0, 'status', 'verified')
|
||||
rollups.setGauge(M.userInventory, '', userRows.unverified, 0, 'status', 'unverified')
|
||||
rollups.setGauge(M.userInventory, '', userRows.banned, 0, 'status', 'banned')
|
||||
rollups.setGauge(M.userInventory, '', userRows.silent, 0, 'status', 'silent')
|
||||
rollups.setGauge(M.userActiveSnapshot, '', activeUserRows.find((row) => row.window === 'mau')?.count ?? 0, 0)
|
||||
for (const row of activeUserRows) rollups.setGauge(M.userActiveSnapshot, '', row.count, 0, 'window', row.window)
|
||||
for (const row of quotaRows) {
|
||||
rollups.setGauge(M.storageUsed, '', 0, 0)
|
||||
rollups.setGauge(M.storageQuota, '', 0, 0)
|
||||
rollups.setGauge(M.storageInventory, '', 0, 0)
|
||||
rollups.setGauge(M.storageTrashSnapshot, '', 0, 0)
|
||||
rollups.setGauge(M.shareInventory, '', 0, 0)
|
||||
rollups.setGauge(M.statsDataQualitySnapshot, '', dataQualityRows.unlocatedShareEvents, 0)
|
||||
rollups.setGauge(M.statsDataQualitySnapshot, '', dataQualityRows.unlocatedShareViews, 0, 'kind', 'share_views')
|
||||
rollups.setGauge(
|
||||
M.statsDataQualitySnapshot,
|
||||
'',
|
||||
dataQualityRows.unlocatedShareDownloads,
|
||||
0,
|
||||
'kind',
|
||||
'share_downloads',
|
||||
)
|
||||
rollups.setGauge(
|
||||
M.statsDataQualitySnapshot,
|
||||
'',
|
||||
dataQualityRows.storageUsageDriftSpaces,
|
||||
dataQualityRows.storageUsageDriftBytes,
|
||||
'kind',
|
||||
'storage_usage_drift',
|
||||
)
|
||||
rollups.setGauge(M.backgroundJobSnapshot, '', 0, 0)
|
||||
rollups.setGauge(M.remoteDownloadTaskSnapshot, '', 0, 0)
|
||||
rollups.setGauge(M.downloaderSnapshot, '', 0, 0)
|
||||
rollups.setGauge(M.trafficReportSnapshot, '', 0, 0)
|
||||
rollups.setGauge(M.webhookSnapshot, '', 0, 0)
|
||||
for (const row of quotaBaseRows) {
|
||||
const effective = effectiveQuotas.get(row.orgId)
|
||||
if (!effective) throw new Error(`stats_effective_quota_missing:${row.orgId}`)
|
||||
const quota = row.quotaId ? effective.quota : 0
|
||||
rollups.setGauge(M.storageUsed, row.orgId, 0, row.used)
|
||||
rollups.setGauge(M.storageQuota, row.orgId, 0, row.quota)
|
||||
rollups.setGauge(M.storageQuota, row.orgId, 0, quota)
|
||||
rollups.incrementGauge(M.storageUsed, '', 0, row.used)
|
||||
rollups.incrementGauge(M.storageQuota, '', 1, row.quota)
|
||||
const status =
|
||||
row.quota > 0 && row.used >= row.quota
|
||||
? 'over'
|
||||
: row.quota > 0 && row.used >= row.quota * 0.8
|
||||
? 'near'
|
||||
: 'healthy'
|
||||
rollups.incrementGauge(M.storageQuota, '', 1, quota)
|
||||
const status = quota <= 0 ? 'invalid' : row.used >= quota ? 'over' : row.used >= quota * 0.8 ? 'near' : 'healthy'
|
||||
rollups.incrementGauge(M.storageQuota, '', 1, 0, 'status', status)
|
||||
}
|
||||
for (const row of [...inventoryRows, ...inventoryByType, ...inventoryBySize, ...inventoryByAge]) {
|
||||
rollups.incrementGauge(M.storageInventory, row.orgId, row.files, row.bytes, row.dimensionKey, row.dimensionValue)
|
||||
rollups.incrementGauge(M.storageInventory, '', row.files, row.bytes, row.dimensionKey, row.dimensionValue)
|
||||
}
|
||||
for (const row of trashRows) {
|
||||
rollups.incrementGauge(M.storageTrashSnapshot, row.orgId, row.files, row.bytes)
|
||||
rollups.incrementGauge(M.storageTrashSnapshot, '', row.files, row.bytes)
|
||||
rollups.incrementGauge(M.storageTrashSnapshot, row.orgId, row.files, row.bytes, 'storage_id', row.storageId)
|
||||
rollups.incrementGauge(M.storageTrashSnapshot, '', row.files, row.bytes, 'storage_id', row.storageId)
|
||||
}
|
||||
for (const row of shareRows) {
|
||||
rollups.incrementGauge(M.shareInventory, row.orgId, Number(row.count), 0, 'lifecycle', row.lifecycle)
|
||||
rollups.incrementGauge(M.shareInventory, '', Number(row.count), 0, 'lifecycle', row.lifecycle)
|
||||
@@ -461,6 +639,89 @@ async function shareSnapshotGroups(
|
||||
.groupBy(shares.orgId, lifecycle)
|
||||
}
|
||||
|
||||
async function dataQualitySnapshot(db: Database): Promise<{
|
||||
unlocatedShareEvents: number
|
||||
unlocatedShareViews: number
|
||||
unlocatedShareDownloads: number
|
||||
storageUsageDriftSpaces: number
|
||||
storageUsageDriftBytes: number
|
||||
}> {
|
||||
const rows = await db.all<{
|
||||
unlocatedShareViews: number
|
||||
unlocatedShareDownloads: number
|
||||
storageUsageDriftSpaces: number
|
||||
storageUsageDriftBytes: number
|
||||
}>(sql`
|
||||
WITH event_counts AS (
|
||||
SELECT
|
||||
COALESCE(CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.shareId') END, target_id) AS share_id,
|
||||
SUM(CASE WHEN action = 'share_view' THEN 1 ELSE 0 END) AS views,
|
||||
SUM(CASE WHEN action = 'share_download' THEN 1 ELSE 0 END) AS downloads
|
||||
FROM activity_events
|
||||
WHERE action IN ('share_view', 'share_download')
|
||||
GROUP BY share_id
|
||||
),
|
||||
billable_storage AS (
|
||||
SELECT org_id, SUM(bytes) AS bytes
|
||||
FROM (
|
||||
SELECT org_id, COALESCE(size, 0) AS bytes
|
||||
FROM matters
|
||||
WHERE status IN ('active', 'trashed') AND dirtype = 0
|
||||
UNION ALL
|
||||
SELECT org_id, COALESCE(size, 0) AS bytes
|
||||
FROM image_hostings
|
||||
WHERE status = 'active'
|
||||
) inventory
|
||||
GROUP BY org_id
|
||||
),
|
||||
storage_drift AS (
|
||||
SELECT ABS(q.used - COALESCE(billable_storage.bytes, 0)) AS bytes
|
||||
FROM org_quotas q
|
||||
JOIN organization o ON o.id = q.org_id
|
||||
LEFT JOIN billable_storage ON billable_storage.org_id = q.org_id
|
||||
WHERE q.used <> COALESCE(billable_storage.bytes, 0)
|
||||
)
|
||||
SELECT
|
||||
COALESCE(SUM(MAX(${shares.views} - COALESCE(event_counts.views, 0), 0)), 0) AS unlocatedShareViews,
|
||||
COALESCE(SUM(MAX(${shares.downloads} - COALESCE(event_counts.downloads, 0), 0)), 0) AS unlocatedShareDownloads,
|
||||
(SELECT COUNT(*) FROM storage_drift) AS storageUsageDriftSpaces,
|
||||
(SELECT COALESCE(SUM(bytes), 0) FROM storage_drift) AS storageUsageDriftBytes
|
||||
FROM ${shares}
|
||||
LEFT JOIN event_counts ON event_counts.share_id = ${shares.id}
|
||||
`)
|
||||
const unlocatedShareViews = Number(rows[0]?.unlocatedShareViews ?? 0)
|
||||
const unlocatedShareDownloads = Number(rows[0]?.unlocatedShareDownloads ?? 0)
|
||||
return {
|
||||
unlocatedShareEvents: unlocatedShareViews + unlocatedShareDownloads,
|
||||
unlocatedShareViews,
|
||||
unlocatedShareDownloads,
|
||||
storageUsageDriftSpaces: Number(rows[0]?.storageUsageDriftSpaces ?? 0),
|
||||
storageUsageDriftBytes: Number(rows[0]?.storageUsageDriftBytes ?? 0),
|
||||
}
|
||||
}
|
||||
|
||||
async function trashInventoryGroups(
|
||||
db: Database,
|
||||
): Promise<Array<{ orgId: string; storageId: string; files: number; bytes: number }>> {
|
||||
const rows = await db
|
||||
.select({
|
||||
orgId: matters.orgId,
|
||||
storageId: matters.storageId,
|
||||
files: sql<number>`COUNT(*)`,
|
||||
bytes: sql<number>`COALESCE(SUM(${matters.size}), 0)`,
|
||||
})
|
||||
.from(matters)
|
||||
.where(
|
||||
and(
|
||||
inArray(matters.status, ['active', 'trashed']),
|
||||
sql`${matters.trashedAt} IS NOT NULL`,
|
||||
eq(matters.dirtype, 0),
|
||||
),
|
||||
)
|
||||
.groupBy(matters.orgId, matters.storageId)
|
||||
return rows.map((row) => ({ ...row, files: Number(row.files), bytes: Number(row.bytes) }))
|
||||
}
|
||||
|
||||
type UserInventorySnapshot = {
|
||||
total: number
|
||||
normal: number
|
||||
@@ -550,7 +811,7 @@ async function inventoryGroups(
|
||||
dimensionValue: dimensionSql,
|
||||
})
|
||||
.from(matters)
|
||||
.where(and(eq(matters.status, 'active'), eq(matters.dirtype, 0)))
|
||||
.where(and(eq(matters.status, 'active'), isNull(matters.trashedAt), eq(matters.dirtype, 0)))
|
||||
.groupBy(matters.orgId, dimensionSql)
|
||||
const imageDimensionSql =
|
||||
dimension === 'file_type_group'
|
||||
|
||||
@@ -6,7 +6,9 @@ import type {
|
||||
AdminDashboardSharingStats,
|
||||
AdminDashboardStorageStats,
|
||||
AdminDashboardTrafficStats,
|
||||
AdminSharingDataQuality,
|
||||
AdminStatsDelta,
|
||||
AdminStorageDataQuality,
|
||||
AdminTopShare,
|
||||
AdminTransferDataQuality,
|
||||
} from '@shared/types'
|
||||
@@ -23,7 +25,7 @@ import { addCalendarDays, statsDayKey as dayKey, utcDateStart } from '../../doma
|
||||
import type { Database } from '../../platform/interface'
|
||||
import type { AdminStatsDateRange, AdminStatsRepo } from '../../usecases/ports'
|
||||
import { AdminStatsHourlyReader } from './admin-stats-hourly'
|
||||
import { rebuildAdminStatsHour } from './admin-stats-rollup'
|
||||
import { captureAdminStatsSnapshot, rebuildAdminStatsHour } from './admin-stats-rollup'
|
||||
|
||||
const DOWNLOAD_ACTIVITY_ACTIONS = ['share_download', 'object_download', 'image_hosting_download', 'webdav_download']
|
||||
const DOWNLOAD_FAILURE_ACTION = 'download_failed'
|
||||
@@ -40,7 +42,8 @@ export function createAdminStatsRepo(db: Database): AdminStatsRepo {
|
||||
}
|
||||
|
||||
async function refreshHourlyRollups(db: Database, now: Date) {
|
||||
const latestClosedHour = new Date(startOfHour(now).getTime() - 3_600_000)
|
||||
const currentHour = startOfHour(now)
|
||||
const latestClosedHour = new Date(currentHour.getTime() - 3_600_000)
|
||||
const repairFrom = new Date(latestClosedHour.getTime() - 47 * 3_600_000)
|
||||
const markers = await db
|
||||
.select({ bucketStart: statsRollupsHourly.bucketStart, metadata: statsRollupsHourly.metadata })
|
||||
@@ -56,7 +59,10 @@ async function refreshHourlyRollups(db: Database, now: Date) {
|
||||
)
|
||||
const completed = new Set(
|
||||
markers
|
||||
.filter((row) => parseAdminStatsRollupMetadata(row.metadata) !== null)
|
||||
.filter((row) => {
|
||||
const scope = parseAdminStatsRollupMetadata(row.metadata)?.scope
|
||||
return scope === 'counters' || scope === 'full'
|
||||
})
|
||||
.map((row) => row.bucketStart.getTime()),
|
||||
)
|
||||
const repairTargets: Date[] = []
|
||||
@@ -64,10 +70,11 @@ async function refreshHourlyRollups(db: Database, now: Date) {
|
||||
if (!completed.has(at)) repairTargets.push(new Date(at))
|
||||
if (repairTargets.length === 3) break
|
||||
}
|
||||
const latest = await rebuildAdminStatsHour(db, latestClosedHour, now, true)
|
||||
const latest = await rebuildAdminStatsHour(db, latestClosedHour, now)
|
||||
const repaired = []
|
||||
for (const bucketStart of repairTargets) repaired.push(await rebuildAdminStatsHour(db, bucketStart, now, false))
|
||||
return [latest, ...repaired]
|
||||
for (const bucketStart of repairTargets) repaired.push(await rebuildAdminStatsHour(db, bucketStart, now))
|
||||
const snapshot = await captureAdminStatsSnapshot(db, currentHour, now)
|
||||
return [latest, ...repaired, snapshot]
|
||||
}
|
||||
|
||||
async function getDashboardOverviewStats(
|
||||
@@ -75,8 +82,9 @@ async function getDashboardOverviewStats(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<AdminDashboardOverviewStats> {
|
||||
const previous = previousRange(range)
|
||||
const reader = new AdminStatsHourlyReader(db, range, now)
|
||||
const effective = effectiveRange(range, now)
|
||||
const previous = previousRange(effective)
|
||||
const reader = new AdminStatsHourlyReader(db, effective, now)
|
||||
const previousReader = new AdminStatsHourlyReader(db, previous, now)
|
||||
const [
|
||||
users,
|
||||
@@ -89,9 +97,13 @@ async function getDashboardOverviewStats(
|
||||
previousTraffic,
|
||||
sharing,
|
||||
previousSharing,
|
||||
sharingDataQuality,
|
||||
previousSharingDataQuality,
|
||||
dataQuality,
|
||||
coverage,
|
||||
comparisonCoverage,
|
||||
snapshotCoverage,
|
||||
comparisonSnapshotCoverage,
|
||||
] = await Promise.all([
|
||||
getUserInventory(reader),
|
||||
getSignupTotal(reader),
|
||||
@@ -103,9 +115,13 @@ async function getDashboardOverviewStats(
|
||||
getTrafficTotals(previousReader),
|
||||
getSharingEventTotals(reader),
|
||||
getSharingComparisonTotals(previousReader),
|
||||
getSharingDataQuality(reader),
|
||||
getSharingDataQuality(previousReader),
|
||||
getTransferDataQuality(reader, previousReader),
|
||||
reader.coverage(),
|
||||
previousReader.coverage(),
|
||||
reader.coverage('counters'),
|
||||
previousReader.coverage('counters'),
|
||||
reader.coverage('snapshots'),
|
||||
previousReader.coverage('snapshots'),
|
||||
])
|
||||
const [trendNewUsers, activeByDay, storageUsedByDay, uploadByDay, downloadByDay] = await Promise.all([
|
||||
getSignupsByDay(reader),
|
||||
@@ -114,37 +130,51 @@ async function getDashboardOverviewStats(
|
||||
getActivityMetricByDay(reader, metricSpec(['upload_confirm']), 'bytes'),
|
||||
getActivityMetricByDay(reader, metricSpec(DOWNLOAD_ACTIVITY_ACTIONS), 'bytes'),
|
||||
])
|
||||
const trends = createDateBuckets(range).map((date) => {
|
||||
const trends = createDateBuckets(effective).map((date) => {
|
||||
return {
|
||||
date,
|
||||
newUsers: trendNewUsers.get(date) ?? 0,
|
||||
activeUsers: activeByDay.get(date) ?? 0,
|
||||
activeUsers: activeByDay.get(date) ?? null,
|
||||
storageUsedBytes: storageUsedByDay.get(date) ?? null,
|
||||
uploadBytes: uploadByDay.get(date) ?? 0,
|
||||
downloadBytes: downloadByDay.get(date) ?? 0,
|
||||
}
|
||||
})
|
||||
const sharingComparable =
|
||||
comparable(coverage, comparisonCoverage) &&
|
||||
hasExactSharingHistory(sharingDataQuality) &&
|
||||
hasExactSharingHistory(previousSharingDataQuality)
|
||||
const validQuotaBytes = quotas && quotas.invalidQuotaSpaces === 0 ? quotas.quotaBytes : null
|
||||
|
||||
return {
|
||||
...statsFrame(now, range, coverage, comparisonCoverage),
|
||||
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
|
||||
dataQuality,
|
||||
totals: {
|
||||
users: users.total,
|
||||
newUsers: delta(newUsers, previousNewUsers),
|
||||
activeUsers: delta(activeUsers.mau, previousActiveUsers.mau),
|
||||
activeUserRate: nullablePercent(activeUsers.mau, users.total),
|
||||
storageUsedBytes: quotas.usedBytes,
|
||||
storageQuotaBytes: quotas.quotaBytes,
|
||||
storageUtilization: nullablePercent(quotas.usedBytes, quotas.quotaBytes),
|
||||
users: users?.total ?? null,
|
||||
newUsers: delta(newUsers, previousNewUsers, comparable(coverage, comparisonCoverage)),
|
||||
activeUsers: delta(
|
||||
activeUsers?.mau ?? null,
|
||||
previousActiveUsers?.mau ?? null,
|
||||
comparable(snapshotCoverage, comparisonSnapshotCoverage),
|
||||
),
|
||||
activeUserRate: nullablePercent(activeUsers?.mau ?? null, users?.total ?? null),
|
||||
storageUsedBytes: quotas?.usedBytes ?? null,
|
||||
storageQuotaBytes: validQuotaBytes,
|
||||
storageUtilization: nullablePercent(quotas?.usedBytes ?? null, validQuotaBytes),
|
||||
trafficBytes: delta(
|
||||
traffic.uploadBytes + traffic.downloadBytes,
|
||||
previousTraffic.uploadBytes + previousTraffic.downloadBytes,
|
||||
comparable(coverage, comparisonCoverage),
|
||||
),
|
||||
uploadBytes: delta(traffic.uploadBytes, previousTraffic.uploadBytes, comparable(coverage, comparisonCoverage)),
|
||||
downloadBytes: delta(
|
||||
traffic.downloadBytes,
|
||||
previousTraffic.downloadBytes,
|
||||
comparable(coverage, comparisonCoverage),
|
||||
),
|
||||
uploadBytes: delta(traffic.uploadBytes, previousTraffic.uploadBytes),
|
||||
downloadBytes: delta(traffic.downloadBytes, previousTraffic.downloadBytes),
|
||||
activeShares: sharing.activeShares,
|
||||
shareViews: delta(sharing.views, previousSharing.views),
|
||||
shareDownloads: delta(sharing.downloads, previousSharing.downloads),
|
||||
shareViews: delta(sharing.views, previousSharing.views, sharingComparable),
|
||||
shareDownloads: delta(sharing.downloads, previousSharing.downloads, sharingComparable),
|
||||
},
|
||||
trends,
|
||||
}
|
||||
@@ -155,7 +185,8 @@ async function getDashboardOperationsStats(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<AdminDashboardOperationsStats> {
|
||||
const reader = new AdminStatsHourlyReader(db, range, now)
|
||||
const effective = effectiveRange(range, now)
|
||||
const reader = new AdminStatsHourlyReader(db, effective, now)
|
||||
const [
|
||||
activeBackgroundJobs,
|
||||
activeRemoteDownloads,
|
||||
@@ -168,6 +199,7 @@ async function getDashboardOperationsStats(
|
||||
backgroundJobsByDay,
|
||||
remoteDownloadsByDay,
|
||||
coverage,
|
||||
snapshotCoverage,
|
||||
] = await Promise.all([
|
||||
getLatestGaugeTotal(reader, ADMIN_STATS_METRICS.backgroundJobSnapshot),
|
||||
getLatestGaugeTotal(reader, ADMIN_STATS_METRICS.remoteDownloadTaskSnapshot),
|
||||
@@ -179,7 +211,8 @@ async function getDashboardOperationsStats(
|
||||
getCloudReportOutcomes(reader),
|
||||
getOperationalOutcomesByDay(reader, 'background_job'),
|
||||
getOperationalOutcomesByDay(reader, 'remote_download'),
|
||||
reader.coverage(),
|
||||
reader.coverage('counters'),
|
||||
reader.coverage('snapshots'),
|
||||
])
|
||||
const completedJobs = backgroundJobOutcomes.get('completed') ?? 0
|
||||
const failedJobs = backgroundJobOutcomes.get('failed') ?? 0
|
||||
@@ -187,12 +220,14 @@ async function getDashboardOperationsStats(
|
||||
const failedRemoteDownloads = remoteDownloadOutcomes.get('failed') ?? 0
|
||||
|
||||
return {
|
||||
...statsFrame(now, range, coverage),
|
||||
...statsFrame(now, effective, coverage, undefined, snapshotCoverage),
|
||||
summary: {
|
||||
activeBackgroundJobs,
|
||||
activeRemoteDownloads,
|
||||
onlineDownloaders: downloaderStatus.get('online') ?? 0,
|
||||
offlineDownloaders: (downloaderStatus.get('offline') ?? 0) + (downloaderStatus.get('disabled') ?? 0),
|
||||
onlineDownloaders: downloaderStatus?.get('online') ?? null,
|
||||
offlineDownloaders: downloaderStatus
|
||||
? (downloaderStatus.get('offline') ?? 0) + (downloaderStatus.get('disabled') ?? 0)
|
||||
: null,
|
||||
backgroundJobFailureRate: nullablePercent(failedJobs, completedJobs + failedJobs),
|
||||
remoteDownloadSuccessRate: nullablePercent(
|
||||
completedRemoteDownloads,
|
||||
@@ -200,9 +235,13 @@ async function getDashboardOperationsStats(
|
||||
),
|
||||
cloudReportBacklog,
|
||||
webhookFailures,
|
||||
alertCount: cloudReportBacklog + webhookFailures,
|
||||
cloudReportDeadLetters: cloudReportStatus?.get('dead_letter') ?? null,
|
||||
alertCount:
|
||||
cloudReportBacklog === null || webhookFailures === null
|
||||
? null
|
||||
: cloudReportBacklog + webhookFailures + (cloudReportStatus?.get('dead_letter') ?? 0),
|
||||
},
|
||||
trend: createDateBuckets(range).map((date) => ({
|
||||
trend: createDateBuckets(effective).map((date) => ({
|
||||
date,
|
||||
completedJobs: backgroundJobsByDay.get(date)?.get('completed') ?? 0,
|
||||
failedJobs: backgroundJobsByDay.get(date)?.get('failed') ?? 0,
|
||||
@@ -211,8 +250,8 @@ async function getDashboardOperationsStats(
|
||||
})),
|
||||
backgroundJobOutcomes: percentRows([...backgroundJobOutcomes].map(([name, value]) => ({ name, value }))),
|
||||
remoteDownloadOutcomes: percentRows([...remoteDownloadOutcomes].map(([name, value]) => ({ name, value }))),
|
||||
downloaderStatus: percentRows([...downloaderStatus].map(([name, value]) => ({ name, value }))),
|
||||
cloudReportStatus: percentRows([...cloudReportStatus].map(([name, value]) => ({ name, value }))),
|
||||
downloaderStatus: percentRows([...(downloaderStatus ?? new Map())].map(([name, value]) => ({ name, value }))),
|
||||
cloudReportStatus: percentRows([...(cloudReportStatus ?? new Map())].map(([name, value]) => ({ name, value }))),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -221,8 +260,9 @@ async function getDashboardGrowthStats(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<AdminDashboardGrowthStats> {
|
||||
const previous = previousRange(range)
|
||||
const reader = new AdminStatsHourlyReader(db, range, now)
|
||||
const effective = effectiveRange(range, now)
|
||||
const previous = previousRange(effective)
|
||||
const reader = new AdminStatsHourlyReader(db, effective, now)
|
||||
const previousReader = new AdminStatsHourlyReader(db, previous, now)
|
||||
const [
|
||||
users,
|
||||
@@ -235,45 +275,55 @@ async function getDashboardGrowthStats(
|
||||
totalsByDay,
|
||||
coverage,
|
||||
comparisonCoverage,
|
||||
snapshotCoverage,
|
||||
comparisonSnapshotCoverage,
|
||||
] = await Promise.all([
|
||||
getUserInventory(reader),
|
||||
getSignupTotal(reader),
|
||||
getSignupTotal(previousReader),
|
||||
getActiveUserSnapshot(reader),
|
||||
getActiveUserSnapshot(previousReader),
|
||||
getRollingActiveUserTrend(reader, range),
|
||||
getRollingActiveUserTrend(reader, effective),
|
||||
getRegistrationSources(reader),
|
||||
getUserTotalsByDay(reader),
|
||||
reader.coverage(),
|
||||
previousReader.coverage(),
|
||||
reader.coverage('counters'),
|
||||
previousReader.coverage('counters'),
|
||||
reader.coverage('snapshots'),
|
||||
previousReader.coverage('snapshots'),
|
||||
])
|
||||
const newUsersByDay = await getSignupsByDay(reader)
|
||||
const userScaleTrend = createDateBuckets(range).map((date) => ({
|
||||
const userScaleTrend = createDateBuckets(effective).map((date) => ({
|
||||
date,
|
||||
newUsers: newUsersByDay.get(date) ?? 0,
|
||||
totalUsers: totalsByDay.get(date) ?? 0,
|
||||
totalUsers: totalsByDay.get(date) ?? null,
|
||||
}))
|
||||
|
||||
return {
|
||||
...statsFrame(now, range, coverage, comparisonCoverage),
|
||||
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
|
||||
summary: {
|
||||
totalUsers: users.total,
|
||||
newUsers: delta(newUsers, previousNewUsers),
|
||||
activeUsers: delta(activeUsers.mau, previousActiveUsers.mau),
|
||||
verifiedUsers: users.verified,
|
||||
bannedUsers: users.banned,
|
||||
silentUsers: users.silent,
|
||||
activeUserRate: nullablePercent(activeUsers.mau, users.total),
|
||||
silentUserRate: nullablePercent(users.silent, users.total),
|
||||
totalUsers: users?.total ?? null,
|
||||
newUsers: delta(newUsers, previousNewUsers, comparable(coverage, comparisonCoverage)),
|
||||
activeUsers: delta(
|
||||
activeUsers?.mau ?? null,
|
||||
previousActiveUsers?.mau ?? null,
|
||||
comparable(snapshotCoverage, comparisonSnapshotCoverage),
|
||||
),
|
||||
verifiedUsers: users?.verified ?? null,
|
||||
bannedUsers: users?.banned ?? null,
|
||||
silentUsers: users?.silent ?? null,
|
||||
activeUserRate: nullablePercent(activeUsers?.mau ?? null, users?.total ?? null),
|
||||
silentUserRate: nullablePercent(users?.silent ?? null, users?.total ?? null),
|
||||
},
|
||||
userScaleTrend,
|
||||
activeUserTrend: activeByDay,
|
||||
userStatus: percentRows([
|
||||
{ name: 'normal', value: users.normal },
|
||||
{ name: 'unverified', value: users.unverified },
|
||||
{ name: 'banned', value: users.banned },
|
||||
{ name: 'silent', value: users.silent },
|
||||
]),
|
||||
userStatus: users
|
||||
? percentRows([
|
||||
{ name: 'normal', value: users.normal },
|
||||
{ name: 'unverified', value: users.unverified },
|
||||
{ name: 'banned', value: users.banned },
|
||||
{ name: 'silent', value: users.silent },
|
||||
])
|
||||
: [],
|
||||
registrationSources,
|
||||
}
|
||||
}
|
||||
@@ -283,12 +333,14 @@ async function getDashboardStorageStats(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<AdminDashboardStorageStats> {
|
||||
const previous = previousRange(range)
|
||||
const reader = new AdminStatsHourlyReader(db, range, now)
|
||||
const effective = effectiveRange(range, now)
|
||||
const previous = previousRange(effective)
|
||||
const reader = new AdminStatsHourlyReader(db, effective, now)
|
||||
const previousReader = new AdminStatsHourlyReader(db, previous, now)
|
||||
const [
|
||||
quotas,
|
||||
inventory,
|
||||
trashInventory,
|
||||
storageUsedByDay,
|
||||
typeBreakdown,
|
||||
newFiles,
|
||||
@@ -297,16 +349,20 @@ async function getDashboardStorageStats(
|
||||
previousUploadBytes,
|
||||
uploadsByDay,
|
||||
uploadFilesByDay,
|
||||
dataQuality,
|
||||
transferDataQuality,
|
||||
storageDataQuality,
|
||||
spaceUsage,
|
||||
quotaPressure,
|
||||
sizeBreakdown,
|
||||
ageBreakdown,
|
||||
coverage,
|
||||
comparisonCoverage,
|
||||
snapshotCoverage,
|
||||
comparisonSnapshotCoverage,
|
||||
] = await Promise.all([
|
||||
getQuotaTotals(reader),
|
||||
getStorageInventory(reader),
|
||||
getStorageTrashInventory(reader),
|
||||
getStorageUsedByDay(reader),
|
||||
getLatestInventoryBreakdown(reader, 'file_type_group'),
|
||||
getActivityMetricTotal(reader, metricSpec(['upload_confirm']), 'count'),
|
||||
@@ -316,14 +372,17 @@ async function getDashboardStorageStats(
|
||||
getActivityMetricByDay(reader, metricSpec(['upload_confirm']), 'bytes'),
|
||||
getActivityMetricByDay(reader, metricSpec(['upload_confirm']), 'count'),
|
||||
getTransferDataQuality(reader, previousReader),
|
||||
getStorageDataQuality(reader),
|
||||
getUsageBySpaceRows(db, reader),
|
||||
getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.storageQuota, 'status'),
|
||||
getLatestInventoryBreakdown(reader, 'size_bucket'),
|
||||
getLatestInventoryBreakdown(reader, 'age_bucket'),
|
||||
reader.coverage(),
|
||||
reader.coverage('counters'),
|
||||
previousReader.coverage('counters'),
|
||||
reader.coverage('snapshots'),
|
||||
previousReader.coverage('snapshots'),
|
||||
])
|
||||
const storageTrend = createDateBuckets(range).map((date) => {
|
||||
const storageTrend = createDateBuckets(effective).map((date) => {
|
||||
return {
|
||||
date,
|
||||
usedBytes: storageUsedByDay.get(date) ?? null,
|
||||
@@ -331,25 +390,31 @@ async function getDashboardStorageStats(
|
||||
newFiles: uploadFilesByDay.get(date) ?? 0,
|
||||
}
|
||||
})
|
||||
const coldFileBytes = ['90-180d', '>180d'].reduce(
|
||||
(total, bucket) => total + (ageBreakdown.find((row) => row.name === bucket)?.bytes ?? 0),
|
||||
0,
|
||||
)
|
||||
const coldFileBytes = inventory
|
||||
? ['90-180d', '>180d'].reduce(
|
||||
(total, bucket) => total + (ageBreakdown.find((row) => row.name === bucket)?.bytes ?? 0),
|
||||
0,
|
||||
)
|
||||
: null
|
||||
const validQuotaBytes = quotas && quotas.invalidQuotaSpaces === 0 ? quotas.quotaBytes : null
|
||||
|
||||
return {
|
||||
...statsFrame(now, range, coverage, comparisonCoverage),
|
||||
dataQuality,
|
||||
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
|
||||
dataQuality: { ...transferDataQuality, ...storageDataQuality },
|
||||
summary: {
|
||||
storageUsedBytes: quotas.usedBytes,
|
||||
quotaBytes: quotas.quotaBytes,
|
||||
fileCount: inventory.files,
|
||||
newFiles: delta(newFiles, previousNewFiles),
|
||||
newBytes: delta(uploadBytes, previousUploadBytes),
|
||||
storageUsedBytes: quotas?.usedBytes ?? null,
|
||||
quotaBytes: validQuotaBytes,
|
||||
fileCount: inventory?.files ?? null,
|
||||
trashFileCount: trashInventory?.files ?? null,
|
||||
trashBytes: trashInventory?.bytes ?? null,
|
||||
newFiles: delta(newFiles, previousNewFiles, comparable(coverage, comparisonCoverage)),
|
||||
newBytes: delta(uploadBytes, previousUploadBytes, comparable(coverage, comparisonCoverage)),
|
||||
coldFileBytes,
|
||||
storageUtilization: nullablePercent(quotas.usedBytes, quotas.quotaBytes),
|
||||
coldFilePercent: nullablePercent(coldFileBytes, quotas.usedBytes),
|
||||
nearQuotaSpaces: quotaPressure.get('near') ?? 0,
|
||||
overQuotaSpaces: quotaPressure.get('over') ?? 0,
|
||||
storageUtilization: nullablePercent(quotas?.usedBytes ?? null, validQuotaBytes),
|
||||
coldFilePercent: nullablePercent(coldFileBytes, inventory?.bytes ?? null),
|
||||
nearQuotaSpaces: quotaPressure?.get('near') ?? null,
|
||||
overQuotaSpaces: quotaPressure?.get('over') ?? null,
|
||||
invalidQuotaSpaces: quotaPressure?.get('invalid') ?? null,
|
||||
},
|
||||
storageTrend,
|
||||
typeBreakdown: typeBreakdown.map(({ name, ...row }) => ({
|
||||
@@ -367,8 +432,9 @@ async function getDashboardTrafficStats(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<AdminDashboardTrafficStats> {
|
||||
const previous = previousRange(range)
|
||||
const reader = new AdminStatsHourlyReader(db, range, now)
|
||||
const effective = effectiveRange(range, now)
|
||||
const previous = previousRange(effective)
|
||||
const reader = new AdminStatsHourlyReader(db, effective, now)
|
||||
const previousReader = new AdminStatsHourlyReader(db, previous, now)
|
||||
const [
|
||||
traffic,
|
||||
@@ -430,13 +496,13 @@ async function getDashboardTrafficStats(
|
||||
item.value += value
|
||||
failureReasonRows.set(reason, item)
|
||||
}
|
||||
const trafficTrend = createDateBuckets(range).map((date) => ({
|
||||
const trafficTrend = createDateBuckets(effective).map((date) => ({
|
||||
date,
|
||||
uploadBytes: uploadByDay.get(date) ?? 0,
|
||||
downloadBytes: downloadByDay.get(date) ?? 0,
|
||||
requests: (uploadRequestsByDay.get(date) ?? 0) + (downloadRequestsByDay.get(date) ?? 0),
|
||||
}))
|
||||
const successTrend = createDateBuckets(range).map((date) => {
|
||||
const successTrend = createDateBuckets(effective).map((date) => {
|
||||
const uploadSuccesses = uploadSuccessByDay.get(date) ?? 0
|
||||
const uploadFailures = uploadFailureByDay.get(date) ?? 0
|
||||
const uploadRequests = uploadSuccesses + uploadFailures
|
||||
@@ -454,14 +520,19 @@ async function getDashboardTrafficStats(
|
||||
const issuedDownloads = Math.max(0, traffic.downloadRequests - blockedDownloads)
|
||||
|
||||
return {
|
||||
...statsFrame(now, range, coverage, comparisonCoverage),
|
||||
...statsFrame(now, effective, coverage, comparisonCoverage),
|
||||
dataQuality,
|
||||
summary: {
|
||||
totalBytes: delta(
|
||||
traffic.uploadBytes + traffic.downloadBytes,
|
||||
previousTraffic.uploadBytes + previousTraffic.downloadBytes,
|
||||
comparable(coverage, comparisonCoverage),
|
||||
),
|
||||
requestCount: delta(
|
||||
totalRequests,
|
||||
previousTraffic.uploadRequests + previousTraffic.downloadRequests,
|
||||
comparable(coverage, comparisonCoverage),
|
||||
),
|
||||
requestCount: delta(totalRequests, previousTraffic.uploadRequests + previousTraffic.downloadRequests),
|
||||
issuedDownloads,
|
||||
blockedDownloads,
|
||||
downloadIssueSuccessRate: nullablePercent(issuedDownloads, issuedDownloads + blockedDownloads),
|
||||
@@ -482,8 +553,9 @@ async function getDashboardSharingStats(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<AdminDashboardSharingStats> {
|
||||
const previous = previousRange(range)
|
||||
const reader = new AdminStatsHourlyReader(db, range, now)
|
||||
const effective = effectiveRange(range, now)
|
||||
const previous = previousRange(effective)
|
||||
const reader = new AdminStatsHourlyReader(db, effective, now)
|
||||
const previousReader = new AdminStatsHourlyReader(db, previous, now)
|
||||
const [
|
||||
sharing,
|
||||
@@ -494,8 +566,12 @@ async function getDashboardSharingStats(
|
||||
saveCount,
|
||||
previousSaveCount,
|
||||
downloadSources,
|
||||
dataQuality,
|
||||
previousDataQuality,
|
||||
coverage,
|
||||
comparisonCoverage,
|
||||
snapshotCoverage,
|
||||
comparisonSnapshotCoverage,
|
||||
] = await Promise.all([
|
||||
getSharingEventTotals(reader),
|
||||
getSharingComparisonTotals(previousReader),
|
||||
@@ -505,17 +581,25 @@ async function getDashboardSharingStats(
|
||||
getActivityMetricTotal(reader, metricSpec(['save_from_share']), 'count'),
|
||||
getActivityMetricTotal(previousReader, metricSpec(['save_from_share']), 'count'),
|
||||
getActivityMetricDimensionTotals(reader, metricSpec(['share_download']), 'source', 'count'),
|
||||
reader.coverage(),
|
||||
getSharingDataQuality(reader),
|
||||
getSharingDataQuality(previousReader),
|
||||
reader.coverage('counters'),
|
||||
previousReader.coverage('counters'),
|
||||
reader.coverage('snapshots'),
|
||||
previousReader.coverage('snapshots'),
|
||||
])
|
||||
const landingDownloads = downloadSources.get('landing_share') ?? 0
|
||||
const directDownloads = downloadSources.get('direct_share') ?? 0
|
||||
const sharingComparable =
|
||||
comparable(coverage, comparisonCoverage) &&
|
||||
hasExactSharingHistory(dataQuality) &&
|
||||
hasExactSharingHistory(previousDataQuality)
|
||||
const [viewsByDay, downloadsByDay, savesByDay] = await Promise.all([
|
||||
getActivityMetricByDay(reader, metricSpec(['share_view']), 'count'),
|
||||
getActivityMetricByDay(reader, metricSpec(['share_download']), 'count'),
|
||||
getActivityMetricByDay(reader, metricSpec(['save_from_share']), 'count'),
|
||||
])
|
||||
const trend = createDateBuckets(range).map((date) => ({
|
||||
const trend = createDateBuckets(effective).map((date) => ({
|
||||
date,
|
||||
views: viewsByDay.get(date) ?? 0,
|
||||
downloads: downloadsByDay.get(date) ?? 0,
|
||||
@@ -527,15 +611,18 @@ async function getDashboardSharingStats(
|
||||
})
|
||||
|
||||
return {
|
||||
...statsFrame(now, range, coverage, comparisonCoverage),
|
||||
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
|
||||
dataQuality,
|
||||
summary: {
|
||||
activeShares: sharing.activeShares,
|
||||
createdShares: delta(createdInRange, createdPrevious),
|
||||
views: delta(sharing.views, previousSharing.views),
|
||||
downloads: delta(sharing.downloads, previousSharing.downloads),
|
||||
saves: delta(saveCount, previousSaveCount),
|
||||
downloadsPer100Views: nullablePercent(landingDownloads, sharing.views),
|
||||
savesPer100Views: nullablePercent(saveCount, sharing.views),
|
||||
createdShares: delta(createdInRange, createdPrevious, comparable(coverage, comparisonCoverage)),
|
||||
views: delta(sharing.views, previousSharing.views, sharingComparable),
|
||||
downloads: delta(sharing.downloads, previousSharing.downloads, sharingComparable),
|
||||
saves: delta(saveCount, previousSaveCount, comparable(coverage, comparisonCoverage)),
|
||||
downloadsPer100Views: hasExactSharingHistory(dataQuality)
|
||||
? nullablePercent(landingDownloads, sharing.views)
|
||||
: null,
|
||||
savesPer100Views: hasExactSharingHistory(dataQuality) ? nullablePercent(saveCount, sharing.views) : null,
|
||||
passwordPasses: sharing.passwordPasses,
|
||||
},
|
||||
trend,
|
||||
@@ -563,11 +650,29 @@ async function getShareCreatedKinds(reader: AdminStatsHourlyReader): Promise<Map
|
||||
return result
|
||||
}
|
||||
|
||||
async function getSharingDataQuality(reader: AdminStatsHourlyReader): Promise<AdminSharingDataQuality> {
|
||||
const dimensions = await getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.statsDataQualitySnapshot, 'kind')
|
||||
if (!dimensions) return { unlocatedViews: null, unlocatedDownloads: null, unlocatedEvents: null }
|
||||
const unlocatedViews = dimensions.get('share_views') ?? 0
|
||||
const unlocatedDownloads = dimensions.get('share_downloads') ?? 0
|
||||
return {
|
||||
unlocatedViews,
|
||||
unlocatedDownloads,
|
||||
unlocatedEvents: unlocatedViews + unlocatedDownloads,
|
||||
}
|
||||
}
|
||||
|
||||
function hasExactSharingHistory(quality: AdminSharingDataQuality): boolean {
|
||||
return quality.unlocatedEvents === 0
|
||||
}
|
||||
|
||||
function statsFrame(
|
||||
now: Date,
|
||||
range: AdminStatsDateRange,
|
||||
coverage: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
|
||||
comparisonCoverage?: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
|
||||
snapshotCoverage?: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
|
||||
comparisonSnapshotCoverage?: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
|
||||
) {
|
||||
return {
|
||||
generatedAt: now.toISOString(),
|
||||
@@ -576,6 +681,8 @@ function statsFrame(
|
||||
timeZone: 'UTC' as const,
|
||||
coverage,
|
||||
comparisonCoverage,
|
||||
snapshotCoverage,
|
||||
comparisonSnapshotCoverage,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -598,7 +705,19 @@ function previousRange(range: AdminStatsDateRange): AdminStatsDateRange {
|
||||
return { from: new Date(to.getTime() - durationMs), to, timeZone: range.timeZone }
|
||||
}
|
||||
|
||||
function delta(value: number, previousValue: number): AdminStatsDelta {
|
||||
function effectiveRange(range: AdminStatsDateRange, now: Date): AdminStatsDateRange {
|
||||
const closedToExclusive = Math.min(range.to.getTime() + 1, startOfHour(now).getTime())
|
||||
return {
|
||||
from: range.from,
|
||||
to: new Date(Math.max(range.from.getTime(), closedToExclusive) - 1),
|
||||
timeZone: range.timeZone,
|
||||
}
|
||||
}
|
||||
|
||||
function delta(value: number | null, previousValue: number | null, canCompare = true): AdminStatsDelta {
|
||||
if (value === null || previousValue === null || !canCompare) {
|
||||
return { value, previousValue: canCompare ? previousValue : null, change: null, changePercent: null }
|
||||
}
|
||||
return {
|
||||
value,
|
||||
previousValue,
|
||||
@@ -607,6 +726,10 @@ function delta(value: number, previousValue: number): AdminStatsDelta {
|
||||
}
|
||||
}
|
||||
|
||||
function comparable(current: { status: string }, previous: { status: string }): boolean {
|
||||
return current.status === 'complete' && previous.status === 'complete'
|
||||
}
|
||||
|
||||
function createDateBuckets(range: AdminStatsDateRange): string[] {
|
||||
const dates = new Set<string>()
|
||||
for (let timestamp = range.from.getTime(); timestamp <= range.to.getTime(); timestamp += 6 * 60 * 60 * 1000) {
|
||||
@@ -616,9 +739,9 @@ function createDateBuckets(range: AdminStatsDateRange): string[] {
|
||||
return [...dates]
|
||||
}
|
||||
|
||||
async function getLatestGaugeTotal(reader: AdminStatsHourlyReader, metric: AdminStatsMetric): Promise<number> {
|
||||
async function getLatestGaugeTotal(reader: AdminStatsHourlyReader, metric: AdminStatsMetric): Promise<number | null> {
|
||||
const rows = await reader.latestRows(metric)
|
||||
return rows.find((row) => row.orgId === '' && row.dimensionKey === '')?.count ?? 0
|
||||
return rows.find((row) => row.orgId === '' && row.dimensionKey === '')?.count ?? null
|
||||
}
|
||||
|
||||
async function getLatestGaugeDimensions(
|
||||
@@ -626,7 +749,9 @@ async function getLatestGaugeDimensions(
|
||||
metric: AdminStatsMetric,
|
||||
dimensionKey: AdminStatsDimension,
|
||||
field: 'count' | 'bytes' = 'count',
|
||||
): Promise<Map<string, number>> {
|
||||
): Promise<Map<string, number> | null> {
|
||||
const baseRows = await reader.latestRows(metric)
|
||||
if (!baseRows.some((row) => row.orgId === '' && row.dimensionKey === '')) return null
|
||||
const result = new Map<string, number>()
|
||||
for (const row of await reader.latestRows(metric, [dimensionKey])) {
|
||||
if (row.orgId === '' && row.dimensionKey === dimensionKey) incrementMap(result, row.dimensionValue, row[field])
|
||||
@@ -639,20 +764,27 @@ async function getLatestGaugeDimensionSum(
|
||||
metric: AdminStatsMetric,
|
||||
dimensionKey: AdminStatsDimension,
|
||||
values: string[],
|
||||
): Promise<number> {
|
||||
): Promise<number | null> {
|
||||
const dimensions = await getLatestGaugeDimensions(reader, metric, dimensionKey)
|
||||
if (!dimensions) return null
|
||||
return values.reduce((sum, value) => sum + (dimensions.get(value) ?? 0), 0)
|
||||
}
|
||||
|
||||
async function getQuotaTotals(reader: AdminStatsHourlyReader): Promise<{ usedBytes: number; quotaBytes: number }> {
|
||||
async function getQuotaTotals(
|
||||
reader: AdminStatsHourlyReader,
|
||||
): Promise<{ usedBytes: number; quotaBytes: number; invalidQuotaSpaces: number } | null> {
|
||||
const [usedRows, quotaRows] = await Promise.all([
|
||||
reader.latestRows(ADMIN_STATS_METRICS.storageUsed),
|
||||
reader.latestRows(ADMIN_STATS_METRICS.storageQuota),
|
||||
reader.latestRows(ADMIN_STATS_METRICS.storageQuota, ['', 'status']),
|
||||
])
|
||||
return {
|
||||
usedBytes: usedRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes ?? 0,
|
||||
quotaBytes: quotaRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes ?? 0,
|
||||
}
|
||||
const usedBytes = usedRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes
|
||||
const quotaBytes = quotaRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes
|
||||
const invalidQuotaSpaces = quotaRows.find(
|
||||
(row) => row.orgId === '' && row.dimensionKey === 'status' && row.dimensionValue === 'invalid',
|
||||
)?.count
|
||||
return usedBytes === undefined || quotaBytes === undefined
|
||||
? null
|
||||
: { usedBytes, quotaBytes, invalidQuotaSpaces: invalidQuotaSpaces ?? 0 }
|
||||
}
|
||||
|
||||
type UserInventory = {
|
||||
@@ -664,13 +796,15 @@ type UserInventory = {
|
||||
verified: number
|
||||
}
|
||||
|
||||
async function getUserInventory(reader: AdminStatsHourlyReader): Promise<UserInventory> {
|
||||
async function getUserInventory(reader: AdminStatsHourlyReader): Promise<UserInventory | null> {
|
||||
const rows = await reader.latestRows(ADMIN_STATS_METRICS.userInventory, ['', 'status'])
|
||||
const total = rows.find((row) => row.dimensionKey === '')?.count
|
||||
if (total === undefined) return null
|
||||
const dimensions = new Map(
|
||||
rows.filter((row) => row.dimensionKey === 'status').map((row) => [row.dimensionValue, row.count]),
|
||||
)
|
||||
return {
|
||||
total: rows.find((row) => row.dimensionKey === '')?.count ?? 0,
|
||||
total,
|
||||
normal: dimensions.get('normal') ?? 0,
|
||||
unverified: dimensions.get('unverified') ?? 0,
|
||||
banned: dimensions.get('banned') ?? 0,
|
||||
@@ -681,8 +815,9 @@ async function getUserInventory(reader: AdminStatsHourlyReader): Promise<UserInv
|
||||
|
||||
type ActiveUserSnapshot = { dau: number; wau: number; mau: number }
|
||||
|
||||
async function getActiveUserSnapshot(reader: AdminStatsHourlyReader): Promise<ActiveUserSnapshot> {
|
||||
async function getActiveUserSnapshot(reader: AdminStatsHourlyReader): Promise<ActiveUserSnapshot | null> {
|
||||
const dimensions = await getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.userActiveSnapshot, 'window')
|
||||
if (!dimensions) return null
|
||||
return { dau: dimensions.get('dau') ?? 0, wau: dimensions.get('wau') ?? 0, mau: dimensions.get('mau') ?? 0 }
|
||||
}
|
||||
|
||||
@@ -697,7 +832,7 @@ async function getUserTotalsByDay(reader: AdminStatsHourlyReader): Promise<Map<s
|
||||
async function getRollingActiveUserTrend(
|
||||
reader: AdminStatsHourlyReader,
|
||||
range: AdminStatsDateRange,
|
||||
): Promise<Array<{ date: string; dau: number; wau: number; mau: number }>> {
|
||||
): Promise<Array<{ date: string; dau: number | null; wau: number | null; mau: number | null }>> {
|
||||
const [dau, wau, mau] = await Promise.all([
|
||||
getLatestGaugeValueByDay(reader, ADMIN_STATS_METRICS.userActiveSnapshot, 'count', 'window', 'dau'),
|
||||
getLatestGaugeValueByDay(reader, ADMIN_STATS_METRICS.userActiveSnapshot, 'count', 'window', 'wau'),
|
||||
@@ -705,9 +840,9 @@ async function getRollingActiveUserTrend(
|
||||
])
|
||||
return createDateBuckets(range).map((date) => ({
|
||||
date,
|
||||
dau: dau.get(date) ?? 0,
|
||||
wau: wau.get(date) ?? 0,
|
||||
mau: mau.get(date) ?? 0,
|
||||
dau: dau.get(date) ?? null,
|
||||
wau: wau.get(date) ?? null,
|
||||
mau: mau.get(date) ?? null,
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -836,7 +971,7 @@ async function getActivityMetricDimensionTotalsFromRollup(
|
||||
return result
|
||||
}
|
||||
|
||||
async function getCloudReportOutcomes(reader: AdminStatsHourlyReader): Promise<Map<string, number>> {
|
||||
async function getCloudReportOutcomes(reader: AdminStatsHourlyReader): Promise<Map<string, number> | null> {
|
||||
return getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.trafficReportSnapshot, 'status')
|
||||
}
|
||||
|
||||
@@ -917,14 +1052,31 @@ async function getStorageUsedByDay(reader: AdminStatsHourlyReader): Promise<Map<
|
||||
return getLatestGaugeValueByDay(reader, ADMIN_STATS_METRICS.storageUsed, 'bytes', '', '')
|
||||
}
|
||||
|
||||
async function getStorageInventory(reader: AdminStatsHourlyReader): Promise<{ files: number; bytes: number }> {
|
||||
async function getStorageInventory(reader: AdminStatsHourlyReader): Promise<{ files: number; bytes: number } | null> {
|
||||
const rows = await reader.latestRows(ADMIN_STATS_METRICS.storageInventory)
|
||||
return rows
|
||||
.filter((row) => row.orgId === '' && row.dimensionKey === '')
|
||||
.reduce((total, row) => ({ files: total.files + row.count, bytes: total.bytes + row.bytes }), {
|
||||
files: 0,
|
||||
bytes: 0,
|
||||
})
|
||||
const row = rows.find((value) => value.orgId === '' && value.dimensionKey === '')
|
||||
return row ? { files: row.count, bytes: row.bytes } : null
|
||||
}
|
||||
|
||||
async function getStorageTrashInventory(
|
||||
reader: AdminStatsHourlyReader,
|
||||
): Promise<{ files: number; bytes: number } | null> {
|
||||
const rows = await reader.latestRows(ADMIN_STATS_METRICS.storageTrashSnapshot)
|
||||
const row = rows.find((value) => value.orgId === '' && value.dimensionKey === '')
|
||||
return row ? { files: row.count, bytes: row.bytes } : null
|
||||
}
|
||||
|
||||
async function getStorageDataQuality(
|
||||
reader: AdminStatsHourlyReader,
|
||||
): Promise<Pick<AdminStorageDataQuality, 'usageDriftSpaces' | 'usageDriftBytes'>> {
|
||||
const rows = await reader.latestRows(ADMIN_STATS_METRICS.statsDataQualitySnapshot, ['', 'kind'])
|
||||
if (!rows.some((row) => row.orgId === '' && row.dimensionKey === '')) {
|
||||
return { usageDriftSpaces: null, usageDriftBytes: null }
|
||||
}
|
||||
const drift = rows.find(
|
||||
(row) => row.orgId === '' && row.dimensionKey === 'kind' && row.dimensionValue === 'storage_usage_drift',
|
||||
)
|
||||
return { usageDriftSpaces: drift?.count ?? 0, usageDriftBytes: drift?.bytes ?? 0 }
|
||||
}
|
||||
|
||||
async function getLatestInventoryBreakdown(
|
||||
@@ -932,7 +1084,9 @@ async function getLatestInventoryBreakdown(
|
||||
dimensionKey: 'file_type_group' | 'size_bucket' | 'age_bucket',
|
||||
): Promise<Array<{ name: string; bytes: number; files: number; percent: number }>> {
|
||||
const values = new Map<string, { name: string; bytes: number; files: number }>()
|
||||
for (const row of await reader.latestRows(ADMIN_STATS_METRICS.storageInventory, [dimensionKey])) {
|
||||
const rows = await reader.latestRows(ADMIN_STATS_METRICS.storageInventory, ['', dimensionKey])
|
||||
if (!rows.some((row) => row.orgId === '' && row.dimensionKey === '')) return []
|
||||
for (const row of rows) {
|
||||
if (row.orgId !== '' || row.dimensionKey !== dimensionKey) continue
|
||||
const value = values.get(row.dimensionValue) ?? { name: row.dimensionValue, bytes: 0, files: 0 }
|
||||
value.bytes += row.bytes
|
||||
@@ -972,7 +1126,7 @@ async function getTrafficTotals(
|
||||
|
||||
async function getSharingEventTotals(
|
||||
reader: AdminStatsHourlyReader,
|
||||
): Promise<{ activeShares: number; views: number; passwordPasses: number; downloads: number }> {
|
||||
): Promise<{ activeShares: number | null; views: number; passwordPasses: number; downloads: number }> {
|
||||
const [activeShares, views, passwordPasses, downloads] = await Promise.all([
|
||||
getLatestGaugeDimensionSum(reader, ADMIN_STATS_METRICS.shareInventory, 'lifecycle', ['usable']),
|
||||
getActivityMetricTotal(reader, metricSpec(['share_view']), 'count'),
|
||||
@@ -1035,7 +1189,19 @@ async function getTopSharesByActivity(db: Database, reader: AdminStatsHourlyRead
|
||||
return topIds.flatMap((id) => {
|
||||
const row = shareById.get(id)
|
||||
const countValue = activityById.get(id)
|
||||
if (!row || !countValue) return []
|
||||
if (!countValue) return []
|
||||
if (!row) {
|
||||
return {
|
||||
id,
|
||||
token: '',
|
||||
name: '已删除的分享',
|
||||
creatorId: '',
|
||||
creatorName: '已删除用户',
|
||||
views: countValue.views,
|
||||
downloads: countValue.downloads,
|
||||
status: 'deleted',
|
||||
}
|
||||
}
|
||||
return {
|
||||
id: row.id,
|
||||
token: row.token,
|
||||
@@ -1074,7 +1240,7 @@ async function getUsageBySpaceRows(
|
||||
...row,
|
||||
orgName: org?.name ?? row.orgId,
|
||||
orgType: org && isPersonalOrgLike({ slug: org.slug, metadata: org.metadata }) ? 'personal' : 'team',
|
||||
utilization: percent(row.usedBytes, row.quotaBytes),
|
||||
utilization: row.quotaBytes > 0 ? percent(row.usedBytes, row.quotaBytes) : null,
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -1123,8 +1289,8 @@ function percent(part: number, total: number): number {
|
||||
return Math.round((part / total) * 1000) / 10
|
||||
}
|
||||
|
||||
function nullablePercent(part: number, total: number): number | null {
|
||||
return total > 0 ? percent(part, total) : null
|
||||
function nullablePercent(part: number | null, total: number | null): number | null {
|
||||
return part !== null && total !== null && total > 0 ? percent(part, total) : null
|
||||
}
|
||||
|
||||
function toNumber(value: unknown): number {
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import type { BackgroundJob, BackgroundJobStatus } from '@shared/types'
|
||||
import { and, count, desc, eq, type SQL } from 'drizzle-orm'
|
||||
import { nanoid } from 'nanoid'
|
||||
import { backgroundJobs } from '../../db/schema'
|
||||
import { activityEvents, backgroundJobs } from '../../db/schema'
|
||||
import { type AtomicQuery, executeWriteTransaction } from '../../db/transaction'
|
||||
import type { Database } from '../../platform/interface'
|
||||
import {
|
||||
BackgroundJobError,
|
||||
@@ -9,6 +10,7 @@ import {
|
||||
type BackgroundJobRepo,
|
||||
type ListBackgroundJobsOptions,
|
||||
} from '../../usecases/ports'
|
||||
import { adminStatsFactValues } from './admin-stats-fact'
|
||||
|
||||
type BackgroundJobRow = typeof backgroundJobs.$inferSelect
|
||||
|
||||
@@ -153,7 +155,25 @@ export function createBackgroundJobRepo(db: Database): BackgroundJobRepo {
|
||||
finishedAt: input.finishedAt === undefined ? finishedAtFor(nextStatus, row.finishedAt, now) : input.finishedAt,
|
||||
updatedAt: now,
|
||||
}
|
||||
await db.update(backgroundJobs).set(values).where(eq(backgroundJobs.id, id))
|
||||
const writes: AtomicQuery[] = [db.update(backgroundJobs).set(values).where(eq(backgroundJobs.id, id))]
|
||||
if (!row.finishedAt && values.finishedAt && ['completed', 'failed', 'canceled'].includes(nextStatus)) {
|
||||
writes.push(
|
||||
db
|
||||
.insert(activityEvents)
|
||||
.values(
|
||||
adminStatsFactValues({
|
||||
action: 'stats_background_job_finished',
|
||||
sourceId: row.id,
|
||||
orgId: row.orgId,
|
||||
targetType: 'background_job',
|
||||
occurredAt: values.finishedAt,
|
||||
metadata: { jobType: row.type, outcome: nextStatus },
|
||||
}),
|
||||
)
|
||||
.onConflictDoNothing(),
|
||||
)
|
||||
}
|
||||
await executeWriteTransaction(db, writes)
|
||||
return repo.get(orgId, id)
|
||||
},
|
||||
|
||||
@@ -164,10 +184,25 @@ export function createBackgroundJobRepo(db: Database): BackgroundJobRepo {
|
||||
throw new BackgroundJobError('not_cancelable')
|
||||
}
|
||||
const now = new Date()
|
||||
await db
|
||||
.update(backgroundJobs)
|
||||
.set({ status: 'canceled', updatedAt: now, finishedAt: now })
|
||||
.where(eq(backgroundJobs.id, id))
|
||||
await executeWriteTransaction(db, [
|
||||
db
|
||||
.update(backgroundJobs)
|
||||
.set({ status: 'canceled', updatedAt: now, finishedAt: now })
|
||||
.where(eq(backgroundJobs.id, id)),
|
||||
db
|
||||
.insert(activityEvents)
|
||||
.values(
|
||||
adminStatsFactValues({
|
||||
action: 'stats_background_job_finished',
|
||||
sourceId: row.id,
|
||||
orgId: row.orgId,
|
||||
targetType: 'background_job',
|
||||
occurredAt: now,
|
||||
metadata: { jobType: row.type, outcome: 'canceled' },
|
||||
}),
|
||||
)
|
||||
.onConflictDoNothing(),
|
||||
])
|
||||
return repo.get(orgId, id)
|
||||
},
|
||||
|
||||
|
||||
@@ -152,6 +152,7 @@ function revokeExistingPlanQueries(
|
||||
eq(orgQuotaEntitlements.resourceType, value.resourceType),
|
||||
eq(orgQuotaEntitlements.entitlementType, 'plan'),
|
||||
eq(orgQuotaEntitlements.status, 'active'),
|
||||
sql`${orgQuotaEntitlements.source} <> 'free_plan'`,
|
||||
sql`${orgQuotaEntitlements.sourceId} != ${value.sourceId}`,
|
||||
),
|
||||
),
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { asc, eq, inArray } from 'drizzle-orm'
|
||||
import { and, asc, eq, isNull, lte, or, sql } from 'drizzle-orm'
|
||||
import { nanoid } from 'nanoid'
|
||||
import { cloudTrafficReports } from '../../db/schema'
|
||||
import type { Database } from '../../platform/interface'
|
||||
@@ -23,6 +23,8 @@ function toRecord(row: typeof cloudTrafficReports.$inferSelect): CloudTrafficRep
|
||||
creditsPerUnit: row.creditsPerUnit,
|
||||
status: row.status as CloudTrafficReportStatus,
|
||||
error: row.error,
|
||||
attemptCount: row.attemptCount,
|
||||
nextRetryAt: row.nextRetryAt,
|
||||
createdAt: row.createdAt,
|
||||
updatedAt: row.updatedAt,
|
||||
}
|
||||
@@ -49,24 +51,43 @@ export function createCloudTrafficReportRepo(db: Database): CloudTrafficReportRe
|
||||
creditsPerUnit: input.creditsPerUnit,
|
||||
status: input.status,
|
||||
error: null,
|
||||
attemptCount: 0,
|
||||
nextRetryAt: null,
|
||||
createdAt: input.now,
|
||||
updatedAt: input.now,
|
||||
})
|
||||
},
|
||||
|
||||
async updateStatus(eventId, status, error, now) {
|
||||
async updateStatus(eventId, status, error, now, retry) {
|
||||
await db
|
||||
.update(cloudTrafficReports)
|
||||
.set({ status, error, updatedAt: now })
|
||||
.set({
|
||||
status,
|
||||
error,
|
||||
updatedAt: now,
|
||||
...(retry ? { attemptCount: retry.attemptCount, nextRetryAt: retry.nextRetryAt } : {}),
|
||||
})
|
||||
.where(eq(cloudTrafficReports.eventId, eventId))
|
||||
},
|
||||
|
||||
async listPending(limit) {
|
||||
async listPending(limit, now) {
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(cloudTrafficReports)
|
||||
.where(inArray(cloudTrafficReports.status, ['pending', 'failed']))
|
||||
.orderBy(asc(cloudTrafficReports.createdAt))
|
||||
.where(
|
||||
or(
|
||||
eq(cloudTrafficReports.status, 'pending'),
|
||||
and(
|
||||
eq(cloudTrafficReports.status, 'failed'),
|
||||
or(isNull(cloudTrafficReports.nextRetryAt), lte(cloudTrafficReports.nextRetryAt, now)),
|
||||
),
|
||||
),
|
||||
)
|
||||
.orderBy(
|
||||
asc(sql`CASE WHEN ${cloudTrafficReports.status} = 'pending' THEN 0 ELSE 1 END`),
|
||||
asc(cloudTrafficReports.nextRetryAt),
|
||||
asc(cloudTrafficReports.createdAt),
|
||||
)
|
||||
.limit(limit)
|
||||
return rows.map(toRecord)
|
||||
},
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { downloadTaskRuntimeSchema } from '@shared/schemas'
|
||||
import type { DownloadTask, DownloadTaskRuntime } from '@shared/types'
|
||||
import { and, asc, count, desc, eq, gte, inArray, isNull, like, ne, notInArray, or, type SQL, sql } from 'drizzle-orm'
|
||||
import { downloaders, downloadTasks } from '../../db/schema'
|
||||
import { activityEvents, downloaders, downloadTasks } from '../../db/schema'
|
||||
import { type AtomicQuery, executeWriteTransaction } from '../../db/transaction'
|
||||
import type { Database } from '../../platform/interface'
|
||||
import {
|
||||
type CreateDownloadTaskRecordInput,
|
||||
@@ -11,6 +12,7 @@ import {
|
||||
type ListDownloadTasksFilters,
|
||||
type UpdateDownloadTaskFields,
|
||||
} from '../../usecases/ports'
|
||||
import { adminStatsFactValues } from './admin-stats-fact'
|
||||
|
||||
type DownloadTaskRow = typeof downloadTasks.$inferSelect
|
||||
|
||||
@@ -244,7 +246,35 @@ export function createDownloadTaskRepo(db: Database): DownloadTaskRepo {
|
||||
},
|
||||
|
||||
async setFields(id, fields: UpdateDownloadTaskFields) {
|
||||
await db.update(downloadTasks).set(fields).where(eq(downloadTasks.id, id))
|
||||
const row = await findRow(id)
|
||||
if (!row) throw new DownloadError('not_found')
|
||||
const nextStatus = fields.status ?? row.status
|
||||
const finishedAt = fields.finishedAt ?? row.finishedAt
|
||||
const writes: AtomicQuery[] = [db.update(downloadTasks).set(fields).where(eq(downloadTasks.id, id))]
|
||||
if (!row.finishedAt && finishedAt && ['completed', 'failed', 'canceled'].includes(nextStatus)) {
|
||||
writes.push(
|
||||
db
|
||||
.insert(activityEvents)
|
||||
.values(
|
||||
adminStatsFactValues({
|
||||
action: 'stats_remote_download_finished',
|
||||
sourceId: `${row.id}:${row.attempt}:${finishedAt.getTime()}:${nextStatus}`,
|
||||
targetId: row.id,
|
||||
orgId: row.orgId,
|
||||
targetType: 'remote_download',
|
||||
occurredAt: finishedAt,
|
||||
metadata: {
|
||||
category: row.category ?? 'uncategorized',
|
||||
downloaderId: row.assignedDownloaderId,
|
||||
outcome: nextStatus,
|
||||
bytes: nextStatus === 'completed' ? (fields.billingChargedBytes ?? row.billingChargedBytes) : 0,
|
||||
},
|
||||
}),
|
||||
)
|
||||
.onConflictDoNothing(),
|
||||
)
|
||||
}
|
||||
await executeWriteTransaction(db, writes)
|
||||
},
|
||||
|
||||
async claimQueued(id, downloaderId, now) {
|
||||
|
||||
@@ -260,6 +260,16 @@ describe('confirmUpload — name conflict', () => {
|
||||
const { db } = await createTestApp()
|
||||
await insertStorage(db)
|
||||
const orgId = nanoid()
|
||||
const now = Date.now()
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES (${nanoid()}, ${orgId}, 0, 0, 0, 0, '2026-05')
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quota_entitlements
|
||||
(id, org_id, resource_type, entitlement_type, source, source_id, bytes, starts_at, status, created_at, updated_at)
|
||||
VALUES (${nanoid()}, ${orgId}, 'storage', 'plan', 'test', ${nanoid()}, 1000000, ${now}, 'active', ${now}, ${now})
|
||||
`)
|
||||
|
||||
const draftId = await makeFile(db, orgId, 'collision.txt', { status: 'draft' })
|
||||
await makeFile(db, orgId, 'collision.txt') // active sibling
|
||||
|
||||
@@ -107,43 +107,43 @@ async function insertDraftFile(
|
||||
// ─── storage usage reservations ───────────────────────────────────────────────
|
||||
|
||||
describe('reserveStorageUsage', () => {
|
||||
it('reserves and increments when no quota row exists (unlimited)', async () => {
|
||||
it('fails closed when no quota row exists', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
const storageId = await insertStorage(db, { id: 'st-ul', used: 0 })
|
||||
|
||||
const result = await reserveStorageUsage(
|
||||
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
|
||||
{ orgId, storageId, bytes: 500 },
|
||||
)
|
||||
|
||||
expect(result).toEqual({ orgId, storageId, bytes: 500 })
|
||||
await expect(
|
||||
reserveStorageUsage(
|
||||
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
|
||||
{ orgId, storageId, bytes: 500 },
|
||||
),
|
||||
).rejects.toThrow(StorageQuotaExceededError)
|
||||
const rows = await db.all<{ used: number }>(sql`SELECT used FROM storages WHERE id = ${storageId}`)
|
||||
expect(rows[0].used).toBe(500)
|
||||
expect(rows[0].used).toBe(0)
|
||||
})
|
||||
|
||||
it('reserves and increments when quota is 0 (unlimited)', async () => {
|
||||
it('fails closed when effective quota is zero', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
const storageId = await insertStorage(db, { id: 'st-q0', used: 100 })
|
||||
await insertOrgQuota(db, orgId, 0, 5000)
|
||||
|
||||
const result = await reserveStorageUsage(
|
||||
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
|
||||
{ orgId, storageId, bytes: 999999 },
|
||||
)
|
||||
|
||||
expect(result).toEqual({ orgId, storageId, bytes: 999999 })
|
||||
await expect(
|
||||
reserveStorageUsage(
|
||||
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
|
||||
{ orgId, storageId, bytes: 999999 },
|
||||
),
|
||||
).rejects.toThrow(StorageQuotaExceededError)
|
||||
const rows = await db.all<{ used: number }>(sql`SELECT used FROM storages WHERE id = ${storageId}`)
|
||||
expect(rows[0].used).toBe(1000099)
|
||||
expect(rows[0].used).toBe(100)
|
||||
})
|
||||
|
||||
it('treats base quota 0 as unlimited without grant aggregation', async () => {
|
||||
it('treats effective storage quota 0 as invalid', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
await insertOrgQuota(db, orgId, 0, 5000)
|
||||
|
||||
await expect(createQuotaRepo(db).hasQuotaForBytes(orgId, 10_000_000)).resolves.toBe(true)
|
||||
await expect(createQuotaRepo(db).hasQuotaForBytes(orgId, 10_000_000)).resolves.toBe(false)
|
||||
await expect(createQuotaRepo(db).getEffectiveQuota(orgId)).resolves.toMatchObject({ baseQuota: 0, quota: 0 })
|
||||
})
|
||||
|
||||
|
||||
@@ -1,11 +1,64 @@
|
||||
import { eq, sql } from 'drizzle-orm'
|
||||
import { nanoid } from 'nanoid'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { orgQuotaEntitlements, orgQuotas } from '../../db/schema.js'
|
||||
import { createTestApp } from '../../test/setup.js'
|
||||
import { orgQuotaEntitlements, orgQuotas, systemOptions } from '../../db/schema.js'
|
||||
import { adminHeaders, createTestApp } from '../../test/setup.js'
|
||||
import { createQuotaRepo } from './quota.js'
|
||||
|
||||
describe('effective quota', () => {
|
||||
it('enforces one quota row per organization', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
await db.insert(orgQuotas).values({ id: nanoid(), orgId })
|
||||
|
||||
await expect(db.insert(orgQuotas).values({ id: nanoid(), orgId })).rejects.toThrow()
|
||||
})
|
||||
|
||||
it('restores missing and revoked Free baselines idempotently', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
await adminHeaders(app)
|
||||
const [{ orgId }] = await db.select({ orgId: orgQuotas.orgId }).from(orgQuotas).limit(1)
|
||||
const now = new Date('2026-07-20T12:00:00.000Z')
|
||||
await db.delete(orgQuotaEntitlements).where(eq(orgQuotaEntitlements.orgId, orgId))
|
||||
await db
|
||||
.insert(systemOptions)
|
||||
.values([
|
||||
{ key: 'default_org_quota', value: '1234' },
|
||||
{ key: 'default_org_monthly_traffic_quota', value: '5678' },
|
||||
])
|
||||
.onConflictDoUpdate({ target: systemOptions.key, set: { value: sql`excluded.value` } })
|
||||
|
||||
const repo = createQuotaRepo(db)
|
||||
await repo.reconcileFreePlanBaselines(now)
|
||||
await repo.reconcileFreePlanBaselines(new Date(now.getTime() + 1000))
|
||||
|
||||
const inserted = await db
|
||||
.select({ resourceType: orgQuotaEntitlements.resourceType, bytes: orgQuotaEntitlements.bytes })
|
||||
.from(orgQuotaEntitlements)
|
||||
.where(eq(orgQuotaEntitlements.orgId, orgId))
|
||||
.orderBy(orgQuotaEntitlements.resourceType)
|
||||
expect(inserted).toEqual([
|
||||
{ resourceType: 'storage', bytes: 1234 },
|
||||
{ resourceType: 'traffic', bytes: 5678 },
|
||||
])
|
||||
|
||||
await db
|
||||
.update(orgQuotaEntitlements)
|
||||
.set({ status: 'revoked', expiresAt: now })
|
||||
.where(eq(orgQuotaEntitlements.orgId, orgId))
|
||||
await repo.reconcileFreePlanBaselines(new Date(now.getTime() + 2000))
|
||||
await expect(
|
||||
db
|
||||
.select({ status: orgQuotaEntitlements.status, expiresAt: orgQuotaEntitlements.expiresAt })
|
||||
.from(orgQuotaEntitlements)
|
||||
.where(eq(orgQuotaEntitlements.orgId, orgId))
|
||||
.orderBy(orgQuotaEntitlements.resourceType),
|
||||
).resolves.toEqual([
|
||||
{ status: 'active', expiresAt: null },
|
||||
{ status: 'active', expiresAt: null },
|
||||
])
|
||||
})
|
||||
|
||||
it('returns storage and traffic quota state', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
@@ -111,6 +164,44 @@ describe('effective quota', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps Free as the active baseline and falls back to it after a paid plan expires', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
const now = new Date('2026-05-06T00:00:00Z')
|
||||
await db.insert(orgQuotas).values({
|
||||
id: nanoid(),
|
||||
orgId,
|
||||
quota: 0,
|
||||
used: 250,
|
||||
trafficQuota: 0,
|
||||
trafficUsed: 0,
|
||||
trafficPeriod: '2026-05',
|
||||
})
|
||||
await db.insert(orgQuotaEntitlements).values([
|
||||
{
|
||||
...entitlement(orgId, 'storage', 'free-storage-plan', 5000, 'active', now, 'Free'),
|
||||
source: 'free_plan',
|
||||
},
|
||||
{
|
||||
...entitlement(orgId, 'storage', `stripe_subscription:sub_storage:${orgId}`, 3000, 'active', now, 'Pro'),
|
||||
expiresAt: new Date('2026-05-07T00:00:00Z'),
|
||||
},
|
||||
])
|
||||
|
||||
await expect(createQuotaRepo(db).getEffectiveQuota(orgId, now)).resolves.toMatchObject({
|
||||
baseQuota: 3000,
|
||||
quota: 3000,
|
||||
storagePlanName: 'Pro',
|
||||
})
|
||||
await expect(createQuotaRepo(db).getEffectiveQuota(orgId, new Date('2026-05-08T00:00:00Z'))).resolves.toMatchObject(
|
||||
{
|
||||
baseQuota: 5000,
|
||||
quota: 5000,
|
||||
storagePlanName: 'Free',
|
||||
},
|
||||
)
|
||||
})
|
||||
|
||||
it('uses a smaller active subscription plan instead of the larger default quota', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
@@ -639,6 +730,9 @@ describe('effective quota', () => {
|
||||
trafficUsed: 900,
|
||||
trafficPeriod: '2026-04',
|
||||
})
|
||||
await db
|
||||
.insert(orgQuotaEntitlements)
|
||||
.values(entitlement(orgId, 'traffic', 'free-traffic-plan', 1000, 'active', now, 'Free'))
|
||||
await db.run(sql`
|
||||
CREATE TRIGGER org_quotas_rollover_race
|
||||
BEFORE UPDATE ON org_quotas
|
||||
@@ -656,13 +750,13 @@ describe('effective quota', () => {
|
||||
expect(rows[0].trafficPeriod).toBe('2026-05')
|
||||
})
|
||||
|
||||
it('allows traffic when no quota row exists', async () => {
|
||||
it('fails closed when no traffic quota row exists', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
const now = new Date('2026-05-06T00:00:00Z')
|
||||
|
||||
await expect(createQuotaRepo(db).hasTrafficQuotaForBytes(orgId, 1024, now)).resolves.toBe(true)
|
||||
await expect(createQuotaRepo(db).consumeTrafficIfQuotaAllows(orgId, 1024, now)).resolves.toBe(true)
|
||||
await expect(createQuotaRepo(db).hasTrafficQuotaForBytes(orgId, 1024, now)).resolves.toBe(false)
|
||||
await expect(createQuotaRepo(db).consumeTrafficIfQuotaAllows(orgId, 1024, now)).resolves.toBe(false)
|
||||
})
|
||||
|
||||
it('treats zero base storage quota as limited when storage entitlements exist', async () => {
|
||||
|
||||
+133
-62
@@ -1,6 +1,7 @@
|
||||
import { DEFAULT_ORG_QUOTA, DEFAULT_ORG_TRAFFIC_QUOTA } from '@shared/constants'
|
||||
import { and, eq, inArray, or, sql } from 'drizzle-orm'
|
||||
import { organization } from '../../db/auth-schema'
|
||||
import { orgQuotaEntitlements, orgQuotas, storages } from '../../db/schema'
|
||||
import { orgQuotaEntitlements, orgQuotas, storages, systemOptions } from '../../db/schema'
|
||||
import { currentTrafficPeriod } from '../../domain/quota'
|
||||
import type { Database } from '../../platform/interface'
|
||||
import type { CurrentStoragePlan, EffectiveQuota, QuotaRepo } from '../../usecases/ports'
|
||||
@@ -15,7 +16,7 @@ async function getEffectiveQuota(db: Database, orgId: string, now = new Date()):
|
||||
// Batch variant of getEffectiveQuota for list views. Resolves every org with two
|
||||
// queries total (quota rows + active entitlements) instead of ~8 per org, then
|
||||
// aggregates in memory. Returns one entry per requested orgId, even with no rows.
|
||||
async function getEffectiveQuotasByOrg(
|
||||
export async function getEffectiveQuotasByOrg(
|
||||
db: Database,
|
||||
orgIds: string[],
|
||||
now = new Date(),
|
||||
@@ -53,6 +54,7 @@ async function getEffectiveQuotasByOrg(
|
||||
orgId: orgQuotaEntitlements.orgId,
|
||||
resourceType: orgQuotaEntitlements.resourceType,
|
||||
entitlementType: orgQuotaEntitlements.entitlementType,
|
||||
source: orgQuotaEntitlements.source,
|
||||
sourceId: orgQuotaEntitlements.sourceId,
|
||||
bytes: orgQuotaEntitlements.bytes,
|
||||
startsAt: orgQuotaEntitlements.startsAt,
|
||||
@@ -125,6 +127,7 @@ function chunk<T>(items: T[], size: number): T[][] {
|
||||
interface EntitlementRow {
|
||||
resourceType: string
|
||||
entitlementType: string
|
||||
source: string
|
||||
sourceId: string
|
||||
bytes: number
|
||||
startsAt: Date
|
||||
@@ -134,10 +137,11 @@ interface EntitlementRow {
|
||||
|
||||
// Mirrors activePlanEntitlement's ORDER BY bytes DESC, startsAt DESC LIMIT 1.
|
||||
function pickPlanEntitlement(ents: EntitlementRow[], resourceType: 'storage' | 'traffic'): PlanEntitlement | null {
|
||||
const plans = ents
|
||||
.filter((e) => e.resourceType === resourceType && e.entitlementType === 'plan')
|
||||
.sort((a, b) => b.bytes - a.bytes || b.startsAt.getTime() - a.startsAt.getTime())
|
||||
const row = plans[0]
|
||||
const plans = ents.filter((e) => e.resourceType === resourceType && e.entitlementType === 'plan')
|
||||
const paidPlans = plans.filter((plan) => plan.source !== 'free_plan')
|
||||
const row = (paidPlans.length > 0 ? paidPlans : plans).sort(
|
||||
(a, b) => b.bytes - a.bytes || b.startsAt.getTime() - a.startsAt.getTime(),
|
||||
)[0]
|
||||
return row ? toPlanEntitlement(row) : null
|
||||
}
|
||||
|
||||
@@ -169,17 +173,92 @@ async function resetExpiredTrafficQuotas(db: Database, now = new Date()): Promis
|
||||
.where(sql`${orgQuotas.trafficPeriod} != ${period}`)
|
||||
}
|
||||
|
||||
async function reconcileFreePlanBaselines(db: Database, now = new Date()): Promise<void> {
|
||||
const rows = await db
|
||||
.select({ key: systemOptions.key, value: systemOptions.value })
|
||||
.from(systemOptions)
|
||||
.where(inArray(systemOptions.key, ['default_org_quota', 'default_team_quota', 'default_org_monthly_traffic_quota']))
|
||||
const options = new Map(rows.map((row) => [row.key, row.value]))
|
||||
const orgQuota = positiveOption(options.get('default_org_quota'), DEFAULT_ORG_QUOTA)
|
||||
const teamQuota = positiveOption(options.get('default_team_quota'), orgQuota)
|
||||
const trafficQuota = nonNegativeOption(options.get('default_org_monthly_traffic_quota'), DEFAULT_ORG_TRAFFIC_QUOTA)
|
||||
const timestamp = now.getTime()
|
||||
|
||||
await db.run(sql`
|
||||
UPDATE org_quota_entitlements
|
||||
SET status = 'active', expires_at = NULL, updated_at = ${timestamp}
|
||||
WHERE source = 'free_plan' AND entitlement_type = 'plan'
|
||||
AND EXISTS (SELECT 1 FROM organization o WHERE o.id = org_quota_entitlements.org_id)
|
||||
AND (status <> 'active' OR expires_at IS NOT NULL)
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quota_entitlements (
|
||||
id, org_id, resource_type, entitlement_type, source, source_id, bytes,
|
||||
starts_at, expires_at, status, metadata, created_at, updated_at
|
||||
)
|
||||
SELECT
|
||||
lower(hex(randomblob(16))), o.id, 'storage', 'plan', 'free_plan', 'free_plan:' || o.id,
|
||||
CASE WHEN json_valid(o.metadata) = 1 AND json_extract(o.metadata, '$.type') = 'team'
|
||||
THEN ${teamQuota} ELSE ${orgQuota} END,
|
||||
${timestamp}, NULL, 'active',
|
||||
json_object(
|
||||
'packageName', 'Free', 'packageId', NULL, 'source', 'free_plan',
|
||||
'settingKey', CASE WHEN json_valid(o.metadata) = 1 AND json_extract(o.metadata, '$.type') = 'team'
|
||||
THEN 'default_team_quota' ELSE 'default_org_quota' END
|
||||
),
|
||||
${timestamp}, ${timestamp}
|
||||
FROM organization o
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM org_quota_entitlements e
|
||||
WHERE e.org_id = o.id AND e.resource_type = 'storage'
|
||||
AND e.entitlement_type = 'plan' AND e.source = 'free_plan'
|
||||
)
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quota_entitlements (
|
||||
id, org_id, resource_type, entitlement_type, source, source_id, bytes,
|
||||
starts_at, expires_at, status, metadata, created_at, updated_at
|
||||
)
|
||||
SELECT
|
||||
lower(hex(randomblob(16))), o.id, 'traffic', 'plan', 'free_plan', 'free_plan:' || o.id,
|
||||
${trafficQuota}, ${timestamp}, NULL, 'active',
|
||||
json_object(
|
||||
'packageName', 'Free', 'packageId', NULL, 'source', 'free_plan',
|
||||
'settingKey', 'default_org_monthly_traffic_quota'
|
||||
),
|
||||
${timestamp}, ${timestamp}
|
||||
FROM organization o
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM org_quota_entitlements e
|
||||
WHERE e.org_id = o.id AND e.resource_type = 'traffic'
|
||||
AND e.entitlement_type = 'plan' AND e.source = 'free_plan'
|
||||
)
|
||||
`)
|
||||
}
|
||||
|
||||
function positiveOption(value: string | undefined, fallback: number): number {
|
||||
const parsed = Number(value)
|
||||
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback
|
||||
}
|
||||
|
||||
function nonNegativeOption(value: string | undefined, fallback: number): number {
|
||||
if (value === undefined || value.trim() === '') return fallback
|
||||
const parsed = Number(value)
|
||||
return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback
|
||||
}
|
||||
|
||||
async function hasQuotaForBytes(db: Database, orgId: string, bytes: number): Promise<boolean> {
|
||||
if (bytes <= 0) return true
|
||||
const quota = await getEffectiveQuota(db, orgId)
|
||||
if (quota.baseQuota === 0 && quota.entitlementQuota === 0) return true
|
||||
if (quota.quota <= 0) return false
|
||||
return quota.used + bytes <= quota.quota
|
||||
}
|
||||
|
||||
async function hasTrafficQuotaForBytes(db: Database, orgId: string, bytes: number, now = new Date()): Promise<boolean> {
|
||||
if (bytes <= 0) return true
|
||||
const quota = await getEffectiveQuota(db, orgId, now)
|
||||
if (quota.baseTrafficQuota === 0 && quota.entitlementTrafficQuota === 0) return true
|
||||
const trafficPlan = await activePlanEntitlement(db, orgId, 'traffic', now)
|
||||
if (quota.trafficQuota === 0) return trafficPlan !== null
|
||||
if ((quota.currentPlan?.trafficOveragePriceCents ?? 0) > 0) return true
|
||||
return quota.trafficUsed + bytes <= quota.trafficQuota
|
||||
}
|
||||
@@ -197,19 +276,14 @@ async function consumeTrafficIfQuotaAllows(
|
||||
.from(orgQuotas)
|
||||
.where(eq(orgQuotas.orgId, orgId))
|
||||
.limit(1)
|
||||
if (quotaRows.length === 0) return true
|
||||
if (quotaRows.length === 0) return false
|
||||
|
||||
const trafficOverageAllowed = await hasActiveTrafficOverage(db, orgId, now)
|
||||
const overageAllowedSql = trafficOverageAllowed ? sql`1 = 1` : sql`1 = 0`
|
||||
|
||||
if (quotaRows[0].trafficPeriod !== period) {
|
||||
const limitBytes = activeEntitlementBytesSql({
|
||||
aggregate: sql`SUM(${orgQuotaEntitlements.bytes})`,
|
||||
orgId,
|
||||
resourceType: 'traffic',
|
||||
now,
|
||||
sourceCondition: sql`1 = 1`,
|
||||
})
|
||||
const limitBytes = activeEntitlementBytesSql(orgId, 'traffic', now)
|
||||
const hasPlan = activePlanEntitlementExistsSql(orgId, 'traffic', now)
|
||||
const updated = await db
|
||||
.update(orgQuotas)
|
||||
.set({ trafficUsed: bytes, trafficPeriod: period })
|
||||
@@ -217,7 +291,7 @@ async function consumeTrafficIfQuotaAllows(
|
||||
sql`${orgQuotas.orgId} = ${orgId}
|
||||
AND ${orgQuotas.trafficPeriod} != ${period}
|
||||
AND (
|
||||
(${limitBytes} = 0)
|
||||
(${limitBytes} = 0 AND ${hasPlan})
|
||||
OR ${overageAllowedSql}
|
||||
OR ${bytes} <= ${limitBytes}
|
||||
)`,
|
||||
@@ -226,13 +300,8 @@ async function consumeTrafficIfQuotaAllows(
|
||||
if (updated.length > 0) return true
|
||||
}
|
||||
|
||||
const limitBytes = activeEntitlementBytesSql({
|
||||
aggregate: sql`SUM(${orgQuotaEntitlements.bytes})`,
|
||||
orgId,
|
||||
resourceType: 'traffic',
|
||||
now,
|
||||
sourceCondition: sql`1 = 1`,
|
||||
})
|
||||
const limitBytes = activeEntitlementBytesSql(orgId, 'traffic', now)
|
||||
const hasPlan = activePlanEntitlementExistsSql(orgId, 'traffic', now)
|
||||
const updated = await db
|
||||
.update(orgQuotas)
|
||||
.set({ trafficUsed: sql`${orgQuotas.trafficUsed} + ${bytes}` })
|
||||
@@ -240,7 +309,7 @@ async function consumeTrafficIfQuotaAllows(
|
||||
sql`${orgQuotas.orgId} = ${orgId}
|
||||
AND ${orgQuotas.trafficPeriod} = ${period}
|
||||
AND (
|
||||
(${limitBytes} = 0)
|
||||
(${limitBytes} = 0 AND ${hasPlan})
|
||||
OR ${overageAllowedSql}
|
||||
OR ${orgQuotas.trafficUsed} + ${bytes} <= ${limitBytes}
|
||||
)`,
|
||||
@@ -276,28 +345,19 @@ async function incrementUsageIfEffectiveQuotaAllows(
|
||||
): Promise<boolean> {
|
||||
if (teamQuotaEnabled) {
|
||||
const rows = await db.select({ id: orgQuotas.id }).from(orgQuotas).where(eq(orgQuotas.orgId, orgId)).limit(1)
|
||||
if (rows.length > 0) {
|
||||
const limitBytes = activeEntitlementBytesSql({
|
||||
aggregate: sql`SUM(${orgQuotaEntitlements.bytes})`,
|
||||
orgId,
|
||||
resourceType: 'storage',
|
||||
now,
|
||||
sourceCondition: sql`1 = 1`,
|
||||
})
|
||||
const updated = await db
|
||||
.update(orgQuotas)
|
||||
.set({ used: sql`${orgQuotas.used} + ${bytes}` })
|
||||
.where(
|
||||
sql`${orgQuotas.orgId} = ${orgId}
|
||||
AND (
|
||||
(${limitBytes} = 0)
|
||||
OR ${orgQuotas.used} + ${bytes} <= ${limitBytes}
|
||||
)`,
|
||||
)
|
||||
.returning({ id: orgQuotas.id })
|
||||
if (rows.length === 0) return false
|
||||
const limitBytes = activeEntitlementBytesSql(orgId, 'storage', now)
|
||||
const updated = await db
|
||||
.update(orgQuotas)
|
||||
.set({ used: sql`${orgQuotas.used} + ${bytes}` })
|
||||
.where(
|
||||
sql`${orgQuotas.orgId} = ${orgId}
|
||||
AND ${limitBytes} > 0
|
||||
AND ${orgQuotas.used} + ${bytes} <= ${limitBytes}`,
|
||||
)
|
||||
.returning({ id: orgQuotas.id })
|
||||
|
||||
if (updated.length === 0) return false
|
||||
}
|
||||
if (updated.length === 0) return false
|
||||
}
|
||||
|
||||
await db
|
||||
@@ -323,7 +383,10 @@ async function activePlanEntitlement(
|
||||
})
|
||||
.from(orgQuotaEntitlements)
|
||||
.where(activePlanEntitlementWhere(orgId, resourceType, now))
|
||||
.orderBy(sql`${orgQuotaEntitlements.bytes} DESC, ${orgQuotaEntitlements.startsAt} DESC`)
|
||||
.orderBy(
|
||||
sql`CASE WHEN ${orgQuotaEntitlements.source} = 'free_plan' THEN 1 ELSE 0 END`,
|
||||
sql`${orgQuotaEntitlements.bytes} DESC, ${orgQuotaEntitlements.startsAt} DESC`,
|
||||
)
|
||||
.limit(1)
|
||||
|
||||
const row = rows[0]
|
||||
@@ -369,29 +432,36 @@ function activeEntitlementWhere(
|
||||
)
|
||||
}
|
||||
|
||||
function activeEntitlementBytesSql({
|
||||
aggregate,
|
||||
orgId,
|
||||
resourceType,
|
||||
now,
|
||||
sourceCondition,
|
||||
}: {
|
||||
aggregate: ReturnType<typeof sql>
|
||||
orgId: string
|
||||
resourceType: 'storage' | 'traffic'
|
||||
now: Date
|
||||
sourceCondition: ReturnType<typeof sql>
|
||||
}) {
|
||||
function activeEntitlementBytesSql(orgId: string, resourceType: 'storage' | 'traffic', now: Date) {
|
||||
const timestamp = now.getTime()
|
||||
return sql`(
|
||||
SELECT COALESCE(${aggregate}, 0)
|
||||
SELECT
|
||||
COALESCE(
|
||||
MAX(CASE WHEN ${orgQuotaEntitlements.entitlementType} = 'plan' AND ${orgQuotaEntitlements.source} <> 'free_plan' THEN ${orgQuotaEntitlements.bytes} END),
|
||||
MAX(CASE WHEN ${orgQuotaEntitlements.entitlementType} = 'plan' THEN ${orgQuotaEntitlements.bytes} END),
|
||||
0
|
||||
)
|
||||
+ COALESCE(SUM(CASE WHEN ${orgQuotaEntitlements.entitlementType} <> 'plan' THEN ${orgQuotaEntitlements.bytes} ELSE 0 END), 0)
|
||||
FROM ${orgQuotaEntitlements}
|
||||
WHERE ${orgQuotaEntitlements.orgId} = ${orgId}
|
||||
AND ${orgQuotaEntitlements.resourceType} = ${resourceType}
|
||||
AND ${orgQuotaEntitlements.status} = 'active'
|
||||
AND ${orgQuotaEntitlements.startsAt} <= ${timestamp}
|
||||
AND (${orgQuotaEntitlements.expiresAt} IS NULL OR ${orgQuotaEntitlements.expiresAt} > ${timestamp})
|
||||
AND ${sourceCondition}
|
||||
)`
|
||||
}
|
||||
|
||||
function activePlanEntitlementExistsSql(orgId: string, resourceType: 'storage' | 'traffic', now: Date) {
|
||||
const timestamp = now.getTime()
|
||||
return sql`EXISTS (
|
||||
SELECT 1
|
||||
FROM ${orgQuotaEntitlements}
|
||||
WHERE ${orgQuotaEntitlements.orgId} = ${orgId}
|
||||
AND ${orgQuotaEntitlements.resourceType} = ${resourceType}
|
||||
AND ${orgQuotaEntitlements.entitlementType} = 'plan'
|
||||
AND ${orgQuotaEntitlements.status} = 'active'
|
||||
AND ${orgQuotaEntitlements.startsAt} <= ${timestamp}
|
||||
AND (${orgQuotaEntitlements.expiresAt} IS NULL OR ${orgQuotaEntitlements.expiresAt} > ${timestamp})
|
||||
)`
|
||||
}
|
||||
|
||||
@@ -466,6 +536,7 @@ export function createQuotaRepo(db: Database): QuotaRepo {
|
||||
listOrgQuotaOverview: () => listOrgQuotaOverview(db),
|
||||
getEffectiveQuota: (orgId, now) => getEffectiveQuota(db, orgId, now),
|
||||
getEffectiveQuotasByOrg: (orgIds, now) => getEffectiveQuotasByOrg(db, orgIds, now),
|
||||
reconcileFreePlanBaselines: (now) => reconcileFreePlanBaselines(db, now),
|
||||
resetExpiredTrafficQuotas: (now) => resetExpiredTrafficQuotas(db, now),
|
||||
hasQuotaForBytes: (orgId, bytes) => hasQuotaForBytes(db, orgId, bytes),
|
||||
hasTrafficQuotaForBytes: (orgId, bytes, now) => hasTrafficQuotaForBytes(db, orgId, bytes, now),
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { eq } from 'drizzle-orm'
|
||||
import { and, eq } from 'drizzle-orm'
|
||||
import { nanoid } from 'nanoid'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { DirType } from '../../../shared/constants'
|
||||
@@ -329,7 +329,10 @@ describe('recordView', () => {
|
||||
const resolved = await resolveShareByToken(db, share.token)
|
||||
if (resolved.status !== 'ok') throw new Error('expected found')
|
||||
expect(resolved.share.views).toBe(2)
|
||||
const events = await db.select().from(activityEvents).where(eq(activityEvents.targetId, share.id))
|
||||
const events = await db
|
||||
.select()
|
||||
.from(activityEvents)
|
||||
.where(and(eq(activityEvents.targetId, share.id), eq(activityEvents.action, 'share_view')))
|
||||
expect(events).toHaveLength(2)
|
||||
expect(events.every((event) => event.action === 'share_view')).toBe(true)
|
||||
})
|
||||
@@ -355,7 +358,12 @@ describe('recordView', () => {
|
||||
const resolved = await resolveShareByToken(db, share.token)
|
||||
if (resolved.status !== 'ok') throw new Error('expected found')
|
||||
expect(resolved.share.views).toBe(0)
|
||||
await expect(db.select().from(activityEvents).where(eq(activityEvents.targetId, share.id))).resolves.toEqual([])
|
||||
await expect(
|
||||
db
|
||||
.select()
|
||||
.from(activityEvents)
|
||||
.where(and(eq(activityEvents.targetId, share.id), eq(activityEvents.action, 'share_view'))),
|
||||
).resolves.toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ import {
|
||||
type ShareResolution,
|
||||
} from '../../usecases/ports'
|
||||
import { activityEventValues } from './activity'
|
||||
import { adminStatsFactValues } from './admin-stats-fact'
|
||||
import { createQuotaRepo } from './quota'
|
||||
|
||||
function buildPath(parent: string, name: string): string {
|
||||
@@ -63,7 +64,22 @@ export function createShareRepo(db: Database): ShareRepo {
|
||||
createdAt: now,
|
||||
}
|
||||
|
||||
const queries: AtomicQuery[] = [db.insert(shares).values(share)]
|
||||
const queries: AtomicQuery[] = [
|
||||
db.insert(shares).values(share),
|
||||
db
|
||||
.insert(activityEvents)
|
||||
.values(
|
||||
adminStatsFactValues({
|
||||
action: 'stats_share_created',
|
||||
sourceId: share.id,
|
||||
orgId: share.orgId,
|
||||
targetType: 'share',
|
||||
occurredAt: now,
|
||||
metadata: { kind: share.kind },
|
||||
}),
|
||||
)
|
||||
.onConflictDoNothing(),
|
||||
]
|
||||
if (input.recipients && input.recipients.length > 0) {
|
||||
const recipientRows = input.recipients.map((r) => ({
|
||||
id: nanoid(),
|
||||
|
||||
@@ -246,7 +246,7 @@ describe('selectStorage', () => {
|
||||
return createStorageRepo(db).get(created.id)
|
||||
}
|
||||
|
||||
it('returns an active storage with unlimited capacity', async () => {
|
||||
it('returns an active storage whose capacity is not reported', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const created = await seedActive(db)
|
||||
const found = await createStorageRepo(db).select()
|
||||
|
||||
+21
-1
@@ -27,6 +27,7 @@ import {
|
||||
import { generateUserOrgSlug, isPersonalOrgLike } from '../shared/org-slugs'
|
||||
import { createEmailGateway } from './adapters/gateways/email'
|
||||
import { createActivityRepo } from './adapters/repos/activity'
|
||||
import { adminStatsFactValues } from './adapters/repos/admin-stats-fact'
|
||||
import { createInviteRepo } from './adapters/repos/invite'
|
||||
import { createLicenseBindingRepo } from './adapters/repos/license-binding'
|
||||
import { createMemberCountRepo } from './adapters/repos/member-count'
|
||||
@@ -35,7 +36,7 @@ import { createOrgRepo } from './adapters/repos/org'
|
||||
import { createSiteInvitationRepo } from './adapters/repos/site-invitations'
|
||||
import { createSystemOptionsRepo } from './adapters/repos/system-options'
|
||||
import * as authSchema from './db/auth-schema'
|
||||
import { orgQuotaEntitlements, orgQuotas, systemOptions } from './db/schema'
|
||||
import { activityEvents, orgQuotaEntitlements, orgQuotas, systemOptions } from './db/schema'
|
||||
import { executeWriteTransaction } from './db/transaction'
|
||||
import { CAPTCHA_AUTH_ENDPOINTS, type CaptchaConfig } from './domain/captcha'
|
||||
import { currentTrafficPeriod } from './domain/quota'
|
||||
@@ -656,6 +657,12 @@ async function createPersonalOrg(
|
||||
const orgSlug = await generateUniqueOrgSlug(db, generateUserOrgSlug)
|
||||
const quotaValues = await createOrgQuotaValues(db, orgId, now)
|
||||
const entitlementValues = await createFreePlanEntitlementValues(db, orgId, now, false)
|
||||
const [account] = await db
|
||||
.select({ providerId: authSchema.account.providerId })
|
||||
.from(authSchema.account)
|
||||
.where(eq(authSchema.account.userId, user.id))
|
||||
.orderBy(authSchema.account.createdAt, authSchema.account.id)
|
||||
.limit(1)
|
||||
|
||||
await executeWriteTransaction(db, [
|
||||
db.insert(authSchema.organization).values({
|
||||
@@ -674,6 +681,19 @@ async function createPersonalOrg(
|
||||
}),
|
||||
db.insert(orgQuotas).values(quotaValues),
|
||||
...entitlementValues.map((value) => db.insert(orgQuotaEntitlements).values(value)),
|
||||
db
|
||||
.insert(activityEvents)
|
||||
.values(
|
||||
adminStatsFactValues({
|
||||
action: 'stats_user_signup',
|
||||
sourceId: user.id,
|
||||
orgId,
|
||||
targetType: 'user',
|
||||
occurredAt: now,
|
||||
metadata: { provider: account?.providerId ?? 'unknown' },
|
||||
}),
|
||||
)
|
||||
.onConflictDoNothing(),
|
||||
])
|
||||
|
||||
return orgId
|
||||
|
||||
+17
-10
@@ -80,15 +80,19 @@ export const storages = sqliteTable('storages', {
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
|
||||
})
|
||||
|
||||
export const orgQuotas = sqliteTable('org_quotas', {
|
||||
id: text('id').primaryKey(),
|
||||
orgId: text('org_id').notNull(),
|
||||
quota: integer('quota').notNull().default(0),
|
||||
used: integer('used').notNull().default(0),
|
||||
trafficQuota: integer('traffic_quota').notNull().default(0),
|
||||
trafficUsed: integer('traffic_used').notNull().default(0),
|
||||
trafficPeriod: text('traffic_period').notNull().default('1970-01'),
|
||||
})
|
||||
export const orgQuotas = sqliteTable(
|
||||
'org_quotas',
|
||||
{
|
||||
id: text('id').primaryKey(),
|
||||
orgId: text('org_id').notNull(),
|
||||
quota: integer('quota').notNull().default(0),
|
||||
used: integer('used').notNull().default(0),
|
||||
trafficQuota: integer('traffic_quota').notNull().default(0),
|
||||
trafficUsed: integer('traffic_used').notNull().default(0),
|
||||
trafficPeriod: text('traffic_period').notNull().default('1970-01'),
|
||||
},
|
||||
(t) => [uniqueIndex('org_quotas_org_uniq').on(t.orgId)],
|
||||
)
|
||||
|
||||
export const cloudTrafficReports = sqliteTable(
|
||||
'cloud_traffic_reports',
|
||||
@@ -105,6 +109,8 @@ export const cloudTrafficReports = sqliteTable(
|
||||
creditsPerUnit: integer('credits_per_unit'),
|
||||
status: text('status').notNull(),
|
||||
error: text('error'),
|
||||
attemptCount: integer('attempt_count').notNull().default(0),
|
||||
nextRetryAt: integer('next_retry_at', { mode: 'timestamp_ms' }),
|
||||
createdAt: integer('created_at', { mode: 'timestamp_ms' }).notNull(),
|
||||
updatedAt: integer('updated_at', { mode: 'timestamp_ms' }).notNull(),
|
||||
},
|
||||
@@ -112,6 +118,7 @@ export const cloudTrafficReports = sqliteTable(
|
||||
uniqueIndex('cloud_traffic_reports_event_uniq').on(t.eventId),
|
||||
index('cloud_traffic_reports_org_period_idx').on(t.orgId, t.period),
|
||||
index('cloud_traffic_reports_status_idx').on(t.status),
|
||||
index('cloud_traffic_reports_retry_idx').on(t.status, t.nextRetryAt, t.createdAt),
|
||||
index('cloud_traffic_reports_updated_idx').on(t.updatedAt),
|
||||
],
|
||||
)
|
||||
@@ -138,7 +145,7 @@ export const orgQuotaEntitlements = sqliteTable(
|
||||
index('org_quota_entitlements_org_type_idx').on(t.orgId, t.resourceType, t.entitlementType, t.status),
|
||||
uniqueIndex('org_quota_entitlements_active_plan_uniq')
|
||||
.on(t.orgId, t.resourceType, t.entitlementType)
|
||||
.where(sql`status = 'active' AND entitlement_type = 'plan'`),
|
||||
.where(sql`status = 'active' AND entitlement_type = 'plan' AND source <> 'free_plan'`),
|
||||
uniqueIndex('org_quota_entitlements_source_resource_uniq').on(t.source, t.sourceId, t.resourceType),
|
||||
],
|
||||
)
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
export const ROLLUP_VERSION = 2
|
||||
export const ROLLUP_VERSION = 3
|
||||
|
||||
export type AdminStatsRollupScope = 'counters' | 'full'
|
||||
export type AdminStatsRollupScope = 'counters' | 'snapshots' | 'full'
|
||||
|
||||
export interface AdminStatsRollupMetadata {
|
||||
version: number
|
||||
scope: AdminStatsRollupScope
|
||||
quality: 'exact' | 'lower_bound'
|
||||
counterQuality?: 'exact' | 'lower_bound'
|
||||
snapshotQuality?: 'exact' | 'lower_bound'
|
||||
snapshotObservedAt?: string
|
||||
}
|
||||
|
||||
export type AdminStatsMetricKind = 'counter' | 'gauge'
|
||||
@@ -47,10 +50,12 @@ export const ADMIN_STATS_METRICS = {
|
||||
sharePasswordPassed: 'share.password_passed',
|
||||
shareSaved: 'share.saved',
|
||||
shareView: 'share.view',
|
||||
statsDataQualitySnapshot: 'stats.data_quality_snapshot',
|
||||
statsMissingBytes: 'stats.quality_missing_bytes',
|
||||
statsRollupRun: 'stats.rollup_run',
|
||||
storageInventory: 'storage.inventory',
|
||||
storageQuota: 'storage.quota',
|
||||
storageTrashSnapshot: 'storage.trash_snapshot',
|
||||
storageUsed: 'storage.used',
|
||||
trafficReportSnapshot: 'traffic.report_snapshot',
|
||||
transferDownloadFailed: 'transfer.download_failed',
|
||||
@@ -85,10 +90,12 @@ export const ADMIN_STATS_METRIC_REGISTRY = {
|
||||
[M.sharePasswordPassed]: counter(['share_id']),
|
||||
[M.shareSaved]: counter(['actor_type', 'share_id'], true),
|
||||
[M.shareView]: counter(['actor_type', 'share_id']),
|
||||
[M.statsDataQualitySnapshot]: gauge(['kind'], 'events', true),
|
||||
[M.statsMissingBytes]: counter(['direction', 'source']),
|
||||
[M.statsRollupRun]: counter(['outcome']),
|
||||
[M.storageInventory]: gauge(['age_bucket', 'file_type_group', 'size_bucket', 'storage_id'], 'entities', true),
|
||||
[M.storageQuota]: gauge(['status'], null, true),
|
||||
[M.storageTrashSnapshot]: gauge(['storage_id'], 'entities', true),
|
||||
[M.storageUsed]: gauge(['storage_id'], null, true),
|
||||
[M.trafficReportSnapshot]: gauge(['status'], 'entities', true),
|
||||
[M.transferDownloadFailed]: counter(['reason', 'source'], true),
|
||||
@@ -131,9 +138,37 @@ export function parseAdminStatsRollupMetadata(metadata: string | null): AdminSta
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) return null
|
||||
const record = value as Record<string, unknown>
|
||||
if (record.version !== ROLLUP_VERSION) return null
|
||||
if (record.scope !== 'counters' && record.scope !== 'full') return null
|
||||
if (record.scope !== 'counters' && record.scope !== 'snapshots' && record.scope !== 'full') return null
|
||||
if (record.quality !== 'exact' && record.quality !== 'lower_bound') return null
|
||||
return { version: record.version, scope: record.scope, quality: record.quality }
|
||||
if (
|
||||
record.counterQuality !== undefined &&
|
||||
record.counterQuality !== 'exact' &&
|
||||
record.counterQuality !== 'lower_bound'
|
||||
) {
|
||||
return null
|
||||
}
|
||||
if (
|
||||
record.snapshotQuality !== undefined &&
|
||||
record.snapshotQuality !== 'exact' &&
|
||||
record.snapshotQuality !== 'lower_bound'
|
||||
) {
|
||||
return null
|
||||
}
|
||||
const snapshotObservedAt = record.snapshotObservedAt ?? record.observedAt
|
||||
if (
|
||||
snapshotObservedAt !== undefined &&
|
||||
(typeof snapshotObservedAt !== 'string' || !Number.isFinite(Date.parse(snapshotObservedAt)))
|
||||
) {
|
||||
return null
|
||||
}
|
||||
return {
|
||||
version: record.version,
|
||||
scope: record.scope,
|
||||
quality: record.quality,
|
||||
counterQuality: record.counterQuality,
|
||||
snapshotQuality: record.snapshotQuality,
|
||||
snapshotObservedAt,
|
||||
}
|
||||
} catch {
|
||||
return null
|
||||
}
|
||||
|
||||
@@ -157,6 +157,7 @@ console.log('stats.rollup.scheduler.started interval=10m')
|
||||
function writeStatsRollup(): void {
|
||||
void (async () => {
|
||||
try {
|
||||
await deps.quota.reconcileFreePlanBaselines()
|
||||
const results = await deps.adminStats.refreshHourlyRollups(new Date())
|
||||
console.log(
|
||||
JSON.stringify({
|
||||
|
||||
@@ -2,7 +2,7 @@ import type { AdminDashboardOverviewStats } from '@shared/types'
|
||||
import { sql } from 'drizzle-orm'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { createAdminStatsRepo, metricSpec } from '../adapters/repos/admin-stats'
|
||||
import { rebuildAdminStatsHour } from '../adapters/repos/admin-stats-rollup'
|
||||
import { captureAdminStatsSnapshot, rebuildAdminStatsHour } from '../adapters/repos/admin-stats-rollup'
|
||||
import { currentTrafficPeriod } from '../domain/quota'
|
||||
import { adminHeaders, createTestApp, seedProLicense } from '../test/setup.js'
|
||||
|
||||
@@ -155,10 +155,10 @@ describe('site stats routes', () => {
|
||||
)
|
||||
VALUES (
|
||||
'storage-marker-2026-01-01', ${Date.UTC(2026, 0, 1)}, '', 'stats.rollup_run', '', '',
|
||||
1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
|
||||
1, 0, 0, '{"version":3,"scope":"full","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
|
||||
), (
|
||||
'storage-used-2026-01-01', ${Date.UTC(2026, 0, 1)}, '', 'storage.used', '', '',
|
||||
0, 4096, 0, '{"version":2,"scope":"full","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
|
||||
0, 4096, 0, '{"version":3,"scope":"snapshots","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
|
||||
)
|
||||
`)
|
||||
|
||||
@@ -191,9 +191,9 @@ describe('site stats routes', () => {
|
||||
count, bytes, unique_count, metadata, updated_at
|
||||
) VALUES
|
||||
('hourly-reader-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
('hourly-reader-upload', ${at}, ${orgId}, 'transfer.upload', 'status', 'success', 1, 999, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000})
|
||||
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000})
|
||||
`)
|
||||
|
||||
const query =
|
||||
@@ -228,18 +228,18 @@ describe('site stats routes', () => {
|
||||
(id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
|
||||
count, bytes, unique_count, metadata, updated_at)
|
||||
VALUES
|
||||
('dimensions-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-share-total', ${at}, ${orgId}, 'share.created', '', '', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-share-kind', ${at}, ${orgId}, 'share.created', 'kind', 'landing', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-job-total', ${at}, ${orgId}, 'background_job.finished', '', '', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-job-outcome', ${at}, ${orgId}, 'background_job.finished', 'outcome', 'failed', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-download-total', ${at}, ${orgId}, 'transfer.download_issued', '', '', 1, 10, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-download-source', ${at}, ${orgId}, 'transfer.download_issued', 'source', 'object_download', 1, 10, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-failure-total', ${at}, ${orgId}, 'transfer.download_failed', '', '', 1, 4, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-failure-reason', ${at}, ${orgId}, 'transfer.download_failed', 'reason', 'network', 1, 4, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-quality-total', ${at}, ${orgId}, 'stats.quality_missing_bytes', '', '', 5, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-quality-upload', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'upload', 2, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-quality-download', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'download', 3, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at})
|
||||
('dimensions-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0, '{"version":3,"scope":"full","quality":"exact"}', ${at}),
|
||||
('dimensions-share-total', ${at}, ${orgId}, 'share.created', '', '', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-share-kind', ${at}, ${orgId}, 'share.created', 'kind', 'landing', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-job-total', ${at}, ${orgId}, 'background_job.finished', '', '', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-job-outcome', ${at}, ${orgId}, 'background_job.finished', 'outcome', 'failed', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-download-total', ${at}, ${orgId}, 'transfer.download_issued', '', '', 1, 10, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-download-source', ${at}, ${orgId}, 'transfer.download_issued', 'source', 'object_download', 1, 10, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-failure-total', ${at}, ${orgId}, 'transfer.download_failed', '', '', 1, 4, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-failure-reason', ${at}, ${orgId}, 'transfer.download_failed', 'reason', 'network', 1, 4, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-quality-total', ${at}, ${orgId}, 'stats.quality_missing_bytes', '', '', 5, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-quality-upload', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'upload', 2, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
|
||||
('dimensions-quality-download', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'download', 3, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at})
|
||||
`)
|
||||
const query = 'from=2026-07-01T10%3A00%3A00.000Z&to=2026-07-01T10%3A59%3A59.999Z&timeZone=UTC'
|
||||
|
||||
@@ -293,13 +293,13 @@ describe('site stats routes', () => {
|
||||
ORDER BY bucket_start, org_id
|
||||
`)
|
||||
|
||||
expect(rows.map((row) => Number(row.bucketStart))).toContain(Date.UTC(2026, 6, 10, 17))
|
||||
expect(rows.map((row) => Number(row.bucketStart))).toContain(Date.UTC(2026, 6, 10, 18))
|
||||
expect(
|
||||
rows
|
||||
.filter((row) => Number(row.bucketStart) === Date.UTC(2026, 6, 10, 17))
|
||||
.filter((row) => Number(row.bucketStart) === Date.UTC(2026, 6, 10, 18))
|
||||
.reduce((sum, row) => sum + row.bytes, 0),
|
||||
).toBe(expected)
|
||||
expect(JSON.parse(rows.at(-1)?.metadata ?? '{}')).toMatchObject({ version: 2, quality: 'exact' })
|
||||
expect(JSON.parse(rows.at(-1)?.metadata ?? '{}')).toMatchObject({ version: 3, quality: 'exact' })
|
||||
})
|
||||
|
||||
it('does not write rollups while serving storage stats', async () => {
|
||||
@@ -362,15 +362,15 @@ describe('site stats routes', () => {
|
||||
count, bytes, unique_count, metadata, updated_at)
|
||||
VALUES
|
||||
('growth-rollup-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
('growth-rollup-total', ${at}, '', 'user.signup', '', '', 2, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000}),
|
||||
('growth-inventory-total', ${at}, '', 'user.inventory', '', '', 2, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
'{"version":3,"scope":"snapshots","quality":"exact"}', ${at + 3_600_000}),
|
||||
('growth-rollup-credential', ${at}, '', 'user.signup', 'provider', 'credential', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000}),
|
||||
('growth-rollup-github', ${at}, '', 'user.signup', 'provider', 'github', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000})
|
||||
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000})
|
||||
`)
|
||||
|
||||
const res = await app.request('/api/site/stats/growth?from=2026-01-01&to=2026-01-01', { headers })
|
||||
@@ -409,7 +409,8 @@ describe('site stats routes', () => {
|
||||
const beforeRes = await app.request('/api/site/stats/storage', { headers })
|
||||
const before = (await beforeRes.json()) as { typeBreakdown: Array<{ type: string; files: number; bytes: number }> }
|
||||
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
|
||||
await captureAdminStatsSnapshot(db, bucketStart, new Date(bucketStart.getTime() + 45 * 60_000))
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date())
|
||||
const afterRes = await app.request('/api/site/stats/storage', { headers })
|
||||
const after = (await afterRes.json()) as { typeBreakdown: Array<{ type: string; files: number; bytes: number }> }
|
||||
|
||||
@@ -425,6 +426,17 @@ describe('site stats routes', () => {
|
||||
await seedProLicense(db)
|
||||
const { bucketStart } = await seedStatsFixture(db)
|
||||
for (let index = 0; index < 12; index += 1) {
|
||||
await db.run(sql`
|
||||
INSERT INTO organization (id, name, slug, metadata, created_at, updated_at)
|
||||
VALUES (
|
||||
${`quota-pressure-org-${index}`},
|
||||
${`Quota Pressure ${index}`},
|
||||
${`quota-pressure-${index}`},
|
||||
'{"type":"team"}',
|
||||
${bucketStart.getTime()},
|
||||
${bucketStart.getTime()}
|
||||
)
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES (
|
||||
@@ -437,19 +449,55 @@ describe('site stats routes', () => {
|
||||
'2026-07'
|
||||
)
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quota_entitlements
|
||||
(id, org_id, resource_type, entitlement_type, source, source_id, bytes, starts_at, status, created_at, updated_at)
|
||||
VALUES (
|
||||
${`quota-pressure-plan-${index}`},
|
||||
${`quota-pressure-org-${index}`},
|
||||
'storage',
|
||||
'plan',
|
||||
'test',
|
||||
${`quota-pressure-source-${index}`},
|
||||
100,
|
||||
${bucketStart.getTime()},
|
||||
'active',
|
||||
${bucketStart.getTime()},
|
||||
${bucketStart.getTime()}
|
||||
)
|
||||
`)
|
||||
}
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
|
||||
await db.run(sql`
|
||||
INSERT INTO organization (id, name, slug, metadata, created_at, updated_at)
|
||||
VALUES ('quota-invalid-org', 'Invalid Quota', 'quota-invalid', '{"type":"team"}', ${bucketStart.getTime()}, ${bucketStart.getTime()})
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES ('quota-invalid', 'quota-invalid-org', 0, 1000, 0, 0, '2026-07')
|
||||
`)
|
||||
await captureAdminStatsSnapshot(db, bucketStart, new Date(bucketStart.getTime() + 45 * 60_000))
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date())
|
||||
|
||||
const res = await app.request('/api/site/stats/storage', { headers })
|
||||
const body = (await res.json()) as {
|
||||
summary: { nearQuotaSpaces: number; overQuotaSpaces: number }
|
||||
topSpaces: Array<{ orgId: string }>
|
||||
summary: {
|
||||
quotaBytes: number | null
|
||||
storageUtilization: number | null
|
||||
nearQuotaSpaces: number
|
||||
overQuotaSpaces: number
|
||||
invalidQuotaSpaces: number
|
||||
}
|
||||
topSpaces: Array<{ orgId: string; utilization: number | null }>
|
||||
}
|
||||
|
||||
expect(res.status).toBe(200)
|
||||
expect(body.summary.nearQuotaSpaces).toBe(9)
|
||||
expect(body.summary.overQuotaSpaces).toBe(3)
|
||||
expect(body.summary.invalidQuotaSpaces).toBe(2)
|
||||
expect(body.summary.quotaBytes).toBeNull()
|
||||
expect(body.summary.storageUtilization).toBeNull()
|
||||
expect(body.topSpaces).toHaveLength(8)
|
||||
expect(body.topSpaces[0]).toMatchObject({ orgId: 'quota-invalid-org', utilization: null })
|
||||
})
|
||||
|
||||
it('reads historical active users from completed snapshots instead of raw activity', async () => {
|
||||
@@ -471,9 +519,11 @@ describe('site stats routes', () => {
|
||||
count, bytes, unique_count, metadata, updated_at)
|
||||
VALUES
|
||||
('active-snapshot-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
'{"version":3,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
|
||||
('active-snapshot-total', ${at}, '', 'user.active_snapshot', '', '', 1, 0, 0,
|
||||
'{"version":3,"scope":"snapshots","quality":"exact"}', ${at + 3_600_000}),
|
||||
('active-snapshot-mau', ${at}, '', 'user.active_snapshot', 'window', 'mau', 1, 0, 0,
|
||||
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000})
|
||||
'{"version":3,"scope":"snapshots","quality":"exact"}', ${at + 3_600_000})
|
||||
`)
|
||||
|
||||
const res = await app.request('/api/site/stats/overview?from=2026-01-02&to=2026-01-02', { headers })
|
||||
@@ -493,7 +543,7 @@ describe('site stats routes', () => {
|
||||
VALUES ('upload-cancel-rate', ${orgId}, ${userId}, 'user', 'upload_cancel', 'file', 'cancel.bin',
|
||||
'{', ${eventSec})
|
||||
`)
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date())
|
||||
|
||||
const current = await app.request('/api/site/stats/traffic', { headers })
|
||||
const currentBody = (await current.json()) as {
|
||||
@@ -523,8 +573,8 @@ describe('site stats routes', () => {
|
||||
('missing-previous-download-bytes', ${orgId}, ${userId}, 'user', 'share_download', 'share', 'previous.bin', '{}',
|
||||
${Math.floor(Date.parse('2026-06-30T12:00:00.000Z') / 1000)})
|
||||
`)
|
||||
await rebuildAdminStatsHour(db, new Date('2026-07-01T12:00:00.000Z'), new Date(), false)
|
||||
await rebuildAdminStatsHour(db, new Date('2026-06-30T12:00:00.000Z'), new Date(), false)
|
||||
await rebuildAdminStatsHour(db, new Date('2026-07-01T12:00:00.000Z'), new Date())
|
||||
await rebuildAdminStatsHour(db, new Date('2026-06-30T12:00:00.000Z'), new Date())
|
||||
|
||||
const res = await app.request('/api/site/stats/overview?from=2026-07-01&to=2026-07-01', { headers })
|
||||
const body = (await res.json()) as { dataQuality: AdminDashboardOverviewStats['dataQuality'] }
|
||||
@@ -611,13 +661,14 @@ describe('site stats routes', () => {
|
||||
VALUES ('operations-cloud-report', ${orgId}, '2026-07', 'object_download', 'stats-file',
|
||||
'operations-cloud-report', 64, 'pending', ${reportAt}, ${reportAt})
|
||||
`)
|
||||
await rebuildAdminStatsHour(db, previousBucket, new Date(reportAt + 3_600_000), true)
|
||||
await rebuildAdminStatsHour(db, previousBucket, new Date(reportAt + 3_600_000))
|
||||
await db.run(sql`
|
||||
UPDATE cloud_traffic_reports
|
||||
SET status = 'reported', updated_at = ${eventMs}
|
||||
WHERE id = 'operations-cloud-report'
|
||||
`)
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
|
||||
await captureAdminStatsSnapshot(db, bucketStart, new Date(bucketStart.getTime() + 45 * 60_000))
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date())
|
||||
|
||||
const res = await app.request('/api/site/stats/operations', { headers })
|
||||
const body = (await res.json()) as {
|
||||
@@ -659,7 +710,12 @@ describe('site stats routes', () => {
|
||||
|
||||
const currentSharingRes = await app.request('/api/site/stats/sharing', { headers })
|
||||
const currentSharing = (await currentSharingRes.json()) as {
|
||||
summary: { views: { value: number }; downloads: { value: number } }
|
||||
dataQuality: { unlocatedViews: number; unlocatedDownloads: number; unlocatedEvents: number }
|
||||
summary: {
|
||||
views: { value: number; change: number | null }
|
||||
downloads: { value: number; change: number | null }
|
||||
downloadsPer100Views: number | null
|
||||
}
|
||||
topShares: Array<{
|
||||
token: string
|
||||
views: number
|
||||
@@ -676,6 +732,10 @@ describe('site stats routes', () => {
|
||||
expect(currentSharingRes.status).toBe(200)
|
||||
expect(currentSharing.summary.views.value).toBe(1)
|
||||
expect(currentSharing.summary.downloads.value).toBe(1)
|
||||
expect(currentSharing.dataQuality).toEqual({ unlocatedViews: 11, unlocatedDownloads: 3, unlocatedEvents: 14 })
|
||||
expect(currentSharing.summary.views.change).toBeNull()
|
||||
expect(currentSharing.summary.downloads.change).toBeNull()
|
||||
expect(currentSharing.summary.downloadsPer100Views).toBeNull()
|
||||
expect(currentSharing.topShares[0]).toMatchObject({
|
||||
token: 'share-token-1',
|
||||
views: 1,
|
||||
@@ -688,6 +748,29 @@ describe('site stats routes', () => {
|
||||
expect(oldSharing.topShares).toEqual([])
|
||||
})
|
||||
|
||||
it('keeps deleted shares in historical rankings without reading their counters live', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
const headers = await adminHeaders(app)
|
||||
await seedProLicense(db)
|
||||
await seedStatsFixture(db)
|
||||
await db.run(sql`DELETE FROM shares WHERE id = 'share-1'`)
|
||||
|
||||
const res = await app.request('/api/site/stats/sharing', { headers })
|
||||
const body = (await res.json()) as {
|
||||
topShares: Array<{ id: string; token: string; name: string; status: string; views: number; downloads: number }>
|
||||
}
|
||||
|
||||
expect(res.status).toBe(200)
|
||||
expect(body.topShares[0]).toMatchObject({
|
||||
id: 'share-1',
|
||||
token: '',
|
||||
name: '已删除的分享',
|
||||
status: 'deleted',
|
||||
views: 1,
|
||||
downloads: 1,
|
||||
})
|
||||
})
|
||||
|
||||
it('orders top share rankings by views before downloads', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
const headers = await adminHeaders(app)
|
||||
@@ -706,14 +789,14 @@ describe('site stats routes', () => {
|
||||
('activity-download-heavy-2', ${orgId}, NULL, 'anonymous', 'share_download', 'share', 'share-download-heavy', 'report.pdf', '{"bytes":512,"source":"landing_share","anonymous":true}', ${eventSec}),
|
||||
('activity-download-heavy-3', ${orgId}, NULL, 'anonymous', 'share_download', 'share', 'share-download-heavy', 'report.pdf', '{"bytes":512,"source":"landing_share","anonymous":true}', ${eventSec})
|
||||
`)
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date())
|
||||
await db.run(sql`
|
||||
INSERT INTO stats_rollups_hourly
|
||||
(id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
|
||||
count, bytes, unique_count, metadata, updated_at)
|
||||
VALUES
|
||||
('ranking-invalid-quality', ${bucketStart.getTime()}, ${orgId}, 'share.view', 'share_id',
|
||||
'share-download-heavy', 1000, 0, 0, '{"version":2,"scope":"full"}', ${bucketStart.getTime()})
|
||||
'share-download-heavy', 1000, 0, 0, '{"version":3,"scope":"counters"}', ${bucketStart.getTime()})
|
||||
`)
|
||||
|
||||
const res = await app.request('/api/site/stats/sharing', { headers })
|
||||
@@ -741,7 +824,7 @@ describe('site stats routes', () => {
|
||||
'{"source":"landing_share"}', ${eventSec})
|
||||
`)
|
||||
}
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date())
|
||||
|
||||
const res = await app.request('/api/site/stats/sharing', { headers })
|
||||
const body = (await res.json()) as { topShares: Array<{ viewPercent: number }> }
|
||||
@@ -801,7 +884,16 @@ async function seedStatsFixture(db: Awaited<ReturnType<typeof createTestApp>>['d
|
||||
('activity-share-view', ${orgId}, NULL, 'anonymous', 'share_view', 'share', 'share-1', 'report.pdf', '{"source":"landing_share","anonymous":true}', ${nowSec}),
|
||||
('activity-share-password', ${orgId}, NULL, 'anonymous', 'share_password_passed', 'share', 'share-1', 'report.pdf', '{"source":"landing_share","anonymous":true}', ${nowSec}),
|
||||
('activity-share-download', ${orgId}, NULL, 'anonymous', 'share_download', 'share', 'share-1', 'report.pdf', '{"bytes":512,"source":"landing_share","anonymous":true}', ${nowSec}),
|
||||
('activity-object-download', ${orgId}, ${userId}, 'user', 'object_download', 'file', 'stats-file', 'report.pdf', '{"bytes":256,"source":"object_download"}', ${nowSec})
|
||||
('activity-object-download', ${orgId}, ${userId}, 'user', 'object_download', 'file', 'stats-file', 'report.pdf', '{"bytes":256,"source":"object_download"}', ${nowSec}),
|
||||
('stats-fixture-share-created', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'share-1', 'share-1', '{"kind":"landing","statsQuality":"exact"}', ${nowSec}),
|
||||
('stats-fixture-task-completed', ${orgId}, NULL, 'system', 'stats_remote_download_finished', 'remote_download', 'task-1', 'task-1', '{"category":"direct","outcome":"completed","bytes":0,"statsQuality":"exact"}', ${nowSec}),
|
||||
('stats-fixture-task-failed', ${orgId}, NULL, 'system', 'stats_remote_download_finished', 'remote_download', 'task-2', 'task-2', '{"category":"direct","outcome":"failed","bytes":0,"statsQuality":"exact"}', ${nowSec}),
|
||||
('stats-fixture-job-failed', ${orgId}, NULL, 'system', 'stats_background_job_finished', 'background_job', 'job-1', 'job-1', '{"jobType":"extract","outcome":"failed","statsQuality":"exact"}', ${nowSec})
|
||||
`)
|
||||
await db.run(sql`
|
||||
UPDATE activity_events
|
||||
SET created_at = ${nowSec}, metadata = '{"provider":"credential","statsQuality":"exact"}'
|
||||
WHERE action = 'stats_user_signup' AND target_id = ${userId}
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO downloaders (id, name, token_hash, token_jti, status, enabled, version, hostname, platform, arch, engine, capabilities, max_concurrent_tasks, current_tasks, download_bps, upload_bps, free_disk_bytes, created_by, last_heartbeat_at, created_at, updated_at)
|
||||
@@ -818,6 +910,7 @@ async function seedStatsFixture(db: Awaited<ReturnType<typeof createTestApp>>['d
|
||||
VALUES ('job-1', ${orgId}, ${userId}, 'extract', 'failed', '', '', NULL, 0, 0, 0, 0, NULL, 'bad zip', NULL, 1, 0, NULL, ${now}, ${now}, ${now}, ${now})
|
||||
`)
|
||||
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(generatedAt), true)
|
||||
await captureAdminStatsSnapshot(db, bucketStart, new Date(now))
|
||||
await rebuildAdminStatsHour(db, bucketStart, new Date(generatedAt))
|
||||
return { orgId, userId, bucketStart, eventMs: now, eventSec: nowSec }
|
||||
}
|
||||
|
||||
@@ -1997,6 +1997,17 @@ describe('Download tasks API integration', () => {
|
||||
runtime: { phase: 'completed' },
|
||||
},
|
||||
})
|
||||
const terminalFacts = await db.all<{ outcome: string; count: number }>(sql`
|
||||
SELECT json_extract(metadata, '$.outcome') AS outcome, COUNT(*) AS count
|
||||
FROM activity_events
|
||||
WHERE action = 'stats_remote_download_finished' AND target_id = ${createdTask.id}
|
||||
GROUP BY outcome
|
||||
ORDER BY outcome
|
||||
`)
|
||||
expect(terminalFacts).toEqual([
|
||||
{ outcome: 'completed', count: 1 },
|
||||
{ outcome: 'failed', count: 1 },
|
||||
])
|
||||
|
||||
const seedingAfterRestartRes = await app.request(`/api/downloads/tasks/${createdTask.id}`, {
|
||||
method: 'PATCH',
|
||||
|
||||
@@ -1381,6 +1381,7 @@ describe('POST /api/objects/:id/transfers', () => {
|
||||
const userId = await getUserIdByEmail(db, 'test@example.com')
|
||||
await insertTeamOrg(db, 'team-a')
|
||||
await insertMember(db, 'team-a', userId, 'editor')
|
||||
await insertStorageEntitlement(db, 'team-a', 10_000_000)
|
||||
await insertFile(db, orgId, { id: 'src-copy', name: 'doc.txt' })
|
||||
|
||||
const res = await transferRequest(app, headers, 'src-copy', { targetOrgId: 'team-a', mode: 'copy' })
|
||||
@@ -1403,10 +1404,12 @@ describe('POST /api/objects/:id/transfers', () => {
|
||||
const userId = await getUserIdByEmail(db, 'test@example.com')
|
||||
await insertTeamOrg(db, 'team-b')
|
||||
await insertMember(db, 'team-b', userId, 'owner')
|
||||
await insertStorageEntitlement(db, 'team-b', 10_000_000)
|
||||
await insertFile(db, orgId, { id: 'src-move', name: 'photo.jpg', size: 1024 })
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES (${`q-${orgId}`}, ${orgId}, ${1024 * 1024}, 1024, 0, 0, '1970-01')
|
||||
UPDATE org_quotas
|
||||
SET quota = ${1024 * 1024}, used = 1024, traffic_quota = 0, traffic_used = 0, traffic_period = '1970-01'
|
||||
WHERE org_id = ${orgId}
|
||||
`)
|
||||
|
||||
const res = await transferRequest(app, headers, 'src-move', { targetOrgId: 'team-b', mode: 'move' })
|
||||
@@ -1430,6 +1433,7 @@ describe('POST /api/objects/:id/transfers', () => {
|
||||
const userId = await getUserIdByEmail(db, 'test@example.com')
|
||||
await insertTeamOrg(db, 'team-c')
|
||||
await insertMember(db, 'team-c', userId, 'editor')
|
||||
await insertStorageEntitlement(db, 'team-c', 10_000_000)
|
||||
await insertFolder(db, orgId, { id: 'fold-1', name: 'Album' })
|
||||
await insertFile(db, orgId, { id: 'in-fold', name: 'pic.png', parent: 'Album' })
|
||||
|
||||
@@ -1738,12 +1742,13 @@ describe('Objects API — quota enforcement', () => {
|
||||
expect(quotaRows[0].used).toBe(500) // unchanged
|
||||
})
|
||||
|
||||
it('returns 201 when no quota row exists (unlimited)', async () => {
|
||||
it('returns 422 when no quota row or entitlement exists', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
const headers = await authedHeaders(app)
|
||||
await insertStorage(db)
|
||||
const orgId = await getOrgId(db)
|
||||
// No org quota row at all — unlimited
|
||||
await db.delete(orgQuotaEntitlements).where(eq(orgQuotaEntitlements.orgId, orgId))
|
||||
await db.delete(orgQuotas).where(eq(orgQuotas.orgId, orgId))
|
||||
await insertFile(db, orgId, { id: 'm-copy-nolimit', name: 'nolimit.txt', size: 100 })
|
||||
|
||||
const res = await app.request('/api/objects/m-copy-nolimit/copies', {
|
||||
@@ -1751,10 +1756,10 @@ describe('Objects API — quota enforcement', () => {
|
||||
headers: { ...headers, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ parent: '' }),
|
||||
})
|
||||
expect(res.status).toBe(201)
|
||||
expect(res.status).toBe(422)
|
||||
})
|
||||
|
||||
it('returns 201 when quota is 0 (unlimited) regardless of file size', async () => {
|
||||
it('returns 422 when effective quota is zero', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
const headers = await authedHeaders(app)
|
||||
await insertStorage(db)
|
||||
@@ -1767,7 +1772,7 @@ describe('Objects API — quota enforcement', () => {
|
||||
headers: { ...headers, 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ parent: '' }),
|
||||
})
|
||||
expect(res.status).toBe(201)
|
||||
expect(res.status).toBe(422)
|
||||
})
|
||||
|
||||
it('returns 404 when source file does not exist', async () => {
|
||||
@@ -2028,7 +2033,7 @@ describe('Objects API — quota enforcement', () => {
|
||||
expect(quotaRows[0]).toEqual({ used: 140, quota: 100 })
|
||||
})
|
||||
|
||||
it('enforces storage entitlements when base quota is unlimited', async () => {
|
||||
it('enforces storage entitlements when the legacy quota column is zero', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
await seedProLicense(db)
|
||||
const headers = await authedHeaders(app)
|
||||
@@ -2092,17 +2097,18 @@ describe('Objects API — quota enforcement', () => {
|
||||
expect(quotaRows[0].used).toBe(50)
|
||||
})
|
||||
|
||||
it('returns 200 when no quota row exists (unlimited)', async () => {
|
||||
it('returns 422 when no quota row or entitlement exists', async () => {
|
||||
const { app, db } = await createTestApp()
|
||||
const headers = await authedHeaders(app)
|
||||
await insertStorage(db)
|
||||
// No quota row — unlimited
|
||||
const orgId = await getOrgId(db)
|
||||
await db.delete(orgQuotaEntitlements).where(eq(orgQuotaEntitlements.orgId, orgId))
|
||||
await db.delete(orgQuotas).where(eq(orgQuotas.orgId, orgId))
|
||||
const ref = await createDraft(app, headers, { name: 'nolimit.txt', size: 5000 })
|
||||
|
||||
const res = await complete(app, headers, ref)
|
||||
expect(res.status).toBe(200)
|
||||
const body = (await res.json()) as Record<string, unknown>
|
||||
expect(body.status).toBe('active')
|
||||
expect(res.status).toBe(422)
|
||||
await expect(res.json()).resolves.toMatchObject({ error: { details: [{ reason: 'QUOTA_EXCEEDED' }] } })
|
||||
})
|
||||
|
||||
it('replaces a same-size file at full quota — net-neutral, incumbent purged', async () => {
|
||||
|
||||
@@ -336,7 +336,7 @@ async function insertStorage(
|
||||
}
|
||||
|
||||
describe('selectStorage service', () => {
|
||||
it('returns the single active storage when capacity is unlimited (0) [spec: storages/select-active]', async () => {
|
||||
it('returns the single active storage when capacity is not reported (0) [spec: storages/select-active]', async () => {
|
||||
const { db } = await createTestApp()
|
||||
await insertStorage(db, { id: 's1', capacity: 0, used: 0 })
|
||||
|
||||
|
||||
@@ -1377,6 +1377,35 @@ describe('Quota Store API', () => {
|
||||
trafficExtraNames: [],
|
||||
currentPlan: { trafficOveragePriceCents: 25 },
|
||||
})
|
||||
|
||||
const freeBaselines = await db.all<{ resourceType: string; status: string }>(sql`
|
||||
SELECT resource_type AS resourceType, status
|
||||
FROM org_quota_entitlements
|
||||
WHERE org_id = ${orgId} AND source = 'free_plan'
|
||||
ORDER BY resource_type
|
||||
`)
|
||||
expect(freeBaselines).toEqual([
|
||||
{ resourceType: 'storage', status: 'active' },
|
||||
{ resourceType: 'traffic', status: 'active' },
|
||||
])
|
||||
|
||||
const revoked = await postWebhook(
|
||||
app,
|
||||
JSON.stringify({
|
||||
eventId: 'evt-subscription-revoked',
|
||||
cloudOrderId: subscriptionSourceId,
|
||||
targetOrgId: orgId,
|
||||
eventType: 'order.quota_changed',
|
||||
direction: 'decrease',
|
||||
storageBytes: 4096,
|
||||
trafficBytes: 2048,
|
||||
source: 'stripe_subscription',
|
||||
}),
|
||||
)
|
||||
expect(revoked.status).toBe(200)
|
||||
const fallbackRes = await app.request('/api/quotas/me', { headers })
|
||||
const fallback = (await fallbackRes.json()) as { baseQuota: number; quota: number; storagePlanName: string }
|
||||
expect(fallback).toMatchObject({ baseQuota: 10 * 1024 * 1024, quota: 10 * 1024 * 1024, storagePlanName: 'Free' })
|
||||
})
|
||||
|
||||
it('renews subscription entitlements by replacing plan bytes and extending expiry [spec: quota-store/webhook-renewal]', async () => {
|
||||
|
||||
@@ -12,7 +12,13 @@ vi.mock('../server/platform/cloudflare', () => ({
|
||||
}))
|
||||
|
||||
const refreshHourlyRollups = vi.fn()
|
||||
const fakeDeps = { instance: 'instance', systemOptions: 'system-options', adminStats: { refreshHourlyRollups } }
|
||||
const reconcileFreePlanBaselines = vi.fn()
|
||||
const fakeDeps = {
|
||||
instance: 'instance',
|
||||
systemOptions: 'system-options',
|
||||
adminStats: { refreshHourlyRollups },
|
||||
quota: { reconcileFreePlanBaselines },
|
||||
}
|
||||
vi.mock('../server/composition', () => ({
|
||||
createDeps: vi.fn(() => fakeDeps),
|
||||
}))
|
||||
@@ -54,6 +60,7 @@ describe('handleScheduled', () => {
|
||||
vi.mocked(runLicensingRefresh).mockReset()
|
||||
mockResetExpiredTrafficQuotas.mockReset()
|
||||
refreshHourlyRollups.mockReset()
|
||||
reconcileFreePlanBaselines.mockReset()
|
||||
})
|
||||
|
||||
it('syncs usage reports on the traffic cron only', async () => {
|
||||
@@ -64,6 +71,7 @@ describe('handleScheduled', () => {
|
||||
cloudBaseUrl: 'https://cloud.example',
|
||||
})
|
||||
expect(refreshHourlyRollups).toHaveBeenCalledOnce()
|
||||
expect(reconcileFreePlanBaselines).toHaveBeenCalledOnce()
|
||||
expect(runLicensingRefresh).not.toHaveBeenCalled()
|
||||
expect(reportInstanceTelemetry).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
@@ -17,6 +17,8 @@ describe('admin stats backfill', () => {
|
||||
const now = new Date('2026-07-10T12:00:00.000Z')
|
||||
const historyStartMs = Date.parse('2026-04-01T00:10:00.000Z')
|
||||
const eventMs = Date.parse('2026-07-10T09:10:00.000Z')
|
||||
const eventHourMs = Date.parse('2026-07-10T09:00:00.000Z')
|
||||
const snapshotObservedAt = '2026-07-10T09:50:00.000Z'
|
||||
const eventSec = Math.floor(eventMs / 1000)
|
||||
const currentHourMs = Date.parse('2026-07-10T12:00:00.000Z')
|
||||
const currentEventSec = Math.floor(Date.parse('2026-07-10T12:10:00.000Z') / 1000)
|
||||
@@ -26,11 +28,14 @@ describe('admin stats backfill', () => {
|
||||
CREATE TABLE user (id TEXT PRIMARY KEY, created_at INTEGER NOT NULL DEFAULT 0);
|
||||
CREATE TABLE account (id TEXT PRIMARY KEY, user_id TEXT NOT NULL, provider_id TEXT NOT NULL, created_at INTEGER NOT NULL);
|
||||
CREATE TABLE session (id TEXT PRIMARY KEY, user_id TEXT NOT NULL, created_at INTEGER NOT NULL);
|
||||
CREATE TABLE organization (id TEXT PRIMARY KEY, created_at INTEGER NOT NULL);
|
||||
CREATE TABLE organization (id TEXT PRIMARY KEY, metadata TEXT, created_at INTEGER NOT NULL);
|
||||
CREATE TABLE member (
|
||||
id TEXT PRIMARY KEY, organization_id TEXT NOT NULL, user_id TEXT NOT NULL, created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE TABLE matters (id TEXT PRIMARY KEY, size INTEGER, dirtype INTEGER);
|
||||
CREATE TABLE shares (
|
||||
id TEXT PRIMARY KEY, kind TEXT NOT NULL, matter_id TEXT NOT NULL, org_id TEXT NOT NULL,
|
||||
status TEXT NOT NULL, expires_at INTEGER, download_limit INTEGER, downloads INTEGER NOT NULL,
|
||||
status TEXT NOT NULL, expires_at INTEGER, download_limit INTEGER, views INTEGER NOT NULL, downloads INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE TABLE activity_events (
|
||||
@@ -69,20 +74,35 @@ describe('admin stats backfill', () => {
|
||||
UNIQUE(bucket_start, org_id, metric_key, dimension_key, dimension_value)
|
||||
);
|
||||
|
||||
INSERT INTO user VALUES ('u0', 0), ('u1', ${historyStartMs});
|
||||
INSERT INTO account VALUES ('a1', 'u1', 'github', ${historyStartMs});
|
||||
INSERT INTO user VALUES ('u0', 0), ('u1', ${historyStartMs}), ('u2', ${historyStartMs + 1000});
|
||||
INSERT INTO account VALUES
|
||||
('a1', 'u1', 'github', ${historyStartMs}),
|
||||
('a2', 'u2', 'github', ${historyStartMs + 1000});
|
||||
INSERT INTO organization VALUES
|
||||
('o1', '{"type":"personal"}', ${historyStartMs}),
|
||||
('o2', '{"type":"personal"}', ${historyStartMs + 1000});
|
||||
INSERT INTO member VALUES
|
||||
('m1', 'o1', 'u1', ${historyStartMs}),
|
||||
('m2', 'o2', 'u2', ${historyStartMs + 1000});
|
||||
INSERT INTO matters VALUES ('f1', 512, 0);
|
||||
INSERT INTO shares VALUES ('s1', 'landing', 'f1', 'o1', 'active', NULL, 10, 1, ${eventSec});
|
||||
INSERT INTO shares VALUES ('s1', 'landing', 'f1', 'o1', 'active', NULL, 10, 0, 1, ${eventSec});
|
||||
INSERT INTO activity_events VALUES
|
||||
('upload-1', 'o1', 'u1', NULL, NULL, 'upload_confirm', 'file', 'f1', 'file.bin', NULL, ${eventSec}),
|
||||
('open-upload', 'o1', 'u1', 'user', NULL, 'upload_confirm', 'file', 'f1', 'file.bin',
|
||||
'{"bytes":512,"source":"upload","status":"success"}', ${currentEventSec}),
|
||||
('share-1', 'o1', NULL, NULL, NULL, 'share_download', 'share', 's1', 'file.bin', '{"anonymous":true}', ${eventSec}),
|
||||
('image-1', 'o1', NULL, NULL, NULL, 'image_hosting_download', 'image', 'img1', 'image.png', NULL, ${eventSec});
|
||||
('image-1', 'o1', NULL, NULL, NULL, 'image_hosting_download', 'image', 'img1', 'image.png', NULL, ${eventSec}),
|
||||
('task-failed', 'o1', 'u1', 'user', NULL, 'download_task_failed', 'remote_download', 't1', 'task', NULL, ${eventSec + 1}),
|
||||
('task-completed', 'o1', 'u1', 'user', NULL, 'download_task_completed', 'remote_download', 't1', 'task', NULL, ${eventSec + 2}),
|
||||
('blocked-download', 'o1', 'u1', 'user', NULL, 'download_failed', 'file', 'f1', 'file.bin',
|
||||
'{"bytes":512,"source":"object_download","reason":"quota_exceeded","trafficEventId":"traffic-3"}', ${eventSec});
|
||||
INSERT INTO cloud_traffic_reports VALUES
|
||||
('r1', 'o1', 'direct_share', 's1', 'traffic-1', 512, NULL, NULL, NULL, 'reported', NULL, ${eventMs}, ${eventMs}),
|
||||
('r2', 'o1', 'image_hosting', 'img1', 'traffic-2', 128, NULL, NULL, NULL, 'reported', NULL, ${eventMs}, ${eventMs});
|
||||
INSERT INTO download_tasks VALUES ('t1', 'o1', 'video', 'url', 'd1', 'completed', 512, ${eventMs}, ${eventMs});
|
||||
('r2', 'o1', 'image_hosting', 'img1', 'traffic-2', 128, NULL, NULL, NULL, 'reported', NULL, ${eventMs}, ${eventMs}),
|
||||
('r3', 'o1', 'object_download', 'f1', 'traffic-3', 512, NULL, NULL, NULL, 'blocked', 'quota_exceeded', ${eventMs}, ${eventMs});
|
||||
INSERT INTO download_tasks VALUES
|
||||
('t1', 'o1', 'video', 'url', 'd1', 'completed', 512, ${eventMs}, ${eventMs}),
|
||||
('t2', 'o1', NULL, 'url', NULL, 'canceled', 0, ${eventMs}, ${eventMs});
|
||||
INSERT INTO org_quotas VALUES ('q1', 512);
|
||||
INSERT INTO stats_rollups_hourly VALUES
|
||||
('legacy-epoch', 0, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
@@ -94,7 +114,11 @@ describe('admin stats backfill', () => {
|
||||
('stale-task', ${eventMs - 3_600_000}, 'o1', 'remote_download.task_finished', '', '', 1, 512, 0,
|
||||
'{"version":2,"scope":"counters","quality":"exact"}', ${eventMs}),
|
||||
('stale-traffic', ${eventMs - 3_600_000}, 'o1', 'traffic.report_sync', '', '', 2, 640, 0,
|
||||
'{"version":2,"scope":"counters","quality":"exact"}', ${eventMs});
|
||||
'{"version":2,"scope":"counters","quality":"exact"}', ${eventMs}),
|
||||
('snapshot-marker', ${eventHourMs}, '', 'stats.rollup_run', '', '', 1, 0, 0,
|
||||
'{"version":3,"scope":"snapshots","quality":"exact","snapshotQuality":"exact","snapshotObservedAt":"${snapshotObservedAt}"}', ${eventMs}),
|
||||
('snapshot-gauge', ${eventHourMs}, '', 'storage.used', '', '', 0, 512, 0,
|
||||
'{"version":3,"scope":"snapshots","quality":"exact","observedAt":"${snapshotObservedAt}"}', ${eventMs});
|
||||
`)
|
||||
|
||||
const sql = buildBackfillSql(now)
|
||||
@@ -127,7 +151,7 @@ describe('admin stats backfill', () => {
|
||||
orphanUserEvents: 0,
|
||||
missingUploadBytes: 0,
|
||||
missingDownloadBytes: 0,
|
||||
trafficEvents: 2,
|
||||
trafficEvents: 3,
|
||||
hourlyRollups: expectedBuckets,
|
||||
rawActiveShares: 1,
|
||||
validActiveShares: 1,
|
||||
@@ -138,14 +162,16 @@ describe('admin stats backfill', () => {
|
||||
openCounterMarkers: 0,
|
||||
rawUploadAttempts: 1,
|
||||
rollupUploadAttempts: 1,
|
||||
rawUserSignups: 1,
|
||||
rollupUserSignups: 1,
|
||||
rawUserSignups: 2,
|
||||
rollupUserSignups: 2,
|
||||
rawSharesCreated: 1,
|
||||
rollupSharesCreated: 1,
|
||||
rawFailedDownloads: 1,
|
||||
rollupFailedDownloads: 1,
|
||||
rawShareDownloads: 1,
|
||||
rollupShareDownloads: 1,
|
||||
rawFinishedDownloadTasks: 1,
|
||||
rollupFinishedDownloadTasks: 1,
|
||||
rawFinishedDownloadTasks: 3,
|
||||
rollupFinishedDownloadTasks: 3,
|
||||
rawMissingByteEvents: 0,
|
||||
rollupMissingByteEvents: 0,
|
||||
})
|
||||
@@ -155,11 +181,21 @@ describe('admin stats backfill', () => {
|
||||
expect(db.prepare("SELECT COUNT(*) AS value FROM activity_events WHERE target_id = 'img1'").get()).toEqual({
|
||||
value: 1,
|
||||
})
|
||||
expect(db.prepare("SELECT COUNT(*) AS value FROM activity_events WHERE id = 'backfill_traffic-3'").get()).toEqual({
|
||||
value: 0,
|
||||
})
|
||||
expect(
|
||||
db
|
||||
.prepare("SELECT json_extract(metadata, '$.scope') AS scope FROM stats_rollups_hourly WHERE id = 'latest-full'")
|
||||
.prepare("SELECT COUNT(*) AS value FROM stats_rollups_hourly WHERE json_extract(metadata, '$.scope') = 'full'")
|
||||
.get(),
|
||||
).toEqual({ scope: 'full' })
|
||||
).toEqual({ value: 1 })
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
"SELECT json_extract(metadata, '$.counterQuality') AS counterQuality, json_extract(metadata, '$.snapshotQuality') AS snapshotQuality, json_extract(metadata, '$.snapshotObservedAt') AS snapshotObservedAt FROM stats_rollups_hourly WHERE id = 'snapshot-marker'",
|
||||
)
|
||||
.get(),
|
||||
).toEqual({ counterQuality: 'lower_bound', snapshotQuality: 'exact', snapshotObservedAt })
|
||||
expect(db.prepare('SELECT COUNT(*) AS value FROM stats_rollups_hourly WHERE bucket_start = 0').get()).toEqual({
|
||||
value: 0,
|
||||
})
|
||||
@@ -172,12 +208,23 @@ describe('admin stats backfill', () => {
|
||||
"SELECT count AS value FROM stats_rollups_hourly WHERE metric_key = 'user.signup' AND dimension_key = 'provider' AND dimension_value = 'github'",
|
||||
)
|
||||
.get(),
|
||||
).toEqual({ value: 1 })
|
||||
).toEqual({ value: 2 })
|
||||
expect(
|
||||
db.prepare("SELECT COUNT(*) AS value FROM stats_rollups_hourly WHERE metric_key = 'traffic.report_sync'").get(),
|
||||
).toEqual({
|
||||
value: 0,
|
||||
})
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
"SELECT json_extract(metadata, '$.outcome') AS outcome, json_extract(metadata, '$.bytes') AS bytes, COUNT(*) AS value FROM activity_events WHERE action = 'stats_remote_download_finished' GROUP BY outcome, bytes ORDER BY outcome",
|
||||
)
|
||||
.all(),
|
||||
).toEqual([
|
||||
{ outcome: 'canceled', bytes: 0, value: 1 },
|
||||
{ outcome: 'completed', bytes: 512, value: 1 },
|
||||
{ outcome: 'failed', bytes: 0, value: 1 },
|
||||
])
|
||||
db.close()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -186,6 +186,7 @@ const APP_SCHEMA_SQL = `
|
||||
traffic_used INTEGER NOT NULL DEFAULT 0,
|
||||
traffic_period TEXT NOT NULL DEFAULT '1970-01'
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS org_quotas_org_uniq ON org_quotas(org_id);
|
||||
CREATE TABLE IF NOT EXISTS cloud_traffic_reports (
|
||||
id TEXT PRIMARY KEY,
|
||||
org_id TEXT NOT NULL,
|
||||
@@ -199,12 +200,15 @@ const APP_SCHEMA_SQL = `
|
||||
credits_per_unit INTEGER,
|
||||
status TEXT NOT NULL,
|
||||
error TEXT,
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
next_retry_at INTEGER,
|
||||
created_at INTEGER NOT NULL,
|
||||
updated_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS cloud_traffic_reports_event_uniq ON cloud_traffic_reports(event_id);
|
||||
CREATE INDEX IF NOT EXISTS cloud_traffic_reports_org_period_idx ON cloud_traffic_reports(org_id, period);
|
||||
CREATE INDEX IF NOT EXISTS cloud_traffic_reports_status_idx ON cloud_traffic_reports(status);
|
||||
CREATE INDEX IF NOT EXISTS cloud_traffic_reports_retry_idx ON cloud_traffic_reports(status, next_retry_at, created_at);
|
||||
CREATE TABLE IF NOT EXISTS org_quota_entitlements (
|
||||
id TEXT PRIMARY KEY,
|
||||
org_id TEXT NOT NULL,
|
||||
@@ -226,7 +230,7 @@ const APP_SCHEMA_SQL = `
|
||||
ON org_quota_entitlements(org_id, resource_type, entitlement_type, status);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS org_quota_entitlements_active_plan_uniq
|
||||
ON org_quota_entitlements(org_id, resource_type, entitlement_type)
|
||||
WHERE status = 'active' AND entitlement_type = 'plan';
|
||||
WHERE status = 'active' AND entitlement_type = 'plan' AND source <> 'free_plan';
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS org_quota_entitlements_source_resource_uniq
|
||||
ON org_quota_entitlements(source, source_id, resource_type);
|
||||
CREATE TABLE IF NOT EXISTS webhook_events (
|
||||
|
||||
@@ -496,11 +496,7 @@ describe('archive processing', () => {
|
||||
it('fails quota checks without creating visible extracted output', async () => {
|
||||
const { db } = await createTestApp()
|
||||
await seedStorage(db)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES ('archive-quota', ${ORG_ID}, 0, 0, 0, 0, '1970-01')
|
||||
`)
|
||||
await seedStoragePlanEntitlement(db, ORG_ID, 4, 'archive-quota-plan')
|
||||
await db.run(sql`UPDATE org_quota_entitlements SET bytes = 4 WHERE id = 'archive-default-quota-plan'`)
|
||||
await seedMatter(db, { id: 'quota-zip', name: 'quota.zip', object: 'source/quota.zip', size: 200 })
|
||||
|
||||
const s3 = new MemoryS3()
|
||||
@@ -522,11 +518,7 @@ describe('archive processing', () => {
|
||||
it('fails compression quota checks and removes streamed output', async () => {
|
||||
const { db } = await createTestApp()
|
||||
await seedStorage(db)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES ('archive-compress-quota', ${ORG_ID}, 0, 0, 0, 0, '1970-01')
|
||||
`)
|
||||
await seedStoragePlanEntitlement(db, ORG_ID, 4, 'archive-compress-quota-plan')
|
||||
await db.run(sql`UPDATE org_quota_entitlements SET bytes = 4 WHERE id = 'archive-default-quota-plan'`)
|
||||
await seedMatter(db, { id: 'file-a', name: 'a.txt', object: 'objects/a.txt', size: 5 })
|
||||
|
||||
const s3 = new MemoryS3()
|
||||
@@ -833,6 +825,11 @@ async function seedStorage(db: TestDb): Promise<void> {
|
||||
INSERT INTO storages (id, bucket, endpoint, region, access_key, secret_key, file_path, custom_host, capacity, used, status, created_at, updated_at)
|
||||
VALUES (${STORAGE_ID}, 'bucket', 'https://s3.example.com', 'auto', 'ak', 'sk', '', '', 0, 0, 'active', ${now}, ${now})
|
||||
`)
|
||||
await db.run(sql`
|
||||
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
|
||||
VALUES ('archive-default-quota', ${ORG_ID}, 0, 0, 0, 0, '1970-01')
|
||||
`)
|
||||
await seedStoragePlanEntitlement(db, ORG_ID, 10 * 1024 ** 3, 'archive-default-quota-plan')
|
||||
}
|
||||
|
||||
async function seedMatter(
|
||||
|
||||
@@ -10,7 +10,7 @@ import { createQuotaRepo } from '../adapters/repos/quota'
|
||||
import { createShareRepo } from '../adapters/repos/share'
|
||||
import { createStorageRepo } from '../adapters/repos/storage'
|
||||
import { createStorageUsageRepo } from '../adapters/repos/storage-usage'
|
||||
import { matters } from '../db/schema'
|
||||
import { matters, orgQuotaEntitlements, orgQuotas } from '../db/schema'
|
||||
import { createCloudflarePlatform } from '../platform/cloudflare'
|
||||
import type { Database } from '../platform/interface'
|
||||
import { type SaveShareInput, type SaveToDriveDeps, saveShareToDrive as saveShareToDriveUseCase } from './object'
|
||||
@@ -44,6 +44,26 @@ async function seedStorage(db: ReturnType<typeof buildDb>, id: string) {
|
||||
)
|
||||
}
|
||||
|
||||
async function seedStorageQuota(db: ReturnType<typeof buildDb>, orgId: string, bytes = 10_000_000) {
|
||||
const now = new Date()
|
||||
await db.insert(orgQuotas).values({ id: nanoid(), orgId, quota: bytes })
|
||||
await db.insert(orgQuotaEntitlements).values({
|
||||
id: nanoid(),
|
||||
orgId,
|
||||
resourceType: 'storage',
|
||||
entitlementType: 'plan',
|
||||
source: 'free_plan',
|
||||
sourceId: `free_plan:${orgId}`,
|
||||
bytes,
|
||||
startsAt: now,
|
||||
expiresAt: null,
|
||||
status: 'active',
|
||||
metadata: JSON.stringify({ packageName: 'Free', source: 'free_plan' }),
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
}
|
||||
|
||||
async function seedMatter(db: ReturnType<typeof buildDb>, orgId: string, dirtype = DirType.FILE) {
|
||||
const now = new Date()
|
||||
const matter = {
|
||||
@@ -129,6 +149,7 @@ describe('[CF] saveShareToDrive — stream copy via D1', () => {
|
||||
const srcOrgId = `src-${nanoid(6)}`
|
||||
const dstOrgId = `dst-${nanoid(6)}`
|
||||
await seedStorage(db, 'cf-storage-1')
|
||||
await seedStorageQuota(db, dstOrgId)
|
||||
|
||||
const matter = await seedMatter(db, srcOrgId)
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'cf-u1', kind: 'landing' })
|
||||
@@ -154,6 +175,7 @@ describe('[CF] saveShareToDrive — stream copy via D1', () => {
|
||||
const srcOrgId = `src-${nanoid(6)}`
|
||||
const dstOrgId = `dst-${nanoid(6)}`
|
||||
await seedStorage(db, 'cf-storage-1')
|
||||
await seedStorageQuota(db, dstOrgId)
|
||||
|
||||
const matter = await seedMatter(db, srcOrgId)
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'cf-u3', kind: 'landing' })
|
||||
@@ -184,6 +206,7 @@ describe('[CF] saveShareToDrive — stream copy via D1', () => {
|
||||
const srcOrgId = `src-${nanoid(6)}`
|
||||
const dstOrgId = `dst-${nanoid(6)}`
|
||||
await seedStorage(db, 'cf-storage-1')
|
||||
await seedStorageQuota(db, dstOrgId)
|
||||
|
||||
// Create folder structure
|
||||
const now = new Date()
|
||||
|
||||
@@ -228,17 +228,17 @@ describe('computeSourceBytes', () => {
|
||||
// ─── isQuotaSufficient ────────────────────────────────────────────────────────
|
||||
|
||||
describe('isQuotaSufficient', () => {
|
||||
it('returns true when no quota row exists (unlimited)', async () => {
|
||||
it('returns false when no quota row exists', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(true)
|
||||
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(false)
|
||||
})
|
||||
|
||||
it('returns true when quota is 0 (unlimited)', async () => {
|
||||
it('returns false when effective quota is zero', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const orgId = nanoid()
|
||||
await seedOrgQuota(db, orgId, 0)
|
||||
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(true)
|
||||
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(false)
|
||||
})
|
||||
|
||||
it('returns true when bytes fit within quota', async () => {
|
||||
@@ -270,6 +270,7 @@ describe('saveShareToDrive', () => {
|
||||
await insertStorage(db)
|
||||
const srcOrgId = nanoid()
|
||||
const dstOrgId = nanoid()
|
||||
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
|
||||
const matter = await seedMatter(db, { orgId: srcOrgId, size: 2048 })
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
|
||||
|
||||
@@ -300,6 +301,7 @@ describe('saveShareToDrive', () => {
|
||||
|
||||
const srcOrgId = nanoid()
|
||||
const dstOrgId = nanoid()
|
||||
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
|
||||
const matter = await seedMatter(db, { orgId: srcOrgId, storageId: STORAGE_ID, size: 1024 })
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
|
||||
|
||||
@@ -324,6 +326,7 @@ describe('saveShareToDrive', () => {
|
||||
await insertStorage(db)
|
||||
const srcOrgId = nanoid()
|
||||
const dstOrgId = nanoid()
|
||||
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
|
||||
const fileName = `photo-${nanoid(4)}.pdf`
|
||||
const matter = await seedMatter(db, { orgId: srcOrgId, name: fileName, size: 1024 })
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
|
||||
@@ -352,6 +355,7 @@ describe('saveShareToDrive', () => {
|
||||
await insertStorage(db)
|
||||
const srcOrgId = nanoid()
|
||||
const dstOrgId = nanoid()
|
||||
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
|
||||
const matter = await seedMatter(db, { orgId: srcOrgId, size: 512 })
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
|
||||
|
||||
@@ -379,6 +383,7 @@ describe('saveShareToDrive', () => {
|
||||
await insertStorage(db)
|
||||
const srcOrgId = nanoid()
|
||||
const dstOrgId = nanoid()
|
||||
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
|
||||
const matter = await seedMatter(db, { orgId: srcOrgId, size: 512 })
|
||||
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
|
||||
|
||||
@@ -461,6 +466,7 @@ describe('saveShareToDrive', () => {
|
||||
await insertStorage(db)
|
||||
const srcOrgId = nanoid()
|
||||
const dstOrgId = nanoid()
|
||||
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
|
||||
|
||||
// Build source tree: Photos/ → [img1.jpg, img2.jpg, Sub/ → [img3.jpg]]
|
||||
const folder = await seedMatter(db, { orgId: srcOrgId, dirtype: DirType.USER_FOLDER, name: 'Photos', size: 0 })
|
||||
|
||||
@@ -6,7 +6,7 @@ export type TrafficReportSource =
|
||||
| 'custom_domain_image'
|
||||
| 'webdav_download'
|
||||
|
||||
export type CloudTrafficReportStatus = 'pending' | 'reported' | 'skipped_unbound' | 'blocked' | 'failed'
|
||||
export type CloudTrafficReportStatus = 'pending' | 'reported' | 'skipped_unbound' | 'blocked' | 'failed' | 'dead_letter'
|
||||
|
||||
export interface CloudTrafficReportRecord {
|
||||
id: string
|
||||
@@ -21,6 +21,8 @@ export interface CloudTrafficReportRecord {
|
||||
creditsPerUnit: number | null
|
||||
status: CloudTrafficReportStatus
|
||||
error: string | null
|
||||
attemptCount: number
|
||||
nextRetryAt: Date | null
|
||||
createdAt: Date
|
||||
updatedAt: Date
|
||||
}
|
||||
@@ -42,6 +44,12 @@ export interface InsertCloudTrafficReportInput {
|
||||
export interface CloudTrafficReportRepo {
|
||||
findByEventId(eventId: string): Promise<CloudTrafficReportRecord | undefined>
|
||||
insert(input: InsertCloudTrafficReportInput): Promise<void>
|
||||
updateStatus(eventId: string, status: CloudTrafficReportStatus, error: string | null, now: Date): Promise<void>
|
||||
listPending(limit: number): Promise<CloudTrafficReportRecord[]>
|
||||
updateStatus(
|
||||
eventId: string,
|
||||
status: CloudTrafficReportStatus,
|
||||
error: string | null,
|
||||
now: Date,
|
||||
retry?: { attemptCount: number; nextRetryAt: Date | null },
|
||||
): Promise<void>
|
||||
listPending(limit: number, now: Date): Promise<CloudTrafficReportRecord[]>
|
||||
}
|
||||
|
||||
@@ -38,6 +38,7 @@ export interface QuotaRepo {
|
||||
listOrgQuotaOverview(): Promise<OrgQuotaOverviewRow[]>
|
||||
getEffectiveQuota(orgId: string, now?: Date): Promise<EffectiveQuota>
|
||||
getEffectiveQuotasByOrg(orgIds: string[], now?: Date): Promise<Map<string, EffectiveQuota>>
|
||||
reconcileFreePlanBaselines(now?: Date): Promise<void>
|
||||
resetExpiredTrafficQuotas(now?: Date): Promise<void>
|
||||
hasQuotaForBytes(orgId: string, bytes: number): Promise<boolean>
|
||||
hasTrafficQuotaForBytes(orgId: string, bytes: number, now?: Date): Promise<boolean>
|
||||
|
||||
@@ -145,7 +145,7 @@ describe('cloud traffic metering', () => {
|
||||
|
||||
const result = await syncPendingCloudTrafficReports({ db, cloudBaseUrl: 'https://cloud.example' })
|
||||
|
||||
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0 })
|
||||
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 })
|
||||
expect(fetch).toHaveBeenCalledTimes(1)
|
||||
const [url, init] = vi.mocked(fetch).mock.calls[0] as [string, RequestInit]
|
||||
expect(url).toBe('https://cloud.example/api/stores/store-test-binding/billing/usage-events')
|
||||
@@ -297,6 +297,7 @@ describe('cloud traffic metering', () => {
|
||||
reported: 1,
|
||||
blocked: 0,
|
||||
failed: 0,
|
||||
deadLetter: 0,
|
||||
})
|
||||
|
||||
const result = await syncPendingCloudTrafficReports({
|
||||
@@ -305,13 +306,75 @@ describe('cloud traffic metering', () => {
|
||||
now: new Date('2026-05-01T00:01:00.000Z'),
|
||||
})
|
||||
|
||||
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0 })
|
||||
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 })
|
||||
expect(fetch).toHaveBeenCalledTimes(2)
|
||||
await expect(db.select().from(cloudTrafficReports)).resolves.toMatchObject([
|
||||
{ status: 'reported', error: null, period: '2026-04' },
|
||||
])
|
||||
})
|
||||
|
||||
it('selects new pending reports before retryable failures', async () => {
|
||||
const { db } = await createTestApp()
|
||||
const repo = createCloudTrafficReportRepo(db)
|
||||
const now = new Date('2026-05-01T00:00:00.000Z')
|
||||
const input = {
|
||||
orgId: 'org_1',
|
||||
period: '2026-05',
|
||||
source: 'object_download' as const,
|
||||
sourceId: 'matter_1',
|
||||
bytes: 1024,
|
||||
storageId: 'storage_1',
|
||||
unitBytes: 1024,
|
||||
creditsPerUnit: 1,
|
||||
status: 'pending' as const,
|
||||
}
|
||||
await repo.insert({ ...input, eventId: 'failed-old', now: new Date(now.getTime() - 60_000) })
|
||||
await repo.updateStatus('failed-old', 'failed', 'offline', now, {
|
||||
attemptCount: 1,
|
||||
nextRetryAt: new Date(now.getTime() - 1),
|
||||
})
|
||||
await repo.insert({ ...input, eventId: 'pending-new', now })
|
||||
|
||||
await expect(repo.listPending(1, now)).resolves.toMatchObject([{ eventId: 'pending-new', status: 'pending' }])
|
||||
})
|
||||
|
||||
it('dead-letters terminal idempotency conflicts without retrying them', async () => {
|
||||
const { db, platform } = await createTestApp({ ZPAN_CLOUD_URL: 'https://cloud.example' })
|
||||
await seedTrafficBinding(db)
|
||||
vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error('usage_idempotency_conflict')))
|
||||
const now = new Date('2026-05-01T00:00:00.000Z')
|
||||
|
||||
await expect(
|
||||
reportTrafficEgress({
|
||||
platform,
|
||||
orgId: 'org_1',
|
||||
bytes: 1024,
|
||||
source: 'object_download',
|
||||
sourceId: 'matter_1',
|
||||
eventId: 'evt_terminal',
|
||||
now,
|
||||
...meteredStorage,
|
||||
}),
|
||||
).resolves.toMatchObject({ status: 'dead_letter', eventId: 'evt_terminal' })
|
||||
await expect(db.select().from(cloudTrafficReports)).resolves.toMatchObject([
|
||||
{
|
||||
eventId: 'evt_terminal',
|
||||
status: 'dead_letter',
|
||||
error: 'usage_idempotency_conflict',
|
||||
attemptCount: 1,
|
||||
nextRetryAt: null,
|
||||
},
|
||||
])
|
||||
await expect(
|
||||
syncPendingCloudTrafficReports({
|
||||
db,
|
||||
cloudBaseUrl: 'https://cloud.example',
|
||||
now: new Date(now.getTime() + 60_000),
|
||||
}),
|
||||
).resolves.toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 })
|
||||
expect(fetch).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('marks reports reported when Cloud returns its own usage event id', async () => {
|
||||
const { db, platform } = await createTestApp({ ZPAN_CLOUD_URL: 'https://cloud.example' })
|
||||
await seedTrafficBinding(db)
|
||||
|
||||
@@ -34,6 +34,9 @@ const usageResponseSchema = z.object({
|
||||
eventId: z.string().min(1),
|
||||
})
|
||||
|
||||
const MAX_TRAFFIC_REPORT_ATTEMPTS = 8
|
||||
const TERMINAL_TRAFFIC_REPORT_ERRORS = new Set(['usage_idempotency_conflict'])
|
||||
|
||||
export async function reportTrafficEgress(
|
||||
deps: CloudTrafficMeteringDeps,
|
||||
params: {
|
||||
@@ -114,16 +117,17 @@ export async function reportTrafficEgress(
|
||||
export async function syncPendingCloudTrafficReports(
|
||||
deps: CloudTrafficMeteringDeps,
|
||||
params: { cloudBaseUrl: string; limit?: number; now?: Date },
|
||||
): Promise<{ attempted: number; reported: number; blocked: number; failed: number }> {
|
||||
): Promise<{ attempted: number; reported: number; blocked: number; failed: number; deadLetter: number }> {
|
||||
const { cloudBaseUrl, limit = 100, now = new Date() } = params
|
||||
if (!hasFeature('quota_store', await loadBindingState(deps)))
|
||||
return { attempted: 0, reported: 0, blocked: 0, failed: 0 }
|
||||
return { attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 }
|
||||
const binding = await deps.licenseBinding.loadActiveLicenseBinding()
|
||||
if (!binding?.refreshToken || !binding.cloudStoreId) return { attempted: 0, reported: 0, blocked: 0, failed: 0 }
|
||||
if (!binding?.refreshToken || !binding.cloudStoreId)
|
||||
return { attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 }
|
||||
|
||||
const reports = await deps.cloudTrafficReports.listPending(limit)
|
||||
const reports = await deps.cloudTrafficReports.listPending(limit, now)
|
||||
|
||||
const result = { attempted: reports.length, reported: 0, blocked: 0, failed: 0 }
|
||||
const result = { attempted: reports.length, reported: 0, blocked: 0, failed: 0, deadLetter: 0 }
|
||||
for (const report of reports) {
|
||||
const status = await syncTrafficReport(deps, {
|
||||
cloudBaseUrl,
|
||||
@@ -132,7 +136,7 @@ export async function syncPendingCloudTrafficReports(
|
||||
report,
|
||||
now,
|
||||
})
|
||||
result[status] += 1
|
||||
result[status === 'dead_letter' ? 'deadLetter' : status] += 1
|
||||
}
|
||||
return result
|
||||
}
|
||||
@@ -146,7 +150,7 @@ async function syncTrafficReport(
|
||||
report: CloudTrafficReportRecord
|
||||
now: Date
|
||||
},
|
||||
): Promise<'reported' | 'blocked' | 'failed'> {
|
||||
): Promise<'reported' | 'blocked' | 'failed' | 'dead_letter'> {
|
||||
const { cloudBaseUrl, refreshToken, storeId, report, now } = params
|
||||
try {
|
||||
const client = deps.licensingCloud.createBoundCloudClient(cloudBaseUrl, refreshToken)
|
||||
@@ -179,19 +183,35 @@ async function syncTrafficReport(
|
||||
usageResponseSchema,
|
||||
)
|
||||
if (!response.accepted) throw new Error('cloud_usage_report_rejected')
|
||||
await deps.cloudTrafficReports.updateStatus(report.eventId, 'reported', null, now)
|
||||
await deps.cloudTrafficReports.updateStatus(report.eventId, 'reported', null, now, {
|
||||
attemptCount: report.attemptCount + 1,
|
||||
nextRetryAt: null,
|
||||
})
|
||||
return 'reported'
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : 'cloud_usage_report_failed'
|
||||
if (message === 'insufficient_credits' || message === 'overage_cap_exceeded') {
|
||||
await deps.cloudTrafficReports.updateStatus(report.eventId, 'blocked', message, now)
|
||||
await deps.cloudTrafficReports.updateStatus(report.eventId, 'blocked', message, now, {
|
||||
attemptCount: report.attemptCount + 1,
|
||||
nextRetryAt: null,
|
||||
})
|
||||
return 'blocked'
|
||||
}
|
||||
await deps.cloudTrafficReports.updateStatus(report.eventId, 'failed', message, now)
|
||||
return 'failed'
|
||||
const attemptCount = report.attemptCount + 1
|
||||
const terminal = TERMINAL_TRAFFIC_REPORT_ERRORS.has(message) || attemptCount >= MAX_TRAFFIC_REPORT_ATTEMPTS
|
||||
await deps.cloudTrafficReports.updateStatus(report.eventId, terminal ? 'dead_letter' : 'failed', message, now, {
|
||||
attemptCount,
|
||||
nextRetryAt: terminal ? null : nextTrafficReportRetryAt(now, attemptCount),
|
||||
})
|
||||
return terminal ? 'dead_letter' : 'failed'
|
||||
}
|
||||
}
|
||||
|
||||
function nextTrafficReportRetryAt(now: Date, attemptCount: number): Date {
|
||||
const delayMinutes = Math.min(360, 2 ** Math.min(attemptCount - 1, 8))
|
||||
return new Date(now.getTime() + delayMinutes * 60_000)
|
||||
}
|
||||
|
||||
function assertSameReport(
|
||||
report: CloudTrafficReportRecord,
|
||||
params: {
|
||||
|
||||
+50
-30
@@ -4,7 +4,7 @@ export interface AdminUsageBySpace {
|
||||
orgType: string
|
||||
usedBytes: number
|
||||
quotaBytes: number
|
||||
utilization: number
|
||||
utilization: number | null
|
||||
}
|
||||
|
||||
export interface AdminTopShare {
|
||||
@@ -25,19 +25,23 @@ export interface AdminStatsRange {
|
||||
timeZone: 'UTC'
|
||||
coverage: AdminStatsCoverage
|
||||
comparisonCoverage?: AdminStatsCoverage
|
||||
snapshotCoverage?: AdminStatsCoverage
|
||||
comparisonSnapshotCoverage?: AdminStatsCoverage
|
||||
}
|
||||
|
||||
export interface AdminStatsCoverage {
|
||||
status: 'complete' | 'partial' | 'empty'
|
||||
expectedBuckets: number
|
||||
completedBuckets: number
|
||||
lowerBoundBuckets: number
|
||||
quality: 'exact' | 'lower_bound'
|
||||
dataThrough: string | null
|
||||
}
|
||||
|
||||
export interface AdminStatsDelta {
|
||||
value: number
|
||||
previousValue: number
|
||||
change: number
|
||||
value: number | null
|
||||
previousValue: number | null
|
||||
change: number | null
|
||||
changePercent: number | null
|
||||
}
|
||||
|
||||
@@ -50,27 +54,38 @@ export interface AdminTransferDataQuality {
|
||||
previousMissingBytesEvents: number
|
||||
}
|
||||
|
||||
export interface AdminSharingDataQuality {
|
||||
unlocatedViews: number | null
|
||||
unlocatedDownloads: number | null
|
||||
unlocatedEvents: number | null
|
||||
}
|
||||
|
||||
export interface AdminStorageDataQuality extends AdminTransferDataQuality {
|
||||
usageDriftSpaces: number | null
|
||||
usageDriftBytes: number | null
|
||||
}
|
||||
|
||||
export interface AdminDashboardOverviewStats extends AdminStatsRange {
|
||||
dataQuality: AdminTransferDataQuality
|
||||
totals: {
|
||||
users: number
|
||||
users: number | null
|
||||
newUsers: AdminStatsDelta
|
||||
activeUsers: AdminStatsDelta
|
||||
activeUserRate: number | null
|
||||
storageUsedBytes: number
|
||||
storageQuotaBytes: number
|
||||
storageUsedBytes: number | null
|
||||
storageQuotaBytes: number | null
|
||||
storageUtilization: number | null
|
||||
trafficBytes: AdminStatsDelta
|
||||
uploadBytes: AdminStatsDelta
|
||||
downloadBytes: AdminStatsDelta
|
||||
activeShares: number
|
||||
activeShares: number | null
|
||||
shareViews: AdminStatsDelta
|
||||
shareDownloads: AdminStatsDelta
|
||||
}
|
||||
trends: Array<{
|
||||
date: string
|
||||
newUsers: number
|
||||
activeUsers: number
|
||||
activeUsers: number | null
|
||||
storageUsedBytes: number | null
|
||||
uploadBytes: number
|
||||
downloadBytes: number
|
||||
@@ -79,34 +94,37 @@ export interface AdminDashboardOverviewStats extends AdminStatsRange {
|
||||
|
||||
export interface AdminDashboardGrowthStats extends AdminStatsRange {
|
||||
summary: {
|
||||
totalUsers: number
|
||||
totalUsers: number | null
|
||||
newUsers: AdminStatsDelta
|
||||
activeUsers: AdminStatsDelta
|
||||
verifiedUsers: number
|
||||
bannedUsers: number
|
||||
silentUsers: number
|
||||
verifiedUsers: number | null
|
||||
bannedUsers: number | null
|
||||
silentUsers: number | null
|
||||
activeUserRate: number | null
|
||||
silentUserRate: number | null
|
||||
}
|
||||
userScaleTrend: Array<{ date: string; newUsers: number; totalUsers: number }>
|
||||
activeUserTrend: Array<{ date: string; dau: number; wau: number; mau: number }>
|
||||
userScaleTrend: Array<{ date: string; newUsers: number; totalUsers: number | null }>
|
||||
activeUserTrend: Array<{ date: string; dau: number | null; wau: number | null; mau: number | null }>
|
||||
userStatus: Array<{ name: string; value: number; percent: number }>
|
||||
registrationSources: Array<{ name: string; value: number; percent: number }>
|
||||
}
|
||||
|
||||
export interface AdminDashboardStorageStats extends AdminStatsRange {
|
||||
dataQuality: AdminTransferDataQuality
|
||||
dataQuality: AdminStorageDataQuality
|
||||
summary: {
|
||||
storageUsedBytes: number
|
||||
quotaBytes: number
|
||||
fileCount: number
|
||||
storageUsedBytes: number | null
|
||||
quotaBytes: number | null
|
||||
fileCount: number | null
|
||||
trashFileCount: number | null
|
||||
trashBytes: number | null
|
||||
newFiles: AdminStatsDelta
|
||||
newBytes: AdminStatsDelta
|
||||
coldFileBytes: number
|
||||
coldFileBytes: number | null
|
||||
storageUtilization: number | null
|
||||
coldFilePercent: number | null
|
||||
nearQuotaSpaces: number
|
||||
overQuotaSpaces: number
|
||||
nearQuotaSpaces: number | null
|
||||
overQuotaSpaces: number | null
|
||||
invalidQuotaSpaces: number | null
|
||||
}
|
||||
storageTrend: Array<{ date: string; usedBytes: number | null; newBytes: number; newFiles: number }>
|
||||
typeBreakdown: Array<{ type: string; bytes: number; files: number; percent: number }>
|
||||
@@ -133,8 +151,9 @@ export interface AdminDashboardTrafficStats extends AdminStatsRange {
|
||||
}
|
||||
|
||||
export interface AdminDashboardSharingStats extends AdminStatsRange {
|
||||
dataQuality: AdminSharingDataQuality
|
||||
summary: {
|
||||
activeShares: number
|
||||
activeShares: number | null
|
||||
createdShares: AdminStatsDelta
|
||||
views: AdminStatsDelta
|
||||
downloads: AdminStatsDelta
|
||||
@@ -151,15 +170,16 @@ export interface AdminDashboardSharingStats extends AdminStatsRange {
|
||||
|
||||
export interface AdminDashboardOperationsStats extends AdminStatsRange {
|
||||
summary: {
|
||||
activeBackgroundJobs: number
|
||||
activeRemoteDownloads: number
|
||||
onlineDownloaders: number
|
||||
offlineDownloaders: number
|
||||
activeBackgroundJobs: number | null
|
||||
activeRemoteDownloads: number | null
|
||||
onlineDownloaders: number | null
|
||||
offlineDownloaders: number | null
|
||||
backgroundJobFailureRate: number | null
|
||||
remoteDownloadSuccessRate: number | null
|
||||
cloudReportBacklog: number
|
||||
webhookFailures: number
|
||||
alertCount: number
|
||||
cloudReportBacklog: number | null
|
||||
cloudReportDeadLetters: number | null
|
||||
webhookFailures: number | null
|
||||
alertCount: number | null
|
||||
}
|
||||
trend: Array<{
|
||||
date: string
|
||||
|
||||
@@ -216,9 +216,11 @@ export type {
|
||||
AdminDashboardSharingStats,
|
||||
AdminDashboardStorageStats,
|
||||
AdminDashboardTrafficStats,
|
||||
AdminSharingDataQuality,
|
||||
AdminStatsCoverage,
|
||||
AdminStatsDelta,
|
||||
AdminStatsRange,
|
||||
AdminStorageDataQuality,
|
||||
AdminTopShare,
|
||||
AdminTransferDataQuality,
|
||||
AdminUsageBySpace,
|
||||
|
||||
@@ -46,7 +46,7 @@ export function QuotaPanel({ enabled }: { enabled: boolean }) {
|
||||
<p className="text-xs text-muted-foreground tabular-nums">
|
||||
{quota.quota > 0
|
||||
? t('quota.usage', { used: formatSize(quota.used), total: formatSize(quota.quota) })
|
||||
: t('quota.usageNoLimit', { used: formatSize(quota.used) })}
|
||||
: t('quota.usageInvalid', { used: formatSize(quota.used) })}
|
||||
</p>
|
||||
) : (
|
||||
<Skeleton className={isLoading ? 'h-3 w-24' : 'h-3 w-16 opacity-50'} />
|
||||
|
||||
@@ -80,6 +80,7 @@ function PlanUsageOverview({ quota }: { quota: UserQuota }) {
|
||||
label={t('storage.storageUsage')}
|
||||
used={quota.used}
|
||||
total={quota.quota}
|
||||
allowUnlimited={false}
|
||||
detail={t('storage.storageUsageDetail', { used: formatSize(quota.used) })}
|
||||
/>
|
||||
<UsageMeter
|
||||
@@ -87,6 +88,7 @@ function PlanUsageOverview({ quota }: { quota: UserQuota }) {
|
||||
label={t('storage.trafficUsage')}
|
||||
used={quota.trafficUsed}
|
||||
total={quota.trafficQuota}
|
||||
allowUnlimited
|
||||
detail={t('storage.trafficPeriodDetail', { period: quota.trafficPeriod })}
|
||||
/>
|
||||
</div>
|
||||
@@ -98,16 +100,19 @@ function UsageMeter({
|
||||
label,
|
||||
used,
|
||||
total,
|
||||
allowUnlimited,
|
||||
detail,
|
||||
}: {
|
||||
icon: React.ReactNode
|
||||
label: string
|
||||
used: number
|
||||
total: number
|
||||
allowUnlimited: boolean
|
||||
detail: string
|
||||
}) {
|
||||
const { t } = useTranslation()
|
||||
const percent = total > 0 ? Math.min(100, (used / total) * 100) : 100
|
||||
const invalid = total <= 0 && !allowUnlimited
|
||||
const percent = total > 0 ? Math.min(100, (used / total) * 100) : invalid ? 0 : 100
|
||||
return (
|
||||
<div className="min-w-0 rounded-md bg-muted/20 p-4">
|
||||
<div className="flex items-center justify-between gap-3">
|
||||
@@ -116,7 +121,9 @@ function UsageMeter({
|
||||
<span>{label}</span>
|
||||
</span>
|
||||
<span className="shrink-0 text-xs text-muted-foreground">
|
||||
{total > 0 ? t('storage.usageTotal', { total: formatSize(total) }) : t('storage.usageNoLimit')}
|
||||
{total > 0
|
||||
? t('storage.usageTotal', { total: formatSize(total) })
|
||||
: t(invalid ? 'storage.usageInvalid' : 'storage.usageNoLimit')}
|
||||
</span>
|
||||
</div>
|
||||
<div className="mt-3 space-y-2">
|
||||
|
||||
@@ -64,7 +64,6 @@ const ADMIN_STORAGES_KEYS = [
|
||||
'admin.storages.customHostPlaceholder',
|
||||
'admin.storages.fieldCapacity',
|
||||
'admin.storages.capacityPlaceholder',
|
||||
'admin.storages.capacityUnlimited',
|
||||
'admin.storages.capacityHint',
|
||||
'admin.storages.egressBilling',
|
||||
'admin.storages.configureEgressBilling',
|
||||
@@ -223,12 +222,10 @@ describe('admin.storages locale keys — English values contract', () => {
|
||||
expect(enLocale['admin.storages.fieldCapacity']).toBe('Capacity')
|
||||
})
|
||||
|
||||
it('admin.storages.capacityUnlimited is "Unlimited"', () => {
|
||||
expect(enLocale['admin.storages.capacityUnlimited']).toBe('Unlimited')
|
||||
})
|
||||
|
||||
it('admin.storages.capacityHint is "Maximum storage space. 0 means unlimited."', () => {
|
||||
expect(enLocale['admin.storages.capacityHint']).toBe('Maximum storage space. 0 means unlimited.')
|
||||
it('admin.storages.capacityHint describes an unreported zero capacity', () => {
|
||||
expect(enLocale['admin.storages.capacityHint']).toBe(
|
||||
'Maximum reported storage space. 0 means capacity is not reported.',
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -380,27 +377,17 @@ describe('admin.storages locale keys — i18n runtime translation', () => {
|
||||
expect(i18n.t('admin.storages.fieldCapacity')).toBe('可用空间')
|
||||
})
|
||||
|
||||
it('translates admin.storages.capacityUnlimited to English', async () => {
|
||||
const { default: i18n } = await import('./index')
|
||||
await i18n.changeLanguage('en')
|
||||
expect(i18n.t('admin.storages.capacityUnlimited')).toBe('Unlimited')
|
||||
})
|
||||
|
||||
it('translates admin.storages.capacityUnlimited to Chinese', async () => {
|
||||
const { default: i18n } = await import('./index')
|
||||
await i18n.changeLanguage('zh')
|
||||
expect(i18n.t('admin.storages.capacityUnlimited')).toBe('不限制')
|
||||
})
|
||||
|
||||
it('translates admin.storages.capacityHint to English', async () => {
|
||||
const { default: i18n } = await import('./index')
|
||||
await i18n.changeLanguage('en')
|
||||
expect(i18n.t('admin.storages.capacityHint')).toBe('Maximum storage space. 0 means unlimited.')
|
||||
expect(i18n.t('admin.storages.capacityHint')).toBe(
|
||||
'Maximum reported storage space. 0 means capacity is not reported.',
|
||||
)
|
||||
})
|
||||
|
||||
it('translates admin.storages.capacityHint to Chinese', async () => {
|
||||
const { default: i18n } = await import('./index')
|
||||
await i18n.changeLanguage('zh')
|
||||
expect(i18n.t('admin.storages.capacityHint')).toBe('最大存储空间,0 表示不限制。')
|
||||
expect(i18n.t('admin.storages.capacityHint')).toBe('存储后端报告的最大空间,0 表示未报告容量。')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -713,13 +713,11 @@
|
||||
"admin.settings.storageProTooltip": "Storage controls are available in ZPan Pro.",
|
||||
"admin.settings.cloudStoreEnabled": "Storage Plans",
|
||||
"admin.settings.cloudStoreEnabledHint": "Allow users to buy additional storage and apply gift cards.",
|
||||
"admin.settings.unlimited": "Unlimited",
|
||||
"admin.settings.previewTitle": "Live summary",
|
||||
"admin.settings.previewDescription": "A quick read on how the current configuration will feel before you save it.",
|
||||
"admin.settings.previewFallback": "Add a short description so users immediately understand what this instance is for.",
|
||||
"admin.settings.previewUrlLabel": "Primary address",
|
||||
"admin.settings.previewQuotaLabel": "New organization quota",
|
||||
"admin.settings.previewQuotaUnlimited": "New organizations will start without a storage cap.",
|
||||
"admin.settings.previewBrandingLabel": "Branding next step",
|
||||
"admin.settings.previewBrandingHint": "Use the branding section below to add your logo, favicon, and footer visibility rules.",
|
||||
"admin.settings.brandingTitle": "Branding",
|
||||
@@ -941,8 +939,7 @@
|
||||
"admin.storages.previewSigningRegion": "Signing region: {{region}}",
|
||||
"admin.storages.fieldCapacity": "Capacity",
|
||||
"admin.storages.capacityPlaceholder": "0",
|
||||
"admin.storages.capacityUnlimited": "Unlimited",
|
||||
"admin.storages.capacityHint": "Maximum storage space. 0 means unlimited.",
|
||||
"admin.storages.capacityHint": "Maximum reported storage space. 0 means capacity is not reported.",
|
||||
"admin.storages.egressBilling": "Traffic Credits billing",
|
||||
"admin.storages.configureEgressBilling": "Configure egress billing",
|
||||
"admin.storages.billingTitle": "Limits and billing",
|
||||
@@ -1051,7 +1048,7 @@
|
||||
"quota.storage": "Storage",
|
||||
"quota.traffic": "Traffic",
|
||||
"quota.usage": "{{used}} / {{total}} used",
|
||||
"quota.usageNoLimit": "{{used}} used",
|
||||
"quota.usageInvalid": "{{used}} used · invalid storage quota",
|
||||
"quota.purchased": "{{amount}} purchased or granted",
|
||||
"quota.purchasedTraffic": "{{amount}} traffic purchased or granted",
|
||||
"quota.cloudStorageEntitlement": "{{amount}} extra storage",
|
||||
@@ -1803,6 +1800,7 @@
|
||||
"storage.freePlanPrice": "Free",
|
||||
"storage.usageTotal": "of {{total}}",
|
||||
"storage.usageNoLimit": "No limit",
|
||||
"storage.usageInvalid": "Invalid quota",
|
||||
"storage.legendUsed": "Used",
|
||||
"storage.legendBaseQuota": "Plan quota",
|
||||
"storage.legendCloudQuota": "Extra quota",
|
||||
|
||||
@@ -713,13 +713,11 @@
|
||||
"admin.settings.storageProTooltip": "Storage 控制项仅在 ZPan Pro 中可用。",
|
||||
"admin.settings.cloudStoreEnabled": "存储套餐",
|
||||
"admin.settings.cloudStoreEnabledHint": "允许用户购买额外存储空间或使用礼品卡。",
|
||||
"admin.settings.unlimited": "不限",
|
||||
"admin.settings.previewTitle": "实时概览",
|
||||
"admin.settings.previewDescription": "在保存之前,快速查看当前配置给用户呈现出来的大致感觉。",
|
||||
"admin.settings.previewFallback": "添加一段简短描述,让用户一进来就知道这个实例是做什么的。",
|
||||
"admin.settings.previewUrlLabel": "主访问地址",
|
||||
"admin.settings.previewQuotaLabel": "新组织默认配额",
|
||||
"admin.settings.previewQuotaUnlimited": "新组织创建后默认不设存储上限。",
|
||||
"admin.settings.previewBrandingLabel": "品牌下一步",
|
||||
"admin.settings.previewBrandingHint": "继续在下方配置 Logo、Favicon 和底部署名显示规则。",
|
||||
"admin.settings.brandingTitle": "品牌设置",
|
||||
@@ -941,8 +939,7 @@
|
||||
"admin.storages.previewSigningRegion": "签名区域:{{region}}",
|
||||
"admin.storages.fieldCapacity": "可用空间",
|
||||
"admin.storages.capacityPlaceholder": "0",
|
||||
"admin.storages.capacityUnlimited": "不限制",
|
||||
"admin.storages.capacityHint": "最大存储空间,0 表示不限制。",
|
||||
"admin.storages.capacityHint": "存储后端报告的最大空间,0 表示未报告容量。",
|
||||
"admin.storages.egressBilling": "流量 Credits 计费",
|
||||
"admin.storages.configureEgressBilling": "配置流量计费",
|
||||
"admin.storages.billingTitle": "容量与计费",
|
||||
@@ -1051,7 +1048,7 @@
|
||||
"quota.storage": "存储空间",
|
||||
"quota.traffic": "下载流量",
|
||||
"quota.usage": "{{used}} / {{total}} 已使用",
|
||||
"quota.usageNoLimit": "{{used}} 已使用",
|
||||
"quota.usageInvalid": "{{used}} 已使用 · 存储额度异常",
|
||||
"quota.purchased": "已购买或获赠 {{amount}}",
|
||||
"quota.purchasedTraffic": "已购买或获赠 {{amount}} 流量",
|
||||
"quota.cloudStorageEntitlement": "额外存储 {{amount}}",
|
||||
@@ -1803,6 +1800,7 @@
|
||||
"storage.freePlanPrice": "免费",
|
||||
"storage.usageTotal": "共 {{total}}",
|
||||
"storage.usageNoLimit": "不限量",
|
||||
"storage.usageInvalid": "额度异常",
|
||||
"storage.legendUsed": "已用",
|
||||
"storage.legendBaseQuota": "套餐额度",
|
||||
"storage.legendCloudQuota": "额外额度",
|
||||
|
||||
+1
-2
@@ -13,9 +13,8 @@ export function getInitials(name: string): string {
|
||||
.join('')
|
||||
}
|
||||
|
||||
// "used / total" with ∞ for an unlimited (0 or negative) quota.
|
||||
export function formatStorageUsage(used: number, total: number): string {
|
||||
return `${formatSize(used)} / ${total <= 0 ? '∞' : formatSize(total)}`
|
||||
return total > 0 ? `${formatSize(used)} / ${formatSize(total)}` : `${formatSize(used)} / —`
|
||||
}
|
||||
|
||||
export function formatSize(bytes: number): string {
|
||||
|
||||
@@ -50,6 +50,8 @@ const overviewStats: AdminDashboardOverviewStats = {
|
||||
status: 'complete',
|
||||
expectedBuckets: 192,
|
||||
completedBuckets: 192,
|
||||
lowerBoundBuckets: 0,
|
||||
quality: 'exact',
|
||||
dataThrough: '2026-07-09T00:00:00.000Z',
|
||||
},
|
||||
dataQuality: {
|
||||
@@ -96,6 +98,8 @@ const operationsStats: AdminDashboardOperationsStats = {
|
||||
status: 'complete',
|
||||
expectedBuckets: 192,
|
||||
completedBuckets: 192,
|
||||
lowerBoundBuckets: 0,
|
||||
quality: 'exact',
|
||||
dataThrough: '2026-07-09T00:00:00.000Z',
|
||||
},
|
||||
summary: {
|
||||
@@ -106,6 +110,7 @@ const operationsStats: AdminDashboardOperationsStats = {
|
||||
backgroundJobFailureRate: 5,
|
||||
remoteDownloadSuccessRate: 95,
|
||||
cloudReportBacklog: 6,
|
||||
cloudReportDeadLetters: 0,
|
||||
webhookFailures: 7,
|
||||
alertCount: 13,
|
||||
},
|
||||
@@ -201,6 +206,8 @@ describe('Admin overview dashboard', () => {
|
||||
status: 'partial',
|
||||
expectedBuckets: 192,
|
||||
completedBuckets: 144,
|
||||
lowerBoundBuckets: 0,
|
||||
quality: 'exact',
|
||||
dataThrough: '2026-06-30T00:00:00.000Z',
|
||||
},
|
||||
})
|
||||
|
||||
@@ -5,6 +5,7 @@ import type {
|
||||
AdminDashboardSharingStats,
|
||||
AdminDashboardStorageStats,
|
||||
AdminDashboardTrafficStats,
|
||||
AdminSharingDataQuality,
|
||||
AdminStatsRange,
|
||||
AdminTransferDataQuality,
|
||||
} from '@shared/types'
|
||||
@@ -68,7 +69,7 @@ import {
|
||||
getAdminDashboardStorageStats,
|
||||
getAdminDashboardTrafficStats,
|
||||
} from '@/lib/api'
|
||||
import { formatSize } from '@/lib/format'
|
||||
import { formatSize as formatByteSize } from '@/lib/format'
|
||||
import { cn } from '@/lib/utils'
|
||||
|
||||
export const Route = createFileRoute('/_authenticated/admin/')({
|
||||
@@ -681,7 +682,7 @@ function StorageSection({ stats }: { stats: AdminDashboardStorageStats }) {
|
||||
label: '配额使用率',
|
||||
value: formatPercent(stats.summary.storageUtilization),
|
||||
},
|
||||
{ label: '文件数', value: formatNumber(stats.summary.fileCount) },
|
||||
{ label: '回收站占用', value: formatSize(stats.summary.trashBytes) },
|
||||
]}
|
||||
/>
|
||||
<StatCard
|
||||
@@ -701,7 +702,7 @@ function StorageSection({ stats }: { stats: AdminDashboardStorageStats }) {
|
||||
icon={Database}
|
||||
metrics={[
|
||||
{ label: '上期新增', value: formatNumber(stats.summary.newFiles.previousValue) },
|
||||
{ label: '文件总数', value: formatNumber(stats.summary.fileCount) },
|
||||
{ label: '活跃文件总数', value: formatNumber(stats.summary.fileCount) },
|
||||
]}
|
||||
/>
|
||||
<StatCard
|
||||
@@ -718,9 +719,10 @@ function StorageSection({ stats }: { stats: AdminDashboardStorageStats }) {
|
||||
/>
|
||||
</div>
|
||||
<TransferDataQualityNotice quality={stats.dataQuality} />
|
||||
<StorageUsageDataQualityNotice quality={stats.dataQuality} />
|
||||
<ChartCard
|
||||
title="空间配额压力"
|
||||
subtitle={`全部空间中 ${stats.summary.nearQuotaSpaces} 个达到 80%,${stats.summary.overQuotaSpaces} 个达到或超过配额。`}
|
||||
subtitle={`全部空间中 ${formatNumber(stats.summary.nearQuotaSpaces)} 个达到 80%,${formatNumber(stats.summary.overQuotaSpaces)} 个达到或超过配额,${formatNumber(stats.summary.invalidQuotaSpaces)} 个额度配置异常。`}
|
||||
contentClassName="h-auto"
|
||||
>
|
||||
{stats.topSpaces.length === 0 ? (
|
||||
@@ -1116,7 +1118,7 @@ function SharingSection({ stats }: { stats: AdminDashboardSharingStats }) {
|
||||
]}
|
||||
/>
|
||||
<StatCard
|
||||
label="访问次数"
|
||||
label="已定位访问"
|
||||
value={formatNumber(stats.summary.views.value)}
|
||||
delta={formatDelta(stats.summary.views)}
|
||||
icon={Activity}
|
||||
@@ -1126,7 +1128,7 @@ function SharingSection({ stats }: { stats: AdminDashboardSharingStats }) {
|
||||
]}
|
||||
/>
|
||||
<StatCard
|
||||
label="下载签发"
|
||||
label="已定位下载签发"
|
||||
value={formatNumber(stats.summary.downloads.value)}
|
||||
delta={formatDelta(stats.summary.downloads)}
|
||||
icon={Download}
|
||||
@@ -1149,8 +1151,9 @@ function SharingSection({ stats }: { stats: AdminDashboardSharingStats }) {
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
<SharingDataQualityNotice quality={stats.dataQuality} />
|
||||
<div className="grid gap-4">
|
||||
<ChartCard title="访问行为趋势" subtitle="这些是独立事件量,不表示同一访客完成了连续漏斗。">
|
||||
<ChartCard title="访问行为趋势" subtitle="仅展示能定位到具体时间的独立事件,不表示同一访客完成了连续漏斗。">
|
||||
<ResponsiveContainer width="100%" height="100%">
|
||||
<AreaChart data={stats.trend}>
|
||||
<CartesianGrid stroke={CHART_GRID_COLOR} strokeDasharray="3 3" vertical={false} />
|
||||
@@ -1236,7 +1239,7 @@ function OperationsSection({ stats }: { stats: AdminDashboardOperationsStats })
|
||||
icon={FileClock}
|
||||
metrics={[
|
||||
{ label: '计量积压', value: formatNumber(stats.summary.cloudReportBacklog) },
|
||||
{ label: 'Webhook 失败', value: formatNumber(stats.summary.webhookFailures) },
|
||||
{ label: '计量死信', value: formatNumber(stats.summary.cloudReportDeadLetters) },
|
||||
]}
|
||||
/>
|
||||
</div>
|
||||
@@ -1530,11 +1533,30 @@ function QueryState<T extends AdminStatsRange>({
|
||||
function StatsCoverageNotice({ stats }: { stats: AdminStatsRange }) {
|
||||
const { coverage } = stats
|
||||
const comparisonCoverage = stats.comparisonCoverage
|
||||
const snapshotCoverage = stats.snapshotCoverage
|
||||
const comparisonSnapshotCoverage = stats.comparisonSnapshotCoverage
|
||||
const through = coverage.dataThrough
|
||||
? new Date(coverage.dataThrough).toISOString().replace('T', ' ').slice(0, 16)
|
||||
: null
|
||||
const snapshotThrough = snapshotCoverage?.dataThrough
|
||||
? new Date(snapshotCoverage.dataThrough).toISOString().replace('T', ' ').slice(0, 16)
|
||||
: null
|
||||
const comparisonIncomplete = comparisonCoverage ? comparisonCoverage.status !== 'complete' : false
|
||||
const incomplete = coverage.status !== 'complete' || comparisonIncomplete
|
||||
const lowerBound =
|
||||
coverage.quality === 'lower_bound' ||
|
||||
comparisonCoverage?.quality === 'lower_bound' ||
|
||||
snapshotCoverage?.quality === 'lower_bound' ||
|
||||
comparisonSnapshotCoverage?.quality === 'lower_bound'
|
||||
const snapshotIncomplete = snapshotCoverage ? snapshotCoverage.status !== 'complete' : false
|
||||
const comparisonSnapshotIncomplete = comparisonSnapshotCoverage
|
||||
? comparisonSnapshotCoverage.status !== 'complete'
|
||||
: false
|
||||
const incomplete =
|
||||
coverage.status !== 'complete' ||
|
||||
comparisonIncomplete ||
|
||||
snapshotIncomplete ||
|
||||
comparisonSnapshotIncomplete ||
|
||||
lowerBound
|
||||
return (
|
||||
<div
|
||||
role="status"
|
||||
@@ -1544,22 +1566,37 @@ function StatsCoverageNotice({ stats }: { stats: AdminStatsRange }) {
|
||||
)}
|
||||
>
|
||||
<span className={incomplete ? 'text-amber-800 dark:text-amber-200' : 'text-muted-foreground'}>
|
||||
{coverage.status === 'empty'
|
||||
? '所选范围还没有可用的离线结果。'
|
||||
: coverage.status === 'partial'
|
||||
? '所选范围存在缺失的小时结果,当前数据不完整。'
|
||||
: comparisonCoverage?.status === 'empty'
|
||||
? '对比区间还没有可用的离线结果,环比不可对账。'
|
||||
: comparisonCoverage?.status === 'partial'
|
||||
? '对比区间存在缺失的小时结果,环比数据不完整。'
|
||||
: '所选范围的离线结果完整。'}
|
||||
{lowerBound
|
||||
? '部分小时只有可验证的数据下限;页面不会把这些数值表述为完整事实。'
|
||||
: coverage.status === 'empty'
|
||||
? '所选范围还没有可用的离线结果。'
|
||||
: coverage.status === 'partial'
|
||||
? '所选范围存在缺失的小时结果,当前数据不完整。'
|
||||
: comparisonCoverage?.status === 'empty'
|
||||
? '对比区间还没有可用的离线结果,环比不可对账。'
|
||||
: comparisonCoverage?.status === 'partial'
|
||||
? '对比区间存在缺失的小时结果,环比数据不完整。'
|
||||
: snapshotCoverage?.status === 'empty'
|
||||
? '所选范围没有可用的状态快照;状态类指标显示为 —,不会伪装成 0。'
|
||||
: snapshotCoverage?.status === 'partial'
|
||||
? '所选范围的事件结果完整,但状态快照仅覆盖部分小时。'
|
||||
: comparisonSnapshotCoverage?.status === 'empty'
|
||||
? '对比区间没有状态快照,状态类环比不可对账。'
|
||||
: comparisonSnapshotCoverage?.status === 'partial'
|
||||
? '对比区间的状态快照不完整,状态类环比不可对账。'
|
||||
: '所选范围的离线结果完整。'}
|
||||
</span>
|
||||
<span className="text-xs text-muted-foreground">
|
||||
{through ? `数据截至 ${through} UTC · ` : ''}
|
||||
当前 {coverage.completedBuckets}/{coverage.expectedBuckets} 小时
|
||||
{coverage.lowerBoundBuckets > 0 ? ` · 下限 ${coverage.lowerBoundBuckets} 小时` : ''}
|
||||
{comparisonCoverage
|
||||
? ` · 对比 ${comparisonCoverage.completedBuckets}/${comparisonCoverage.expectedBuckets} 小时`
|
||||
: ''}
|
||||
{snapshotCoverage
|
||||
? ` · 快照 ${snapshotCoverage.completedBuckets}/${snapshotCoverage.expectedBuckets} 小时`
|
||||
: ''}
|
||||
{snapshotThrough ? ` · 快照采样于 ${snapshotThrough} UTC` : ''}
|
||||
</span>
|
||||
</div>
|
||||
)
|
||||
@@ -1599,6 +1636,42 @@ function TransferDataQualityNotice({ quality }: { quality: AdminTransferDataQual
|
||||
)
|
||||
}
|
||||
|
||||
function SharingDataQualityNotice({ quality }: { quality: AdminSharingDataQuality }) {
|
||||
if (quality.unlocatedEvents === null || quality.unlocatedEvents === 0) return null
|
||||
return (
|
||||
<div
|
||||
role="status"
|
||||
className="flex flex-col gap-2 rounded-lg border border-amber-500/40 bg-amber-500/10 px-4 py-3 text-sm sm:flex-row sm:items-center"
|
||||
>
|
||||
<Badge variant="outline" className="w-fit border-amber-600/50 text-amber-700 dark:text-amber-300">
|
||||
历史事件无法定位
|
||||
</Badge>
|
||||
<span className="text-muted-foreground">
|
||||
现存分享累计计数中有 {formatNumber(quality.unlocatedViews)} 次访问和 {formatNumber(quality.unlocatedDownloads)}{' '}
|
||||
次下载没有可信时间戳;区间值仅为已定位下限,相关环比和每百次访问比例已隐藏。
|
||||
</span>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function StorageUsageDataQualityNotice({ quality }: { quality: AdminDashboardStorageStats['dataQuality'] }) {
|
||||
if (quality.usageDriftSpaces === null || quality.usageDriftSpaces === 0) return null
|
||||
return (
|
||||
<div
|
||||
role="status"
|
||||
className="flex flex-col gap-2 rounded-lg border border-red-500/40 bg-red-500/10 px-4 py-3 text-sm sm:flex-row sm:items-center"
|
||||
>
|
||||
<Badge variant="outline" className="w-fit border-red-600/50 text-red-700 dark:text-red-300">
|
||||
占用账本异常
|
||||
</Badge>
|
||||
<span className="text-muted-foreground">
|
||||
{formatNumber(quality.usageDriftSpaces)} 个空间的配额占用账本与可计费对象相差{' '}
|
||||
{formatSize(quality.usageDriftBytes)};修复前不要把配额水位作为准确事实。
|
||||
</span>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
function EmptyState() {
|
||||
return (
|
||||
<div className="rounded-lg border border-dashed border-border/70 bg-muted/10 p-8 text-center text-sm text-muted-foreground">
|
||||
@@ -1646,10 +1719,15 @@ function normalizeDateRange(range: DateRange): DateRange {
|
||||
return { from: range.to, to: range.from }
|
||||
}
|
||||
|
||||
function formatNumber(value: number): string {
|
||||
function formatNumber(value: number | null): string {
|
||||
if (value === null) return '—'
|
||||
return new Intl.NumberFormat().format(value)
|
||||
}
|
||||
|
||||
function formatSize(value: number | null): string {
|
||||
return value === null ? '—' : formatByteSize(value)
|
||||
}
|
||||
|
||||
function formatChartDate(value: unknown): string {
|
||||
const text = String(value)
|
||||
const match = /^(\d{4})-(\d{2})-(\d{2})/.exec(text)
|
||||
@@ -1671,9 +1749,10 @@ function formatPer100(value: number | null): string {
|
||||
}
|
||||
|
||||
function formatDelta(
|
||||
delta: { value: number; previousValue: number; change: number; changePercent: number | null },
|
||||
delta: { value: number | null; previousValue: number | null; change: number | null; changePercent: number | null },
|
||||
valueFormatter: (value: number) => string = formatNumber,
|
||||
): string {
|
||||
if (delta.change === null) return '—'
|
||||
const sign = delta.change >= 0 ? '+' : ''
|
||||
return `${sign}${valueFormatter(delta.change)} (${formatPercent(delta.changePercent)})`
|
||||
}
|
||||
|
||||
@@ -374,7 +374,7 @@ describe('StoragePage', () => {
|
||||
expect(view.getByText('storage.trafficPeriodDetail:2026-05')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('does not mark unlimited quota usage as over cap', async () => {
|
||||
it('renders invalid storage quota separately from unlimited traffic', async () => {
|
||||
vi.mocked(getUserQuota).mockResolvedValue({
|
||||
orgId: 'org-1',
|
||||
baseQuota: 0,
|
||||
@@ -404,7 +404,8 @@ describe('StoragePage', () => {
|
||||
|
||||
await waitFor(() => expect(view.getAllByText('1.5 KB').length).toBeGreaterThan(0))
|
||||
expect(view.queryByText('storage.overCap')).toBeNull()
|
||||
expect(view.getAllByText('storage.usageNoLimit')).toHaveLength(2)
|
||||
expect(view.getByText('storage.usageInvalid')).toBeTruthy()
|
||||
expect(view.getByText('storage.usageNoLimit')).toBeTruthy()
|
||||
})
|
||||
|
||||
it('hides self-service forms when storage purchases are disabled', async () => {
|
||||
|
||||
@@ -34,6 +34,7 @@ export async function handleScheduled(event: ScheduledTrigger, env: ScheduledEnv
|
||||
const deps = createDeps(platform)
|
||||
const cloudBaseUrl = env.ZPAN_CLOUD_URL ?? ZPAN_CLOUD_URL_DEFAULT
|
||||
if (event.cron === TRAFFIC_SYNC_CRON) {
|
||||
await deps.quota.reconcileFreePlanBaselines()
|
||||
await Promise.all([
|
||||
syncPendingCloudTrafficReports(deps, { cloudBaseUrl }),
|
||||
syncPendingRemoteDownloadUsageReports(deps, { cloudBaseUrl }),
|
||||
|
||||
Reference in New Issue
Block a user