fix(analytics): enforce trustworthy offline statistics (#510)

* fix(analytics): enforce trustworthy offline statistics

Separate immutable counters from point-in-time snapshots, expose incomplete coverage instead of synthetic zeroes, and keep browser analytics result-only.

Restore finite Free quota baselines, fail closed for invalid storage quota, reconcile traffic reports fairly, and add production-safe backfill and data-quality diagnostics.

* fix(analytics): preserve global backfill totals

Group generated hourly backfill rows by their projected values so SQLite cannot resolve output aliases to source organization columns and overwrite cross-organization totals.
This commit is contained in:
Jasper Van
2026-07-20 10:23:22 -04:00
committed by GitHub
parent 3dc4b170d6
commit c85e60f200
57 changed files with 10729 additions and 661 deletions
+16 -7
View File
@@ -13,7 +13,7 @@ The dashboard is read-only and does not support report or CSV export.
## Fact writes
Metrics derived from activity facts use a validated producer contract. New transfer and sharing facts fail fast when required fields such as `bytes`, `source`, `trafficEventId`, or `shareId` are missing. Historical rows that predate this contract remain diagnosable through quality metrics.
Metrics derived from activity facts use a validated producer contract. New transfer and sharing facts fail fast when required fields such as `bytes`, `source`, `trafficEventId`, or `shareId` are missing. User signup, share creation, background-job completion, and remote-download completion each write a deterministic immutable stats fact in the same transaction as the business change. Deleting the mutable user/share/job/task row therefore does not rewrite history. Historical rows that predate this contract remain diagnosable through quality metrics.
A deduplicated share view updates `shares.views` and inserts its `share_view` fact in one database transaction. Download flows record an issued fact only after quota/metering and presigning succeed; a failed fact records the stable reason and traffic event id. If the issued-fact write fails, the flow refunds the counters it already reserved.
@@ -22,17 +22,19 @@ A deduplicated share view updates `shares.views` and inserts its `share_view` fa
Metric and dimension combinations are declared in `server/domain/admin-stats-metrics.ts`:
- Counters are additive event totals for a closed hour.
- Gauges are point-in-time snapshots written for that hour's close.
- Gauges are point-in-time snapshots captured during the current open hour and continuously replaced in that same bucket.
- High-volume source rows are grouped by the database before the Worker builds result rows; the scheduler does not load a whole hour of raw events into memory.
The ten-minute scheduler always rebuilds the latest closed UTC hour with counters and snapshots. It also checks the previous 48 hours and repairs at most three missing counter buckets per run. Historical repair does not invent old snapshots from current state.
The ten-minute scheduler finalizes counters for the latest closed UTC hour, repairs at most three missing counter buckets from the previous 48 hours, and then captures a fresh gauge snapshot into the current open UTC hour. When the next hour starts, finalization keeps the snapshot captured in its original hour, preserves its exact `snapshotObservedAt`, and upgrades that bucket's completion marker. Historical repair has no API for generating snapshots and cannot copy current state into an old bucket. The dashboard reports the actual snapshot observation time instead of presenting the bucket end as the observation time.
Each row and `stats.rollup_run` marker carries:
- the rollup schema version;
- `scope=full` for counters plus snapshots, or `scope=counters` for a historical counter repair;
- `scope=snapshots` for an open-hour snapshot, `scope=full` for a closed bucket with both counters and a previously captured snapshot, or `scope=counters` for a closed bucket whose historical snapshot is unavailable;
- `quality=exact` or `quality=lower_bound`.
Full markers carry separate `counterQuality` and `snapshotQuality` values so an incomplete historical transfer does not incorrectly downgrade an exact inventory snapshot.
A transaction deletes and reinserts the affected result scope, including its marker, so retries are idempotent. Readers reject old-version rows, rows without valid metadata, orphan rows without a compatible marker, and gauges backed only by a counter-repair marker.
## Query model
@@ -50,7 +52,7 @@ Responses with period-over-period deltas also include `comparisonCoverage`. A de
fully reconcilable when both the requested range and its comparison range are complete; the UI
shows either side's missing buckets instead of presenting a partial comparison as authoritative.
Event-only traffic coverage accepts `counters` and `full` markers. Views that depend on snapshots require `full` markers.
Event-only traffic coverage accepts `counters` and `full` markers. Closed-hour views that depend on snapshots require `full` markers; open `snapshots` markers are never queryable.
Request-time work is bounded and result-only. Charts and totals read only the aggregate dimensions they need; high-cardinality share dimensions are not loaded when querying totals. Top-share and top-space queries aggregate the result table and return at most eight ids; only those ids are hydrated from the operational tables for display names. No dashboard endpoint scans users, files, activities, shares, jobs, tasks, reports, or downloaders.
@@ -59,16 +61,23 @@ Request-time work is bounded and result-only. Charts and totals read only the ag
- Upload bytes mean successfully confirmed upload bytes.
- Download bytes and counts mean issued downloads, not client-completed transfers.
- Cloud traffic report status is a closed-hour snapshot of the current metering queue and belongs to Operations, not download success. Mutable report rows are never modeled as additive counters.
- Remote-download outcomes count terminal run transitions. A failed run followed by a successful retry contributes one failure and one completion; deleting the task does not erase either outcome.
- User, storage, share lifecycle, active job/task, downloader, report, and webhook state comes from closed-hour snapshots, not live request-time reads.
- Storage inventory includes normal files and image-hosting objects.
- Storage inventory includes non-trashed normal files and image-hosting objects.
- Quota usage includes files retained in trash until purge. `storage.trash_snapshot` exposes that file and byte total separately, so active-file inventory is not expected to equal the quota waterline by itself.
- `stats.data_quality_snapshot/storage_usage_drift` compares the quota usage ledger with all currently billable database objects. A non-zero space or byte drift is shown as an error instead of silently presenting the waterline as authoritative.
- Workspace storage quota is the active paid plan, or the active Free baseline when no paid plan is valid, plus active non-plan grants. Missing or zero effective storage entitlement is invalid and always fails closed; it never means unlimited space.
- The ten-minute maintenance cycle idempotently restores missing or accidentally revoked Free baseline entitlements before capturing quota snapshots. It preserves each existing baseline size and uses the configured finite defaults only when a baseline row is absent.
- Files older than 90 days are an age cohort, not proven cold data; the UI labels this explicitly.
- Sharing views, downloads, and saves are independent event totals. Downloads/saves per 100 views are intensity ratios, not user-level conversion funnels.
Rows with transfer events but no recoverable byte count carry `quality=lower_bound`. `stats.quality_missing_bytes` exposes affected event counts by direction and source.
Legacy share counters can exceed the number of timestamped activity facts. The snapshot metric `stats.data_quality_snapshot` records the currently provable gap for views and downloads. The sharing dashboard labels its range values as located-event lower bounds and suppresses period comparisons and per-view ratios while either gap is non-zero; it never assigns those legacy counts to invented dates.
## Backfill and validation
Run `pnpm stats:backfill -- --apply ...` after the migration in each environment. The script repairs recoverable historical fact metadata, removes incompatible result versions, and rebuilds historical counters idempotently. It writes a continuous `scope=counters` marker for every closed UTC hour from the first available fact through the latest closed hour, including hours with zero activity, and rejects missing or open-hour markers during validation. Existing `scope=full` markers and snapshot rows remain intact.
Run `pnpm stats:backfill -- --apply ...` after the migration in each environment. The script repairs recoverable historical fact metadata, creates deterministic lower-bound facts for currently recoverable signup/share/job/task history, removes incompatible result versions, and rebuilds historical counters idempotently. It writes a continuous `scope=counters` marker for every closed UTC hour from the first available immutable fact through the latest closed hour, including hours with zero activity, and rejects missing or open-hour markers during validation. Incompatible older snapshots are removed rather than relabeled as current-version facts.
The script does not fabricate historical inventory or active-user snapshots. Snapshot-backed views therefore report partial or empty coverage before full rollups began, while event-backed views can report complete historical coverage. Facts whose original byte size can no longer be recovered remain visible as `quality=lower_bound` instead of being guessed.
+2 -1
View File
@@ -89,7 +89,8 @@ Ownership:
- **Cloud is merchant of record in v2.6.** Cloud owns Stripe Checkout, Stripe subscriptions, Stripe paid webhooks, credit grants, gift-card redemption, usage debit, order state, and webhook retries.
- **The ZPan operator defines the package catalog per instance.** Package name, description, bytes, amount, currency, active state, and sort order are configured in ZPan admin.
- **Terminal buyers do not need Cloud accounts.** ZPan creates checkout requests through the existing paid-tier Cloud binding and sends the user to Cloud only for checkout handling.
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** ZPan keeps admin base quota separate from Cloud-delivered storage and traffic entitlements, then exposes effective quota as `base + entitlement`. Editing the admin base quota changes only the local base quota. The existing `0` base quota sentinel means unlimited only when no active Cloud entitlement exists for that resource.
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** Every organization keeps a Free baseline plan. An active paid plan overrides that baseline while grants add to it; expiry or revocation therefore falls back to Free automatically. Missing or zero effective storage entitlement fails closed and never means unlimited storage.
- A scheduled, idempotent reconciler restores missing or accidentally revoked Free baselines before quota snapshots while preserving existing baseline sizes.
- **Monthly storage package subscriptions stay Cloud-owned.** Cloud owns Stripe subscriptions and sends authenticated `order.quota_changed` deliveries only for entitlement-changing subscription states: initial entitlement, positive create/update deltas, renewals that replace the current entitlement period, and terminal reversals. ZPan records deliveries idempotently under a stable subscription entitlement id such as `stripe_subscription:<subscriptionId>:<orgId>` and recomputes effective quota from active entitlements.
- **Metered traffic billing is credit-backed and usage-event based.** Credits are unitless internal ZPan Cloud units, not USD and not Stripe Customer Balance. ZPan increments local monthly traffic usage before issuing a presigned download URL, reports `traffic_egress` usage to Cloud with an idempotency key, and denies the request if Cloud returns an insufficient-credit decision. ZPan does not price or debit credits.
- **Subscriptions and credits are separate.** Stripe subscriptions remain normal USD subscriptions and may include credits per billing period. Free or fully couponed subscriptions still grant included credits when the Cloud subscription or invoice state qualifies. Credits can also come from top-up purchases, gift-card redemption, and admin grants, but credits only pay usage-based charges. They do not pay subscriptions or fixed package purchases.
+2 -1
View File
@@ -43,7 +43,8 @@ The quota store lets a Pro instance operator sell monthly storage packages and m
- **Cloud is merchant of record in v2.6.** Stripe Checkout, Stripe subscriptions, Stripe webhooks, Cloud order state, credit grants, gift-card redemption, and usage debits run through ZPan Cloud.
- **The ZPan operator defines the package catalog per instance.** Admins create packages in ZPan with name, description, bytes, price, currency, active state, and sort order.
- **Terminal buyers do not need Cloud accounts.** ZPan creates binding-authenticated checkout requests through the Pro Cloud binding; Cloud uses those requests to create orders.
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** ZPan keeps admin base quota separate from Cloud-delivered storage and traffic entitlements, then exposes the effective quota as `base + entitlement`. Editing the admin base quota changes only the local base quota. The existing `0` base quota sentinel means unlimited only when no active Cloud entitlement exists for that resource.
- **Purchased storage and traffic are delivered as Cloud entitlements in v2.6.** Every organization keeps a Free baseline plan. An active paid plan overrides that baseline while grants add to it; expiry or revocation therefore falls back to Free automatically. Missing or zero effective storage entitlement fails closed and never means unlimited storage.
- The scheduled quota reconciler restores missing or previously revoked Free baseline rows before analytics snapshots, so installations affected by the old paid-plan replacement logic heal automatically after upgrade.
- **Monthly storage package subscriptions are Cloud-owned.** ZPan does not run subscription billing locally. Cloud owns Stripe subscriptions and sends authenticated `order.quota_changed` deliveries only for entitlement-changing subscription states: initial entitlement, positive create/update deltas, renewals that replace the current entitlement period, and terminal reversals. ZPan records deliveries idempotently under a stable subscription entitlement id such as `stripe_subscription:<subscriptionId>:<orgId>` and recomputes the effective quota from active entitlements.
- **Metered traffic billing is credit-backed and usage-event based.** Credits are unitless internal ZPan Cloud units, not USD and not Stripe Customer Balance. ZPan increments local monthly traffic usage before issuing a presigned download URL, reports `traffic_egress` usage to Cloud with an idempotency key, and denies the request if Cloud returns an insufficient-credit decision. ZPan does not price or debit credits.
- **Subscriptions and credits are separate.** Stripe subscriptions remain normal USD subscriptions and may include credits per billing period. Free or fully couponed subscriptions still grant included credits when the Cloud subscription or invoice state qualifies. Credits can also come from top-up purchases, gift-card redemption, and admin grants, but credits only pay usage-based charges. They do not pay subscriptions or fixed package purchases.
@@ -0,0 +1,5 @@
DROP INDEX `org_quota_entitlements_active_plan_uniq`;--> statement-breakpoint
CREATE UNIQUE INDEX `org_quota_entitlements_active_plan_uniq` ON `org_quota_entitlements` (`org_id`,`resource_type`,`entitlement_type`) WHERE status = 'active' AND entitlement_type = 'plan' AND source <> 'free_plan';--> statement-breakpoint
ALTER TABLE `cloud_traffic_reports` ADD `attempt_count` integer DEFAULT 0 NOT NULL;--> statement-breakpoint
ALTER TABLE `cloud_traffic_reports` ADD `next_retry_at` integer;--> statement-breakpoint
CREATE INDEX `cloud_traffic_reports_retry_idx` ON `cloud_traffic_reports` (`status`,`next_retry_at`,`created_at`);
@@ -0,0 +1 @@
CREATE UNIQUE INDEX `org_quotas_org_uniq` ON `org_quotas` (`org_id`);
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+14
View File
@@ -386,6 +386,20 @@
"when": 1783697537563,
"tag": "0055_hourly-admin-stats",
"breakpoints": true
},
{
"idx": 56,
"version": "6",
"when": 1784524502907,
"tag": "0056_stats-correctness-round-1",
"breakpoints": true
},
{
"idx": 57,
"version": "6",
"when": 1784526402754,
"tag": "0057_stats-quota-uniqueness",
"breakpoints": true
}
]
}
+299 -71
View File
@@ -143,6 +143,7 @@ SET metadata = json_set(
ELSE ''
END
)
AND ctr.status <> 'blocked'
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
ORDER BY ctr.created_at, ctr.event_id
LIMIT 1
@@ -159,6 +160,7 @@ SET metadata = json_set(
ELSE ''
END
)
AND ctr.status <> 'blocked'
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
ORDER BY ctr.created_at, ctr.event_id
LIMIT 1
@@ -176,6 +178,7 @@ SET metadata = json_set(
ELSE ''
END
)
AND ctr.status <> 'blocked'
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
ORDER BY ctr.created_at, ctr.event_id
LIMIT 1
@@ -196,6 +199,7 @@ WHERE action IN ('share_download', 'object_download', 'image_hosting_download',
ELSE ''
END
)
AND ctr.status <> 'blocked'
AND ABS(CAST(ctr.created_at / 1000 AS INTEGER) - ae.created_at) <= 5
);
@@ -237,6 +241,7 @@ WHERE ctr.source IN ('direct_share', 'landing_share', 'image_hosting', 'object_d
OR (
ae.target_id = ctr.source_id
AND ae.action = CASE
WHEN ctr.status = 'blocked' THEN 'download_failed'
WHEN ctr.source IN ('direct_share', 'landing_share') THEN 'share_download'
WHEN ctr.source = 'image_hosting' THEN 'image_hosting_download'
WHEN ctr.source = 'object_download' THEN 'object_download'
@@ -246,6 +251,8 @@ WHERE ctr.source IN ('direct_share', 'landing_share', 'image_hosting', 'object_d
)
);
${buildImmutableFactBackfillSql()}
${purgeLegacyRollupsSql()}
${purgeOpenRollupsSql(now)}
${purgeCounterRollupsSql()}
@@ -254,6 +261,127 @@ ${buildHourlyBackfillSql(now)}
`
}
function buildImmutableFactBackfillSql(): string {
return `INSERT OR IGNORE INTO activity_events (
id, org_id, user_id, actor_type, actor_ref, action, target_type,
target_id, target_name, metadata, created_at
)
SELECT
'stats:stats_share_created:' || ae.target_id,
ae.org_id, NULL, 'system', 'stats-backfill', 'stats_share_created', 'share',
ae.target_id, ae.target_id,
json_object(
'kind', COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END, 'unknown'),
'statsQuality', 'lower_bound'
),
ae.created_at
FROM activity_events ae
WHERE ae.action = 'share_create' AND ae.target_id IS NOT NULL;
INSERT OR IGNORE INTO activity_events (
id, org_id, user_id, actor_type, actor_ref, action, target_type,
target_id, target_name, metadata, created_at
)
SELECT
'stats:stats_share_created:' || s.id,
s.org_id, NULL, 'system', 'stats-backfill', 'stats_share_created', 'share',
s.id, s.id, json_object('kind', s.kind, 'statsQuality', 'lower_bound'), s.created_at
FROM shares s;
INSERT OR IGNORE INTO activity_events (
id, org_id, user_id, actor_type, actor_ref, action, target_type,
target_id, target_name, metadata, created_at
)
SELECT
'stats:stats_user_signup:' || u.id,
COALESCE((
SELECT m.organization_id
FROM member m
JOIN organization o ON o.id = m.organization_id
WHERE m.user_id = u.id AND o.metadata LIKE '%"type":"personal"%'
ORDER BY m.created_at, m.id
LIMIT 1
), ''),
NULL, 'system', 'stats-backfill', 'stats_user_signup', 'user',
u.id, u.id,
json_object(
'provider', COALESCE((
SELECT a.provider_id FROM account a WHERE a.user_id = u.id ORDER BY a.created_at, a.id LIMIT 1
), 'direct'),
'statsQuality', 'lower_bound'
),
CAST(u.created_at / 1000 AS INTEGER)
FROM user u;
INSERT OR IGNORE INTO activity_events (
id, org_id, user_id, actor_type, actor_ref, action, target_type,
target_id, target_name, metadata, created_at
)
SELECT
'stats:stats_background_job_finished:' || bj.id,
bj.org_id, NULL, 'system', 'stats-backfill', 'stats_background_job_finished', 'background_job',
bj.id, bj.id,
json_object('jobType', bj.type, 'outcome', bj.status, 'statsQuality', 'lower_bound'),
CAST(bj.finished_at / 1000 AS INTEGER)
FROM background_jobs bj
WHERE bj.finished_at IS NOT NULL;
INSERT OR IGNORE INTO activity_events (
id, org_id, user_id, actor_type, actor_ref, action, target_type,
target_id, target_name, metadata, created_at
)
SELECT
'stats:stats_remote_download_finished:event:' || ae.id,
ae.org_id, NULL, 'system', 'stats-backfill', 'stats_remote_download_finished', 'remote_download',
ae.target_id, ae.target_id,
json_object(
'category', COALESCE(dt.category, 'uncategorized'),
'downloaderId', dt.assigned_downloader_id,
'outcome', CASE ae.action
WHEN 'download_task_completed' THEN 'completed'
WHEN 'download_task_failed' THEN 'failed'
ELSE 'canceled'
END,
'bytes', CASE WHEN ae.action = 'download_task_completed' THEN COALESCE(dt.billing_charged_bytes, 0) ELSE 0 END,
'statsQuality', 'lower_bound'
),
ae.created_at
FROM activity_events ae
LEFT JOIN download_tasks dt ON dt.id = ae.target_id
WHERE ae.action IN ('download_task_completed', 'download_task_failed', 'download_task_canceled')
AND ae.target_id IS NOT NULL;
INSERT OR IGNORE INTO activity_events (
id, org_id, user_id, actor_type, actor_ref, action, target_type,
target_id, target_name, metadata, created_at
)
SELECT
'stats:stats_remote_download_finished:' || dt.id,
dt.org_id, NULL, 'system', 'stats-backfill', 'stats_remote_download_finished', 'remote_download',
dt.id, dt.id,
json_object(
'category', COALESCE(dt.category, 'uncategorized'),
'downloaderId', dt.assigned_downloader_id,
'outcome', dt.status,
'bytes', dt.billing_charged_bytes,
'statsQuality', 'lower_bound'
),
CAST(dt.finished_at / 1000 AS INTEGER)
FROM download_tasks dt
WHERE dt.finished_at IS NOT NULL
AND dt.status IN ('completed', 'failed', 'canceled')
AND NOT EXISTS (
SELECT 1 FROM activity_events ae
WHERE ae.target_id = dt.id
AND ae.action = CASE dt.status
WHEN 'completed' THEN 'download_task_completed'
WHEN 'failed' THEN 'download_task_failed'
WHEN 'canceled' THEN 'download_task_canceled'
ELSE ''
END
);`
}
type HourlySource = {
metric: string
source: string
@@ -270,6 +398,22 @@ type HourlySource = {
function buildHourlyBackfillSql(now: Date): string {
const currentHour = Math.floor(now.getTime() / 3_600_000) * 3_600_000
const missingShareViewsQuality = `CASE WHEN EXISTS (
SELECT 1 FROM shares s
WHERE s.views > (
SELECT COUNT(*) FROM activity_events history
WHERE history.action = 'share_view'
AND COALESCE(CASE WHEN json_valid(history.metadata) = 1 THEN json_extract(history.metadata, '$.shareId') END, history.target_id) = s.id
)
) THEN 'lower_bound' ELSE 'exact' END`
const missingShareDownloadsQuality = `CASE WHEN EXISTS (
SELECT 1 FROM shares s
WHERE s.downloads > (
SELECT COUNT(*) FROM activity_events history
WHERE history.action = 'share_download'
AND COALESCE(CASE WHEN json_valid(history.metadata) = 1 THEN json_extract(history.metadata, '$.shareId') END, history.target_id) = s.id
)
) THEN 'lower_bound' ELSE 'exact' END`
const sources: HourlySource[] = [
{
metric: 'transfer.upload',
@@ -293,7 +437,10 @@ function buildHourlyBackfillSql(now: Date): string {
org: 'ae.org_id',
where: "ae.action IN ('share_download','object_download','image_hosting_download','webdav_download')",
bytes: "SUM(CASE WHEN json_valid(ae.metadata) = 1 THEN COALESCE(json_extract(ae.metadata, '$.bytes'), 0) ELSE 0 END)",
quality: "CASE WHEN SUM(CASE WHEN ae.metadata IS NULL OR json_valid(ae.metadata) = 0 OR json_type(ae.metadata, '$.bytes') IS NULL THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
quality: `CASE WHEN
SUM(CASE WHEN ae.metadata IS NULL OR json_valid(ae.metadata) = 0 OR json_type(ae.metadata, '$.bytes') IS NULL THEN 1 ELSE 0 END) > 0
OR (${missingShareDownloadsQuality}) = 'lower_bound'
THEN 'lower_bound' ELSE 'exact' END`,
dimensions: {
source: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.source') END, CASE ae.action WHEN 'object_download' THEN 'object_download' WHEN 'image_hosting_download' THEN 'image_hosting' WHEN 'webdav_download' THEN 'webdav_download' ELSE 'landing_share' END)",
actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)",
@@ -314,31 +461,45 @@ function buildHourlyBackfillSql(now: Date): string {
},
{
metric: 'share.created',
source: 'shares s',
timestampMs: 's.created_at * 1000',
org: 's.org_id',
dimensions: { kind: 's.kind' },
source: 'activity_events ae',
timestampMs: 'ae.created_at * 1000',
org: 'ae.org_id',
where: "ae.action = 'stats_share_created'",
quality:
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
dimensions: { kind: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END, 'unknown')" },
},
activitySource('share.view', "ae.action = 'share_view'", { share_id: 'ae.target_id', actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }),
activitySource('share.download_issued', "ae.action = 'share_download'", { share_id: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.shareId') END, ae.target_id)", kind: "CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END", source: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.source') END, 'landing_share')", actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, true),
activitySource('share.view', "ae.action = 'share_view'", { share_id: 'ae.target_id', actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, false, missingShareViewsQuality),
activitySource('share.download_issued', "ae.action = 'share_download'", { share_id: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.shareId') END, ae.target_id)", kind: "CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.kind') END", source: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.source') END, 'landing_share')", actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, true, missingShareDownloadsQuality),
activitySource('share.saved', "ae.action = 'save_from_share'", { share_id: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.shareId') END, ae.target_id)", actor_type: "COALESCE(ae.actor_type, CASE WHEN ae.user_id IS NULL THEN 'anonymous' ELSE 'user' END)" }, true),
activitySource('share.password_passed', "ae.action = 'share_password_passed'", { share_id: 'ae.target_id' }),
{
metric: 'remote_download.task_finished',
source: 'download_tasks dt',
timestampMs: 'dt.finished_at',
org: 'dt.org_id',
where: 'dt.finished_at IS NOT NULL',
bytes: 'SUM(dt.billing_charged_bytes)',
dimensions: { category: "COALESCE(dt.category, 'uncategorized')", downloader_id: 'dt.assigned_downloader_id', outcome: 'dt.status' },
source: 'activity_events ae',
timestampMs: 'ae.created_at * 1000',
org: 'ae.org_id',
where: "ae.action = 'stats_remote_download_finished'",
bytes: "SUM(CASE WHEN json_valid(ae.metadata) = 1 THEN COALESCE(json_extract(ae.metadata, '$.bytes'), 0) ELSE 0 END)",
quality:
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
dimensions: {
category: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.category') END, 'uncategorized')",
downloader_id: "CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.downloaderId') END",
outcome: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.outcome') END, 'unknown')",
},
},
{
metric: 'background_job.finished',
source: 'background_jobs bj',
timestampMs: 'bj.finished_at',
org: 'bj.org_id',
where: 'bj.finished_at IS NOT NULL',
dimensions: { job_type: 'bj.type', outcome: 'bj.status' },
source: 'activity_events ae',
timestampMs: 'ae.created_at * 1000',
org: 'ae.org_id',
where: "ae.action = 'stats_background_job_finished'",
quality:
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
dimensions: {
job_type: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.jobType') END, 'unknown')",
outcome: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.outcome') END, 'unknown')",
},
},
{
metric: 'stats.quality_missing_bytes',
@@ -361,8 +522,8 @@ function buildHourlyBackfillSql(now: Date): string {
function purgeLegacyRollupsSql(): string {
return `DELETE FROM stats_rollups_hourly
WHERE CASE WHEN json_valid(metadata) = 1 THEN
json_extract(metadata, '$.version') = 2
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
json_extract(metadata, '$.version') = 3
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
ELSE 0 END = 0;`
}
@@ -386,7 +547,13 @@ WHERE metric_key IN (
);`
}
function activitySource(metric: string, where: string, dimensions: Record<string, string>, bytes = false): HourlySource {
function activitySource(
metric: string,
where: string,
dimensions: Record<string, string>,
bytes = false,
quality?: string,
): HourlySource {
return {
metric,
source: 'activity_events ae',
@@ -394,6 +561,7 @@ function activitySource(metric: string, where: string, dimensions: Record<string
org: 'ae.org_id',
where,
bytes: bytes ? "SUM(CASE WHEN json_valid(ae.metadata) = 1 THEN COALESCE(json_extract(ae.metadata, '$.bytes'), 0) ELSE 0 END)" : undefined,
quality,
dimensions,
}
}
@@ -428,14 +596,14 @@ SELECT
CAST(bucket_start AS TEXT) || ':' || COALESCE(NULLIF(org_id, ''), 'global') || ':${source.metric}:${dimensionKey || 'all'}:' || hex(dimension_value),
bucket_start, org_id, '${source.metric}', '${dimensionKey}', dimension_value,
count_value, bytes_value, unique_value,
json_object('version', 2, 'scope', '${source.scope ?? 'counters'}', 'quality', quality_value),
json_object('version', 3, 'scope', '${source.scope ?? 'counters'}', 'quality', quality_value),
bucket_start + 3600000
FROM (
SELECT ${bucket} AS bucket_start, ${source.org} AS org_id, ${dimension} AS dimension_value,
${count} AS count_value, ${bytes} AS bytes_value, ${uniqueCount} AS unique_value, ${quality} AS quality_value
FROM ${source.source}
${where ? `WHERE ${where}` : ''}
GROUP BY bucket_start, org_id${dimensionKey ? ', dimension_value' : ''}
GROUP BY 1, 2${dimensionKey ? ', 3' : ''}
) rollup
WHERE true
ON CONFLICT(bucket_start, org_id, metric_key, dimension_key, dimension_value)
@@ -447,12 +615,15 @@ WHERE count <> excluded.count OR bytes <> excluded.bytes OR unique_count <> excl
function userSignupBackfillSql(before: number): string {
return hourlyStatements({
metric: 'user.signup',
source: `user u LEFT JOIN account a ON a.id = (
SELECT a2.id FROM account a2 WHERE a2.user_id = u.id ORDER BY a2.created_at, a2.id LIMIT 1
)`,
timestampMs: 'u.created_at',
source: 'activity_events ae',
timestampMs: 'ae.created_at * 1000',
org: "''",
dimensions: { provider: "COALESCE(a.provider_id, 'direct')" },
where: "ae.action = 'stats_user_signup'",
quality:
"CASE WHEN SUM(CASE WHEN json_valid(ae.metadata) = 1 AND json_extract(ae.metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) > 0 THEN 'lower_bound' ELSE 'exact' END",
dimensions: {
provider: "COALESCE(CASE WHEN json_valid(ae.metadata) = 1 THEN json_extract(ae.metadata, '$.provider') END, 'unknown')",
},
}, before).join('\n\n')
}
@@ -477,50 +648,100 @@ buckets AS (
FROM bounds
JOIN numbers ON start_at + numbers.n * 3600000 <= end_at
WHERE start_at IS NOT NULL AND start_at <= end_at
),
snapshot_markers AS MATERIALIZED (
SELECT
bucket_start,
COALESCE(json_extract(metadata, '$.snapshotQuality'), json_extract(metadata, '$.quality'), 'exact') AS quality,
COALESCE(json_extract(metadata, '$.snapshotObservedAt'), json_extract(metadata, '$.observedAt')) AS observed_at
FROM stats_rollups_hourly
WHERE metric_key = 'stats.rollup_run' AND org_id = '' AND dimension_key = '' AND dimension_value = ''
AND json_valid(metadata) = 1
AND json_extract(metadata, '$.version') = 3
AND json_extract(metadata, '$.scope') IN ('snapshots', 'full')
AND COALESCE(json_extract(metadata, '$.snapshotObservedAt'), json_extract(metadata, '$.observedAt')) IS NOT NULL
)
INSERT INTO stats_rollups_hourly (
id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
count, bytes, unique_count, metadata, updated_at
)
SELECT
CAST(bucket_start AS TEXT) || ':global:stats.rollup_run:all:all',
bucket_start, '', 'stats.rollup_run', '', '', 1, 0, 0,
json_object('version', 2, 'scope', 'counters', 'quality', 'exact'),
bucket_start + 3600000
CAST(buckets.bucket_start AS TEXT) || ':global:stats.rollup_run:all:all',
buckets.bucket_start, '', 'stats.rollup_run', '', '', 1, 0, 0,
CASE WHEN snapshot_markers.bucket_start IS NULL THEN
json_object(
'version', 3,
'scope', 'counters',
'quality', CASE WHEN EXISTS (
SELECT 1 FROM stats_rollups_hourly result
WHERE result.bucket_start = buckets.bucket_start
AND result.metric_key <> 'stats.rollup_run'
AND json_valid(result.metadata) = 1
AND json_extract(result.metadata, '$.version') = 3
AND json_extract(result.metadata, '$.scope') = 'counters'
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
) THEN 'lower_bound' ELSE 'exact' END,
'counterQuality', CASE WHEN EXISTS (
SELECT 1 FROM stats_rollups_hourly result
WHERE result.bucket_start = buckets.bucket_start
AND result.metric_key <> 'stats.rollup_run'
AND json_valid(result.metadata) = 1
AND json_extract(result.metadata, '$.version') = 3
AND json_extract(result.metadata, '$.scope') = 'counters'
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
) THEN 'lower_bound' ELSE 'exact' END
)
ELSE
json_object(
'version', 3,
'scope', 'full',
'quality', CASE WHEN snapshot_markers.quality = 'lower_bound' OR EXISTS (
SELECT 1 FROM stats_rollups_hourly result
WHERE result.bucket_start = buckets.bucket_start
AND result.metric_key <> 'stats.rollup_run'
AND json_valid(result.metadata) = 1
AND json_extract(result.metadata, '$.version') = 3
AND json_extract(result.metadata, '$.scope') = 'counters'
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
) THEN 'lower_bound' ELSE 'exact' END,
'counterQuality', CASE WHEN EXISTS (
SELECT 1 FROM stats_rollups_hourly result
WHERE result.bucket_start = buckets.bucket_start
AND result.metric_key <> 'stats.rollup_run'
AND json_valid(result.metadata) = 1
AND json_extract(result.metadata, '$.version') = 3
AND json_extract(result.metadata, '$.scope') = 'counters'
AND json_extract(result.metadata, '$.quality') = 'lower_bound'
) THEN 'lower_bound' ELSE 'exact' END,
'snapshotQuality', snapshot_markers.quality,
'snapshotObservedAt', snapshot_markers.observed_at
)
END,
buckets.bucket_start + 3600000
FROM buckets
LEFT JOIN snapshot_markers ON snapshot_markers.bucket_start = buckets.bucket_start
WHERE true
ON CONFLICT(bucket_start, org_id, metric_key, dimension_key, dimension_value)
DO UPDATE SET count = excluded.count, bytes = excluded.bytes, unique_count = excluded.unique_count,
metadata = excluded.metadata, updated_at = excluded.updated_at
WHERE CASE WHEN json_valid(stats_rollups_hourly.metadata) = 1 THEN
json_extract(stats_rollups_hourly.metadata, '$.version') = 2
AND json_extract(stats_rollups_hourly.metadata, '$.scope') = 'full'
AND json_extract(stats_rollups_hourly.metadata, '$.quality') IN ('exact', 'lower_bound')
ELSE 0 END = 0
AND (
count <> excluded.count OR bytes <> excluded.bytes OR unique_count <> excluded.unique_count
OR metadata <> excluded.metadata OR updated_at <> excluded.updated_at
);`
WHERE count <> excluded.count OR bytes <> excluded.bytes OR unique_count <> excluded.unique_count
OR metadata <> excluded.metadata OR updated_at <> excluded.updated_at;`
}
function statsHistoryStartSql(): string {
const missing = '9223372036854775807'
return `NULLIF(MIN(
COALESCE((SELECT CAST(MIN(u.created_at) / 3600000 AS INTEGER) * 3600000
FROM user u WHERE u.created_at >= ${MIN_VALID_TIMESTAMP_MS}), ${missing}),
COALESCE((SELECT CAST((MIN(s.created_at) * 1000) / 3600000 AS INTEGER) * 3600000
FROM shares s WHERE s.created_at >= ${MIN_VALID_TIMESTAMP_SECONDS}), ${missing}),
COALESCE((SELECT CAST((MIN(ae.created_at) * 1000) / 3600000 AS INTEGER) * 3600000
FROM activity_events ae WHERE ae.created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND ae.action IN (
return `NULLIF(COALESCE((
SELECT CAST((MIN(ae.created_at) * 1000) / 3600000 AS INTEGER) * 3600000
FROM activity_events ae
WHERE ae.created_at >= ${MIN_VALID_TIMESTAMP_SECONDS}
AND ae.action IN (
'upload_confirm', 'upload_cancel', 'upload_failed',
'share_download', 'object_download', 'image_hosting_download', 'webdav_download', 'download_failed',
'share_view', 'save_from_share', 'share_password_passed'
)), ${missing}),
COALESCE((SELECT CAST(MIN(dt.finished_at) / 3600000 AS INTEGER) * 3600000
FROM download_tasks dt WHERE dt.finished_at IS NOT NULL AND dt.finished_at >= ${MIN_VALID_TIMESTAMP_MS}), ${missing}),
COALESCE((SELECT CAST(MIN(bj.finished_at) / 3600000 AS INTEGER) * 3600000
FROM background_jobs bj WHERE bj.finished_at IS NOT NULL AND bj.finished_at >= ${MIN_VALID_TIMESTAMP_MS}), ${missing})
), ${missing})`
'share_view', 'save_from_share', 'share_password_passed',
'stats_user_signup', 'stats_share_created',
'stats_remote_download_finished', 'stats_background_job_finished'
)
), ${missing}), ${missing})`
}
export function buildValidationSql(now = new Date()): string {
@@ -589,12 +810,14 @@ export function buildValidationSql(now = new Date()): string {
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
),
'rawUserSignups', (
SELECT COUNT(*) FROM user
WHERE created_at >= ${MIN_VALID_TIMESTAMP_MS} AND created_at < ${currentHour}
SELECT COUNT(*) FROM activity_events
WHERE action = 'stats_user_signup'
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
),
'rawSharesCreated', (
SELECT COUNT(*) FROM shares
WHERE created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
SELECT COUNT(*) FROM activity_events
WHERE action = 'stats_share_created'
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
),
'rawShareDownloads', (
SELECT COUNT(*) FROM activity_events
@@ -617,12 +840,14 @@ export function buildValidationSql(now = new Date()): string {
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
),
'rawFinishedDownloadTasks', (
SELECT COUNT(*) FROM download_tasks
WHERE finished_at >= ${MIN_VALID_TIMESTAMP_MS} AND finished_at < ${currentHour}
SELECT COUNT(*) FROM activity_events
WHERE action = 'stats_remote_download_finished'
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
),
'rawFinishedBackgroundJobs', (
SELECT COUNT(*) FROM background_jobs
WHERE finished_at >= ${MIN_VALID_TIMESTAMP_MS} AND finished_at < ${currentHour}
SELECT COUNT(*) FROM activity_events
WHERE action = 'stats_background_job_finished'
AND created_at >= ${MIN_VALID_TIMESTAMP_SECONDS} AND created_at * 1000 < ${currentHour}
),
'rawMissingByteEvents', (
SELECT COUNT(*) FROM activity_events
@@ -636,8 +861,8 @@ export function buildValidationSql(now = new Date()): string {
SELECT *
FROM stats_rollups_hourly
WHERE CASE WHEN json_valid(metadata) = 1 THEN
json_extract(metadata, '$.version') = 2
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
json_extract(metadata, '$.version') = 3
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
ELSE 0 END = 1
),
@@ -686,8 +911,8 @@ SELECT json_object(
'legacyRollupRows', (
SELECT COUNT(*) FROM stats_rollups_hourly
WHERE CASE WHEN json_valid(metadata) = 1 THEN
json_extract(metadata, '$.version') = 2
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
json_extract(metadata, '$.version') = 3
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
ELSE 0 END = 0
)
@@ -697,14 +922,15 @@ SELECT json_object(
SELECT *
FROM stats_rollups_hourly
WHERE CASE WHEN json_valid(metadata) = 1 THEN
json_extract(metadata, '$.version') = 2
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
json_extract(metadata, '$.version') = 3
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
ELSE 0 END = 1
),
counter_markers AS MATERIALIZED (
SELECT bucket_start FROM valid_rollups
WHERE metric_key = 'stats.rollup_run' AND org_id = '' AND dimension_key = '' AND dimension_value = ''
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
),
counter_rows AS MATERIALIZED (
SELECT result.* FROM valid_rollups result
@@ -759,10 +985,11 @@ SELECT json_object(
FROM stats_rollups_hourly
WHERE metric_key = 'stats.rollup_run' AND org_id = '' AND dimension_key = '' AND dimension_value = ''
AND CASE WHEN json_valid(metadata) = 1 THEN
json_extract(metadata, '$.version') = 2
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
json_extract(metadata, '$.version') = 3
AND json_extract(metadata, '$.scope') IN ('counters', 'snapshots', 'full')
AND json_extract(metadata, '$.quality') IN ('exact', 'lower_bound')
ELSE 0 END = 1
AND json_extract(metadata, '$.scope') IN ('counters', 'full')
),
coverage AS MATERIALIZED (
SELECT ${statsHistoryStartSql()} AS start_at, ${latestClosedHour} AS end_at
@@ -828,6 +1055,7 @@ SELECT json_object(
OR (
ae.target_id = ctr.source_id
AND ae.action = CASE
WHEN ctr.status = 'blocked' THEN 'download_failed'
WHEN ctr.source IN ('direct_share', 'landing_share') THEN 'share_download'
WHEN ctr.source = 'image_hosting' THEN 'image_hosting_download'
WHEN ctr.source = 'object_download' THEN 'object_download'
+10 -3
View File
@@ -1,10 +1,11 @@
import { and, count, desc, eq, gte, lte } from 'drizzle-orm'
import { and, count, desc, eq, gte, lte, notInArray } from 'drizzle-orm'
import { nanoid } from 'nanoid'
import { organization, user } from '../../db/auth-schema'
import { activityEvents } from '../../db/schema'
import { assertAdminStatsEvent } from '../../domain/admin-stats-events'
import type { Database } from '../../platform/interface'
import type { ActivityActorType, ActivityRepo, RecordActivityInput } from '../../usecases/ports'
import { ADMIN_STATS_FACT_ACTIONS } from './admin-stats-fact'
export function activityEventValues(event: RecordActivityInput): typeof activityEvents.$inferInsert {
assertAdminStatsEvent(event.action, event.metadata)
@@ -47,7 +48,11 @@ export function createActivityRepo(db: Database): ActivityRepo {
const pageSize = opts.pageSize ?? 20
const offset = (page - 1) * pageSize
const countRows = await db.select({ count: count() }).from(activityEvents).where(eq(activityEvents.orgId, orgId))
const visible = and(
eq(activityEvents.orgId, orgId),
notInArray(activityEvents.action, [...ADMIN_STATS_FACT_ACTIONS]),
)
const countRows = await db.select({ count: count() }).from(activityEvents).where(visible)
const total = countRows[0]?.count ?? 0
const rows = await db
@@ -68,7 +73,7 @@ export function createActivityRepo(db: Database): ActivityRepo {
})
.from(activityEvents)
.leftJoin(user, eq(activityEvents.userId, user.id))
.where(eq(activityEvents.orgId, orgId))
.where(visible)
.orderBy(desc(activityEvents.createdAt))
.limit(pageSize)
.offset(offset)
@@ -104,6 +109,7 @@ export function createActivityRepo(db: Database): ActivityRepo {
const offset = (page - 1) * pageSize
const filters = [
notInArray(activityEvents.action, [...ADMIN_STATS_FACT_ACTIONS]),
opts.orgId ? eq(activityEvents.orgId, opts.orgId) : undefined,
opts.userId ? eq(activityEvents.userId, opts.userId) : undefined,
opts.action ? eq(activityEvents.action, opts.action) : undefined,
@@ -176,6 +182,7 @@ export function createActivityRepo(db: Database): ActivityRepo {
eq(activityEvents.orgId, opts.orgId),
eq(activityEvents.targetType, opts.targetType),
eq(activityEvents.targetId, opts.targetId),
notInArray(activityEvents.action, [...ADMIN_STATS_FACT_ACTIONS]),
)
const [countRows, rows] = await Promise.all([
+35
View File
@@ -0,0 +1,35 @@
import type { activityEvents } from '../../db/schema'
export const ADMIN_STATS_FACT_ACTIONS = [
'stats_user_signup',
'stats_share_created',
'stats_background_job_finished',
'stats_remote_download_finished',
] as const
export type AdminStatsFactAction = (typeof ADMIN_STATS_FACT_ACTIONS)[number]
export function adminStatsFactValues(input: {
action: AdminStatsFactAction
sourceId: string
targetId?: string
orgId: string
targetType: string
occurredAt: Date
metadata: Record<string, unknown>
}): typeof activityEvents.$inferInsert {
const targetId = input.targetId ?? input.sourceId
return {
id: `stats:${input.action}:${input.sourceId}`,
orgId: input.orgId,
userId: null,
actorType: 'system',
actorRef: 'admin-stats',
action: input.action,
targetType: input.targetType,
targetId,
targetName: targetId,
metadata: JSON.stringify({ ...input.metadata, statsQuality: 'exact' }),
createdAt: input.occurredAt,
}
}
+60 -29
View File
@@ -5,6 +5,7 @@ import {
ADMIN_STATS_METRICS,
type AdminStatsDimension,
type AdminStatsMetric,
type AdminStatsRollupMetadata,
type AdminStatsRollupScope,
assertMetricDimension,
metricDefinition,
@@ -33,7 +34,7 @@ export class AdminStatsHourlyReader {
private readonly queryFrom: Date
private readonly queryTo: Date
private readonly metricRows = new Map<string, Promise<HourlyMetricRow[]>>()
private readonly markerBucketsPromises = new Map<AdminStatsRollupScope, Promise<Set<number>>>()
private readonly markerRowsPromises = new Map<AdminStatsRollupScope, Promise<CompatibleMarkerRow[]>>()
constructor(
private readonly db: Database,
@@ -121,19 +122,19 @@ export class AdminStatsHourlyReader {
AND bucket_start >= ${this.queryFrom.getTime()}
AND bucket_start < ${this.queryTo.getTime()}
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.version') END = ${ROLLUP_VERSION}
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.scope') END = 'full'
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.scope') END IN ('snapshots', 'full')
AND CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.quality') END IN ('exact', 'lower_bound')
)
SELECT used.org_id AS orgId, used.bytes AS usedBytes, COALESCE(quota.bytes, 0) AS quotaBytes
SELECT used.org_id AS orgId, used.bytes AS usedBytes, quota.bytes AS quotaBytes
FROM stats_rollups_hourly used
LEFT JOIN stats_rollups_hourly quota
INNER JOIN stats_rollups_hourly quota
ON quota.bucket_start = used.bucket_start
AND quota.org_id = used.org_id
AND quota.metric_key = ${ADMIN_STATS_METRICS.storageQuota}
AND quota.dimension_key = ''
AND quota.dimension_value = ''
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.version') END = ${ROLLUP_VERSION}
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.scope') END = 'full'
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.scope') END IN ('snapshots', 'full')
AND CASE WHEN json_valid(quota.metadata) = 1 THEN json_extract(quota.metadata, '$.quality') END IN ('exact', 'lower_bound')
WHERE used.bucket_start = (SELECT bucketStart FROM latest)
AND used.org_id <> ''
@@ -141,7 +142,7 @@ export class AdminStatsHourlyReader {
AND used.dimension_key = ''
AND used.dimension_value = ''
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.version') END = ${ROLLUP_VERSION}
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.scope') END = 'full'
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.scope') END IN ('snapshots', 'full')
AND CASE WHEN json_valid(used.metadata) = 1 THEN json_extract(used.metadata, '$.quality') END IN ('exact', 'lower_bound')
ORDER BY used.bytes DESC, used.org_id
LIMIT ${DASHBOARD_RANKING_LIMIT}
@@ -155,14 +156,22 @@ export class AdminStatsHourlyReader {
async coverage(requiredScope: AdminStatsRollupScope = 'full'): Promise<AdminStatsCoverage> {
const expectedBuckets = Math.max(0, Math.floor((this.queryTo.getTime() - this.queryFrom.getTime()) / HOUR_MS))
const markerBuckets = await this.markerBuckets(requiredScope)
const completedBuckets = markerBuckets.size
const latest = Math.max(...markerBuckets, Number.NEGATIVE_INFINITY)
const markerRows = await this.markerRows(requiredScope)
const completedBuckets = markerRows.length
const lowerBoundBuckets = markerRows.filter(
(row) => qualityForScope(row.metadata, requiredScope) === 'lower_bound',
).length
const latest = markerRows.reduce<CompatibleMarkerRow | null>(
(value, row) => (!value || row.bucketStart.getTime() > value.bucketStart.getTime() ? row : value),
null,
)
return {
status: completedBuckets === 0 ? 'empty' : completedBuckets === expectedBuckets ? 'complete' : 'partial',
expectedBuckets,
completedBuckets,
dataThrough: Number.isFinite(latest) ? new Date(latest + HOUR_MS).toISOString() : null,
lowerBoundBuckets,
quality: lowerBoundBuckets > 0 ? 'lower_bound' : 'exact',
dataThrough: latest ? dataThroughForScope(latest, requiredScope) : null,
}
}
@@ -179,7 +188,7 @@ export class AdminStatsHourlyReader {
dimensionKeys: readonly (AdminStatsDimension | '')[],
): Promise<HourlyMetricRow[]> {
if (this.queryFrom >= this.queryTo) return []
const requiredScope = metricDefinition(metric).kind === 'gauge' ? 'full' : 'counters'
const requiredScope = metricDefinition(metric).kind === 'gauge' ? 'snapshots' : 'counters'
const markerBuckets = await this.markerBuckets(requiredScope)
if (markerBuckets.size === 0) return []
const rows = await this.db
@@ -201,8 +210,8 @@ export class AdminStatsHourlyReader {
gte(statsRollupsHourly.bucketStart, this.queryFrom),
lt(statsRollupsHourly.bucketStart, this.queryTo),
sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.version') END = ${ROLLUP_VERSION}`,
requiredScope === 'full'
? sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.scope') END = 'full'`
requiredScope === 'snapshots'
? sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.scope') END IN ('snapshots', 'full')`
: sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.scope') END IN ('counters', 'full')`,
sql`CASE WHEN json_valid(${statsRollupsHourly.metadata}) = 1 THEN json_extract(${statsRollupsHourly.metadata}, '$.quality') END IN ('exact', 'lower_bound')`,
),
@@ -225,15 +234,19 @@ export class AdminStatsHourlyReader {
}
private markerBuckets(requiredScope: AdminStatsRollupScope): Promise<Set<number>> {
const cached = this.markerBucketsPromises.get(requiredScope)
if (cached) return cached
const buckets = this.loadMarkerBuckets(requiredScope)
this.markerBucketsPromises.set(requiredScope, buckets)
return buckets
return this.markerRows(requiredScope).then((rows) => new Set(rows.map((row) => row.bucketStart.getTime())))
}
private async loadMarkerBuckets(requiredScope: AdminStatsRollupScope): Promise<Set<number>> {
if (this.queryFrom >= this.queryTo) return new Set()
private markerRows(requiredScope: AdminStatsRollupScope): Promise<CompatibleMarkerRow[]> {
const cached = this.markerRowsPromises.get(requiredScope)
if (cached) return cached
const rows = this.loadMarkerRows(requiredScope)
this.markerRowsPromises.set(requiredScope, rows)
return rows
}
private async loadMarkerRows(requiredScope: AdminStatsRollupScope): Promise<CompatibleMarkerRow[]> {
if (this.queryFrom >= this.queryTo) return []
const rows = await this.db
.select({ bucketStart: statsRollupsHourly.bucketStart, metadata: statsRollupsHourly.metadata })
.from(statsRollupsHourly)
@@ -246,19 +259,37 @@ export class AdminStatsHourlyReader {
lt(statsRollupsHourly.bucketStart, this.queryTo),
),
)
return new Set(
rows
.filter((row) => {
const metadata = parseAdminStatsRollupMetadata(row.metadata)
return supportsScope(metadata?.scope, requiredScope)
})
.map((row) => row.bucketStart.getTime()),
)
return rows.flatMap((row) => {
const metadata = parseAdminStatsRollupMetadata(row.metadata)
return metadata && supportsScope(metadata.scope, requiredScope)
? [{ bucketStart: row.bucketStart, metadata }]
: []
})
}
}
type CompatibleMarkerRow = { bucketStart: Date; metadata: AdminStatsRollupMetadata }
function qualityForScope(
metadata: AdminStatsRollupMetadata,
requiredScope: AdminStatsRollupScope,
): 'exact' | 'lower_bound' {
if (requiredScope === 'counters') return metadata.counterQuality ?? metadata.quality
if (requiredScope === 'snapshots') return metadata.snapshotQuality ?? metadata.quality
return metadata.quality
}
function dataThroughForScope(row: CompatibleMarkerRow, requiredScope: AdminStatsRollupScope): string {
if (requiredScope === 'snapshots' && row.metadata.snapshotObservedAt) return row.metadata.snapshotObservedAt
return new Date(row.bucketStart.getTime() + HOUR_MS).toISOString()
}
function supportsScope(scope: AdminStatsRollupScope | undefined, requiredScope: AdminStatsRollupScope): boolean {
return scope === 'full' || (requiredScope === 'counters' && scope === 'counters')
return (
scope === 'full' ||
(requiredScope === 'counters' && scope === 'counters') ||
(requiredScope === 'snapshots' && scope === 'snapshots')
)
}
function floorHour(date: Date): Date {
@@ -3,7 +3,11 @@ import { describe, expect, it } from 'vitest'
import { assertMetricDimension, ADMIN_STATS_METRICS as M, metricDefinition } from '../../domain/admin-stats-metrics'
import { adminHeaders, createTestApp } from '../../test/setup.js'
import { AdminStatsHourlyReader } from './admin-stats-hourly'
import { ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE, rebuildAdminStatsHour } from './admin-stats-rollup'
import {
ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE,
captureAdminStatsSnapshot,
rebuildAdminStatsHour,
} from './admin-stats-rollup'
type RollupRow = {
orgId: string
@@ -48,9 +52,18 @@ describe('admin hourly stats rollup', () => {
`)
await db.run(sql`
UPDATE org_quotas
SET used = 500, quota = 1000, traffic_used = 300, traffic_quota = 2000
SET used = 1300, quota = 1000, traffic_used = 300, traffic_quota = 2000
WHERE org_id = ${orgId}
`)
await db.run(sql`
UPDATE org_quota_entitlements
SET bytes = 1000, starts_at = ${atMs}
WHERE org_id = ${orgId} AND resource_type = 'storage' AND source = 'free_plan' AND status = 'active'
`)
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES ('rollup-orphan-quota', 'deleted-org', 999999, 999999, 0, 0, '2026-07')
`)
await db.run(sql`
INSERT INTO storages
(id, bucket, endpoint, region, access_key, secret_key, file_path, custom_host, capacity, used, status, created_at, updated_at)
@@ -60,11 +73,11 @@ describe('admin hourly stats rollup', () => {
`)
await db.run(sql`
INSERT INTO matters
(id, org_id, alias, name, type, size, dirtype, parent, object, storage_id, status, created_at, updated_at)
(id, org_id, alias, name, type, size, dirtype, parent, object, storage_id, status, trashed_at, created_at, updated_at)
VALUES
('rollup-file', ${orgId}, 'rollup-file', 'video.mp4', 'video/mp4', 200, 0, '', 'video.mp4', 'rollup-storage', 'active', ${atSec}, ${atSec}),
('rollup-dir', ${orgId}, 'rollup-dir', 'folder', '', 0, 1, '', '', 'rollup-storage', 'active', ${atSec}, ${atSec}),
('rollup-trashed', ${orgId}, 'rollup-trashed', 'old.bin', 'application/octet-stream', 800, 0, '', 'old.bin', 'rollup-storage', 'trashed', ${atSec}, ${atSec})
('rollup-file', ${orgId}, 'rollup-file', 'video.mp4', 'video/mp4', 200, 0, '', 'video.mp4', 'rollup-storage', 'active', NULL, ${atSec}, ${atSec}),
('rollup-dir', ${orgId}, 'rollup-dir', 'folder', '', 0, 1, '', '', 'rollup-storage', 'active', NULL, ${atSec}, ${atSec}),
('rollup-trashed', ${orgId}, 'rollup-trashed', 'old.bin', 'application/octet-stream', 800, 0, '', 'old.bin', 'rollup-storage', 'trashed', ${atMs}, ${atSec}, ${atSec})
`)
await db.run(sql`
INSERT INTO image_hostings
@@ -114,7 +127,23 @@ describe('admin hourly stats rollup', () => {
('rollup-team-join', ${orgId}, ${userId}, 'user', 'team_member_join', 'organization', 'rollup-team', 'team', NULL, ${atSec}),
('rollup-team-remove', ${orgId}, ${userId}, 'user', 'team_member_remove', 'organization', 'rollup-team', 'team', NULL, ${atSec}),
('rollup-license', ${orgId}, NULL, 'system', 'license_refresh', 'license', NULL, 'license',
'{"status":"failed"}', ${atSec})
'{"status":"failed"}', ${atSec}),
('rollup-signup-credential', ${orgId}, NULL, 'system', 'stats_user_signup', 'user', ${userId}, ${userId},
'{"provider":"credential","statsQuality":"exact"}', ${atSec}),
('rollup-signup-direct', '', NULL, 'system', 'stats_user_signup', 'user', 'rollup-direct-user', 'rollup-direct-user',
'{"provider":"direct","statsQuality":"exact"}', ${atSec}),
('rollup-share-created-usable', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-usable', 'rollup-share-usable',
'{"kind":"landing","statsQuality":"exact"}', ${atSec}),
('rollup-share-created-revoked', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-revoked', 'rollup-share-revoked',
'{"kind":"direct","statsQuality":"exact"}', ${atSec}),
('rollup-share-created-expired', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-expired', 'rollup-share-expired',
'{"kind":"landing","statsQuality":"exact"}', ${atSec}),
('rollup-share-created-limited', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'rollup-share-limited', 'rollup-share-limited',
'{"kind":"direct","statsQuality":"exact"}', ${atSec}),
('rollup-task-finished-fact', ${orgId}, NULL, 'system', 'stats_remote_download_finished', 'remote_download', 'rollup-task-finished', 'rollup-task-finished',
'{"category":"uncategorized","downloaderId":"rollup-downloader","outcome":"completed","bytes":60,"statsQuality":"exact"}', ${atSec}),
('rollup-job-finished-fact', ${orgId}, NULL, 'system', 'stats_background_job_finished', 'background_job', 'rollup-job-finished', 'rollup-job-finished',
'{"jobType":"archive","outcome":"failed","statsQuality":"exact"}', ${atSec})
`)
await db.run(sql`
INSERT INTO cloud_traffic_reports
@@ -160,7 +189,8 @@ describe('admin hourly stats rollup', () => {
VALUES ('rollup-webhook', 'cloud', 'webhook-event', 'order.quota_changed', 'hash', '{}', 'processed', ${atMs}, ${atMs})
`)
const first = await rebuildAdminStatsHour(db, bucketStart, generatedAt, true)
await captureAdminStatsSnapshot(db, bucketStart, generatedAt)
const first = await rebuildAdminStatsHour(db, bucketStart, generatedAt)
const rows = await db.all<RollupRow>(sql`
SELECT org_id AS orgId, metric_key AS metric, dimension_key AS dimensionKey,
dimension_value AS dimensionValue, count, bytes, unique_count AS uniqueCount, metadata
@@ -179,9 +209,10 @@ describe('admin hourly stats rollup', () => {
expect(first).toMatchObject({
bucketStart,
bucketEnd: new Date('2026-07-10T13:00:00.000Z'),
rows: rows.length,
rows: expect.any(Number),
lowerBoundRows: expect.any(Number),
})
expect(rows.length).toBeGreaterThan(first.rows)
expect(first.lowerBoundRows).toBeGreaterThan(0)
expect(row(M.transferUpload)).toMatchObject({ count: 4, bytes: 100 })
expect(row(M.transferDownloadIssued)).toMatchObject({ count: 4, bytes: 90 })
@@ -194,12 +225,19 @@ describe('admin hourly stats rollup', () => {
expect(row(M.remoteDownloadTaskFinished)).toMatchObject({ count: 1, bytes: 60 })
expect(row(M.backgroundJobFinished)).toMatchObject({ count: 1 })
expect(row(M.storageInventory)).toMatchObject({ count: 2, bytes: 500 })
expect(row(M.storageUsed)).toMatchObject({ bytes: 500 })
expect(row(M.storageUsed)).toMatchObject({ bytes: 1300 })
expect(row(M.storageQuota)).toMatchObject({ bytes: 1000 })
expect(row(M.storageQuota, 'status', 'over', '')).toMatchObject({ count: 1 })
expect(row(M.storageQuota, 'status', 'invalid', '')).toMatchObject({ count: 1 })
expect(row(M.storageTrashSnapshot)).toMatchObject({ count: 1, bytes: 800 })
expect(row(M.statsDataQualitySnapshot, 'kind', 'storage_usage_drift', '')).toMatchObject({ count: 0, bytes: 0 })
expect(row(M.shareInventory, 'lifecycle', 'usable')).toMatchObject({ count: 1 })
expect(row(M.shareInventory, 'lifecycle', 'revoked')).toMatchObject({ count: 1 })
expect(row(M.shareInventory, 'lifecycle', 'expired')).toMatchObject({ count: 1 })
expect(row(M.shareInventory, 'lifecycle', 'download_limit_reached')).toMatchObject({ count: 1 })
expect(row(M.statsDataQualitySnapshot, '', '', '')).toMatchObject({ count: 1 })
expect(row(M.statsDataQualitySnapshot, 'kind', 'share_downloads', '')).toMatchObject({ count: 1 })
expect(row(M.statsDataQualitySnapshot, 'kind', 'share_views', '')).toMatchObject({ count: 0 })
expect(row(M.backgroundJobSnapshot)).toMatchObject({ count: 1 })
expect(row(M.remoteDownloadTaskSnapshot)).toMatchObject({ count: 1 })
expect(row(M.downloaderSnapshot, '', '', '')).toMatchObject({ count: 2 })
@@ -208,21 +246,40 @@ describe('admin hourly stats rollup', () => {
expect(row(M.trafficReportSnapshot, '', '', '')).toMatchObject({ count: 2, bytes: 300 })
expect(row(M.webhookSnapshot, 'status', 'processed', '')).toMatchObject({ count: 1 })
expect(row(M.statsRollupRun, '', '', '')).toMatchObject({ count: 1 })
expect(JSON.parse(row(M.transferUpload)?.metadata ?? '{}')).toMatchObject({ version: 2, quality: 'lower_bound' })
expect(JSON.parse(row(M.transferUpload)?.metadata ?? '{}')).toMatchObject({ version: 3, quality: 'lower_bound' })
expect(JSON.parse(row(M.statsRollupRun, '', '', '')?.metadata ?? '{}')).toMatchObject({
version: 3,
scope: 'full',
quality: 'lower_bound',
counterQuality: 'lower_bound',
snapshotQuality: 'exact',
snapshotObservedAt: generatedAt.toISOString(),
})
const second = await rebuildAdminStatsHour(db, bucketStart, generatedAt, true)
const reader = new AdminStatsHourlyReader(
db,
{ from: bucketStart, to: new Date(bucketStart.getTime() + 3_600_000 - 1), timeZone: 'UTC' },
new Date(bucketStart.getTime() + 2 * 3_600_000),
)
await expect(reader.coverage('counters')).resolves.toMatchObject({ quality: 'lower_bound' })
await expect(reader.coverage('snapshots')).resolves.toMatchObject({
quality: 'exact',
dataThrough: generatedAt.toISOString(),
})
const second = await rebuildAdminStatsHour(db, bucketStart, generatedAt)
const [{ count: storedRows }] = await db.all<{ count: number }>(sql`
SELECT COUNT(*) AS count FROM stats_rollups_hourly WHERE bucket_start = ${bucketStart.getTime()}
`)
expect(second.rows).toBe(first.rows)
expect(storedRows).toBe(first.rows)
expect(storedRows).toBeGreaterThan(first.rows)
})
it('rejects buckets that are not aligned to a UTC hour', async () => {
const { db } = await createTestApp()
await expect(
rebuildAdminStatsHour(db, new Date('2026-07-10T12:00:00.001Z'), new Date('2026-07-10T12:30:00Z'), false),
rebuildAdminStatsHour(db, new Date('2026-07-10T12:00:00.001Z'), new Date('2026-07-10T12:30:00Z')),
).rejects.toThrow('stats_bucket_must_align_to_utc_hour')
})
@@ -231,8 +288,8 @@ describe('admin hourly stats rollup', () => {
await db.run(sql`DROP TABLE org_quotas`)
await expect(
rebuildAdminStatsHour(db, new Date('2026-07-10T12:00:00Z'), new Date('2026-07-10T12:30:00Z'), true),
).rejects.toThrow('stats_rollup_query_failed:quota')
captureAdminStatsSnapshot(db, new Date('2026-07-10T12:00:00Z'), new Date('2026-07-10T12:30:00Z')),
).rejects.toThrow(/stats_rollup_query_failed:(quota|data-quality)/)
})
it('reads only current-version result rows with a compatible completion scope', async () => {
@@ -244,15 +301,15 @@ describe('admin hourly stats rollup', () => {
count, bytes, unique_count, metadata, updated_at)
VALUES
('quality-marker', ${bucketStart}, '', 'stats.rollup_run', '', '', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart}),
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart}),
('quality-null', ${bucketStart}, 'org-null', 'transfer.upload', '', '', 1, 1, 0, NULL, ${bucketStart}),
('quality-array', ${bucketStart}, 'org-array', 'transfer.upload', '', '', 1, 2, 0, '[]', ${bucketStart}),
('quality-v1', ${bucketStart}, 'org-v1', 'transfer.upload', '', '', 1, 3, 0,
'{"version":1,"scope":"full","quality":"exact"}', ${bucketStart}),
('quality-exact', ${bucketStart}, 'org-exact', 'transfer.upload', '', '', 1, 4, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart}),
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart}),
('quality-lower', ${bucketStart}, 'org-lower', 'transfer.upload', '', '', 1, 5, 0,
'{"version":2,"scope":"full","quality":"lower_bound"}', ${bucketStart})
'{"version":3,"scope":"full","quality":"lower_bound"}', ${bucketStart})
`)
const reader = new AdminStatsHourlyReader(
db,
@@ -278,7 +335,7 @@ describe('admin hourly stats rollup', () => {
it('keeps counter-only repairs separate from full snapshot results', async () => {
const { db } = await createTestApp()
const bucketStart = Date.parse('2026-07-10T09:00:00.000Z')
const metadata = '{"version":2,"scope":"counters","quality":"exact"}'
const metadata = '{"version":3,"scope":"counters","quality":"exact"}'
await db.run(sql`
INSERT INTO stats_rollups_hourly
(id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
@@ -287,7 +344,7 @@ describe('admin hourly stats rollup', () => {
('counter-marker', ${bucketStart}, '', 'stats.rollup_run', '', '', 1, 0, 0, ${metadata}, ${bucketStart}),
('counter-result', ${bucketStart}, '', 'transfer.upload', '', '', 1, 42, 0, ${metadata}, ${bucketStart}),
('orphan-gauge', ${bucketStart}, '', 'storage.used', '', '', 0, 99, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart})
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart})
`)
const reader = new AdminStatsHourlyReader(
db,
@@ -314,9 +371,9 @@ describe('admin hourly stats rollup', () => {
count, bytes, unique_count, metadata, updated_at)
VALUES
('current-hour-marker', ${bucketStart}, '', 'stats.rollup_run', '', '', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart}),
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart}),
('current-hour-rollup', ${bucketStart}, '', 'transfer.upload', '', '', 1, 42, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${bucketStart})
'{"version":3,"scope":"full","quality":"exact"}', ${bucketStart})
`)
const reader = new AdminStatsHourlyReader(
db,
+350 -89
View File
@@ -1,4 +1,5 @@
import { and, eq, gte, inArray, lt, sql } from 'drizzle-orm'
import { and, eq, inArray, isNull, sql } from 'drizzle-orm'
import { organization } from '../../db/auth-schema'
import {
backgroundJobs,
cloudTrafficReports,
@@ -17,9 +18,11 @@ import {
type AdminStatsMetric,
assertMetricDimension,
ADMIN_STATS_METRICS as M,
parseAdminStatsRollupMetadata,
ROLLUP_VERSION,
} from '../../domain/admin-stats-metrics'
import type { Database } from '../../platform/interface'
import { getEffectiveQuotasByOrg } from './quota'
const HOUR_MS = 3_600_000
const D1_MAX_BOUND_PARAMS = 100
@@ -41,6 +44,21 @@ const COUNTER_METRICS: AdminStatsMetric[] = [
M.transferUpload,
M.userSignup,
]
const GAUGE_METRICS: AdminStatsMetric[] = [
M.backgroundJobSnapshot,
M.downloaderSnapshot,
M.remoteDownloadTaskSnapshot,
M.shareInventory,
M.statsDataQualitySnapshot,
M.storageInventory,
M.storageQuota,
M.storageTrashSnapshot,
M.storageUsed,
M.trafficReportSnapshot,
M.userActiveSnapshot,
M.userInventory,
M.webhookSnapshot,
]
type RollupValue = {
metric: AdminStatsMetric
@@ -66,21 +84,21 @@ export async function rebuildAdminStatsHour(
db: Database,
bucketStartInput: Date,
generatedAt: Date,
includeSnapshots: boolean,
): Promise<AdminStatsHourlyRollupResult> {
const bucketStart = startOfHour(bucketStartInput)
if (bucketStart.getTime() !== bucketStartInput.getTime()) throw new Error('stats_bucket_must_align_to_utc_hour')
const bucketEnd = new Date(bucketStart.getTime() + HOUR_MS)
const rollups = new RollupAccumulator()
const capturedSnapshot = await compatibleSnapshotMarker(db, bucketStart)
await addEventMetrics(db, rollups, bucketStart, bucketEnd)
await addUserMetrics(db, rollups, bucketStart, bucketEnd)
await addOperationalMetrics(db, rollups, bucketStart, bucketEnd)
if (includeSnapshots) await addSnapshotMetrics(db, rollups, generatedAt)
const values = rollups.values()
const lowerBoundRows = values.filter((row) => row.lowerBound).length
rollups.add(M.statsRollupRun, '', 1, 0, { outcome: 'success' })
rollups.add(M.statsRollupRun, '', 1, 0, { outcome: 'success' }, lowerBoundRows > 0)
const completionScope = capturedSnapshot ? 'full' : 'counters'
const counterQuality = lowerBoundRows > 0 ? 'lower_bound' : 'exact'
const updatedAt = generatedAt
const rows = rollups.values().map((row) => ({
@@ -93,19 +111,39 @@ export async function rebuildAdminStatsHour(
count: row.count,
bytes: row.bytes,
uniqueCount: row.uniqueCount,
metadata: JSON.stringify({
version: ROLLUP_VERSION,
scope: includeSnapshots ? 'full' : 'counters',
quality: row.lowerBound ? 'lower_bound' : 'exact',
generatedAt: generatedAt.toISOString(),
}),
metadata: JSON.stringify(
row.metric === M.statsRollupRun
? {
version: ROLLUP_VERSION,
scope: completionScope,
quality:
counterQuality === 'lower_bound' || capturedSnapshot?.quality === 'lower_bound' ? 'lower_bound' : 'exact',
counterQuality,
...(capturedSnapshot
? {
snapshotQuality: capturedSnapshot.quality,
snapshotObservedAt: capturedSnapshot.observedAt,
}
: {}),
generatedAt: generatedAt.toISOString(),
}
: {
version: ROLLUP_VERSION,
scope: 'counters',
quality: row.lowerBound ? 'lower_bound' : 'exact',
generatedAt: generatedAt.toISOString(),
},
),
updatedAt,
}))
const deleteWhere = includeSnapshots
? eq(statsRollupsHourly.bucketStart, bucketStart)
: and(eq(statsRollupsHourly.bucketStart, bucketStart), inArray(statsRollupsHourly.metricKey, COUNTER_METRICS))
const writes: AtomicQuery[] = [db.delete(statsRollupsHourly).where(deleteWhere)]
const writes: AtomicQuery[] = [
db
.delete(statsRollupsHourly)
.where(
and(eq(statsRollupsHourly.bucketStart, bucketStart), inArray(statsRollupsHourly.metricKey, COUNTER_METRICS)),
),
]
for (let index = 0; index < rows.length; index += ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE) {
writes.push(db.insert(statsRollupsHourly).values(rows.slice(index, index + ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE)))
}
@@ -113,6 +151,79 @@ export async function rebuildAdminStatsHour(
return { bucketStart, bucketEnd, rows: rows.length, lowerBoundRows }
}
export async function captureAdminStatsSnapshot(
db: Database,
bucketStartInput: Date,
observedAt: Date,
): Promise<AdminStatsHourlyRollupResult> {
const bucketStart = startOfHour(bucketStartInput)
if (bucketStart.getTime() !== bucketStartInput.getTime()) throw new Error('stats_bucket_must_align_to_utc_hour')
if (startOfHour(observedAt).getTime() !== bucketStart.getTime()) {
throw new Error('stats_snapshot_must_be_captured_in_bucket')
}
const bucketEnd = new Date(bucketStart.getTime() + HOUR_MS)
const rollups = new RollupAccumulator()
await addSnapshotMetrics(db, rollups, observedAt)
rollups.add(M.statsRollupRun, '', 1, 0, { outcome: 'success' })
const rows = rollups.values().map((row) => ({
id: rollupId(bucketStart, row),
bucketStart,
orgId: row.orgId,
metricKey: row.metric,
dimensionKey: row.dimensionKey,
dimensionValue: row.dimensionValue,
count: row.count,
bytes: row.bytes,
uniqueCount: row.uniqueCount,
metadata: JSON.stringify({
version: ROLLUP_VERSION,
scope: 'snapshots',
quality: 'exact',
...(row.metric === M.statsRollupRun
? { snapshotQuality: 'exact', snapshotObservedAt: observedAt.toISOString() }
: { observedAt: observedAt.toISOString() }),
}),
updatedAt: observedAt,
}))
const writes: AtomicQuery[] = [
db
.delete(statsRollupsHourly)
.where(
and(
eq(statsRollupsHourly.bucketStart, bucketStart),
inArray(statsRollupsHourly.metricKey, [...GAUGE_METRICS, M.statsRollupRun]),
),
),
]
for (let index = 0; index < rows.length; index += ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE) {
writes.push(db.insert(statsRollupsHourly).values(rows.slice(index, index + ADMIN_STATS_ROLLUP_WRITE_BATCH_SIZE)))
}
await executeWriteTransaction(db, writes)
return { bucketStart, bucketEnd, rows: rows.length, lowerBoundRows: 0 }
}
async function compatibleSnapshotMarker(
db: Database,
bucketStart: Date,
): Promise<{ quality: 'exact' | 'lower_bound'; observedAt: string } | null> {
const rows = await db
.select({ metadata: statsRollupsHourly.metadata })
.from(statsRollupsHourly)
.where(
and(
eq(statsRollupsHourly.bucketStart, bucketStart),
eq(statsRollupsHourly.orgId, ''),
eq(statsRollupsHourly.metricKey, M.statsRollupRun),
eq(statsRollupsHourly.dimensionKey, ''),
eq(statsRollupsHourly.dimensionValue, ''),
),
)
.limit(1)
const metadata = parseAdminStatsRollupMetadata(rows[0]?.metadata ?? null)
if ((metadata?.scope !== 'snapshots' && metadata?.scope !== 'full') || !metadata.snapshotObservedAt) return null
return { quality: metadata.snapshotQuality ?? metadata.quality, observedAt: metadata.snapshotObservedAt }
}
async function addEventMetrics(db: Database, rollups: RollupAccumulator, from: Date, to: Date): Promise<void> {
const rows = await db.all<EventMetricGroup>(sql`
WITH facts AS (
@@ -241,70 +352,90 @@ type EventMetricGroup = {
}
async function addUserMetrics(db: Database, rollups: RollupAccumulator, from: Date, to: Date): Promise<void> {
const [signupRows, shareRows] = await Promise.all([
db.all<{ provider: string; count: number }>(sql`
SELECT provider, COUNT(*) AS count
FROM (
SELECT COALESCE((
SELECT a.provider_id FROM account a
WHERE a.user_id = u.id
ORDER BY a.created_at, a.id
LIMIT 1
), 'direct') AS provider
FROM "user" u
WHERE u.created_at >= ${from.getTime()} AND u.created_at < ${to.getTime()}
) signups
GROUP BY provider
`),
db
.select({ orgId: shares.orgId, kind: shares.kind, count: sql<number>`COUNT(*)` })
.from(shares)
.where(and(gte(shares.createdAt, from), lt(shares.createdAt, to)))
.groupBy(shares.orgId, shares.kind),
])
const rows = await db.all<{
action: string
orgId: string
provider: string | null
kind: string | null
count: number
lowerBound: number
}>(sql`
SELECT
action,
org_id AS orgId,
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.provider') END AS provider,
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.kind') END AS kind,
COUNT(*) AS count,
SUM(CASE WHEN json_valid(metadata) = 1 AND json_extract(metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) AS lowerBound
FROM activity_events
WHERE created_at >= ${Math.floor(from.getTime() / 1000)}
AND created_at < ${Math.floor(to.getTime() / 1000)}
AND action IN ('stats_user_signup', 'stats_share_created')
GROUP BY action, org_id, provider, kind
`)
for (const row of signupRows) rollups.add(M.userSignup, '', Number(row.count), 0, { provider: row.provider })
for (const row of shareRows) rollups.add(M.shareCreated, row.orgId, Number(row.count), 0, { kind: row.kind })
for (const row of rows) {
if (row.action === 'stats_user_signup') {
rollups.add(M.userSignup, '', Number(row.count), 0, { provider: row.provider ?? 'unknown' }, row.lowerBound > 0)
} else {
rollups.add(M.shareCreated, row.orgId, Number(row.count), 0, { kind: row.kind ?? 'unknown' }, row.lowerBound > 0)
}
}
}
async function addOperationalMetrics(db: Database, rollups: RollupAccumulator, from: Date, to: Date): Promise<void> {
const [taskFinishedRows, jobRows] = await Promise.all([
db
.select({
orgId: downloadTasks.orgId,
category: downloadTasks.category,
downloaderId: downloadTasks.assignedDownloaderId,
status: downloadTasks.status,
count: sql<number>`COUNT(*)`,
bytes: sql<number>`COALESCE(SUM(${downloadTasks.billingChargedBytes}), 0)`,
})
.from(downloadTasks)
.where(and(gte(downloadTasks.finishedAt, from), lt(downloadTasks.finishedAt, to)))
.groupBy(downloadTasks.orgId, downloadTasks.category, downloadTasks.assignedDownloaderId, downloadTasks.status),
db
.select({
orgId: backgroundJobs.orgId,
type: backgroundJobs.type,
status: backgroundJobs.status,
count: sql<number>`COUNT(*)`,
})
.from(backgroundJobs)
.where(and(gte(backgroundJobs.finishedAt, from), lt(backgroundJobs.finishedAt, to)))
.groupBy(backgroundJobs.orgId, backgroundJobs.type, backgroundJobs.status),
])
const rows = await db.all<{
action: string
orgId: string
category: string | null
downloaderId: string | null
jobType: string | null
outcome: string | null
count: number
bytes: number
lowerBound: number
}>(sql`
SELECT
action,
org_id AS orgId,
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.category') END AS category,
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.downloaderId') END AS downloaderId,
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.jobType') END AS jobType,
CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.outcome') END AS outcome,
COUNT(*) AS count,
SUM(CASE WHEN json_valid(metadata) = 1 THEN COALESCE(json_extract(metadata, '$.bytes'), 0) ELSE 0 END) AS bytes,
SUM(CASE WHEN json_valid(metadata) = 1 AND json_extract(metadata, '$.statsQuality') = 'lower_bound' THEN 1 ELSE 0 END) AS lowerBound
FROM activity_events
WHERE created_at >= ${Math.floor(from.getTime() / 1000)}
AND created_at < ${Math.floor(to.getTime() / 1000)}
AND action IN ('stats_background_job_finished', 'stats_remote_download_finished')
GROUP BY action, org_id, category, downloaderId, jobType, outcome
`)
for (const row of taskFinishedRows) {
rollups.add(M.remoteDownloadTaskFinished, row.orgId, Number(row.count), Number(row.bytes), {
category: row.category ?? 'uncategorized',
downloader_id: row.downloaderId,
outcome: row.status,
})
}
for (const row of jobRows) {
rollups.add(M.backgroundJobFinished, row.orgId, Number(row.count), 0, {
job_type: row.type,
outcome: row.status,
})
for (const row of rows) {
if (row.action === 'stats_remote_download_finished') {
rollups.add(
M.remoteDownloadTaskFinished,
row.orgId,
Number(row.count),
Number(row.bytes),
{
category: row.category ?? 'uncategorized',
downloader_id: row.downloaderId,
outcome: row.outcome ?? 'unknown',
},
row.lowerBound > 0,
)
} else {
rollups.add(
M.backgroundJobFinished,
row.orgId,
Number(row.count),
0,
{ job_type: row.jobType ?? 'unknown', outcome: row.outcome ?? 'unknown' },
row.lowerBound > 0,
)
}
}
}
@@ -312,12 +443,14 @@ async function addSnapshotMetrics(db: Database, rollups: RollupAccumulator, now:
const [
userRows,
activeUserRows,
quotaRows,
quotaBaseRows,
inventoryRows,
inventoryByType,
inventoryBySize,
inventoryByAge,
trashRows,
shareRows,
dataQualityRows,
jobRows,
taskRows,
downloaderRows,
@@ -330,17 +463,20 @@ async function addSnapshotMetrics(db: Database, rollups: RollupAccumulator, now:
'quota',
db
.select({
orgId: orgQuotas.orgId,
used: orgQuotas.used,
quota: orgQuotas.quota,
orgId: organization.id,
quotaId: orgQuotas.id,
used: sql<number>`COALESCE(${orgQuotas.used}, 0)`,
})
.from(orgQuotas),
.from(organization)
.leftJoin(orgQuotas, eq(orgQuotas.orgId, organization.id)),
),
queryStage('inventory-base', inventoryGroups(db, now, 'base')),
queryStage('inventory-type', inventoryGroups(db, now, 'file_type_group')),
queryStage('inventory-size', inventoryGroups(db, now, 'size_bucket')),
queryStage('inventory-age', inventoryGroups(db, now, 'age_bucket')),
queryStage('trash-inventory', trashInventoryGroups(db)),
queryStage('shares', shareSnapshotGroups(db, now)),
queryStage('data-quality', dataQualitySnapshot(db)),
queryStage(
'jobs',
db
@@ -394,30 +530,72 @@ async function addSnapshotMetrics(db: Database, rollups: RollupAccumulator, now:
),
])
const effectiveQuotas = await queryStage(
'effective-quota',
getEffectiveQuotasByOrg(
db,
quotaBaseRows.map((row) => row.orgId),
now,
),
)
rollups.setGauge(M.userInventory, '', userRows.total, 0)
rollups.setGauge(M.userInventory, '', userRows.normal, 0, 'status', 'normal')
rollups.setGauge(M.userInventory, '', userRows.verified, 0, 'status', 'verified')
rollups.setGauge(M.userInventory, '', userRows.unverified, 0, 'status', 'unverified')
rollups.setGauge(M.userInventory, '', userRows.banned, 0, 'status', 'banned')
rollups.setGauge(M.userInventory, '', userRows.silent, 0, 'status', 'silent')
rollups.setGauge(M.userActiveSnapshot, '', activeUserRows.find((row) => row.window === 'mau')?.count ?? 0, 0)
for (const row of activeUserRows) rollups.setGauge(M.userActiveSnapshot, '', row.count, 0, 'window', row.window)
for (const row of quotaRows) {
rollups.setGauge(M.storageUsed, '', 0, 0)
rollups.setGauge(M.storageQuota, '', 0, 0)
rollups.setGauge(M.storageInventory, '', 0, 0)
rollups.setGauge(M.storageTrashSnapshot, '', 0, 0)
rollups.setGauge(M.shareInventory, '', 0, 0)
rollups.setGauge(M.statsDataQualitySnapshot, '', dataQualityRows.unlocatedShareEvents, 0)
rollups.setGauge(M.statsDataQualitySnapshot, '', dataQualityRows.unlocatedShareViews, 0, 'kind', 'share_views')
rollups.setGauge(
M.statsDataQualitySnapshot,
'',
dataQualityRows.unlocatedShareDownloads,
0,
'kind',
'share_downloads',
)
rollups.setGauge(
M.statsDataQualitySnapshot,
'',
dataQualityRows.storageUsageDriftSpaces,
dataQualityRows.storageUsageDriftBytes,
'kind',
'storage_usage_drift',
)
rollups.setGauge(M.backgroundJobSnapshot, '', 0, 0)
rollups.setGauge(M.remoteDownloadTaskSnapshot, '', 0, 0)
rollups.setGauge(M.downloaderSnapshot, '', 0, 0)
rollups.setGauge(M.trafficReportSnapshot, '', 0, 0)
rollups.setGauge(M.webhookSnapshot, '', 0, 0)
for (const row of quotaBaseRows) {
const effective = effectiveQuotas.get(row.orgId)
if (!effective) throw new Error(`stats_effective_quota_missing:${row.orgId}`)
const quota = row.quotaId ? effective.quota : 0
rollups.setGauge(M.storageUsed, row.orgId, 0, row.used)
rollups.setGauge(M.storageQuota, row.orgId, 0, row.quota)
rollups.setGauge(M.storageQuota, row.orgId, 0, quota)
rollups.incrementGauge(M.storageUsed, '', 0, row.used)
rollups.incrementGauge(M.storageQuota, '', 1, row.quota)
const status =
row.quota > 0 && row.used >= row.quota
? 'over'
: row.quota > 0 && row.used >= row.quota * 0.8
? 'near'
: 'healthy'
rollups.incrementGauge(M.storageQuota, '', 1, quota)
const status = quota <= 0 ? 'invalid' : row.used >= quota ? 'over' : row.used >= quota * 0.8 ? 'near' : 'healthy'
rollups.incrementGauge(M.storageQuota, '', 1, 0, 'status', status)
}
for (const row of [...inventoryRows, ...inventoryByType, ...inventoryBySize, ...inventoryByAge]) {
rollups.incrementGauge(M.storageInventory, row.orgId, row.files, row.bytes, row.dimensionKey, row.dimensionValue)
rollups.incrementGauge(M.storageInventory, '', row.files, row.bytes, row.dimensionKey, row.dimensionValue)
}
for (const row of trashRows) {
rollups.incrementGauge(M.storageTrashSnapshot, row.orgId, row.files, row.bytes)
rollups.incrementGauge(M.storageTrashSnapshot, '', row.files, row.bytes)
rollups.incrementGauge(M.storageTrashSnapshot, row.orgId, row.files, row.bytes, 'storage_id', row.storageId)
rollups.incrementGauge(M.storageTrashSnapshot, '', row.files, row.bytes, 'storage_id', row.storageId)
}
for (const row of shareRows) {
rollups.incrementGauge(M.shareInventory, row.orgId, Number(row.count), 0, 'lifecycle', row.lifecycle)
rollups.incrementGauge(M.shareInventory, '', Number(row.count), 0, 'lifecycle', row.lifecycle)
@@ -461,6 +639,89 @@ async function shareSnapshotGroups(
.groupBy(shares.orgId, lifecycle)
}
async function dataQualitySnapshot(db: Database): Promise<{
unlocatedShareEvents: number
unlocatedShareViews: number
unlocatedShareDownloads: number
storageUsageDriftSpaces: number
storageUsageDriftBytes: number
}> {
const rows = await db.all<{
unlocatedShareViews: number
unlocatedShareDownloads: number
storageUsageDriftSpaces: number
storageUsageDriftBytes: number
}>(sql`
WITH event_counts AS (
SELECT
COALESCE(CASE WHEN json_valid(metadata) = 1 THEN json_extract(metadata, '$.shareId') END, target_id) AS share_id,
SUM(CASE WHEN action = 'share_view' THEN 1 ELSE 0 END) AS views,
SUM(CASE WHEN action = 'share_download' THEN 1 ELSE 0 END) AS downloads
FROM activity_events
WHERE action IN ('share_view', 'share_download')
GROUP BY share_id
),
billable_storage AS (
SELECT org_id, SUM(bytes) AS bytes
FROM (
SELECT org_id, COALESCE(size, 0) AS bytes
FROM matters
WHERE status IN ('active', 'trashed') AND dirtype = 0
UNION ALL
SELECT org_id, COALESCE(size, 0) AS bytes
FROM image_hostings
WHERE status = 'active'
) inventory
GROUP BY org_id
),
storage_drift AS (
SELECT ABS(q.used - COALESCE(billable_storage.bytes, 0)) AS bytes
FROM org_quotas q
JOIN organization o ON o.id = q.org_id
LEFT JOIN billable_storage ON billable_storage.org_id = q.org_id
WHERE q.used <> COALESCE(billable_storage.bytes, 0)
)
SELECT
COALESCE(SUM(MAX(${shares.views} - COALESCE(event_counts.views, 0), 0)), 0) AS unlocatedShareViews,
COALESCE(SUM(MAX(${shares.downloads} - COALESCE(event_counts.downloads, 0), 0)), 0) AS unlocatedShareDownloads,
(SELECT COUNT(*) FROM storage_drift) AS storageUsageDriftSpaces,
(SELECT COALESCE(SUM(bytes), 0) FROM storage_drift) AS storageUsageDriftBytes
FROM ${shares}
LEFT JOIN event_counts ON event_counts.share_id = ${shares.id}
`)
const unlocatedShareViews = Number(rows[0]?.unlocatedShareViews ?? 0)
const unlocatedShareDownloads = Number(rows[0]?.unlocatedShareDownloads ?? 0)
return {
unlocatedShareEvents: unlocatedShareViews + unlocatedShareDownloads,
unlocatedShareViews,
unlocatedShareDownloads,
storageUsageDriftSpaces: Number(rows[0]?.storageUsageDriftSpaces ?? 0),
storageUsageDriftBytes: Number(rows[0]?.storageUsageDriftBytes ?? 0),
}
}
async function trashInventoryGroups(
db: Database,
): Promise<Array<{ orgId: string; storageId: string; files: number; bytes: number }>> {
const rows = await db
.select({
orgId: matters.orgId,
storageId: matters.storageId,
files: sql<number>`COUNT(*)`,
bytes: sql<number>`COALESCE(SUM(${matters.size}), 0)`,
})
.from(matters)
.where(
and(
inArray(matters.status, ['active', 'trashed']),
sql`${matters.trashedAt} IS NOT NULL`,
eq(matters.dirtype, 0),
),
)
.groupBy(matters.orgId, matters.storageId)
return rows.map((row) => ({ ...row, files: Number(row.files), bytes: Number(row.bytes) }))
}
type UserInventorySnapshot = {
total: number
normal: number
@@ -550,7 +811,7 @@ async function inventoryGroups(
dimensionValue: dimensionSql,
})
.from(matters)
.where(and(eq(matters.status, 'active'), eq(matters.dirtype, 0)))
.where(and(eq(matters.status, 'active'), isNull(matters.trashedAt), eq(matters.dirtype, 0)))
.groupBy(matters.orgId, dimensionSql)
const imageDimensionSql =
dimension === 'file_type_group'
+290 -124
View File
@@ -6,7 +6,9 @@ import type {
AdminDashboardSharingStats,
AdminDashboardStorageStats,
AdminDashboardTrafficStats,
AdminSharingDataQuality,
AdminStatsDelta,
AdminStorageDataQuality,
AdminTopShare,
AdminTransferDataQuality,
} from '@shared/types'
@@ -23,7 +25,7 @@ import { addCalendarDays, statsDayKey as dayKey, utcDateStart } from '../../doma
import type { Database } from '../../platform/interface'
import type { AdminStatsDateRange, AdminStatsRepo } from '../../usecases/ports'
import { AdminStatsHourlyReader } from './admin-stats-hourly'
import { rebuildAdminStatsHour } from './admin-stats-rollup'
import { captureAdminStatsSnapshot, rebuildAdminStatsHour } from './admin-stats-rollup'
const DOWNLOAD_ACTIVITY_ACTIONS = ['share_download', 'object_download', 'image_hosting_download', 'webdav_download']
const DOWNLOAD_FAILURE_ACTION = 'download_failed'
@@ -40,7 +42,8 @@ export function createAdminStatsRepo(db: Database): AdminStatsRepo {
}
async function refreshHourlyRollups(db: Database, now: Date) {
const latestClosedHour = new Date(startOfHour(now).getTime() - 3_600_000)
const currentHour = startOfHour(now)
const latestClosedHour = new Date(currentHour.getTime() - 3_600_000)
const repairFrom = new Date(latestClosedHour.getTime() - 47 * 3_600_000)
const markers = await db
.select({ bucketStart: statsRollupsHourly.bucketStart, metadata: statsRollupsHourly.metadata })
@@ -56,7 +59,10 @@ async function refreshHourlyRollups(db: Database, now: Date) {
)
const completed = new Set(
markers
.filter((row) => parseAdminStatsRollupMetadata(row.metadata) !== null)
.filter((row) => {
const scope = parseAdminStatsRollupMetadata(row.metadata)?.scope
return scope === 'counters' || scope === 'full'
})
.map((row) => row.bucketStart.getTime()),
)
const repairTargets: Date[] = []
@@ -64,10 +70,11 @@ async function refreshHourlyRollups(db: Database, now: Date) {
if (!completed.has(at)) repairTargets.push(new Date(at))
if (repairTargets.length === 3) break
}
const latest = await rebuildAdminStatsHour(db, latestClosedHour, now, true)
const latest = await rebuildAdminStatsHour(db, latestClosedHour, now)
const repaired = []
for (const bucketStart of repairTargets) repaired.push(await rebuildAdminStatsHour(db, bucketStart, now, false))
return [latest, ...repaired]
for (const bucketStart of repairTargets) repaired.push(await rebuildAdminStatsHour(db, bucketStart, now))
const snapshot = await captureAdminStatsSnapshot(db, currentHour, now)
return [latest, ...repaired, snapshot]
}
async function getDashboardOverviewStats(
@@ -75,8 +82,9 @@ async function getDashboardOverviewStats(
now: Date,
range: AdminStatsDateRange,
): Promise<AdminDashboardOverviewStats> {
const previous = previousRange(range)
const reader = new AdminStatsHourlyReader(db, range, now)
const effective = effectiveRange(range, now)
const previous = previousRange(effective)
const reader = new AdminStatsHourlyReader(db, effective, now)
const previousReader = new AdminStatsHourlyReader(db, previous, now)
const [
users,
@@ -89,9 +97,13 @@ async function getDashboardOverviewStats(
previousTraffic,
sharing,
previousSharing,
sharingDataQuality,
previousSharingDataQuality,
dataQuality,
coverage,
comparisonCoverage,
snapshotCoverage,
comparisonSnapshotCoverage,
] = await Promise.all([
getUserInventory(reader),
getSignupTotal(reader),
@@ -103,9 +115,13 @@ async function getDashboardOverviewStats(
getTrafficTotals(previousReader),
getSharingEventTotals(reader),
getSharingComparisonTotals(previousReader),
getSharingDataQuality(reader),
getSharingDataQuality(previousReader),
getTransferDataQuality(reader, previousReader),
reader.coverage(),
previousReader.coverage(),
reader.coverage('counters'),
previousReader.coverage('counters'),
reader.coverage('snapshots'),
previousReader.coverage('snapshots'),
])
const [trendNewUsers, activeByDay, storageUsedByDay, uploadByDay, downloadByDay] = await Promise.all([
getSignupsByDay(reader),
@@ -114,37 +130,51 @@ async function getDashboardOverviewStats(
getActivityMetricByDay(reader, metricSpec(['upload_confirm']), 'bytes'),
getActivityMetricByDay(reader, metricSpec(DOWNLOAD_ACTIVITY_ACTIONS), 'bytes'),
])
const trends = createDateBuckets(range).map((date) => {
const trends = createDateBuckets(effective).map((date) => {
return {
date,
newUsers: trendNewUsers.get(date) ?? 0,
activeUsers: activeByDay.get(date) ?? 0,
activeUsers: activeByDay.get(date) ?? null,
storageUsedBytes: storageUsedByDay.get(date) ?? null,
uploadBytes: uploadByDay.get(date) ?? 0,
downloadBytes: downloadByDay.get(date) ?? 0,
}
})
const sharingComparable =
comparable(coverage, comparisonCoverage) &&
hasExactSharingHistory(sharingDataQuality) &&
hasExactSharingHistory(previousSharingDataQuality)
const validQuotaBytes = quotas && quotas.invalidQuotaSpaces === 0 ? quotas.quotaBytes : null
return {
...statsFrame(now, range, coverage, comparisonCoverage),
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
dataQuality,
totals: {
users: users.total,
newUsers: delta(newUsers, previousNewUsers),
activeUsers: delta(activeUsers.mau, previousActiveUsers.mau),
activeUserRate: nullablePercent(activeUsers.mau, users.total),
storageUsedBytes: quotas.usedBytes,
storageQuotaBytes: quotas.quotaBytes,
storageUtilization: nullablePercent(quotas.usedBytes, quotas.quotaBytes),
users: users?.total ?? null,
newUsers: delta(newUsers, previousNewUsers, comparable(coverage, comparisonCoverage)),
activeUsers: delta(
activeUsers?.mau ?? null,
previousActiveUsers?.mau ?? null,
comparable(snapshotCoverage, comparisonSnapshotCoverage),
),
activeUserRate: nullablePercent(activeUsers?.mau ?? null, users?.total ?? null),
storageUsedBytes: quotas?.usedBytes ?? null,
storageQuotaBytes: validQuotaBytes,
storageUtilization: nullablePercent(quotas?.usedBytes ?? null, validQuotaBytes),
trafficBytes: delta(
traffic.uploadBytes + traffic.downloadBytes,
previousTraffic.uploadBytes + previousTraffic.downloadBytes,
comparable(coverage, comparisonCoverage),
),
uploadBytes: delta(traffic.uploadBytes, previousTraffic.uploadBytes, comparable(coverage, comparisonCoverage)),
downloadBytes: delta(
traffic.downloadBytes,
previousTraffic.downloadBytes,
comparable(coverage, comparisonCoverage),
),
uploadBytes: delta(traffic.uploadBytes, previousTraffic.uploadBytes),
downloadBytes: delta(traffic.downloadBytes, previousTraffic.downloadBytes),
activeShares: sharing.activeShares,
shareViews: delta(sharing.views, previousSharing.views),
shareDownloads: delta(sharing.downloads, previousSharing.downloads),
shareViews: delta(sharing.views, previousSharing.views, sharingComparable),
shareDownloads: delta(sharing.downloads, previousSharing.downloads, sharingComparable),
},
trends,
}
@@ -155,7 +185,8 @@ async function getDashboardOperationsStats(
now: Date,
range: AdminStatsDateRange,
): Promise<AdminDashboardOperationsStats> {
const reader = new AdminStatsHourlyReader(db, range, now)
const effective = effectiveRange(range, now)
const reader = new AdminStatsHourlyReader(db, effective, now)
const [
activeBackgroundJobs,
activeRemoteDownloads,
@@ -168,6 +199,7 @@ async function getDashboardOperationsStats(
backgroundJobsByDay,
remoteDownloadsByDay,
coverage,
snapshotCoverage,
] = await Promise.all([
getLatestGaugeTotal(reader, ADMIN_STATS_METRICS.backgroundJobSnapshot),
getLatestGaugeTotal(reader, ADMIN_STATS_METRICS.remoteDownloadTaskSnapshot),
@@ -179,7 +211,8 @@ async function getDashboardOperationsStats(
getCloudReportOutcomes(reader),
getOperationalOutcomesByDay(reader, 'background_job'),
getOperationalOutcomesByDay(reader, 'remote_download'),
reader.coverage(),
reader.coverage('counters'),
reader.coverage('snapshots'),
])
const completedJobs = backgroundJobOutcomes.get('completed') ?? 0
const failedJobs = backgroundJobOutcomes.get('failed') ?? 0
@@ -187,12 +220,14 @@ async function getDashboardOperationsStats(
const failedRemoteDownloads = remoteDownloadOutcomes.get('failed') ?? 0
return {
...statsFrame(now, range, coverage),
...statsFrame(now, effective, coverage, undefined, snapshotCoverage),
summary: {
activeBackgroundJobs,
activeRemoteDownloads,
onlineDownloaders: downloaderStatus.get('online') ?? 0,
offlineDownloaders: (downloaderStatus.get('offline') ?? 0) + (downloaderStatus.get('disabled') ?? 0),
onlineDownloaders: downloaderStatus?.get('online') ?? null,
offlineDownloaders: downloaderStatus
? (downloaderStatus.get('offline') ?? 0) + (downloaderStatus.get('disabled') ?? 0)
: null,
backgroundJobFailureRate: nullablePercent(failedJobs, completedJobs + failedJobs),
remoteDownloadSuccessRate: nullablePercent(
completedRemoteDownloads,
@@ -200,9 +235,13 @@ async function getDashboardOperationsStats(
),
cloudReportBacklog,
webhookFailures,
alertCount: cloudReportBacklog + webhookFailures,
cloudReportDeadLetters: cloudReportStatus?.get('dead_letter') ?? null,
alertCount:
cloudReportBacklog === null || webhookFailures === null
? null
: cloudReportBacklog + webhookFailures + (cloudReportStatus?.get('dead_letter') ?? 0),
},
trend: createDateBuckets(range).map((date) => ({
trend: createDateBuckets(effective).map((date) => ({
date,
completedJobs: backgroundJobsByDay.get(date)?.get('completed') ?? 0,
failedJobs: backgroundJobsByDay.get(date)?.get('failed') ?? 0,
@@ -211,8 +250,8 @@ async function getDashboardOperationsStats(
})),
backgroundJobOutcomes: percentRows([...backgroundJobOutcomes].map(([name, value]) => ({ name, value }))),
remoteDownloadOutcomes: percentRows([...remoteDownloadOutcomes].map(([name, value]) => ({ name, value }))),
downloaderStatus: percentRows([...downloaderStatus].map(([name, value]) => ({ name, value }))),
cloudReportStatus: percentRows([...cloudReportStatus].map(([name, value]) => ({ name, value }))),
downloaderStatus: percentRows([...(downloaderStatus ?? new Map())].map(([name, value]) => ({ name, value }))),
cloudReportStatus: percentRows([...(cloudReportStatus ?? new Map())].map(([name, value]) => ({ name, value }))),
}
}
@@ -221,8 +260,9 @@ async function getDashboardGrowthStats(
now: Date,
range: AdminStatsDateRange,
): Promise<AdminDashboardGrowthStats> {
const previous = previousRange(range)
const reader = new AdminStatsHourlyReader(db, range, now)
const effective = effectiveRange(range, now)
const previous = previousRange(effective)
const reader = new AdminStatsHourlyReader(db, effective, now)
const previousReader = new AdminStatsHourlyReader(db, previous, now)
const [
users,
@@ -235,45 +275,55 @@ async function getDashboardGrowthStats(
totalsByDay,
coverage,
comparisonCoverage,
snapshotCoverage,
comparisonSnapshotCoverage,
] = await Promise.all([
getUserInventory(reader),
getSignupTotal(reader),
getSignupTotal(previousReader),
getActiveUserSnapshot(reader),
getActiveUserSnapshot(previousReader),
getRollingActiveUserTrend(reader, range),
getRollingActiveUserTrend(reader, effective),
getRegistrationSources(reader),
getUserTotalsByDay(reader),
reader.coverage(),
previousReader.coverage(),
reader.coverage('counters'),
previousReader.coverage('counters'),
reader.coverage('snapshots'),
previousReader.coverage('snapshots'),
])
const newUsersByDay = await getSignupsByDay(reader)
const userScaleTrend = createDateBuckets(range).map((date) => ({
const userScaleTrend = createDateBuckets(effective).map((date) => ({
date,
newUsers: newUsersByDay.get(date) ?? 0,
totalUsers: totalsByDay.get(date) ?? 0,
totalUsers: totalsByDay.get(date) ?? null,
}))
return {
...statsFrame(now, range, coverage, comparisonCoverage),
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
summary: {
totalUsers: users.total,
newUsers: delta(newUsers, previousNewUsers),
activeUsers: delta(activeUsers.mau, previousActiveUsers.mau),
verifiedUsers: users.verified,
bannedUsers: users.banned,
silentUsers: users.silent,
activeUserRate: nullablePercent(activeUsers.mau, users.total),
silentUserRate: nullablePercent(users.silent, users.total),
totalUsers: users?.total ?? null,
newUsers: delta(newUsers, previousNewUsers, comparable(coverage, comparisonCoverage)),
activeUsers: delta(
activeUsers?.mau ?? null,
previousActiveUsers?.mau ?? null,
comparable(snapshotCoverage, comparisonSnapshotCoverage),
),
verifiedUsers: users?.verified ?? null,
bannedUsers: users?.banned ?? null,
silentUsers: users?.silent ?? null,
activeUserRate: nullablePercent(activeUsers?.mau ?? null, users?.total ?? null),
silentUserRate: nullablePercent(users?.silent ?? null, users?.total ?? null),
},
userScaleTrend,
activeUserTrend: activeByDay,
userStatus: percentRows([
{ name: 'normal', value: users.normal },
{ name: 'unverified', value: users.unverified },
{ name: 'banned', value: users.banned },
{ name: 'silent', value: users.silent },
]),
userStatus: users
? percentRows([
{ name: 'normal', value: users.normal },
{ name: 'unverified', value: users.unverified },
{ name: 'banned', value: users.banned },
{ name: 'silent', value: users.silent },
])
: [],
registrationSources,
}
}
@@ -283,12 +333,14 @@ async function getDashboardStorageStats(
now: Date,
range: AdminStatsDateRange,
): Promise<AdminDashboardStorageStats> {
const previous = previousRange(range)
const reader = new AdminStatsHourlyReader(db, range, now)
const effective = effectiveRange(range, now)
const previous = previousRange(effective)
const reader = new AdminStatsHourlyReader(db, effective, now)
const previousReader = new AdminStatsHourlyReader(db, previous, now)
const [
quotas,
inventory,
trashInventory,
storageUsedByDay,
typeBreakdown,
newFiles,
@@ -297,16 +349,20 @@ async function getDashboardStorageStats(
previousUploadBytes,
uploadsByDay,
uploadFilesByDay,
dataQuality,
transferDataQuality,
storageDataQuality,
spaceUsage,
quotaPressure,
sizeBreakdown,
ageBreakdown,
coverage,
comparisonCoverage,
snapshotCoverage,
comparisonSnapshotCoverage,
] = await Promise.all([
getQuotaTotals(reader),
getStorageInventory(reader),
getStorageTrashInventory(reader),
getStorageUsedByDay(reader),
getLatestInventoryBreakdown(reader, 'file_type_group'),
getActivityMetricTotal(reader, metricSpec(['upload_confirm']), 'count'),
@@ -316,14 +372,17 @@ async function getDashboardStorageStats(
getActivityMetricByDay(reader, metricSpec(['upload_confirm']), 'bytes'),
getActivityMetricByDay(reader, metricSpec(['upload_confirm']), 'count'),
getTransferDataQuality(reader, previousReader),
getStorageDataQuality(reader),
getUsageBySpaceRows(db, reader),
getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.storageQuota, 'status'),
getLatestInventoryBreakdown(reader, 'size_bucket'),
getLatestInventoryBreakdown(reader, 'age_bucket'),
reader.coverage(),
reader.coverage('counters'),
previousReader.coverage('counters'),
reader.coverage('snapshots'),
previousReader.coverage('snapshots'),
])
const storageTrend = createDateBuckets(range).map((date) => {
const storageTrend = createDateBuckets(effective).map((date) => {
return {
date,
usedBytes: storageUsedByDay.get(date) ?? null,
@@ -331,25 +390,31 @@ async function getDashboardStorageStats(
newFiles: uploadFilesByDay.get(date) ?? 0,
}
})
const coldFileBytes = ['90-180d', '>180d'].reduce(
(total, bucket) => total + (ageBreakdown.find((row) => row.name === bucket)?.bytes ?? 0),
0,
)
const coldFileBytes = inventory
? ['90-180d', '>180d'].reduce(
(total, bucket) => total + (ageBreakdown.find((row) => row.name === bucket)?.bytes ?? 0),
0,
)
: null
const validQuotaBytes = quotas && quotas.invalidQuotaSpaces === 0 ? quotas.quotaBytes : null
return {
...statsFrame(now, range, coverage, comparisonCoverage),
dataQuality,
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
dataQuality: { ...transferDataQuality, ...storageDataQuality },
summary: {
storageUsedBytes: quotas.usedBytes,
quotaBytes: quotas.quotaBytes,
fileCount: inventory.files,
newFiles: delta(newFiles, previousNewFiles),
newBytes: delta(uploadBytes, previousUploadBytes),
storageUsedBytes: quotas?.usedBytes ?? null,
quotaBytes: validQuotaBytes,
fileCount: inventory?.files ?? null,
trashFileCount: trashInventory?.files ?? null,
trashBytes: trashInventory?.bytes ?? null,
newFiles: delta(newFiles, previousNewFiles, comparable(coverage, comparisonCoverage)),
newBytes: delta(uploadBytes, previousUploadBytes, comparable(coverage, comparisonCoverage)),
coldFileBytes,
storageUtilization: nullablePercent(quotas.usedBytes, quotas.quotaBytes),
coldFilePercent: nullablePercent(coldFileBytes, quotas.usedBytes),
nearQuotaSpaces: quotaPressure.get('near') ?? 0,
overQuotaSpaces: quotaPressure.get('over') ?? 0,
storageUtilization: nullablePercent(quotas?.usedBytes ?? null, validQuotaBytes),
coldFilePercent: nullablePercent(coldFileBytes, inventory?.bytes ?? null),
nearQuotaSpaces: quotaPressure?.get('near') ?? null,
overQuotaSpaces: quotaPressure?.get('over') ?? null,
invalidQuotaSpaces: quotaPressure?.get('invalid') ?? null,
},
storageTrend,
typeBreakdown: typeBreakdown.map(({ name, ...row }) => ({
@@ -367,8 +432,9 @@ async function getDashboardTrafficStats(
now: Date,
range: AdminStatsDateRange,
): Promise<AdminDashboardTrafficStats> {
const previous = previousRange(range)
const reader = new AdminStatsHourlyReader(db, range, now)
const effective = effectiveRange(range, now)
const previous = previousRange(effective)
const reader = new AdminStatsHourlyReader(db, effective, now)
const previousReader = new AdminStatsHourlyReader(db, previous, now)
const [
traffic,
@@ -430,13 +496,13 @@ async function getDashboardTrafficStats(
item.value += value
failureReasonRows.set(reason, item)
}
const trafficTrend = createDateBuckets(range).map((date) => ({
const trafficTrend = createDateBuckets(effective).map((date) => ({
date,
uploadBytes: uploadByDay.get(date) ?? 0,
downloadBytes: downloadByDay.get(date) ?? 0,
requests: (uploadRequestsByDay.get(date) ?? 0) + (downloadRequestsByDay.get(date) ?? 0),
}))
const successTrend = createDateBuckets(range).map((date) => {
const successTrend = createDateBuckets(effective).map((date) => {
const uploadSuccesses = uploadSuccessByDay.get(date) ?? 0
const uploadFailures = uploadFailureByDay.get(date) ?? 0
const uploadRequests = uploadSuccesses + uploadFailures
@@ -454,14 +520,19 @@ async function getDashboardTrafficStats(
const issuedDownloads = Math.max(0, traffic.downloadRequests - blockedDownloads)
return {
...statsFrame(now, range, coverage, comparisonCoverage),
...statsFrame(now, effective, coverage, comparisonCoverage),
dataQuality,
summary: {
totalBytes: delta(
traffic.uploadBytes + traffic.downloadBytes,
previousTraffic.uploadBytes + previousTraffic.downloadBytes,
comparable(coverage, comparisonCoverage),
),
requestCount: delta(
totalRequests,
previousTraffic.uploadRequests + previousTraffic.downloadRequests,
comparable(coverage, comparisonCoverage),
),
requestCount: delta(totalRequests, previousTraffic.uploadRequests + previousTraffic.downloadRequests),
issuedDownloads,
blockedDownloads,
downloadIssueSuccessRate: nullablePercent(issuedDownloads, issuedDownloads + blockedDownloads),
@@ -482,8 +553,9 @@ async function getDashboardSharingStats(
now: Date,
range: AdminStatsDateRange,
): Promise<AdminDashboardSharingStats> {
const previous = previousRange(range)
const reader = new AdminStatsHourlyReader(db, range, now)
const effective = effectiveRange(range, now)
const previous = previousRange(effective)
const reader = new AdminStatsHourlyReader(db, effective, now)
const previousReader = new AdminStatsHourlyReader(db, previous, now)
const [
sharing,
@@ -494,8 +566,12 @@ async function getDashboardSharingStats(
saveCount,
previousSaveCount,
downloadSources,
dataQuality,
previousDataQuality,
coverage,
comparisonCoverage,
snapshotCoverage,
comparisonSnapshotCoverage,
] = await Promise.all([
getSharingEventTotals(reader),
getSharingComparisonTotals(previousReader),
@@ -505,17 +581,25 @@ async function getDashboardSharingStats(
getActivityMetricTotal(reader, metricSpec(['save_from_share']), 'count'),
getActivityMetricTotal(previousReader, metricSpec(['save_from_share']), 'count'),
getActivityMetricDimensionTotals(reader, metricSpec(['share_download']), 'source', 'count'),
reader.coverage(),
getSharingDataQuality(reader),
getSharingDataQuality(previousReader),
reader.coverage('counters'),
previousReader.coverage('counters'),
reader.coverage('snapshots'),
previousReader.coverage('snapshots'),
])
const landingDownloads = downloadSources.get('landing_share') ?? 0
const directDownloads = downloadSources.get('direct_share') ?? 0
const sharingComparable =
comparable(coverage, comparisonCoverage) &&
hasExactSharingHistory(dataQuality) &&
hasExactSharingHistory(previousDataQuality)
const [viewsByDay, downloadsByDay, savesByDay] = await Promise.all([
getActivityMetricByDay(reader, metricSpec(['share_view']), 'count'),
getActivityMetricByDay(reader, metricSpec(['share_download']), 'count'),
getActivityMetricByDay(reader, metricSpec(['save_from_share']), 'count'),
])
const trend = createDateBuckets(range).map((date) => ({
const trend = createDateBuckets(effective).map((date) => ({
date,
views: viewsByDay.get(date) ?? 0,
downloads: downloadsByDay.get(date) ?? 0,
@@ -527,15 +611,18 @@ async function getDashboardSharingStats(
})
return {
...statsFrame(now, range, coverage, comparisonCoverage),
...statsFrame(now, effective, coverage, comparisonCoverage, snapshotCoverage, comparisonSnapshotCoverage),
dataQuality,
summary: {
activeShares: sharing.activeShares,
createdShares: delta(createdInRange, createdPrevious),
views: delta(sharing.views, previousSharing.views),
downloads: delta(sharing.downloads, previousSharing.downloads),
saves: delta(saveCount, previousSaveCount),
downloadsPer100Views: nullablePercent(landingDownloads, sharing.views),
savesPer100Views: nullablePercent(saveCount, sharing.views),
createdShares: delta(createdInRange, createdPrevious, comparable(coverage, comparisonCoverage)),
views: delta(sharing.views, previousSharing.views, sharingComparable),
downloads: delta(sharing.downloads, previousSharing.downloads, sharingComparable),
saves: delta(saveCount, previousSaveCount, comparable(coverage, comparisonCoverage)),
downloadsPer100Views: hasExactSharingHistory(dataQuality)
? nullablePercent(landingDownloads, sharing.views)
: null,
savesPer100Views: hasExactSharingHistory(dataQuality) ? nullablePercent(saveCount, sharing.views) : null,
passwordPasses: sharing.passwordPasses,
},
trend,
@@ -563,11 +650,29 @@ async function getShareCreatedKinds(reader: AdminStatsHourlyReader): Promise<Map
return result
}
async function getSharingDataQuality(reader: AdminStatsHourlyReader): Promise<AdminSharingDataQuality> {
const dimensions = await getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.statsDataQualitySnapshot, 'kind')
if (!dimensions) return { unlocatedViews: null, unlocatedDownloads: null, unlocatedEvents: null }
const unlocatedViews = dimensions.get('share_views') ?? 0
const unlocatedDownloads = dimensions.get('share_downloads') ?? 0
return {
unlocatedViews,
unlocatedDownloads,
unlocatedEvents: unlocatedViews + unlocatedDownloads,
}
}
function hasExactSharingHistory(quality: AdminSharingDataQuality): boolean {
return quality.unlocatedEvents === 0
}
function statsFrame(
now: Date,
range: AdminStatsDateRange,
coverage: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
comparisonCoverage?: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
snapshotCoverage?: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
comparisonSnapshotCoverage?: Awaited<ReturnType<AdminStatsHourlyReader['coverage']>>,
) {
return {
generatedAt: now.toISOString(),
@@ -576,6 +681,8 @@ function statsFrame(
timeZone: 'UTC' as const,
coverage,
comparisonCoverage,
snapshotCoverage,
comparisonSnapshotCoverage,
}
}
@@ -598,7 +705,19 @@ function previousRange(range: AdminStatsDateRange): AdminStatsDateRange {
return { from: new Date(to.getTime() - durationMs), to, timeZone: range.timeZone }
}
function delta(value: number, previousValue: number): AdminStatsDelta {
function effectiveRange(range: AdminStatsDateRange, now: Date): AdminStatsDateRange {
const closedToExclusive = Math.min(range.to.getTime() + 1, startOfHour(now).getTime())
return {
from: range.from,
to: new Date(Math.max(range.from.getTime(), closedToExclusive) - 1),
timeZone: range.timeZone,
}
}
function delta(value: number | null, previousValue: number | null, canCompare = true): AdminStatsDelta {
if (value === null || previousValue === null || !canCompare) {
return { value, previousValue: canCompare ? previousValue : null, change: null, changePercent: null }
}
return {
value,
previousValue,
@@ -607,6 +726,10 @@ function delta(value: number, previousValue: number): AdminStatsDelta {
}
}
function comparable(current: { status: string }, previous: { status: string }): boolean {
return current.status === 'complete' && previous.status === 'complete'
}
function createDateBuckets(range: AdminStatsDateRange): string[] {
const dates = new Set<string>()
for (let timestamp = range.from.getTime(); timestamp <= range.to.getTime(); timestamp += 6 * 60 * 60 * 1000) {
@@ -616,9 +739,9 @@ function createDateBuckets(range: AdminStatsDateRange): string[] {
return [...dates]
}
async function getLatestGaugeTotal(reader: AdminStatsHourlyReader, metric: AdminStatsMetric): Promise<number> {
async function getLatestGaugeTotal(reader: AdminStatsHourlyReader, metric: AdminStatsMetric): Promise<number | null> {
const rows = await reader.latestRows(metric)
return rows.find((row) => row.orgId === '' && row.dimensionKey === '')?.count ?? 0
return rows.find((row) => row.orgId === '' && row.dimensionKey === '')?.count ?? null
}
async function getLatestGaugeDimensions(
@@ -626,7 +749,9 @@ async function getLatestGaugeDimensions(
metric: AdminStatsMetric,
dimensionKey: AdminStatsDimension,
field: 'count' | 'bytes' = 'count',
): Promise<Map<string, number>> {
): Promise<Map<string, number> | null> {
const baseRows = await reader.latestRows(metric)
if (!baseRows.some((row) => row.orgId === '' && row.dimensionKey === '')) return null
const result = new Map<string, number>()
for (const row of await reader.latestRows(metric, [dimensionKey])) {
if (row.orgId === '' && row.dimensionKey === dimensionKey) incrementMap(result, row.dimensionValue, row[field])
@@ -639,20 +764,27 @@ async function getLatestGaugeDimensionSum(
metric: AdminStatsMetric,
dimensionKey: AdminStatsDimension,
values: string[],
): Promise<number> {
): Promise<number | null> {
const dimensions = await getLatestGaugeDimensions(reader, metric, dimensionKey)
if (!dimensions) return null
return values.reduce((sum, value) => sum + (dimensions.get(value) ?? 0), 0)
}
async function getQuotaTotals(reader: AdminStatsHourlyReader): Promise<{ usedBytes: number; quotaBytes: number }> {
async function getQuotaTotals(
reader: AdminStatsHourlyReader,
): Promise<{ usedBytes: number; quotaBytes: number; invalidQuotaSpaces: number } | null> {
const [usedRows, quotaRows] = await Promise.all([
reader.latestRows(ADMIN_STATS_METRICS.storageUsed),
reader.latestRows(ADMIN_STATS_METRICS.storageQuota),
reader.latestRows(ADMIN_STATS_METRICS.storageQuota, ['', 'status']),
])
return {
usedBytes: usedRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes ?? 0,
quotaBytes: quotaRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes ?? 0,
}
const usedBytes = usedRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes
const quotaBytes = quotaRows.find((row) => row.orgId === '' && row.dimensionKey === '')?.bytes
const invalidQuotaSpaces = quotaRows.find(
(row) => row.orgId === '' && row.dimensionKey === 'status' && row.dimensionValue === 'invalid',
)?.count
return usedBytes === undefined || quotaBytes === undefined
? null
: { usedBytes, quotaBytes, invalidQuotaSpaces: invalidQuotaSpaces ?? 0 }
}
type UserInventory = {
@@ -664,13 +796,15 @@ type UserInventory = {
verified: number
}
async function getUserInventory(reader: AdminStatsHourlyReader): Promise<UserInventory> {
async function getUserInventory(reader: AdminStatsHourlyReader): Promise<UserInventory | null> {
const rows = await reader.latestRows(ADMIN_STATS_METRICS.userInventory, ['', 'status'])
const total = rows.find((row) => row.dimensionKey === '')?.count
if (total === undefined) return null
const dimensions = new Map(
rows.filter((row) => row.dimensionKey === 'status').map((row) => [row.dimensionValue, row.count]),
)
return {
total: rows.find((row) => row.dimensionKey === '')?.count ?? 0,
total,
normal: dimensions.get('normal') ?? 0,
unverified: dimensions.get('unverified') ?? 0,
banned: dimensions.get('banned') ?? 0,
@@ -681,8 +815,9 @@ async function getUserInventory(reader: AdminStatsHourlyReader): Promise<UserInv
type ActiveUserSnapshot = { dau: number; wau: number; mau: number }
async function getActiveUserSnapshot(reader: AdminStatsHourlyReader): Promise<ActiveUserSnapshot> {
async function getActiveUserSnapshot(reader: AdminStatsHourlyReader): Promise<ActiveUserSnapshot | null> {
const dimensions = await getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.userActiveSnapshot, 'window')
if (!dimensions) return null
return { dau: dimensions.get('dau') ?? 0, wau: dimensions.get('wau') ?? 0, mau: dimensions.get('mau') ?? 0 }
}
@@ -697,7 +832,7 @@ async function getUserTotalsByDay(reader: AdminStatsHourlyReader): Promise<Map<s
async function getRollingActiveUserTrend(
reader: AdminStatsHourlyReader,
range: AdminStatsDateRange,
): Promise<Array<{ date: string; dau: number; wau: number; mau: number }>> {
): Promise<Array<{ date: string; dau: number | null; wau: number | null; mau: number | null }>> {
const [dau, wau, mau] = await Promise.all([
getLatestGaugeValueByDay(reader, ADMIN_STATS_METRICS.userActiveSnapshot, 'count', 'window', 'dau'),
getLatestGaugeValueByDay(reader, ADMIN_STATS_METRICS.userActiveSnapshot, 'count', 'window', 'wau'),
@@ -705,9 +840,9 @@ async function getRollingActiveUserTrend(
])
return createDateBuckets(range).map((date) => ({
date,
dau: dau.get(date) ?? 0,
wau: wau.get(date) ?? 0,
mau: mau.get(date) ?? 0,
dau: dau.get(date) ?? null,
wau: wau.get(date) ?? null,
mau: mau.get(date) ?? null,
}))
}
@@ -836,7 +971,7 @@ async function getActivityMetricDimensionTotalsFromRollup(
return result
}
async function getCloudReportOutcomes(reader: AdminStatsHourlyReader): Promise<Map<string, number>> {
async function getCloudReportOutcomes(reader: AdminStatsHourlyReader): Promise<Map<string, number> | null> {
return getLatestGaugeDimensions(reader, ADMIN_STATS_METRICS.trafficReportSnapshot, 'status')
}
@@ -917,14 +1052,31 @@ async function getStorageUsedByDay(reader: AdminStatsHourlyReader): Promise<Map<
return getLatestGaugeValueByDay(reader, ADMIN_STATS_METRICS.storageUsed, 'bytes', '', '')
}
async function getStorageInventory(reader: AdminStatsHourlyReader): Promise<{ files: number; bytes: number }> {
async function getStorageInventory(reader: AdminStatsHourlyReader): Promise<{ files: number; bytes: number } | null> {
const rows = await reader.latestRows(ADMIN_STATS_METRICS.storageInventory)
return rows
.filter((row) => row.orgId === '' && row.dimensionKey === '')
.reduce((total, row) => ({ files: total.files + row.count, bytes: total.bytes + row.bytes }), {
files: 0,
bytes: 0,
})
const row = rows.find((value) => value.orgId === '' && value.dimensionKey === '')
return row ? { files: row.count, bytes: row.bytes } : null
}
async function getStorageTrashInventory(
reader: AdminStatsHourlyReader,
): Promise<{ files: number; bytes: number } | null> {
const rows = await reader.latestRows(ADMIN_STATS_METRICS.storageTrashSnapshot)
const row = rows.find((value) => value.orgId === '' && value.dimensionKey === '')
return row ? { files: row.count, bytes: row.bytes } : null
}
async function getStorageDataQuality(
reader: AdminStatsHourlyReader,
): Promise<Pick<AdminStorageDataQuality, 'usageDriftSpaces' | 'usageDriftBytes'>> {
const rows = await reader.latestRows(ADMIN_STATS_METRICS.statsDataQualitySnapshot, ['', 'kind'])
if (!rows.some((row) => row.orgId === '' && row.dimensionKey === '')) {
return { usageDriftSpaces: null, usageDriftBytes: null }
}
const drift = rows.find(
(row) => row.orgId === '' && row.dimensionKey === 'kind' && row.dimensionValue === 'storage_usage_drift',
)
return { usageDriftSpaces: drift?.count ?? 0, usageDriftBytes: drift?.bytes ?? 0 }
}
async function getLatestInventoryBreakdown(
@@ -932,7 +1084,9 @@ async function getLatestInventoryBreakdown(
dimensionKey: 'file_type_group' | 'size_bucket' | 'age_bucket',
): Promise<Array<{ name: string; bytes: number; files: number; percent: number }>> {
const values = new Map<string, { name: string; bytes: number; files: number }>()
for (const row of await reader.latestRows(ADMIN_STATS_METRICS.storageInventory, [dimensionKey])) {
const rows = await reader.latestRows(ADMIN_STATS_METRICS.storageInventory, ['', dimensionKey])
if (!rows.some((row) => row.orgId === '' && row.dimensionKey === '')) return []
for (const row of rows) {
if (row.orgId !== '' || row.dimensionKey !== dimensionKey) continue
const value = values.get(row.dimensionValue) ?? { name: row.dimensionValue, bytes: 0, files: 0 }
value.bytes += row.bytes
@@ -972,7 +1126,7 @@ async function getTrafficTotals(
async function getSharingEventTotals(
reader: AdminStatsHourlyReader,
): Promise<{ activeShares: number; views: number; passwordPasses: number; downloads: number }> {
): Promise<{ activeShares: number | null; views: number; passwordPasses: number; downloads: number }> {
const [activeShares, views, passwordPasses, downloads] = await Promise.all([
getLatestGaugeDimensionSum(reader, ADMIN_STATS_METRICS.shareInventory, 'lifecycle', ['usable']),
getActivityMetricTotal(reader, metricSpec(['share_view']), 'count'),
@@ -1035,7 +1189,19 @@ async function getTopSharesByActivity(db: Database, reader: AdminStatsHourlyRead
return topIds.flatMap((id) => {
const row = shareById.get(id)
const countValue = activityById.get(id)
if (!row || !countValue) return []
if (!countValue) return []
if (!row) {
return {
id,
token: '',
name: '已删除的分享',
creatorId: '',
creatorName: '已删除用户',
views: countValue.views,
downloads: countValue.downloads,
status: 'deleted',
}
}
return {
id: row.id,
token: row.token,
@@ -1074,7 +1240,7 @@ async function getUsageBySpaceRows(
...row,
orgName: org?.name ?? row.orgId,
orgType: org && isPersonalOrgLike({ slug: org.slug, metadata: org.metadata }) ? 'personal' : 'team',
utilization: percent(row.usedBytes, row.quotaBytes),
utilization: row.quotaBytes > 0 ? percent(row.usedBytes, row.quotaBytes) : null,
}
})
}
@@ -1123,8 +1289,8 @@ function percent(part: number, total: number): number {
return Math.round((part / total) * 1000) / 10
}
function nullablePercent(part: number, total: number): number | null {
return total > 0 ? percent(part, total) : null
function nullablePercent(part: number | null, total: number | null): number | null {
return part !== null && total !== null && total > 0 ? percent(part, total) : null
}
function toNumber(value: unknown): number {
+41 -6
View File
@@ -1,7 +1,8 @@
import type { BackgroundJob, BackgroundJobStatus } from '@shared/types'
import { and, count, desc, eq, type SQL } from 'drizzle-orm'
import { nanoid } from 'nanoid'
import { backgroundJobs } from '../../db/schema'
import { activityEvents, backgroundJobs } from '../../db/schema'
import { type AtomicQuery, executeWriteTransaction } from '../../db/transaction'
import type { Database } from '../../platform/interface'
import {
BackgroundJobError,
@@ -9,6 +10,7 @@ import {
type BackgroundJobRepo,
type ListBackgroundJobsOptions,
} from '../../usecases/ports'
import { adminStatsFactValues } from './admin-stats-fact'
type BackgroundJobRow = typeof backgroundJobs.$inferSelect
@@ -153,7 +155,25 @@ export function createBackgroundJobRepo(db: Database): BackgroundJobRepo {
finishedAt: input.finishedAt === undefined ? finishedAtFor(nextStatus, row.finishedAt, now) : input.finishedAt,
updatedAt: now,
}
await db.update(backgroundJobs).set(values).where(eq(backgroundJobs.id, id))
const writes: AtomicQuery[] = [db.update(backgroundJobs).set(values).where(eq(backgroundJobs.id, id))]
if (!row.finishedAt && values.finishedAt && ['completed', 'failed', 'canceled'].includes(nextStatus)) {
writes.push(
db
.insert(activityEvents)
.values(
adminStatsFactValues({
action: 'stats_background_job_finished',
sourceId: row.id,
orgId: row.orgId,
targetType: 'background_job',
occurredAt: values.finishedAt,
metadata: { jobType: row.type, outcome: nextStatus },
}),
)
.onConflictDoNothing(),
)
}
await executeWriteTransaction(db, writes)
return repo.get(orgId, id)
},
@@ -164,10 +184,25 @@ export function createBackgroundJobRepo(db: Database): BackgroundJobRepo {
throw new BackgroundJobError('not_cancelable')
}
const now = new Date()
await db
.update(backgroundJobs)
.set({ status: 'canceled', updatedAt: now, finishedAt: now })
.where(eq(backgroundJobs.id, id))
await executeWriteTransaction(db, [
db
.update(backgroundJobs)
.set({ status: 'canceled', updatedAt: now, finishedAt: now })
.where(eq(backgroundJobs.id, id)),
db
.insert(activityEvents)
.values(
adminStatsFactValues({
action: 'stats_background_job_finished',
sourceId: row.id,
orgId: row.orgId,
targetType: 'background_job',
occurredAt: now,
metadata: { jobType: row.type, outcome: 'canceled' },
}),
)
.onConflictDoNothing(),
])
return repo.get(orgId, id)
},
+1
View File
@@ -152,6 +152,7 @@ function revokeExistingPlanQueries(
eq(orgQuotaEntitlements.resourceType, value.resourceType),
eq(orgQuotaEntitlements.entitlementType, 'plan'),
eq(orgQuotaEntitlements.status, 'active'),
sql`${orgQuotaEntitlements.source} <> 'free_plan'`,
sql`${orgQuotaEntitlements.sourceId} != ${value.sourceId}`,
),
),
+27 -6
View File
@@ -1,4 +1,4 @@
import { asc, eq, inArray } from 'drizzle-orm'
import { and, asc, eq, isNull, lte, or, sql } from 'drizzle-orm'
import { nanoid } from 'nanoid'
import { cloudTrafficReports } from '../../db/schema'
import type { Database } from '../../platform/interface'
@@ -23,6 +23,8 @@ function toRecord(row: typeof cloudTrafficReports.$inferSelect): CloudTrafficRep
creditsPerUnit: row.creditsPerUnit,
status: row.status as CloudTrafficReportStatus,
error: row.error,
attemptCount: row.attemptCount,
nextRetryAt: row.nextRetryAt,
createdAt: row.createdAt,
updatedAt: row.updatedAt,
}
@@ -49,24 +51,43 @@ export function createCloudTrafficReportRepo(db: Database): CloudTrafficReportRe
creditsPerUnit: input.creditsPerUnit,
status: input.status,
error: null,
attemptCount: 0,
nextRetryAt: null,
createdAt: input.now,
updatedAt: input.now,
})
},
async updateStatus(eventId, status, error, now) {
async updateStatus(eventId, status, error, now, retry) {
await db
.update(cloudTrafficReports)
.set({ status, error, updatedAt: now })
.set({
status,
error,
updatedAt: now,
...(retry ? { attemptCount: retry.attemptCount, nextRetryAt: retry.nextRetryAt } : {}),
})
.where(eq(cloudTrafficReports.eventId, eventId))
},
async listPending(limit) {
async listPending(limit, now) {
const rows = await db
.select()
.from(cloudTrafficReports)
.where(inArray(cloudTrafficReports.status, ['pending', 'failed']))
.orderBy(asc(cloudTrafficReports.createdAt))
.where(
or(
eq(cloudTrafficReports.status, 'pending'),
and(
eq(cloudTrafficReports.status, 'failed'),
or(isNull(cloudTrafficReports.nextRetryAt), lte(cloudTrafficReports.nextRetryAt, now)),
),
),
)
.orderBy(
asc(sql`CASE WHEN ${cloudTrafficReports.status} = 'pending' THEN 0 ELSE 1 END`),
asc(cloudTrafficReports.nextRetryAt),
asc(cloudTrafficReports.createdAt),
)
.limit(limit)
return rows.map(toRecord)
},
+32 -2
View File
@@ -1,7 +1,8 @@
import { downloadTaskRuntimeSchema } from '@shared/schemas'
import type { DownloadTask, DownloadTaskRuntime } from '@shared/types'
import { and, asc, count, desc, eq, gte, inArray, isNull, like, ne, notInArray, or, type SQL, sql } from 'drizzle-orm'
import { downloaders, downloadTasks } from '../../db/schema'
import { activityEvents, downloaders, downloadTasks } from '../../db/schema'
import { type AtomicQuery, executeWriteTransaction } from '../../db/transaction'
import type { Database } from '../../platform/interface'
import {
type CreateDownloadTaskRecordInput,
@@ -11,6 +12,7 @@ import {
type ListDownloadTasksFilters,
type UpdateDownloadTaskFields,
} from '../../usecases/ports'
import { adminStatsFactValues } from './admin-stats-fact'
type DownloadTaskRow = typeof downloadTasks.$inferSelect
@@ -244,7 +246,35 @@ export function createDownloadTaskRepo(db: Database): DownloadTaskRepo {
},
async setFields(id, fields: UpdateDownloadTaskFields) {
await db.update(downloadTasks).set(fields).where(eq(downloadTasks.id, id))
const row = await findRow(id)
if (!row) throw new DownloadError('not_found')
const nextStatus = fields.status ?? row.status
const finishedAt = fields.finishedAt ?? row.finishedAt
const writes: AtomicQuery[] = [db.update(downloadTasks).set(fields).where(eq(downloadTasks.id, id))]
if (!row.finishedAt && finishedAt && ['completed', 'failed', 'canceled'].includes(nextStatus)) {
writes.push(
db
.insert(activityEvents)
.values(
adminStatsFactValues({
action: 'stats_remote_download_finished',
sourceId: `${row.id}:${row.attempt}:${finishedAt.getTime()}:${nextStatus}`,
targetId: row.id,
orgId: row.orgId,
targetType: 'remote_download',
occurredAt: finishedAt,
metadata: {
category: row.category ?? 'uncategorized',
downloaderId: row.assignedDownloaderId,
outcome: nextStatus,
bytes: nextStatus === 'completed' ? (fields.billingChargedBytes ?? row.billingChargedBytes) : 0,
},
}),
)
.onConflictDoNothing(),
)
}
await executeWriteTransaction(db, writes)
},
async claimQueued(id, downloaderId, now) {
@@ -260,6 +260,16 @@ describe('confirmUpload — name conflict', () => {
const { db } = await createTestApp()
await insertStorage(db)
const orgId = nanoid()
const now = Date.now()
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES (${nanoid()}, ${orgId}, 0, 0, 0, 0, '2026-05')
`)
await db.run(sql`
INSERT INTO org_quota_entitlements
(id, org_id, resource_type, entitlement_type, source, source_id, bytes, starts_at, status, created_at, updated_at)
VALUES (${nanoid()}, ${orgId}, 'storage', 'plan', 'test', ${nanoid()}, 1000000, ${now}, 'active', ${now}, ${now})
`)
const draftId = await makeFile(db, orgId, 'collision.txt', { status: 'draft' })
await makeFile(db, orgId, 'collision.txt') // active sibling
@@ -107,43 +107,43 @@ async function insertDraftFile(
// ─── storage usage reservations ───────────────────────────────────────────────
describe('reserveStorageUsage', () => {
it('reserves and increments when no quota row exists (unlimited)', async () => {
it('fails closed when no quota row exists', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
const storageId = await insertStorage(db, { id: 'st-ul', used: 0 })
const result = await reserveStorageUsage(
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
{ orgId, storageId, bytes: 500 },
)
expect(result).toEqual({ orgId, storageId, bytes: 500 })
await expect(
reserveStorageUsage(
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
{ orgId, storageId, bytes: 500 },
),
).rejects.toThrow(StorageQuotaExceededError)
const rows = await db.all<{ used: number }>(sql`SELECT used FROM storages WHERE id = ${storageId}`)
expect(rows[0].used).toBe(500)
expect(rows[0].used).toBe(0)
})
it('reserves and increments when quota is 0 (unlimited)', async () => {
it('fails closed when effective quota is zero', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
const storageId = await insertStorage(db, { id: 'st-q0', used: 100 })
await insertOrgQuota(db, orgId, 0, 5000)
const result = await reserveStorageUsage(
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
{ orgId, storageId, bytes: 999999 },
)
expect(result).toEqual({ orgId, storageId, bytes: 999999 })
await expect(
reserveStorageUsage(
{ quota: createQuotaRepo(db), storageUsage: createStorageUsageRepo(db) },
{ orgId, storageId, bytes: 999999 },
),
).rejects.toThrow(StorageQuotaExceededError)
const rows = await db.all<{ used: number }>(sql`SELECT used FROM storages WHERE id = ${storageId}`)
expect(rows[0].used).toBe(1000099)
expect(rows[0].used).toBe(100)
})
it('treats base quota 0 as unlimited without grant aggregation', async () => {
it('treats effective storage quota 0 as invalid', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
await insertOrgQuota(db, orgId, 0, 5000)
await expect(createQuotaRepo(db).hasQuotaForBytes(orgId, 10_000_000)).resolves.toBe(true)
await expect(createQuotaRepo(db).hasQuotaForBytes(orgId, 10_000_000)).resolves.toBe(false)
await expect(createQuotaRepo(db).getEffectiveQuota(orgId)).resolves.toMatchObject({ baseQuota: 0, quota: 0 })
})
+99 -5
View File
@@ -1,11 +1,64 @@
import { eq, sql } from 'drizzle-orm'
import { nanoid } from 'nanoid'
import { describe, expect, it } from 'vitest'
import { orgQuotaEntitlements, orgQuotas } from '../../db/schema.js'
import { createTestApp } from '../../test/setup.js'
import { orgQuotaEntitlements, orgQuotas, systemOptions } from '../../db/schema.js'
import { adminHeaders, createTestApp } from '../../test/setup.js'
import { createQuotaRepo } from './quota.js'
describe('effective quota', () => {
it('enforces one quota row per organization', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
await db.insert(orgQuotas).values({ id: nanoid(), orgId })
await expect(db.insert(orgQuotas).values({ id: nanoid(), orgId })).rejects.toThrow()
})
it('restores missing and revoked Free baselines idempotently', async () => {
const { app, db } = await createTestApp()
await adminHeaders(app)
const [{ orgId }] = await db.select({ orgId: orgQuotas.orgId }).from(orgQuotas).limit(1)
const now = new Date('2026-07-20T12:00:00.000Z')
await db.delete(orgQuotaEntitlements).where(eq(orgQuotaEntitlements.orgId, orgId))
await db
.insert(systemOptions)
.values([
{ key: 'default_org_quota', value: '1234' },
{ key: 'default_org_monthly_traffic_quota', value: '5678' },
])
.onConflictDoUpdate({ target: systemOptions.key, set: { value: sql`excluded.value` } })
const repo = createQuotaRepo(db)
await repo.reconcileFreePlanBaselines(now)
await repo.reconcileFreePlanBaselines(new Date(now.getTime() + 1000))
const inserted = await db
.select({ resourceType: orgQuotaEntitlements.resourceType, bytes: orgQuotaEntitlements.bytes })
.from(orgQuotaEntitlements)
.where(eq(orgQuotaEntitlements.orgId, orgId))
.orderBy(orgQuotaEntitlements.resourceType)
expect(inserted).toEqual([
{ resourceType: 'storage', bytes: 1234 },
{ resourceType: 'traffic', bytes: 5678 },
])
await db
.update(orgQuotaEntitlements)
.set({ status: 'revoked', expiresAt: now })
.where(eq(orgQuotaEntitlements.orgId, orgId))
await repo.reconcileFreePlanBaselines(new Date(now.getTime() + 2000))
await expect(
db
.select({ status: orgQuotaEntitlements.status, expiresAt: orgQuotaEntitlements.expiresAt })
.from(orgQuotaEntitlements)
.where(eq(orgQuotaEntitlements.orgId, orgId))
.orderBy(orgQuotaEntitlements.resourceType),
).resolves.toEqual([
{ status: 'active', expiresAt: null },
{ status: 'active', expiresAt: null },
])
})
it('returns storage and traffic quota state', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
@@ -111,6 +164,44 @@ describe('effective quota', () => {
})
})
it('keeps Free as the active baseline and falls back to it after a paid plan expires', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
const now = new Date('2026-05-06T00:00:00Z')
await db.insert(orgQuotas).values({
id: nanoid(),
orgId,
quota: 0,
used: 250,
trafficQuota: 0,
trafficUsed: 0,
trafficPeriod: '2026-05',
})
await db.insert(orgQuotaEntitlements).values([
{
...entitlement(orgId, 'storage', 'free-storage-plan', 5000, 'active', now, 'Free'),
source: 'free_plan',
},
{
...entitlement(orgId, 'storage', `stripe_subscription:sub_storage:${orgId}`, 3000, 'active', now, 'Pro'),
expiresAt: new Date('2026-05-07T00:00:00Z'),
},
])
await expect(createQuotaRepo(db).getEffectiveQuota(orgId, now)).resolves.toMatchObject({
baseQuota: 3000,
quota: 3000,
storagePlanName: 'Pro',
})
await expect(createQuotaRepo(db).getEffectiveQuota(orgId, new Date('2026-05-08T00:00:00Z'))).resolves.toMatchObject(
{
baseQuota: 5000,
quota: 5000,
storagePlanName: 'Free',
},
)
})
it('uses a smaller active subscription plan instead of the larger default quota', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
@@ -639,6 +730,9 @@ describe('effective quota', () => {
trafficUsed: 900,
trafficPeriod: '2026-04',
})
await db
.insert(orgQuotaEntitlements)
.values(entitlement(orgId, 'traffic', 'free-traffic-plan', 1000, 'active', now, 'Free'))
await db.run(sql`
CREATE TRIGGER org_quotas_rollover_race
BEFORE UPDATE ON org_quotas
@@ -656,13 +750,13 @@ describe('effective quota', () => {
expect(rows[0].trafficPeriod).toBe('2026-05')
})
it('allows traffic when no quota row exists', async () => {
it('fails closed when no traffic quota row exists', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
const now = new Date('2026-05-06T00:00:00Z')
await expect(createQuotaRepo(db).hasTrafficQuotaForBytes(orgId, 1024, now)).resolves.toBe(true)
await expect(createQuotaRepo(db).consumeTrafficIfQuotaAllows(orgId, 1024, now)).resolves.toBe(true)
await expect(createQuotaRepo(db).hasTrafficQuotaForBytes(orgId, 1024, now)).resolves.toBe(false)
await expect(createQuotaRepo(db).consumeTrafficIfQuotaAllows(orgId, 1024, now)).resolves.toBe(false)
})
it('treats zero base storage quota as limited when storage entitlements exist', async () => {
+133 -62
View File
@@ -1,6 +1,7 @@
import { DEFAULT_ORG_QUOTA, DEFAULT_ORG_TRAFFIC_QUOTA } from '@shared/constants'
import { and, eq, inArray, or, sql } from 'drizzle-orm'
import { organization } from '../../db/auth-schema'
import { orgQuotaEntitlements, orgQuotas, storages } from '../../db/schema'
import { orgQuotaEntitlements, orgQuotas, storages, systemOptions } from '../../db/schema'
import { currentTrafficPeriod } from '../../domain/quota'
import type { Database } from '../../platform/interface'
import type { CurrentStoragePlan, EffectiveQuota, QuotaRepo } from '../../usecases/ports'
@@ -15,7 +16,7 @@ async function getEffectiveQuota(db: Database, orgId: string, now = new Date()):
// Batch variant of getEffectiveQuota for list views. Resolves every org with two
// queries total (quota rows + active entitlements) instead of ~8 per org, then
// aggregates in memory. Returns one entry per requested orgId, even with no rows.
async function getEffectiveQuotasByOrg(
export async function getEffectiveQuotasByOrg(
db: Database,
orgIds: string[],
now = new Date(),
@@ -53,6 +54,7 @@ async function getEffectiveQuotasByOrg(
orgId: orgQuotaEntitlements.orgId,
resourceType: orgQuotaEntitlements.resourceType,
entitlementType: orgQuotaEntitlements.entitlementType,
source: orgQuotaEntitlements.source,
sourceId: orgQuotaEntitlements.sourceId,
bytes: orgQuotaEntitlements.bytes,
startsAt: orgQuotaEntitlements.startsAt,
@@ -125,6 +127,7 @@ function chunk<T>(items: T[], size: number): T[][] {
interface EntitlementRow {
resourceType: string
entitlementType: string
source: string
sourceId: string
bytes: number
startsAt: Date
@@ -134,10 +137,11 @@ interface EntitlementRow {
// Mirrors activePlanEntitlement's ORDER BY bytes DESC, startsAt DESC LIMIT 1.
function pickPlanEntitlement(ents: EntitlementRow[], resourceType: 'storage' | 'traffic'): PlanEntitlement | null {
const plans = ents
.filter((e) => e.resourceType === resourceType && e.entitlementType === 'plan')
.sort((a, b) => b.bytes - a.bytes || b.startsAt.getTime() - a.startsAt.getTime())
const row = plans[0]
const plans = ents.filter((e) => e.resourceType === resourceType && e.entitlementType === 'plan')
const paidPlans = plans.filter((plan) => plan.source !== 'free_plan')
const row = (paidPlans.length > 0 ? paidPlans : plans).sort(
(a, b) => b.bytes - a.bytes || b.startsAt.getTime() - a.startsAt.getTime(),
)[0]
return row ? toPlanEntitlement(row) : null
}
@@ -169,17 +173,92 @@ async function resetExpiredTrafficQuotas(db: Database, now = new Date()): Promis
.where(sql`${orgQuotas.trafficPeriod} != ${period}`)
}
async function reconcileFreePlanBaselines(db: Database, now = new Date()): Promise<void> {
const rows = await db
.select({ key: systemOptions.key, value: systemOptions.value })
.from(systemOptions)
.where(inArray(systemOptions.key, ['default_org_quota', 'default_team_quota', 'default_org_monthly_traffic_quota']))
const options = new Map(rows.map((row) => [row.key, row.value]))
const orgQuota = positiveOption(options.get('default_org_quota'), DEFAULT_ORG_QUOTA)
const teamQuota = positiveOption(options.get('default_team_quota'), orgQuota)
const trafficQuota = nonNegativeOption(options.get('default_org_monthly_traffic_quota'), DEFAULT_ORG_TRAFFIC_QUOTA)
const timestamp = now.getTime()
await db.run(sql`
UPDATE org_quota_entitlements
SET status = 'active', expires_at = NULL, updated_at = ${timestamp}
WHERE source = 'free_plan' AND entitlement_type = 'plan'
AND EXISTS (SELECT 1 FROM organization o WHERE o.id = org_quota_entitlements.org_id)
AND (status <> 'active' OR expires_at IS NOT NULL)
`)
await db.run(sql`
INSERT INTO org_quota_entitlements (
id, org_id, resource_type, entitlement_type, source, source_id, bytes,
starts_at, expires_at, status, metadata, created_at, updated_at
)
SELECT
lower(hex(randomblob(16))), o.id, 'storage', 'plan', 'free_plan', 'free_plan:' || o.id,
CASE WHEN json_valid(o.metadata) = 1 AND json_extract(o.metadata, '$.type') = 'team'
THEN ${teamQuota} ELSE ${orgQuota} END,
${timestamp}, NULL, 'active',
json_object(
'packageName', 'Free', 'packageId', NULL, 'source', 'free_plan',
'settingKey', CASE WHEN json_valid(o.metadata) = 1 AND json_extract(o.metadata, '$.type') = 'team'
THEN 'default_team_quota' ELSE 'default_org_quota' END
),
${timestamp}, ${timestamp}
FROM organization o
WHERE NOT EXISTS (
SELECT 1 FROM org_quota_entitlements e
WHERE e.org_id = o.id AND e.resource_type = 'storage'
AND e.entitlement_type = 'plan' AND e.source = 'free_plan'
)
`)
await db.run(sql`
INSERT INTO org_quota_entitlements (
id, org_id, resource_type, entitlement_type, source, source_id, bytes,
starts_at, expires_at, status, metadata, created_at, updated_at
)
SELECT
lower(hex(randomblob(16))), o.id, 'traffic', 'plan', 'free_plan', 'free_plan:' || o.id,
${trafficQuota}, ${timestamp}, NULL, 'active',
json_object(
'packageName', 'Free', 'packageId', NULL, 'source', 'free_plan',
'settingKey', 'default_org_monthly_traffic_quota'
),
${timestamp}, ${timestamp}
FROM organization o
WHERE NOT EXISTS (
SELECT 1 FROM org_quota_entitlements e
WHERE e.org_id = o.id AND e.resource_type = 'traffic'
AND e.entitlement_type = 'plan' AND e.source = 'free_plan'
)
`)
}
function positiveOption(value: string | undefined, fallback: number): number {
const parsed = Number(value)
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback
}
function nonNegativeOption(value: string | undefined, fallback: number): number {
if (value === undefined || value.trim() === '') return fallback
const parsed = Number(value)
return Number.isInteger(parsed) && parsed >= 0 ? parsed : fallback
}
async function hasQuotaForBytes(db: Database, orgId: string, bytes: number): Promise<boolean> {
if (bytes <= 0) return true
const quota = await getEffectiveQuota(db, orgId)
if (quota.baseQuota === 0 && quota.entitlementQuota === 0) return true
if (quota.quota <= 0) return false
return quota.used + bytes <= quota.quota
}
async function hasTrafficQuotaForBytes(db: Database, orgId: string, bytes: number, now = new Date()): Promise<boolean> {
if (bytes <= 0) return true
const quota = await getEffectiveQuota(db, orgId, now)
if (quota.baseTrafficQuota === 0 && quota.entitlementTrafficQuota === 0) return true
const trafficPlan = await activePlanEntitlement(db, orgId, 'traffic', now)
if (quota.trafficQuota === 0) return trafficPlan !== null
if ((quota.currentPlan?.trafficOveragePriceCents ?? 0) > 0) return true
return quota.trafficUsed + bytes <= quota.trafficQuota
}
@@ -197,19 +276,14 @@ async function consumeTrafficIfQuotaAllows(
.from(orgQuotas)
.where(eq(orgQuotas.orgId, orgId))
.limit(1)
if (quotaRows.length === 0) return true
if (quotaRows.length === 0) return false
const trafficOverageAllowed = await hasActiveTrafficOverage(db, orgId, now)
const overageAllowedSql = trafficOverageAllowed ? sql`1 = 1` : sql`1 = 0`
if (quotaRows[0].trafficPeriod !== period) {
const limitBytes = activeEntitlementBytesSql({
aggregate: sql`SUM(${orgQuotaEntitlements.bytes})`,
orgId,
resourceType: 'traffic',
now,
sourceCondition: sql`1 = 1`,
})
const limitBytes = activeEntitlementBytesSql(orgId, 'traffic', now)
const hasPlan = activePlanEntitlementExistsSql(orgId, 'traffic', now)
const updated = await db
.update(orgQuotas)
.set({ trafficUsed: bytes, trafficPeriod: period })
@@ -217,7 +291,7 @@ async function consumeTrafficIfQuotaAllows(
sql`${orgQuotas.orgId} = ${orgId}
AND ${orgQuotas.trafficPeriod} != ${period}
AND (
(${limitBytes} = 0)
(${limitBytes} = 0 AND ${hasPlan})
OR ${overageAllowedSql}
OR ${bytes} <= ${limitBytes}
)`,
@@ -226,13 +300,8 @@ async function consumeTrafficIfQuotaAllows(
if (updated.length > 0) return true
}
const limitBytes = activeEntitlementBytesSql({
aggregate: sql`SUM(${orgQuotaEntitlements.bytes})`,
orgId,
resourceType: 'traffic',
now,
sourceCondition: sql`1 = 1`,
})
const limitBytes = activeEntitlementBytesSql(orgId, 'traffic', now)
const hasPlan = activePlanEntitlementExistsSql(orgId, 'traffic', now)
const updated = await db
.update(orgQuotas)
.set({ trafficUsed: sql`${orgQuotas.trafficUsed} + ${bytes}` })
@@ -240,7 +309,7 @@ async function consumeTrafficIfQuotaAllows(
sql`${orgQuotas.orgId} = ${orgId}
AND ${orgQuotas.trafficPeriod} = ${period}
AND (
(${limitBytes} = 0)
(${limitBytes} = 0 AND ${hasPlan})
OR ${overageAllowedSql}
OR ${orgQuotas.trafficUsed} + ${bytes} <= ${limitBytes}
)`,
@@ -276,28 +345,19 @@ async function incrementUsageIfEffectiveQuotaAllows(
): Promise<boolean> {
if (teamQuotaEnabled) {
const rows = await db.select({ id: orgQuotas.id }).from(orgQuotas).where(eq(orgQuotas.orgId, orgId)).limit(1)
if (rows.length > 0) {
const limitBytes = activeEntitlementBytesSql({
aggregate: sql`SUM(${orgQuotaEntitlements.bytes})`,
orgId,
resourceType: 'storage',
now,
sourceCondition: sql`1 = 1`,
})
const updated = await db
.update(orgQuotas)
.set({ used: sql`${orgQuotas.used} + ${bytes}` })
.where(
sql`${orgQuotas.orgId} = ${orgId}
AND (
(${limitBytes} = 0)
OR ${orgQuotas.used} + ${bytes} <= ${limitBytes}
)`,
)
.returning({ id: orgQuotas.id })
if (rows.length === 0) return false
const limitBytes = activeEntitlementBytesSql(orgId, 'storage', now)
const updated = await db
.update(orgQuotas)
.set({ used: sql`${orgQuotas.used} + ${bytes}` })
.where(
sql`${orgQuotas.orgId} = ${orgId}
AND ${limitBytes} > 0
AND ${orgQuotas.used} + ${bytes} <= ${limitBytes}`,
)
.returning({ id: orgQuotas.id })
if (updated.length === 0) return false
}
if (updated.length === 0) return false
}
await db
@@ -323,7 +383,10 @@ async function activePlanEntitlement(
})
.from(orgQuotaEntitlements)
.where(activePlanEntitlementWhere(orgId, resourceType, now))
.orderBy(sql`${orgQuotaEntitlements.bytes} DESC, ${orgQuotaEntitlements.startsAt} DESC`)
.orderBy(
sql`CASE WHEN ${orgQuotaEntitlements.source} = 'free_plan' THEN 1 ELSE 0 END`,
sql`${orgQuotaEntitlements.bytes} DESC, ${orgQuotaEntitlements.startsAt} DESC`,
)
.limit(1)
const row = rows[0]
@@ -369,29 +432,36 @@ function activeEntitlementWhere(
)
}
function activeEntitlementBytesSql({
aggregate,
orgId,
resourceType,
now,
sourceCondition,
}: {
aggregate: ReturnType<typeof sql>
orgId: string
resourceType: 'storage' | 'traffic'
now: Date
sourceCondition: ReturnType<typeof sql>
}) {
function activeEntitlementBytesSql(orgId: string, resourceType: 'storage' | 'traffic', now: Date) {
const timestamp = now.getTime()
return sql`(
SELECT COALESCE(${aggregate}, 0)
SELECT
COALESCE(
MAX(CASE WHEN ${orgQuotaEntitlements.entitlementType} = 'plan' AND ${orgQuotaEntitlements.source} <> 'free_plan' THEN ${orgQuotaEntitlements.bytes} END),
MAX(CASE WHEN ${orgQuotaEntitlements.entitlementType} = 'plan' THEN ${orgQuotaEntitlements.bytes} END),
0
)
+ COALESCE(SUM(CASE WHEN ${orgQuotaEntitlements.entitlementType} <> 'plan' THEN ${orgQuotaEntitlements.bytes} ELSE 0 END), 0)
FROM ${orgQuotaEntitlements}
WHERE ${orgQuotaEntitlements.orgId} = ${orgId}
AND ${orgQuotaEntitlements.resourceType} = ${resourceType}
AND ${orgQuotaEntitlements.status} = 'active'
AND ${orgQuotaEntitlements.startsAt} <= ${timestamp}
AND (${orgQuotaEntitlements.expiresAt} IS NULL OR ${orgQuotaEntitlements.expiresAt} > ${timestamp})
AND ${sourceCondition}
)`
}
function activePlanEntitlementExistsSql(orgId: string, resourceType: 'storage' | 'traffic', now: Date) {
const timestamp = now.getTime()
return sql`EXISTS (
SELECT 1
FROM ${orgQuotaEntitlements}
WHERE ${orgQuotaEntitlements.orgId} = ${orgId}
AND ${orgQuotaEntitlements.resourceType} = ${resourceType}
AND ${orgQuotaEntitlements.entitlementType} = 'plan'
AND ${orgQuotaEntitlements.status} = 'active'
AND ${orgQuotaEntitlements.startsAt} <= ${timestamp}
AND (${orgQuotaEntitlements.expiresAt} IS NULL OR ${orgQuotaEntitlements.expiresAt} > ${timestamp})
)`
}
@@ -466,6 +536,7 @@ export function createQuotaRepo(db: Database): QuotaRepo {
listOrgQuotaOverview: () => listOrgQuotaOverview(db),
getEffectiveQuota: (orgId, now) => getEffectiveQuota(db, orgId, now),
getEffectiveQuotasByOrg: (orgIds, now) => getEffectiveQuotasByOrg(db, orgIds, now),
reconcileFreePlanBaselines: (now) => reconcileFreePlanBaselines(db, now),
resetExpiredTrafficQuotas: (now) => resetExpiredTrafficQuotas(db, now),
hasQuotaForBytes: (orgId, bytes) => hasQuotaForBytes(db, orgId, bytes),
hasTrafficQuotaForBytes: (orgId, bytes, now) => hasTrafficQuotaForBytes(db, orgId, bytes, now),
@@ -1,4 +1,4 @@
import { eq } from 'drizzle-orm'
import { and, eq } from 'drizzle-orm'
import { nanoid } from 'nanoid'
import { describe, expect, it } from 'vitest'
import { DirType } from '../../../shared/constants'
@@ -329,7 +329,10 @@ describe('recordView', () => {
const resolved = await resolveShareByToken(db, share.token)
if (resolved.status !== 'ok') throw new Error('expected found')
expect(resolved.share.views).toBe(2)
const events = await db.select().from(activityEvents).where(eq(activityEvents.targetId, share.id))
const events = await db
.select()
.from(activityEvents)
.where(and(eq(activityEvents.targetId, share.id), eq(activityEvents.action, 'share_view')))
expect(events).toHaveLength(2)
expect(events.every((event) => event.action === 'share_view')).toBe(true)
})
@@ -355,7 +358,12 @@ describe('recordView', () => {
const resolved = await resolveShareByToken(db, share.token)
if (resolved.status !== 'ok') throw new Error('expected found')
expect(resolved.share.views).toBe(0)
await expect(db.select().from(activityEvents).where(eq(activityEvents.targetId, share.id))).resolves.toEqual([])
await expect(
db
.select()
.from(activityEvents)
.where(and(eq(activityEvents.targetId, share.id), eq(activityEvents.action, 'share_view'))),
).resolves.toEqual([])
})
})
+17 -1
View File
@@ -16,6 +16,7 @@ import {
type ShareResolution,
} from '../../usecases/ports'
import { activityEventValues } from './activity'
import { adminStatsFactValues } from './admin-stats-fact'
import { createQuotaRepo } from './quota'
function buildPath(parent: string, name: string): string {
@@ -63,7 +64,22 @@ export function createShareRepo(db: Database): ShareRepo {
createdAt: now,
}
const queries: AtomicQuery[] = [db.insert(shares).values(share)]
const queries: AtomicQuery[] = [
db.insert(shares).values(share),
db
.insert(activityEvents)
.values(
adminStatsFactValues({
action: 'stats_share_created',
sourceId: share.id,
orgId: share.orgId,
targetType: 'share',
occurredAt: now,
metadata: { kind: share.kind },
}),
)
.onConflictDoNothing(),
]
if (input.recipients && input.recipients.length > 0) {
const recipientRows = input.recipients.map((r) => ({
id: nanoid(),
@@ -246,7 +246,7 @@ describe('selectStorage', () => {
return createStorageRepo(db).get(created.id)
}
it('returns an active storage with unlimited capacity', async () => {
it('returns an active storage whose capacity is not reported', async () => {
const { db } = await createTestApp()
const created = await seedActive(db)
const found = await createStorageRepo(db).select()
+21 -1
View File
@@ -27,6 +27,7 @@ import {
import { generateUserOrgSlug, isPersonalOrgLike } from '../shared/org-slugs'
import { createEmailGateway } from './adapters/gateways/email'
import { createActivityRepo } from './adapters/repos/activity'
import { adminStatsFactValues } from './adapters/repos/admin-stats-fact'
import { createInviteRepo } from './adapters/repos/invite'
import { createLicenseBindingRepo } from './adapters/repos/license-binding'
import { createMemberCountRepo } from './adapters/repos/member-count'
@@ -35,7 +36,7 @@ import { createOrgRepo } from './adapters/repos/org'
import { createSiteInvitationRepo } from './adapters/repos/site-invitations'
import { createSystemOptionsRepo } from './adapters/repos/system-options'
import * as authSchema from './db/auth-schema'
import { orgQuotaEntitlements, orgQuotas, systemOptions } from './db/schema'
import { activityEvents, orgQuotaEntitlements, orgQuotas, systemOptions } from './db/schema'
import { executeWriteTransaction } from './db/transaction'
import { CAPTCHA_AUTH_ENDPOINTS, type CaptchaConfig } from './domain/captcha'
import { currentTrafficPeriod } from './domain/quota'
@@ -656,6 +657,12 @@ async function createPersonalOrg(
const orgSlug = await generateUniqueOrgSlug(db, generateUserOrgSlug)
const quotaValues = await createOrgQuotaValues(db, orgId, now)
const entitlementValues = await createFreePlanEntitlementValues(db, orgId, now, false)
const [account] = await db
.select({ providerId: authSchema.account.providerId })
.from(authSchema.account)
.where(eq(authSchema.account.userId, user.id))
.orderBy(authSchema.account.createdAt, authSchema.account.id)
.limit(1)
await executeWriteTransaction(db, [
db.insert(authSchema.organization).values({
@@ -674,6 +681,19 @@ async function createPersonalOrg(
}),
db.insert(orgQuotas).values(quotaValues),
...entitlementValues.map((value) => db.insert(orgQuotaEntitlements).values(value)),
db
.insert(activityEvents)
.values(
adminStatsFactValues({
action: 'stats_user_signup',
sourceId: user.id,
orgId,
targetType: 'user',
occurredAt: now,
metadata: { provider: account?.providerId ?? 'unknown' },
}),
)
.onConflictDoNothing(),
])
return orgId
+17 -10
View File
@@ -80,15 +80,19 @@ export const storages = sqliteTable('storages', {
updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull(),
})
export const orgQuotas = sqliteTable('org_quotas', {
id: text('id').primaryKey(),
orgId: text('org_id').notNull(),
quota: integer('quota').notNull().default(0),
used: integer('used').notNull().default(0),
trafficQuota: integer('traffic_quota').notNull().default(0),
trafficUsed: integer('traffic_used').notNull().default(0),
trafficPeriod: text('traffic_period').notNull().default('1970-01'),
})
export const orgQuotas = sqliteTable(
'org_quotas',
{
id: text('id').primaryKey(),
orgId: text('org_id').notNull(),
quota: integer('quota').notNull().default(0),
used: integer('used').notNull().default(0),
trafficQuota: integer('traffic_quota').notNull().default(0),
trafficUsed: integer('traffic_used').notNull().default(0),
trafficPeriod: text('traffic_period').notNull().default('1970-01'),
},
(t) => [uniqueIndex('org_quotas_org_uniq').on(t.orgId)],
)
export const cloudTrafficReports = sqliteTable(
'cloud_traffic_reports',
@@ -105,6 +109,8 @@ export const cloudTrafficReports = sqliteTable(
creditsPerUnit: integer('credits_per_unit'),
status: text('status').notNull(),
error: text('error'),
attemptCount: integer('attempt_count').notNull().default(0),
nextRetryAt: integer('next_retry_at', { mode: 'timestamp_ms' }),
createdAt: integer('created_at', { mode: 'timestamp_ms' }).notNull(),
updatedAt: integer('updated_at', { mode: 'timestamp_ms' }).notNull(),
},
@@ -112,6 +118,7 @@ export const cloudTrafficReports = sqliteTable(
uniqueIndex('cloud_traffic_reports_event_uniq').on(t.eventId),
index('cloud_traffic_reports_org_period_idx').on(t.orgId, t.period),
index('cloud_traffic_reports_status_idx').on(t.status),
index('cloud_traffic_reports_retry_idx').on(t.status, t.nextRetryAt, t.createdAt),
index('cloud_traffic_reports_updated_idx').on(t.updatedAt),
],
)
@@ -138,7 +145,7 @@ export const orgQuotaEntitlements = sqliteTable(
index('org_quota_entitlements_org_type_idx').on(t.orgId, t.resourceType, t.entitlementType, t.status),
uniqueIndex('org_quota_entitlements_active_plan_uniq')
.on(t.orgId, t.resourceType, t.entitlementType)
.where(sql`status = 'active' AND entitlement_type = 'plan'`),
.where(sql`status = 'active' AND entitlement_type = 'plan' AND source <> 'free_plan'`),
uniqueIndex('org_quota_entitlements_source_resource_uniq').on(t.source, t.sourceId, t.resourceType),
],
)
+39 -4
View File
@@ -1,11 +1,14 @@
export const ROLLUP_VERSION = 2
export const ROLLUP_VERSION = 3
export type AdminStatsRollupScope = 'counters' | 'full'
export type AdminStatsRollupScope = 'counters' | 'snapshots' | 'full'
export interface AdminStatsRollupMetadata {
version: number
scope: AdminStatsRollupScope
quality: 'exact' | 'lower_bound'
counterQuality?: 'exact' | 'lower_bound'
snapshotQuality?: 'exact' | 'lower_bound'
snapshotObservedAt?: string
}
export type AdminStatsMetricKind = 'counter' | 'gauge'
@@ -47,10 +50,12 @@ export const ADMIN_STATS_METRICS = {
sharePasswordPassed: 'share.password_passed',
shareSaved: 'share.saved',
shareView: 'share.view',
statsDataQualitySnapshot: 'stats.data_quality_snapshot',
statsMissingBytes: 'stats.quality_missing_bytes',
statsRollupRun: 'stats.rollup_run',
storageInventory: 'storage.inventory',
storageQuota: 'storage.quota',
storageTrashSnapshot: 'storage.trash_snapshot',
storageUsed: 'storage.used',
trafficReportSnapshot: 'traffic.report_snapshot',
transferDownloadFailed: 'transfer.download_failed',
@@ -85,10 +90,12 @@ export const ADMIN_STATS_METRIC_REGISTRY = {
[M.sharePasswordPassed]: counter(['share_id']),
[M.shareSaved]: counter(['actor_type', 'share_id'], true),
[M.shareView]: counter(['actor_type', 'share_id']),
[M.statsDataQualitySnapshot]: gauge(['kind'], 'events', true),
[M.statsMissingBytes]: counter(['direction', 'source']),
[M.statsRollupRun]: counter(['outcome']),
[M.storageInventory]: gauge(['age_bucket', 'file_type_group', 'size_bucket', 'storage_id'], 'entities', true),
[M.storageQuota]: gauge(['status'], null, true),
[M.storageTrashSnapshot]: gauge(['storage_id'], 'entities', true),
[M.storageUsed]: gauge(['storage_id'], null, true),
[M.trafficReportSnapshot]: gauge(['status'], 'entities', true),
[M.transferDownloadFailed]: counter(['reason', 'source'], true),
@@ -131,9 +138,37 @@ export function parseAdminStatsRollupMetadata(metadata: string | null): AdminSta
if (!value || typeof value !== 'object' || Array.isArray(value)) return null
const record = value as Record<string, unknown>
if (record.version !== ROLLUP_VERSION) return null
if (record.scope !== 'counters' && record.scope !== 'full') return null
if (record.scope !== 'counters' && record.scope !== 'snapshots' && record.scope !== 'full') return null
if (record.quality !== 'exact' && record.quality !== 'lower_bound') return null
return { version: record.version, scope: record.scope, quality: record.quality }
if (
record.counterQuality !== undefined &&
record.counterQuality !== 'exact' &&
record.counterQuality !== 'lower_bound'
) {
return null
}
if (
record.snapshotQuality !== undefined &&
record.snapshotQuality !== 'exact' &&
record.snapshotQuality !== 'lower_bound'
) {
return null
}
const snapshotObservedAt = record.snapshotObservedAt ?? record.observedAt
if (
snapshotObservedAt !== undefined &&
(typeof snapshotObservedAt !== 'string' || !Number.isFinite(Date.parse(snapshotObservedAt)))
) {
return null
}
return {
version: record.version,
scope: record.scope,
quality: record.quality,
counterQuality: record.counterQuality,
snapshotQuality: record.snapshotQuality,
snapshotObservedAt,
}
} catch {
return null
}
+1
View File
@@ -157,6 +157,7 @@ console.log('stats.rollup.scheduler.started interval=10m')
function writeStatsRollup(): void {
void (async () => {
try {
await deps.quota.reconcileFreePlanBaselines()
const results = await deps.adminStats.refreshHourlyRollups(new Date())
console.log(
JSON.stringify({
+135 -42
View File
@@ -2,7 +2,7 @@ import type { AdminDashboardOverviewStats } from '@shared/types'
import { sql } from 'drizzle-orm'
import { describe, expect, it } from 'vitest'
import { createAdminStatsRepo, metricSpec } from '../adapters/repos/admin-stats'
import { rebuildAdminStatsHour } from '../adapters/repos/admin-stats-rollup'
import { captureAdminStatsSnapshot, rebuildAdminStatsHour } from '../adapters/repos/admin-stats-rollup'
import { currentTrafficPeriod } from '../domain/quota'
import { adminHeaders, createTestApp, seedProLicense } from '../test/setup.js'
@@ -155,10 +155,10 @@ describe('site stats routes', () => {
)
VALUES (
'storage-marker-2026-01-01', ${Date.UTC(2026, 0, 1)}, '', 'stats.rollup_run', '', '',
1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
1, 0, 0, '{"version":3,"scope":"full","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
), (
'storage-used-2026-01-01', ${Date.UTC(2026, 0, 1)}, '', 'storage.used', '', '',
0, 4096, 0, '{"version":2,"scope":"full","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
0, 4096, 0, '{"version":3,"scope":"snapshots","quality":"exact"}', ${Date.UTC(2026, 0, 2)}
)
`)
@@ -191,9 +191,9 @@ describe('site stats routes', () => {
count, bytes, unique_count, metadata, updated_at
) VALUES
('hourly-reader-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
'{"version":3,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
('hourly-reader-upload', ${at}, ${orgId}, 'transfer.upload', 'status', 'success', 1, 999, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000})
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000})
`)
const query =
@@ -228,18 +228,18 @@ describe('site stats routes', () => {
(id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
count, bytes, unique_count, metadata, updated_at)
VALUES
('dimensions-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-share-total', ${at}, ${orgId}, 'share.created', '', '', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-share-kind', ${at}, ${orgId}, 'share.created', 'kind', 'landing', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-job-total', ${at}, ${orgId}, 'background_job.finished', '', '', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-job-outcome', ${at}, ${orgId}, 'background_job.finished', 'outcome', 'failed', 1, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-download-total', ${at}, ${orgId}, 'transfer.download_issued', '', '', 1, 10, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-download-source', ${at}, ${orgId}, 'transfer.download_issued', 'source', 'object_download', 1, 10, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-failure-total', ${at}, ${orgId}, 'transfer.download_failed', '', '', 1, 4, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-failure-reason', ${at}, ${orgId}, 'transfer.download_failed', 'reason', 'network', 1, 4, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-quality-total', ${at}, ${orgId}, 'stats.quality_missing_bytes', '', '', 5, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-quality-upload', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'upload', 2, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at}),
('dimensions-quality-download', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'download', 3, 0, 0, '{"version":2,"scope":"full","quality":"exact"}', ${at})
('dimensions-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0, '{"version":3,"scope":"full","quality":"exact"}', ${at}),
('dimensions-share-total', ${at}, ${orgId}, 'share.created', '', '', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-share-kind', ${at}, ${orgId}, 'share.created', 'kind', 'landing', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-job-total', ${at}, ${orgId}, 'background_job.finished', '', '', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-job-outcome', ${at}, ${orgId}, 'background_job.finished', 'outcome', 'failed', 1, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-download-total', ${at}, ${orgId}, 'transfer.download_issued', '', '', 1, 10, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-download-source', ${at}, ${orgId}, 'transfer.download_issued', 'source', 'object_download', 1, 10, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-failure-total', ${at}, ${orgId}, 'transfer.download_failed', '', '', 1, 4, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-failure-reason', ${at}, ${orgId}, 'transfer.download_failed', 'reason', 'network', 1, 4, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-quality-total', ${at}, ${orgId}, 'stats.quality_missing_bytes', '', '', 5, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-quality-upload', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'upload', 2, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at}),
('dimensions-quality-download', ${at}, ${orgId}, 'stats.quality_missing_bytes', 'direction', 'download', 3, 0, 0, '{"version":3,"scope":"counters","quality":"exact"}', ${at})
`)
const query = 'from=2026-07-01T10%3A00%3A00.000Z&to=2026-07-01T10%3A59%3A59.999Z&timeZone=UTC'
@@ -293,13 +293,13 @@ describe('site stats routes', () => {
ORDER BY bucket_start, org_id
`)
expect(rows.map((row) => Number(row.bucketStart))).toContain(Date.UTC(2026, 6, 10, 17))
expect(rows.map((row) => Number(row.bucketStart))).toContain(Date.UTC(2026, 6, 10, 18))
expect(
rows
.filter((row) => Number(row.bucketStart) === Date.UTC(2026, 6, 10, 17))
.filter((row) => Number(row.bucketStart) === Date.UTC(2026, 6, 10, 18))
.reduce((sum, row) => sum + row.bytes, 0),
).toBe(expected)
expect(JSON.parse(rows.at(-1)?.metadata ?? '{}')).toMatchObject({ version: 2, quality: 'exact' })
expect(JSON.parse(rows.at(-1)?.metadata ?? '{}')).toMatchObject({ version: 3, quality: 'exact' })
})
it('does not write rollups while serving storage stats', async () => {
@@ -362,15 +362,15 @@ describe('site stats routes', () => {
count, bytes, unique_count, metadata, updated_at)
VALUES
('growth-rollup-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
'{"version":3,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
('growth-rollup-total', ${at}, '', 'user.signup', '', '', 2, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000}),
('growth-inventory-total', ${at}, '', 'user.inventory', '', '', 2, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
'{"version":3,"scope":"snapshots","quality":"exact"}', ${at + 3_600_000}),
('growth-rollup-credential', ${at}, '', 'user.signup', 'provider', 'credential', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000}),
('growth-rollup-github', ${at}, '', 'user.signup', 'provider', 'github', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000})
'{"version":3,"scope":"counters","quality":"exact"}', ${at + 3_600_000})
`)
const res = await app.request('/api/site/stats/growth?from=2026-01-01&to=2026-01-01', { headers })
@@ -409,7 +409,8 @@ describe('site stats routes', () => {
const beforeRes = await app.request('/api/site/stats/storage', { headers })
const before = (await beforeRes.json()) as { typeBreakdown: Array<{ type: string; files: number; bytes: number }> }
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
await captureAdminStatsSnapshot(db, bucketStart, new Date(bucketStart.getTime() + 45 * 60_000))
await rebuildAdminStatsHour(db, bucketStart, new Date())
const afterRes = await app.request('/api/site/stats/storage', { headers })
const after = (await afterRes.json()) as { typeBreakdown: Array<{ type: string; files: number; bytes: number }> }
@@ -425,6 +426,17 @@ describe('site stats routes', () => {
await seedProLicense(db)
const { bucketStart } = await seedStatsFixture(db)
for (let index = 0; index < 12; index += 1) {
await db.run(sql`
INSERT INTO organization (id, name, slug, metadata, created_at, updated_at)
VALUES (
${`quota-pressure-org-${index}`},
${`Quota Pressure ${index}`},
${`quota-pressure-${index}`},
'{"type":"team"}',
${bucketStart.getTime()},
${bucketStart.getTime()}
)
`)
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES (
@@ -437,19 +449,55 @@ describe('site stats routes', () => {
'2026-07'
)
`)
await db.run(sql`
INSERT INTO org_quota_entitlements
(id, org_id, resource_type, entitlement_type, source, source_id, bytes, starts_at, status, created_at, updated_at)
VALUES (
${`quota-pressure-plan-${index}`},
${`quota-pressure-org-${index}`},
'storage',
'plan',
'test',
${`quota-pressure-source-${index}`},
100,
${bucketStart.getTime()},
'active',
${bucketStart.getTime()},
${bucketStart.getTime()}
)
`)
}
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
await db.run(sql`
INSERT INTO organization (id, name, slug, metadata, created_at, updated_at)
VALUES ('quota-invalid-org', 'Invalid Quota', 'quota-invalid', '{"type":"team"}', ${bucketStart.getTime()}, ${bucketStart.getTime()})
`)
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES ('quota-invalid', 'quota-invalid-org', 0, 1000, 0, 0, '2026-07')
`)
await captureAdminStatsSnapshot(db, bucketStart, new Date(bucketStart.getTime() + 45 * 60_000))
await rebuildAdminStatsHour(db, bucketStart, new Date())
const res = await app.request('/api/site/stats/storage', { headers })
const body = (await res.json()) as {
summary: { nearQuotaSpaces: number; overQuotaSpaces: number }
topSpaces: Array<{ orgId: string }>
summary: {
quotaBytes: number | null
storageUtilization: number | null
nearQuotaSpaces: number
overQuotaSpaces: number
invalidQuotaSpaces: number
}
topSpaces: Array<{ orgId: string; utilization: number | null }>
}
expect(res.status).toBe(200)
expect(body.summary.nearQuotaSpaces).toBe(9)
expect(body.summary.overQuotaSpaces).toBe(3)
expect(body.summary.invalidQuotaSpaces).toBe(2)
expect(body.summary.quotaBytes).toBeNull()
expect(body.summary.storageUtilization).toBeNull()
expect(body.topSpaces).toHaveLength(8)
expect(body.topSpaces[0]).toMatchObject({ orgId: 'quota-invalid-org', utilization: null })
})
it('reads historical active users from completed snapshots instead of raw activity', async () => {
@@ -471,9 +519,11 @@ describe('site stats routes', () => {
count, bytes, unique_count, metadata, updated_at)
VALUES
('active-snapshot-marker', ${at}, '', 'stats.rollup_run', '', '', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
'{"version":3,"scope":"full","quality":"exact"}', ${at + 3_600_000}),
('active-snapshot-total', ${at}, '', 'user.active_snapshot', '', '', 1, 0, 0,
'{"version":3,"scope":"snapshots","quality":"exact"}', ${at + 3_600_000}),
('active-snapshot-mau', ${at}, '', 'user.active_snapshot', 'window', 'mau', 1, 0, 0,
'{"version":2,"scope":"full","quality":"exact"}', ${at + 3_600_000})
'{"version":3,"scope":"snapshots","quality":"exact"}', ${at + 3_600_000})
`)
const res = await app.request('/api/site/stats/overview?from=2026-01-02&to=2026-01-02', { headers })
@@ -493,7 +543,7 @@ describe('site stats routes', () => {
VALUES ('upload-cancel-rate', ${orgId}, ${userId}, 'user', 'upload_cancel', 'file', 'cancel.bin',
'{', ${eventSec})
`)
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
await rebuildAdminStatsHour(db, bucketStart, new Date())
const current = await app.request('/api/site/stats/traffic', { headers })
const currentBody = (await current.json()) as {
@@ -523,8 +573,8 @@ describe('site stats routes', () => {
('missing-previous-download-bytes', ${orgId}, ${userId}, 'user', 'share_download', 'share', 'previous.bin', '{}',
${Math.floor(Date.parse('2026-06-30T12:00:00.000Z') / 1000)})
`)
await rebuildAdminStatsHour(db, new Date('2026-07-01T12:00:00.000Z'), new Date(), false)
await rebuildAdminStatsHour(db, new Date('2026-06-30T12:00:00.000Z'), new Date(), false)
await rebuildAdminStatsHour(db, new Date('2026-07-01T12:00:00.000Z'), new Date())
await rebuildAdminStatsHour(db, new Date('2026-06-30T12:00:00.000Z'), new Date())
const res = await app.request('/api/site/stats/overview?from=2026-07-01&to=2026-07-01', { headers })
const body = (await res.json()) as { dataQuality: AdminDashboardOverviewStats['dataQuality'] }
@@ -611,13 +661,14 @@ describe('site stats routes', () => {
VALUES ('operations-cloud-report', ${orgId}, '2026-07', 'object_download', 'stats-file',
'operations-cloud-report', 64, 'pending', ${reportAt}, ${reportAt})
`)
await rebuildAdminStatsHour(db, previousBucket, new Date(reportAt + 3_600_000), true)
await rebuildAdminStatsHour(db, previousBucket, new Date(reportAt + 3_600_000))
await db.run(sql`
UPDATE cloud_traffic_reports
SET status = 'reported', updated_at = ${eventMs}
WHERE id = 'operations-cloud-report'
`)
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
await captureAdminStatsSnapshot(db, bucketStart, new Date(bucketStart.getTime() + 45 * 60_000))
await rebuildAdminStatsHour(db, bucketStart, new Date())
const res = await app.request('/api/site/stats/operations', { headers })
const body = (await res.json()) as {
@@ -659,7 +710,12 @@ describe('site stats routes', () => {
const currentSharingRes = await app.request('/api/site/stats/sharing', { headers })
const currentSharing = (await currentSharingRes.json()) as {
summary: { views: { value: number }; downloads: { value: number } }
dataQuality: { unlocatedViews: number; unlocatedDownloads: number; unlocatedEvents: number }
summary: {
views: { value: number; change: number | null }
downloads: { value: number; change: number | null }
downloadsPer100Views: number | null
}
topShares: Array<{
token: string
views: number
@@ -676,6 +732,10 @@ describe('site stats routes', () => {
expect(currentSharingRes.status).toBe(200)
expect(currentSharing.summary.views.value).toBe(1)
expect(currentSharing.summary.downloads.value).toBe(1)
expect(currentSharing.dataQuality).toEqual({ unlocatedViews: 11, unlocatedDownloads: 3, unlocatedEvents: 14 })
expect(currentSharing.summary.views.change).toBeNull()
expect(currentSharing.summary.downloads.change).toBeNull()
expect(currentSharing.summary.downloadsPer100Views).toBeNull()
expect(currentSharing.topShares[0]).toMatchObject({
token: 'share-token-1',
views: 1,
@@ -688,6 +748,29 @@ describe('site stats routes', () => {
expect(oldSharing.topShares).toEqual([])
})
it('keeps deleted shares in historical rankings without reading their counters live', async () => {
const { app, db } = await createTestApp()
const headers = await adminHeaders(app)
await seedProLicense(db)
await seedStatsFixture(db)
await db.run(sql`DELETE FROM shares WHERE id = 'share-1'`)
const res = await app.request('/api/site/stats/sharing', { headers })
const body = (await res.json()) as {
topShares: Array<{ id: string; token: string; name: string; status: string; views: number; downloads: number }>
}
expect(res.status).toBe(200)
expect(body.topShares[0]).toMatchObject({
id: 'share-1',
token: '',
name: '已删除的分享',
status: 'deleted',
views: 1,
downloads: 1,
})
})
it('orders top share rankings by views before downloads', async () => {
const { app, db } = await createTestApp()
const headers = await adminHeaders(app)
@@ -706,14 +789,14 @@ describe('site stats routes', () => {
('activity-download-heavy-2', ${orgId}, NULL, 'anonymous', 'share_download', 'share', 'share-download-heavy', 'report.pdf', '{"bytes":512,"source":"landing_share","anonymous":true}', ${eventSec}),
('activity-download-heavy-3', ${orgId}, NULL, 'anonymous', 'share_download', 'share', 'share-download-heavy', 'report.pdf', '{"bytes":512,"source":"landing_share","anonymous":true}', ${eventSec})
`)
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
await rebuildAdminStatsHour(db, bucketStart, new Date())
await db.run(sql`
INSERT INTO stats_rollups_hourly
(id, bucket_start, org_id, metric_key, dimension_key, dimension_value,
count, bytes, unique_count, metadata, updated_at)
VALUES
('ranking-invalid-quality', ${bucketStart.getTime()}, ${orgId}, 'share.view', 'share_id',
'share-download-heavy', 1000, 0, 0, '{"version":2,"scope":"full"}', ${bucketStart.getTime()})
'share-download-heavy', 1000, 0, 0, '{"version":3,"scope":"counters"}', ${bucketStart.getTime()})
`)
const res = await app.request('/api/site/stats/sharing', { headers })
@@ -741,7 +824,7 @@ describe('site stats routes', () => {
'{"source":"landing_share"}', ${eventSec})
`)
}
await rebuildAdminStatsHour(db, bucketStart, new Date(), true)
await rebuildAdminStatsHour(db, bucketStart, new Date())
const res = await app.request('/api/site/stats/sharing', { headers })
const body = (await res.json()) as { topShares: Array<{ viewPercent: number }> }
@@ -801,7 +884,16 @@ async function seedStatsFixture(db: Awaited<ReturnType<typeof createTestApp>>['d
('activity-share-view', ${orgId}, NULL, 'anonymous', 'share_view', 'share', 'share-1', 'report.pdf', '{"source":"landing_share","anonymous":true}', ${nowSec}),
('activity-share-password', ${orgId}, NULL, 'anonymous', 'share_password_passed', 'share', 'share-1', 'report.pdf', '{"source":"landing_share","anonymous":true}', ${nowSec}),
('activity-share-download', ${orgId}, NULL, 'anonymous', 'share_download', 'share', 'share-1', 'report.pdf', '{"bytes":512,"source":"landing_share","anonymous":true}', ${nowSec}),
('activity-object-download', ${orgId}, ${userId}, 'user', 'object_download', 'file', 'stats-file', 'report.pdf', '{"bytes":256,"source":"object_download"}', ${nowSec})
('activity-object-download', ${orgId}, ${userId}, 'user', 'object_download', 'file', 'stats-file', 'report.pdf', '{"bytes":256,"source":"object_download"}', ${nowSec}),
('stats-fixture-share-created', ${orgId}, NULL, 'system', 'stats_share_created', 'share', 'share-1', 'share-1', '{"kind":"landing","statsQuality":"exact"}', ${nowSec}),
('stats-fixture-task-completed', ${orgId}, NULL, 'system', 'stats_remote_download_finished', 'remote_download', 'task-1', 'task-1', '{"category":"direct","outcome":"completed","bytes":0,"statsQuality":"exact"}', ${nowSec}),
('stats-fixture-task-failed', ${orgId}, NULL, 'system', 'stats_remote_download_finished', 'remote_download', 'task-2', 'task-2', '{"category":"direct","outcome":"failed","bytes":0,"statsQuality":"exact"}', ${nowSec}),
('stats-fixture-job-failed', ${orgId}, NULL, 'system', 'stats_background_job_finished', 'background_job', 'job-1', 'job-1', '{"jobType":"extract","outcome":"failed","statsQuality":"exact"}', ${nowSec})
`)
await db.run(sql`
UPDATE activity_events
SET created_at = ${nowSec}, metadata = '{"provider":"credential","statsQuality":"exact"}'
WHERE action = 'stats_user_signup' AND target_id = ${userId}
`)
await db.run(sql`
INSERT INTO downloaders (id, name, token_hash, token_jti, status, enabled, version, hostname, platform, arch, engine, capabilities, max_concurrent_tasks, current_tasks, download_bps, upload_bps, free_disk_bytes, created_by, last_heartbeat_at, created_at, updated_at)
@@ -818,6 +910,7 @@ async function seedStatsFixture(db: Awaited<ReturnType<typeof createTestApp>>['d
VALUES ('job-1', ${orgId}, ${userId}, 'extract', 'failed', '', '', NULL, 0, 0, 0, 0, NULL, 'bad zip', NULL, 1, 0, NULL, ${now}, ${now}, ${now}, ${now})
`)
await rebuildAdminStatsHour(db, bucketStart, new Date(generatedAt), true)
await captureAdminStatsSnapshot(db, bucketStart, new Date(now))
await rebuildAdminStatsHour(db, bucketStart, new Date(generatedAt))
return { orgId, userId, bucketStart, eventMs: now, eventSec: nowSec }
}
@@ -1997,6 +1997,17 @@ describe('Download tasks API integration', () => {
runtime: { phase: 'completed' },
},
})
const terminalFacts = await db.all<{ outcome: string; count: number }>(sql`
SELECT json_extract(metadata, '$.outcome') AS outcome, COUNT(*) AS count
FROM activity_events
WHERE action = 'stats_remote_download_finished' AND target_id = ${createdTask.id}
GROUP BY outcome
ORDER BY outcome
`)
expect(terminalFacts).toEqual([
{ outcome: 'completed', count: 1 },
{ outcome: 'failed', count: 1 },
])
const seedingAfterRestartRes = await app.request(`/api/downloads/tasks/${createdTask.id}`, {
method: 'PATCH',
+19 -13
View File
@@ -1381,6 +1381,7 @@ describe('POST /api/objects/:id/transfers', () => {
const userId = await getUserIdByEmail(db, 'test@example.com')
await insertTeamOrg(db, 'team-a')
await insertMember(db, 'team-a', userId, 'editor')
await insertStorageEntitlement(db, 'team-a', 10_000_000)
await insertFile(db, orgId, { id: 'src-copy', name: 'doc.txt' })
const res = await transferRequest(app, headers, 'src-copy', { targetOrgId: 'team-a', mode: 'copy' })
@@ -1403,10 +1404,12 @@ describe('POST /api/objects/:id/transfers', () => {
const userId = await getUserIdByEmail(db, 'test@example.com')
await insertTeamOrg(db, 'team-b')
await insertMember(db, 'team-b', userId, 'owner')
await insertStorageEntitlement(db, 'team-b', 10_000_000)
await insertFile(db, orgId, { id: 'src-move', name: 'photo.jpg', size: 1024 })
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES (${`q-${orgId}`}, ${orgId}, ${1024 * 1024}, 1024, 0, 0, '1970-01')
UPDATE org_quotas
SET quota = ${1024 * 1024}, used = 1024, traffic_quota = 0, traffic_used = 0, traffic_period = '1970-01'
WHERE org_id = ${orgId}
`)
const res = await transferRequest(app, headers, 'src-move', { targetOrgId: 'team-b', mode: 'move' })
@@ -1430,6 +1433,7 @@ describe('POST /api/objects/:id/transfers', () => {
const userId = await getUserIdByEmail(db, 'test@example.com')
await insertTeamOrg(db, 'team-c')
await insertMember(db, 'team-c', userId, 'editor')
await insertStorageEntitlement(db, 'team-c', 10_000_000)
await insertFolder(db, orgId, { id: 'fold-1', name: 'Album' })
await insertFile(db, orgId, { id: 'in-fold', name: 'pic.png', parent: 'Album' })
@@ -1738,12 +1742,13 @@ describe('Objects API — quota enforcement', () => {
expect(quotaRows[0].used).toBe(500) // unchanged
})
it('returns 201 when no quota row exists (unlimited)', async () => {
it('returns 422 when no quota row or entitlement exists', async () => {
const { app, db } = await createTestApp()
const headers = await authedHeaders(app)
await insertStorage(db)
const orgId = await getOrgId(db)
// No org quota row at all — unlimited
await db.delete(orgQuotaEntitlements).where(eq(orgQuotaEntitlements.orgId, orgId))
await db.delete(orgQuotas).where(eq(orgQuotas.orgId, orgId))
await insertFile(db, orgId, { id: 'm-copy-nolimit', name: 'nolimit.txt', size: 100 })
const res = await app.request('/api/objects/m-copy-nolimit/copies', {
@@ -1751,10 +1756,10 @@ describe('Objects API — quota enforcement', () => {
headers: { ...headers, 'Content-Type': 'application/json' },
body: JSON.stringify({ parent: '' }),
})
expect(res.status).toBe(201)
expect(res.status).toBe(422)
})
it('returns 201 when quota is 0 (unlimited) regardless of file size', async () => {
it('returns 422 when effective quota is zero', async () => {
const { app, db } = await createTestApp()
const headers = await authedHeaders(app)
await insertStorage(db)
@@ -1767,7 +1772,7 @@ describe('Objects API — quota enforcement', () => {
headers: { ...headers, 'Content-Type': 'application/json' },
body: JSON.stringify({ parent: '' }),
})
expect(res.status).toBe(201)
expect(res.status).toBe(422)
})
it('returns 404 when source file does not exist', async () => {
@@ -2028,7 +2033,7 @@ describe('Objects API — quota enforcement', () => {
expect(quotaRows[0]).toEqual({ used: 140, quota: 100 })
})
it('enforces storage entitlements when base quota is unlimited', async () => {
it('enforces storage entitlements when the legacy quota column is zero', async () => {
const { app, db } = await createTestApp()
await seedProLicense(db)
const headers = await authedHeaders(app)
@@ -2092,17 +2097,18 @@ describe('Objects API — quota enforcement', () => {
expect(quotaRows[0].used).toBe(50)
})
it('returns 200 when no quota row exists (unlimited)', async () => {
it('returns 422 when no quota row or entitlement exists', async () => {
const { app, db } = await createTestApp()
const headers = await authedHeaders(app)
await insertStorage(db)
// No quota row — unlimited
const orgId = await getOrgId(db)
await db.delete(orgQuotaEntitlements).where(eq(orgQuotaEntitlements.orgId, orgId))
await db.delete(orgQuotas).where(eq(orgQuotas.orgId, orgId))
const ref = await createDraft(app, headers, { name: 'nolimit.txt', size: 5000 })
const res = await complete(app, headers, ref)
expect(res.status).toBe(200)
const body = (await res.json()) as Record<string, unknown>
expect(body.status).toBe('active')
expect(res.status).toBe(422)
await expect(res.json()).resolves.toMatchObject({ error: { details: [{ reason: 'QUOTA_EXCEEDED' }] } })
})
it('replaces a same-size file at full quota — net-neutral, incumbent purged', async () => {
@@ -336,7 +336,7 @@ async function insertStorage(
}
describe('selectStorage service', () => {
it('returns the single active storage when capacity is unlimited (0) [spec: storages/select-active]', async () => {
it('returns the single active storage when capacity is not reported (0) [spec: storages/select-active]', async () => {
const { db } = await createTestApp()
await insertStorage(db, { id: 's1', capacity: 0, used: 0 })
@@ -1377,6 +1377,35 @@ describe('Quota Store API', () => {
trafficExtraNames: [],
currentPlan: { trafficOveragePriceCents: 25 },
})
const freeBaselines = await db.all<{ resourceType: string; status: string }>(sql`
SELECT resource_type AS resourceType, status
FROM org_quota_entitlements
WHERE org_id = ${orgId} AND source = 'free_plan'
ORDER BY resource_type
`)
expect(freeBaselines).toEqual([
{ resourceType: 'storage', status: 'active' },
{ resourceType: 'traffic', status: 'active' },
])
const revoked = await postWebhook(
app,
JSON.stringify({
eventId: 'evt-subscription-revoked',
cloudOrderId: subscriptionSourceId,
targetOrgId: orgId,
eventType: 'order.quota_changed',
direction: 'decrease',
storageBytes: 4096,
trafficBytes: 2048,
source: 'stripe_subscription',
}),
)
expect(revoked.status).toBe(200)
const fallbackRes = await app.request('/api/quotas/me', { headers })
const fallback = (await fallbackRes.json()) as { baseQuota: number; quota: number; storagePlanName: string }
expect(fallback).toMatchObject({ baseQuota: 10 * 1024 * 1024, quota: 10 * 1024 * 1024, storagePlanName: 'Free' })
})
it('renews subscription entitlements by replacing plan bytes and extending expiry [spec: quota-store/webhook-renewal]', async () => {
+9 -1
View File
@@ -12,7 +12,13 @@ vi.mock('../server/platform/cloudflare', () => ({
}))
const refreshHourlyRollups = vi.fn()
const fakeDeps = { instance: 'instance', systemOptions: 'system-options', adminStats: { refreshHourlyRollups } }
const reconcileFreePlanBaselines = vi.fn()
const fakeDeps = {
instance: 'instance',
systemOptions: 'system-options',
adminStats: { refreshHourlyRollups },
quota: { reconcileFreePlanBaselines },
}
vi.mock('../server/composition', () => ({
createDeps: vi.fn(() => fakeDeps),
}))
@@ -54,6 +60,7 @@ describe('handleScheduled', () => {
vi.mocked(runLicensingRefresh).mockReset()
mockResetExpiredTrafficQuotas.mockReset()
refreshHourlyRollups.mockReset()
reconcileFreePlanBaselines.mockReset()
})
it('syncs usage reports on the traffic cron only', async () => {
@@ -64,6 +71,7 @@ describe('handleScheduled', () => {
cloudBaseUrl: 'https://cloud.example',
})
expect(refreshHourlyRollups).toHaveBeenCalledOnce()
expect(reconcileFreePlanBaselines).toHaveBeenCalledOnce()
expect(runLicensingRefresh).not.toHaveBeenCalled()
expect(reportInstanceTelemetry).not.toHaveBeenCalled()
})
+64 -17
View File
@@ -17,6 +17,8 @@ describe('admin stats backfill', () => {
const now = new Date('2026-07-10T12:00:00.000Z')
const historyStartMs = Date.parse('2026-04-01T00:10:00.000Z')
const eventMs = Date.parse('2026-07-10T09:10:00.000Z')
const eventHourMs = Date.parse('2026-07-10T09:00:00.000Z')
const snapshotObservedAt = '2026-07-10T09:50:00.000Z'
const eventSec = Math.floor(eventMs / 1000)
const currentHourMs = Date.parse('2026-07-10T12:00:00.000Z')
const currentEventSec = Math.floor(Date.parse('2026-07-10T12:10:00.000Z') / 1000)
@@ -26,11 +28,14 @@ describe('admin stats backfill', () => {
CREATE TABLE user (id TEXT PRIMARY KEY, created_at INTEGER NOT NULL DEFAULT 0);
CREATE TABLE account (id TEXT PRIMARY KEY, user_id TEXT NOT NULL, provider_id TEXT NOT NULL, created_at INTEGER NOT NULL);
CREATE TABLE session (id TEXT PRIMARY KEY, user_id TEXT NOT NULL, created_at INTEGER NOT NULL);
CREATE TABLE organization (id TEXT PRIMARY KEY, created_at INTEGER NOT NULL);
CREATE TABLE organization (id TEXT PRIMARY KEY, metadata TEXT, created_at INTEGER NOT NULL);
CREATE TABLE member (
id TEXT PRIMARY KEY, organization_id TEXT NOT NULL, user_id TEXT NOT NULL, created_at INTEGER NOT NULL
);
CREATE TABLE matters (id TEXT PRIMARY KEY, size INTEGER, dirtype INTEGER);
CREATE TABLE shares (
id TEXT PRIMARY KEY, kind TEXT NOT NULL, matter_id TEXT NOT NULL, org_id TEXT NOT NULL,
status TEXT NOT NULL, expires_at INTEGER, download_limit INTEGER, downloads INTEGER NOT NULL,
status TEXT NOT NULL, expires_at INTEGER, download_limit INTEGER, views INTEGER NOT NULL, downloads INTEGER NOT NULL,
created_at INTEGER NOT NULL
);
CREATE TABLE activity_events (
@@ -69,20 +74,35 @@ describe('admin stats backfill', () => {
UNIQUE(bucket_start, org_id, metric_key, dimension_key, dimension_value)
);
INSERT INTO user VALUES ('u0', 0), ('u1', ${historyStartMs});
INSERT INTO account VALUES ('a1', 'u1', 'github', ${historyStartMs});
INSERT INTO user VALUES ('u0', 0), ('u1', ${historyStartMs}), ('u2', ${historyStartMs + 1000});
INSERT INTO account VALUES
('a1', 'u1', 'github', ${historyStartMs}),
('a2', 'u2', 'github', ${historyStartMs + 1000});
INSERT INTO organization VALUES
('o1', '{"type":"personal"}', ${historyStartMs}),
('o2', '{"type":"personal"}', ${historyStartMs + 1000});
INSERT INTO member VALUES
('m1', 'o1', 'u1', ${historyStartMs}),
('m2', 'o2', 'u2', ${historyStartMs + 1000});
INSERT INTO matters VALUES ('f1', 512, 0);
INSERT INTO shares VALUES ('s1', 'landing', 'f1', 'o1', 'active', NULL, 10, 1, ${eventSec});
INSERT INTO shares VALUES ('s1', 'landing', 'f1', 'o1', 'active', NULL, 10, 0, 1, ${eventSec});
INSERT INTO activity_events VALUES
('upload-1', 'o1', 'u1', NULL, NULL, 'upload_confirm', 'file', 'f1', 'file.bin', NULL, ${eventSec}),
('open-upload', 'o1', 'u1', 'user', NULL, 'upload_confirm', 'file', 'f1', 'file.bin',
'{"bytes":512,"source":"upload","status":"success"}', ${currentEventSec}),
('share-1', 'o1', NULL, NULL, NULL, 'share_download', 'share', 's1', 'file.bin', '{"anonymous":true}', ${eventSec}),
('image-1', 'o1', NULL, NULL, NULL, 'image_hosting_download', 'image', 'img1', 'image.png', NULL, ${eventSec});
('image-1', 'o1', NULL, NULL, NULL, 'image_hosting_download', 'image', 'img1', 'image.png', NULL, ${eventSec}),
('task-failed', 'o1', 'u1', 'user', NULL, 'download_task_failed', 'remote_download', 't1', 'task', NULL, ${eventSec + 1}),
('task-completed', 'o1', 'u1', 'user', NULL, 'download_task_completed', 'remote_download', 't1', 'task', NULL, ${eventSec + 2}),
('blocked-download', 'o1', 'u1', 'user', NULL, 'download_failed', 'file', 'f1', 'file.bin',
'{"bytes":512,"source":"object_download","reason":"quota_exceeded","trafficEventId":"traffic-3"}', ${eventSec});
INSERT INTO cloud_traffic_reports VALUES
('r1', 'o1', 'direct_share', 's1', 'traffic-1', 512, NULL, NULL, NULL, 'reported', NULL, ${eventMs}, ${eventMs}),
('r2', 'o1', 'image_hosting', 'img1', 'traffic-2', 128, NULL, NULL, NULL, 'reported', NULL, ${eventMs}, ${eventMs});
INSERT INTO download_tasks VALUES ('t1', 'o1', 'video', 'url', 'd1', 'completed', 512, ${eventMs}, ${eventMs});
('r2', 'o1', 'image_hosting', 'img1', 'traffic-2', 128, NULL, NULL, NULL, 'reported', NULL, ${eventMs}, ${eventMs}),
('r3', 'o1', 'object_download', 'f1', 'traffic-3', 512, NULL, NULL, NULL, 'blocked', 'quota_exceeded', ${eventMs}, ${eventMs});
INSERT INTO download_tasks VALUES
('t1', 'o1', 'video', 'url', 'd1', 'completed', 512, ${eventMs}, ${eventMs}),
('t2', 'o1', NULL, 'url', NULL, 'canceled', 0, ${eventMs}, ${eventMs});
INSERT INTO org_quotas VALUES ('q1', 512);
INSERT INTO stats_rollups_hourly VALUES
('legacy-epoch', 0, '', 'stats.rollup_run', '', '', 1, 0, 0,
@@ -94,7 +114,11 @@ describe('admin stats backfill', () => {
('stale-task', ${eventMs - 3_600_000}, 'o1', 'remote_download.task_finished', '', '', 1, 512, 0,
'{"version":2,"scope":"counters","quality":"exact"}', ${eventMs}),
('stale-traffic', ${eventMs - 3_600_000}, 'o1', 'traffic.report_sync', '', '', 2, 640, 0,
'{"version":2,"scope":"counters","quality":"exact"}', ${eventMs});
'{"version":2,"scope":"counters","quality":"exact"}', ${eventMs}),
('snapshot-marker', ${eventHourMs}, '', 'stats.rollup_run', '', '', 1, 0, 0,
'{"version":3,"scope":"snapshots","quality":"exact","snapshotQuality":"exact","snapshotObservedAt":"${snapshotObservedAt}"}', ${eventMs}),
('snapshot-gauge', ${eventHourMs}, '', 'storage.used', '', '', 0, 512, 0,
'{"version":3,"scope":"snapshots","quality":"exact","observedAt":"${snapshotObservedAt}"}', ${eventMs});
`)
const sql = buildBackfillSql(now)
@@ -127,7 +151,7 @@ describe('admin stats backfill', () => {
orphanUserEvents: 0,
missingUploadBytes: 0,
missingDownloadBytes: 0,
trafficEvents: 2,
trafficEvents: 3,
hourlyRollups: expectedBuckets,
rawActiveShares: 1,
validActiveShares: 1,
@@ -138,14 +162,16 @@ describe('admin stats backfill', () => {
openCounterMarkers: 0,
rawUploadAttempts: 1,
rollupUploadAttempts: 1,
rawUserSignups: 1,
rollupUserSignups: 1,
rawUserSignups: 2,
rollupUserSignups: 2,
rawSharesCreated: 1,
rollupSharesCreated: 1,
rawFailedDownloads: 1,
rollupFailedDownloads: 1,
rawShareDownloads: 1,
rollupShareDownloads: 1,
rawFinishedDownloadTasks: 1,
rollupFinishedDownloadTasks: 1,
rawFinishedDownloadTasks: 3,
rollupFinishedDownloadTasks: 3,
rawMissingByteEvents: 0,
rollupMissingByteEvents: 0,
})
@@ -155,11 +181,21 @@ describe('admin stats backfill', () => {
expect(db.prepare("SELECT COUNT(*) AS value FROM activity_events WHERE target_id = 'img1'").get()).toEqual({
value: 1,
})
expect(db.prepare("SELECT COUNT(*) AS value FROM activity_events WHERE id = 'backfill_traffic-3'").get()).toEqual({
value: 0,
})
expect(
db
.prepare("SELECT json_extract(metadata, '$.scope') AS scope FROM stats_rollups_hourly WHERE id = 'latest-full'")
.prepare("SELECT COUNT(*) AS value FROM stats_rollups_hourly WHERE json_extract(metadata, '$.scope') = 'full'")
.get(),
).toEqual({ scope: 'full' })
).toEqual({ value: 1 })
expect(
db
.prepare(
"SELECT json_extract(metadata, '$.counterQuality') AS counterQuality, json_extract(metadata, '$.snapshotQuality') AS snapshotQuality, json_extract(metadata, '$.snapshotObservedAt') AS snapshotObservedAt FROM stats_rollups_hourly WHERE id = 'snapshot-marker'",
)
.get(),
).toEqual({ counterQuality: 'lower_bound', snapshotQuality: 'exact', snapshotObservedAt })
expect(db.prepare('SELECT COUNT(*) AS value FROM stats_rollups_hourly WHERE bucket_start = 0').get()).toEqual({
value: 0,
})
@@ -172,12 +208,23 @@ describe('admin stats backfill', () => {
"SELECT count AS value FROM stats_rollups_hourly WHERE metric_key = 'user.signup' AND dimension_key = 'provider' AND dimension_value = 'github'",
)
.get(),
).toEqual({ value: 1 })
).toEqual({ value: 2 })
expect(
db.prepare("SELECT COUNT(*) AS value FROM stats_rollups_hourly WHERE metric_key = 'traffic.report_sync'").get(),
).toEqual({
value: 0,
})
expect(
db
.prepare(
"SELECT json_extract(metadata, '$.outcome') AS outcome, json_extract(metadata, '$.bytes') AS bytes, COUNT(*) AS value FROM activity_events WHERE action = 'stats_remote_download_finished' GROUP BY outcome, bytes ORDER BY outcome",
)
.all(),
).toEqual([
{ outcome: 'canceled', bytes: 0, value: 1 },
{ outcome: 'completed', bytes: 512, value: 1 },
{ outcome: 'failed', bytes: 0, value: 1 },
])
db.close()
})
})
+5 -1
View File
@@ -186,6 +186,7 @@ const APP_SCHEMA_SQL = `
traffic_used INTEGER NOT NULL DEFAULT 0,
traffic_period TEXT NOT NULL DEFAULT '1970-01'
);
CREATE UNIQUE INDEX IF NOT EXISTS org_quotas_org_uniq ON org_quotas(org_id);
CREATE TABLE IF NOT EXISTS cloud_traffic_reports (
id TEXT PRIMARY KEY,
org_id TEXT NOT NULL,
@@ -199,12 +200,15 @@ const APP_SCHEMA_SQL = `
credits_per_unit INTEGER,
status TEXT NOT NULL,
error TEXT,
attempt_count INTEGER NOT NULL DEFAULT 0,
next_retry_at INTEGER,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE UNIQUE INDEX IF NOT EXISTS cloud_traffic_reports_event_uniq ON cloud_traffic_reports(event_id);
CREATE INDEX IF NOT EXISTS cloud_traffic_reports_org_period_idx ON cloud_traffic_reports(org_id, period);
CREATE INDEX IF NOT EXISTS cloud_traffic_reports_status_idx ON cloud_traffic_reports(status);
CREATE INDEX IF NOT EXISTS cloud_traffic_reports_retry_idx ON cloud_traffic_reports(status, next_retry_at, created_at);
CREATE TABLE IF NOT EXISTS org_quota_entitlements (
id TEXT PRIMARY KEY,
org_id TEXT NOT NULL,
@@ -226,7 +230,7 @@ const APP_SCHEMA_SQL = `
ON org_quota_entitlements(org_id, resource_type, entitlement_type, status);
CREATE UNIQUE INDEX IF NOT EXISTS org_quota_entitlements_active_plan_uniq
ON org_quota_entitlements(org_id, resource_type, entitlement_type)
WHERE status = 'active' AND entitlement_type = 'plan';
WHERE status = 'active' AND entitlement_type = 'plan' AND source <> 'free_plan';
CREATE UNIQUE INDEX IF NOT EXISTS org_quota_entitlements_source_resource_uniq
ON org_quota_entitlements(source, source_id, resource_type);
CREATE TABLE IF NOT EXISTS webhook_events (
+7 -10
View File
@@ -496,11 +496,7 @@ describe('archive processing', () => {
it('fails quota checks without creating visible extracted output', async () => {
const { db } = await createTestApp()
await seedStorage(db)
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES ('archive-quota', ${ORG_ID}, 0, 0, 0, 0, '1970-01')
`)
await seedStoragePlanEntitlement(db, ORG_ID, 4, 'archive-quota-plan')
await db.run(sql`UPDATE org_quota_entitlements SET bytes = 4 WHERE id = 'archive-default-quota-plan'`)
await seedMatter(db, { id: 'quota-zip', name: 'quota.zip', object: 'source/quota.zip', size: 200 })
const s3 = new MemoryS3()
@@ -522,11 +518,7 @@ describe('archive processing', () => {
it('fails compression quota checks and removes streamed output', async () => {
const { db } = await createTestApp()
await seedStorage(db)
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES ('archive-compress-quota', ${ORG_ID}, 0, 0, 0, 0, '1970-01')
`)
await seedStoragePlanEntitlement(db, ORG_ID, 4, 'archive-compress-quota-plan')
await db.run(sql`UPDATE org_quota_entitlements SET bytes = 4 WHERE id = 'archive-default-quota-plan'`)
await seedMatter(db, { id: 'file-a', name: 'a.txt', object: 'objects/a.txt', size: 5 })
const s3 = new MemoryS3()
@@ -833,6 +825,11 @@ async function seedStorage(db: TestDb): Promise<void> {
INSERT INTO storages (id, bucket, endpoint, region, access_key, secret_key, file_path, custom_host, capacity, used, status, created_at, updated_at)
VALUES (${STORAGE_ID}, 'bucket', 'https://s3.example.com', 'auto', 'ak', 'sk', '', '', 0, 0, 'active', ${now}, ${now})
`)
await db.run(sql`
INSERT INTO org_quotas (id, org_id, quota, used, traffic_quota, traffic_used, traffic_period)
VALUES ('archive-default-quota', ${ORG_ID}, 0, 0, 0, 0, '1970-01')
`)
await seedStoragePlanEntitlement(db, ORG_ID, 10 * 1024 ** 3, 'archive-default-quota-plan')
}
async function seedMatter(
+24 -1
View File
@@ -10,7 +10,7 @@ import { createQuotaRepo } from '../adapters/repos/quota'
import { createShareRepo } from '../adapters/repos/share'
import { createStorageRepo } from '../adapters/repos/storage'
import { createStorageUsageRepo } from '../adapters/repos/storage-usage'
import { matters } from '../db/schema'
import { matters, orgQuotaEntitlements, orgQuotas } from '../db/schema'
import { createCloudflarePlatform } from '../platform/cloudflare'
import type { Database } from '../platform/interface'
import { type SaveShareInput, type SaveToDriveDeps, saveShareToDrive as saveShareToDriveUseCase } from './object'
@@ -44,6 +44,26 @@ async function seedStorage(db: ReturnType<typeof buildDb>, id: string) {
)
}
async function seedStorageQuota(db: ReturnType<typeof buildDb>, orgId: string, bytes = 10_000_000) {
const now = new Date()
await db.insert(orgQuotas).values({ id: nanoid(), orgId, quota: bytes })
await db.insert(orgQuotaEntitlements).values({
id: nanoid(),
orgId,
resourceType: 'storage',
entitlementType: 'plan',
source: 'free_plan',
sourceId: `free_plan:${orgId}`,
bytes,
startsAt: now,
expiresAt: null,
status: 'active',
metadata: JSON.stringify({ packageName: 'Free', source: 'free_plan' }),
createdAt: now,
updatedAt: now,
})
}
async function seedMatter(db: ReturnType<typeof buildDb>, orgId: string, dirtype = DirType.FILE) {
const now = new Date()
const matter = {
@@ -129,6 +149,7 @@ describe('[CF] saveShareToDrive — stream copy via D1', () => {
const srcOrgId = `src-${nanoid(6)}`
const dstOrgId = `dst-${nanoid(6)}`
await seedStorage(db, 'cf-storage-1')
await seedStorageQuota(db, dstOrgId)
const matter = await seedMatter(db, srcOrgId)
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'cf-u1', kind: 'landing' })
@@ -154,6 +175,7 @@ describe('[CF] saveShareToDrive — stream copy via D1', () => {
const srcOrgId = `src-${nanoid(6)}`
const dstOrgId = `dst-${nanoid(6)}`
await seedStorage(db, 'cf-storage-1')
await seedStorageQuota(db, dstOrgId)
const matter = await seedMatter(db, srcOrgId)
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'cf-u3', kind: 'landing' })
@@ -184,6 +206,7 @@ describe('[CF] saveShareToDrive — stream copy via D1', () => {
const srcOrgId = `src-${nanoid(6)}`
const dstOrgId = `dst-${nanoid(6)}`
await seedStorage(db, 'cf-storage-1')
await seedStorageQuota(db, dstOrgId)
// Create folder structure
const now = new Date()
+10 -4
View File
@@ -228,17 +228,17 @@ describe('computeSourceBytes', () => {
// ─── isQuotaSufficient ────────────────────────────────────────────────────────
describe('isQuotaSufficient', () => {
it('returns true when no quota row exists (unlimited)', async () => {
it('returns false when no quota row exists', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(true)
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(false)
})
it('returns true when quota is 0 (unlimited)', async () => {
it('returns false when effective quota is zero', async () => {
const { db } = await createTestApp()
const orgId = nanoid()
await seedOrgQuota(db, orgId, 0)
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(true)
expect(await isQuotaSufficient(db, orgId, 9999999)).toBe(false)
})
it('returns true when bytes fit within quota', async () => {
@@ -270,6 +270,7 @@ describe('saveShareToDrive', () => {
await insertStorage(db)
const srcOrgId = nanoid()
const dstOrgId = nanoid()
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
const matter = await seedMatter(db, { orgId: srcOrgId, size: 2048 })
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
@@ -300,6 +301,7 @@ describe('saveShareToDrive', () => {
const srcOrgId = nanoid()
const dstOrgId = nanoid()
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
const matter = await seedMatter(db, { orgId: srcOrgId, storageId: STORAGE_ID, size: 1024 })
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
@@ -324,6 +326,7 @@ describe('saveShareToDrive', () => {
await insertStorage(db)
const srcOrgId = nanoid()
const dstOrgId = nanoid()
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
const fileName = `photo-${nanoid(4)}.pdf`
const matter = await seedMatter(db, { orgId: srcOrgId, name: fileName, size: 1024 })
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
@@ -352,6 +355,7 @@ describe('saveShareToDrive', () => {
await insertStorage(db)
const srcOrgId = nanoid()
const dstOrgId = nanoid()
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
const matter = await seedMatter(db, { orgId: srcOrgId, size: 512 })
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
@@ -379,6 +383,7 @@ describe('saveShareToDrive', () => {
await insertStorage(db)
const srcOrgId = nanoid()
const dstOrgId = nanoid()
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
const matter = await seedMatter(db, { orgId: srcOrgId, size: 512 })
const share = await createShare(db, { matterId: matter.id, orgId: srcOrgId, creatorId: 'u1', kind: 'landing' })
@@ -461,6 +466,7 @@ describe('saveShareToDrive', () => {
await insertStorage(db)
const srcOrgId = nanoid()
const dstOrgId = nanoid()
await seedOrgQuota(db, dstOrgId, 1_000_000_000)
// Build source tree: Photos/ → [img1.jpg, img2.jpg, Sub/ → [img3.jpg]]
const folder = await seedMatter(db, { orgId: srcOrgId, dirtype: DirType.USER_FOLDER, name: 'Photos', size: 0 })
+11 -3
View File
@@ -6,7 +6,7 @@ export type TrafficReportSource =
| 'custom_domain_image'
| 'webdav_download'
export type CloudTrafficReportStatus = 'pending' | 'reported' | 'skipped_unbound' | 'blocked' | 'failed'
export type CloudTrafficReportStatus = 'pending' | 'reported' | 'skipped_unbound' | 'blocked' | 'failed' | 'dead_letter'
export interface CloudTrafficReportRecord {
id: string
@@ -21,6 +21,8 @@ export interface CloudTrafficReportRecord {
creditsPerUnit: number | null
status: CloudTrafficReportStatus
error: string | null
attemptCount: number
nextRetryAt: Date | null
createdAt: Date
updatedAt: Date
}
@@ -42,6 +44,12 @@ export interface InsertCloudTrafficReportInput {
export interface CloudTrafficReportRepo {
findByEventId(eventId: string): Promise<CloudTrafficReportRecord | undefined>
insert(input: InsertCloudTrafficReportInput): Promise<void>
updateStatus(eventId: string, status: CloudTrafficReportStatus, error: string | null, now: Date): Promise<void>
listPending(limit: number): Promise<CloudTrafficReportRecord[]>
updateStatus(
eventId: string,
status: CloudTrafficReportStatus,
error: string | null,
now: Date,
retry?: { attemptCount: number; nextRetryAt: Date | null },
): Promise<void>
listPending(limit: number, now: Date): Promise<CloudTrafficReportRecord[]>
}
+1
View File
@@ -38,6 +38,7 @@ export interface QuotaRepo {
listOrgQuotaOverview(): Promise<OrgQuotaOverviewRow[]>
getEffectiveQuota(orgId: string, now?: Date): Promise<EffectiveQuota>
getEffectiveQuotasByOrg(orgIds: string[], now?: Date): Promise<Map<string, EffectiveQuota>>
reconcileFreePlanBaselines(now?: Date): Promise<void>
resetExpiredTrafficQuotas(now?: Date): Promise<void>
hasQuotaForBytes(orgId: string, bytes: number): Promise<boolean>
hasTrafficQuotaForBytes(orgId: string, bytes: number, now?: Date): Promise<boolean>
+65 -2
View File
@@ -145,7 +145,7 @@ describe('cloud traffic metering', () => {
const result = await syncPendingCloudTrafficReports({ db, cloudBaseUrl: 'https://cloud.example' })
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0 })
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 })
expect(fetch).toHaveBeenCalledTimes(1)
const [url, init] = vi.mocked(fetch).mock.calls[0] as [string, RequestInit]
expect(url).toBe('https://cloud.example/api/stores/store-test-binding/billing/usage-events')
@@ -297,6 +297,7 @@ describe('cloud traffic metering', () => {
reported: 1,
blocked: 0,
failed: 0,
deadLetter: 0,
})
const result = await syncPendingCloudTrafficReports({
@@ -305,13 +306,75 @@ describe('cloud traffic metering', () => {
now: new Date('2026-05-01T00:01:00.000Z'),
})
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0 })
expect(result).toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 })
expect(fetch).toHaveBeenCalledTimes(2)
await expect(db.select().from(cloudTrafficReports)).resolves.toMatchObject([
{ status: 'reported', error: null, period: '2026-04' },
])
})
it('selects new pending reports before retryable failures', async () => {
const { db } = await createTestApp()
const repo = createCloudTrafficReportRepo(db)
const now = new Date('2026-05-01T00:00:00.000Z')
const input = {
orgId: 'org_1',
period: '2026-05',
source: 'object_download' as const,
sourceId: 'matter_1',
bytes: 1024,
storageId: 'storage_1',
unitBytes: 1024,
creditsPerUnit: 1,
status: 'pending' as const,
}
await repo.insert({ ...input, eventId: 'failed-old', now: new Date(now.getTime() - 60_000) })
await repo.updateStatus('failed-old', 'failed', 'offline', now, {
attemptCount: 1,
nextRetryAt: new Date(now.getTime() - 1),
})
await repo.insert({ ...input, eventId: 'pending-new', now })
await expect(repo.listPending(1, now)).resolves.toMatchObject([{ eventId: 'pending-new', status: 'pending' }])
})
it('dead-letters terminal idempotency conflicts without retrying them', async () => {
const { db, platform } = await createTestApp({ ZPAN_CLOUD_URL: 'https://cloud.example' })
await seedTrafficBinding(db)
vi.stubGlobal('fetch', vi.fn().mockRejectedValue(new Error('usage_idempotency_conflict')))
const now = new Date('2026-05-01T00:00:00.000Z')
await expect(
reportTrafficEgress({
platform,
orgId: 'org_1',
bytes: 1024,
source: 'object_download',
sourceId: 'matter_1',
eventId: 'evt_terminal',
now,
...meteredStorage,
}),
).resolves.toMatchObject({ status: 'dead_letter', eventId: 'evt_terminal' })
await expect(db.select().from(cloudTrafficReports)).resolves.toMatchObject([
{
eventId: 'evt_terminal',
status: 'dead_letter',
error: 'usage_idempotency_conflict',
attemptCount: 1,
nextRetryAt: null,
},
])
await expect(
syncPendingCloudTrafficReports({
db,
cloudBaseUrl: 'https://cloud.example',
now: new Date(now.getTime() + 60_000),
}),
).resolves.toEqual({ attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 })
expect(fetch).toHaveBeenCalledTimes(1)
})
it('marks reports reported when Cloud returns its own usage event id', async () => {
const { db, platform } = await createTestApp({ ZPAN_CLOUD_URL: 'https://cloud.example' })
await seedTrafficBinding(db)
+31 -11
View File
@@ -34,6 +34,9 @@ const usageResponseSchema = z.object({
eventId: z.string().min(1),
})
const MAX_TRAFFIC_REPORT_ATTEMPTS = 8
const TERMINAL_TRAFFIC_REPORT_ERRORS = new Set(['usage_idempotency_conflict'])
export async function reportTrafficEgress(
deps: CloudTrafficMeteringDeps,
params: {
@@ -114,16 +117,17 @@ export async function reportTrafficEgress(
export async function syncPendingCloudTrafficReports(
deps: CloudTrafficMeteringDeps,
params: { cloudBaseUrl: string; limit?: number; now?: Date },
): Promise<{ attempted: number; reported: number; blocked: number; failed: number }> {
): Promise<{ attempted: number; reported: number; blocked: number; failed: number; deadLetter: number }> {
const { cloudBaseUrl, limit = 100, now = new Date() } = params
if (!hasFeature('quota_store', await loadBindingState(deps)))
return { attempted: 0, reported: 0, blocked: 0, failed: 0 }
return { attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 }
const binding = await deps.licenseBinding.loadActiveLicenseBinding()
if (!binding?.refreshToken || !binding.cloudStoreId) return { attempted: 0, reported: 0, blocked: 0, failed: 0 }
if (!binding?.refreshToken || !binding.cloudStoreId)
return { attempted: 0, reported: 0, blocked: 0, failed: 0, deadLetter: 0 }
const reports = await deps.cloudTrafficReports.listPending(limit)
const reports = await deps.cloudTrafficReports.listPending(limit, now)
const result = { attempted: reports.length, reported: 0, blocked: 0, failed: 0 }
const result = { attempted: reports.length, reported: 0, blocked: 0, failed: 0, deadLetter: 0 }
for (const report of reports) {
const status = await syncTrafficReport(deps, {
cloudBaseUrl,
@@ -132,7 +136,7 @@ export async function syncPendingCloudTrafficReports(
report,
now,
})
result[status] += 1
result[status === 'dead_letter' ? 'deadLetter' : status] += 1
}
return result
}
@@ -146,7 +150,7 @@ async function syncTrafficReport(
report: CloudTrafficReportRecord
now: Date
},
): Promise<'reported' | 'blocked' | 'failed'> {
): Promise<'reported' | 'blocked' | 'failed' | 'dead_letter'> {
const { cloudBaseUrl, refreshToken, storeId, report, now } = params
try {
const client = deps.licensingCloud.createBoundCloudClient(cloudBaseUrl, refreshToken)
@@ -179,19 +183,35 @@ async function syncTrafficReport(
usageResponseSchema,
)
if (!response.accepted) throw new Error('cloud_usage_report_rejected')
await deps.cloudTrafficReports.updateStatus(report.eventId, 'reported', null, now)
await deps.cloudTrafficReports.updateStatus(report.eventId, 'reported', null, now, {
attemptCount: report.attemptCount + 1,
nextRetryAt: null,
})
return 'reported'
} catch (error) {
const message = error instanceof Error ? error.message : 'cloud_usage_report_failed'
if (message === 'insufficient_credits' || message === 'overage_cap_exceeded') {
await deps.cloudTrafficReports.updateStatus(report.eventId, 'blocked', message, now)
await deps.cloudTrafficReports.updateStatus(report.eventId, 'blocked', message, now, {
attemptCount: report.attemptCount + 1,
nextRetryAt: null,
})
return 'blocked'
}
await deps.cloudTrafficReports.updateStatus(report.eventId, 'failed', message, now)
return 'failed'
const attemptCount = report.attemptCount + 1
const terminal = TERMINAL_TRAFFIC_REPORT_ERRORS.has(message) || attemptCount >= MAX_TRAFFIC_REPORT_ATTEMPTS
await deps.cloudTrafficReports.updateStatus(report.eventId, terminal ? 'dead_letter' : 'failed', message, now, {
attemptCount,
nextRetryAt: terminal ? null : nextTrafficReportRetryAt(now, attemptCount),
})
return terminal ? 'dead_letter' : 'failed'
}
}
function nextTrafficReportRetryAt(now: Date, attemptCount: number): Date {
const delayMinutes = Math.min(360, 2 ** Math.min(attemptCount - 1, 8))
return new Date(now.getTime() + delayMinutes * 60_000)
}
function assertSameReport(
report: CloudTrafficReportRecord,
params: {
+50 -30
View File
@@ -4,7 +4,7 @@ export interface AdminUsageBySpace {
orgType: string
usedBytes: number
quotaBytes: number
utilization: number
utilization: number | null
}
export interface AdminTopShare {
@@ -25,19 +25,23 @@ export interface AdminStatsRange {
timeZone: 'UTC'
coverage: AdminStatsCoverage
comparisonCoverage?: AdminStatsCoverage
snapshotCoverage?: AdminStatsCoverage
comparisonSnapshotCoverage?: AdminStatsCoverage
}
export interface AdminStatsCoverage {
status: 'complete' | 'partial' | 'empty'
expectedBuckets: number
completedBuckets: number
lowerBoundBuckets: number
quality: 'exact' | 'lower_bound'
dataThrough: string | null
}
export interface AdminStatsDelta {
value: number
previousValue: number
change: number
value: number | null
previousValue: number | null
change: number | null
changePercent: number | null
}
@@ -50,27 +54,38 @@ export interface AdminTransferDataQuality {
previousMissingBytesEvents: number
}
export interface AdminSharingDataQuality {
unlocatedViews: number | null
unlocatedDownloads: number | null
unlocatedEvents: number | null
}
export interface AdminStorageDataQuality extends AdminTransferDataQuality {
usageDriftSpaces: number | null
usageDriftBytes: number | null
}
export interface AdminDashboardOverviewStats extends AdminStatsRange {
dataQuality: AdminTransferDataQuality
totals: {
users: number
users: number | null
newUsers: AdminStatsDelta
activeUsers: AdminStatsDelta
activeUserRate: number | null
storageUsedBytes: number
storageQuotaBytes: number
storageUsedBytes: number | null
storageQuotaBytes: number | null
storageUtilization: number | null
trafficBytes: AdminStatsDelta
uploadBytes: AdminStatsDelta
downloadBytes: AdminStatsDelta
activeShares: number
activeShares: number | null
shareViews: AdminStatsDelta
shareDownloads: AdminStatsDelta
}
trends: Array<{
date: string
newUsers: number
activeUsers: number
activeUsers: number | null
storageUsedBytes: number | null
uploadBytes: number
downloadBytes: number
@@ -79,34 +94,37 @@ export interface AdminDashboardOverviewStats extends AdminStatsRange {
export interface AdminDashboardGrowthStats extends AdminStatsRange {
summary: {
totalUsers: number
totalUsers: number | null
newUsers: AdminStatsDelta
activeUsers: AdminStatsDelta
verifiedUsers: number
bannedUsers: number
silentUsers: number
verifiedUsers: number | null
bannedUsers: number | null
silentUsers: number | null
activeUserRate: number | null
silentUserRate: number | null
}
userScaleTrend: Array<{ date: string; newUsers: number; totalUsers: number }>
activeUserTrend: Array<{ date: string; dau: number; wau: number; mau: number }>
userScaleTrend: Array<{ date: string; newUsers: number; totalUsers: number | null }>
activeUserTrend: Array<{ date: string; dau: number | null; wau: number | null; mau: number | null }>
userStatus: Array<{ name: string; value: number; percent: number }>
registrationSources: Array<{ name: string; value: number; percent: number }>
}
export interface AdminDashboardStorageStats extends AdminStatsRange {
dataQuality: AdminTransferDataQuality
dataQuality: AdminStorageDataQuality
summary: {
storageUsedBytes: number
quotaBytes: number
fileCount: number
storageUsedBytes: number | null
quotaBytes: number | null
fileCount: number | null
trashFileCount: number | null
trashBytes: number | null
newFiles: AdminStatsDelta
newBytes: AdminStatsDelta
coldFileBytes: number
coldFileBytes: number | null
storageUtilization: number | null
coldFilePercent: number | null
nearQuotaSpaces: number
overQuotaSpaces: number
nearQuotaSpaces: number | null
overQuotaSpaces: number | null
invalidQuotaSpaces: number | null
}
storageTrend: Array<{ date: string; usedBytes: number | null; newBytes: number; newFiles: number }>
typeBreakdown: Array<{ type: string; bytes: number; files: number; percent: number }>
@@ -133,8 +151,9 @@ export interface AdminDashboardTrafficStats extends AdminStatsRange {
}
export interface AdminDashboardSharingStats extends AdminStatsRange {
dataQuality: AdminSharingDataQuality
summary: {
activeShares: number
activeShares: number | null
createdShares: AdminStatsDelta
views: AdminStatsDelta
downloads: AdminStatsDelta
@@ -151,15 +170,16 @@ export interface AdminDashboardSharingStats extends AdminStatsRange {
export interface AdminDashboardOperationsStats extends AdminStatsRange {
summary: {
activeBackgroundJobs: number
activeRemoteDownloads: number
onlineDownloaders: number
offlineDownloaders: number
activeBackgroundJobs: number | null
activeRemoteDownloads: number | null
onlineDownloaders: number | null
offlineDownloaders: number | null
backgroundJobFailureRate: number | null
remoteDownloadSuccessRate: number | null
cloudReportBacklog: number
webhookFailures: number
alertCount: number
cloudReportBacklog: number | null
cloudReportDeadLetters: number | null
webhookFailures: number | null
alertCount: number | null
}
trend: Array<{
date: string
+2
View File
@@ -216,9 +216,11 @@ export type {
AdminDashboardSharingStats,
AdminDashboardStorageStats,
AdminDashboardTrafficStats,
AdminSharingDataQuality,
AdminStatsCoverage,
AdminStatsDelta,
AdminStatsRange,
AdminStorageDataQuality,
AdminTopShare,
AdminTransferDataQuality,
AdminUsageBySpace,
+1 -1
View File
@@ -46,7 +46,7 @@ export function QuotaPanel({ enabled }: { enabled: boolean }) {
<p className="text-xs text-muted-foreground tabular-nums">
{quota.quota > 0
? t('quota.usage', { used: formatSize(quota.used), total: formatSize(quota.quota) })
: t('quota.usageNoLimit', { used: formatSize(quota.used) })}
: t('quota.usageInvalid', { used: formatSize(quota.used) })}
</p>
) : (
<Skeleton className={isLoading ? 'h-3 w-24' : 'h-3 w-16 opacity-50'} />
+9 -2
View File
@@ -80,6 +80,7 @@ function PlanUsageOverview({ quota }: { quota: UserQuota }) {
label={t('storage.storageUsage')}
used={quota.used}
total={quota.quota}
allowUnlimited={false}
detail={t('storage.storageUsageDetail', { used: formatSize(quota.used) })}
/>
<UsageMeter
@@ -87,6 +88,7 @@ function PlanUsageOverview({ quota }: { quota: UserQuota }) {
label={t('storage.trafficUsage')}
used={quota.trafficUsed}
total={quota.trafficQuota}
allowUnlimited
detail={t('storage.trafficPeriodDetail', { period: quota.trafficPeriod })}
/>
</div>
@@ -98,16 +100,19 @@ function UsageMeter({
label,
used,
total,
allowUnlimited,
detail,
}: {
icon: React.ReactNode
label: string
used: number
total: number
allowUnlimited: boolean
detail: string
}) {
const { t } = useTranslation()
const percent = total > 0 ? Math.min(100, (used / total) * 100) : 100
const invalid = total <= 0 && !allowUnlimited
const percent = total > 0 ? Math.min(100, (used / total) * 100) : invalid ? 0 : 100
return (
<div className="min-w-0 rounded-md bg-muted/20 p-4">
<div className="flex items-center justify-between gap-3">
@@ -116,7 +121,9 @@ function UsageMeter({
<span>{label}</span>
</span>
<span className="shrink-0 text-xs text-muted-foreground">
{total > 0 ? t('storage.usageTotal', { total: formatSize(total) }) : t('storage.usageNoLimit')}
{total > 0
? t('storage.usageTotal', { total: formatSize(total) })
: t(invalid ? 'storage.usageInvalid' : 'storage.usageNoLimit')}
</span>
</div>
<div className="mt-3 space-y-2">
+8 -21
View File
@@ -64,7 +64,6 @@ const ADMIN_STORAGES_KEYS = [
'admin.storages.customHostPlaceholder',
'admin.storages.fieldCapacity',
'admin.storages.capacityPlaceholder',
'admin.storages.capacityUnlimited',
'admin.storages.capacityHint',
'admin.storages.egressBilling',
'admin.storages.configureEgressBilling',
@@ -223,12 +222,10 @@ describe('admin.storages locale keys — English values contract', () => {
expect(enLocale['admin.storages.fieldCapacity']).toBe('Capacity')
})
it('admin.storages.capacityUnlimited is "Unlimited"', () => {
expect(enLocale['admin.storages.capacityUnlimited']).toBe('Unlimited')
})
it('admin.storages.capacityHint is "Maximum storage space. 0 means unlimited."', () => {
expect(enLocale['admin.storages.capacityHint']).toBe('Maximum storage space. 0 means unlimited.')
it('admin.storages.capacityHint describes an unreported zero capacity', () => {
expect(enLocale['admin.storages.capacityHint']).toBe(
'Maximum reported storage space. 0 means capacity is not reported.',
)
})
})
@@ -380,27 +377,17 @@ describe('admin.storages locale keys — i18n runtime translation', () => {
expect(i18n.t('admin.storages.fieldCapacity')).toBe('可用空间')
})
it('translates admin.storages.capacityUnlimited to English', async () => {
const { default: i18n } = await import('./index')
await i18n.changeLanguage('en')
expect(i18n.t('admin.storages.capacityUnlimited')).toBe('Unlimited')
})
it('translates admin.storages.capacityUnlimited to Chinese', async () => {
const { default: i18n } = await import('./index')
await i18n.changeLanguage('zh')
expect(i18n.t('admin.storages.capacityUnlimited')).toBe('不限制')
})
it('translates admin.storages.capacityHint to English', async () => {
const { default: i18n } = await import('./index')
await i18n.changeLanguage('en')
expect(i18n.t('admin.storages.capacityHint')).toBe('Maximum storage space. 0 means unlimited.')
expect(i18n.t('admin.storages.capacityHint')).toBe(
'Maximum reported storage space. 0 means capacity is not reported.',
)
})
it('translates admin.storages.capacityHint to Chinese', async () => {
const { default: i18n } = await import('./index')
await i18n.changeLanguage('zh')
expect(i18n.t('admin.storages.capacityHint')).toBe('最大存储空间,0 表示不限制。')
expect(i18n.t('admin.storages.capacityHint')).toBe('存储后端报告的最大空间,0 表示未报告容量。')
})
})
+3 -5
View File
@@ -713,13 +713,11 @@
"admin.settings.storageProTooltip": "Storage controls are available in ZPan Pro.",
"admin.settings.cloudStoreEnabled": "Storage Plans",
"admin.settings.cloudStoreEnabledHint": "Allow users to buy additional storage and apply gift cards.",
"admin.settings.unlimited": "Unlimited",
"admin.settings.previewTitle": "Live summary",
"admin.settings.previewDescription": "A quick read on how the current configuration will feel before you save it.",
"admin.settings.previewFallback": "Add a short description so users immediately understand what this instance is for.",
"admin.settings.previewUrlLabel": "Primary address",
"admin.settings.previewQuotaLabel": "New organization quota",
"admin.settings.previewQuotaUnlimited": "New organizations will start without a storage cap.",
"admin.settings.previewBrandingLabel": "Branding next step",
"admin.settings.previewBrandingHint": "Use the branding section below to add your logo, favicon, and footer visibility rules.",
"admin.settings.brandingTitle": "Branding",
@@ -941,8 +939,7 @@
"admin.storages.previewSigningRegion": "Signing region: {{region}}",
"admin.storages.fieldCapacity": "Capacity",
"admin.storages.capacityPlaceholder": "0",
"admin.storages.capacityUnlimited": "Unlimited",
"admin.storages.capacityHint": "Maximum storage space. 0 means unlimited.",
"admin.storages.capacityHint": "Maximum reported storage space. 0 means capacity is not reported.",
"admin.storages.egressBilling": "Traffic Credits billing",
"admin.storages.configureEgressBilling": "Configure egress billing",
"admin.storages.billingTitle": "Limits and billing",
@@ -1051,7 +1048,7 @@
"quota.storage": "Storage",
"quota.traffic": "Traffic",
"quota.usage": "{{used}} / {{total}} used",
"quota.usageNoLimit": "{{used}} used",
"quota.usageInvalid": "{{used}} used · invalid storage quota",
"quota.purchased": "{{amount}} purchased or granted",
"quota.purchasedTraffic": "{{amount}} traffic purchased or granted",
"quota.cloudStorageEntitlement": "{{amount}} extra storage",
@@ -1803,6 +1800,7 @@
"storage.freePlanPrice": "Free",
"storage.usageTotal": "of {{total}}",
"storage.usageNoLimit": "No limit",
"storage.usageInvalid": "Invalid quota",
"storage.legendUsed": "Used",
"storage.legendBaseQuota": "Plan quota",
"storage.legendCloudQuota": "Extra quota",
+3 -5
View File
@@ -713,13 +713,11 @@
"admin.settings.storageProTooltip": "Storage 控制项仅在 ZPan Pro 中可用。",
"admin.settings.cloudStoreEnabled": "存储套餐",
"admin.settings.cloudStoreEnabledHint": "允许用户购买额外存储空间或使用礼品卡。",
"admin.settings.unlimited": "不限",
"admin.settings.previewTitle": "实时概览",
"admin.settings.previewDescription": "在保存之前,快速查看当前配置给用户呈现出来的大致感觉。",
"admin.settings.previewFallback": "添加一段简短描述,让用户一进来就知道这个实例是做什么的。",
"admin.settings.previewUrlLabel": "主访问地址",
"admin.settings.previewQuotaLabel": "新组织默认配额",
"admin.settings.previewQuotaUnlimited": "新组织创建后默认不设存储上限。",
"admin.settings.previewBrandingLabel": "品牌下一步",
"admin.settings.previewBrandingHint": "继续在下方配置 Logo、Favicon 和底部署名显示规则。",
"admin.settings.brandingTitle": "品牌设置",
@@ -941,8 +939,7 @@
"admin.storages.previewSigningRegion": "签名区域:{{region}}",
"admin.storages.fieldCapacity": "可用空间",
"admin.storages.capacityPlaceholder": "0",
"admin.storages.capacityUnlimited": "不限制",
"admin.storages.capacityHint": "最大存储空间,0 表示不限制。",
"admin.storages.capacityHint": "存储后端报告的最大空间,0 表示未报告容量。",
"admin.storages.egressBilling": "流量 Credits 计费",
"admin.storages.configureEgressBilling": "配置流量计费",
"admin.storages.billingTitle": "容量与计费",
@@ -1051,7 +1048,7 @@
"quota.storage": "存储空间",
"quota.traffic": "下载流量",
"quota.usage": "{{used}} / {{total}} 已使用",
"quota.usageNoLimit": "{{used}} 已使用",
"quota.usageInvalid": "{{used}} 已使用 · 存储额度异常",
"quota.purchased": "已购买或获赠 {{amount}}",
"quota.purchasedTraffic": "已购买或获赠 {{amount}} 流量",
"quota.cloudStorageEntitlement": "额外存储 {{amount}}",
@@ -1803,6 +1800,7 @@
"storage.freePlanPrice": "免费",
"storage.usageTotal": "共 {{total}}",
"storage.usageNoLimit": "不限量",
"storage.usageInvalid": "额度异常",
"storage.legendUsed": "已用",
"storage.legendBaseQuota": "套餐额度",
"storage.legendCloudQuota": "额外额度",
+1 -2
View File
@@ -13,9 +13,8 @@ export function getInitials(name: string): string {
.join('')
}
// "used / total" with ∞ for an unlimited (0 or negative) quota.
export function formatStorageUsage(used: number, total: number): string {
return `${formatSize(used)} / ${total <= 0 ? '∞' : formatSize(total)}`
return total > 0 ? `${formatSize(used)} / ${formatSize(total)}` : `${formatSize(used)} / —`
}
export function formatSize(bytes: number): string {
@@ -50,6 +50,8 @@ const overviewStats: AdminDashboardOverviewStats = {
status: 'complete',
expectedBuckets: 192,
completedBuckets: 192,
lowerBoundBuckets: 0,
quality: 'exact',
dataThrough: '2026-07-09T00:00:00.000Z',
},
dataQuality: {
@@ -96,6 +98,8 @@ const operationsStats: AdminDashboardOperationsStats = {
status: 'complete',
expectedBuckets: 192,
completedBuckets: 192,
lowerBoundBuckets: 0,
quality: 'exact',
dataThrough: '2026-07-09T00:00:00.000Z',
},
summary: {
@@ -106,6 +110,7 @@ const operationsStats: AdminDashboardOperationsStats = {
backgroundJobFailureRate: 5,
remoteDownloadSuccessRate: 95,
cloudReportBacklog: 6,
cloudReportDeadLetters: 0,
webhookFailures: 7,
alertCount: 13,
},
@@ -201,6 +206,8 @@ describe('Admin overview dashboard', () => {
status: 'partial',
expectedBuckets: 192,
completedBuckets: 144,
lowerBoundBuckets: 0,
quality: 'exact',
dataThrough: '2026-06-30T00:00:00.000Z',
},
})
+99 -20
View File
@@ -5,6 +5,7 @@ import type {
AdminDashboardSharingStats,
AdminDashboardStorageStats,
AdminDashboardTrafficStats,
AdminSharingDataQuality,
AdminStatsRange,
AdminTransferDataQuality,
} from '@shared/types'
@@ -68,7 +69,7 @@ import {
getAdminDashboardStorageStats,
getAdminDashboardTrafficStats,
} from '@/lib/api'
import { formatSize } from '@/lib/format'
import { formatSize as formatByteSize } from '@/lib/format'
import { cn } from '@/lib/utils'
export const Route = createFileRoute('/_authenticated/admin/')({
@@ -681,7 +682,7 @@ function StorageSection({ stats }: { stats: AdminDashboardStorageStats }) {
label: '配额使用率',
value: formatPercent(stats.summary.storageUtilization),
},
{ label: '文件数', value: formatNumber(stats.summary.fileCount) },
{ label: '回收站占用', value: formatSize(stats.summary.trashBytes) },
]}
/>
<StatCard
@@ -701,7 +702,7 @@ function StorageSection({ stats }: { stats: AdminDashboardStorageStats }) {
icon={Database}
metrics={[
{ label: '上期新增', value: formatNumber(stats.summary.newFiles.previousValue) },
{ label: '文件总数', value: formatNumber(stats.summary.fileCount) },
{ label: '活跃文件总数', value: formatNumber(stats.summary.fileCount) },
]}
/>
<StatCard
@@ -718,9 +719,10 @@ function StorageSection({ stats }: { stats: AdminDashboardStorageStats }) {
/>
</div>
<TransferDataQualityNotice quality={stats.dataQuality} />
<StorageUsageDataQualityNotice quality={stats.dataQuality} />
<ChartCard
title="空间配额压力"
subtitle={`全部空间中 ${stats.summary.nearQuotaSpaces} 个达到 80%${stats.summary.overQuotaSpaces} 个达到或超过配额。`}
subtitle={`全部空间中 ${formatNumber(stats.summary.nearQuotaSpaces)} 个达到 80%${formatNumber(stats.summary.overQuotaSpaces)} 个达到或超过配额${formatNumber(stats.summary.invalidQuotaSpaces)} 个额度配置异常`}
contentClassName="h-auto"
>
{stats.topSpaces.length === 0 ? (
@@ -1116,7 +1118,7 @@ function SharingSection({ stats }: { stats: AdminDashboardSharingStats }) {
]}
/>
<StatCard
label="访问次数"
label="已定位访问"
value={formatNumber(stats.summary.views.value)}
delta={formatDelta(stats.summary.views)}
icon={Activity}
@@ -1126,7 +1128,7 @@ function SharingSection({ stats }: { stats: AdminDashboardSharingStats }) {
]}
/>
<StatCard
label="下载签发"
label="已定位下载签发"
value={formatNumber(stats.summary.downloads.value)}
delta={formatDelta(stats.summary.downloads)}
icon={Download}
@@ -1149,8 +1151,9 @@ function SharingSection({ stats }: { stats: AdminDashboardSharingStats }) {
]}
/>
</div>
<SharingDataQualityNotice quality={stats.dataQuality} />
<div className="grid gap-4">
<ChartCard title="访问行为趋势" subtitle="这些是独立事件,不表示同一访客完成了连续漏斗。">
<ChartCard title="访问行为趋势" subtitle="仅展示能定位到具体时间的独立事件,不表示同一访客完成了连续漏斗。">
<ResponsiveContainer width="100%" height="100%">
<AreaChart data={stats.trend}>
<CartesianGrid stroke={CHART_GRID_COLOR} strokeDasharray="3 3" vertical={false} />
@@ -1236,7 +1239,7 @@ function OperationsSection({ stats }: { stats: AdminDashboardOperationsStats })
icon={FileClock}
metrics={[
{ label: '计量积压', value: formatNumber(stats.summary.cloudReportBacklog) },
{ label: 'Webhook 失败', value: formatNumber(stats.summary.webhookFailures) },
{ label: '计量死信', value: formatNumber(stats.summary.cloudReportDeadLetters) },
]}
/>
</div>
@@ -1530,11 +1533,30 @@ function QueryState<T extends AdminStatsRange>({
function StatsCoverageNotice({ stats }: { stats: AdminStatsRange }) {
const { coverage } = stats
const comparisonCoverage = stats.comparisonCoverage
const snapshotCoverage = stats.snapshotCoverage
const comparisonSnapshotCoverage = stats.comparisonSnapshotCoverage
const through = coverage.dataThrough
? new Date(coverage.dataThrough).toISOString().replace('T', ' ').slice(0, 16)
: null
const snapshotThrough = snapshotCoverage?.dataThrough
? new Date(snapshotCoverage.dataThrough).toISOString().replace('T', ' ').slice(0, 16)
: null
const comparisonIncomplete = comparisonCoverage ? comparisonCoverage.status !== 'complete' : false
const incomplete = coverage.status !== 'complete' || comparisonIncomplete
const lowerBound =
coverage.quality === 'lower_bound' ||
comparisonCoverage?.quality === 'lower_bound' ||
snapshotCoverage?.quality === 'lower_bound' ||
comparisonSnapshotCoverage?.quality === 'lower_bound'
const snapshotIncomplete = snapshotCoverage ? snapshotCoverage.status !== 'complete' : false
const comparisonSnapshotIncomplete = comparisonSnapshotCoverage
? comparisonSnapshotCoverage.status !== 'complete'
: false
const incomplete =
coverage.status !== 'complete' ||
comparisonIncomplete ||
snapshotIncomplete ||
comparisonSnapshotIncomplete ||
lowerBound
return (
<div
role="status"
@@ -1544,22 +1566,37 @@ function StatsCoverageNotice({ stats }: { stats: AdminStatsRange }) {
)}
>
<span className={incomplete ? 'text-amber-800 dark:text-amber-200' : 'text-muted-foreground'}>
{coverage.status === 'empty'
? '所选范围还没有可用的离线结果。'
: coverage.status === 'partial'
? '所选范围存在缺失的小时结果,当前数据不完整。'
: comparisonCoverage?.status === 'empty'
? '对比区间还没有可用的离线结果,环比不可对账。'
: comparisonCoverage?.status === 'partial'
? '对比区间存在缺失的小时结果,环比数据不完整。'
: '所选范围的离线结果完整。'}
{lowerBound
? '部分小时只有可验证的数据下限;页面不会把这些数值表述为完整事实。'
: coverage.status === 'empty'
? '所选范围还没有可用的离线结果。'
: coverage.status === 'partial'
? '所选范围存在缺失的小时结果,当前数据不完整。'
: comparisonCoverage?.status === 'empty'
? '对比区间还没有可用的离线结果,环比不可对账。'
: comparisonCoverage?.status === 'partial'
? '对比区间存在缺失的小时结果,环比数据不完整。'
: snapshotCoverage?.status === 'empty'
? '所选范围没有可用的状态快照;状态类指标显示为 —,不会伪装成 0。'
: snapshotCoverage?.status === 'partial'
? '所选范围的事件结果完整,但状态快照仅覆盖部分小时。'
: comparisonSnapshotCoverage?.status === 'empty'
? '对比区间没有状态快照,状态类环比不可对账。'
: comparisonSnapshotCoverage?.status === 'partial'
? '对比区间的状态快照不完整,状态类环比不可对账。'
: '所选范围的离线结果完整。'}
</span>
<span className="text-xs text-muted-foreground">
{through ? `数据截至 ${through} UTC · ` : ''}
{coverage.completedBuckets}/{coverage.expectedBuckets}
{coverage.lowerBoundBuckets > 0 ? ` · 下限 ${coverage.lowerBoundBuckets} 小时` : ''}
{comparisonCoverage
? ` · 对比 ${comparisonCoverage.completedBuckets}/${comparisonCoverage.expectedBuckets} 小时`
: ''}
{snapshotCoverage
? ` · 快照 ${snapshotCoverage.completedBuckets}/${snapshotCoverage.expectedBuckets} 小时`
: ''}
{snapshotThrough ? ` · 快照采样于 ${snapshotThrough} UTC` : ''}
</span>
</div>
)
@@ -1599,6 +1636,42 @@ function TransferDataQualityNotice({ quality }: { quality: AdminTransferDataQual
)
}
function SharingDataQualityNotice({ quality }: { quality: AdminSharingDataQuality }) {
if (quality.unlocatedEvents === null || quality.unlocatedEvents === 0) return null
return (
<div
role="status"
className="flex flex-col gap-2 rounded-lg border border-amber-500/40 bg-amber-500/10 px-4 py-3 text-sm sm:flex-row sm:items-center"
>
<Badge variant="outline" className="w-fit border-amber-600/50 text-amber-700 dark:text-amber-300">
</Badge>
<span className="text-muted-foreground">
{formatNumber(quality.unlocatedViews)} 访 {formatNumber(quality.unlocatedDownloads)}{' '}
访
</span>
</div>
)
}
function StorageUsageDataQualityNotice({ quality }: { quality: AdminDashboardStorageStats['dataQuality'] }) {
if (quality.usageDriftSpaces === null || quality.usageDriftSpaces === 0) return null
return (
<div
role="status"
className="flex flex-col gap-2 rounded-lg border border-red-500/40 bg-red-500/10 px-4 py-3 text-sm sm:flex-row sm:items-center"
>
<Badge variant="outline" className="w-fit border-red-600/50 text-red-700 dark:text-red-300">
</Badge>
<span className="text-muted-foreground">
{formatNumber(quality.usageDriftSpaces)} {' '}
{formatSize(quality.usageDriftBytes)}
</span>
</div>
)
}
function EmptyState() {
return (
<div className="rounded-lg border border-dashed border-border/70 bg-muted/10 p-8 text-center text-sm text-muted-foreground">
@@ -1646,10 +1719,15 @@ function normalizeDateRange(range: DateRange): DateRange {
return { from: range.to, to: range.from }
}
function formatNumber(value: number): string {
function formatNumber(value: number | null): string {
if (value === null) return '—'
return new Intl.NumberFormat().format(value)
}
function formatSize(value: number | null): string {
return value === null ? '—' : formatByteSize(value)
}
function formatChartDate(value: unknown): string {
const text = String(value)
const match = /^(\d{4})-(\d{2})-(\d{2})/.exec(text)
@@ -1671,9 +1749,10 @@ function formatPer100(value: number | null): string {
}
function formatDelta(
delta: { value: number; previousValue: number; change: number; changePercent: number | null },
delta: { value: number | null; previousValue: number | null; change: number | null; changePercent: number | null },
valueFormatter: (value: number) => string = formatNumber,
): string {
if (delta.change === null) return '—'
const sign = delta.change >= 0 ? '+' : ''
return `${sign}${valueFormatter(delta.change)} (${formatPercent(delta.changePercent)})`
}
+3 -2
View File
@@ -374,7 +374,7 @@ describe('StoragePage', () => {
expect(view.getByText('storage.trafficPeriodDetail:2026-05')).toBeTruthy()
})
it('does not mark unlimited quota usage as over cap', async () => {
it('renders invalid storage quota separately from unlimited traffic', async () => {
vi.mocked(getUserQuota).mockResolvedValue({
orgId: 'org-1',
baseQuota: 0,
@@ -404,7 +404,8 @@ describe('StoragePage', () => {
await waitFor(() => expect(view.getAllByText('1.5 KB').length).toBeGreaterThan(0))
expect(view.queryByText('storage.overCap')).toBeNull()
expect(view.getAllByText('storage.usageNoLimit')).toHaveLength(2)
expect(view.getByText('storage.usageInvalid')).toBeTruthy()
expect(view.getByText('storage.usageNoLimit')).toBeTruthy()
})
it('hides self-service forms when storage purchases are disabled', async () => {
+1
View File
@@ -34,6 +34,7 @@ export async function handleScheduled(event: ScheduledTrigger, env: ScheduledEnv
const deps = createDeps(platform)
const cloudBaseUrl = env.ZPAN_CLOUD_URL ?? ZPAN_CLOUD_URL_DEFAULT
if (event.cron === TRAFFIC_SYNC_CRON) {
await deps.quota.reconcileFreePlanBaselines()
await Promise.all([
syncPendingCloudTrafficReports(deps, { cloudBaseUrl }),
syncPendingRemoteDownloadUsageReports(deps, { cloudBaseUrl }),