the client_credentials grant could never complete: the authorization
insert left principal_type null, state/code/refresh columns were
declared not null for a grant that never sets them, and the connection
grant type silently fell back to authorization_code because builders
match on the enum value, not the name
- stamp the service-account principal type on the cc authorization
- bind explicit nulls for grant-specific columns in insertauthorization
- make state_hash, authorization_code_hash and the refresh hashes
nullable in seed ddl and resync the embedded schema fingerprint
Join 1242 keyword-split statement lines (COMMENT/CREATE/ALTER/LOAD), move the dc3_identity_audit_log table section and the schema fingerprint INSERT so every script orders DDL before seed data, and drop one duplicated COMMENT ON COLUMN. Statement multiset verified identical apart from the intended dedup. The canonical fingerprint changes to e81e3562... and is synchronized via schema_fingerprint.py --sync into the embedded ddl_hash and all four distribution points; existing initialized databases require re-initialization (flag day).
- migrate auth, manager, data, and agentic persistence to R2DBC
- hard cut pagination, facade, gateway, web, and CLI contracts
- remove legacy database and TSDB adapters and document validation gates
BREAKING CHANGE: platform persistence now requires PostgreSQL 18 with TimescaleDB and the new reactive API contracts.
R2c of docs/design/storage-abstraction.md §3. dc3-db-tck runs one set of
mapper-level assertions against both engines with the real mapper XML —
the databaseIdProvider routing is part of what gets certified, not just
the SQL: the fenced latest-value upsert's three-level tiebreak (fencing
token, then time, then sequence, then message id), the state upsert +
same-transaction re-select shape, the three-step expired-lease claim
(lock-and-read FOR UPDATE SKIP LOCKED, flip offline, derive the
post-update view), the lease upserts' row-local increments (token moves
only on ownership change), the row-level revision triggers (insert and
meaningful update bump, trivial update does not), and the tool-catalog
query's encoded-JSON double-hop.
Harness notes worth keeping: one SqlSessionFactory per database mirrors
the production @DS wiring (state/latest/leases/catalog span four
databases with unqualified table names); the mapper XML loads through
XMLMapperBuilder with the affectData mybatis-plus DTD extension
stripped; postgres registers the timestamptz type handler. Both
harnesses learned to distrust podman: log-following never fires, so
mysql waits on the listening TCP port and postgres polls a real JDBC
handshake (the port opens during initdb's local-only phase and closes
again for the restart). The postgres harness mounts seed files 01..07
individually — 00's age/pgvector extensions exist only in the
production base image — with 05 trimmed at runtime to the
dc3_point_latest projection (the hypertable half needs the base
image's TimescaleDB function set; the dialect contract doesn't touch
it).
The catalog fixture plants its api row through a parameterized insert:
the escaped-JSON value must not travel through SQL text (MySQL's
default backslash escapes eat the \" the value depends on) nor bind
uniformly (PG rejects varchar->json, MySQL rejects the binary
Types.OTHER binding) — the fixture branches on the engine.
Selection guide published at docs/db-dialects.md (dialect matrix,
swap procedure, the semantic decisions the suite locks); the storage
design's status header moves to Implemented with an R1/R2 closeout
note.
Gates: db-tck 16/16 (8 postgres + 8 mysql), data/auth/manager suites
green (274 in data), e2e 26/26, full-repo compile green.
R2b of docs/design/storage-abstraction.md §3:
- PostgreSQL seed: the three driver-lease sequences retire. The fencing
token and assignment version default to 1 and advance row-locally —
per-device / per-driver monotonicity is the property the checks rely
on, which the global sequences only delivered incidentally, and MySQL
has no sequence objects. The three statement-level revision triggers
(transition tables + INSERT .. ON CONFLICT, none of which MySQL can
express) collapse into one row-level function with row-local +1 upserts
— a shape both engines share.
- MySQL seed under dc3/dependencies/mysql/initdb/ (00-06), derived by
dc3/dependencies/mysql/pg2mysql_seed.py: schemas become databases,
TIMESTAMPTZ becomes DATETIME(6), TEXT/JSON literal defaults become
expression defaults (MySQL 8 refuses literal defaults on BLOB/TEXT/
JSON), operate-time plpgsql triggers become the ON UPDATE column
attribute (explicit SETs keep winning), the revision family becomes
row-level DELIMITER triggers, keyed TEXT columns widen to VARCHAR(191),
partial indexes drop, and the embedded-JSON seed values load under
NO_BACKSLASH_ESCAPES (MySQL string literals eat the backslashes the
escaped-JSON content values depend on; PostgreSQL does not). 05 is
hand-maintained: only the dc3_point_latest projection — a MySQL core
requires an external time-series store, so the hypertable and cagg DDL
have no counterpart by design.
- Compose: the optional stack gains a mysql:8.4 service (utf8mb4, seed
mounted at docker-entrypoint-initdb.d, healthcheck); the app stack
passes DC3_DB_TYPE through.
Verified by loading all seven files into a fresh mysql:8.4 container —
zero errors, five databases, table counts aligned with the PostgreSQL
seed (manager 24 incl. common, auth 24, data 12, agentic 6, history 1),
and the revision trigger proven live: insert bumps the owning driver's
revision to 1, an enable_flag flip bumps it again to 2.
The timescale adapter now owns the S16 lifecycle: raw 30 days (seed 05,
was 180) -> 1-minute tier 1 year (configurable via
dc3.tsdb.timescale.rollup.minute-keep-days) -> 1-hour tier forever.
bucketedAggregate and bucketedCount serve bucket widths at or above a
tier's granularity from that tier with exactly composable expressions
(AVG recombines as SUM(num_sum)/SUM(num_count), COUNT as
SUM(sample_count)); PERCENTILE and FIRST/LAST stay on the raw path — the
shared caggs carry textual first/last only, and tiered numeric first/last
would version-skew deployments created before such columns existed.
Single-window aggregate() stays raw by design: alarm windows must never
read through materialization lag.
The tiers are the observability pipeline's existing real-time continuous
aggregates (cagg_point_value_1m/1h, queried by Grafana) — one structure
serves both consumers; the adapter bootstraps the same names with an
identical shape via IF NOT EXISTS, so embedded seed boots, standalone
adapter boots and already-running deployments all converge on it without
rebuilding anything. Real-time mode (materialized_only=FALSE, no longer
the TS 2.13+ default) keeps tier reads correct immediately after an
append; the refresh policies only move aggregation work off the read
path. deleteRange refreshes both tiers afterwards.
Also fixed along the way: TDengine's bucketedAggregate PERCENTILE hit the
same "single table query" limit as aggregate() and now routes per-series
subtables.
Gates: timescale TCK 24/24 including the new rollup case (tiered
COUNT/AVG/LAST/bucketedCount and bucketed P50 all verified against raw
scans before any refresh materialized); TDengine TCK 24/24 with the same
case passing via honest raw degradation (2 declared-capability skips).
Phase 1b of docs/design/tsdb-abstraction.md: every point-value read and
write now flows through the dc3-tsdb port, and dc3-common-repository is
deleted outright (no compat aliases).
Write path — new PointValueIngestService orchestration in dc3-common-data:
the stale-owner lease guard leaves the history INSERT's cross-schema join
and resolves the active owner per distinct device through the existing
DeviceFacade.getActiveOwner chain; a caffeine-backed ingest idempotency
window (dc3.data.ingest.*, marked only after both writes commit) absorbs
MQ redeliveries now that the message_id unique index is retired; the
batch lands via TsdbStore.append (natural series+time upsert, INGEST_ORDER
kept for the fenced dc3_point_latest upsert).
Read path — history via last(), page() resolves name/enable filters to
series keys through relational metadata (tenant-wide when unrestricted,
cursor offset emulation capped at 10k) with count() for totals; the alarm
window backend becomes TsdbWindowDataSource over aggregate()/history().
Dashboards — all nine data-side hypertable statements move to the S13
analytics facet (count/bucketedCount/countByDimension/latencyHistogram/
lastSeenPerSeries) plus the dc3_point_latest projection for latestStream;
the manager topology statement crosses into the data center via a new
ListSeriesVolumes RPC backed by the new S13-5 seriesCounts primitive (a
point shared by several devices needs per-series counts, which
single-dimension grouping cannot reconstruct); both cross-schema joins
are gone.
Port/adapter — seriesCounts added with a TCK case (23 total); series
filters compile to row-value IN lists chunked at 500 pairs; bootstrap
adds the quality column idempotently and retires uk_point_value_event so
existing deployments converge on the new upsert; the adapter binds the
application-provided tsdbDataSource bean (the history dynamic-datasource
entry) instead of the routing primary.
Timestamps — BO<->Instant conversion is pinned to
TimeConstant.DEFAULT_ZONEID (the platform canonical zone the latest
projection type handler already writes with); the design note's "lock
UTC" is amended accordingly, together with the lease-guard and S13-5
deviations recorded in §6.2.
Seed/deploy — 05-iot-dc3-history.sql gains the quality column and swaps
the unique index; compose carries DC3_TSDB_TYPE; make deploy publishes
dc3-tsdb.
Also fixed along the way: root pom never managed dc3-mq-core (latent,
masked by full-reactor builds); PointValueMapper ran unrouted on the
master datasource (now @DS("history")); latestStream name enrichment
looked up a Long-keyed map with String keys.
Gates: tsdb TCK 23/23, dc3-common-data 262/262, e2e 26/26 including the
unmodified PostgresHypertableIT; full reactor green except dc3-mq-tck's
kafka/pulsar container-startup flake, reproduced identically twice and
unrelated to this change.
Use PostgreSQL leases and fencing for distributed ownership, require a durable SQLite outbox before RabbitMQ publication, and make Data Center ingestion transactional and idempotent.
BREAKING CHANGE: drivers require mandatory durable outbox configuration and use lease-aware ownership and telemetry contracts.
Add dc3_menu rows for the MCP Connection / MCP Client / MCP Tool settings pages (codes settingsMcpConnection/Client/Tool under the Integration group), matching the new frontend management views.
Reorganize dc3_menu (and mirrored dc3_resource) under Settings into Identity / Access / Model / Alarm / Event&Command / Audit / Integration / System, matching the frontend settingsNav.ts. Replaces the flat, out-of-order layout and retires the legacy Event/Command group containers.
pnoker