the repo-wide reformat mangled every go template directive in the
chart ({{ }} split to { { } }, {{- to {{ -, headers indented into the
yaml, ports list items misaligned under conditional blocks), leaving
the chart unrenderable since it landed; restore the directives, align
the conditional grpc/https port entries with the http entry, and drop
the stray blank lines - helm lint now passes and the full chart
renders 66 resources with both default and production values
the historical defaults dc3.security.key.2026.io.github.pnoker and
io.github.pnoker.dc3 are public in this repository, so any deployment
that never overrode them shipped a known hmac signing key
- app/scale/swarm compose: fail fast (${var:?}) when dc3_security_key
or auth_hmac_secret is unset; dev compose keeps a clearly-marked
dev-only default
- helm production/default values and k8s secret.env ship empty values:
pre/pro startup rejects blank and known-default hmac keys, forcing
--set-string or an existingsecret; document the per-key failure mode
honestly instead of overpromising startup rejection for both
- dev env files: rotate both keys to machine-generated dev-only values
and add throwaway rs256 material for local oauth flows
- fix the chart maintainer email to a syntactically valid address
the client_credentials grant could never complete: the authorization
insert left principal_type null, state/code/refresh columns were
declared not null for a grant that never sets them, and the connection
grant type silently fell back to authorization_code because builders
match on the enum value, not the name
- stamp the service-account principal type on the cc authorization
- bind explicit nulls for grant-specific columns in insertauthorization
- make state_hash, authorization_code_hash and the refresh hashes
nullable in seed ddl and resync the embedded schema fingerprint
AGENTS.md stays lean with always-relevant rules and a task routing
table; the deep per-area conventions move to
dc3/agents/{backend,frontend,cli,testing}.md to be read on demand.
CONTRIBUTING.md documents the changelog-only commit rule and the
generated-CHANGE.md editing policy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Land the v7 three-terminal UX decision: drop the width/pointer-based
44px touch enlargement so one 32px density holds across desktop,
tablet, and mobile, and rebuild entity pages on a responsive record
list (ResponsiveRecordList + EntityMobileCard + per-entity mobile
summaries). Tighten the nginx security headers (CSP frame-ancestors
'none', X-Frame-Options DENY) to match the static e2e server.
Complete the dc3-web audit remediation, including the remaining LOW
items from docs/dc3-web-low-defects-handoff.md: streaming-safe
message loads in the agentic store, mock menu reparent cycle
rejection, paged-list page clamping, fetch/axios mock request
normalization, per-table attribute catalogs, post-flush alarm
watchers, rowKey-agnostic delete state, localized card text
fallbacks, and e2e-server port fail-fast.
Gate: vue-tsc, eslint, vitest 687 passed, responsive e2e 64 passed.
Co-Authored-By: Claude Code <noreply@anthropic.com>
The type-level Javadoc check walked annotations line by line and stopped
inside multiline annotation arguments, reporting false negatives such as
AuthInitRunner. Reuse the bracket-aware _annotation_block_start helper so
Javadoc attached above a multiline annotation block is recognized.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Add Dockerfiles under dc3/dependencies for MariaDB and MySQL that layer on the dc3-mariadb-base/dc3-mysql-base images published by dc3-docker (2026.6), matching the dc3-postgres layering convention. Nothing is layered yet since DC3 runs on the PostgreSQL dialect; credentials, seeds, and entrypoint overrides land here when a MariaDB/MySQL profile is needed.
Serve the dc3-web image over plain HTTP on port 80 only: remove the ssl_certificate/ssl_certificate_key directives, the 443 listeners and the HSTS header from default.conf, drop the 8443:443 mapping and the certificate COPY from the Dockerfile, and remove the acme.sh README. The retired dc3.site certificate is preserved as reference material at nginx/certs/ssl-certs.md.
Join 1242 keyword-split statement lines (COMMENT/CREATE/ALTER/LOAD), move the dc3_identity_audit_log table section and the schema fingerprint INSERT so every script orders DDL before seed data, and drop one duplicated COMMENT ON COLUMN. Statement multiset verified identical apart from the intended dedup. The canonical fingerprint changes to e81e3562... and is synchronized via schema_fingerprint.py --sync into the embedded ddl_hash and all four distribution points; existing initialized databases require re-initialization (flag day).
- migrate auth, manager, data, and agentic persistence to R2DBC
- hard cut pagination, facade, gateway, web, and CLI contracts
- remove legacy database and TSDB adapters and document validation gates
BREAKING CHANGE: platform persistence now requires PostgreSQL 18 with TimescaleDB and the new reactive API contracts.
- gateway container env passes DC3_GATEWAY_OAUTH_ENABLED/JWKS_URL so
deployments can turn RS256 verification on without image changes
- PrincipalHeader regains a 7-arg constructor delegating scopes=null,
keeping ~15 existing call sites compiling against the new field
- driver replicas are per-instance nodes with their own RabbitMQ
command queues (group=dc3.driver.client, key service.node), not
workers over shared queues; FAQ now also states Compose/Swarm keep
protocol drivers at 1 replica (shared driver_data / SQLite outbox)
- fail-fast gate covers AUTH_HMAC_SECRET only; DC3_SECURITY_KEY is
non-empty-checked, so a weak default still boots
- 6271 is intended as the device UDP channel but every publish site
omits /udp / protocol: UDP and actually publishes TCP - say so
- k8s scaling note: scaled driver pods are independent nodes, not one
shared worker pool
- swarm runbook: pushing dependency images is a multi-node requirement,
a local build is fine on a single node
- USAGE.md: NODE_ENV bullet narrowed to weak AUTH_HMAC_SECRET rejected
- include dc3-db in the make deploy artifact set alongside common/mq/tsdb
- sample DC3_DB_TYPE/DC3_TSDB_TYPE and external-store ports in .env.example
- expose mysql/mariadb/tdengine/influxdb/iotdb/jaeger in the compose stacks
- update AGENTS.md module map and module-ownership guidance
- add the missing AGPL header to the oracle driver service
- let TSDB failures propagate in the alarm window data source: swallowing
them made an outage read as no-data and silently missed alarms
- dedupe ingest batches by natural key before store upserts (same-row
writes poison timescale ON CONFLICT batches)
- turn unsupported countByDimension into a structured refusal (analytics)
or capability-honest empty panel (dashboard) instead of a raw 500
- pin LocalDateTime.now to DEFAULT_ZONEID on the tsdb read paths
- adapter fixes: influxdb/iotdb/tdengine/timescale store and auto-config
corrections, iotdb system properties, capability declarations on
TsdbStore, new timescale store unit tests
- derive pagination DbType from the selected dc3-db adapter with a fail-fast
when a non-default dc3.db.type lacks its adapter jar
- skip postgres profile activation for mariadb, not just mysql
- fork OAuthMcpMapper/AlertMapper statements for mysql and mariadb
(CAST AS JSON, ::text casts, ODKU value references)
- honor GET_LOCK verdict in resource registry sync instead of assuming
acquisition on every engine
- pg2mysql_seed.py: convert partial unique indexes to stored generated guard
columns instead of dropping them; add 99-iot-dc3-grants.sql
- add DC3_DB_URL/DC3_DB_HISTORY_URL per-service overrides in center configs
- cover the new mapper forks in the db TCK; fix AGPL headers in spring
metadata files
- add the canonical AGPL header to the six agentic application yml files and the two python scripts that lacked one
- add the header to all 79 kustomize, helm, and grafana yaml files, including the k8s driver template which regenerates the 36 driver manifests
- verified: the k8s yaml files parse (multi-document aware), the python scripts compile, and the regenerated drivers differ only by the header
- add ble, can, dlt645, dnp3, iec61850, kafka, knx, lorawan, mbus, redis, and zigbee to the k8s driver list, kustomization, helm values, and compose service stacks
- regenerate k8s driver deployments and preserve the listening-virtual ports in the generator
- replace the per-driver named volumes with one shared driver_data volume isolated by mount path, matching the scale/swarm stacks
- correct scaling docs: compose scale/swarm drivers stay at one replica because replicas would share one SQLite outbox file
A comment-health audit driven by an AST scanner (regex scanners kept
false-positiving on annotation-separated javadoc; the AST pass with
constructor and @Override-implementation exemptions is what produced a
trustworthy worklist) ended with a much smaller true gap than the raw
numbers suggested: class-level javadoc is already at 100% repository-wide.
What actually gets filled here:
- AnalyticsModel: all 30 nested records of the S19 analytics facet get
maintainer-voice javadoc (the @Schema descriptions serve the API side;
these serve code readers).
- The whole entity/ext family (29 files, 47 nested classes): every
Content/Ui/Validation/Security/AppliesTo/Template/Dedup/... schema
class states what it actually holds — read from the fields, not
templated.
- Adapter families: the @Bean methods of all six MQ adapters, all four
TSDB adapters and the three DB dialect adapters get one-line
maintainer docs (what binds to what, what is overridable).
- Nested support types across data/agentic/constant/public/facade:
NotifyBindKey, CacheTuning, Credential, dashboard VOs' Item/BucketVO,
ExpireListener (plus its null-doc placeholder javadoc replaced with a
real contract), agentic chat/stream VOs' choice/delta/usage shapes,
tool-context records, RequestHeader envelopes, FleetSummary,
visualization Type/Scale codes.
- PointValueSampleConverter's boundary methods now state the port-side
contract (quality default injection point, batch order preservation).
Comment drift fixed: the class javadoc of the can/mqtt/opc-da driver
skeletons claimed "see TODO markers in method bodies" — those markers
do not exist; the claims are corrected to plain work-in-progress notes.
The genuine TODOs (zigbee API verification, lwm2m lookup, ethernet-ip
CIP framing, CAN JNI) stay: they mark real unimplemented behavior.
Vendored-code boundary: the opc-da module carries a ported
org.openscada library (46 files, package org.openscada.*). Its trivial
accessors stay bare on purpose — filler comments on get/set pairs are
exactly the kind of noise this pass exists to remove — but the
non-obvious JIStruct wire-decoder methods (getStruct/fromStruct
FILETIME + three DO wrappers) now document the COM bridge shape.
Gates: full-repo compile green; data/model/constant/agentic/public
test suites green.
dc3-db-mariadb brings MariaDB 10.6+ behind dc3.db.type=mariadb with the
official mariadb-java-client and the same dual-dialect contract suite.
MariaDB shares the MySQL seed wholesale (utf8mb4_unicode_ci — the 0900
collation is MySQL-8-only and MariaDB rejects it outright, so the seed
translator and the hand-written 05 both moved to the portably-supported
collation, re-verified against mysql:8.4).
The mapper forks differ from the mysql twins in exactly one dimension,
verified against a live 10.11 container first: MariaDB never adopted
the AS-new ODKU row alias (syntax error on 10.11), so its 15 twins
reference VALUES(col) instead — everything else (GET_LOCK/RELEASE_LOCK,
FOR UPDATE SKIP LOCKED, recursive CTEs, INTERVAL ? SECOND, the
JSON_UNQUOTE(JSON_EXTRACT(...)) double-hop, REGEXP_REPLACE without the
g flag, DAYOFWEEK/HOUR units) proved compatible and rides the same
statement bodies.
Scope decision recorded in docs/db-dialects.md: DB2, H2 and HSQLDB are
explicitly out (dialect cost with zero demand; embedded test databases
unfit for multi-service concurrent IoT writes), SQLite equally unfit
for the multi-service shape (database-level write lock, no network
listener). Oracle and SQL Server stay on the roadmap with their real
obstacles spelled out (MERGE-heavy rewrite; no ARM64 mssql image for
local certification on Apple Silicon) — not started without a demand
signal.
Gates: db-tck 24/24 (postgres + mysql + mariadb, 8 each),
data/auth/manager suites green (274 in data), full-repo compile green.
R2c of docs/design/storage-abstraction.md §3. dc3-db-tck runs one set of
mapper-level assertions against both engines with the real mapper XML —
the databaseIdProvider routing is part of what gets certified, not just
the SQL: the fenced latest-value upsert's three-level tiebreak (fencing
token, then time, then sequence, then message id), the state upsert +
same-transaction re-select shape, the three-step expired-lease claim
(lock-and-read FOR UPDATE SKIP LOCKED, flip offline, derive the
post-update view), the lease upserts' row-local increments (token moves
only on ownership change), the row-level revision triggers (insert and
meaningful update bump, trivial update does not), and the tool-catalog
query's encoded-JSON double-hop.
Harness notes worth keeping: one SqlSessionFactory per database mirrors
the production @DS wiring (state/latest/leases/catalog span four
databases with unqualified table names); the mapper XML loads through
XMLMapperBuilder with the affectData mybatis-plus DTD extension
stripped; postgres registers the timestamptz type handler. Both
harnesses learned to distrust podman: log-following never fires, so
mysql waits on the listening TCP port and postgres polls a real JDBC
handshake (the port opens during initdb's local-only phase and closes
again for the restart). The postgres harness mounts seed files 01..07
individually — 00's age/pgvector extensions exist only in the
production base image — with 05 trimmed at runtime to the
dc3_point_latest projection (the hypertable half needs the base
image's TimescaleDB function set; the dialect contract doesn't touch
it).
The catalog fixture plants its api row through a parameterized insert:
the escaped-JSON value must not travel through SQL text (MySQL's
default backslash escapes eat the \" the value depends on) nor bind
uniformly (PG rejects varchar->json, MySQL rejects the binary
Types.OTHER binding) — the fixture branches on the engine.
Selection guide published at docs/db-dialects.md (dialect matrix,
swap procedure, the semantic decisions the suite locks); the storage
design's status header moves to Implemented with an R1/R2 closeout
note.
Gates: db-tck 16/16 (8 postgres + 8 mysql), data/auth/manager suites
green (274 in data), e2e 26/26, full-repo compile green.
R2b of docs/design/storage-abstraction.md §3:
- PostgreSQL seed: the three driver-lease sequences retire. The fencing
token and assignment version default to 1 and advance row-locally —
per-device / per-driver monotonicity is the property the checks rely
on, which the global sequences only delivered incidentally, and MySQL
has no sequence objects. The three statement-level revision triggers
(transition tables + INSERT .. ON CONFLICT, none of which MySQL can
express) collapse into one row-level function with row-local +1 upserts
— a shape both engines share.
- MySQL seed under dc3/dependencies/mysql/initdb/ (00-06), derived by
dc3/dependencies/mysql/pg2mysql_seed.py: schemas become databases,
TIMESTAMPTZ becomes DATETIME(6), TEXT/JSON literal defaults become
expression defaults (MySQL 8 refuses literal defaults on BLOB/TEXT/
JSON), operate-time plpgsql triggers become the ON UPDATE column
attribute (explicit SETs keep winning), the revision family becomes
row-level DELIMITER triggers, keyed TEXT columns widen to VARCHAR(191),
partial indexes drop, and the embedded-JSON seed values load under
NO_BACKSLASH_ESCAPES (MySQL string literals eat the backslashes the
escaped-JSON content values depend on; PostgreSQL does not). 05 is
hand-maintained: only the dc3_point_latest projection — a MySQL core
requires an external time-series store, so the hypertable and cagg DDL
have no counterpart by design.
- Compose: the optional stack gains a mysql:8.4 service (utf8mb4, seed
mounted at docker-entrypoint-initdb.d, healthcheck); the app stack
passes DC3_DB_TYPE through.
Verified by loading all seven files into a fresh mysql:8.4 container —
zero errors, five databases, table counts aligned with the PostgreSQL
seed (manager 24 incl. common, auth 24, data 12, agentic 6, history 1),
and the revision trigger proven live: insert bumps the owning driver's
revision to 1, an enable_flag flip bumps it again to 2.
Phase 3 of docs/design/tsdb-abstraction.md: the store family reaches four
certified adapters, the selection guide publishes the real capability
matrix, and external stores become first-class compose services.
dc3-tsdb-influxdb (3.11.2-core) talks the documented v3 HTTP APIs
directly — line protocol writes, query_sql CSV reads, zero client
dependencies. Integer fields carry the i suffix from the first write (a
bare number binds the column to Float64 forever and destroys ns
timestamps); timestamps cross the wire as CAST(... AS BIGINT)
nanoseconds because JSON renders them in scientific notation; PERCENTILE
is approximate-only and deleteRange does not exist in Core — both
declared honestly (false), the analytics facade computes exact
percentiles from bounded pulls.
dc3-tsdb-iotdb (2.0.10-standalone) maps series onto tree paths
root.dc3.t<tenant>.d<device>.p<point> — path nodes cannot be purely
numeric, so the design's raw-id mapping is amended. The server must run
with timestamp_precision=us and dn_rpc_address=0.0.0.0 (a two-line
properties override shipped under dc3/dependencies/iotdb); WHERE time
literals must use the 2026-08-20T12:59:59+00:00 form because bare epoch
numbers are parsed as milliseconds regardless of the server precision —
a microsecond literal silently matches nothing. Sessions disable
redirection: node discovery hands out internal cluster addresses that
break behind port mappings. GROUP BY LEVEL columns come back
wildcard-shaped and are parsed per column; COUNT rides the INT32 quality
measurement (never null); null measurements are omitted per record — a
null value in insertRecords makes the whole record vanish.
Capability matrix published in docs/tsdb-stores.md per adapter-declared
values (skip counts in the TCK are declared-unsupported cases, degraded
by facades, never wrong data). External store services (tdengine,
influxdb, iotdb) join the optional compose stack with healthchecks and
DC3_TSDB_* passthrough in the app stack.
Gates: timescale 24/24, tdengine 24/24 (2 skips), influxdb 24/24 (2
skips), iotdb 24/24 (3 skips); full-repo compile green.
The timescale adapter now owns the S16 lifecycle: raw 30 days (seed 05,
was 180) -> 1-minute tier 1 year (configurable via
dc3.tsdb.timescale.rollup.minute-keep-days) -> 1-hour tier forever.
bucketedAggregate and bucketedCount serve bucket widths at or above a
tier's granularity from that tier with exactly composable expressions
(AVG recombines as SUM(num_sum)/SUM(num_count), COUNT as
SUM(sample_count)); PERCENTILE and FIRST/LAST stay on the raw path — the
shared caggs carry textual first/last only, and tiered numeric first/last
would version-skew deployments created before such columns existed.
Single-window aggregate() stays raw by design: alarm windows must never
read through materialization lag.
The tiers are the observability pipeline's existing real-time continuous
aggregates (cagg_point_value_1m/1h, queried by Grafana) — one structure
serves both consumers; the adapter bootstraps the same names with an
identical shape via IF NOT EXISTS, so embedded seed boots, standalone
adapter boots and already-running deployments all converge on it without
rebuilding anything. Real-time mode (materialized_only=FALSE, no longer
the TS 2.13+ default) keeps tier reads correct immediately after an
append; the refresh policies only move aggregation work off the read
path. deleteRange refreshes both tiers afterwards.
Also fixed along the way: TDengine's bucketedAggregate PERCENTILE hit the
same "single table query" limit as aggregate() and now routes per-series
subtables.
Gates: timescale TCK 24/24 including the new rollup case (tiered
COUNT/AVG/LAST/bucketedCount and bucketed P50 all verified against raw
scans before any refresh materialized); TDengine TCK 24/24 with the same
case passing via honest raw degradation (2 declared-capability skips).
dc3-tsdb-tdengine maps the port onto TDengine 3.x: one supertable
point_value tagged by (tenant, device, point), one deterministic subtable
pv_<t>_<d>_<p> auto-created on first insert via USING TAGS, database
created with PRECISION 'us' and KEEP 180. Connection settings live under
dc3.tsdb.tdengine.* (REST driver, own Hikari pool); selected by
dc3.tsdb.type=tdengine. Like the broker family, only the default adapter
(timescale) ships inside consumers — switching deployments add this
dependency plus DC3_TSDB_TYPE/DC3_TSDB_TDENGINE_URL.
The supertable mapping grilled four real issues out of the port
surface, each fixed and locked by the TCK:
- Timestamps never touch string form. The REST driver serializes
Timestamp parameters in the client JVM zone while the server parses
them as UTC — every write would shift by the deployment zone and
cursor pagination would drift per page without end. Instants now
travel as epoch-micro integer literals and reads return
CAST(ts AS BIGINT) — verified symmetric against the image.
- AS value collides with a reserved word (agg_value now);
INTERVAL takes a bare number read in the database (micro) precision.
- PERCENTILE only runs on single tables, so single-series percentiles
query the deterministic subtable directly; tenant-wide percentile is
refused rather than approximated.
- REST readiness must be probed with POST /rest/sql + basic auth; the
GET path-style route 404s even on a healthy server.
Honest capabilities: latencyHistogram=false (dashboard degrades to
zero-filled bins via a capability check) and correlation=false (future
analytics facade computes from bucketed pulls); rollups stay NONE until
S16 lands stream computing. TCK result: 23 tests, 0 failures, 0 errors,
2 capability-gated skips; the timescale suite re-ran 23/23 unchanged.
The design doc's capability matrix now reflects the adapter's declared
values instead of the pre-implementation estimates.
Phase 1b of docs/design/tsdb-abstraction.md: every point-value read and
write now flows through the dc3-tsdb port, and dc3-common-repository is
deleted outright (no compat aliases).
Write path — new PointValueIngestService orchestration in dc3-common-data:
the stale-owner lease guard leaves the history INSERT's cross-schema join
and resolves the active owner per distinct device through the existing
DeviceFacade.getActiveOwner chain; a caffeine-backed ingest idempotency
window (dc3.data.ingest.*, marked only after both writes commit) absorbs
MQ redeliveries now that the message_id unique index is retired; the
batch lands via TsdbStore.append (natural series+time upsert, INGEST_ORDER
kept for the fenced dc3_point_latest upsert).
Read path — history via last(), page() resolves name/enable filters to
series keys through relational metadata (tenant-wide when unrestricted,
cursor offset emulation capped at 10k) with count() for totals; the alarm
window backend becomes TsdbWindowDataSource over aggregate()/history().
Dashboards — all nine data-side hypertable statements move to the S13
analytics facet (count/bucketedCount/countByDimension/latencyHistogram/
lastSeenPerSeries) plus the dc3_point_latest projection for latestStream;
the manager topology statement crosses into the data center via a new
ListSeriesVolumes RPC backed by the new S13-5 seriesCounts primitive (a
point shared by several devices needs per-series counts, which
single-dimension grouping cannot reconstruct); both cross-schema joins
are gone.
Port/adapter — seriesCounts added with a TCK case (23 total); series
filters compile to row-value IN lists chunked at 500 pairs; bootstrap
adds the quality column idempotently and retires uk_point_value_event so
existing deployments converge on the new upsert; the adapter binds the
application-provided tsdbDataSource bean (the history dynamic-datasource
entry) instead of the routing primary.
Timestamps — BO<->Instant conversion is pinned to
TimeConstant.DEFAULT_ZONEID (the platform canonical zone the latest
projection type handler already writes with); the design note's "lock
UTC" is amended accordingly, together with the lease-guard and S13-5
deviations recorded in §6.2.
Seed/deploy — 05-iot-dc3-history.sql gains the quality column and swaps
the unique index; compose carries DC3_TSDB_TYPE; make deploy publishes
dc3-tsdb.
Also fixed along the way: root pom never managed dc3-mq-core (latent,
masked by full-reactor builds); PointValueMapper ran unrouted on the
master datasource (now @DS("history")); latestStream name enrichment
looked up a Long-keyed map with String keys.
Gates: tsdb TCK 23/23, dc3-common-data 262/262, e2e 26/26 including the
unmodified PostgresHypertableIT; full reactor green except dc3-mq-tck's
kafka/pulsar container-startup flake, reproduced identically twice and
unrelated to this change.
Close out the mq abstraction rollout:
- docs/mq-brokers.md: user-facing broker selection guide — how to pick
(DC3_MQ_TYPE + connection settings), the six certified brokers, the
capability matrix, and the at-least-once/outbox notes
- .env.example + dc3/env/dev.env: DC3_MQ_TYPE and the per-adapter
connection variables with safe rabbitmq defaults
- docker-compose.yml: DC3_MQ_TYPE and adapter endpoints flow through the
shared x-app-runtime-env anchor so every service sees the selection
- adapter configurations resolve connection settings from the canonical
dc3.mq.* property first, then the DC3_MQ_* environment variable, then
the transport-standard property (spring.kafka.bootstrap-servers),
making containerized selection deterministic
The gated E2E gate passes unmodified through the port: RabbitDeliveryIT
6/6, CommandCallE2eIT 4/4, EventReportE2eIT 3/3 (full suite 24/24
including postgres and harness ITs) against testcontainers brokers.
pnoker