- let TSDB failures propagate in the alarm window data source: swallowing
them made an outage read as no-data and silently missed alarms
- dedupe ingest batches by natural key before store upserts (same-row
writes poison timescale ON CONFLICT batches)
- turn unsupported countByDimension into a structured refusal (analytics)
or capability-honest empty panel (dashboard) instead of a raw 500
- pin LocalDateTime.now to DEFAULT_ZONEID on the tsdb read paths
- adapter fixes: influxdb/iotdb/tdengine/timescale store and auto-config
corrections, iotdb system properties, capability declarations on
TsdbStore, new timescale store unit tests
- fill the missing Javadoc on public methods, records, and types across the dc3-common, dc3-mq, and dc3-tsdb core modules
- document about 350 public constants and enum constants in dc3-common-constant, including the MCP, RabbitMQ, agentic, manager, auth, and data surfaces
- document exception constructors and replace the lombok @RequiredArgsConstructor with explicit documented constructors in ThreadPoolConfig and QuartzService
- translate the remaining non-English section comments in the TSDB store adapters
- verified: check_documentation.py passes, javadoc doclint=all reports zero warnings across all aggregators, and the touched modules compile
- add READMEs for the dc3-mq core, six broker adapters, and the contract suite
- add READMEs for the dc3-tsdb core, four store adapters, and the contract suite
- add READMEs for the dc3-db core, three dialect adapters, and the contract suite
A comment-health audit driven by an AST scanner (regex scanners kept
false-positiving on annotation-separated javadoc; the AST pass with
constructor and @Override-implementation exemptions is what produced a
trustworthy worklist) ended with a much smaller true gap than the raw
numbers suggested: class-level javadoc is already at 100% repository-wide.
What actually gets filled here:
- AnalyticsModel: all 30 nested records of the S19 analytics facet get
maintainer-voice javadoc (the @Schema descriptions serve the API side;
these serve code readers).
- The whole entity/ext family (29 files, 47 nested classes): every
Content/Ui/Validation/Security/AppliesTo/Template/Dedup/... schema
class states what it actually holds — read from the fields, not
templated.
- Adapter families: the @Bean methods of all six MQ adapters, all four
TSDB adapters and the three DB dialect adapters get one-line
maintainer docs (what binds to what, what is overridable).
- Nested support types across data/agentic/constant/public/facade:
NotifyBindKey, CacheTuning, Credential, dashboard VOs' Item/BucketVO,
ExpireListener (plus its null-doc placeholder javadoc replaced with a
real contract), agentic chat/stream VOs' choice/delta/usage shapes,
tool-context records, RequestHeader envelopes, FleetSummary,
visualization Type/Scale codes.
- PointValueSampleConverter's boundary methods now state the port-side
contract (quality default injection point, batch order preservation).
Comment drift fixed: the class javadoc of the can/mqtt/opc-da driver
skeletons claimed "see TODO markers in method bodies" — those markers
do not exist; the claims are corrected to plain work-in-progress notes.
The genuine TODOs (zigbee API verification, lwm2m lookup, ethernet-ip
CIP framing, CAN JNI) stay: they mark real unimplemented behavior.
Vendored-code boundary: the opc-da module carries a ported
org.openscada library (46 files, package org.openscada.*). Its trivial
accessors stay bare on purpose — filler comments on get/set pairs are
exactly the kind of noise this pass exists to remove — but the
non-obvious JIStruct wire-decoder methods (getStruct/fromStruct
FILETIME + three DO wrappers) now document the COM bridge shape.
Gates: full-repo compile green; data/model/constant/agentic/public
test suites green.
Phase 3 of docs/design/tsdb-abstraction.md: the store family reaches four
certified adapters, the selection guide publishes the real capability
matrix, and external stores become first-class compose services.
dc3-tsdb-influxdb (3.11.2-core) talks the documented v3 HTTP APIs
directly — line protocol writes, query_sql CSV reads, zero client
dependencies. Integer fields carry the i suffix from the first write (a
bare number binds the column to Float64 forever and destroys ns
timestamps); timestamps cross the wire as CAST(... AS BIGINT)
nanoseconds because JSON renders them in scientific notation; PERCENTILE
is approximate-only and deleteRange does not exist in Core — both
declared honestly (false), the analytics facade computes exact
percentiles from bounded pulls.
dc3-tsdb-iotdb (2.0.10-standalone) maps series onto tree paths
root.dc3.t<tenant>.d<device>.p<point> — path nodes cannot be purely
numeric, so the design's raw-id mapping is amended. The server must run
with timestamp_precision=us and dn_rpc_address=0.0.0.0 (a two-line
properties override shipped under dc3/dependencies/iotdb); WHERE time
literals must use the 2026-08-20T12:59:59+00:00 form because bare epoch
numbers are parsed as milliseconds regardless of the server precision —
a microsecond literal silently matches nothing. Sessions disable
redirection: node discovery hands out internal cluster addresses that
break behind port mappings. GROUP BY LEVEL columns come back
wildcard-shaped and are parsed per column; COUNT rides the INT32 quality
measurement (never null); null measurements are omitted per record — a
null value in insertRecords makes the whole record vanish.
Capability matrix published in docs/tsdb-stores.md per adapter-declared
values (skip counts in the TCK are declared-unsupported cases, degraded
by facades, never wrong data). External store services (tdengine,
influxdb, iotdb) join the optional compose stack with healthchecks and
DC3_TSDB_* passthrough in the app stack.
Gates: timescale 24/24, tdengine 24/24 (2 skips), influxdb 24/24 (2
skips), iotdb 24/24 (3 skips); full-repo compile green.
The timescale adapter now owns the S16 lifecycle: raw 30 days (seed 05,
was 180) -> 1-minute tier 1 year (configurable via
dc3.tsdb.timescale.rollup.minute-keep-days) -> 1-hour tier forever.
bucketedAggregate and bucketedCount serve bucket widths at or above a
tier's granularity from that tier with exactly composable expressions
(AVG recombines as SUM(num_sum)/SUM(num_count), COUNT as
SUM(sample_count)); PERCENTILE and FIRST/LAST stay on the raw path — the
shared caggs carry textual first/last only, and tiered numeric first/last
would version-skew deployments created before such columns existed.
Single-window aggregate() stays raw by design: alarm windows must never
read through materialization lag.
The tiers are the observability pipeline's existing real-time continuous
aggregates (cagg_point_value_1m/1h, queried by Grafana) — one structure
serves both consumers; the adapter bootstraps the same names with an
identical shape via IF NOT EXISTS, so embedded seed boots, standalone
adapter boots and already-running deployments all converge on it without
rebuilding anything. Real-time mode (materialized_only=FALSE, no longer
the TS 2.13+ default) keeps tier reads correct immediately after an
append; the refresh policies only move aggregation work off the read
path. deleteRange refreshes both tiers afterwards.
Also fixed along the way: TDengine's bucketedAggregate PERCENTILE hit the
same "single table query" limit as aggregate() and now routes per-series
subtables.
Gates: timescale TCK 24/24 including the new rollup case (tiered
COUNT/AVG/LAST/bucketedCount and bucketed P50 all verified against raw
scans before any refresh materialized); TDengine TCK 24/24 with the same
case passing via honest raw degradation (2 declared-capability skips).
dc3-tsdb-tdengine maps the port onto TDengine 3.x: one supertable
point_value tagged by (tenant, device, point), one deterministic subtable
pv_<t>_<d>_<p> auto-created on first insert via USING TAGS, database
created with PRECISION 'us' and KEEP 180. Connection settings live under
dc3.tsdb.tdengine.* (REST driver, own Hikari pool); selected by
dc3.tsdb.type=tdengine. Like the broker family, only the default adapter
(timescale) ships inside consumers — switching deployments add this
dependency plus DC3_TSDB_TYPE/DC3_TSDB_TDENGINE_URL.
The supertable mapping grilled four real issues out of the port
surface, each fixed and locked by the TCK:
- Timestamps never touch string form. The REST driver serializes
Timestamp parameters in the client JVM zone while the server parses
them as UTC — every write would shift by the deployment zone and
cursor pagination would drift per page without end. Instants now
travel as epoch-micro integer literals and reads return
CAST(ts AS BIGINT) — verified symmetric against the image.
- AS value collides with a reserved word (agg_value now);
INTERVAL takes a bare number read in the database (micro) precision.
- PERCENTILE only runs on single tables, so single-series percentiles
query the deterministic subtable directly; tenant-wide percentile is
refused rather than approximated.
- REST readiness must be probed with POST /rest/sql + basic auth; the
GET path-style route 404s even on a healthy server.
Honest capabilities: latencyHistogram=false (dashboard degrades to
zero-filled bins via a capability check) and correlation=false (future
analytics facade computes from bucketed pulls); rollups stay NONE until
S16 lands stream computing. TCK result: 23 tests, 0 failures, 0 errors,
2 capability-gated skips; the timescale suite re-ran 23/23 unchanged.
The design doc's capability matrix now reflects the adapter's declared
values instead of the pre-implementation estimates.
Phase 1b of docs/design/tsdb-abstraction.md: every point-value read and
write now flows through the dc3-tsdb port, and dc3-common-repository is
deleted outright (no compat aliases).
Write path — new PointValueIngestService orchestration in dc3-common-data:
the stale-owner lease guard leaves the history INSERT's cross-schema join
and resolves the active owner per distinct device through the existing
DeviceFacade.getActiveOwner chain; a caffeine-backed ingest idempotency
window (dc3.data.ingest.*, marked only after both writes commit) absorbs
MQ redeliveries now that the message_id unique index is retired; the
batch lands via TsdbStore.append (natural series+time upsert, INGEST_ORDER
kept for the fenced dc3_point_latest upsert).
Read path — history via last(), page() resolves name/enable filters to
series keys through relational metadata (tenant-wide when unrestricted,
cursor offset emulation capped at 10k) with count() for totals; the alarm
window backend becomes TsdbWindowDataSource over aggregate()/history().
Dashboards — all nine data-side hypertable statements move to the S13
analytics facet (count/bucketedCount/countByDimension/latencyHistogram/
lastSeenPerSeries) plus the dc3_point_latest projection for latestStream;
the manager topology statement crosses into the data center via a new
ListSeriesVolumes RPC backed by the new S13-5 seriesCounts primitive (a
point shared by several devices needs per-series counts, which
single-dimension grouping cannot reconstruct); both cross-schema joins
are gone.
Port/adapter — seriesCounts added with a TCK case (23 total); series
filters compile to row-value IN lists chunked at 500 pairs; bootstrap
adds the quality column idempotently and retires uk_point_value_event so
existing deployments converge on the new upsert; the adapter binds the
application-provided tsdbDataSource bean (the history dynamic-datasource
entry) instead of the routing primary.
Timestamps — BO<->Instant conversion is pinned to
TimeConstant.DEFAULT_ZONEID (the platform canonical zone the latest
projection type handler already writes with); the design note's "lock
UTC" is amended accordingly, together with the lease-guard and S13-5
deviations recorded in §6.2.
Seed/deploy — 05-iot-dc3-history.sql gains the quality column and swaps
the unique index; compose carries DC3_TSDB_TYPE; make deploy publishes
dc3-tsdb.
Also fixed along the way: root pom never managed dc3-mq-core (latent,
masked by full-reactor builds); PointValueMapper ran unrouted on the
master datasource (now @DS("history")); latestStream name enrichment
looked up a Long-keyed map with String keys.
Gates: tsdb TCK 23/23, dc3-common-data 262/262, e2e 26/26 including the
unmodified PostgresHypertableIT; full reactor green except dc3-mq-tck's
kafka/pulsar container-startup flake, reproduced identically twice and
unrelated to this change.
First vertical slice of the tsdb abstraction (docs/design/
tsdb-abstraction.md phase 1): the dc3-tsdb top-level family lands with
the S19-final port and a fully certified TimescaleDB adapter.
dc3-tsdb-core — the port, zero store dependencies:
- TsdbModel: SeriesKey, the unified SeriesFilter (single series / series
set / tenant-wide as one shape), PointValueSample with both timestamps
(S9) and the quality flag (S17), AggregateFunction incl. FIRST/LAST
(M4) and gated PERCENTILE, cursor records, analytics records
- TsdbStore SPI: append / last / cursor history / aggregate /
bucketedAggregate / count, the S13 analytics facet
(bucketedCount, countByDimension, lastSeenPerSeries,
latencyHistogram), listSeries, deleteRange, correlation; every read
carries TsdbDeadline
dc3-tsdb-timescale — the reference adapter:
- unnest single-statement batch append (one round trip) with natural
upsert on (series, deviceTime); batches chunked at the declared
maxAppendBatch
- read paths through SeriesFilter-shaped SQL: ROW_NUMBER per-series
last-N, global (create_time, message_id) descending cursor history,
time_bucket bucketed aggregates, percentile_cont, aligned-bucket
corr() correlation, CASE-binned latency histogram
- idempotent schema bootstrap incl. initial-chunk priming (a sentinel
row at a fixed early instant forces TimescaleDB's initial chunk
creation at bootstrap with a controlled boundary)
- capability negotiation logged at startup; rollupSupport NONE in this
extraction (S16 continuous aggregates arrive with phase 2)
dc3-tsdb-tck — the 22-case contract suite on Testcontainers:
append-readback fidelity (every field incl. both timestamps and
quality), newest-first last-N with exact limit, cursor pagination
without skip or duplicate, NULL-skipping aggregates, epoch-anchored
bucket boundaries, series and tenant-wide counts, duplicate-timestamp
last-write-wins, backfill acceptance, cross-tenant isolation,
microsecond precision, 5k-sample burst, the four analytics ops,
multi-series isolation, FIRST/LAST M4, percentile tolerance, quality
round-trip, deadline-bounded reads, and known-correlation detection.
Two debugging lessons worth recording (both fixed and TCK-locked):
- Spring's RowCallbackHandler fires once PER ROW; a while(rs.next())
inside it silently skips every other row — the original cause of all
'vanishing row' symptoms, initially misattributed to TimescaleDB
- TimescaleDB sizes the initial hypertable chunk around the first
inserted row; priming at bootstrap avoids boundary anomalies
Timescale contract suite: 22/22 green against timescale-ha:pg18.
pnoker