five drivers carried private bytesToHex copies and three carried
hexToBytes variants, while mbus/dlt645 frames each implemented the same
sum checksum:
- codecutil.bytesToHex now uses hexformat uppercase, matching what all
five private copies already emitted (zero display change)
- codecutil.hexToBytes added: whitespace-tolerant, blank-to-empty,
mixed-case input
- codecutil.sumChecksum added as the shared mbus/dlt645 frame checksum
and both frame classes delegate to it
- dlt645's inline hex loop replaced by codecutil.hexTobytes
- fins' private bcd copy kept deliberately: codecutil's bcd variant
strips leading zeros, a different protocol semantic
- springdoc: declare api-docs/swagger-ui enabled explicitly; the boot
reminder only fires on unset properties, so the intent-declaring
values keep behavior identical and quiet all six services
- netty: add the macos dns native resolver to the gateway so the
resolver does not log an error at boot on mac dev machines (the
no-classifier jar bundles every arch and is inert on linux)
- hibernate-validator HV000271: spring-cloud-gateway 4.3.0 still uses
container-level @Valid on its List properties. logger-level rules
cannot suppress it because spring boot's logging re-init installs a
second loggercontext that resets them, so drop the exact message
with a turbo filter attached to every context loading this config;
the class documents the upstream trigger for removal
live verification: five services plus gateway boot with zero warn and
zero error lines, virtual driver included; salt/login probe 200
sweep every module to satisfy the new method-name rule, renaming
about 40 methods together with their interface, implementation, test
and cli call sites: store find() lookups become get(), analytics
queryLatest/queryHistory become getLatest/getHistory, sql helper
query() becomes spec()/sql(), jwks fetch() becomes load(), and the
cursor record component queryDigest becomes requestDigest
no behavior changes; full unit suite (5278 tests) and both quality
gates stay green
a live-stack sweep of all 36 drivers surfaced five sdk-level defects
that left reads dead or the whole fleet unbootable:
- drop the empty spring.main stub in the shared driver profile; the
boot 4.1 binder rejects null-valued keys and every driver died at
startup with a bind exception
- rename the schedule enable flag to enable to match what every driver
application.yml actually spells; with the old enabled key the read
and custom quartz jobs were silently never scheduled
- treat proto3 zero-versions as unset when reconciling registration
so a driver restart re-registers instead of failing the optimistic
lock with 'driver version conflict'
- ignore the driver's own registration event that can be consumed
before applymetadata finishes instead of logging an error
- initialize jdbc/serial connection maps eagerly: the registration
event handler invalidates pools and nped on a lazy-null map
- ble: exclude the guava 18 pin dragged in by bluetooth-manager
- kafka: use the starter (boot 4 split autoconfigurations out of the
bare jar) and read the compose-provided bootstrap variable
- lwm2m: pin californium to the 3.x api leshan m14 needs and
configure an explicit coap endpoint
- mqtt: replace the orphaned spring-integration gateway with a paho
publisher so the send path exists again
- boot sweep result: 36/36 drivers start, 0 errors
the client_credentials grant could never complete: the authorization
insert left principal_type null, state/code/refresh columns were
declared not null for a grant that never sets them, and the connection
grant type silently fell back to authorization_code because builders
match on the enum value, not the name
- stamp the service-account principal type on the cc authorization
- bind explicit nulls for grant-specific columns in insertauthorization
- make state_hash, authorization_code_hash and the refresh hashes
nullable in seed ddl and resync the embedded schema fingerprint
the /api/v3/auth/token/** wildcard also routed check and cancel
without the authentic filter, so those @PreAuthorize endpoints could
never establish a caller identity and always returned 401 (the logout
denylist never fired); enumerate salt, generate and change_password
explicitly and let the rest fall through to the authenticated route
token/cancel accepted any name+tenant pair, so a principal holding
token:delete could log out arbitrary users; compare the resolved
credential against the caller principal and deny otherwise
role_principal_bind:add alone could attach the wildcard administrator
role to any principal, and role_resource_bind:add could attach the
wildcard resource to a held role; require the caller to already hold
the granted role/resource code or the wildcard authority before the
insert proceeds (permission resolution goes through the same
authpermissionprovider used by @perm.can)
the public token/change_password endpoint verified the current
password but never recorded failed attempts, leaving it a brute-force
surface the login endpoint does not have; route wrong-password
attempts through recordFailed so the existing lockout applies
the compact constructors rejected a null tenantId during jackson
deserialization, so any authenticated request body with filters failed
with http 400 before withTenant could inject the caller tenant; keep
only the page-bounds validation at parse time
the command/event detail drawers render tenantId but the vos never
returned it; add a server-resolved read-only field (inbound binding
stays ignored by the existing mapstruct mapping) and refresh the
openapi-manager snapshot
- command_history/get_by_record_id now binds the declared wire name
record_id instead of the raw parameter name recordId, matching the
event_history twin and the frontend/CLI clients
- point_command_history/get_by_command_id likewise binds command_id,
and the accepted statusUri built by point_command write now uses it
- openapi-data snapshot updated for both query params
- cli command history and web mock handlers aligned to the same names
- web mock registers command_history under the data service prefix
Backfill the one-line Javadoc required by check_documentation on public
top-level types and non-override public methods (1305 sites across 313
files), matching the existing single-line contract style: each doc states
the behaviour contract without @param/@return tags.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Relocate all 64 R2dbc*Store implementations from dc3-common-{auth,manager,data,agentic} repositories into new dc3-db-postgres-{auth,manager,data,agentic} adapter modules (package io.github.pnoker.db.postgres.<domain>), each registered through its own AutoConfiguration. Reactive*Store ports and filter types stay in the business modules; services are untouched. Drops stale MyBatis wording from AGENTS.md and init runner javadoc.
- migrate auth, manager, data, and agentic persistence to R2DBC
- hard cut pagination, facade, gateway, web, and CLI contracts
- remove legacy database and TSDB adapters and document validation gates
BREAKING CHANGE: platform persistence now requires PostgreSQL 18 with TimescaleDB and the new reactive API contracts.
- gateway container env passes DC3_GATEWAY_OAUTH_ENABLED/JWKS_URL so
deployments can turn RS256 verification on without image changes
- PrincipalHeader regains a 7-arg constructor delegating scopes=null,
keeping ~15 existing call sites compiling against the new field
- dc3-common-auth imports com.baomidou.mybatisplus.spring.service
directly since the 3.5.17 split but never declared the artifact;
out-of-reactor consumers hit FileNotFoundException at boot. Manage
it in the root BOM alongside the existing entry.
- ci-cli.yml gates dc3-cli (lint, test, build, help smoke) on PRs
touching dc3-cli/**
- adapt AuthenticGatewayFilterTest to the extended constructor
Phase 2 of token-unification-mcp-first-cli.md:
- capScopesByRbac narrows requested scopes at authorization time to
what the principal's bound resources cover via the tool catalog
(additive per risk class; never widens). principals with no
bindings degrade to legacy behaviour so incomplete RBAC setups
cannot silently zero out clients; toggle dc3.oauth.rbac-scoped-scopes
- risk-ladder lifetime: call-capable tickets keep the 15m default,
list/read-only tickets get a new readOnlyAccessTokenTtl of 1h
Phase 1 of token-unification-mcp-first-cli.md:
- shared OAuthJwtVerifier in dc3-common-public; auth center's
parseAccessToken now delegates to it
- gateway TokenResolver chain: optional OAuthTokenResolver verifies
bearer tickets against a cached JWKS (unknown kid forces one
refresh) and projects claims onto the extended PrincipalHeader
(new scopes field); behind dc3.gateway.oauth.enabled, default off
- platform login-token lifetime configurable via DC3_TOKEN_TTL_HOURS /
dc3.token.ttl-hours; denylist cache follows ttl+1h so revocation
keeps outliving tokens
Replace limit-truncated listing with real server-side paging
(LIMIT/OFFSET plus a shared count query) behind Page envelopes:
tool/list accepts a nested Pages block and audit/list takes
current/size params, both keeping the legacy limit as a compatibility
page size. The frontend configs forward pager state and consume
{records, total} instead of faking a single page; the assignment
drawer keeps its full-catalog semantics with one 500-row page (the
service cap). Adds a mapper XML grammar gate and audit page/count
symmetry tests.
Co-Authored-By: Claude <noreply@anthropic.com>
The "(? IS NULL OR ? = '' OR ...)" filter shape breaks on Postgres with
stringtype=unspecified: a bare bind parameter in an IS NULL context has
no inferable type, so opening the MCP tool page with a blank risk-level
filter failed with "could not determine data type of parameter $1". The
dynamic <if> form drops the dead branches entirely and is
dialect-neutral, so the mysql/mariadb forks of the statement are gone.
Co-Authored-By: Claude <noreply@anthropic.com>
affectData is not declared for insert by the MyBatis mapper DTD, so
the attribute only surfaces as a startup error after the affected
module deploys. Remove it from every upsertLeaseState variant and
document that the atomically advanced assignment version is re-selected
in the same transaction rather than returned.
Add ManagerMapperXmlGrammarTest to parse every mapping/*.xml against
the bundled DTD so future invalid attributes fail the build instead of
application startup.
Co-Authored-By: Claude <noreply@anthropic.com>
The auth filter sits in front of the routing chain, so TCP connect
refusals and timeouts propagate through it and masquerade as an
authentication failure answered with 500. Classify ConnectException
and ConnectTimeoutException (including wrapped causes) as downstream
unavailability and answer 503 Service Unavailable instead.
Co-Authored-By: Claude <noreply@anthropic.com>
The tsdbDataSource bean breaks by-type injection: the dynamic routing
dataSource carries no @Primary, so a candidate-eligible second
DataSource throws NoUniqueBeanDefinitionException in consumers that
cannot take a qualifier (MyBatis-Plus sqlSessionFactory, Quartz, the
health check). Mark it defaultCandidate = false so those consumers see
exactly one candidate again while the TSDB adapter keeps resolving it
through @Qualifier("tsdbDataSource").
The same second bean makes @ConditionalOnSingleCandidate(DataSource)
fail, so DataSourceTransactionManagerAutoConfiguration silently backs
off and no TransactionTemplate is auto-configured. Declare the
DataSourceTransactionManager explicitly, bound to the routing
datasource with local-transaction semantics: switching routes inside
an open transaction does not take effect.
Co-Authored-By: Claude <noreply@anthropic.com>
- Spring Boot 4.0.7 -> 4.1.1, Spring Cloud 2025.1.2 -> 2025.1.3,
Spring AI 2.0.0 -> 2.0.1, springdoc 3.0.3 -> 3.1.0
- Migrate gRPC starters to the Boot-owned spring-boot-starter-grpc-client/server
line (org.springframework.grpc starter stops at 1.0.3 and mixes badly with the
spring-grpc-core 1.1.1 managed by the Boot 4.1 BOM), and move the
spring.grpc.channels.* YAML keys to the new spring.grpc.channel.* format
- Refresh driver protocol libraries (modbus4j 3.1.0, jSerialComm 2.11.4,
bacnet4j 6.1.1, gurux-dlms 4.0.96, j60870 1.8.0, snmp4j 3.13.1) and drop the
flow-control overrides removed from the modbus4j SerialPortWrapper interface
- Refresh jackson 2.x/3.x, protobuf, okhttp, micrometer, oshi, sqlite-jdbc,
swagger-core, instancio, mockito, logstash-logback-encoder 9.0 and the
graalvm native plugin; keep commons-io (broken timestamp-version candidate)
and j-interop 2.0.4 (legacy major) pinned
Co-Authored-By: Claude <noreply@anthropic.com>
- let TSDB failures propagate in the alarm window data source: swallowing
them made an outage read as no-data and silently missed alarms
- dedupe ingest batches by natural key before store upserts (same-row
writes poison timescale ON CONFLICT batches)
- turn unsupported countByDimension into a structured refusal (analytics)
or capability-honest empty panel (dashboard) instead of a raw 500
- pin LocalDateTime.now to DEFAULT_ZONEID on the tsdb read paths
- adapter fixes: influxdb/iotdb/tdengine/timescale store and auto-config
corrections, iotdb system properties, capability declarations on
TsdbStore, new timescale store unit tests
- derive pagination DbType from the selected dc3-db adapter with a fail-fast
when a non-default dc3.db.type lacks its adapter jar
- skip postgres profile activation for mariadb, not just mysql
- fork OAuthMcpMapper/AlertMapper statements for mysql and mariadb
(CAST AS JSON, ::text casts, ODKU value references)
- honor GET_LOCK verdict in resource registry sync instead of assuming
acquisition on every engine
- pg2mysql_seed.py: convert partial unique indexes to stored generated guard
columns instead of dropping them; add 99-iot-dc3-grants.sql
- add DC3_DB_URL/DC3_DB_HISTORY_URL per-service overrides in center configs
- cover the new mapper forks in the db TCK; fix AGPL headers in spring
metadata files
- add the canonical AGPL header to the six agentic application yml files and the two python scripts that lacked one
- add the header to all 79 kustomize, helm, and grafana yaml files, including the k8s driver template which regenerates the 36 driver manifests
- verified: the k8s yaml files parse (multi-document aware), the python scripts compile, and the regenerated drivers differ only by the header
- fill the missing Javadoc on public methods, records, and types across the dc3-common, dc3-mq, and dc3-tsdb core modules
- document about 350 public constants and enum constants in dc3-common-constant, including the MCP, RabbitMQ, agentic, manager, auth, and data surfaces
- document exception constructors and replace the lombok @RequiredArgsConstructor with explicit documented constructors in ThreadPoolConfig and QuartzService
- translate the remaining non-English section comments in the TSDB store adapters
- verified: check_documentation.py passes, javadoc doclint=all reports zero warnings across all aggregators, and the touched modules compile
- add ble, can, dlt645, dnp3, iec61850, kafka, knx, lorawan, mbus, redis, and zigbee to the k8s driver list, kustomization, helm values, and compose service stacks
- regenerate k8s driver deployments and preserve the listening-virtual ports in the generator
- replace the per-driver named volumes with one shared driver_data volume isolated by mount path, matching the scale/swarm stacks
- correct scaling docs: compose scale/swarm drivers stay at one replica because replicas would share one SQLite outbox file
Follow-up review of the comment pass found three spots where new
javadoc landed next to pre-existing blocks:
- AnalyticsModel: SeriesSelector/TimeRange/HistoryPoint carried the same
doc twice (the batch injector did not notice the originals).
- MybatisPlusConfig: an orphaned copy of the old mybatisPlusInterceptor
doc sat above databaseIdProvider's own javadoc.
- EntityStateMapper: the parameter javadoc of the retired
claimExpiredDevices method was left dangling above
selectExpiredForClaim's doc.
All three removed; the original mybatisPlusInterceptor doc survives
(the one-line summary I added earlier was replaced by restoring the
full contract note in place). Verified by a repository-wide scan for
adjacent javadoc pairs — now zero. data suite green, compile green.
A comment-health audit driven by an AST scanner (regex scanners kept
false-positiving on annotation-separated javadoc; the AST pass with
constructor and @Override-implementation exemptions is what produced a
trustworthy worklist) ended with a much smaller true gap than the raw
numbers suggested: class-level javadoc is already at 100% repository-wide.
What actually gets filled here:
- AnalyticsModel: all 30 nested records of the S19 analytics facet get
maintainer-voice javadoc (the @Schema descriptions serve the API side;
these serve code readers).
- The whole entity/ext family (29 files, 47 nested classes): every
Content/Ui/Validation/Security/AppliesTo/Template/Dedup/... schema
class states what it actually holds — read from the fields, not
templated.
- Adapter families: the @Bean methods of all six MQ adapters, all four
TSDB adapters and the three DB dialect adapters get one-line
maintainer docs (what binds to what, what is overridable).
- Nested support types across data/agentic/constant/public/facade:
NotifyBindKey, CacheTuning, Credential, dashboard VOs' Item/BucketVO,
ExpireListener (plus its null-doc placeholder javadoc replaced with a
real contract), agentic chat/stream VOs' choice/delta/usage shapes,
tool-context records, RequestHeader envelopes, FleetSummary,
visualization Type/Scale codes.
- PointValueSampleConverter's boundary methods now state the port-side
contract (quality default injection point, batch order preservation).
Comment drift fixed: the class javadoc of the can/mqtt/opc-da driver
skeletons claimed "see TODO markers in method bodies" — those markers
do not exist; the claims are corrected to plain work-in-progress notes.
The genuine TODOs (zigbee API verification, lwm2m lookup, ethernet-ip
CIP framing, CAN JNI) stay: they mark real unimplemented behavior.
Vendored-code boundary: the opc-da module carries a ported
org.openscada library (46 files, package org.openscada.*). Its trivial
accessors stay bare on purpose — filler comments on get/set pairs are
exactly the kind of noise this pass exists to remove — but the
non-obvious JIStruct wire-decoder methods (getStruct/fromStruct
FILETIME + three DO wrappers) now document the COM bridge shape.
Gates: full-repo compile green; data/model/constant/agentic/public
test suites green.
Apply queryTimeout as a statement timeout instead of a connection timeout, validate point queries against the point read/write flag, close all connection pools on driver metadata events, and remove the unused DriverSenderService dependency. Rename the module unit test so surefire executes it (26 tests) instead of excluding it via the Abstract* pattern.
dc3-common-jdbc was the family's core in disguise — MyBatis-Plus wiring,
the tenant-line handler, the databaseIdProvider, the jdbc profile. It now
lives as dc3-db/dc3-db-core, matching the dc3-mq-core / dc3-tsdb-core
shape: one aggregator, port first, adapters after, tck last. Artifact
renamed, packages deliberately untouched (same convention as the mq
family move); twelve poms rewired, the family aggregator puts core
first, and the db-tck now depends on the core explicitly instead of
riding transitives.
dc3-common-sql stays where it is on purpose: it is not platform dialect
plumbing but the base class of the four JDBC *device drivers*
(mysql/oracle/postgresql/sqlserver drivers that connect to external
databases being collected from) and depends on the driver SDK — under
dc3-db it would claim a meaning it does not have.
Gates: db-tck 24/24 (postgres + mysql + mariadb), data 274/274,
full-repo compile green.
dc3-db-mariadb brings MariaDB 10.6+ behind dc3.db.type=mariadb with the
official mariadb-java-client and the same dual-dialect contract suite.
MariaDB shares the MySQL seed wholesale (utf8mb4_unicode_ci — the 0900
collation is MySQL-8-only and MariaDB rejects it outright, so the seed
translator and the hand-written 05 both moved to the portably-supported
collation, re-verified against mysql:8.4).
The mapper forks differ from the mysql twins in exactly one dimension,
verified against a live 10.11 container first: MariaDB never adopted
the AS-new ODKU row alias (syntax error on 10.11), so its 15 twins
reference VALUES(col) instead — everything else (GET_LOCK/RELEASE_LOCK,
FOR UPDATE SKIP LOCKED, recursive CTEs, INTERVAL ? SECOND, the
JSON_UNQUOTE(JSON_EXTRACT(...)) double-hop, REGEXP_REPLACE without the
g flag, DAYOFWEEK/HOUR units) proved compatible and rides the same
statement bodies.
Scope decision recorded in docs/db-dialects.md: DB2, H2 and HSQLDB are
explicitly out (dialect cost with zero demand; embedded test databases
unfit for multi-service concurrent IoT writes), SQLite equally unfit
for the multi-service shape (database-level write lock, no network
listener). Oracle and SQL Server stay on the roadmap with their real
obstacles spelled out (MERGE-heavy rewrite; no ARM64 mssql image for
local certification on Apple Silicon) — not started without a demand
signal.
Gates: db-tck 24/24 (postgres + mysql + mariadb, 8 each),
data/auth/manager suites green (274 in data), full-repo compile green.
R2c of docs/design/storage-abstraction.md §3. dc3-db-tck runs one set of
mapper-level assertions against both engines with the real mapper XML —
the databaseIdProvider routing is part of what gets certified, not just
the SQL: the fenced latest-value upsert's three-level tiebreak (fencing
token, then time, then sequence, then message id), the state upsert +
same-transaction re-select shape, the three-step expired-lease claim
(lock-and-read FOR UPDATE SKIP LOCKED, flip offline, derive the
post-update view), the lease upserts' row-local increments (token moves
only on ownership change), the row-level revision triggers (insert and
meaningful update bump, trivial update does not), and the tool-catalog
query's encoded-JSON double-hop.
Harness notes worth keeping: one SqlSessionFactory per database mirrors
the production @DS wiring (state/latest/leases/catalog span four
databases with unqualified table names); the mapper XML loads through
XMLMapperBuilder with the affectData mybatis-plus DTD extension
stripped; postgres registers the timestamptz type handler. Both
harnesses learned to distrust podman: log-following never fires, so
mysql waits on the listening TCP port and postgres polls a real JDBC
handshake (the port opens during initdb's local-only phase and closes
again for the restart). The postgres harness mounts seed files 01..07
individually — 00's age/pgvector extensions exist only in the
production base image — with 05 trimmed at runtime to the
dc3_point_latest projection (the hypertable half needs the base
image's TimescaleDB function set; the dialect contract doesn't touch
it).
The catalog fixture plants its api row through a parameterized insert:
the escaped-JSON value must not travel through SQL text (MySQL's
default backslash escapes eat the \" the value depends on) nor bind
uniformly (PG rejects varchar->json, MySQL rejects the binary
Types.OTHER binding) — the fixture branches on the engine.
Selection guide published at docs/db-dialects.md (dialect matrix,
swap procedure, the semantic decisions the suite locks); the storage
design's status header moves to Implemented with an R1/R2 closeout
note.
Gates: db-tck 16/16 (8 postgres + 8 mysql), data/auth/manager suites
green (274 in data), e2e 26/26, full-repo compile green.
R2a of docs/design/storage-abstraction.md §3. MySQL 8 becomes a
first-class relational deployment behind dc3.db.type=mysql:
- dc3-db-mysql module: mysql-connector-j, MYSQL pagination DbType, the
mysql profile (driver + jdbc-type-for-null), activated only when
dc3.db.type=mysql — the postgres profile and DbType stay the default
for unset values, so existing deployments are untouched. The neutral
jdbc module now exposes a VendorDatabaseIdProvider (postgres/mysql
aliases) so mapper XML can fork statements by engine.
- RETURNING decoupled where MySQL has no equivalent: the state upsert
and the lease-state upsert no longer RETURN rows — callers re-select
by the natural key in the same transaction (portable shape, one extra
keyed read on both engines). The expired-lease claim becomes three
portable steps in one TransactionTemplate: lock-and-read (FOR UPDATE
SKIP LOCKED works identically on both engines), flip offline (INTERVAL
expression is the only forked fragment), derive the post-update view
in Java from the locked rows — the update is deterministic per column.
- Sequences retired in favor of row-local increments: the assignment
version and the fencing token advance as col = col + 1 on every
change. Per-driver / per-device monotonicity is the property callers
and stale-writer checks rely on; the global sequences only ever
delivered it incidentally, and MySQL has no sequence objects.
- Forked statements (databaseId twins): the point-latest fencing upsert
(row-value tuple guard expanded into per-column IF chains — MySQL has
neither row comparisons nor WHERE on ON DUPLICATE KEY UPDATE), the
driver-instance upsert, both lease upserts, the driver lock (MySQL
serializes on the driver row itself: SELECT ... FOR UPDATE, scoped to
the transaction like the advisory lock it replaces), the registry
lock (GET_LOCK + an explicitly paired advisoryUnlock — a no-op on
postgres; the sync service wraps its critical section in try/finally),
the tool-catalog candidate query (|| -> CONCAT, regexp_replace flag,
and the (ext ->> 'content')::json double-hop re-expressed through
JSON_UNQUOTE(JSON_EXTRACT(...)) — the content value is a JSON-encoded
string, not a nested object, verified against live data), four alert
statements' ->> extractions, hourly counts (DATE_FORMAT hour bucket),
and the activity heatmap (DAYOFWEEK/HOUR units, dow 0 = Sunday).
- Portable rewrites that needed no fork: token_metadata::TEXT ->
CONCAT(token_metadata, '') (both engines return text; only PG JDBC
needs the nudge past PGobject), typeDistribution's GROUP BY now
references the renamed bucket_key alias (the bare `key` was a MySQL
reserved-word landmine).
Gates: data/auth/manager suites green (274 in data), e2e 26/26.
MySQL seed and the dual-dialect DAL TCK follow in R2b/R2c.
R1 of docs/design/storage-abstraction.md §3 — zero behavior change:
Portable SQL rewrites (verified against a live PostgreSQL before landing):
- COUNT(*) FILTER (WHERE) x3 -> COALESCE(SUM(CASE WHEN ... THEN 1 ELSE 0
END), 0); the COALESCE keeps the empty-set result at 0 where bare SUM
would return NULL.
- dailyTrend's generate_series + date_trunc calendar spine -> a
RECURSIVE CTE day ladder (CAST(day + INTERVAL '1' DAY AS DATE) keeps
the column type stable on both engines — the bare date+interval form
widens to timestamp on PostgreSQL and fails the recursive term's type
check, caught live); day rendering via CONCAT instead of ::text.
- OAuthMcpMapper: '{} '::JSON literals -> '{}' (unknown literals coerce
to the JSON column on both engines), NULL::BIGINT -> NULL in the
UNION arm, #{riskLevel}::text parameter casts dropped (the text column
compares untyped parameters fine on both).
- The reserved-word alias AS key (a MySQL landmine hiding inside three
otherwise-portable statements) renamed to bucket_key / bucketKey
across both Dashboard mappers and BucketRow.
Module split: dc3-common-postgres -> dc3-common-jdbc (MyBatis-Plus
wiring, tenant-line handler, generator utils, the new dialect-neutral
jdbc profile carrying the shared mybatis-plus/hikari plumbing) + the
top-level dc3-db family's dc3-db-postgres (driver, timestamptz type
handler, postgres profile delta, and the pagination DbType bean the
neutral config now injects — defaulting to PostgreSQL so a bare
jdbc-only classpath behaves exactly as before). Packages deliberately
unchanged, mirroring the mq/tsdb family moves; all nine dependent poms
rewired; make deploy publishes the dc3-db family.
The jdbc module's interceptor integration test now pins an explicit
embedded H2 datasource — the module carries no driver on its main
classpath, so context inference would fail without it.
Gates: data/auth/manager/jdbc/db-postgres unit suites green (274 in
data), full-repo compile green, e2e 26/26.
DataAnalyticsService (biz/analytics) composes TSDB port primitives with
relational metadata into the nine coarse-grained reads of
docs/design/tsdb-abstraction.md §9.7: latest values, raw/M4 history,
per-series stats with percentiles, period comparison, entity ranking
(activity or mean/max/min), least-squares trends, threshold exceedance
reports with merged intervals, Pearson correlation, and a tenant-level
quality overview (coverage, silence, quality-code census).
Exposed as nine /analytics/* REST endpoints. The MCP gateway's tools/call
dispatches over HTTP to catalog rows synthesized from OpenAPI snapshots,
so these endpoints (x-dc3-ai annotated, fully @Schema-documented — the
data annotation gate enforces it) become the MCP tool surface after
make openapi + refreshToolCatalog; no Java tool classes are needed.
Hard guarantees: the tenant id always comes from the authenticated
principal, never the body, and the facade re-validates it (S11 double
gate); scan volume is bounded (20 series, 5000 samples per series, 90-day
windows) with structured over-budget rejections and honest degradation
notes instead of silent truncation; ambiguous name resolution fails with
the candidate list so an agent can disambiguate on the next turn.
correlate is dual-path — the store primitive when declared, otherwise
Pearson over aligned bucket pulls (verified against TDengine's
capability=false, exact on a perfectly-correlated fixture).
Gates: dc3-common-data 274/274 including 12 new analytics tests;
dc3-center-data assembly compiles.
dc3-tsdb-tdengine maps the port onto TDengine 3.x: one supertable
point_value tagged by (tenant, device, point), one deterministic subtable
pv_<t>_<d>_<p> auto-created on first insert via USING TAGS, database
created with PRECISION 'us' and KEEP 180. Connection settings live under
dc3.tsdb.tdengine.* (REST driver, own Hikari pool); selected by
dc3.tsdb.type=tdengine. Like the broker family, only the default adapter
(timescale) ships inside consumers — switching deployments add this
dependency plus DC3_TSDB_TYPE/DC3_TSDB_TDENGINE_URL.
The supertable mapping grilled four real issues out of the port
surface, each fixed and locked by the TCK:
- Timestamps never touch string form. The REST driver serializes
Timestamp parameters in the client JVM zone while the server parses
them as UTC — every write would shift by the deployment zone and
cursor pagination would drift per page without end. Instants now
travel as epoch-micro integer literals and reads return
CAST(ts AS BIGINT) — verified symmetric against the image.
- AS value collides with a reserved word (agg_value now);
INTERVAL takes a bare number read in the database (micro) precision.
- PERCENTILE only runs on single tables, so single-series percentiles
query the deterministic subtable directly; tenant-wide percentile is
refused rather than approximated.
- REST readiness must be probed with POST /rest/sql + basic auth; the
GET path-style route 404s even on a healthy server.
Honest capabilities: latencyHistogram=false (dashboard degrades to
zero-filled bins via a capability check) and correlation=false (future
analytics facade computes from bucketed pulls); rollups stay NONE until
S16 lands stream computing. TCK result: 23 tests, 0 failures, 0 errors,
2 capability-gated skips; the timescale suite re-ran 23/23 unchanged.
The design doc's capability matrix now reflects the adapter's declared
values instead of the pre-implementation estimates.
Phase 1b of docs/design/tsdb-abstraction.md: every point-value read and
write now flows through the dc3-tsdb port, and dc3-common-repository is
deleted outright (no compat aliases).
Write path — new PointValueIngestService orchestration in dc3-common-data:
the stale-owner lease guard leaves the history INSERT's cross-schema join
and resolves the active owner per distinct device through the existing
DeviceFacade.getActiveOwner chain; a caffeine-backed ingest idempotency
window (dc3.data.ingest.*, marked only after both writes commit) absorbs
MQ redeliveries now that the message_id unique index is retired; the
batch lands via TsdbStore.append (natural series+time upsert, INGEST_ORDER
kept for the fenced dc3_point_latest upsert).
Read path — history via last(), page() resolves name/enable filters to
series keys through relational metadata (tenant-wide when unrestricted,
cursor offset emulation capped at 10k) with count() for totals; the alarm
window backend becomes TsdbWindowDataSource over aggregate()/history().
Dashboards — all nine data-side hypertable statements move to the S13
analytics facet (count/bucketedCount/countByDimension/latencyHistogram/
lastSeenPerSeries) plus the dc3_point_latest projection for latestStream;
the manager topology statement crosses into the data center via a new
ListSeriesVolumes RPC backed by the new S13-5 seriesCounts primitive (a
point shared by several devices needs per-series counts, which
single-dimension grouping cannot reconstruct); both cross-schema joins
are gone.
Port/adapter — seriesCounts added with a TCK case (23 total); series
filters compile to row-value IN lists chunked at 500 pairs; bootstrap
adds the quality column idempotently and retires uk_point_value_event so
existing deployments converge on the new upsert; the adapter binds the
application-provided tsdbDataSource bean (the history dynamic-datasource
entry) instead of the routing primary.
Timestamps — BO<->Instant conversion is pinned to
TimeConstant.DEFAULT_ZONEID (the platform canonical zone the latest
projection type handler already writes with); the design note's "lock
UTC" is amended accordingly, together with the lease-guard and S13-5
deviations recorded in §6.2.
Seed/deploy — 05-iot-dc3-history.sql gains the quality column and swaps
the unique index; compose carries DC3_TSDB_TYPE; make deploy publishes
dc3-tsdb.
Also fixed along the way: root pom never managed dc3-mq-core (latent,
masked by full-reactor builds); PointValueMapper ran unrouted on the
master datasource (now @DS("history")); latestStream name enrichment
looked up a Long-keyed map with String keys.
Gates: tsdb TCK 23/23, dc3-common-data 262/262, e2e 26/26 including the
unmodified PostgresHypertableIT; full reactor green except dc3-mq-tck's
kafka/pulsar container-startup flake, reproduced identically twice and
unrelated to this change.
The broker-selection family (port + six certified adapters + tck) is a
deployment selection, not shared plumbing, so it moves out of dc3-common
into a top-level dc3-mq aggregator beside dc3-api — the same standing as
the gRPC contract layer, and a template for future pluggable families
(storage is next on the design board).
- dc3-common-mq -> dc3-mq/dc3-mq-core (port: API + core runtime)
- dc3-common-mq-{rabbitmq,kafka,rocketmq,pulsar,activemq,mqtt,tck} ->
dc3-mq/dc3-mq-{...}; no compatibility aliases — every coordinate is
renamed everywhere in one commit (business poms, coverage, root
dependency management)
- dc3-common sheds eight modules back to plain shared plumbing; the
device-access dc3-common-mqtt stays by design (protocol stack, not a
broker selection)
- Java packages deliberately stay io.github.pnoker.common.mq.* — the
facade modules set the precedent that packages do not track module
homes, keeping this a pom-only, zero-source refactor
- make deploy now publishes dc3-api + dc3-common + dc3-mq (third-party
driver authors compile against the port transitively)
- docs/mq-brokers.md and the design's module layout follow the new home
Verified: full-repo test-compile green; constant/data/driver/manager
unit suites unchanged and green (255/144/291); the rabbitmq contract
suite passes 12/12 from the new coordinates against a live broker,
proving adapter wiring survived the move.
Close out the mq abstraction rollout:
- docs/mq-brokers.md: user-facing broker selection guide — how to pick
(DC3_MQ_TYPE + connection settings), the six certified brokers, the
capability matrix, and the at-least-once/outbox notes
- .env.example + dc3/env/dev.env: DC3_MQ_TYPE and the per-adapter
connection variables with safe rabbitmq defaults
- docker-compose.yml: DC3_MQ_TYPE and adapter endpoints flow through the
shared x-app-runtime-env anchor so every service sees the selection
- adapter configurations resolve connection settings from the canonical
dc3.mq.* property first, then the DC3_MQ_* environment variable, then
the transport-standard property (spring.kafka.bootstrap-servers),
making containerized selection deterministic
The gated E2E gate passes unmodified through the port: RabbitDeliveryIT
6/6, CommandCallE2eIT 4/4, EventReportE2eIT 3/3 (full suite 24/24
including postgres and harness ITs) against testcontainers brokers.
Phase 3 (4/4) and the final piece of the mq abstraction: every broker
named in the design is now a TCK-certified selection behind
dc3.mq.type. Pulsar suite 11 pass + 1 documented skip on the first run.
- topics map to persistent://public/default/dc3-<topic>, logical dead
topics to dc3-<topic>-dlq; LOAD_BALANCE rides a durable shared
subscription named after the consumer group (competing consumers,
retained while offline), BROADCAST rides an exclusive per-instance
subscription
- pulsar subscriptions start at the latest position natively, matching
the rabbit fresh-queue / kafka latest semantics with no seeding
- acknowledge(List<MessageId>) commits deliveries, negativeAcknowledge
requeues; reject(false) republishes to the -dlq topic and commits;
poison and exhausted batches dead-letter through the shared
synchronous bounded-retry semantics
- sends complete with the broker-issued message id (publisher
confirmation); batchReceive is the native batch (capability true);
arbitrary delays ride the port fallback like the other adapters
- tck harness against apachepulsar/pulsar standalone, or TCK_PULSAR_URL
for an externally managed broker
pnoker