OpenTelemetry Java 1.44.0 ~ 1.46.x ships OtlpGrpcMetricExporter with
MemoryMode.REUSABLE_DATA by default. The underlying
MetricReusableDataMarshaler.marshalerPool is a non-thread-safe
ArrayDeque accessed concurrently by the reader thread (poll) and the
OkHttp callback thread (add, via whenComplete). With
BatchSplittingMetricExporter issuing N concurrent sub-batch exports
per cycle, the pool races and leaks marshalers (~132 KiB each) until
OOM. Fixed upstream in 1.47.0 via open-telemetry/opentelemetry-java#7041
(ArrayDeque -> ConcurrentLinkedDeque).
- Bump OpenTelemetry to 1.47.0 in pom.xml so the upstream race fix is
in effect.
- Default OtlpGrpcMetricExporter to MemoryMode.IMMUTABLE_DATA to
preserve the pre-1.44 default behavior; exposed via
brokerConfig.metricsExportOtelMemoryMode ("IMMUTABLE_DATA" /
"REUSABLE_DATA", case-insensitive). Operators may opt in to
REUSABLE_DATA when running on OTel >= 1.47.
- Cap concurrent in-flight sub-batches in BatchSplittingMetricExporter
with a Semaphore controlled by
brokerConfig.metricsExportBatchMaxConcurrent (default 4; set to 1
to serialize and match pre-batch behavior; 0 or Integer.MAX_VALUE
means unlimited).
- Add brokerConfig.metricsExportBatchSplitEnabled (default true) as
an escape hatch to bypass BatchSplittingMetricExporter entirely,
restoring the raw OtlpGrpcMetricExporter wiring.
- Defensively snapshot MetricData points before export to avoid
ArrayIndexOutOfBoundsException in NumberDataPointMarshaler when
async instrument callbacks mutate point collections during export.
* feat: use data version from master while sync slave and fix delete config while sync
Change-Id: I42b2e7b1acc6836d3c90973801c9defba5f1325c
* fix: assign new version using master while sync slave
Change-Id: I7ec20607a84499fe5a6607763013c59d726aedc3
* feat: allow set dataVersion directly for topic/group config sync
Change-Id: Ic845794350e8bdaa847bdd0ae4b3e40ab1ad6311
* feat: set data version directly while sync from master
Change-Id: I39e78477a5223b578a4ede3e5cb76f04368d1ca3
* test: adjust slave sync test for version
Change-Id: I9e835568912928ddf6e81816095ee3ed8f93afc0
* Add BatchSplittingMetricExporter to prevent OTLP gRPC export failures
When high-cardinality metrics (consumer_group x topic) produce OTLP export
payloads exceeding the gRPC 32MB limit or SLS per-RPC processing limit,
all metrics fail to export. This adds a MetricExporter decorator that:
- Splits large batches of MetricData objects into smaller sub-batches
- Splits single oversized MetricData objects by their internal data points
into multiple smaller MetricData objects (supports all 7 MetricDataType)
- Configurable via BrokerConfig.metricsExportBatchMaxDataPoints (default 1000)
- Fast path with zero overhead when data points are within threshold
- Logs failed batch details for debugging
* fix(metrics): snapshot MetricData points before export to prevent AIOOBE
The OTel SDK's NumberDataPointMarshaler.createRepeated allocates an
array based on points.size() then iterates. If callback threads
concurrently add data points between size() and iteration, an
ArrayIndexOutOfBoundsException occurs. This adds a defensive snapshot
of all data point collections at the start of export(), ensuring
the delegate exporter always receives immutable point collections.
* test(metrics): add unit tests for snapshot defensive copy
- testSnapshotCreatesNewMetricData: verify delegate receives
snapshotted MetricData, not the original reference
- testSnapshotFallsBackToOriginal: verify catch block falls
back to original when snapshot fails (e.g., mock without type)
- testSnapshotPointsAreIndependentCopy: verify the snapshotted
points collection is a separate instance from the original
- Add wildcard (*) subscription support for liteTopic
- Implement consume suspend mechanism with invalid scan count threshold
- Refactor subscriber query interface with SubscriberWrapper for flexible retrieval
- Add wildcard client cache with 30s TTL for performance optimization
- Update related components and enhance test coverage
Change-Id: I4ecaceec7daa2f4364d911437007df98dc49d542
* When IndexRocksDBEnable or TransRocksDBEnable are enabled, we need to take these two offsets into account to accelerate recovery.
* Add UTs
* Refactor the code based on the review comments
* Revert "[ISSUE #8127]Optimize the metric calculation logic of the time wheel"
* Remove useless import
* Refactor Code
* Refactor Code
* Refactor Code
* Refactor Code
* Refactor Code
* Implement accelerated recovery for the file-based ConsumeQueue.
* Implement accelerated recovery for the file-based ConsumeQueue.
Change-Id: Ieac45d0582f2f83d977aeb8e6f5084268b7f8752
* Implement accelerated recovery for the file-based ConsumeQueue.
* Ignore testTruncateCQ UT
---------
Co-authored-by: RongtongJin <user@example.com>
yx9o