* chore: bump DLedger to 0.3.3.4 (maintenance line)
Update the DLedger dependency from 0.3.2 to 0.3.3.4, the latest
release on the 0.3.x maintenance line published on Maven Central.
The 0.3.3.4 release is API-compatible with 0.3.2 and only carries
dependency hygiene fixes (aligning fastjson2 and excluding fastjson1
from its rocketmq-remoting transitive path).
Also update the Bazel WORKSPACE artifact coordinate to keep the two
build systems in sync.
* chore: fully remove fastjson1 from the build
DLedger 0.3.3.4 already excludes com.alibaba:fastjson from its
rocketmq-remoting dependency and its bytecode only references
fastjson2, but rocketmq still declared fastjson1 itself, so it kept
leaking onto every module compile classpath through rocketmq-common.
- drop the com.alibaba:fastjson property and dependencyManagement
entry from the root pom
- drop the unused com.alibaba:fastjson dependency from rocketmq-common
(no main source imports com.alibaba.fastjson.*)
- drop the fastjson1 pin from WORKSPACE and the fastjson1 dep from the
remoting test target in remoting/BUILD.bazel
- migrate RemotingSerializableCompatTest to fastjson2: use the
fastjson2 JSONField annotation (which is what the protocol classes
actually carry) and round trip through RemotingSerializable instead
of com.alibaba.fastjson.JSON
fastjson1 wire-format coverage is retained by the frozen fastjson1
payload already asserted in testCompatibilityCheckWithBitSet.
* fix(bazel): exclude rocketmq-remoting from coursier resolution
dledger 0.3.3.4 depends on rocketmq-remoting:5.5.0, which Coursier
fetches from Maven Central. That artifact transitively brings
grpc-netty-shaded:1.53.0 with a strict [1.53.0] range on grpc-core,
conflicting with the WORKSPACE-declared grpc 1.47.0 artifacts.
Since rocketmq-remoting is built from source in this repo (//remoting),
exclude it from external resolution — matching what PR #10947 already
does for the DLedger 0.4.x line.
---------
Co-authored-by: 通融 <rongtong.jrt@alibaba-inc.com>
OpenTelemetry Java 1.44.0 ~ 1.46.x ships OtlpGrpcMetricExporter with
MemoryMode.REUSABLE_DATA by default. The underlying
MetricReusableDataMarshaler.marshalerPool is a non-thread-safe
ArrayDeque accessed concurrently by the reader thread (poll) and the
OkHttp callback thread (add, via whenComplete). With
BatchSplittingMetricExporter issuing N concurrent sub-batch exports
per cycle, the pool races and leaks marshalers (~132 KiB each) until
OOM. Fixed upstream in 1.47.0 via open-telemetry/opentelemetry-java#7041
(ArrayDeque -> ConcurrentLinkedDeque).
- Bump OpenTelemetry to 1.47.0 in pom.xml so the upstream race fix is
in effect.
- Default OtlpGrpcMetricExporter to MemoryMode.IMMUTABLE_DATA to
preserve the pre-1.44 default behavior; exposed via
brokerConfig.metricsExportOtelMemoryMode ("IMMUTABLE_DATA" /
"REUSABLE_DATA", case-insensitive). Operators may opt in to
REUSABLE_DATA when running on OTel >= 1.47.
- Cap concurrent in-flight sub-batches in BatchSplittingMetricExporter
with a Semaphore controlled by
brokerConfig.metricsExportBatchMaxConcurrent (default 4; set to 1
to serialize and match pre-batch behavior; 0 or Integer.MAX_VALUE
means unlimited).
- Add brokerConfig.metricsExportBatchSplitEnabled (default true) as
an escape hatch to bypass BatchSplittingMetricExporter entirely,
restoring the raw OtlpGrpcMetricExporter wiring.
- Defensively snapshot MetricData points before export to avoid
ArrayIndexOutOfBoundsException in NumberDataPointMarshaler when
async instrument callbacks mutate point collections during export.
- Add wildcard (*) subscription support for liteTopic
- Implement consume suspend mechanism with invalid scan count threshold
- Refactor subscriber query interface with SubscriberWrapper for flexible retrieval
- Add wildcard client cache with 30s TTL for performance optimization
- Update related components and enhance test coverage
Change-Id: I4ecaceec7daa2f4364d911437007df98dc49d542
* feat: support LMQ dispatch
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: introduce group-commit for batch insertion of RocksDB KV pairs
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: propagate store error to broker module
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* chore: fix all Bazel warning and errors
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>
* fix: remove unnecessary batch-ops when writing RocksDB using atomic flush
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: find a writable directory for RocksDB logs
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* chore: clean up ConfigHelperTest
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: truncate consume queues in case commit log records are truncated
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: truncate LMQ max offsets
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: correct truncate boundary of consume queues
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* fix: correct MessageExt encoding
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* chore: remove unused import
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
---------
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>
* typo int readme[ecosystem]
* rocksdb metadata
* add unit test
* fix testOffsetPersistInMemory
* fix unit test
* fix unit test
* remove unused import
* move RocksDBOffsetSerialize to broker moudle
* Fix bazel build scripts
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* Flag QueryMsgByKeyIT as flaky as it fails at frequency: 5 out of 32
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* change public to private of some inner method
---------
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
* support s3 backend in tiered storage
* refactor(tieredstorage): Unify all object storage configuration properties
1. Unify all object storage configuration properties
* refactor(tieredstorage): replace some lambda function with more simple expression
1. replace some lambda function with more simple expression
* style(tieredstorage): perfect comments on ChunkMetadata
1. perfect comments on ChunkMetadata
* refactor(tieredstorage): perfect lambda expression
1. perfect lambda expression
* fix(tieredstorage): fix unmatched config attributes in brokerS3.conf
1. fix unmatched config attributes in brokerS3.conf
* feat(tieredstorage): More context in logging output
1. More context in logging output
* fix(tieredstorage): fix wrong concurrently put
1. fix wrong concurrently put
* test(tieredstorage): add UT to verify TieredFileSegmentInputStream
1. add UT to verify TieredFileSegmentInputStream
* refactor(tieredstorage): better code placement
1. better code placement
* refactor(tieredstorage): refactor TieredFileSegmentInputStream for better understandability
1. refactor TieredFileSegmentInputStream for better understandability
* feat(tieredstorage): support `reset` of TieredFileSegmentInputStream
1. support `reset` of TieredFileSegmentInputStream
* fix(tieredstorage): fix wrong position when failed in `S3FileSegment#commit0`
1. fix wrong position when failed in `S3FileSegment#commit0`
* fix(tieredstorage): fix still have upload buffer when already seal the segment
1. fix still have upload buffer when already seal the segment
* test(tieredstorage): fix wrong assertion
1. fix wrong assertion
* feat(tieredstorage): support switch to enable merge chunks into segment
1. support switch to enable merge chunks into segment
* feat(tieredstorage): add more debug log in TieredMessageStore
1. add more debug log in TieredMessageStore
* style(tieredstorage): use rmq code style
1. use rmq code style
* feat(tieredstorage): add metrics for S3 provider
1. add metrics for S3 provider
* fix(tieredstorage): resolve conflicts after rebasing master
1. resolve conflicts after rebasing master
Closes https://github.com/apache/rocketmq/issues/6624
* style(tieredstorage): change log level
1. change log level
Closes https://github.com/apache/rocketmq/issues/6154
* build(controller): build tieredstorage with bazel
1. build tieredstorage with bazel
* build(controller): build tieredstorage with bazel
1. build tieredstorage with bazel
* style(tieredstorage): change log level
1. change log level
Closes https://github.com/apache/rocketmq/issues/6154
* test(tieredstorage): ignore tests about S3Mock
1. ignore tests about S3Mock
* test(tieredstorage): ignore tests about S3Mock
1. ignore tests about S3Mock
* [ISSUE #6414] polish MQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer parameters
* Fix to make Bazel CI pass
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* Fix typo
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
---------
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
* refactor: simplify getPID (#5962)
* [ISSUE #5923] Add example tiered storage backend service provider (#5926)
* implement example file segment
* add metrics
* add readme
* fix license
* fix tests
* fix links in README.md
* add comment to PosixFileSegment and mark as experimental
* fix test
* optimize image quality
* Remove the useless exception class: MQRedirectException #5963
* [ISSUE #5965] Fix lmqTopicQueueTable initialization (#5968)
* [ISSUE #5965] Fix lmqTopicQueueTable initialization
* [ISSUE #5965] Fix lmqTopicQueueTable initialization
* [ISSUE #5890] Fix dledger logging (#5959)
* Fix dledger logging
* Add bridge into store module
* [ISSUE #5860] Set the value of order when create or update topic (#5861)
* [ISSUE #5939]Adjust the MQClientInstance#sendHeartbeatToAllBroker catch code block log print level from info to warn (#5940)
* [ISSUE #5924] Optimize UtilAll#sleep method (#5925)
* [ISSUE #5924]Optimize UtilAll#sleep method
* polish code
* [ISSUE #5986] optimize the BrokerOuterAPITest class code
Co-authored-by: zhouyunpeng <2474138779@qq.com>
* [ISSUE #5971] Make the internal logs related to the dledger in the controller print to a file separately (#5972)
* Make the internal logs related to the dledger in the controller print to a file separately
* Make the internal logs related to the dledger in the controller print to a file separately
* [ISSUE #5969] Remvoe duplicate deleteUnusedStats in admin processor (#5973)
* [ISSUE #5847] Add checkBlock for hasMsgFromQueue
* [ISSUE #5983] Make consumer support flow control code better (#5984)
* When encountering the flow control code, pull it after 20ms instead of 3s
* When encountering the flow control code, pull it after 20ms instead of 3s
* [ISSUE #5896] feat:add pop consumer example (#5991)
* feat:add pop consumer
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* Update example/src/main/java/org/apache/rocketmq/example/simple/PushConsumer.java
Co-authored-by: Oliver <wqdyxnbd@163.com>
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
* feat:fix
---------
Co-authored-by: mahaitao617 <mahaitao617@mahaitao617deMacBook-Pro.local>
Co-authored-by: Oliver <wqdyxnbd@163.com>
* [ISSUE #5942] Fix the produce count include the quantity of the system topic(#5943)
* [ISSUE #5999] Fix the TopicQueueMappingUtils comments typo (#6000)
* [ISSUE #5996] Optimize the RemotingSerializable class code (#5998)
* simplified RemotingSerializable null check
* optimize the RemotingSerializable class code
* [ISSUE #5994] [RIP-46] add pop and timer metrics (#5995)
* add pop and timer metrics
* fix according to review comment
* test(broker): add ReplicasManagerRegisterTest to test the register process
1. add ReplicasManagerRegisterTest to test the register process
* chore(pom): modify pom.xml to replace mockito with powermock
1. modify pom.xml to replace mockito with powermock
* build(bazel): export powermock in bazel
1. export powermock in bazel
---------
Co-authored-by: Xinda <xdshent@gmail.com>
Co-authored-by: SSpirits <admin@lv5.moe>
Co-authored-by: loboxu <loboxu@tencent.com>
Co-authored-by: pingww <pingw002@gmail.com>
Co-authored-by: Aaron Ai <yangkun.ayk@gmail.com>
Co-authored-by: Slideee <yechun@corp.netease.com>
Co-authored-by: mxsm <ljbmxsm@gmail.com>
Co-authored-by: hardyfish <85128645+hardyfish@users.noreply.github.com>
Co-authored-by: zhouyunpeng <2474138779@qq.com>
Co-authored-by: rongtong <jinrongtong5@163.com>
Co-authored-by: zhiliatom <87265072+zhiliatom@users.noreply.github.com>
Co-authored-by: zhouxiang <zhouxiang.zzx@alibaba-inc.com>
Co-authored-by: mahaitao <15828010639@163.com>
Co-authored-by: mahaitao617 <mahaitao617@mahaitao617deMacBook-Pro.local>
Co-authored-by: Oliver <wqdyxnbd@163.com>
* Add long polling
* Change rocketmq-proto version to 2.0.2
* fix checkstyle
* Fix rocketmq-proto version
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* Change pollTime to timeRemaining
* fix test
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
* Fix compilation warning
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* Skip unstable test case HATest#testSemiSyncReplicaWhenSlaveActingMaster on macOS
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
* [RIP-57] Add asynchronous interfaces to MessageStore
* fix unit test
* Modified according to review comments
* Remove unnecessary stubbing
* Add some other methods to MessageStore
* fix bazel test
* Skip failing test case
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
* Remove ClientLogger
* WIP
* WIP
* Rename logger to log
* Make it compile
* WIP
* Fix bazel to use shaded slf4j
* Fix DeleteExpiredCommitLogSubCommandTest
* Fix pom.xml duplication
* Fix maven deps
* Add test logback configuration file
* Fix unit test output
* Fix logback configuration file
* All logging are made on top of slf4j
* Fix test log configuration file name
* Fix test log configuration file name for test module
* All logging are shaded slf4j targeted
* DLedger has an explicit dependency on slf4j
* Fix DLedger
* Fix DLeader issue
* Move logback configuration files to each module
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
* Rule out tests that are flaky. Aka they failed to pass 64 sequential and concurrent runs
* Change default log root directory for tests
* Follow test encyclopedia
* Fix test cases, allowing OS pick up available port to listen to
* Add sha256
* Use pull_request
* Support build with Bazel (#4865)
* Support build with Bazel, fixing tests to make them capable of running in concurrency and hermetic.
* Make test cases capable of running in parallel. (#4874)
rongtong