730 Commits

Author SHA1 Message Date
lizhimins 854369aa38 [ISSUE #9339] Fix pop update consumption offset when message are filtered (#9340) 2025-04-16 16:00:37 +08:00
ymwneu d5b474d9d3 [ISSUE #9297] Supports outputting topic put TPS in TopicStatusSubCommand (#9298) 2025-04-02 18:59:45 +08:00
ymwneu 586a274839 [ISSUE #9279] Restrict system subscription group creation and add pull request rejection policy (#9280) 2025-04-01 17:12:36 +08:00
ymwneu 830e89eed1 [ISSUE #9284] When pullMessage overflow one, should refresh recordDiskFallBehind data (#9285) 2025-04-01 13:52:26 +08:00
ymwneu b5ab93ce7d [ISSUE #9286] Counting the filtered message when filter by SQL92 (#9287) 2025-04-01 13:48:04 +08:00
ymwneu 4a15256f37 [ISSUE #9300] Periodic cleanup of inactive items in StatsItemSet (#9301) 2025-04-01 13:43:56 +08:00
ymwneu c5be5c8f2e [ISSUE #9304] Resolve cold data read control issue in DefaultMessageStore (#9305) 2025-04-01 13:38:46 +08:00
half 62be041617 [ISSUE #7948] Prevent invoking the queryMessage method lead to OOM (#9265)
Co-authored-by: yangxiaohui <yangxiaohui@didachuxing.com>
2025-03-26 10:31:02 +08:00
mxsm 34c32d8c80 [ISSUE #9259] Remove duplicate flushing operation of StoreCheckpoint(#9260) 2025-03-20 11:44:03 +08:00
qianye cee19630ad [ISSUE #9241] RocksDBConsumeQueueStore do not need to update StoreCheckpoint (#9242) 2025-03-17 17:30:48 +08:00
hqbfz 343ed4ff46 [ISSUE #8127] Optimize the metric calculation logic of the time wheel (#8128)
* Fix the metric of the time wheel was incorrectly calculated

* Fix the metric of the time wheel was incorrectly calculated

---------

Co-authored-by: wanghuaiyuan <wanghuaiyuan@xiaomi.com>
2025-03-04 15:53:28 +08:00
qianye 66d4a26eaa Optimize RocksDB CQ shutdown when using DoubleWriteCQ #9212 2025-02-28 14:19:38 +08:00
hqbfz d9d5e93135 [ISSUE #9177] Fix unstable tests in AdaptiveLockTest.testAdaptiveLock (#9178) 2025-02-12 16:44:34 +08:00
Kris20030907 a2755104b0 fix(comment): correct typos in ConsumeQueueExt class. (#9124) 2025-01-15 14:51:09 +08:00
qianye 94d9185f2c [ISSUE #8895] Fix NPE when broker shutdown and optimize the log #9094 2025-01-14 14:50:56 +08:00
guyinyou 119a0bec66 [ISSUE #9121] Fix CRC32 Check Failing When Value is 0 2025-01-11 10:27:46 +08:00
lizhimins a3afb05cb3 [ISSUE #9025] [RIP-73] Modify Pop Consumption rocksdb init config (#9100) 2025-01-03 19:39:35 +08:00
Lei Zhiyuan 16b6e53263 [ISSUE #9002] when bytebuffer is not enough, do not throw exception (#9003)
* fix: when bytebuffer is not enough,we should wait for next instead of throw exception

* fix: when bytebuffer is not enough,we should wait for next instead of throw exception
2024-12-19 11:44:32 +08:00
guyinyou 9aa081b8ac [ISSUE #8988] Support dispatchBehindMilliseconds (#8989)
* support dispatchBehindMilliseconds

* Modify the initial value of currentReputTimestamp

---------

Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2024-12-10 20:05:44 +08:00
小陈 7da9ad4fa5 [ISSUE #9042] Update createTimerMessageStore call with new parameter (#9041)
* Update createTimerMessageStore call with new parameter

* Reduce unit test execution time
2024-12-10 14:40:27 +08:00
imzs bfb3d17ef1 [ISSUE #8974] Support recalling of delay message (#8975) 2024-12-09 16:56:12 +08:00
asapple 4ea8d1ff6c refactor(LmqBrokerStatsManager): extract common method to eliminate duplicate logic (#9034)
Extracted the duplicated logic of replacing group and topic with LMQ_PREFIX based on configuration into a common method, improving code structure and maintainability.
2024-12-07 17:59:46 +08:00
小陈 d1fd7af3f1 [ISSUE #8979] Add configurable switch for timer message retry logic (#8980) 2024-12-05 17:49:03 +08:00
rongtong f4c498433d [ISSUE #7480] Fix the offset in the timerCheckPoint will not be corrected when the commitlog and consumeQueue are truncated (#7488) 2024-12-04 15:46:56 +08:00
Zhanhui Li 9a891f1d49 fix: multiple patches during long running tests for LMQ over RocksDB (#8915)
* fix: multiple patches during long running tests for LMQ over RocksDB

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: fix a bug in RocksGroupCommitService; remove RocksDBConsumeQueueStore#findConsumeQueueMap override

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: async fsync on RocksDB WAL flush

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: use a dedicated thread to flush and sync RocksDB WAL

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: trigger WAL rolling according to estimated WAL file size

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: add doc, explaining config RocksDB instance flush/sync strategy

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: data-version should be per table

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: test case: RocksdbTransferOffsetAndCqTest

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
2024-12-02 10:07:54 +08:00
rongtong 715dd5a885 Adding the EnableLmqStats option allows monitoring of LMQ statistics at runtime (#8973) 2024-11-22 12:57:38 +08:00
Zhanhui Li 8505482c0b fix: avoid memory overhead when there is large number of LMQ ConsumeQueue (#8956) 2024-11-20 14:55:30 +08:00
hqbfz c13f051eb8 Improve IO for asynchronous delivery processes (#8954)
Co-authored-by: wanghuaiyuan <wanghuaiyuan@xiaomi.com>
2024-11-20 11:13:12 +08:00
yuz10 ae7179d75e [ISSUE #8765] fix low performance of delay message when enable rocksdb consume queue (#8766)
* #7538 fix wrong cachedMsgSize if msg body is changed in consumer callback
* [ISSUE #8765] fix low performance of delay message when enable rocksdb consume queue
* remove prefetch
2024-11-20 11:10:01 +08:00
Ji Juntao 78575af0b8 [ISSUE #8935] Fix behind metrics unit error in timer message store (#8936) 2024-11-19 17:35:10 +08:00
hqbfz c961edd2fd [ISSUE #8945] Remove unnecessary operations from the critical section
Co-authored-by: wanghuaiyuan <wanghuaiyuan@xiaomi.com>
2024-11-18 16:46:34 +08:00
rongtong 4e8a5ca48f Add incGroupAckNums and incGroupCkNums to LmqBrokerStatsManager (#8943) 2024-11-18 14:23:00 +08:00
Drizzle 60e68dadd9 [ISSUE #8921] Add isWakeCommitWhenPutMessage for AIO
Co-authored-by: drizzle.zk <drizzle.zk@alibaba-inc.com>
2024-11-14 19:42:25 +08:00
Crazywen fe8077250b [ISSUE #8875] Fix HAConnection leak 2024-10-30 16:42:30 +08:00
LetLetMe 0b24768500 [ISSUE #8822] Double write cq, reduce unnecessary switches (#8823)
* Reduce unnecessary switches
2024-10-29 19:09:35 +08:00
hqbfz 95b88ff8fc [ISSUE #8442][RIP-70-3] Extract adaptive lock mechanism (#8663)
* extract the adaptive lock

* extract the adaptive lock

* feat(): perfect the adaptive lock

* feat(): perfect the adaptive lock

* Optimized code type

* Optimized code type

* Optimized code type

* fix fail test

* Optimize the adaptive locking mechanism logic

* Optimize the adaptive locking mechanism logic

* feat:Adaptive locking mechanism adjustment

* feat:Adaptive locking mechanism adjustment

* feat:Adaptive locking mechanism adjustment

* Optimize the adaptive locking mechanism logic

* Optimize the adaptive locking mechanism logic

* Optimize the adaptive locking mechanism logic

* feat:Supports the hot activation of ABS locks

* feat:Supports the hot activation of ABS locks

* feat:Supports the hot activation of ABS locks

* feat:Supports the hot activation of ABS locks

* Optimize code style

* Optimize code style

* Optimize code style

* Optimize code style

* Optimize code style

* Optimize code style

* Updated the locking mechanism name

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* delete unused import

* Optimize the logic of switching to spin locks

* Revert "Optimize the logic of switching to spin locks"

This reverts commit 1d7bac5c2f.

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimize the logic of switching to spin locks

* Optimized locking logic

* Optimized locking logic

* Optimized locking logic

* fix test

* fix test

* fix test

* fix test

* Optimize code style

* Optimize code style

* fix test

* fix test

* optimize client rebalancing logic

---------

Co-authored-by: wanghuaiyuan <wanghuaiyuan@xiaomi.com>
2024-10-23 19:17:37 +08:00
Zhanhui Li 738c9f3a95 [ISSUE #8829]feat: add test case to RocksDBConsumeQueueOffsetTable (#8857)
* feat: add test case to RocksDBConsumeQueueOffsetTable, verifying forEach works properly for long topic-name

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: change OpenJDK distribution from adopt to corretto as the previous one is not updated anymore

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: unify set-java version and distribution

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
2024-10-23 19:17:05 +08:00
Zhanhui Li b86059c7c1 Support LMQ dispatch in case if Consume Queue Store is RocksDB-based (#8842)
* feat: support LMQ dispatch

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: introduce group-commit for batch insertion of RocksDB KV pairs

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: propagate store error to broker module

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: fix all Bazel warning and errors

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>

* fix: remove unnecessary batch-ops when writing RocksDB using atomic flush

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: find a writable directory for RocksDB logs

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: clean up ConfigHelperTest

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: truncate consume queues in case commit log records are truncated

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: truncate LMQ max offsets

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: correct truncate boundary of consume queues

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: correct MessageExt encoding

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: remove unused import

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>
2024-10-23 09:56:37 +08:00
mawen12 c033c3e85f [ISSUE #8848] Fix log typo 2024-10-23 08:29:35 +08:00
Zhanhui Li 1e0b08f56a fix: make ConsumeQueueStore bottom most compression type configurable (#8841)
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
2024-10-18 16:50:13 +08:00
Zhanhui Li 49e23e1e0b fix: atomic flush incorrect use and clean up code (#8830)
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
2024-10-18 14:30:38 +08:00
LetLetMe 2fa7513a2b [ISSUE #8764] Implement consume lag estimation in cq rocksdb store (#8800) 2024-10-15 14:14:31 +08:00
luozongle01 c5ac8a47ea [ISSUE #8782] Fix log typo (#8783) 2024-10-05 10:09:07 +08:00
yuz10 551c8c3d0d [ISSUE #8736] fix searchOffset corner case in rocksdb consume queue (#8737)
* fix searchOffset  for ConsumeQueue backed by RocksDB
2024-09-29 15:10:06 +08:00
LetLetMe 3b5cbf86df [ISSUE #8698] Remove batch write in kv cq store and update rocksdb cq check tool (#8739) 2024-09-25 19:32:47 +08:00
LetLetMe 525f877f3b [ISSUE #8589] Support file format CQ and json format offset in-place upgrade to rocksdb management (#8600) 2024-09-23 19:24:15 +08:00
Liu Shengzhong 280804c559 [ISSUE #8693] Fix checking MultiDispatchMessage when appending commitlog 2024-09-18 13:57:20 +08:00
Dongyuan Pan ba8148f746 [ISSUE #8669] Fix crc 32 overflow when lmq 2024-09-11 17:06:40 +08:00
Vincent Lee adf110fbaa [ISSUE #8660] Should use read only getConsumeQueue instead of findConsumeQueue in read only func (#8659) 2024-09-09 16:59:44 +08:00
小陈 4c51706b28 [ISSUE #8623] Temporarily skip flaky unit tests on macOS (#8633)
* Skip flaky tests on macOS

* Trigger ci

* Remove branch trigger
2024-09-04 11:14:34 +08:00