129 Commits

Author SHA1 Message Date
Quan eddb235c1b [ISSUE #10722] Lite Topic may not be cleaned up when message store timestamp is unavailable (#10723) 2026-08-03 14:04:44 +08:00
Houlong66 d8c3f9e735 [ISSUE #10572] Add dual TTL policy for tiered storage read-ahead cache (#10573) 2026-07-25 16:07:45 +08:00
fancy 3a568241f9 docs: fix dead link to tiered store config in tieredstore README (#10484)
The 'Configuration' section linked to TieredMessageStoreConfig.java under
the tieredstore/common package, but that class has been renamed to
MessageStoreConfig and moved to the tieredstore package root, so the link
returns 404. Point it at the current MessageStoreConfig.java, which holds
the tiered store configuration fields documented in the table below
(tieredStoreFilePath, tieredBackendServiceProvider, tieredStorageLevel, ...).

Signed-off-by: fancy-agent <fancyboi66@gmail.com>
2026-06-13 11:17:28 +08:00
lizhimins 91cb333716 [ISSUE #10462] Improve error handling in tiered storage (#10473) 2026-06-12 15:45:06 +08:00
lizhimins cf9113fce9 [ISSUE #10462] Fix resource leaks and lifecycle issues in tiered storage (#10476) 2026-06-11 20:47:54 +08:00
lizhimins 78e680b811 [ISSUE #10462] Standardize log format across tiered storage module (#10474) 2026-06-11 19:23:52 +08:00
lizhimins 84fc240acc [ISSUE #10462] Optimize thread pool sizes for commit and fetch executors (#10472) 2026-06-11 19:21:09 +08:00
lizhimins 21562bba0e [ISSUE #10462] Fix concurrency bugs in tiered storage (#10471) 2026-06-11 15:29:56 +08:00
lizhimins 50ae7c5093 [ISSUE #10462] Add memory backpressure to dispatch (#10470) 2026-06-11 14:14:11 +08:00
lizhimins 9443c6d9db [ISSUE #10383] Align Bazel dependencies with Maven pom.xml (#10404) 2026-05-29 10:04:20 +08:00
lizhimins 980f3d7f55 [ISSUE #10373] Quarantine flaky tests and add detection plan docs (#10374)
* [ISSUE #10373] Quarantine flaky tests and add detection plan docs

Ran all RocketMQ module tests 100x across 10 ECS nodes to identify
non-deterministic failures. Quarantined methods with @Ignore across
broker, client, filter, and tieredstore modules.

Flaky tests quarantined:
- broker: LiteLifecycleManagerTest#testCleanByParentTopic (2%)
- broker: ConsumerOrderInfoManagerLockFreeNotifyTest#testRecover (2%)
- broker: TransactionalMessageServiceImplTest#testDeletePrepareMessage_maxSize (1%)
- client: DefaultMQConsumerWithTraceTest#testPullMessage_WithTrace_Success (1%)
- client: DefaultMQLitePullConsumerWithTraceTest#testSubscribe_PollMessageSuccess_WithCustomizedTraceTopic (5%)
- client: DefaultMQLitePullConsumerWithTraceTest#testSubscribe_PollMessageSuccess_WithDefaultTraceTopic (6%)
- filter: BloomFilterTest#testCheckFalseHit (1%)
- tieredstore: IndexStoreServiceTest#queryCrossFileBoundaryTest (35%)
- tieredstore: IndexStoreServiceTest#concurrentGetTest (1.5%)

Additional changes:
- LiteLifecycleManagerTest: Switch to MockitoJUnitRunner.Silent
- Add flaky test detection plan docs (CN + EN)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [ISSUE #10373] Quarantine flaky PopPriorityIT and fix test cases

- Quarantine PopPriorityIT at class level (multiple methods fail
  intermittently with 'expected:<8> but was:<2>' due to async race)
- Fix ConsumerOrderInfoManagerLockFreeNotifyTest
- Fix IndexStoreServiceTest

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [ISSUE #10373] Fix flaky test detection plan docs path and naming

Move English doc from docs/cn/ to docs/en/ and rename both files
to match existing docs naming convention (underscore + PascalCase).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-25 17:36:36 +08:00
lizhimins 94fbfcf342 [ISSUE #10268] Fix incorrect time range file selection in IndexStoreService.queryAsync (#10269)
Co-authored-by: lizhimins <lizhimins@users.noreply.github.com>
2026-04-21 19:15:26 +08:00
lizhimins e5d33722d9 [ISSUE #10223] Not query the index of system topics in tiered storage (#10224) 2026-03-30 13:34:53 +08:00
SHI 7742b7c12d [ISSUE #10157] Fix tiered metadata leak after topic delete (#10158)
Co-authored-by: shixiaoxiao <shixiaoxiao@bytedance.com>
2026-03-17 10:15:00 +08:00
lizhimins 4eead1364d [ISSUE #9912] Reduce excessive requests for consumer offset timestamps in tiered storage (#9991)
* [ISSUE #9912] Reduce excessive requests for consumer offset timestamps in tiered storage

Signed-off-by: terrance.lzm <terrance.lzm@alibaba-inc.com>

* [ISSUE #9912] Reduce excessive requests for consumer offset timestamps in tiered storage

---------

Signed-off-by: terrance.lzm <terrance.lzm@alibaba-inc.com>
2026-01-13 09:56:22 +08:00
Drizzle fe5c17cb8b Optimized the function naming (#9935)
* add isWakeCommitWhenPutMessage  for AIO

* optimzie the Function name

Change-Id: Id91e3eb9c4488fb9804fb2c105082657e66c44c0

* optimized the function naming

Change-Id: Ifc482f91220ff328e5c5425a57a04ac627e8d469

---------

Co-authored-by: drizzle.zk <drizzle.zk@alibaba-inc.com>
2025-12-19 16:03:49 +08:00
SHI 15796e73d3 [ISSUE #9912] Fix consumer offset timestamp displayed incorrect in tiered store (#9913)
Co-authored-by: shixiaoxiao <shixiaoxiao@bytedance.com>
2025-12-12 16:10:09 +08:00
Drizzle 7a08b4cb68 [RIP-82] Implement Timer message, transaction message, and index based on RocksDB (#9787)
* add isWakeCommitWhenPutMessage  for AIO

* Implemented Timer message, Transaction Message, Index based on rocksdb

* add timeline roll interval can be config

* optimize the code

* optimize the code

* add sone logic for recover commitlog

* optimize the code

* update

* update

* optimize the code

* update

* update

* optimize the code

* optimize the code

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

* update

Change-Id: I700798ef625187ac32c16c8ea5aae5f2debfdddf

* update

Change-Id: I1d2dfa24317d2ad7c0ca41d90cd2b3c1ba260b0b

* update

Change-Id: I974e30eb909722ea2a17a14e505759bf20f4f860

* update

Change-Id: Ica01fbd08a2d60e9a0f4ab54b41ae4768cc4fa5d

* update

Change-Id: I764ce6cea3159b025bf1e6e65b60e20514cecb4f

* update

Change-Id: I0cc735896e1266934293ec06cb4f07746a1894a1

* update

Change-Id: I6538b6a60631c62e162bf511ae20185a07b72508

* update

Change-Id: I03f83b9e8199f3595ca28471df4a2e94671fbaae

* update

Change-Id: I5b19b671fa0b40cd2aba63f1d0eb654b92775cb4

* update the code

Change-Id: I22788d112b7f214310719e2d3ebad839ef818e67

* update

Change-Id: I7da89fe635b58ac5026095a8e484fa46f68521e7

* update the code

Change-Id: I5eade6c31803b8cf05c1a4bb83c226d57f4980c7

* update

Change-Id: Id67f600fad1a876926cd505dba047cc82011160c

* update

Change-Id: Ia4b0e2a21aa5e12570a80713432fd48ddfb210e6

* optimize the code

Change-Id: Idf8080c7b17c25e14fe34ef6bad1e1150dab58d6

* update

Change-Id: I904f0120a728b4eb87226159f9c0cde3d18ef26a

* update code

Change-Id: I21653d6bcad905daa8f10d12298d1d27535af9d9

* update

Change-Id: I712b9cde9a18c730fd020ea76f05560e594a9edd

* update

Change-Id: Iafe10c007e9b5057212e2dcfb21671f970b1d344

* update

Change-Id: I40c1c7f54852ee033e0caa22573436ee505a27b8

* optimize the code

Change-Id: Iae006b01608df4f363c63a0ae876dbf63cd56fbd

---------

Co-authored-by: drizzle.zk <drizzle.zk@alibaba-inc.com>
Co-authored-by: zhouli11 <04081337@163.com>
2025-12-11 15:13:56 +08:00
majialong f73fe6c4fb [ISSUE #9885] Fix tiered store cache count and bytes metrics (#9886) 2025-12-11 10:27:08 +08:00
yx9o d7e27d6d69 [ISSUE #9396] Use fastjson2 in all modules (#9397)
* Use fastjson2 in all modules

* Update test

* Update test

* Update test

* Add serialization compatibility test tool class

* Update RemotingSerializableCompatTest.java

* Update RemotingSerializableCompatTest.java

* Update RemotingSerializableCompatTest.java

* Update BitSet problem

* Update

* Update

* Update test

* Update test

* Update BUILD.bazel

* Update BUILD.bazel

* Update test

* Update BitSet problem

* Add test

* Add compat test

* merge develop

* Update test

* merge develop
2025-12-04 19:09:08 +08:00
rongtong 69e2524c45 Fix resource leak in IndexStoreFile shutdown (#9840)
* Fix resource leak in IndexStoreFile shutdown

Add cleanResources() calls after shutdown() for both mappedFile and compactMappedFile to ensure proper cleanup of memory-mapped buffers and file channels.

Change-Id: I2716b4e3b0cd281e89a9d5a00a389dc6048de3e7
Co-developed-by: Cursor <noreply@cursor.com>

* Add a FileNotFoundException check to allow breaking out of the loop after shutdown

Change-Id: Icc0063544ef91de8b2bd96a80f3829a6922cb0e6

* Add a FileNotFoundException check to allow breaking out of the loop after shutdown.

Change-Id: I53a7d2800775d80c1db8feb159c5c258b3517f09

---------

Co-authored-by: RongtongJin <user@example.com>
Co-authored-by: ShannonDing <dinglei@apache.org>
2025-11-14 11:32:35 +08:00
rongtong 2017630383 [ISSUE #9834] Support writeWithoutMmap in IndexStoreFile (#9835)
* Use FileChannel for writing when writeWithoutMmap is enabled in IndexStoreFile

Change-Id: I23ac2a4dfb8286cd8c3e51aeeb2d54d91136bc03
Co-developed-by: Cursor <noreply@cursor.com>

* Use FileChannel for writing when writeWithoutMmap is enabled in IndexStoreFile

Change-Id: I6a541d13c81a16f39b88ac91ce770717f60d64ff
Co-developed-by: Cursor <noreply@cursor.com>

* Remove unnecessary rewind() call in IndexStoreFile when using FileChannel

Change-Id: Id6deca99d34736761fd1d937f5eb5e75506ba1cc
Co-developed-by: Cursor <noreply@cursor.com>

* Add parameterized test for writeWithoutMmap in IndexStoreFileTest

Change-Id: I1ce92e4f728a018ff7bd3160995f395b480345ea
Co-developed-by: Cursor <noreply@cursor.com>

* Fix compactToNewFile to read data from file when using FileChannel

Change-Id: I9058b08f1bbcf646b5e7913260a5eefe853b9b16
Co-developed-by: Cursor <noreply@cursor.com>

* Fix newBuffer position and limit in compactToNewFile

Change-Id: I1a0100b6159d636a8db1124e3de416b3a2e62bc5
Co-developed-by: Cursor <noreply@cursor.com>

* Update test parameter order in IndexStoreFileTest

Change-Id: Ie01656574c1866badf726ead58a8ad2ca3d01cf7
Co-developed-by: Cursor <noreply@cursor.com>

---------

Co-authored-by: RongtongJin <user@example.com>
2025-11-13 17:25:31 +08:00
rongtong e60b67afe7 [ISSUE #9838] IndexStoreService use forceShutdown when disk is not writable (#9839)
* Update store and tieredstore implementation

Change-Id: I29c931148110c4238ce9a256ab2bf69d7b6139a8
Co-developed-by: Cursor <noreply@cursor.com>

* Fix UT NPE

Change-Id: I699ec538118166743308af9e78da50f2aa1b56b3

---------

Co-authored-by: RongtongJin <user@example.com>
2025-11-13 17:25:09 +08:00
lizhimins 35a747452c [ISSUE #9802] Fix the issue of delete logic in tiered storage index service (#9803) 2025-11-10 10:29:53 +08:00
lizhimins 455305a9d4 [ISSUE #9813] Fix buffer fetch thread pool starvation in tiered storage (#9818) 2025-11-07 11:51:34 +08:00
Sebastien Dionne 4b5f8665f9 Fix typos and linguistic errors in documentation (#9748)
Signed-off-by: Sebastien Dionne <survivant00@gmail.com>
2025-10-09 13:58:56 +08:00
cvictory 1c23a8094a [ISSUE #9663] Adopt Maven's revision mechanism (requires Maven >= 3.5.0) (#9665)
Co-authored-by: cvictory <shengli.caosl@alibaba-inc.com>
2025-09-04 09:51:02 +08:00
Duxuwei 74ab3ae112 [ISSUE #9648] Fix getOffsetInQueueByTime missing boundaryType in tieredMessageStore (#9649)
* [ISSUE #9648] Fix getOffsetInQueueByTime missing in tieredMessageStore

* Update TieredMessageStoreTest.java

* Delete inappropriate UT

* Remove unused import
2025-09-02 19:49:40 +08:00
Hongxu Xu 7ab7ddd6d2 [ISSUE #9609] Fix bazel CI and reduce dependencies (#9610) 2025-08-22 16:35:39 +08:00
lizhimins d8fa28c472 [ISSUE #9544] Must correct file size when init the file segment (#9545) 2025-07-14 14:08:35 +08:00
lizhimins cd6bec6291 [ISSUE #9441] Add file length check in IndexService (#9442) 2025-06-03 09:56:23 +08:00
lizhimins 286557a204 [ISSUE #9379] Set compact timestamp when timeStoreTable empty (#9419) 2025-05-21 17:25:19 +08:00
lizhimins aecb97ec44 [maven-release-plugin] prepare for next development iteration (#9389) 2025-05-07 16:34:04 +08:00
lizhimins 29d90a4ff9 [maven-release-plugin] prepare release rocketmq-all-5.3.3 (#9388) 2025-05-07 09:48:32 +08:00
lizhimins c66e758a1c [ISSUE #9379] Fix timeStoreTable delete logic in IndexService (#9384)
* [ISSUE #9379] Fix timeStoreTable delete logic in IndexService

* [ISSUE #9379] Fix delete logic from TimeStoreTable in IndexService
2025-05-06 15:31:03 +08:00
dingshuangxi888 6819b4d684 [ISSUE #9233] Fix query time boundary calculation in tiered storage (#9374) 2025-04-29 19:41:05 +08:00
lizhimins 8c623df7ad [ISSUE #9371] Delete ConsumeQueue index before CommitLog in tiered storage (#9372) 2025-04-29 10:31:49 +08:00
bxfjb 1babb32954 [ISSUE #9358] Timediff should multiply 1000 when query message from tiered storage (#9359) 2025-04-23 10:33:49 +08:00
bxfjb 729275c0c7 [ISSUE #9271] Enhance tiered storage getQueueOffsetByTimeAsync (#9272) 2025-03-27 13:57:47 +08:00
bxfjb 622c807653 [ISSUE #9233] Query message in tiered storage may fail for the first correct index file was not selected (#9234) 2025-03-13 17:02:58 +08:00
lizhimins 47fe6b2ba6 [maven-release-plugin] prepare for next development iteration (#9232) 2025-03-08 17:20:49 +08:00
lizhimins 2baaf044ea [maven-release-plugin] prepare release rocketmq-all-5.3.2 (#9231) 2025-03-08 14:07:17 +08:00
lizhimins 53fdc4ad3d [ISSUE #9213] Fix get the earliest time error when data is clean up in tiered storage (#9214)
* [ISSUE #9213] Fix get the earliest time error when data is clean up in tiered storag
2025-03-03 09:58:06 +08:00
lizhimins 7722ce78ee [ISSUE #9080] Not hold final message store config in fetcher (#9086) 2024-12-30 15:21:25 +08:00
yuz10 2089abd3be [ISSUE #9080] Fix tranfer logic when get large messages from cache in tiered storage (#9079) 2024-12-26 14:49:00 +08:00
wangshaojie4039 152a955ca4 [ISSUE #9069] Fix the IndexFile ConcurrentModificationException in tiered storage (#9071) 2024-12-25 17:02:06 +08:00
lizhimins fc2283008b [ISSUE #8976] Modify file segment construct method (#8977) 2024-11-26 14:16:38 +08:00
lizhimins e876bed084 [ISSUE #8955] Fix message buffer not release and dispatch thread exit in tiered storage (#8965) 2024-11-22 11:02:50 +08:00
Zhanhui Li b86059c7c1 Support LMQ dispatch in case if Consume Queue Store is RocksDB-based (#8842)
* feat: support LMQ dispatch

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: introduce group-commit for batch insertion of RocksDB KV pairs

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: propagate store error to broker module

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: fix all Bazel warning and errors

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>

* fix: remove unnecessary batch-ops when writing RocksDB using atomic flush

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: find a writable directory for RocksDB logs

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: clean up ConfigHelperTest

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: truncate consume queues in case commit log records are truncated

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: truncate LMQ max offsets

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: correct truncate boundary of consume queues

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: correct MessageExt encoding

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* chore: remove unused import

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>
2024-10-23 09:56:37 +08:00
lizhimins daf3d1a666 [maven-release-plugin] prepare for next development iteration (#8750) 2024-09-26 09:33:52 +08:00