mirror of
https://github.com/apache/rocketmq.git
synced 2026-08-28 20:09:14 +08:00
lizhimins
980f3d7f55
* [ISSUE #10373] Quarantine flaky tests and add detection plan docs Ran all RocketMQ module tests 100x across 10 ECS nodes to identify non-deterministic failures. Quarantined methods with @Ignore across broker, client, filter, and tieredstore modules. Flaky tests quarantined: - broker: LiteLifecycleManagerTest#testCleanByParentTopic (2%) - broker: ConsumerOrderInfoManagerLockFreeNotifyTest#testRecover (2%) - broker: TransactionalMessageServiceImplTest#testDeletePrepareMessage_maxSize (1%) - client: DefaultMQConsumerWithTraceTest#testPullMessage_WithTrace_Success (1%) - client: DefaultMQLitePullConsumerWithTraceTest#testSubscribe_PollMessageSuccess_WithCustomizedTraceTopic (5%) - client: DefaultMQLitePullConsumerWithTraceTest#testSubscribe_PollMessageSuccess_WithDefaultTraceTopic (6%) - filter: BloomFilterTest#testCheckFalseHit (1%) - tieredstore: IndexStoreServiceTest#queryCrossFileBoundaryTest (35%) - tieredstore: IndexStoreServiceTest#concurrentGetTest (1.5%) Additional changes: - LiteLifecycleManagerTest: Switch to MockitoJUnitRunner.Silent - Add flaky test detection plan docs (CN + EN) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [ISSUE #10373] Quarantine flaky PopPriorityIT and fix test cases - Quarantine PopPriorityIT at class level (multiple methods fail intermittently with 'expected:<8> but was:<2>' due to async race) - Fix ConsumerOrderInfoManagerLockFreeNotifyTest - Fix IndexStoreServiceTest Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * [ISSUE #10373] Fix flaky test detection plan docs path and naming Move English doc from docs/cn/ to docs/en/ and rename both files to match existing docs naming convention (underscore + PascalCase). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Tiered storage for RocketMQ (Technical preview)
RocketMQ tiered storage allows users to offload message data from the local disk to other cheaper and larger storage mediums. So that users can extend the message reserve time at a lower cost. And different topics can flexibly specify different TTL as needed.
This article is a cookbook for RocketMQ tiered storage.
Architecture
Quick start
Use the following steps to easily use tiered storage
- Change
messageStorePlugIntoorg.apache.rocketmq.tieredstore.TieredMessageStorein yourbroker.conf. - Configure your backend service provider. Change
tieredBackendServiceProviderto your storage medium implementation. We provide a default implementation: POSIX provider, and you need to changetieredStoreFilePathto the mount point of the storage medium for tiered storage. - Start the broker and enjoy!
Configuration
The following are some core configurations, for more details, see TieredMessageStoreConfig
| Configuration | Default value | Unit | Function |
|---|---|---|---|
| messageStorePlugIn | Set to org.apache.rocketmq.tieredstore.TieredMessageStore to use tiered storage | ||
| tieredMetadataServiceProvider | org.apache.rocketmq.tieredstore.metadata.DefaultMetadataStore | Select your metadata provider | |
| tieredBackendServiceProvider | org.apache.rocketmq.tieredstore.provider.PosixFileSegment | Select your backend service provider | |
| tieredStoreFilePath | Select the directory used for tiered storage, only for POSIX provider. | ||
| tieredStorageLevel | NOT_IN_DISK | The options are DISABLE, NOT_IN_DISK, NOT_IN_MEM, FORCE | |
| tieredStoreFileReservedTime | 72 | hour | Default topic TTL in tiered storage |
| tieredStoreGroupCommitCount | 2500 | The number of messages that trigger one batch transfer | |
| tieredStoreGroupCommitSize | 33554432 | byte | The size of messages that trigger one batch transfer, 32M by default |
| tieredStoreMaxGroupCommitCount | 10000 | The maximum number of messages waiting to be transferred per queue | |
| readAheadCacheExpireDuration | 1000 | millisecond | Read-ahead cache expiration time |
| readAheadCacheSizeThresholdRate | 0.3 | The maximum heap space occupied by the read-ahead cache |
Metrics
Tiered storage provides some useful metrics, see RIP-46 for details.
| Type | Name | Unit |
|---|---|---|
| Histogram | rocketmq_tiered_store_api_latency | milliseconds |
| Histogram | rocketmq_tiered_store_provider_rpc_latency | milliseconds |
| Histogram | rocketmq_tiered_store_provider_upload_bytes | byte |
| Histogram | rocketmq_tiered_store_provider_download_bytes | byte |
| Gauge | rocketmq_tiered_store_dispatch_behind | |
| Gauge | rocketmq_tiered_store_dispatch_latency | milliseconds |
| Counter | rocketmq_tiered_store_messages_dispatch_total | |
| Counter | rocketmq_tiered_store_messages_out_total | |
| Counter | rocketmq_tiered_store_get_message_fallback_total | |
| Gauge | rocketmq_tiered_store_read_ahead_cache_count | |
| Gauge | rocketmq_tiered_store_read_ahead_cache_bytes | bytes |
| Counter | rocketmq_tiered_store_read_ahead_cache_access_total | |
| Counter | rocketmq_tiered_store_read_ahead_cache_hit_total | |
| Gauge | rocketmq_storage_message_reserve_time | milliseconds |
How to contribute
We need community participation to add more backend service providers for tiered storage. PosixFileSegment, the implementation provided by default is just an example. People who want to contribute can follow it to implement their own providers, such as S3FileSegment, OSSFileSegment, and MinIOFileSegment. Here are some guidelines:
- Extend FileSegment and implement the methods of FileSegmentProvider interface.
- Record metrics where appropriate. See
rocketmq_tiered_store_provider_rpc_latency,rocketmq_tiered_store_provider_upload_bytes, androcketmq_tiered_store_provider_download_bytes - No need to maintain your own cache and avoid polluting the page cache. It already has the read-ahead cache.