Commit Graph

9164 Commits

Author SHA1 Message Date
Quan 577b89f2cd [ISSUE #10644] Add LMQ number gauge metric to Broker observability (#10645)
- Add GAUGE_LMQ_NUM constant in BrokerMetricsConstant
- Register ObservableLongGauge in BrokerMetricsManager reading from messageStore.getQueueStore().getLmqNum()
- Expose rocketmq_lmq_number metric for monitoring LMQ resource usage
2026-07-22 11:43:58 +08:00
Quan 4f9b187fd7 [ISSUE #10569] Support Lite Simple Consumer on server side (#10570)
- Broker: add PopLiteLongPollingService to NotificationProcessor for lite consumer notification polling
- Broker: LiteEventDispatcher notifies both PopLiteMessageProcessor and NotificationProcessor polling services
- Broker: add hasEvents(clientId) API to LiteEventDispatcher for message availability check
- Proxy: unify lite/normal pop paths into single popMessage call, route by ProxyContext.isLiteConsumer()
- Proxy: register LITE_SIMPLE_CONSUMER in ClientActivity and GrpcClientSettingsManager lifecycle
- Proxy: pass liteTopic property only for lite consumers in filter and response writer
- Proxy: remove standalone popLiteMessage from MessagingProcessor/ConsumerProcessor
- Remoting: add isLiteConsumer and clientId fields to NotificationRequestHeader
2026-07-20 10:46:46 +08:00
Quan 26cfb5f60e [ISSUE #10622] Resolve ConsumerOffsetManager dynamically in LiteEventDispatcher (#10623)
- Remove cached consumerOffsetManager field from LiteEventDispatcher
- Replace all usages with brokerController.getConsumerOffsetManager() for lazy resolution
- Allows downstream projects to swap ConsumerOffsetManager after broker init
2026-07-17 22:10:39 +08:00
Jiahua Wang 3373f284e6 [ISSUE #10613] Fix async request-reply RequestCallback firing multiple times (#10614)
- DefaultMQProducerImpl#request(Message, RequestCallback, long): drop the
  executeRequestCallback() call in the async send onSuccess so a send success
  no longer delivers a premature onSuccess(null); align with the other async
  request overloads which only set sendRequestOk here.
- RequestResponseFuture: add an AtomicBoolean executeCallbackOnlyOnce guard so
  the callback fires at most once even if the reply and timeout paths race.
- RequestFutureHolder#scanExpiredRequest: use ConcurrentHashMap.remove(key) to
  atomically claim ownership instead of iterator.remove(); also fix the log
  placeholder concatenation.
- ClientRemotingProcessor#processReplyMessage: use atomic remove(correlationId)
  and route the reply through executeRequestCallback so the single-shot guard
  covers the reply-success path too.
- Add RequestResponseFutureTest cases for success-then-timeout and concurrent
  single-callback semantics.

Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
2026-07-17 18:43:41 +08:00
imzs 45254d5e13 [ISSUE #10615] Fix missing long-polling notification under CombineConsumeQueue selective double-write (#10616) 2026-07-14 13:39:34 +08:00
Jiahua Wang 74e90f7c28 [ISSUE #10446] Support batch deletion of topics and subscription groups in broker (#10448)
* [ISSUE #10446] Support batch deletion of topics and subscription groups in broker

- Add DELETE_TOPIC_IN_BROKER_LIST and DELETE_SUBSCRIPTION_GROUP_LIST
  request codes with corresponding RequestBody types
- Implement deleteTopicList / deleteSubscriptionGroupList in
  AdminBrokerProcessor with deduplication and one-shot persist
- Add deleteTopicConfigList / deleteSubscriptionGroupConfigList in
  TopicConfigManager / SubscriptionGroupManager (single persist per batch)
- Add MQClientAPIImpl#deleteTopicInBrokerList /
  deleteSubscriptionGroupList client APIs
- Cover changes with unit tests in AdminBrokerProcessorTest

* fix: persist  config after batch delete

* fix: preserve batch delete failure response

---------

Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
Co-authored-by: fuchong <yubao.fyb@alibaba-inc.com>
2026-07-13 10:07:43 +08:00
imzs c4ce550a57 [ISSUE #10604] Fix TopicConfig DataVersion not persisted in split registration (#10605) 2026-07-10 18:09:59 +08:00
aias00 0e4ccf1b66 [ISSUE #10578] Fix master election maxOffset comparator overflow
DefaultElectPolicy sorted broker candidates by subtracting maxOffset values and casting the long delta to int. Large offset gaps can overflow the comparator result and rank a lower-offset broker first. This replaces subtraction with safe comparator helpers and adds a focused overflow regression test.

Constraint: Preserve existing election order: higher epoch, higher maxOffset, lower electionPriority

Rejected: Keep subtraction comparator | unsafe for long offset deltas greater than Integer.MAX_VALUE

Confidence: high

Scope-risk: narrow

Tested: mvn -q -pl controller -DskipTests=false -Dtest=DefaultElectPolicyTest -Djacoco.skip=true test

Tested: mvn -q -pl controller -DskipTests compile -Dspotbugs.skip=true -Dcheckstyle.skip=true

Tested: mvn -q -DskipTests compile -Dspotbugs.skip=true -Dcheckstyle.skip=true

Not-tested: Full controller suite on local JDK due existing JaCoCo/Hessian module-access failures

Related: #10578
2026-07-07 19:37:13 +08:00
yx9o 2af604f3af [ISSUE #10593] Rename CLusterSendMsgRTCommand to ClusterSendMsgRTCommand (#10594)
* [ISSUE #10593] Rename CLusterSendMsgRTCommand to ClusterSendMsgRTCommand

* Update
2026-07-06 10:27:33 +08:00
Quan 8242c1e9d2 [ISSUE #10562] Optimize PopConsumerServiceTest#transferToFsStoreTest to avoid unnecessary background threads (#10563)
- Replace consumerService.start() with consumerService.getPopConsumerStore().start()
- Replace consumerService.shutdown() with consumerService.getPopConsumerStore().shutdown()
- Avoid starting revive/cache background threads that are not needed for transferToFsStore verification
- Reduces test execution time from ~60s to ~1.3s
2026-07-01 14:03:54 +08:00
Quan 1730dbc040 [ISSUE #10560] Remove enableLiteEventMode config switch (#10561)
* [ISSUE #10560] Remove enableLiteEventMode config switch

- Remove enableLiteEventMode field, getter and setter from BrokerConfig
- Remove 5 early-return guard checks in LiteEventDispatcher
- Remove dead condition in PopLiteMessageProcessor.popLiteTopic
- Simplify getEventIterator to always use event-set path
- Delete unused LiteSubscriptionIterator inner class
- Remove disabled-mode test cases and stale Javadoc references

* chore: retrigger CI
2026-07-01 13:50:53 +08:00
SSpirits 88709c56aa [ISSUE #10543] resolve data race in ServiceThread wakeup mechanism (#10544) 2026-06-30 10:26:23 +08:00
Surya Kant f4c0c4d3a4 docs: format README for consistent Markdown style (#10551)
Normalize horizontal rules, blank lines around headings and code
blocks, list markers, nested list indentation, and trailing whitespace
without changing content.
2026-06-29 19:16:54 +08:00
Quan 49102a9985 [ISSUE #10549] Fix lite topic reset offset: memory leak, FIFO block bypass, and offset-0 reset failure (#10550)
* [ISSUE #10549] Fix lite topic reset offset: memory leak, FIFO block bypass, and offset-0 reset failure

- Fix memory leak in removeResetOffset: clean up empty inner map entries from resetOffsetTable
- Add eraseResetOffset for precise cleanup on lite topic removal
- Skip FIFO block check in isFifoBlocked when server-side reset offset is pending
- Fix ResetOffsetByTimeCommand: change resetOffset > 0 to >= 0 to allow resetting to offset 0
- Add unit tests for eraseResetOffset and isFifoBlocked reset bypass

* chore: empty commit to trigger CI pipeline
2026-06-29 15:34:00 +08:00
Quan 10d498c68a [ISSUE-10546] Add default implementation for rejectRequest() in NettyRequestProcessor (#10547) 2026-06-25 10:22:35 +08:00
chenxu80 2e6632ff40 [ISSUE #10417] Fix stale minOffset after consumeQueue truncation (#10418) 2026-06-23 17:10:16 +08:00
rongtong b5bc1ff5d6 Fix flaky HATest semi-sync replication (#10495) 2026-06-21 09:37:25 +08:00
LeiXiaoGao c4ae761f09 docs: fix broker container reference and producer typos (#10449) 2026-06-20 10:54:59 +08:00
echooymxq bfbbb2a955 [#10421] Fix Timer message rocksdb use wrong cache key. (#10422) 2026-06-20 10:44:49 +08:00
yin_bo_ e213756971 [ISSUE #10419] Fix NPE when class filter route data is missing (#10420)
* [ISSUE #10419] Fix NPE when class filter route data is missing

* [ISSUE #10419] Add class filter route fallback tests
2026-06-20 10:27:55 +08:00
Jiahua Wang c270c60f85 [ISSUE #10441] Reduce per-RPC allocation in metrics by caching static AttributeKey instances (#10443)
Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
2026-06-19 09:41:57 +08:00
Jiahua Wang 5977e9cb1f [ISSUE #10522] Reduce Remoting header encoding allocation via FastCodesHeader/RocketMQSerializable optimizations
- Add writeDecimalLong/writeDecimalInt to RocketMQSerializable for direct primitive decimal encoding

- Add writeLong/writeInt helpers to FastCodesHeader

- Optimize writeIfNotNull to route Long/Integer through primitive writers

- Cache single-byte ASCII strings in RocketMQSerializable.readStr

- Right-size HashMap initial capacity in mapDeserialize (128 -> 24)

Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
2026-06-19 09:41:19 +08:00
lizhimins f941dced39 [ISSUE #10521] Use madvise(MADV_RANDOM) to disable kernel read-ahead during correctMinOffset binary search (#10523) 2026-06-17 19:43:16 +08:00
Jiahua Wang 971d568433 [ISSUE #10515] Fix TransactionMetricsFlushService busy spin bug (#10517)
Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
2026-06-17 15:01:48 +08:00
Jiahua Wang 9b91157f30 [ISSUE #10486] Add getMessageType(Map) overload to eliminate redundant properties decode (#10487)
BrokerMetricsManager.getMessageType(SendMessageRequestHeader) is called
once per send to classify the message. It internally decodes the
properties String into a HashMap, but the typical caller
(SendMessageProcessor) has already decoded the same String moments
before. The result is a redundant decode allocation per send (one
HashMap + ~14 String substrings + one Node[]).

This commit adds a public overload getMessageType(Map<String, String>)
that lets callers pass an already-decoded Map and reuse it. The
existing SendMessageRequestHeader overload now delegates to the new
overload; behavior is unchanged for callers that don't have a decoded
Map. Downstream callers (e.g. SendMessageProcessor) can switch to the
new overload in a separate broker-layer commit.

Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
2026-06-17 14:11:01 +08:00
Jiahua Wang e18f0d2c2c [ISSUE #10511] Replace Enum.values() loop with static array lookup in LanguageCode and SerializeType (#10513)
Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
2026-06-17 14:10:28 +08:00
KaiQuan Chen 59a70d9b80 Update ACL mqadmin commands in Chinese doc Section 7 (#10504)
Replace five outdated commands (updateAclConfig, deleteAccessConfig,
updateGlobalWhiteAddr, clusterAclConfigVersion, getAclConfig) with
the six commands available in RocketMQ 5.5.0:

  createAcl, updateAcl, deleteAcl, getAcl, listAcl, copyAcl

Verified: all five old commands return 'sub command not exist' on
RocketMQ 5.5.0. Each new command documented with parameters and
usage examples from actual mqadmin -h output.

Fixes #10502
2026-06-16 13:55:38 +08:00
KaiQuan Chen 226e24faf6 Fix outdated constructor signatures in trace documentation (#10501)
The two-argument constructors DefaultMQProducer(String, boolean) and
DefaultMQPushConsumer(String, boolean) do not exist in 5.x.
Updated to use the three-argument form with null for the default
trace topic.

Fixes #10500
2026-06-15 20:25:30 +08:00
KaiQuan Chen 4082e31d88 [ISSUE #8262]Fix bitsarray off by one (#10506)
* Update ACL mqadmin commands in Chinese doc Section 7

Replace five outdated commands (updateAclConfig, deleteAccessConfig,
updateGlobalWhiteAddr, clusterAclConfigVersion, getAclConfig) with
the six commands available in RocketMQ 5.5.0:

  createAcl, updateAcl, deleteAcl, getAcl, listAcl, copyAcl

Verified: all five old commands return 'sub command not exist' on
RocketMQ 5.5.0. Each new command documented with parameters and
usage examples from actual mqadmin -h output.

Fixes #10502

* Fix off-by-one in BitsArray boundary checks and add unit tests

**Bug fix**
checkBytePosition and checkBitPosition used '>' instead of '>=',
allowing positions equal to array length to pass validation and
cause ArrayIndexOutOfBoundsException instead of the intended
IllegalArgumentException.

**Production impact if not fixed**
Minimal. getByte/setByte are only called internally by xor/or/and
with safe bounds. getBit/setBit positions come from BloomFilterData
hash functions which always produce in-range values. The incorrect
check has no known production trigger.

**Impact of fix**
Zero negative impact. All valid positions (0 to length-1) unchanged.
Edge case now correctly throws IllegalArgumentException instead of
ArrayIndexOutOfBoundsException.

**Tests added**
- BitsArrayTest: 27 tests covering create, bit/byte ops, boundary,
  including regression tests for the fixed checks
- PlainAccessConfigTest: 12 tests for getters/setters, equals/hashCode

References #8262

* Remove self-comparison assertion flagged as error by CI
2026-06-15 20:23:50 +08:00
RockteMQ-AI 7347c44b06 [ISSUE #10510] Add gRPC server permit keepalive configuration (#10516) 2026-06-15 16:43:51 +08:00
SGloria b4d92c06b7 [ISSUE #10492] Add debug logging for swallowed exceptions in StatsAllSubCommand (#10493)
Previously, multiple catch blocks in StatsAllSubCommand silently
swallowed exceptions, making it difficult to diagnose issues when
the `mqadmin statsAll` command failed to retrieve statistics from
certain brokers.

This commit adds debug-level logging to all empty catch blocks,
recording the exception details along with contextual information
(topic, group, broker address) to improve observability.

Co-authored-by: fire <gloria_404@126.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-15 13:42:09 +08:00
SGloria da58317516 [ISSUE #10216] Fix potential NPE in EscapeBridge when topicPublishInfo is null (#10507)
In asyncPutMessage() and asyncRemotePutMessageToSpecificQueue(), the
return value of tryToFindTopicPublishInfo() is dereferenced without a
null check, causing NullPointerException when topic route information
is unavailable (e.g., during startup or after nameserver disconnection).

This commit adds null/validity checks consistent with the existing
pattern in putMessageToRemoteBroker(), returning PUT_TO_REMOTE_BROKER_FAIL
with a warning log instead of crashing with NPE.

Co-authored-by: fire <gloria_404@126.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-15 13:40:42 +08:00
quzizii c4f18d96bb [DOCS] Fix typos in documentation filenames (#10505)
Rename Design_LoadBlancing.md to Design_LoadBalancing.md
Rename Design_Trancation.md to Design_Transaction.md
2026-06-15 09:43:26 +08:00
fancy 3a568241f9 docs: fix dead link to tiered store config in tieredstore README (#10484)
The 'Configuration' section linked to TieredMessageStoreConfig.java under
the tieredstore/common package, but that class has been renamed to
MessageStoreConfig and moved to the tieredstore package root, so the link
returns 404. Point it at the current MessageStoreConfig.java, which holds
the tiered store configuration fields documented in the table below
(tieredStoreFilePath, tieredBackendServiceProvider, tieredStorageLevel, ...).

Signed-off-by: fancy-agent <fancyboi66@gmail.com>
2026-06-13 11:17:28 +08:00
fancy c8301910cb docs: remove dead Slack invite link from README (#10483)
The Slack invite link (rocketmq-invite-automation.herokuapp.com) returns
HTTP 404 — the Heroku-hosted invite app is gone (Heroku retired free
dynos in Nov 2022). The project site no longer lists Slack anywhere
(home, docs, and the contact page have no Slack reference), so this
removes the dead link to keep the contact list accurate.

Signed-off-by: fancy-agent <fancyboi66@gmail.com>
2026-06-13 11:17:10 +08:00
lizhimins 91cb333716 [ISSUE #10462] Improve error handling in tiered storage (#10473) 2026-06-12 15:45:06 +08:00
imzs 1ab9689f34 [ISSUE #10450] Selective double-write in CombineConsumeQueueStore (#10452) 2026-06-11 21:14:18 +08:00
imzs 8a589c6a53 [ISSUE #10453] Fix unexpected decrement of lmq counter (#10454) 2026-06-11 21:11:08 +08:00
lizhimins cf9113fce9 [ISSUE #10462] Fix resource leaks and lifecycle issues in tiered storage (#10476) 2026-06-11 20:47:54 +08:00
Jiahua Wang 801d6c0a2f [ISSUE #10467] Reduce per-message allocation in createUniqID and valueOfMagicCode (#10469)
- MessageClientIDSetter.createUniqID(): reuse a ThreadLocal<char[]>
  instead of allocating a new char[LEN*2] on every send.
- MessageVersion.valueOfMagicCode(): replace Enum.values() array copy +
  O(n) loop with direct if-else on the two known magic codes.

Co-authored-by: wangjiahua.wjh <wangjiahua.wjh@alibaba-inc.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-11 20:26:12 +08:00
lizhimins 78e680b811 [ISSUE #10462] Standardize log format across tiered storage module (#10474) 2026-06-11 19:23:52 +08:00
lizhimins 84fc240acc [ISSUE #10462] Optimize thread pool sizes for commit and fetch executors (#10472) 2026-06-11 19:21:09 +08:00
Harsh Mehta 68cc743881 [Fix #10460]: Pre-size ProxyContext map to avoid HashMap resize on request path (#10461)
* fix: initialize HashMap with an initial capacity in ProxyContext

Signed-off-by: Harsh Mehta <harshmehta010102@gmail.com>

* fix: initialize HashMap with an initial capacity in ProxyContext

Signed-off-by: Harsh Mehta <harshmehta010102@gmail.com>

* fix: changed initial capacity to 64

Signed-off-by: Harsh Mehta <harshmehta010102@gmail.com>

---------

Signed-off-by: Harsh Mehta <harshmehta010102@gmail.com>
2026-06-11 16:14:41 +08:00
Chuan 976e9a3edd [ISSUE #10435] Fix shared ClientChannelInfo preventing inactive channel expiry (#10437) 2026-06-11 15:36:39 +08:00
lizhimins 21562bba0e [ISSUE #10462] Fix concurrency bugs in tiered storage (#10471) 2026-06-11 15:29:56 +08:00
Chuan da9b26bd2d [ISSUE #10438] Fix buildTopicConfigSerializeWrapper exposing live topicConfigTable reference (#10439) 2026-06-11 14:52:43 +08:00
quzizii 86d1df4917 [ISSUE #10475] Add unit tests for CleanupPolicyUtils 2026-06-11 14:24:09 +08:00
lizhimins 50ae7c5093 [ISSUE #10462] Add memory backpressure to dispatch (#10470) 2026-06-11 14:14:11 +08:00
Aurélien Pupier be38a8211b Configure GitHub workflows to use concurrency cancel-in-progress for (#10445)
See recommended best practices at Apache
https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=INFRA&title=GitHub+Actions+Recommended+Practices

Signed-off-by: Aurélien Pupier <apupier@ibm.com>
2026-06-11 10:53:22 +08:00
LeiXiaoGao 8997e8cdf0 fix: validate message batch input (#10459) 2026-06-11 10:12:22 +08:00