829 Commits

Author SHA1 Message Date
lizhimins daf3d1a666 [maven-release-plugin] prepare for next development iteration (#8750) 2024-09-26 09:33:52 +08:00
lizhimins b122c8109d [maven-release-plugin] prepare release rocketmq-all-5.3.1 (#8749) 2024-09-25 23:24:56 +08:00
LetLetMe 3b5cbf86df [ISSUE #8698] Remove batch write in kv cq store and update rocksdb cq check tool (#8739) 2024-09-25 19:32:47 +08:00
LetLetMe 525f877f3b [ISSUE #8589] Support file format CQ and json format offset in-place upgrade to rocksdb management (#8600) 2024-09-23 19:24:15 +08:00
Liu Shengzhong 280804c559 [ISSUE #8693] Fix checking MultiDispatchMessage when appending commitlog 2024-09-18 13:57:20 +08:00
Dongyuan Pan ba8148f746 [ISSUE #8669] Fix crc 32 overflow when lmq 2024-09-11 17:06:40 +08:00
Vincent Lee adf110fbaa [ISSUE #8660] Should use read only getConsumeQueue instead of findConsumeQueue in read only func (#8659) 2024-09-09 16:59:44 +08:00
小陈 4c51706b28 [ISSUE #8623] Temporarily skip flaky unit tests on macOS (#8633)
* Skip flaky tests on macOS

* Trigger ci

* Remove branch trigger
2024-09-04 11:14:34 +08:00
Lei Zhiyuan 1a15729ca9 [ISSUE #8601]When isPopShouldStop hit,unlock queueLockManager (#8602)
* fix:when isPopShouldStop hit, unlock queueLockManager

* fix:when isPopShouldStop hit, unlock queueLockManager

* fix: limit rate of appending commit in case of DLedger commit-log

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>

---------

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>
Co-authored-by: Zhanhui Li <lizhanhui@gmail.com>
2024-08-30 13:43:01 +08:00
LetLetMe 2a938fb1bc [ISSUE #8058] Support for upgrading metadata in json to rocksdb (#8571) 2024-08-23 11:18:03 +08:00
Lei Zhiyuan 09beca60eb [ISSUE #8549] Ipv6 enabled in broker, pickupStoreTimestamp size should be 20 (#8567)
* fix: when ipv6 enabled in broker, pickupStoreTimestamp size should be 12(20-8)

* fix: when ipv6 enabled in broker, pickupStoreTimestamp size should be 20
2024-08-21 13:44:26 +08:00
imzs aab646ce83 [ISSUE #8460] Improve the pop revive process when reading biz messages from a remote broker (#8475)
* [ISSUE #8460] part1: add extra information to the call chain of remote message reading

* [ISSUE #8460] part2: add exponential backoff and ending condition of CK rewrite, and fix checkstyle

* [ISSUE #8460] exclude test, BrokerOuterAPITest passed locally, but failed on bazel.
2024-08-02 14:43:21 +08:00
rongtong 3696be0632 [ISSUE #8463] Some statistical items should also be deleted to prevent memory leakage when a topic or group is deleted (#8464)
* Some important statistical items should also be deleted to prevent memory leakage when a topic or group is deleted

* Add UTs
2024-08-01 10:04:11 +08:00
Qiu 263f0fbdd6 [ISSUE #8448] commitlog class comment optimize 2024-07-29 13:46:46 +08:00
rongtong 86d59d2485 Add the ability to write ConsumeQueue using fileChannel to prevent JVM crashes in some situations (#8403) 2024-07-22 17:20:11 +08:00
Dongyuan Pan 259efdb086 [ISSUE #8350] Properties store error: crc32ReservedLength make undefine memory in properties when no enable AppendPropCRC (#8351) 2024-07-16 17:24:14 +08:00
lizhimins 5dc0c6a8b5 [maven-release-plugin] prepare for next development iteration (#8382) 2024-07-09 01:30:43 +08:00
lizhimins 7ee50ec32e [maven-release-plugin] prepare release rocketmq-all-5.3.0 (#8379) 2024-07-08 23:15:33 +08:00
rongtong 3aa5d1936f Adjust the default value of ackMessageThreadPoolNums to 16 to prevent performance bottlenecks during high traffic. (#8337) 2024-07-02 17:08:54 +08:00
rongtong d60198f621 [ISSUE #8239] Fix the issue of potential message loss after a crash under synchronous disk flushing configuration. (#8240) 2024-06-07 13:44:13 +08:00
mxsm 144b22ba7d [ISSUE #8235]Add @Override annotation for handleDiskFlush method (#8236) 2024-06-02 07:13:15 +08:00
yx9o 152055632d [ISSUE #8222] Fix spelling errors in comments (#8224) 2024-05-30 10:10:03 +08:00
oopooa c00fac3c77 [ISSUE #8148] Fix variable typo 2024-05-16 09:56:09 +08:00
mxsm b37d283793 [ISSUE #8044]Add Override annotation for AllocateMappedFileService#run (#8045) 2024-04-23 23:33:55 +08:00
littleboy 9d07cc7c51 [ISSUE #8032] Set checkDupInfo value from config 2024-04-18 14:20:16 +08:00
hqbfz bf24ffde88 [ISSUE #7979] Fix timerWheel message metric (#7980)
* fix metric in TimerWheel

* fix metric in TimerWheel

* fix message metric in TimerWheel

* fix message metric in TimerWheel
2024-04-01 10:47:35 +08:00
koado fb6f9e44f4 [ISSUE #7961] use BoundaryType in binarySearchInCQByTime
Co-authored-by: 凯铎 <zhengkaifeng.zkf@alibaba-inc.com>
2024-03-28 16:59:20 +08:00
Lei Zhiyuan 27d8375940 [ISSUE #7836] flush_behind_bytes wrong in transientStorePoolEnable not enable 2024-03-24 18:58:19 +08:00
Humkum 3c27bfe5ed [ISSUE #7902] fix:reput thread may quit by throwing error 2024-03-14 16:03:51 +08:00
ChineseTony 3877dcd134 [ISSUE #7853] Fix the manualDeleteFileSeveralTimes count 2024-02-29 10:48:31 +08:00
rongtong f41160210a [maven-release-plugin] prepare for next development iteration (#7828) 2024-02-12 11:07:03 +08:00
rongtong 281a7b32f0 [maven-release-plugin] prepare release rocketmq-all-5.2.0 (#7827) 2024-02-11 17:53:50 +08:00
bazelisky af1936d93a [ISSUE #7715] Update Bazel toolchain to make the CI work (#7714)
* Update bazel toolchain

Signed-off-by: Bazelisky <bazelisk@hotmail.com>

* fix a bad test case as it violates requirement of ConcurrentHashMap#computeIfAbsent

Signed-off-by: Bazelisky <bazelisk@hotmail.com>

* Exclude flaky tests from test suite

Signed-off-by: bazelisky <155456828+bazelisky@users.noreply.github.com>

* fix: format issue

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>

* fix: Remove unused import

* fix deps issue

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>

* Expose data property of java_test, allowing to specify run_file

Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>

---------

Signed-off-by: Bazelisky <bazelisk@hotmail.com>
Signed-off-by: bazelisky <155456828+bazelisky@users.noreply.github.com>
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com>
Co-authored-by: Zhanhui Li <lizhanhui@gmail.com>
2024-02-01 16:52:55 +08:00
schopenhauerz 7b64f50137 [ISSUE #7446] Add config of maxFilterMessageSize (#7447)
* bugfix broker boot succes but get fail ip addr

bug:
broker ip addr(IPV4) get fail  after broker start up;
fix:
add compare ,continue  when ip is start with '0.' ;
like :
The broker[broker-a, 0.0.1.1:10911] boot success. serializeType=JSON and name server is 127.0.0.1:9876;

* add config  maxFilterMessageSize , not hard code

* Pass the code check style

---------

Co-authored-by: mingzhi.zhang <mingzhi.zhang@jiduauto.com>
Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
2024-01-31 03:34:27 +08:00
Ji Juntao 6d7513425c [RIP-46] Enhanced metrics for timing and transactional messages (#7500)
* add request codes' distribution and timing messages' distribution

* remove the requestCode distribution.

* add delay message latency distribution.

* add transaction metrics

* transaction metric of topics finished, v1.

* add the transaction metrics, to be tested.

* fix the judgement of putMessageResult

* optimize.

* add config.

* fix test case.

* add unit tests for transactionMetrics.

* remove chinese character

* add rocksdb metrics.

* add more rocksdb metrics.

* fix NPE

* avoid the total time is 0.

* add license

* remove useless import.
2024-01-22 16:56:05 +08:00
Lei Zhiyuan 920dc32d27 [ISSUE #7760] make timerSkipUnknownError can be set by config file 2024-01-19 09:59:23 +08:00
bxfjb 646e2a4942 [ISSUE #7355] fix dledger recover abnormally may lost consume queue of tail (#7599)
* fix dledger recover abnormally may lost consume queue of tail

* fix correct storeTimestampPosition when bornhost is v6

* fix correct SYSFLAG offset

---------

Co-authored-by: 赵宇晗 <zhaoyuhan@xiaomi.com>
2024-01-18 15:43:01 +08:00
Zhouxiang Zhan b1d8d306a5 [ISSUE #7699] Add namespace v2 in client (#7700)
* Add namespace v2

* Add NamespaceRpcHook

* Refector extends header

* Use Boolean in request header to remove unnecessary encode

* Add unit test

* Add NamespaceRpcHookTest

* Remove GrpcConverter#wrapResourceWithNamespace

* Optimize readability of RpcRequestHeader
2024-01-12 10:46:15 +08:00
Ji Juntao 2c898c9b31 [ISSUE #7689] In Controller mode, messages may lost due to sharing the same cq offset (#7690)
* fix the reput bug.

* add more logs.

* refactor the method of compensating for HA.

* not modify the imports.

* refactor the log.

* refactor the log.
2023-12-21 14:41:51 +08:00
EvanMi f0a3e933b9 [ISSUE #7684] Fix iterator.remove() bug (#7682)
* bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating

* add testcase

---------

Co-authored-by: mipengcheng3 <mipengcheng3@jd.com>
2023-12-20 14:34:53 +08:00
YASH PATEL 9dec4cf5fe [ISSUE #7669] map variable delayLevelTable changed to ConcurrentSkipListMap from ConcurrentHashMap (#7675) 2023-12-19 19:47:13 +08:00
rongtong 80c0330f75 ConfirmOffset directly takes the max offset when allAckInSyncStateSet is false (#7657) 2023-12-18 10:10:28 +08:00
rongtong 1e69e3904f Fix flaky test testSemiSyncReplicaWhenAdaptiveDegradation (#7631) 2023-12-12 14:17:24 +08:00
Zhanhui Li bcc9db5cba [ISSUE #7614] Fix flaky test RocksDBMessageStoreTest (#7625)
* fix #7614 Flaky test RocksDBMessageStoreTest

Signed-off-by: lizhanhui <lizhanhui@gmail.com>

* fix: give TimerMessageStoreTest#testStateAndRecover more time for Awaitability to poll and check

Signed-off-by: lizhanhui <lizhanhui@gmail.com>

* clean up exclude test list of bazel

Signed-off-by: lizhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: lizhanhui <lizhanhui@gmail.com>
2023-12-11 17:37:01 +08:00
gaoyf faae64715d [ISSUE #7601] Fix slave acting master bug (#7603)
* fix NullPointerException when message escape to remote

* fix NumberFormatException when message retry to escape to remote

* fix timerCheckPoint of the master is not updated, causing the timer message to be replayed after master is restarted

* Use properties copies instead of referencing the same map when converting message
2023-12-07 11:25:22 +08:00
YASH PATEL 56e886bf70 [ISSUE #7592] testCleanBuffer unit test modifies, changed non-direct … (#7593)
* [ISSUE #7592] testCleanBuffer unit test modifies, changed non-direct to direct buffer allocation

* fix: consolidate UtilAll#cleanBuffer by checking if the given buffer is direct or not

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
2023-11-29 12:32:38 +08:00
AYue e955e4399c [ISSUE #7577] SlaveActingMaster Timer Message retry without escape logic (#7578)
Co-authored-by: ayue <ericyu0421@163.com>
2023-11-24 10:47:08 +08:00
guyinyou 8e7e2b5f50 [ISSUE #7574] Fix RunningFlags conflict
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-11-20 19:32:57 +08:00
Zhanhui Li 15d32db03b [ISSUE #7547] Let consumer be aware of message queue assignment change (#7548)
* let consumer be aware of message queue assignment change

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* add unit test for DefaultMQPushConsumer#setMessageQueueListener

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: bazel build warnings

Signed-off-by: Zhanhui Li <lizhanhui@apache.org>

* fix: set MixCommitlogTest test size as medium

Signed-off-by: Zhanhui Li <lizhanhui@apache.org>

* allow cache bazel test results

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix code style issue by removing unused imports

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix #7552

Signed-off-by: Zhanhui Li <lizhanhui@apache.org>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Zhanhui Li <lizhanhui@apache.org>
2023-11-13 09:44:25 +08:00
Dongyuan Pan ead3d90501 [ISSUE #7511] Lock granularity issue causing LMQ message loss (#7525)
* bug fix: assignOffset and increaseOffset in LMQ has concurrency issues in topicQueueLock, should be in putMessageLock

* fix MultiDispatchTest

* fix MultiDispatchTest

* fix unit test
2023-11-08 10:40:52 +08:00