Commit Graph
580 Commits
Author SHA1 Message Date
guyinyouandguyinyou 72befa0fb4 1. change the compactionTopic dispatch process from asynchronous to synchronous (#6530)
2. compactionTopic dispatch performance improvement

Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-04-04 10:39:04 +08:00
mxsm 66cf564c67 [ISSUE #6523] Schematic diagram of adding BatchConsumeQueue storage unit (#6524) 2023-04-02 09:52:48 +08:00
guyinyouandguyinyou e7f29798ec [ISSUE #6518] Fix bug that multi-threaded using bytebuffer
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-03-31 17:03:37 +08:00
fujian-zfj 0d516ba708 [ISSUE #6473] Fix multi dispatch error when enableMultiDispatch=true and enableLmq=true (#6476)
* typo int readme[ecosystem]

* remove multi_dispatch to avoid building cq again

* if topic starts with %RETRY%, do not process INNER_MULTI_DISPATCH

* quickstart consumer test revert
2023-03-29 14:05:07 +08:00
guyinyouandguyinyou cbd1d38199 flush() in compactionTopic (#6498)
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-03-29 11:46:03 +08:00
fuyou001andRongtongJin 17aa86c618 [ISSUE #6324] Improving compact topic stability (#6353)
Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
2023-03-28 10:41:23 +08:00
guyinyouandguyinyou 9ccfcfbd13 [ISSUE #6478] Fix outTPS in mqadmin when using compactionTopic
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-03-27 16:24:08 +08:00
guyinyouandguyinyou 654c12ed56 [ISSUE #6430] Scan topic.json to find compactionTopic and copy it
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-03-27 15:25:29 +08:00
Ji JuntaoandLi Zhanhui 1818c2d507 [ISSUE #4890] BugFix: fix binary search consume offset by time (#6429)
* BugFix: fix binary search consume offset by time

* polish codes.

* modify the test case: when timestamp is skewing, only find lower bound.

* modify the test case: when timestamp is skewing, only find lower bound.

* modify the test case: when timestamp is skewing, only find lower bound.

---------

Co-authored-by: Li Zhanhui <shutian.lzh@alibaba-inc.com>
2023-03-24 10:15:41 +08:00
hiyoandLi Zhanhui 0faa0a2c7d [ISSUE #6414] polish MQClientAPIImpl.getDefaultTopicRouteInfoFromName… (#6452)
* [ISSUE #6414] polish MQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer parameters

* Fix to make Bazel CI pass

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* Fix typo

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
2023-03-23 20:30:21 +08:00
guyinyouandguyinyou c17baf1e33 [ISSUE #6386] Some improvements for compactionTopic (#6387)
* 1、Graceful shutdown needs to wait for the compactionTopic to build the index
2、Exception handling that may occur when adding pullMessageFromMaster
3、Update usage docs

* fix doc

---------

Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
2023-03-22 11:23:13 +08:00
TheR1sing3un d75ab111f3 [ISSUE #6406] Add more visual comments on IndexFile & IndexHeader & ConsumeQueue
1. add more visual comments
2023-03-21 08:42:40 +08:00
fuyou001 7dbcdc8f96 [ISSUE #6372] new scheduledExecutor clean consume queue (#6376) 2023-03-17 16:48:50 +08:00
fujian-zfj 4e0a723d22 [ISSUE #6344] queueOffsets in topicQueueTable rollback when master changes to master in ha mode (#6345)
* typo int readme[ecosystem]

* master to master, no need to recover topicQueueTable

* revert

* optimize

* fix bug found by test

* remove unused log
2023-03-17 09:26:38 +08:00
Ji Juntao 63a014b810 [ISSUE#6342] Local SyncStatSet sync to remote value when changeToMaster (#6352)
* fix ISSUE#6342

* refactor

* correct the syncStateSet in elect process.

* move syncStateSet into request/response's body.

* optimize the broker electing switch's branch.
2023-03-16 11:48:04 +08:00
Lei Zhiyuan 3db8d039d1 [ISSUE #6347] fix: use mutable list to avoid start fail when enable tiered store 2023-03-16 11:01:34 +08:00
redlsz 4a4626df63 [ISSUE #6343] fix consumeQueueExtDir not deleted when deleting topic (#6351) 2023-03-15 16:08:20 +08:00
hardyfish 94db42ade7 [ISSUE #6292] fix typos (#6293)
* fix typos

* fix typos

* fix typos
2023-03-15 15:41:01 +08:00
RongtongJin fac699ea4d Fix some spelling problems 2023-03-14 09:40:15 +08:00
TheR1sing3un d802231163 fix(store): fix conflicts after rebase
1. fix conflicts after rebase
2023-03-14 09:40:15 +08:00
TheR1sing3un ce418c907c feat(broker): add more logs when broker register to controller
1. add more logs when broker register to controller
2023-03-14 09:40:15 +08:00
RongtongJin 578a42803c Fix bug that state not match when handshake 2023-03-14 09:40:15 +08:00
RongtongJin bbf54f8a71 Modify StorePathMetadata to StorePathBrokerIdentity 2023-03-14 09:40:15 +08:00
RongtongJin 3a141868b7 Fix some typos and log output 2023-03-14 09:40:15 +08:00
TheR1sing3un 5a2aea7997 refactor(controller): remove meaningless attribute MessageStoreConfig#storePathTempMetadata
1. remove meaningless attribute
`MessageStoreConfig#storePathTempMetadata`
2023-03-14 09:40:15 +08:00
TheR1sing3un 92fac4c10f fix(store): fix wrong value of MessageStoreConfig#storePathMetadata and storePathTempMetadata
1. fix wrong value of MessageStoreConfig#storePathMetadata and
storePathTempMetadata
2023-03-14 09:40:15 +08:00
TheR1sing3un c494a43c52 refactor(controller): optimize some code suggested by reviewer
1. optimize some code suggested by reviewer
2023-03-14 09:40:15 +08:00
TheR1sing3un 0c2f480c3a fix(broker): mark unless attribute with @Deprecated
1. mark unless attribute with @Deprecated
2023-03-14 09:40:15 +08:00
+6 3a69aae701 [ISSUE #6023] Add a unit test to verify new register process in broker with controller mode (#6024)
* refactor: simplify getPID (#5962)

* [ISSUE #5923] Add example tiered storage backend service provider (#5926)

* implement example file segment

* add metrics

* add readme

* fix license

* fix tests

* fix links in README.md

* add comment to PosixFileSegment and mark as experimental

* fix test

* optimize image quality

* Remove the useless exception class: MQRedirectException #5963

* [ISSUE #5965] Fix lmqTopicQueueTable initialization (#5968)

* [ISSUE #5965] Fix lmqTopicQueueTable initialization

* [ISSUE #5965] Fix lmqTopicQueueTable initialization

* [ISSUE #5890] Fix dledger logging (#5959)

* Fix dledger logging

* Add bridge into store module

* [ISSUE #5860] Set the value of order when create or update topic (#5861)

* [ISSUE #5939]Adjust the MQClientInstance#sendHeartbeatToAllBroker catch code block log print level from info to warn (#5940)

* [ISSUE #5924] Optimize UtilAll#sleep method (#5925)

* [ISSUE #5924]Optimize UtilAll#sleep method

* polish code

* [ISSUE #5986] optimize the BrokerOuterAPITest class code

Co-authored-by: zhouyunpeng <2474138779@qq.com>

* [ISSUE #5971] Make the internal logs related to the dledger in the controller print to a file separately (#5972)

* Make the internal logs related to the dledger in the controller print to a file separately

* Make the internal logs related to the dledger in the controller print to a file separately

* [ISSUE #5969] Remvoe duplicate deleteUnusedStats in admin processor (#5973)

* [ISSUE #5847] Add checkBlock for hasMsgFromQueue

* [ISSUE #5983] Make consumer support flow control code better (#5984)

* When encountering the flow control code, pull it after 20ms instead of 3s

* When encountering the flow control code, pull it after 20ms instead of 3s

* [ISSUE #5896] feat:add pop consumer example (#5991)

* feat:add pop consumer

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* Update example/src/main/java/org/apache/rocketmq/example/simple/PushConsumer.java

Co-authored-by: Oliver <wqdyxnbd@163.com>

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

* feat:fix

---------

Co-authored-by: mahaitao617 <mahaitao617@mahaitao617deMacBook-Pro.local>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* [ISSUE #5942] Fix the produce count include the quantity of the system topic(#5943)

* [ISSUE #5999] Fix the TopicQueueMappingUtils comments typo (#6000)

* [ISSUE #5996] Optimize the RemotingSerializable class code (#5998)

* simplified RemotingSerializable null check

* optimize the RemotingSerializable class code

* [ISSUE #5994] [RIP-46] add pop and timer metrics (#5995)

* add pop and timer metrics
* fix according to review comment

* test(broker): add ReplicasManagerRegisterTest to test the register process

1. add ReplicasManagerRegisterTest to test the register process

* chore(pom): modify pom.xml to replace mockito with powermock

1. modify pom.xml to replace mockito with powermock

* build(bazel): export powermock in bazel

1. export powermock in bazel

---------

Co-authored-by: Xinda <xdshent@gmail.com>
Co-authored-by: SSpirits <admin@lv5.moe>
Co-authored-by: loboxu <loboxu@tencent.com>
Co-authored-by: pingww <pingw002@gmail.com>
Co-authored-by: Aaron Ai <yangkun.ayk@gmail.com>
Co-authored-by: Slideee <yechun@corp.netease.com>
Co-authored-by: mxsm <ljbmxsm@gmail.com>
Co-authored-by: hardyfish <85128645+hardyfish@users.noreply.github.com>
Co-authored-by: zhouyunpeng <2474138779@qq.com>
Co-authored-by: rongtong <jinrongtong5@163.com>
Co-authored-by: zhiliatom <87265072+zhiliatom@users.noreply.github.com>
Co-authored-by: zhouxiang <zhouxiang.zzx@alibaba-inc.com>
Co-authored-by: mahaitao <15828010639@163.com>
Co-authored-by: mahaitao617 <mahaitao617@mahaitao617deMacBook-Pro.local>
Co-authored-by: Oliver <wqdyxnbd@163.com>
2023-03-14 09:40:15 +08:00
TheR1sing3un c9bf9d7899 feat(broker): perfect logic test in broker
1. perfect logic test in broker
2023-03-14 09:40:15 +08:00
TheR1sing3un cfdfa0ea51 feat(broker): implement the general register to controller protocol
1. implement the general register to controller protocol
2023-03-14 09:40:15 +08:00
TheR1sing3un f750dec0c7 refactor code in module: store/ha for persistence broker-id
1. refactor code in module: store/ha for persistence broker-id
2023-03-14 09:40:15 +08:00
hzh0425 2cc899fc35 [ISSUE #5663] Fix Messages may be lost when SyncStateSet expand in extreme scenarios (#5798)
* Add a new state 'isSynchronizingSyncStateSet' to Solve the problem of missing messages

* pass checkstyle

* Using readWriteLock to replace synchronized in AutoSwitchHAService

* Fix lock issue

* Remove unnecessary 'remoteSyncStateSet.clear'

* optimize import
2023-03-13 14:22:29 +08:00
rongtong 20dc5c96ea [ISSUE #6316] Nameserver should choose a master with a larger epoch when there are two masters in controller mode (#6317) 2023-03-13 11:16:35 +08:00
fujian-zfj 7b23042eb2 [ISSUE #6306] Fix unexpected state from slave (#6307)
* typo int readme[ecosystem]

* fix unexpected state from slave
2023-03-10 13:39:28 +08:00
rongtong ba4dd5a21e [ISSUE #6283] Fix the bug that single replica cannot flush data when transientStorePoolEnable is true
[ISSUE #6283] Fix the bug that single replica cannot flush data when transientStorePoolEnable is true
2023-03-09 11:43:38 +08:00
rongtong 9e0fb1b029 [ISSUE #6169] Fix NPE when timerWheel disabled (#6184)
* Fix some NPE when timerWheel disabled

* Add UT
2023-03-08 17:23:06 +08:00
cc8823cf62 [ISSUE #5883] DLedger commit log should override the getData (#5879)
* fix: dledger commit log should override the getData

Change-Id: I665c5b9d7e96c5b7ccd035b32272be7223d7f454

* fix: adjust timer test case for dledger

* test: remove some test case to speed up unit test

Change-Id: I5aa348ab17977ce1f0e080f7801b28797ef82af9

* test: remove other timer dledger test

* Revert TimerMessageStoreTest

---------

Co-authored-by: liwen.2022 <liwen.2022@bytedance.com>
Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
2023-03-06 09:20:41 +08:00
TheR1sing3un 18fcce664c [ISSUE #6250] Fix wrong value of MessageStoreConfig#storePathEpochFile
1. fix wrong value of MessageStoreConfig#storePathEpochFile
2023-03-05 21:57:24 +08:00
hardyfish ce99a7fe8a [ISSUE #6246] Optimize AcceptSocketService#run 2023-03-05 11:21:44 +08:00
rongtong d5d5a637f6 [ISSUE #6226] Shutdown flowMonitor when connection disconnect (#6227)
* Shutdown flowMonitor when connection disconnect

* Whether the master is existed has more accurate judgment

* Shutdown flowMonitor when HAClient service end

* Pass the check style

* Modify according to comments
2023-03-03 22:28:15 +08:00
TheR1sing3un 487473eb74 [ISSUE #6121] Optimize some code style in store module (#6122)
* style(store): optimize some typos in store module

1. optimize some typos in store module

* style(store): rename DefaultHAClient#REPORT_HEADER to DefaultHAClient#REPORT_HEADER_SIZE

1. rename DefaultHAClient#REPORT_HEADER to
DefaultHAClient#REPORT_HEADER_SIZE

* style(store): optimize comment about DefaultHAClient#REPORT_HEADER_SIZE and DefaultHAConnection#TRANSFER_HEADER_SIZE

1. optimize comment about DefaultHAClient#REPORT_HEADER_SIZE and
DefaultHAConnection#TRANSFER_HEADER_SIZE
2023-02-27 11:40:20 +08:00
zhouxiang 0f878b5157 [maven-release-plugin] prepare for next development iteration 2023-02-16 12:35:58 +08:00
zhouxiang b8e9334d2a [maven-release-plugin] prepare release rocketmq-all-5.1.0 2023-02-16 12:35:58 +08:00
zhouxiang 83d6778747 [ISSUE #6030] Update version to 5.1.0-SNAPSHOT 2023-02-16 12:35:58 +08:00
nowinkey 7d58f68819 Modify abbreviated variables and add comments 2023-02-14 13:50:47 +08:00
nowinkey 07e65d37d0 Put batchDispatchRequestThreadPoolNums config to MessageStoreConfig.java 2023-02-14 13:50:47 +08:00
nowinkey ee6cddfdb2 Combine the process of decoding byteBuffer into preCheckMessageAndReturnSize method 2023-02-14 13:50:47 +08:00
nowinkey c2ea0922b7 Remove enableBuildConsumeQueueConcurrently attribute from BrokerConfig class 2023-02-14 13:50:47 +08:00
nowinkey dcf84e4125 Add integration test and add enableBuildConsumeQueueConcurrently attribute to BrokerConfig class 2023-02-14 13:50:47 +08:00