opensource/rocketmq
3
0
Fork 0
mirror of https://github.com/apache/rocketmq.git synced 2026-08-28 20:09:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,189 Commits 72 Branches 40 Tags
develop
Commit Graph

829 Commits

Author SHA1 Message Date
fujian-zfj 46962c262c [ISSUE #7523] Message will flush timeout when transientStorePoolEnable=true and flushDiskType=SYNC_FLUSH (#7524) 
* typo int readme[ecosystem]

* enableTransientPool and sync_flush will cause flush_time_out

* polish

* add log
 2023-11-02 13:47:16 +08:00
guyinyou 91349f30b9 [ISSUE #7437] Add the CRC check of commitlog (#7468) 
* Added CRC32 check for full data

* add unit test

* add MessageExtBrokerInnerTest

* fix codestyle

* fix codestyle

---------

Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
 2023-10-25 14:54:00 +08:00
fujian-zfj f565654351 [ISSUE #7064] [RIP-66-2] Support KV(RocksDB) Storage for ConsumeQueue (#7120) 
* typo int readme[ecosystem]

* consumequue support rocksdb

* fix rocksdb test

* fix rocksdb test

* remove unused method

* split into two tables

* CqUnit in Rocksdb [phyOffset, bodySize, tagHashCode, msgStoreTime]

* fix build.baze in tieredMessageStore

* skip RocksDBMessageStoreTest bazel

* skip RocksDBMessageStoreTest bazel

* Rocksdb TimerMessageStore

* fix unit test bazel

* fix store build.bazel

* fix store build.bazel

* optimize

* optimize

* polish

* build bytebuffer pair inner

* remove unused code

* DataConverter.CHARSET_UTF8

* fix comment

* fix comment

* rebuild test

* rebuild test

* optimize

* rebuild test

* polish

* polish

* rebuild test

* merge develop

* rebuild test

* rebuild test

* fix getConsumeQueue not find cq

* fix test

* rocksdb new version

* rebuild test

* fix bug

* Resolve conflicts after merging develop

* fix updateCqOffset

* fix recoverAbnormally

* fix recoverAbnormally

* polish

* polish

* polish

* remove exception in cleanunusedTopic

* remove exception in cleanunusedTopic

* remove exception in cleanunusedTopic

---------

Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
 2023-10-14 11:04:04 +08:00
fujian-zfj 7e4879a3bc [ISSUE #7441] Fix log "Init the confirmOffset" keep printing error in controller mode (#7442) 
* typo int readme[ecosystem]

* fix keep printing log problem
 2023-10-11 14:45:07 +08:00
guyinyou d7e5c4d1a4 [ISSUE #7389] Fix the problem that getLastMappedFile function affects performance 
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
 2023-09-25 13:37:36 +08:00
Ji Juntao 88a9d939ce [ISSUE #7381] Fix the problem of inaccurate timer message metric (#7382) 
* correct the timerMetrics' result.

* for further extension.

* checkstyle.

* use toLong.
 2023-09-23 17:38:27 +08:00
lk 73b3fde837 [maven-release-plugin] prepare for next development iteration (#7379) 2023-09-22 10:08:59 +08:00
lk 1a681bdf9b [maven-release-plugin] prepare release rocketmq-all-5.1.4 (#7377) 2023-09-21 19:58:29 +08:00
rongtong a9e353285c Add the configuration of topicQueueLock number to better support different scenarios (#7317) 2023-09-11 15:55:18 +08:00
guyinyou c100d815d7 [ISSUE #7328] Convergent thread pool creation (#7329) 
* Convergence thread pool creation to facilitate subsequent iteration management

* Convergence thread pool creation in ThreadPoolMonitor.java

* fix unit test

* Convergence ThreadPool constructor

* Convergence ScheduledThreadPool constructor

* remove unused import

* Convergence ScheduledThreadPool constructor

* remove unused import

---------
 2023-09-11 10:13:56 +08:00
yuz10 d000ef947d [ISSUE #7283] Incorrect dledger commitlog min offset after mappedFile re delete failed (#7284) 2023-08-31 09:41:33 +08:00
mxsm b4f73e2aab [ISSUE #7271] Optimize the configuration for setting the quantity of TimerDequeuePutMessageService (#7272) 2023-08-27 20:58:58 +08:00
echooymxq 017ad11047 [ISSUE #7219] Fix Concurrent modify syncStateSet and Mark synchronizing frequently when shrink. (#7220) 2023-08-23 11:11:42 +08:00
guyinyou 69c26d3d29 [ISSUE #7228] Converge the use of some important variables for some class 2023-08-23 10:27:52 +08:00
guyinyou 3df1b9232a [ISSUE #7201] Remove the DefaultMessageStore.class dependency in TransientStorePool 
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
 2023-08-17 13:59:04 +08:00
Jixiang Jin 50d1050437 To config the cardinalityLimit for openTelemetry metrics exporting and fix logging config for metrics (#7196) 2023-08-16 21:15:00 +08:00
fujian-zfj 3a6ef0400c [ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata (#7092) 
* typo int readme[ecosystem]

* rocksdb metadata

* add unit test

* fix testOffsetPersistInMemory

* fix unit test

* fix unit test

* remove unused import

* move RocksDBOffsetSerialize to broker moudle

* Fix bazel build scripts

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* Flag QueryMsgByKeyIT as flaky as it fails at frequency: 5 out of 32

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* change public to private of some inner method

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
 2023-08-05 00:32:11 +08:00
koado 1fe5d62334 [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp (#7082) 
* add new interface for searching offset with boundary type

* format code

* fix failed test

* unify two BoundaryType class

* add interface getOffsetInQueueByTime(long timestamp, BoundaryType boundaryTYpe) in ConsumeQueueInterface

* fix AdminBrokerProcessorTest unnecessary Mockito stubbings
 2023-08-03 11:40:16 +08:00
Ji Juntao 1f0f3b2d6d [ISSUE #7013] Polish ColdDataCheckService's logic (#7014) 
* polish coldCtrl

* remove the catch.
 2023-07-11 21:13:06 +08:00
mxsm 4f840afcb0 [ISSUE #6960] Added Slot formatting sketch comments (#6961) 2023-07-05 11:03:52 +08:00
yuz10 f3ce3e8fb9 [ISSUE #6940] change dataReadAheadEnable default to false (#6944) 
* [ISSUE #6390] Add break to the exception of WHEEL_TIMER_NOT_ENABLE.

* fix broker start fail if mapped file size is 0

* log

* only delete the last empty file

* change dataReadAheadEnable default to true
 2023-06-26 17:10:26 +08:00
rongtong e369d7deac [maven-release-plugin] prepare for next development iteration (#6939) 2023-06-24 14:31:42 +08:00
rongtong f7425f52f9 [maven-release-plugin] prepare release rocketmq-all-5.1.3 (#6938) 2023-06-24 12:23:11 +08:00
yuz10 4e09a9cfe9 [ISSUE #6904] Fix fail to start broker if mapped file size is 0 (#6903) 2023-06-22 23:43:59 +08:00
rongtong ad4cc957fb [maven-release-plugin] prepare for next development iteration (#6895) 2023-06-12 19:12:25 +08:00
rongtong 749967baa8 [maven-release-plugin] prepare release rocketmq-all-5.1.2 (#6892) 2023-06-12 17:44:39 +08:00
Vincent Lee 91d8ee16a0 [ISSUE #6800] Change the config variable isEnableBatchPush to enableBatchPush (#6801) 
* fix: wrong setter for config isEnableBatchPush

Change-Id: I1ebf4748c26c42efb058c11a8b5084b69213892c

* fix: change isEnableBatchPush to enableBatchPush

Change-Id: I13db49a30841fe5ff9a548f02afa0cc272ba027d
 2023-06-12 13:59:11 +08:00
lizhimins 6eac107005 [ISSUE #6633] [RIP-65] Improving Tiered Storage Implementation (#6781) 
[ISSUE #6633] [RIP-65] Improving Tiered Storage Implementation
 2023-06-09 21:07:53 +08:00
Drizzle bee5077bcb [ISSUE #6336] [RIP-62] Cold Read Control (#6507) 2023-06-05 17:09:31 +08:00
cnScarb 1d62b74ab6 [ISSUE #6779] Support scheduled message for LMQ (#6780) 2023-06-02 10:36:09 +08:00
Ji Juntao acd8b099df [ISSUE #6849] Fix the issue of increasing RT in three replicators 2023-06-02 09:57:46 +08:00
Dongyuan Pan 985319ba2a [ISSUE #6797]Support batch ack when reput buffer ak to store in PopBufferMergeService (#6798) 
* add back for PopReviveService

* add batch ack for PopReviveService
 2023-05-24 11:22:18 +08:00
Humkum 5f747f5c24 [ISSUE #6747] Add INNER_MULTI_DISPATCH check, can not contains path separator 2023-05-16 08:26:17 +08:00
rongtong 8aea26ef8f [maven-release-plugin] prepare for next development iteration 2023-05-15 19:48:34 +08:00
rongtong e244ebb99a [maven-release-plugin] prepare release rocketmq-all-5.1.1 (#6763) 2023-05-15 18:51:01 +08:00
Ji Juntao 5dc2e20efc [ISSUE #6728] Compute the confirmOffset without considering new connections (#6729) 
* 1. When compute the confirmOffset, dismiss the ackOffset of new connections. 2. When compute the confirmOffset, use getConfirmOffsetDirectly() to avoid the endless calling.

* use the calculated slaveAckOffset

* optimize the logic.
 2023-05-10 20:30:16 -07:00
rongtong 41e9b8cf06 [ISSUE #6703] Fix the incorrect of confirmOffset when recovering abnormally caused by message loss due to asynchronous flushing after restarting 2023-05-06 17:12:47 +08:00
Ji Juntao e73a9aa8a7 [ISSUE #6706] brokerPermission cannot stop messages flow in unwritable brokers 2023-05-06 16:49:55 +08:00
Ji Juntao 709537595c [ISSUE #6665] Optimize the process of truncateInvalidMsgs() (#6666) 
* 1. Fix the bug that confirmOffset may be larger than ReputOffset in some cases. 2. Optimize the logic when truncate dirty files.

* Add more logs while expanding SyncStateSet.

* reset the confirmOffset to the truncated offset.

* optimize the log.

* move setConfirmOffset() process into commitLog.

* add the judge.
 2023-05-04 11:55:04 +08:00
rongtong b410557e87 [ISSUE #6609] Fix the issue that consume queue building exceeds confirmOffset when node restarts to recover (#6618) 
* Fix the issue that consume queue building exceeds confirmOffset when node restarts to recover
 2023-05-04 09:39:51 +08:00
Ji Juntao 7903c70412 [ISSUE #6662] Optimize the process of HA's confirmOffset calculation (#6663) 
* When compute confirmOffset, judge whether the slaves in syncStateSet all connect to the master.

* fix the brokerId to brokerControllerId.

* optimize the code: 1. Set brokerId when the replicasManager inited. 2. Rename the confusing name.

* set haService's brokerControllerId after registered.

* set the haService's brokerControllerId when the ReplicasManager start without brokerIdentity file.
 2023-05-01 14:35:52 +08:00
rongtong e3b8178871 [ISSUE #6570] Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal (#6641) 
* Fix the issue that expectLogicOffset is greater than currentLogicOffset in consumeQueue build when the message is illegal

* Add new UT

* Fix bug that UT can not pass

* Polish the variable name

* Polish the comment

* Add more comments
 2023-04-26 11:24:47 +08:00
rongtong 14ad16b0d5 [ISSUE #6634] Polish the HA logs to better troubleshoot issues (#6635) 2023-04-24 10:11:01 +08:00
rongtong 073da6b971 Fix the issue broker startup failed when version upgrade from 5.1.0 to latest develop because compactionThreadNum is 0 (#6611) 2023-04-19 09:54:52 +08:00
Ao Qiao 3fe81bfbac [ISSUE #6545] Remove getTopicConfigs method in interface MessageStore (#6531) 
* change map -> lambda

* f

* fix unit test

* remove getTopicConfig function

* Update MultiDispatchTest.java

* Update CompactionStore.java

* update

* update test

* update test

* Update BatchConsumeMessageTest.java

* Update BrokerController.java

* Update BrokerController.java

* check

* Update BrokerController.java

* Update BatchConsumeMessageTest.java
 2023-04-15 18:22:06 +08:00
Abhijeet Mishra d1b14b0ff9 [ISSUE #6525]Make ConsumeQueueInterface extends from FileQueueLifeCycle (#6534) 
* [#6525] Make ConsumeQueueInterface extends from FileQueueLifeCycle

* [apache#6525] forced typecast is removed

* [#6525] remove all the forced convert
 2023-04-15 18:21:03 +08:00
guyinyou d3a8fe0c5b [ISSUE #6587] fix getMax() in SparseConsumeQueue 
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
 2023-04-14 11:26:11 +08:00
mxsm d9a7315ffe [ISSUE #6584] Fix AppendMessageCallback comments typo (#6585) 2023-04-13 10:37:26 +08:00
hardyfish 9574165fe3 [ISSUE #6299] Remove TimerFlushService#run duplicate code (#6300) 2023-04-06 10:19:04 +08:00
guyinyou 72befa0fb4 1. change the compactionTopic dispatch process from asynchronous to synchronous (#6530) 
2. compactionTopic dispatch performance improvement

Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
 2023-04-04 10:39:04 +08:00