1a15729ca9
[ISSUE #8601 ]When isPopShouldStop hit,unlock queueLockManager ( #8602 )
...
* fix:when isPopShouldStop hit， unlock queueLockManager
* fix:when isPopShouldStop hit， unlock queueLockManager
* fix: limit rate of appending commit in case of DLedger commit-log
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com >
---------
Signed-off-by: Zhanhui Li <lizhanhui@gmail.com >
Co-authored-by: Zhanhui Li <lizhanhui@gmail.com >
2024-08-30 13:43:01 +08:00
2a938fb1bc
[ISSUE #8058 ] Support for upgrading metadata in json to rocksdb ( #8571 )
2024-08-23 11:18:03 +08:00
09beca60eb
[ISSUE #8549 ] Ipv6 enabled in broker, pickupStoreTimestamp size should be 20 ( #8567 )
...
* fix: when ipv6 enabled in broker, pickupStoreTimestamp size should be 12(20-8)
* fix: when ipv6 enabled in broker, pickupStoreTimestamp size should be 20
2024-08-21 13:44:26 +08:00
aab646ce83
[ISSUE #8460 ] Improve the pop revive process when reading biz messages from a remote broker ( #8475 )
...
* [ISSUE #8460 ] part1: add extra information to the call chain of remote message reading
* [ISSUE #8460 ] part2: add exponential backoff and ending condition of CK rewrite, and fix checkstyle
* [ISSUE #8460 ] exclude test, BrokerOuterAPITest passed locally, but failed on bazel.
2024-08-02 14:43:21 +08:00
3696be0632
[ISSUE #8463 ] Some statistical items should also be deleted to prevent memory leakage when a topic or group is deleted ( #8464 )
...
* Some important statistical items should also be deleted to prevent memory leakage when a topic or group is deleted
* Add UTs
2024-08-01 10:04:11 +08:00
263f0fbdd6
[ISSUE #8448 ] commitlog class comment optimize
2024-07-29 13:46:46 +08:00
86d59d2485
Add the ability to write ConsumeQueue using fileChannel to prevent JVM crashes in some situations ( #8403 )
2024-07-22 17:20:11 +08:00
259efdb086
[ISSUE #8350 ] Properties store error: crc32ReservedLength make undefine memory in properties when no enable AppendPropCRC ( #8351 )
2024-07-16 17:24:14 +08:00
3aa5d1936f
Adjust the default value of ackMessageThreadPoolNums to 16 to prevent performance bottlenecks during high traffic. ( #8337 )
2024-07-02 17:08:54 +08:00
d60198f621
[ISSUE #8239 ] Fix the issue of potential message loss after a crash under synchronous disk flushing configuration. ( #8240 )
2024-06-07 13:44:13 +08:00
144b22ba7d
[ISSUE #8235 ]Add @Override annotation for handleDiskFlush method ( #8236 )
2024-06-02 07:13:15 +08:00
152055632d
[ISSUE #8222 ] Fix spelling errors in comments ( #8224 )
2024-05-30 10:10:03 +08:00
c00fac3c77
[ISSUE #8148 ] Fix variable typo
2024-05-16 09:56:09 +08:00
b37d283793
[ISSUE #8044 ]Add Override annotation for AllocateMappedFileService#run ( #8045 )
2024-04-23 23:33:55 +08:00
9d07cc7c51
[ISSUE #8032 ] Set checkDupInfo value from config
2024-04-18 14:20:16 +08:00
bf24ffde88
[ISSUE #7979 ] Fix timerWheel message metric ( #7980 )
...
* fix metric in TimerWheel
* fix metric in TimerWheel
* fix message metric in TimerWheel
* fix message metric in TimerWheel
2024-04-01 10:47:35 +08:00
fb6f9e44f4
[ISSUE #7961 ] use BoundaryType in binarySearchInCQByTime
...
Co-authored-by: 凯铎 <zhengkaifeng.zkf@alibaba-inc.com >
2024-03-28 16:59:20 +08:00
27d8375940
[ISSUE #7836 ] flush_behind_bytes wrong in transientStorePoolEnable not enable
2024-03-24 18:58:19 +08:00
3c27bfe5ed
[ISSUE #7902 ] fix:reput thread may quit by throwing error
2024-03-14 16:03:51 +08:00
3877dcd134
[ISSUE #7853 ] Fix the manualDeleteFileSeveralTimes count
2024-02-29 10:48:31 +08:00
7b64f50137
[ISSUE #7446 ] Add config of maxFilterMessageSize ( #7447 )
...
* bugfix broker boot succes but get fail ip addr
bug:
broker ip addr(IPV4) get fail after broker start up;
fix:
add compare ,continue when ip is start with '0.' ;
like :
The broker[broker-a, 0.0.1.1:10911] boot success. serializeType=JSON and name server is 127.0.0.1:9876;
* add config maxFilterMessageSize , not hard code
* Pass the code check style
---------
Co-authored-by: mingzhi.zhang <mingzhi.zhang@jiduauto.com >
Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn >
2024-01-31 03:34:27 +08:00
6d7513425c
[RIP-46] Enhanced metrics for timing and transactional messages ( #7500 )
...
* add request codes' distribution and timing messages' distribution
* remove the requestCode distribution.
* add delay message latency distribution.
* add transaction metrics
* transaction metric of topics finished, v1.
* add the transaction metrics, to be tested.
* fix the judgement of putMessageResult
* optimize.
* add config.
* fix test case.
* add unit tests for transactionMetrics.
* remove chinese character
* add rocksdb metrics.
* add more rocksdb metrics.
* fix NPE
* avoid the total time is 0.
* add license
* remove useless import.
2024-01-22 16:56:05 +08:00
920dc32d27
[ISSUE #7760 ] make timerSkipUnknownError can be set by config file
2024-01-19 09:59:23 +08:00
646e2a4942
[ISSUE #7355 ] fix dledger recover abnormally may lost consume queue of tail ( #7599 )
...
* fix dledger recover abnormally may lost consume queue of tail
* fix correct storeTimestampPosition when bornhost is v6
* fix correct SYSFLAG offset
---------
Co-authored-by: 赵宇晗 <zhaoyuhan@xiaomi.com >
2024-01-18 15:43:01 +08:00
b1d8d306a5
[ISSUE #7699 ] Add namespace v2 in client ( #7700 )
...
* Add namespace v2
* Add NamespaceRpcHook
* Refector extends header
* Use Boolean in request header to remove unnecessary encode
* Add unit test
* Add NamespaceRpcHookTest
* Remove GrpcConverter#wrapResourceWithNamespace
* Optimize readability of RpcRequestHeader
2024-01-12 10:46:15 +08:00
2c898c9b31
[ISSUE #7689 ] In Controller mode, messages may lost due to sharing the same cq offset ( #7690 )
...
* fix the reput bug.
* add more logs.
* refactor the method of compensating for HA.
* not modify the imports.
* refactor the log.
* refactor the log.
2023-12-21 14:41:51 +08:00
f0a3e933b9
[ISSUE #7684 ] Fix iterator.remove() bug ( #7682 )
...
* bugfix: CopyOnWriteArray#listIterator do not support remove action when iterating
* add testcase
---------
Co-authored-by: mipengcheng3 <mipengcheng3@jd.com >
2023-12-20 14:34:53 +08:00
9dec4cf5fe
[ISSUE #7669 ] map variable delayLevelTable changed to ConcurrentSkipListMap from ConcurrentHashMap ( #7675 )
2023-12-19 19:47:13 +08:00
80c0330f75
ConfirmOffset directly takes the max offset when allAckInSyncStateSet is false ( #7657 )
2023-12-18 10:10:28 +08:00
1e69e3904f
Fix flaky test testSemiSyncReplicaWhenAdaptiveDegradation ( #7631 )
2023-12-12 14:17:24 +08:00
bcc9db5cba
[ISSUE #7614 ] Fix flaky test RocksDBMessageStoreTest ( #7625 )
...
* fix #7614 Flaky test RocksDBMessageStoreTest
Signed-off-by: lizhanhui <lizhanhui@gmail.com >
* fix: give TimerMessageStoreTest#testStateAndRecover more time for Awaitability to poll and check
Signed-off-by: lizhanhui <lizhanhui@gmail.com >
* clean up exclude test list of bazel
Signed-off-by: lizhanhui <lizhanhui@gmail.com >
---------
Signed-off-by: lizhanhui <lizhanhui@gmail.com >
2023-12-11 17:37:01 +08:00
faae64715d
[ISSUE #7601 ] Fix slave acting master bug ( #7603 )
...
* fix NullPointerException when message escape to remote
* fix NumberFormatException when message retry to escape to remote
* fix timerCheckPoint of the master is not updated, causing the timer message to be replayed after master is restarted
* Use properties copies instead of referencing the same map when converting message
2023-12-07 11:25:22 +08:00
56e886bf70
[ISSUE #7592 ] testCleanBuffer unit test modifies, changed non-direct … ( #7593 )
...
* [ISSUE #7592 ] testCleanBuffer unit test modifies, changed non-direct to direct buffer allocation
* fix: consolidate UtilAll#cleanBuffer by checking if the given buffer is direct or not
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com >
---------
Signed-off-by: Li Zhanhui <lizhanhui@gmail.com >
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com >
2023-11-29 12:32:38 +08:00
e955e4399c
[ISSUE #7577 ] SlaveActingMaster Timer Message retry without escape logic ( #7578 )
...
Co-authored-by: ayue <ericyu0421@163.com >
2023-11-24 10:47:08 +08:00
8e7e2b5f50
[ISSUE #7574 ] Fix RunningFlags conflict
...
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com >
2023-11-20 19:32:57 +08:00
ead3d90501
[ISSUE #7511 ] Lock granularity issue causing LMQ message loss ( #7525 )
...
* bug fix: assignOffset and increaseOffset in LMQ has concurrency issues in topicQueueLock, should be in putMessageLock
* fix MultiDispatchTest
* fix MultiDispatchTest
* fix unit test
2023-11-08 10:40:52 +08:00
46962c262c
[ISSUE #7523 ] Message will flush timeout when transientStorePoolEnable=true and flushDiskType=SYNC_FLUSH ( #7524 )
...
* typo int readme[ecosystem]
* enableTransientPool and sync_flush will cause flush_time_out
* polish
* add log
2023-11-02 13:47:16 +08:00
91349f30b9
[ISSUE #7437 ] Add the CRC check of commitlog ( #7468 )
...
* Added CRC32 check for full data
* add unit test
* add MessageExtBrokerInnerTest
* fix codestyle
* fix codestyle
---------
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com >
2023-10-25 14:54:00 +08:00
f565654351
[ISSUE #7064 ] [RIP-66-2] Support KV(RocksDB) Storage for ConsumeQueue ( #7120 )
...
* typo int readme[ecosystem]
* consumequue support rocksdb
* fix rocksdb test
* fix rocksdb test
* remove unused method
* split into two tables
* CqUnit in Rocksdb [phyOffset, bodySize, tagHashCode, msgStoreTime]
* fix build.baze in tieredMessageStore
* skip RocksDBMessageStoreTest bazel
* skip RocksDBMessageStoreTest bazel
* Rocksdb TimerMessageStore
* fix unit test bazel
* fix store build.bazel
* fix store build.bazel
* optimize
* optimize
* polish
* build bytebuffer pair inner
* remove unused code
* DataConverter.CHARSET_UTF8
* fix comment
* fix comment
* rebuild test
* rebuild test
* optimize
* rebuild test
* polish
* polish
* rebuild test
* merge develop
* rebuild test
* rebuild test
* fix getConsumeQueue not find cq
* fix test
* rocksdb new version
* rebuild test
* fix bug
* Resolve conflicts after merging develop
* fix updateCqOffset
* fix recoverAbnormally
* fix recoverAbnormally
* polish
* polish
* polish
* remove exception in cleanunusedTopic
* remove exception in cleanunusedTopic
* remove exception in cleanunusedTopic
---------
Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn >
2023-10-14 11:04:04 +08:00
7e4879a3bc
[ISSUE #7441 ] Fix log "Init the confirmOffset" keep printing error in controller mode ( #7442 )
...
* typo int readme[ecosystem]
* fix keep printing log problem
2023-10-11 14:45:07 +08:00
d7e5c4d1a4
[ISSUE #7389 ] Fix the problem that getLastMappedFile function affects performance
...
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com >
2023-09-25 13:37:36 +08:00
88a9d939ce
[ISSUE #7381 ] Fix the problem of inaccurate timer message metric ( #7382 )
...
* correct the timerMetrics' result.
* for further extension.
* checkstyle.
* use toLong.
2023-09-23 17:38:27 +08:00
a9e353285c
Add the configuration of topicQueueLock number to better support different scenarios ( #7317 )
2023-09-11 15:55:18 +08:00
c100d815d7
[ISSUE #7328 ] Convergent thread pool creation ( #7329 )
...
* Convergence thread pool creation to facilitate subsequent iteration management
* Convergence thread pool creation in ThreadPoolMonitor.java
* fix unit test
* Convergence ThreadPool constructor
* Convergence ScheduledThreadPool constructor
* remove unused import
* Convergence ScheduledThreadPool constructor
* remove unused import
---------
2023-09-11 10:13:56 +08:00
d000ef947d
[ISSUE #7283 ] Incorrect dledger commitlog min offset after mappedFile re delete failed ( #7284 )
2023-08-31 09:41:33 +08:00
b4f73e2aab
[ISSUE #7271 ] Optimize the configuration for setting the quantity of TimerDequeuePutMessageService ( #7272 )
2023-08-27 20:58:58 +08:00
017ad11047
[ISSUE #7219 ] Fix Concurrent modify syncStateSet and Mark synchronizing frequently when shrink. ( #7220 )
2023-08-23 11:11:42 +08:00
69c26d3d29
[ISSUE #7228 ] Converge the use of some important variables for some class
2023-08-23 10:27:52 +08:00
3df1b9232a
[ISSUE #7201 ] Remove the DefaultMessageStore.class dependency in TransientStorePool
...
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com >
2023-08-17 13:59:04 +08:00
50d1050437
To config the cardinalityLimit for openTelemetry metrics exporting and fix logging config for metrics ( #7196 )
2023-08-16 21:15:00 +08:00
Lei Zhiyuan