444 Commits

Author SHA1 Message Date
yulangz 8df53dfde8 [ISSUE #7300] jRaft-Controller Implemention (#7301)
* jRaft-Controller Implemention

* fix bazel build

* reformat code

* remove fury dependence

* fix bazel build

* resolve conflict

* clear code

* Optimize code style

* Optimize code style

* Optimize code style

* revert style only change
fix code style
rollback an unexpected modification.

* Update Producer.java

* chore: move raft startup to start method

* chore: use jraftconfig to collect all configs about jraft

* fix: fix wrong store path because init use string constant

* fix: fix CONTROLLER_NOT_LEADER error in follower

* chore: seperate jraft and controller log

* chore: fix conflict with develop

* chore: add comment to clear the filter logic

* feat: triggerElectMaster will retry when failed

* feat: when controller all restart, we use a timestamp to trace the first heartbeat, avoid to elect again

* fix: implements Serializable to enable snapshot serialize

* fix: use for loop to simple the elect retry

* chore: update jraft version

* chore: opt import

---------

Co-authored-by: leizhiyuan <leizhiyuan@gmail.com>
2024-01-29 20:45:20 +08:00
dingshuangxi888 92e058c823 [ISSUE #7774] Make the handle of ppv2 tlv more extendable (#7775)
* Fix ascii validate for ppv2 tls.
* make the handle of ppv2 tlv extendable.
2024-01-24 09:50:15 +08:00
Zhouxiang Zhan 5262358140 [ISSUE #7699] Refector NamespaceRpcHook (#7769)
* [ISSUE #7699] Refector NamespaceRpcHook

* fix
2024-01-23 13:56:26 +08:00
Ji Juntao 6d7513425c [RIP-46] Enhanced metrics for timing and transactional messages (#7500)
* add request codes' distribution and timing messages' distribution

* remove the requestCode distribution.

* add delay message latency distribution.

* add transaction metrics

* transaction metric of topics finished, v1.

* add the transaction metrics, to be tested.

* fix the judgement of putMessageResult

* optimize.

* add config.

* fix test case.

* add unit tests for transactionMetrics.

* remove chinese character

* add rocksdb metrics.

* add more rocksdb metrics.

* fix NPE

* avoid the total time is 0.

* add license

* remove useless import.
2024-01-22 16:56:05 +08:00
道君 7a36d4d736 [ISSUE #7757] Use CompositeByteBuf to prevent memory copy. (#7694)
* Use CompositeByteBuf to prevent mem_copy.

* Fix code

* Add tests

* Remove useless UTs

* Remove unused imports.

---------

Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
2024-01-17 09:49:23 +08:00
dingshuangxi888 89fdab4d58 [ISSUE #7752] Fix ppv2 tls ascii start with (byte)2 (#7753)
* Fix ascii validate for ppv2 tls.

* fix ppv2 tls ascii check

---------

Co-authored-by: ShuangxiDing <shuangxi.dsx@alibaba-inc.com>
2024-01-15 15:18:45 +08:00
Zhouxiang Zhan b1d8d306a5 [ISSUE #7699] Add namespace v2 in client (#7700)
* Add namespace v2

* Add NamespaceRpcHook

* Refector extends header

* Use Boolean in request header to remove unnecessary encode

* Add unit test

* Add NamespaceRpcHookTest

* Remove GrpcConverter#wrapResourceWithNamespace

* Optimize readability of RpcRequestHeader
2024-01-12 10:46:15 +08:00
Zhouxiang Zhan 8a36471a19 [ISSUE #7543] Add enableRetryTopicV2 brokerConfig (#7734)
* Add enableRetryTopicV2
2024-01-11 11:11:53 +08:00
mxsm 403ad6f66c [ISSUE #7679] Optimize the serialization of RemotingCommand processTimer property (#7683) 2023-12-20 08:45:08 +08:00
keranbingaa 65faea22fd [ISSUE #7534] Use high performance concurrent set to replace copyonwriteset (#7583)
* fix ISSUE #7534

* reformat code

* Remove the useless unit test

---------

Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
2023-12-01 10:05:16 +08:00
Jixiang Jin 651a5ca992 [ISSUE #7562] BugFix for estimating message accumulation correctly (#7563) 2023-11-16 10:16:16 +08:00
Zhanhui Li 15d32db03b [ISSUE #7547] Let consumer be aware of message queue assignment change (#7548)
* let consumer be aware of message queue assignment change

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* add unit test for DefaultMQPushConsumer#setMessageQueueListener

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix: bazel build warnings

Signed-off-by: Zhanhui Li <lizhanhui@apache.org>

* fix: set MixCommitlogTest test size as medium

Signed-off-by: Zhanhui Li <lizhanhui@apache.org>

* allow cache bazel test results

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix code style issue by removing unused imports

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* fix #7552

Signed-off-by: Zhanhui Li <lizhanhui@apache.org>

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Signed-off-by: Zhanhui Li <lizhanhui@apache.org>
2023-11-13 09:44:25 +08:00
rongtong 38d267672a Remove getBrokerClusterAclConfig from admin and server (#7486)
* Remove getBrokerClusterAclConfig from admin and server

* Add @Deprecated annotation to GET_BROKER_CLUSTER_ACL_CONFIG request code
2023-10-24 15:15:08 +08:00
Zhouxiang Zhan 2043dd5034 [ISSUE #7493] Introduce a new event NettyEventType.ACTIVE (#7494)
* [ISSUE #7493] Introduce a new event NettyEventType.ACTIVE for ChannelEventListener

* introduce a new event NettyEventType.ACTIVE,

* implement channelActive interface for NettyRemotingClient#NettyConnectManageHandler

* add onChannelActive for ChannelEventListener interface.

* Move send heartbeat to onChannelActive
2023-10-24 10:29:43 +08:00
weihubeats 3968c186a5 [ISSUE #7231] Fix: proxy client language error (#7200)
* Adding null does not update

* add langeuga code

* add langeuga code

* add langeuga code

* add langeuga code

* add langeuga code

* Rerun ci

* Rerun ci

* Rerun ci

* remove redundant package imports

* redundant line

* modify the parameter passed as proxyContext to language

* format
2023-10-20 14:49:00 +08:00
guyinyou f0f15b5e21 [ISSUE #7454] Utilizing cache to avoid duplicate parsing (#7455)
* Utilizing cache to avoid duplicate parsing

* add a method argument to decide cacheable

* Renaming variable names from cacheAble to isCached

---------

Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
Co-authored-by: RongtongJin <jinrongtong16@mails.ucas.ac.cn>
2023-10-19 10:14:29 +08:00
Zhouxiang Zhan d73b601382 [ISSUE #7330] Fix channel connect issue for goaway (#7467)
* add waitChannelFuture for goaway

* add body for retry channel
2023-10-16 19:06:40 +08:00
LetLetMe dc62d7f2e1 [ISSUE #7451] Override toString for TopicConfigAndQueueMapping 2023-10-13 13:45:48 +08:00
Zhouxiang Zhan 4acb43ecee [ISSUE #7330] Add goaway and reconnection mechanism (#7331)
* Add shutdown wait for NettyRemotingServer

* Add goaway and reconnection mechanism

* Add client version check

* Add enableTransparentRetry for NettyClientConfig

* Add enableReconnectForGoAway for NettyClientConfig

* fix unit test

* fix client version check
2023-10-10 13:54:01 +08:00
Zhouxiang Zhan b9ffe0f957 [ISSUE #7296] Add ChannelEventListener for MQClientAPIImpl (#7324)
* Add ChannelEventListener for MQClientAPIImpl

* add heartbeat when channel connect

* remove log

* Add enableHeartbeatChannelEventListener for ClientConfig
2023-10-08 16:33:44 +08:00
Zhouxiang Zhan 84156084a4 [ISSUE #7321] Refector NettyRemotingAbstract with unify future implementation (#7322)
* Refector NettyRemotingAbstract

* Add invoke with future method

* Deprecate InvokeCallback#operationComplete

* Add operationSuccess and operationException for InvokeCallback

* fix unit test

* fix unit test

* Keep InvokeCallback#operationComplete

* Optimize invokeAsyncImpl operationComplete

* Add unit test for NettyRemotingClient

* fix checkstyle
2023-10-08 11:13:25 +08:00
lk 73b3fde837 [maven-release-plugin] prepare for next development iteration (#7379) 2023-09-22 10:08:59 +08:00
lk 1a681bdf9b [maven-release-plugin] prepare release rocketmq-all-5.1.4 (#7377) 2023-09-21 19:58:29 +08:00
guyinyou c100d815d7 [ISSUE #7328] Convergent thread pool creation (#7329)
* Convergence thread pool creation to facilitate subsequent iteration management

* Convergence thread pool creation in ThreadPoolMonitor.java

* fix unit test

* Convergence ThreadPool constructor

* Convergence ScheduledThreadPool constructor

* remove unused import

* Convergence ScheduledThreadPool constructor

* remove unused import

---------
2023-09-11 10:13:56 +08:00
PiteXChen bd0e9c09db [ISSUE #7111] Remove responseFuture from the responseTable when exception occurs (#7112)
* remove responseFuture when exception
* Empty-Commit

---------
Co-authored-by: chenyong152 <chenyong152@midea.com>
2023-08-29 19:39:27 +08:00
Ji Juntao b028277018 [ISSUE #6567] [RIP-63] Queue Selection Strategy Optimization (#6568)
Optimize the proxy's and client's selection strategy for brokers when sending messages, and use multiple selection strategies as a pipeline to filter suitable queues.
2023-08-29 16:13:38 +08:00
ShuangxiDing 9f34f55e1d [ISSUE #7226] Filter tlvs in ppv2 which contents not are spec-compliant ASCII characters and space (#7227)
Filter tlvs in ppv2 which not are spec-compliant ASCII characters and space
2023-08-29 15:48:46 +08:00
yao-wenbin 05e7cde610 [ISSUE #7042] maven-compile job failed, Because TlsTest's serverRejectsSSLClient test case will throw TooLongFrameException (#7179) 2023-08-18 09:49:59 +08:00
Jixiang Jin 50d1050437 To config the cardinalityLimit for openTelemetry metrics exporting and fix logging config for metrics (#7196) 2023-08-16 21:15:00 +08:00
redlsz 55e0cdb2af fix: IndexOutOfBoundsException when process pop response (#7003) 2023-08-15 19:19:45 +08:00
fujian-zfj 3a6ef0400c [ISSUE #7064] [RIP-66-1] Support KV(RocksDB) Storage for Metadata (#7092)
* typo int readme[ecosystem]

* rocksdb metadata

* add unit test

* fix testOffsetPersistInMemory

* fix unit test

* fix unit test

* remove unused import

* move RocksDBOffsetSerialize to broker moudle

* Fix bazel build scripts

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* Flag QueryMsgByKeyIT as flaky as it fails at frequency: 5 out of 32

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>

* change public to private of some inner method

---------

Signed-off-by: Li Zhanhui <lizhanhui@gmail.com>
Co-authored-by: Li Zhanhui <lizhanhui@gmail.com>
2023-08-05 00:32:11 +08:00
koado 1fe5d62334 [ISSUE #7074] Allow a BoundaryType to be specified when retrieving offset based on the timestamp (#7082)
* add new interface for searching offset with boundary type

* format code

* fix failed test

* unify two BoundaryType class

* add interface getOffsetInQueueByTime(long timestamp, BoundaryType boundaryTYpe) in ConsumeQueueInterface

* fix AdminBrokerProcessorTest unnecessary Mockito stubbings
2023-08-03 11:40:16 +08:00
Vincent Lee d797377880 [ISSUE #7056] Avoid close success channel if invokeSync most time cost on get connection for channel (#7057)
* fix: avoid close success channel if invokeSync most time cost on get channel

Change-Id: I29741cf55ac6333bfa30fef755357b78a22b1325

* fix: ci style

Change-Id: I8c9b86e9cb6f1463bf213e64c9b8c139afa794c8
2023-07-27 10:51:51 +08:00
gaoyf ebad3c8a6b [ISSUE #7047] NettyRemotingClient#invokeOneway throw Exception with address 2023-07-20 10:59:40 +08:00
ShuangxiDing 70a66eda2c Fix LEAK: HAProxyMessage.release() was not called before it's garbage-collected (#7025)
Call HAProxyMessage.release() after reading it.

---------

Co-authored-by: 徒钟 <shuangxi.dsx@alibaba-inc.com>
2023-07-14 16:46:40 +08:00
ShuangxiDing b4496be687 [ISSUE #7010] Fix the HandshakeHandler returns when detect haproxy version need more data (#7011)
* Support dynamic modification of grpc tls mode to improve the scalability of ProtocolNegotiator

* Support dynamic modification of grpc tls mode to improve the scalability of ProtocolNegotiator

* [ISSUE #6866] Move the judgment logic of grpc TLS mode to improve the scalability of ProtocolNegotiator

* [ISSUE #6866] Move the judgment logic of grpc TLS mode to improve the scalability of ProtocolNegotiator

* [ISSUE #6866] Move the judgment logic of grpc TLS mode to improve the scalability of ProtocolNegotiator

* [ISSUE #6866] Move the judgment logic of grpc TLS mode to improve the scalability of ProtocolNegotiator

* Support proxy protocol for gRPC server.

* Support proxy protocol for gRPC server.

* Support proxy protocol for gRPC server.

* Support proxy protocol for gRPC server.

* Support proxy protocol for gRPC server.

* Support proxy protocol for gRPC and Remoting server.

* 回滚netty的升级

* Support proxy protocol for gRPC and Remoting server.

* Support proxy protocol for gRPC and Remoting server.

* Support proxy protocol for gRPC and Remoting server.

* add grpc-netty-codec-haproxy in bazel

* add grpc-netty-codec-haproxy in bazel

* Support proxy protocol for gRPC and Remoting server.

* Fix Test

* add grpc-netty-codec-haproxy in bazel

* add ProxyProtocolTest for Remoting

* Move AttributeKey from RemotingHelper to AttributeKey.

* Fix the needs more data for HandshakeHandler.

* Fix the needs more data for HandshakeHandler.

* Fix the needs more data for HandshakeHandler.

* Fix the needs more data for HandshakeHandler.

---------

Co-authored-by: 徒钟 <shuangxi.dsx@alibaba-inc.com>
2023-07-11 19:09:09 +08:00
rongtong 58550f074e [ISSUE #7008] Fix the issue of protocol parsing failure when using haproxy and tls together (#7009) 2023-07-10 14:13:18 +08:00
ShuangxiDing 00fc42b8be [ISSUE #6957] Support Proxy Protocol for gRPC and Remoting Server (#6958) 2023-07-04 18:02:16 +08:00
lk bbbe737e4e [ISSUE #6964] use the correct context in telemetry; polish the code structure (#6965) 2023-06-29 15:27:30 +08:00
rongtong e369d7deac [maven-release-plugin] prepare for next development iteration (#6939) 2023-06-24 14:31:42 +08:00
rongtong f7425f52f9 [maven-release-plugin] prepare release rocketmq-all-5.1.3 (#6938) 2023-06-24 12:23:11 +08:00
Quan 6238caaac9 [ISSUE #6841] new feature: pop batch ack implementation broker-side (#6842) 2023-06-19 11:06:50 +08:00
Drizzle 3dda55e812 [RIP-64] Heartbeat Optimization (#6724) 2023-06-14 15:00:24 +08:00
Zhouxiang Zhan aaa4a4b5fc Add attribute for SubscriptionGroupConfig (#6891) 2023-06-14 11:46:46 +08:00
imzs 4291971c03 [ISSUE #5866] fix client-side memory leak of inactive channel (#5867)
* fix #5866 Client-side memory leak of inactive channel

* also accept RemotingSendRequestException when NettyRemotingAbstract#failFast invoked.

---------

Co-authored-by: moling <zhangshuo.zs@alibaba-inc.com>
2023-06-13 16:47:10 +08:00
rongtong ad4cc957fb [maven-release-plugin] prepare for next development iteration (#6895) 2023-06-12 19:12:25 +08:00
rongtong 749967baa8 [maven-release-plugin] prepare release rocketmq-all-5.1.2 (#6892) 2023-06-12 17:44:39 +08:00
Drizzle bee5077bcb [ISSUE #6336] [RIP-62] Cold Read Control (#6507) 2023-06-05 17:09:31 +08:00
lk 9dfd4a9599 [ISSUE #6761] Support reentrant notification orderly for broker (#6762) 2023-05-16 11:32:11 +08:00
rongtong 8aea26ef8f [maven-release-plugin] prepare for next development iteration 2023-05-15 19:48:34 +08:00