* add isWakeCommitWhenPutMessage for AIO
* optimzie the Function name
Change-Id: Id91e3eb9c4488fb9804fb2c105082657e66c44c0
* optimized the function naming
Change-Id: Ifc482f91220ff328e5c5425a57a04ac627e8d469
---------
Co-authored-by: drizzle.zk <drizzle.zk@alibaba-inc.com>
* Use fastjson2 in all modules
* Update test
* Update test
* Update test
* Add serialization compatibility test tool class
* Update RemotingSerializableCompatTest.java
* Update RemotingSerializableCompatTest.java
* Update RemotingSerializableCompatTest.java
* Update BitSet problem
* Update
* Update
* Update test
* Update test
* Update BUILD.bazel
* Update BUILD.bazel
* Update test
* Update BitSet problem
* Add test
* Add compat test
* merge develop
* Update test
* merge develop
* fix(DirectBuffer): issue#9779,fix the promblem of 'DirectBuffer will cause error at 9+ JDK version'.
Change-Id: I4657ecc401046a3b0d29b466ee68845f45d34105
* fix(DirectBuffer): issue#9779,fix the promblem of 'DirectBuffer will cause error at 9+ JDK version'.
Change-Id: Iff0880f694cccfa86b812b81260dfe09e4763fa9
- Create StoreMetricsManager interface to define unified metrics management methods
- Add getStoreMetricsManager() method to MessageStore interface
- Add flushBehindBytes() method to MessageStore interface to fix missing method calls
- Refactor DefaultStoreMetricsManager to implement StoreMetricsManager interface
- Update all MessageStore implementations to support the new interface
- Modify TimerMessageStore to use getStoreMetricsManager() instead of direct type checking
This refactoring resolves the issue where incTimerDequeueCount and incTimerEnqueueCount
methods cannot be called when messageStore is not a DefaultMessageStore, improving
code extensibility and decoupling.
* refactor: convert RemotingMetricsManager from static to instance-based
- Remove all static global instance related code
- Convert all static methods to instance methods
- Remove static wrapper methods for backward compatibility
- Add constructor and getter/setter methods for instance variables
- This ensures each broker instance manages its own metrics in multi-broker scenarios
* refactor: convert PopMetricsManager from static to instance-based
- Remove all static global instance related code
- Convert all static methods to instance methods
- Remove static wrapper methods for backward compatibility
- Add constructor and getter/setter methods for instance variables
- This ensures each broker instance manages its own Pop metrics in multi-broker scenarios
* refactor: implement instance-based metrics management for multi-broker scenarios
- Add RemotingMetricsManager instance variable to NettyRemotingAbstract
- Add setter/getter methods for RemotingMetricsManager in NettyRemotingAbstract
- Update writeResponse method to use instance-based metrics recording
- Add writeResponse method to RemotingServer interface
- Update BrokerController to set RemotingMetricsManager on NettyRemotingServer instances
- Update InnerBrokerController to set RemotingMetricsManager on SubRemotingServer instances
- Update all processor classes to use instance-based writeResponse calls
- Update RocksDBStoreMetricsManager integration in DefaultStoreMetricsManager
- Remove unused imports and clean up code
This ensures each broker instance manages its own metrics independently,
solving the static variable conflict issue in multi-broker processes.
* refactor: fix test failures caused by metrics manager refactoring
- Fix ChangeInvisibleTimeProcessorTest by using @Spy instead of @Mock for BrokerController
- Inject BrokerMetricsManager into BrokerController using reflection
- Remove unnecessary mock stubbings
- Update all processors to use new static writeResponse method
- Fix BrokerMetricsManager initialization issues in AckMessageProcessorTest and PopReviveServiceTest
These changes ensure all related tests pass after refactoring metrics managers from static variables to instance variables.
* fix: resolve checkstyle errors and ensure compilation success
- Remove duplicate import in PopMessageProcessor.java
- Clean up unused imports in ChangeInvisibleTimeProcessorTest.java
- Clean up unused imports in AckMessageProcessorTest.java
- Ensure all modules compile successfully without checkstyle violations
These changes complete the metrics manager refactoring from static variables to instance variables, resolving statistical information conflicts in multi-broker processes.
* Fix PopReviveServiceTest by adding proper PopMetricsManager mock
- Add PopMetricsManager mock to PopReviveServiceTest
- Fix NullPointerException in reviveRetry method
- Ensure rePutCK method is called correctly when retry fails
- Test now passes successfully with proper metrics manager setup
* Remove debug code from PopReviveService
- Remove debug logging statements from reviveMsgFromCk method
- Clean up test output for better readability
* fix
* fix
- Add isRunningDequeue() check in checkDequeueLatch while loop
- Exit loop immediately when service is stopped to avoid thread blocking
- Ensure moveReadTime is not executed during shutdown process
- Improve system stability and reliability
* refactor: replace RandomAccessFile with FileChannel for better I/O performance
- Remove RandomAccessFile field and related logic completely
- Use FileChannel for all write operations when writeWithoutMmap is enabled
- Change SharedByteBuffer to use direct memory allocation (ByteBuffer.allocateDirect)
- Add RunningFlags support for better error handling
- Improve constructor design with better parameter handling
- Fix SharedByteBuffer write operation to ensure correct byte count
This change improves I/O performance by:
1. Eliminating the overhead of RandomAccessFile
2. Using direct memory allocation for better memory management
3. Providing more consistent I/O operations through FileChannel
4. Better error handling with RunningFlags integration
* writeWithoutMmap and transientStorePoolEnable cannot be used together. If both are enabled, only transientStorePoolEnable will take effect.
* Fix config comment
- Add configuration to control runningFlags usage in CommitLog (default: false)
- Update CommitLog and AllocateMappedFileService to respect the configuration
- Users can enable runningFlags validation as needed
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
* fix: prevent broker startup failure after power outage
- Add atomic file backup mechanism in persist() methods
- Delete corrupted config files during startup to avoid bak file pollution
- Add directory sync to ensure file operations visibility
Fixes: TimerMetrics#persist, TransactionMetrics#persist, ConfigManager#persist
* remove "Shutdown" implements
* empty commit
* add SuppressWarnings
* fix ut
* fix ut
* pass rocksdb ut when isMac()
* pass rocksdb ut when isMac()
* pass rocksdb ut when isMac()
* pass rocksdb ut when isMac()
* pass RocksdbGroupConfigTransferTest ut when isWindows()
* pass Rocksdb ut when isMac()
---------
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
* Optimize shutdown process and resource management
- Improve BrokerController shutdown flow for graceful shutdown
- Optimize BrokerStartup startup and shutdown logic
- Enhance ClientHousekeepingService resource cleanup
- Improve shutdown handling for various processors
- Optimize resource management in storage layer components
- Enhance lifecycle management for statistics manager
- Improve shutdown flow for timer components
* Fix this.popMessageProcessor.getPopLongPollingService() not shutdown
* Fix test shutdown state transition issue
- Add proper null checks and exception handling in test cleanup
- Prevent IllegalStateException during test teardown
- Ensure graceful test cleanup without state conflicts
* Fix DefaultMessageStoreCleanFilesTest can not pass
* Fix CombineConsumeQueueStoreTest can not pass
* Polish the code
* Polish the code
* Ignore flaky test first
- Move FLUSHED_POSITION_UPDATER.set() inside try block
- Prevent false positive flush success when actual flush operation fails
- Ensure data consistency and prevent potential data loss on system crash
Previously, flushed position was updated even when flush operation failed,
which could lead to data loss as the system would incorrectly assume
data was persisted to disk when it was still in memory.
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
* Add RunningFlags support to MappedFileQueue
- Integrate RunningFlags throughout MappedFileQueue hierarchy
- Add writeable state checking and error handling in DefaultMappedFile
- Update MappedFile interface and constructors to support RunningFlags
- Implement proper error state management during flush operations
* fix ut
* fix ut
---------
Co-authored-by: guyinyou <guyinyou.gyy@alibaba-inc.com>
* Cherry-pick partial changes from 67db1757df - BrokerMetricsManager refactoring
- Excluded DefaultMappedFile, METRICS_REFACTORING_GUIDE.md, TimerMetrics, and RocksDB files per requirements
- Successfully applied changes to most broker and store modules
- Compilation errors in TimerMessageStore.java will be fixed in next commit
* fix: Resolve metrics static variable conflicts in BrokerContainer mode
Convert static metrics variables to instance-level to fix resource leaks and
data conflicts in BrokerContainer scenarios with multiple broker instances.
## Problem Statement
In BrokerContainer mode, multiple broker instances share static metrics variables
from BrokerMetricsManager and DefaultStoreMetricsManager, causing:
- Metrics data conflicts between different broker instances
- Resource leaks during frequent addBroker/removeBroker operations
- Incorrect metrics aggregation across multiple brokers
## Solution
- Convert static metrics variables to instance-level variables
- Add proper getter methods for external access
- Ensure each broker instance maintains isolated metrics
- Apply instanceof checks for type safety in TimerMessageStore
## Files Modified
- broker/src/main/java/org/apache/rocketmq/broker/metrics/BrokerMetricsManager.java
- store/src/main/java/org/apache/rocketmq/store/timer/TimerMessageStore.java
- METRICS_REFACTORING_GUIDE.md (documentation)
## Key Benefits
✅ Eliminates metrics conflicts between broker instances
✅ Prevents resource leaks in dynamic broker scenarios
✅ Maintains proper metrics isolation per broker
✅ Supports BrokerContainer mode with multiple brokers
✅ Backward compatible with existing functionality
Resolves metrics static variable issues in multi-broker container environments.
* Delete useless file
* Fix test can not pass
* Fix test can not pass
* feat: Add accelerated startup recovery feature
Add accelerated startup recovery functionality when using RocksDB store with SYNC_FLUSH configuration:
- Add enableAcceleratedRecovery configuration option in MessageStoreConfig
- Implement accelerated recovery logic in CommitLog for both normal and abnormal recovery
- Add protective fallback mechanism to handle edge cases
- Improve isMappedFileMatchedRecover method for better robustness
- Add comprehensive unit tests for the accelerated recovery feature
This feature significantly reduces startup time when recovering from RocksDB-based storage
with synchronous flushing enabled, while maintaining data consistency and safety.
* Fix the issue of accelerated startup failure
* refactor: Remove problematic unit test
Remove AcceleratedRecoveryTest.java as the test implementation was not appropriate for the accelerated startup recovery feature.
* Delete useless code
littleboy