opensource/rocketmq
3
0
Fork 0
mirror of https://github.com/apache/rocketmq.git synced 2026-08-28 20:09:14 +08:00
Code Issues Packages Projects Releases Wiki Activity

chore: exclude local planning artifacts

Browse Source
This commit is contained in:
通融
2026-08-16 09:50:15 +08:00
parent f644676f64
commit cfc61dfc1d
2 changed files with 0 additions and 3593 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-08-14-dledger-latest-pr336-fastjson2.md
-3477
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/specs/2026-08-14-dledger-latest-pr336-fastjson2-design.md
-116
View File
@@ -1,116 +0,0 @@
# Latest DLedger PR #336 RocketMQ Fastjson2 Migration Design
**Date:** 2026-08-14
## Context
RocketMQ still consumes DLedger 0.3.2. Pull request apache/rocketmq#10650 upgrades that dependency to DLedger 0.3.3-SNAPSHOT and adapts RocketMQ to the current DLedger API, but it is based on an old RocketMQ commit and leaves Maven/Bazel dependency and compatibility-test gaps.
DLedger pull request openmessaging/dledger#336 fixes a regression in current DLedger master: a locally successful append returns an `AppendFuture` whose position is still `-1`. RocketMQ reads that position before quorum completion in both the commit log and controller paths, so the regression makes a valid append look like a failure.
The existing 0.3.2-maintenance solution remains unchanged on its own branches. This design creates an independent latest-code validation path.
## Source Baselines
- RocketMQ: `develop` at `293f5885719fc4aa3619446a1900f58ccfcfdd29`.
- DLedger: current `master` at `2834424cec15670c1b1108378b0b0a3a5fd57791` plus PR #336 head `a2555fa78fcda379279a815357717283d0f6c56b`.
- fastjson2: `2.0.64` for Maven and Bazel, matching apache/rocketmq#10928 and avoiding the JDK 8 initialization-order regression in 2.0.61 through 2.0.63.
The DLedger validation artifact will use a unique local version such as `0.3.3-pr336-f2-64-SNAPSHOT`. Version and fastjson2 metadata changes are build-only overlays; PR #336 remains a focused append-position fix.
## Goals
1. Port the useful DLedger API adaptation from RocketMQ PR #10650 onto current RocketMQ develop.
2. Consume current DLedger plus PR #336 and prove the append position is available immediately for single and batch appends.
3. Remove fastjson1 from RocketMQ and DLedger resolved dependencies, test classpaths, Bazel artifacts, and packaged bytecode.
4. Preserve the ability to read representative JSON produced by fastjson 1.2.83 without executing fastjson1 in tests.
5. Validate RocketMQ behavior across storage, controller, broker lifecycle, production/consumption, failover, and restart paths.
## Non-Goals
- Mixed-version rolling operation between DLedger 0.3.2 and current master.
- Direct downgrade from current DLedger data to 0.3.2.
- Push-protocol compatibility bridging between the old `entry`/`batchEntry` schema and the new `entries`/`preLogIndex`/`preLogTerm` schema.
- Unrelated RocketMQ or DLedger refactoring.
The result may support a coordinated full-stop upgrade after validation, but it must not claim rolling-upgrade compatibility.
## RocketMQ Changes
### DLedger API adaptation
Port PR #10650 by intent rather than cherry-picking its stale commit:
- Import `AppendFuture` and `BatchAppendFuture` from `io.openmessaging.storage.dledger.common`.
- Replace `setEnableBatchPush` with `setEnableBatchAppend`.
- Read the committed index through `MemberState` where the latest store API removed it.
- Adapt the controller state machine from `CommittedEntryIterator` to `ApplyEntryIterator` and unwrap `ApplyEntry`.
- Derive a committed physical position from the committed entry because `getCommittedPos()` no longer exists.
- Update Maven and Bazel DLedger coordinates together.
No PR #10650 code is accepted blindly. Each change is checked against the exact PR #336 artifact and current RocketMQ source.
### Complete fastjson1 removal
- Remove `com.alibaba:fastjson` from root Maven dependency management and `common/pom.xml`.
- Remove the fastjson1 artifact from `WORKSPACE` and its test dependency from `remoting/BUILD.bazel`.
- Pin fastjson2 2.0.64 in both Maven and Bazel.
- Preserve the DLedger POM exclusion for `com.alibaba:fastjson`; the exclusion is allowed as guard metadata and must not resolve an artifact.
- Replace the live fastjson1 compatibility oracle with checked-in JSON literals generated by fastjson 1.2.83.
Golden fixtures cover `BatchAck`, `SubscriptionData`, and `ConsumerConnection`, including byte/bit encoding, nested collections, enums, renamed fields, and fields marked `serialize = false`.
## Data Flow and Correctness
For a RocketMQ write, `DLedgerServer` appends locally before quorum acknowledgement. PR #336 sets the returned future position to the appended entry position before returning. RocketMQ then computes the message physical offset without mistaking the append for `OS_PAGE_CACHE_BUSY`. Batch append uses the last entry as the base future position and retains every individual position in `BatchAppendFuture#getPositions()`.
Controller append follows the same immediate-position contract. State-machine application uses the latest `ApplyEntryIterator` API and applies the embedded DLedger entry body exactly once.
Committed-position derivation receives additional restart coverage. Current DLedger initializes `MemberState.committedIndex` to `-1` and no longer restores the old checkpoint value. The validation must therefore cover an existing-data full restart before any new user write. If that test exposes an incorrect max offset, read result, or role transition, a minimal RocketMQ fix will be designed from the failing test without treating an uncommitted tail as committed.
## TDD and Verification Strategy
### DLedger gates
- Reproduce the current-master failure where single and batch futures return `-1` immediately after local append.
- Verify PR #336 makes the same assertions pass before quorum completion.
- Run the DLedger unit reactor and package the unique local artifact with JDK 8 and fastjson2 2.0.64.
### RocketMQ compile and dependency gates
- First compile current RocketMQ against the PR #336 artifact to capture the expected latest-API failures.
- Port the minimum API changes until Maven and Bazel compile.
- Verify Maven dependency trees resolve one DLedger artifact and fastjson2 2.0.64, with no fastjson1.
- Verify Bazel resolves the same versions and excludes DLedger's external RocketMQ remoting artifact in favor of the reactor target.
### Focused behavior tests
- `RemotingSerializableCompatTest`, including the three fastjson1 golden fixtures and fresh-JVM JDK 8 cold start.
- DLedger controller append and state-machine application tests.
- DLedger commit-log single and batch append tests, including immediate future positions.
- Broker container add/remove DLedger broker lifecycle.
- Connection-failure callbacks for DLedger RPC futures.
### Integration and recovery gates
- Real DLedger produce-and-consume integration test.
- Three-node single and batch append, leader election, leader shutdown, follower promotion, and continued production/consumption.
- Controller leader change and committed-index convergence.
- Existing data full-cluster shutdown and restart. Before a new write, verify max offset, message reads, dispatch progress, and broker role transition; then append and consume new messages and restart once more.
- Run relevant Maven packaging/tests and Bazel builds/tests on JDK 8.
Tests that create DLedger stores use the low-usage isolated APFS volume. A `DISK_FULL` result on the host data volume is an environment failure and is rerun only after the test JVM is proven to use the isolated volume.
## Acceptance Criteria
- All targeted Maven and Bazel builds and tests pass.
- DLedger single and batch futures expose correct positions before quorum completion.
- RocketMQ production, consumption, controller, failover, and existing-data restart scenarios pass.
- `mvn dependency:tree`, Bazel resolution output, source scans, JAR entry scans, and class constant-pool scans contain no resolved or embedded fastjson1.
- fastjson1 1.2.83 golden JSON remains readable through production `RemotingSerializable` APIs.
- Maven and Bazel both use fastjson2 2.0.64 and the same unique DLedger validation coordinate.
- The existing maintenance-line solution and its branches remain unchanged.
## Delivery Boundary
The first result is a local validation branch and evidence report. It is not pushed as a RocketMQ community PR while it depends on a local-only DLedger coordinate. After PR #336 is merged and a consumable DLedger version exists, the coordinate is replaced with the public version and the clean Maven/Bazel/integration matrix is rerun before publication.