4166b9503f
* fix(ecstore): recover crash-interrupted delete retries Retain set-quorum cleanup proof and restore metadata before dropping rollback markers. Reclaim eligible exact-key residue during explicit delete retries, with bucket identity, version, retention and namespace fencing guards. Preserve prefix children and snapshot ownership through batched physical cleanup, and cover interrupted transactions and failure boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route recovery test imports through contracts Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve missing-key deletes at quorum * feat(ecstore): add certified set-wide delete recovery Atomically prepare deletion journals and exact targets, publish durable set receipts, and recover through one fenced disk/RPC core used by startup and scoped administrator healing. Preserve returning generations, ordinary missing-key quorum semantics, and legacy rollback wire compatibility. Retain namespace and volume ownership through late physical operations, order related transactions causally, and keep proof until full convergence. Remove the superseded local-only recovery path and add storage, protocol, RPC, durability and isolated Docker regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim completed multipart delete sidecars * fix(ecstore): recover multipart sidecars with verified proofs Recognize native multipart sidecars at the shared recovery boundary and restore them without treating metadata or proof files as shard payloads. Validate required proof generations and bind every present live/staged shard copy to its metadata-bound Merkle proof before restoring metadata. Cover protected native MPU rollback, split source layouts, missing or corrupt proofs, and payload corruption with recomputed bitrot checksums. Add real multipart Docker regressions in both integrity modes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): settle superseded committed version deletes Allow a validated committed decision to finish its own data cleanup after a later decision removes current metadata. Prepared copies need an inspected same-transaction receipt matching immutable local intent before promotion, and retain their local journal without importing peer hashes. Reuse the authority matcher for dry-run and settlement. Cover chained explicit-version deletions, partial receipt publication, mismatch rejection and no-resurrection, while retaining the no-decision absence protection. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): clarify delete recovery case label Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): restore recovery CI guard coverage Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): preserve owned OIDC validation errors Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): unify recovery authority and snapshot work Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve delete recovery invariants --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com>
Documentation
Use the focused indexes rather than treating this directory as an unordered collection:
Operations
-
External certificate issuance and TLS rotation — publication, native reload, trust separation and failure handling.
-
Architecture guard troubleshooting — diagnose repository guard failures.
-
Logging governance — broad event audits, migrations, and guardrail changes.
-
Multipart upload memory diagnosis — allocator attribution, Docker reproduction and sustained upload checks.
For the logical per-operation io_uring read cap, see io_uring read chunk size.
For bounded local read-backend startup and cancellation, see io_uring backend initialization.
For legacy protection assessment and bounded protected copies, see Object integrity inventory, audit, and migration.
For encrypted listing compatibility and older stored fingerprints, see Encrypted object ETags.
Operational runbooks live under operations/. Replication
operators should start with:
| Runbook | Use it for |
|---|---|
| Site replication operations | Health fields, pending operations, outage recovery, re-pair admission, IAM/SSE boundaries, and upgrades. |
| Replication target check | Validating an S3 destination and version fidelity before enabling replication. |
| Replication object size limits | Multipart routing, large-object limits, and retry characteristics. |
| Replication outbound transport | Integrity headers, generic target behavior, and transport knobs. |
For the erasure-coded cluster lifecycle (planning, parity and EC:0,
expansion, rebalance, decommission, heal, drive replacement, restart
recovery, and the rc CLI mapping), start with
Cluster and erasure-coding lifecycle operations.
For concurrent bucket creation and deletion, see Bucket operation admission.
For persisted administrator bucket tasks and bucket recreation, see Bucket heal recovery.
For interrupted object deletions and returning drives, see Delete transaction recovery.
For disk replacement across VM restarts and schema 5/6 maintenance migration, see Replacement generation recovery.
For historical GET timeouts during PUT or Heal, see Object lock contention diagnostics.
Other runbooks remain grouped by filename in operations/;
architecture pages link to the relevant runbook where a cross-boundary
procedure is required.
For storage dashboards, see Storage metrics and observer selection: drive ownership, snapshot freshness, counter queries, and rolling upgrades.
For optional shard commitments, see Independent shard integrity rollout: activation, legacy repair results, multipart mode changes, and rollback limits.
For crates.io publication of workspace crates, see Workspace Cargo Publish: dependency ordering, dry-run, publish, and failure handling.