Reject causal cycles before source repair and share conservative backup
classification between discovery and inspection. Fence public raw deletion
and derive exact-key removals from successful settlement without an extra
metadata read. Preserve the existing physical phases and strict source checks.
Make outage and renewal fixtures deterministic without weakening their
convergence assertions, and complete strict TLS/RPC lint cleanup.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bound sentinel tests by the lock wait, not recreation IO
Both bucket incarnation sentinel tests gave the bucket recreation that
follows the paused delete (force delete_bucket, make_bucket, put_object
on four fsync'ing disks) 10 s in total. The delete releases the sentinel
milliseconds after the barrier, but on a runner starved for IO the
recreation alone can exceed 10 s:
delete_objects_holds_bucket_incarnation_sentinel_through_commit failed
that way on main in run 37760321449 (job 113259251979).
That bound is not what catches a kept sentinel: delete_bucket waits for
it on its lifecycle write lock, which fails after the lock acquire
timeout, and the recreation's expect reports the lock error. Keep that
check and give the recreation 120 s only as a hang guard.
The recreation starts that lock wait before the barrier is released, so
the wait also has to cover the delete's own commit, and the 5 s default
leaves a starved runner little room for it. Run both tests with
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30 and bound the delete, which was
not bounded, by the same 30 s. A kept sentinel still fails the tests,
after 30 s instead of 5 s.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* test(ecstore): keep background recovery out of dispatch rollback tests
dispatch_manifest_detached_workers_are_store_bounded_and_shutdown_cancellable
seeds six aborting dispatch manifests, installs a pause hook and expects
its own recovery passes to park four detached workers on it. Its first
store also ran the background tier-delete journal recovery loop, which
runs a pass every TIER_DELETE_JOURNAL_RECOVERY_INTERVAL (60 s). When the
setup took longer than that, a background pass rolled the manifests
back before the hook was installed, no worker paused, and the 30 s wait
failed. This happened on main on a runner starved for IO, in run
37760321449 (job 113259251979).
Build the first store with background recovery suppressed, as the
neighbouring dispatch tests do. The restarted store keeps it: the second
half of the test checks that it picks up the retained manifests.
dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member
and dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence
have the same exposure: each seeds an aborting manifest on a store with
background recovery and only then installs its slow-delete hook. They
have not failed in CI, but with the setup stretched past 60 s they fail
the same way. Suppress background recovery for them too.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
---------
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): recover crash-interrupted delete retries
Retain set-quorum cleanup proof and restore metadata before dropping rollback
markers. Reclaim eligible exact-key residue during explicit delete retries,
with bucket identity, version, retention and namespace fencing guards.
Preserve prefix children and snapshot ownership through batched physical
cleanup, and cover interrupted transactions and failure boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): route recovery test imports through contracts
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve missing-key deletes at quorum
* feat(ecstore): add certified set-wide delete recovery
Atomically prepare deletion journals and exact targets, publish durable set
receipts, and recover through one fenced disk/RPC core used by startup and
scoped administrator healing. Preserve returning generations, ordinary
missing-key quorum semantics, and legacy rollback wire compatibility.
Retain namespace and volume ownership through late physical operations,
order related transactions causally, and keep proof until full convergence.
Remove the superseded local-only recovery path and add storage, protocol,
RPC, durability and isolated Docker regression coverage.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): reclaim completed multipart delete sidecars
* fix(ecstore): recover multipart sidecars with verified proofs
Recognize native multipart sidecars at the shared recovery boundary and
restore them without treating metadata or proof files as shard payloads.
Validate required proof generations and bind every present live/staged
shard copy to its metadata-bound Merkle proof before restoring metadata.
Cover protected native MPU rollback, split source layouts, missing or
corrupt proofs, and payload corruption with recomputed bitrot checksums.
Add real multipart Docker regressions in both integrity modes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): settle superseded committed version deletes
Allow a validated committed decision to finish its own data cleanup after
a later decision removes current metadata. Prepared copies need an
inspected same-transaction receipt matching immutable local intent before
promotion, and retain their local journal without importing peer hashes.
Reuse the authority matcher for dry-run and settlement. Cover chained
explicit-version deletions, partial receipt publication, mismatch rejection
and no-resurrection, while retaining the no-decision absence protection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): clarify delete recovery case label
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): restore recovery CI guard coverage
Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(admin): preserve owned OIDC validation errors
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): unify recovery authority and snapshot work
Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): preserve delete recovery invariants
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
* feat(tls): reload external certificate snapshots coherently
Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(tls): publish server certificates through a fenced admin API
Add opt-in server-leaf publication with dedicated IAM actions, actual TLS
transport checks, immutable generations, durable captured-token CAS and
idempotent receipts. Preserve independent trust and client identity sources
and distinguish visible publication state from the full serving chain.
Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test,
and eight live local API scenarios including least privilege and restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(admin): route certificate handler types through S3 facade
Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(admin): include TLS certificate routes in route matrix
Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tls): reconcile startup trust before skipping reloads
Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(tls): reuse pinned generations and prepared listener state
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tls): preserve startup trust source requirements
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
A child prefix can be deleted after its parent listed it and before scan_folder opens it. read_dir then returned NotFound, which propagated through the compacted subtree and dropped the whole top-level child (child_scan_failed), so the cycle never produced a durable usage snapshot under continuous deletes.
Treat NotFound on open like a directory that vanished during iteration for non-root folders. A missing bucket root still reports NotFound so namespace removal handling is unchanged.
Refs #8478
Co-authored-by: Anton <5527364+basic-system@users.noreply.github.com>
use_stdout was forced to true whenever the root RUSTFS_OBS_ENDPOINT was
empty, so a deployment that sets only a per-signal endpoint such as
RUSTFS_OBS_TRACE_ENDPOINT could not turn the stdout exporters off: every
exported span was also printed to stdout as a multi-line text block.
Count a non-empty trace, metric or log endpoint as a configured endpoint.
With no endpoint at all the stdout default is still forced on. Add a test
covering each per-signal endpoint and an empty value.
* fix(storage): preserve heal failures and timeout responses
Stop repairs with no writable target before opening source readers. Consolidate target failure aggregation, preserve typed causes, and distinguish all-ignored diagnostics from actual successful nil entries. Map typed disk timeouts to retryable S3 responses without changing quorum decisions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(heal): unify staging failure cleanup
Route data-stage failures through one local result boundary after temporary
readers and writers are released. Preserve typed report failures, outer
reconstruction errors, partial repair, and the existing commit fences.
Exercise actual source loss after second-part writer admission using the
existing test pause registry. Assert all staged parts are removed, committed
metadata stays unchanged, and restoring the source allows a full-byte retry.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(scanner): bound old-cache lookups and recover checkpoint saves
Keep missing child lookups read-only and preserve the checkpoint receiver
after ordinary persistence errors. Reload on stale CAS or indeterminate
revision refresh, and report confirmed periodic saves separately.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): remove redundant checkpoint error conversion
Return the existing StorageError directly so strict Clippy passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs(scanner): clarify checkpoint persistence guarantees
Distinguish a confirmed periodic save from validated restart coverage.
Keep adoption and subtree-skip requirements explicit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): preserve resumable periodic coverage
Materialize active ancestor accounting in bounded periodic snapshots and
seal completed coverage independently of raw enumeration state. Preserve
durable coverage during early resumed enumeration and exclude unfinished
out-of-order heal work from periodic and cancellation receipts.
Reuse ancestor frame slots to avoid growing child-set copies on descent,
retain compacted accumulator safety, and report checkpoint adoption resets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): reject covered active ancestors in checkpoints
Check every suspended ancestor alongside the current folder before
certifying completed coverage. An unfinished covered ancestor must not
become a resumable subtree merely because its descendant sorts later.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain an incomplete root for missing namespaces
Create the empty bucket root explicitly at the missing-namespace boundary.
Preserve nonempty durable caches and keep namespace results unpublished.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: repair s3s footprint ratchet violation, clippy warnings, and stack overflow
- Replace s3_error! macro with S3Error::with_message in ecfs.rs to keep
s3s footprint ratchet at baseline (1586) after retention validation refactor
- Remove redundant clone on Copy type in list_objects.rs (clippy::redundant_clone)
- Remove clone_on_copy calls in access.rs test (clippy::clone_on_copy)
- Use manual thread with 8MB stack for heal_concurrent_delete_test to fix
stack overflow on macOS (consistent with sibling tests in same file)
* test: align CI assertions with version authorization rules
* fix(listing): continue versioned delimiter listings from raw key
Versioned listings with a delimiter could end a page with
IsTruncated=false while keys remained past the folded scan window.
Continue from the last raw scanned key; a folded prefix cannot
advance forward_past on versioned walks.
Fixes#8423
* fix(listing): filter resumed version prefixes
Apply the ListObjectVersions CommonPrefixes key-marker boundary before
page truncation and resume at the last raw key represented by the final
returned prefix.
Refs rustfs/backlog#2791
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(lifecycle): coordinate tier free-version recovery scans
* fix(lifecycle): remove obsolete recovery enqueue entry point
* test(upgrade): wait for writers after each rolling replacement
* fix(ecstore): keep object deletes running when the caller is dropped
DeleteObject and DeleteObjects commit on every disk and then run a
separate cleanup pass, both inside the request future. When the client
disconnects, hyper drops that future and the delete stops between the
two passes: xl.meta is gone, but the rollback backup and the marked data
dir stay on disk, hidden from listings and never reclaimed (#6898).
Run the ECStore delete entry points on a detached task tracked by the
instance context. The task owns the object locks, both disk passes and
any rollback, so dropping the caller only stops the wait. Server
shutdown waits up to 30s for detached deletes after the HTTP drain.
Two smaller leaks of the same kind: a cancelled data-dir delete no
longer leaves its snapshot-lease entry marked as deleting forever, and
DiskStore::delete_versions releases its waiting slot through a guard.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): finish committed delete residue on repeated deletes
Residue that an older build stranded (#6898) has no xl.meta, so deleting
the key again reports it missing and leaves everything in place. On a
disk, a delete-data.<T> marker in one of the object's data dirs proves
that transaction T removed the metadata there and committed: the data it
marked can no longer be read, healed or rolled back on that disk.
When delete_versions or delete_version finds no xl.meta, remove what the
cleanup pass of every transaction proven that way would have removed,
under the object's metadata lease. The rollback backup goes before the
marked data dirs, so an attempt cut short still leaves the marker for
the next one. Reserved markers, rollback backups without a committed
marker and directories that also hold xl.meta or other keys are left
alone. The disk result stays FileNotFound; the extra cost is one
directory read per missing key inside the disk call that already runs,
with no new RPC, and the read stops at the first child that is not a
data dir. Finished transactions are counted in
rustfs_disk_delete_residue_finished_total instead of logged per object.
A single DeleteObject in an unversioned bucket answers a missing key
from the metadata read quorum without reaching the disks, so existing
residue is reclaimed by DeleteObjects or by the empty-listing purge.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): purge the rollback backup of a committed delete on listing
After an empty listing, the orphan purge (#7955, #8261) already removes
data dirs that carry a committed delete-data.<T> marker, but it treated
the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data
and blocked the whole object directory and its ancestors. Residue left
by an abandoned delete (#6898) therefore survived the listing, and
DeleteBucket kept failing with BucketNotEmpty.
Treat a UUID dir that holds only xl.meta.bkp as purgeable when a
purgeable data dir of the same object on the same disk carries the
committed marker of that transaction. A backup without such a marker
still blocks as before, since it may be needed to roll the delete back.
The other purge gates are unchanged: every disk of the set must be
online, and the backoff applies. xl.meta or unmarked data on any disk
blocks that directory and its ancestors on every disk. Like a marked
data dir before, a backup below such a directory is still removed from
a disk that holds no xl.meta there and proves the transaction with its
own marker.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(ecstore): move caller-held delete locks into the detached delete
A force (recursive) delete takes the bucket lifecycle write lock in the
S3 handler and passes only its fence down, so the store does not take
that lock again. Since the delete runs on a detached task, dropping the
request released the lock while the delete kept running: a PUT under
the same prefix could succeed and then be removed by the old delete.
Lifecycle expiry had the same shape with its table-bucket publication
read lock, of which only the lock-lost signal reaches the delete.
Add ECStore::delete_object_with_tier_delete_journal_and_guards. It
moves the caller's guards into the detached task and releases them
after the delete and its listing bookkeeping. The force-delete handler
and lifecycle expiry hand their guards over; DeleteObjects holds none.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* fix(startup): keep unclean-shutdown markers while deletes still run
Shutdown waits up to 30s for detached deletes, but it cleared the
unclean-shutdown markers even when that wait timed out. A delete then
cut short by the runtime teardown went unnoticed: the next startup
skipped the erasure-set heal that the marker triggers.
Clear the markers only when the heal handoff completed and every
detached mutation finished, and log that they are kept otherwise. Wait
after the FTP, FTPS, SFTP and WebDAV servers have stopped as well,
since their deletes take the same detached path.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
* test(ecstore): simulate tier-delete crashes below the detached delete
The chunk crash-boundary tests model a crash by aborting the task that
runs delete_object_with_tier_delete_journal while it is paused at a
TierDeleteChunkTestBarrier. That entry point now runs the delete on a
detached task, so aborting the caller no longer stops it: the delete
resumed on the old store once the barrier was dropped, and building the
restarted store on the same disks tripped the tier-delete journal proof
assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources
failed in CI).
Run the delete through handle_delete_object_with_journal, the body
below the detached task, so the abort stops it at the barrier as before.
tier_delete_chunk_missing_child_with_journals_fails_closed used the same
pattern; its resumed delete only failed fast because the tier rejects
removes, so it gets the same change.
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
---------
Signed-off-by: Petr Slavyagin <disident47@gmail.com>
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints.
