* test(heal): relative disk target and fail fast on terminal-but-short
The absolute 40 GiB heal target was calibrated to the background scanner
(auto-heal), which is now disabled for determinism; with only the explicit
heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the
success criterion relative: the outage node must reach at least 90% of the
least-used surviving node (absolute HEAL_TARGET_GB floor optional, default
0 = relative only).
Also fail fast when the heal task reaches a terminal success but the disk
target is not met (previously the monitor kept polling until timeout), and
drop the misleading 'progress absent' warning on the final (cleaned) task
response — mid-run progress is reported correctly.
Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB,
vm002=40GB (target 36GB), test PASSED.
* test(heal): gate success on server verdict + data read-back, drop disk GB gate
The per-node disk-usage target (40 GiB / 90% of survivors) is not a
code-level invariant: EC distributes different shards per node, so the
final GB per node depends on the layout, not on heal correctness. Gate the
test on what the server actually verifies:
- Heal task terminal success (finished/completed) with objectsFailed == 0
(the server's per-object scan/repair verdict).
- S3 read-back verification: list the test bucket and GET a sample of
objects, requiring HTTP 200 for every read (end-to-end proof the data is
still reconstructable after repair). The GET uses a discard mode so
binary bodies are not captured (no null-byte warnings / SIGPIPE).
Per-node disk usage stays in the output as observability (with a warning if
the outage node gained no usage), not as the pass/fail gate. Removes the
heal_target_gb input and the relative-target logic.
Validated live: heal summary=finished, 0 failed, 20/20 objects read back,
vm002_used=40GB, PASS.
Add a default-on, size-aware foreground PUT admission policy so large or
unknown-size PutObject requests are backpressured before body ingest and
erasure/RPC fan-out. Preserve the explicit strict gate semantics, including
limit=0 as an opt-out, and keep small PUTs on the legacy fast path.
Closesrustfs/backlog#2038
Co-authored-by: heihutu <heihutu@gmail.com>
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A fixed /tmp path (log file, final heal status, warp log) can be owned by
another user on the shared runner (e.g. a previous root run), which made the
github-runner user fail: tee could not append the test log, the final heal
status write killed step 6 with EACCES, and upload-artifact could not read
stale root-owned warp logs. The heal scenario itself had passed
(summary=finished, vm002 reached the target) before the status-save died.
- Log files fall back to a unique mktemp path when the configured path is not
writable (heal + pool scripts).
- The final heal status is written to a mktemp file (best effort).
- Workflow artifact uploads use globs for the fallback names.
* test(heal): add node-outage heal E2E script and workflow
RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.
Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.
The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.
* ci(pool-test): chain heal regression after the pool test
The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:
1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
(if: always()): a pool failure makes the run red but does not block the
heal regression. Runs the heal script (reset -> install/start 3x4 ->
write/outage -> heal -> verify -> reset).
* test(heal): address review — camelCase progress, fail-closed, workflow hygiene
- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
objectsFailed/progressPercentage); read them with a snake_case fallback and
distinguish null (absent) progress from zero, logging null as evidence
(rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
the write target must be reached, and an unobserved outage or incomplete
write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
an exact successful nightly conclusion; checkout is pinned to the triggering
SHA; comma-separated step args are quoted (actionlint SC2054).
Keep /health/ready from riding the generic internode lock RPC and channel keepalive budgets when a peer host is unreachable. Add a health-specific lock online timeout, route ping failures through the existing remote lock RPC eviction path, cache the static ping payload for readiness fan-out, and cover hanging cached channels with focused tests.
Refs rustfs/backlog#2033
Refs rustfs/rustfs#6286
Co-authored-by: heihutu <heihutu@gmail.com>
Config::new() and the external decode path read the process-global
DEFAULT_KVS OnceLock at call time, and config::tests in the same test
binary register it via crate::config::init() mid-run. Several com.rs
tests asserted on unregistered state (heal section absence, equality
with a later Config::new()), so they could flip depending on thread
scheduling under cargo test -p rustfs-ecstore --lib config::.
Assert on the semantic heal diff instead of section presence, normalize
compared configs with a single DEFAULT_KVS snapshot taken after both
sides exist, and compare the snapshot transaction test against the
persisted baseline bytes.
* test(pool): fix warp log path and retry rebalance start
- warp writes now use a unique mktemp log file instead of a fixed
/tmp/rustfs-warp.log: the runner user could not write the stale
root-owned file, which made the background warp process die instantly
(warp never ran). The workflow uploads /tmp/rustfs-warp.*.log.
- rebalance start is retried (6x, 20s apart): nightly builds gate
rebalance activation on a live cross-pool fence fleet capability proof
that takes ~10-20s to re-establish after a pool joins. Verified live:
attempt 1 fails with 500 'pool activation requires a live fleet
capability proof', attempt 2 succeeds.
* test(pool): annotate known server-side issues in failure output
When a node fails to start, grab the rustfs journal tail and match known
server-side error signatures (e.g. the fleet capability proof cold-start
regression, rustfs/backlog#2031), printing a hint with the tracking issue.
Also annotate the rebalance-start retry exhaustion and the rc.3 decommission
metacache-listing failure with actionable guidance.
* fix(ecstore): defer rebalance activation without fleet proof
---------
Co-authored-by: 马登山 <cxymds@qq.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Preview tags stay as the traceability record for the validated commit, but their GitHub Releases are internal validation state and should not accumulate on the Releases page next to real deliverables.
Add a cleanup-preview-releases job that runs after publish-release succeeds for a release or prerelease tag and deletes every Release whose tag is exactly <target>-preview.<digits>. The tags themselves are kept: the job never passes --cleanup-tag. Tag matching uses jq string operations rather than a regex over the version, so dots in the version cannot widen the match, and the release listing is fetched before filtering so an API failure aborts the job instead of looking like there was nothing to clean up.
Extend the preview release workflow guard with the job condition, the delete invocation, both tag-matching filters, and an absent check for --cleanup-tag.
* test(ecstore): decouple server config snapshot test from global defaults
The final assertion of server_config_snapshot_serializes_read_modify_write_transactions
compared the second snapshot against a fresh Config::new(). Config::new()
reads the process-global DEFAULT_KVS OnceLock, which a sibling test in the
same process can register mid-run (crate::config::init()), so the in-process
run 'cargo test -p rustfs-ecstore --lib config::' failed while nextest's
process-per-test isolation hid the coupling. Assert on the snapshot's raw
bytes against the baseline blob instead, which is deterministic and matches
the invariant under test: the second transaction observes the store unchanged
by the first.
* test: deflake presigned tamper helper and relocated-pool resume staging
tamper_signature only remapped '0' and 'a', so a signature containing
neither (about 1 in 5000) left the URI unchanged and tripped the helper's
own guard assert in CI. Complement every hex digit (15 - v) instead: the
map has no fixed point, so the tamper always changes the value while
keeping length and hex shape.
execute_get_object_resumes_from_relocated_pool_without_splicing_body
staged the relocation by reading xl.meta from every source-pool disk, but
a write-quorum commit legitimately leaves a lagging minority disk without
the object directory (#6701) — the test already tolerates that gap when
normalizing the upload pool, and CI suite IO load hit the same gap in the
staging loop. Skip sourceless disks, carry the staged metadata path
explicitly, and assert a write-quorum majority was staged.
* refactor(site-replication): move business tests next to the service module
backlog#1840 PR5: 79 business-logic tests (plus 12 helpers, 6 of them small fixtures kept on both sides) move from the admin handler file's test module into rustfs/src/site_replication/tests.rs, next to the code they exercise: peer connection/TLS/DNS/egress validation, the peer client cache and payload wire contract, retry-queue classification/settlement/escalation/backoff, the repair state machine, bootstrap-plan construction, lifecycle expiry subsetting, bucket-target reconciliation, endpoint/identity normalization, and state serialization. The 149 tests that exercise the admin handlers, apply/reconcile paths, status/resync builders, and the four include_str! tripwires stay in rustfs/src/admin/handlers/site_replication.rs with their subjects (229 total conserved: 149 + 79 + 1).
The issue's PR5 also called for converting the source-order tripwire at the old file's line 11339 into a behavior test; both adversarial review passes re-derived all four tripwires against the shrunken file and found them non-vacuous and byte-identical in the regions they guard (the handler bodies, which did not move), so they stay as source-text assertions.
Supporting changes: the root facade's site_replication consumer gains cfg(test) re-exports (endpoint types, merge_incoming_replication_config, five lifecycle DTO types) so the relocated tests stay off the direct s3s/admin surfaces — including rewriting the one inline crate::admin BucketMetadata path a moved test carried over (review finding); tests.rs joins the logging-guardrail checked list; the embedded-secrets guard comment follows the validate_peer_connection_inner fixtures to their new file.
Verified: cargo check -p rustfs --all-targets clean; cargo nextest run -p rustfs --lib 3856/3856 passed; relocated tests run under site_replication::tests::; make pre-commit green including the s3s footprint ratchet; logging and embedded-secrets guards green.
Refs rustfs/backlog#1840
* style(site-replication): apply rustfmt import ordering
* test(targets): ship a builder MockTarget testkit and retire the in-crate Target mocks
Adds crates/targets/src/testkit.rs with a builder-style MockTarget implementing Target<E> for every E: PluginEvent, with orthogonal off-by-default knobs: disabled/active override, health delay plus health-started signal plus a drop-guard counter proving a cancelled probe future was dropped, an init failure budget (usize::MAX = always fail) plus blocking init plus an init counter, a close counter/signal/semaphore gate with a runtime block toggle, a save counter plus save failure budget, caller-supplied store and failed-store handles, and a shared final-failure counter. Clones and clone_dyn share all counters, so an observer clone keeps watching a target after it is boxed into a runtime.
The module is gated as #[cfg(any(test, feature = "test-support"))]: in-crate unit tests get it via cfg(test), and downstream test suites opt in through the new off-by-default test-support cargo feature (test-support = [], activating no dependencies).
Migrates the five in-crate duplicate mocks onto it: the plugin.rs registry-factory TestTarget, the runtime/adapter.rs lifecycle TestTarget (init/close/store knobs), the runtime/mod.rs TestTarget plus HealthDropGuard (close gating and health-probe tests), the target/mod.rs MoveTestTarget (folded into the test_support helper constructors used by the NATS JetStream failed-store tests), and the target/mod.rs StoreBackedTarget (the default send_from_store purge test; the mock deliberately does not override send_from_store or handle_terminal_failure). The forced init failure now uses TargetError::Initialization instead of the old adapter mock's Configuration; the adapter derives its redacted failure summary from the target id alone, so the migrated assertions are unchanged.
Leak guard: testkit unit tests assert the crate manifest still declares default = [] and that test-support = [] stays a pure cfg gate, complementing the compile-level cfg gate that keeps the mock out of production builds.
Part of rustfs/backlog#1846 (cluster 3, step 1).
* test(notify,audit): migrate the Target mocks onto the shared testkit
Retires the hand-written Target mocks in crates/notify and crates/audit in favor of rustfs_targets::testkit::MockTarget: notify's notifier.rs TestTarget/DeferredTestTarget/ClosableTestTarget, lifecycle.rs BlockingInitTarget/RetryInitTarget (rebuilt as observed MockTarget templates cloned by their plugin-descriptor factories, sharing the init signal, close counter, and single-failure init budget across generations), runtime_view.rs TestTarget, runtime_facade.rs TestTarget, and audit's pipeline.rs MockTarget, system.rs TestTarget, registry.rs CloseTestTarget, plus TestTarget and FailingTarget in audit/tests/pipeline_layer_test.rs. Removing the two integration-test mocks also removes their respelled PluginEvent bound (E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned), which the plugin-contract rules require to be spelled only via PluginEvent.
lifecycle.rs ReplayTarget stays bespoke on purpose: its generation tags, mpsc observation channels, gated send_raw delivery, and ObservedQueueStore model the replay pipeline itself and would contort a general-purpose mock. The other bespoke mocks named out of scope in PR-3a (ProgrammedTarget, ClassifyingTarget, the ReloadableTargetTls fakes) are likewise untouched.
New testkit knobs, each defaulted off and unit-tested: with_id (rename a clone while keeping the shared counters, for factory templates), with_first_save_gate (the first save notifies entered and waits on release; several mocks may share one pair), with_health_gate (is_active waits on a release handle after notifying health_started), with_delivery_snapshot (fixed snapshot overriding the store-derived default), with_close_failures (close-failure budget, default TargetError::Storage) with with_close_failure_error to shape the variant (audit's registry test pins TargetError::Unknown), and an always-on is_enabled call counter exposed as enabled_call_count (notifier's generation tests count dispatcher selections through it).
Both crates enable the testkit through a dev-dependency on rustfs-targets with the test-support feature; the feature stays out of default and activates no dependencies, so production builds are unchanged.
Part of rustfs/backlog#1846 (cluster 3, step 2).
fix(test): wait for EC write materialization before staging relocated-pool fixture
An erasure-coded write returns once write-quorum disks commit, so a lagging disk can legally still be missing its xl.meta when the relocated-pool resume test starts staging its fixture by iterating every disk of the owning pool. Under CI load this raced into a NotFound panic in the staging loop. Add a bounded readiness poll that waits for xl.meta on every pool disk before the normalization and staging steps.
Fixes#6703
refactor(s3-client): remove the superseded per-algorithm checksum plumbing
Deletes the write-only RequestMetadata.add_crc pipeline (assigned but never read since the port), the dead MinIO-parity Checksum constructors and CompletePart accessor, and key_capitalized (identical to key). The five hand-rolled x-amz-checksum-* response-header if-lets in the streaming and multipart paths collapse into one checksum_header_value helper, ChecksumMode's inherent to_string becomes a Display impl, and checksum.rs drops its file-wide allow blanket now that the file is lint-clean.
Refs rustfs/backlog#1844 (PR2 of 3).
Mechanical move-only extraction for backlog#1840 PR1+PR4: the site-replication state (load/parse/persist/RMW transaction), repair state machine, peer transport (client cache, DNS resolver, send_peer_* family), retry queue, and the four storage-side hooks move from rustfs/src/admin/handlers/site_replication.rs into the new infra-layer module rustfs/src/site_replication/ ({mod,state,state_lock,identity,transport,retry,repair,hooks}.rs). The admin handler file keeps route registration, all Operation impls, request/response glue, and the in-file test module, and re-exports the moved items so existing paths keep resolving. admin/site_replication_identity.rs and admin/site_replication_state.rs relocate wholesale as identity.rs/state_lock.rs.
Storage access from the moved code goes through a new site_replication consumer module in the root facade (rustfs/src/storage_api.rs), including an s3 shim so the module stays off the direct s3s surface (file count stays at the 215 baseline). The three admin runtime-source wrappers the moved code needs (outbound TLS generation incl. the test atomic, outbound TLS state, runtime port) are reproduced locally; the TLS-generation trio moves out of admin/runtime_sources.rs since site replication was its only consumer. The one non-verbatim rewrite: site_replication_peer_payload inlines encrypt_stream_io in its encrypted branch, which is provably the branch encode_compatible_admin_payload always took for the /minio/admin peer-join wire path.
app/bucket_usecase.rs now imports the three bucket hooks from crate::site_replication, deleting the three app->interface entries from the layer baseline (shrink-only). The peer-client cache test moves with the owner-local SITE_REPLICATION_PEER_CLIENT static into transport.rs (228+1 = 229 tests conserved). New module files are added to the logging-guardrail checked list; the s3_error! line baseline tightens 1620 -> 1619; global-state/config-consumer inventories and ARCHITECTURE.md pointers updated.
Verified: cargo check -p rustfs --all-targets clean; cargo clippy --workspace --all-targets clean; cargo nextest run -p rustfs --lib 3852/3852 passed; make pre-commit green; scripts/check_layer_dependencies.sh green with baseline-only deletions; line-multiset conservation audit over the moved code accounts for every non-verbatim line (visibility bumps, import rewrites, fmt reflow).
Refs rustfs/backlog#1840
ci(pool-test): fix scheduled runs and read env from secrets or vars
workflow_dispatch inputs are empty for schedule events, so the scheduled
pool test built a broken package URL (--version "") and failed preflight.
Fall back to the latest nightly deb (R2) when no version/package_url input
is given, default the thresholds/duration/pools, and default cleanup to
enabled. Also read RUSTFS_API_ENDPOINT / RUSTFS_NODES / RUSTFS_SSH_USER
from secrets first (variables as fallback) so either configuration works.
refactor(ecstore): retire the set_disk lint blankets by making the prelude explicit
backlog#1823 step 1 / backlog#2029 road 2. Removes the last two module-level lint blankets in ecstore: set_disk/mod.rs #![allow(unused_imports)] and #![allow(unused_variables)], restoring both lints for the whole 40K-line subtree, and deletes the register line for the unused_variables blanket in the same diff (the guard from #6155 is a bidirectional exact match).
The unused_imports blanket existed because 14 submodules consumed mod.rs as a glob prelude (use super::* / use super::super::*), and rustc does not track consumption through glob re-exports. Each glob is now an explicit use super::{...} list, keeping mod.rs as the single import hub while making every import lint-checkable. Names consumed only by test or test-util units carry #[cfg(test)] / #[cfg(all(test, feature = "test-util"))] / #[cfg(any(test, feature = "test-util"))] gates matching their consumers; storage-api traits are routed through the storage_api_contracts facade per the architecture guard.
The sweep then deleted the genuinely dead imports the blanket was hiding (chrono::Utc, glob::Pattern, futures::task::AtomicWaker, rustfs_lock LocalLock, AsyncBatchProcessor, rand::Rng, std::future::Future among others in mod.rs, plus stale scoped imports and one empty test module shell across the subtree). One unused_variables finding surfaced: flush_read_version_coalescer_pending's lane_key is read only by the #[cfg(test)] counter block, handled with the cfg(not(test)) let _ pattern established in #6158.
Verification: cargo check zero warnings versus the 9cf276ed2 baseline on five lanes (default lib / --tests / rio-v2 --tests / test-util --tests / test-util,rio-v2 --tests; the --tests lane keeps the same three pre-existing core/pools.rs and store/object.rs dead-code warnings main already has); clippy --lib --tests -D warnings clean with test-util,rio-v2; cargo nextest run 4567 passed; make pre-commit exit 0.
The s3-client ChecksumMode previously duplicated per-algorithm header names, wire names, digest lengths, and checksum-type capability tables in EnumSet-mask matches. ChecksumAlgorithm in rustfs-checksums now owns that metadata behind exhaustive matches (a new variant fails to compile until its metadata is decided), and ChecksumMode delegates through a single algorithm() bridge. Wire behaviour is pinned unchanged by tests on both sides.
Refs rustfs/backlog#1844 (PR1 of 3).
The admin surface had accumulated one near-identical response helper per handler file. This folds the byte-equivalent ones into `rustfs/src/admin/utils.rs` so the wire shape of an admin JSON answer is pinned in one place instead of being re-derived twelve times.
Folded into `crate::admin::utils`:
- `json_response(status, &value)` — 9 local definitions removed: batch_job.rs, kms_backup.rs, oidc.rs, diagnostics.rs (identical signature), object_data_cache.rs and site_replication.rs (hard-coded `StatusCode::OK`, whose call sites now pass `StatusCode::OK` explicitly), ilm_transition.rs (arguments were `(&value, status)` and are swapped at every call site), and kms_key_metadata.rs / kms_key_lifecycle.rs (concrete response types now covered by the generic helper).
- `empty_response(status)` — 2 local definitions removed: site_replication.rs (`Body::empty()`) and table_catalog/mod.rs (`Body::default()`); `Body::empty()` is defined as `Body::default()`, so the two were already the same response.
- `extract_query_params(uri)` — 4 local definitions removed: kms_keys.rs (was `pub(super)`), replication.rs, batch_job.rs, config_admin.rs. All four bodies were behaviourally identical (`form_urlencoded::parse` over `uri.query()`, last-wins on repeated keys, valueless parameters kept as empty strings); they differed only in blank lines. kms_key_lifecycle.rs, which imported the kms_keys copy, now imports the shared one.
Intentionally left alone:
- heal.rs `json_response` — different shape: returns a bare `S3Response` (not `S3Result`) and additionally sets `CONTENT_LENGTH`.
- kms_rekey.rs `json_response` — same divergent shape as heal.rs: bare `S3Response` over already-serialized `Vec<u8>`.
- idp_compat.rs `json_response` — encrypts the payload via `encode_compatible_admin_payload`; it is not a duplicate of the plain JSON helper.
- scanner.rs `json_response` — takes raw `Vec<u8>`, and `ScannerCycleStateResetHandler` genuinely passes a byte literal rather than a serializable value, so the local helper stays.
- oidc.rs `extract_query_param` — singular, returns `Option<String>` for one key, hand-rolls its own splitting via the `urlencoding` crate; a different function, not a variant of the map builder.
Wire behaviour on the success path is byte-identical everywhere: same status, same `Content-Type: application/json` (every local copy spelled the same value, whether via a per-file `JSON_CONTENT_TYPE`/`CONTENT_TYPE_JSON` constant, `HeaderValue::from_static`, or `"application/json".parse()`), same serialized body bytes, and no other header. The only behavioural change is the message text on the serde-serialization-failure arm, which is now uniformly `failed to serialize response: {e}`; that arm is unreachable for these owned response structs and the acceptance criteria pin only status and content type.
No `include_str!` self-grep assertion needed updating: the affected tests in ilm_transition.rs, site_replication.rs, kms_keys.rs, kms_key_metadata.rs, kms_key_lifecycle.rs, object_data_cache.rs, and table_catalog/tests.rs are all bounded by handler `impl Operation` / entry-point markers that sit well after the removed helpers, and none of them assert on a `json_response`, `empty_response`, or `extract_query_params` string.
Tests: `rustfs/src/admin/utils.rs` gains `json_response_carries_status_content_type_and_serialized_body`, `json_response_reports_serialization_failure_as_internal_error`, `empty_response_has_no_body_and_no_headers`, `extract_query_params_decodes_percent_escapes`, and `extract_query_params_keeps_valueless_parameters_and_survives_no_query`. The percent-decoding coverage previously in batch_job's `extract_query_params_decodes_job_id` moves there, and batch_job keeps its own end-to-end coverage as `require_job_id_decodes_and_rejects_missing_and_empty`.
Reference: rustfs/backlog#1829 T6
hector