- common.rs gains an AdminTransport knob (Signed | Awscurl) with admin_execute_at plus three family wrappers: admin_create_user_via, admin_add_canned_policy_via, admin_attach_user_policy_via; the existing admin_create_user now delegates over the Signed transport.
- Deleted the four signed admin request clones in admin_mfa_test, admin_auth_test, reliant/tiering, and inline_fast_path_cluster_test; each keeps a thin local wrapper over common::admin_request so call sites keep their Option<&str> body shape.
- Deduped the notification_webhook signer onto common::signed_request and the webdav_core signer plus its three admin helpers onto the shared _via helpers.
- Consolidated the S3-client-with-credentials builders: admin_auth s3_client_with, existing_object_tag user_client/sts_session_client, bucket_policy_check create_user_client, and the create_user_s3_client copies in group_delete_test and replication_extension_test now delegate to create_s3_client_with_credentials / build_test_s3_config; replication_extension admin_add_canned_policy and admin_attach_policy_to_user route through the _via helpers on the Signed transport.
- The awscurl-gated suites (existing_object_tag_policy, bucket_policy_check, policy/policy_variables) keep going through the external awscurl binary via AdminTransport::Awscurl, preserving their wire behavior.
Part of rustfs/backlog#1846 (cluster 2).
* fix(s3): round-trip null-version delete-marker identity through listing and delete responses
On a versioning-suspended bucket, a null delete marker's identity was lost on the way back to the client at three points (issue #6745): ListObjectVersions advertised the marker's VersionId as the literal nil UUID instead of null; deleting by that id succeeded but the DeleteObjects/DeleteObject response reported the identity as null with no way to correlate it to the request; and the response lacked DeleteMarker/DeleteMarkerVersionId because the marker-ness comparison mixed the client-facing identity (Some(nil)) with the storage identity (None), so the removal also mis-recorded accounting and fired DeleteMarkerCreated semantics on later paths.
- Listing (bucket_usecase, s3_api/bucket, build_list_versions_next_marker) now maps the synthesized nil UUID to the literal null everywhere it reaches the wire, and VersionMarker::parse folds a nil-UUID marker from older listings into VersionMarker::Null so pagination resumes correctly.
- delete_objects normalizes both sides of the marker-ness comparison via delete_file_info_version_id (matching the adjacent explicit_delete_marker admission check) and reports DeleteMarkerVersionId as null for an explicit null-marker removal.
- resolve_delete_version_state reports delete_marker for an explicit-version delete whose target is a delete marker even when the bucket is versioning-suspended, fixing x-amz-delete-marker on the single-object path.
- The DeleteObjects response entry echoes the version identity the request addressed for marker removals, marker-removal accounting no longer records a marker creation, and notification events fire DeleteMarkerCreated only for actual marker creation.
Fixes#6745
* fix(s3): keep null-marker removal write shape undeleted and report marker semantics response-side
The first cut marked the storage delete request deleted for a null-marker removal, which FileMeta::delete_version interprets as the suspended-bucket delete-mints-a-marker write and re-creates the marker just removed. Carry marker-ness to responses via explicit_delete_removed_marker (single path) and a response-only branch flag (batch path) instead, keeping every storage write shape byte-identical to the pre-fix behavior. Adds an embedded end-to-end regression test covering the full issue #6745 round trip.
WarmBackendS3::new already rejects loopback, private, link-local, and
cloud metadata-service endpoints via validate_outbound_url, but the
Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS, and GCS warm
backend constructors built their transition clients directly from
conf.endpoint without the same check.
The endpoint comes from the AddTier admin API, gated only by
SetTierAction, which can be a narrower IAM grant than root. Any
principal holding it could point one of these eight tier types at an
internal address (loopback, RFC1918, link-local, or a cloud metadata
IP) and have the server issue authenticated outbound requests to it, a
server-side SSRF vector that the S3 and Wasabi tier types were already
closed against.
Apply the same validate_outbound_url check at construction time for
all eight providers, before any credentials or network client are
built, mirroring the existing WarmBackendS3 pattern. GCS keeps its
default-endpoint behavior when conf.endpoint is empty and only
validates an explicitly configured endpoint.
Add a regression test per provider asserting that a loopback endpoint
is rejected before any backend/network setup, matching the existing
WarmBackendS3 coverage.
Update the error(format!) ratchet baseline: these are one-shot admin
tier-configuration validation errors returned once per AddTier call,
not per-disk I/O errors that flow through reduce_errs quorum
aggregation (backlog#1845), so the new ::other(format!) call sites do
not introduce a quorum-bucketing hazard. They mirror the pre-existing,
already-baselined warm_backend_s3.rs call site.
TransitionClient::new() in crates/s3-client/src/transition_api.rs computes
trailing_header_support = opts.trailing_headers && override_signer_type == SignatureV4,
but override_signer_type is hardcoded to SignatureDefault at construction
and never mutated afterwards, so the expression is always false regardless
of opts.trailing_headers. The resulting field also has no live reader: its
only reference is inside PutObjectOptions::validate() in
crates/s3-client/src/api_put_object.rs, which is itself
#[allow(dead_code, reason = "MinIO-parity ... no caller in this port")],
and even there the reference to trailing_header_support is commented out.
So trailing_headers: true in the seven warm_backend_*.rs constructors has
never had any effect on request signing or chunked/trailing-header
behavior (stream_sha256 signing is gated separately by
metadata.stream_sha256 && !self.secure). Remove the misleading dead
configuration from the seven provider constructors so it doesn't look
like intentional, load-bearing behavior to future readers.
Found during adversarial self-check while implementing rustfs/backlog#2040 (out of that issue's scope).
* refactor(ecstore,rustfs): reuse canonical starts_with_ignore_ascii_case
`crates/utils/src/http/metadata_compat.rs` owns the internal metadata key helpers, including `starts_with_ignore_ascii_case`. Two files carried their own byte-identical copies of that predicate: `SetDisks::starts_with_ignore_ascii_case` in ecstore and a free function in the S3 options layer. Both drive internal metadata key classification (`internal_metadata_suffix` and quorum hashing on one side, `should_skip_object_metadata_key` and `is_reserved_user_metadata_key` on the other), so keeping three implementations of one predicate is an avoidable drift risk on a path that decides whether an internal key is treated as user metadata.
Delete both local copies and call the canonical implementation. Every prefix used at these call sites is an ASCII constant or literal, where the canonical byte-slice comparison and the removed `str::get(..n)` form are equivalent; that equivalence was checked differentially over 4.6M (key, prefix) pairs, including keys with multi-byte characters straddling the prefix boundary. No other logic in `internal_metadata_suffix` or `should_skip_object_metadata_key` changed.
Add regression tests on both sides pinning the two properties the switch depends on: internal prefixes match case-insensitively (a mixed-case `X-RustFS-Internal-*` key stays internal), and keys shorter than a prefix never match (they stay ordinary user metadata).
Refs rustfs/backlog#2051
* fix(rustfs): avoid typos-checker false positive in prefix-length test
The test literal "x-rustfs-encryptio" (a deliberate truncation of the
x-rustfs-encryption- prefix, used to assert that a key shorter than every
internal prefix falls through to user metadata) reads as a likely typo of
"encryption" to the repo's typos CI check. Derive it from
RUSTFS_ENCRYPTION_PREFIX via slicing instead of a hand-typed literal, which
both satisfies the linter and ties the truncation to the real constant
instead of a copy-typed guess.
Refs rustfs/backlog#2051
chacha20 0.10.1 was yanked on crates.io today, which fails the Cargo Deny gate (error[yanked]) on every branch. cargo update -p chacha20 to 0.10.2; no API change, all dependents are semver-compatible.
Verification: cargo check -p rustfs-crypto; the Cargo Deny job on this PR is the authoritative gate.
`crates/lifecycle/src/tagging.rs` carried a byte-identical copy of the `form_urlencoded` tag decoder already owned by `rustfs-replication`, plus a duplicate of its test. Since `crates/lifecycle` already depends on `rustfs-replication`, replace the copy with a `pub(crate) use` re-export: no new crate edge, one parser, and no second implementation to drift from the replication contract. The `rule.rs` call site is unchanged.
Also drop `crates/ecstore/src/bucket/lifecycle/tagging_boundary.rs`, a migration-era boundary shim with zero call sites in the tree.
The import_bucket_metadata handler carried eight match arms whose bodies were
byte-identical apart from the type a payload is validated against and the pair
of BucketMetadata fields it lands in, so every arm repeated the same warn! call
and the same metadata lookup. Fold them into one or-pattern arm backed by
apply_imported_bucket_config, where a single conf_name match owns both the
validated type and the destination field pair and can no longer drift apart.
Validation still runs before the metadata lookup, the warn! event, fields, and
label are unchanged, and the BUCKET_POLICY_CONFIG and BUCKET_QUOTA_CONFIG_FILE
arms keep their own handling. Regression tests drive the full mapping table:
each config file's payload lands only in the field it owns, an unparsable
payload leaves the field untouched, and a rejected entry does not stop the
remaining ones from being imported.
Refs rustfs/backlog#2052
crates/s3-client/src/utils.rs carried a verbatim copy of the header
classification tables and predicates owned by
crates/utils/src/http/headers.rs: SUPPORTED_HEADERS (same 11 keys),
SUPPORTED_QUERY_VALUES (same 9 keys), and is_standard_header /
is_storageclass_header / is_amz_header / is_rustfs_header /
is_minio_header with byte-identical bodies. The duplication was already
half-resolved and inconsistent — the local is_amz_header called
rustfs_utils::http::is_sse_header while consulting its own tables — and
s3-client already depends on rustfs-utils with the "full" feature, so
reusing the canonical owner adds no crate edge.
The sole caller, PutObjectOptions::header(), now imports the five
predicates from rustfs_utils::http. Semantics are unchanged: both sides
normalize with to_lowercase(), return false for unknown keys, and the
storage-class constants are the same string ("x-amz-storage-class" from
s3s::header::X_AMZ_STORAGE_CLASS vs rustfs_utils AMZ_STORAGE_CLASS), so
the set of user-metadata headers passed through verbatim rather than
prefixed with x-amz-meta- is identical.
SUPPORTED_QUERY_VALUES is deleted outright: s3-client had no reader for
it (utils consumes its own copy via is_standard_query_value). The
base64_encode/base64_decode helpers and their rustfs/rustfs#4811
regression test stay untouched, and lazy_static remains a dependency
because crates/s3-client/src/constants.rs still uses it.
Refs rustfs/backlog#2050
ForegroundPressure had two definitions with byte-identical pressure computation: one in ecstore data-movement backpressure and one in the heal manager queue. That duplication is a violation of the ARCHITECTURE.md invariant that each type has exactly one definition, and it means any future change to the utilization math has to land twice.
Add the canonical `ForegroundPressure` and a `foreground_pressure(snapshot, read_threshold_pct, write_threshold_pct)` function to `crates/concurrency/src/workload.rs`, which already owns `WorkloadClass`, `AdmissionState`, and the admission snapshot contract. Both existing consumers already depend on `rustfs-concurrency`, so no crate edge is added.
The filter_map pipeline is transferred verbatim, preserving all five boundary behaviors (zero threshold, zero limit, missing entry, missing active count, and the `Saturated` full-utilization special case), the mul-before-div percentage normalization, the `>=` threshold comparison, and the read-then-write ordering that makes `max_by_key` break utilization ties toward the write class. The enable switch is deliberately left out: ecstore gates on `config.enabled` while heal gates on `mainline_throttle_enable` plus a both-thresholds-zero check, so each call site keeps its own condition.
This is the expand step only. The ecstore and heal copies are untouched and are removed by the follow-up migrate task.
Refs rustfs/backlog#2047
The seven S3-compatible warm backend providers (Aliyun, Azure, Huaweicloud, Tencent, MinIO, R2, RustFS) each carry a byte-identical copy of the same statically-credentialed TransitionClient construction and of the same optimal_part_size helper. Add both to the module that already owns the WarmBackend trait and WarmBackendS3, so the per-provider migrate step can drop its duplicate without redesigning anything.
bucket_lookup is a parameter rather than a constant because the providers split into two families: Aliyun, Azure, Huaweicloud, and Tencent pin BucketLookupDNS, while MinIO, R2, and RustFS leave it at the BucketLookupAuto default. Hardcoding either value would silently change bucket addressing for the other family during the migrate step.
Error texts, validation order, prefix and host/port normalization are reproduced exactly from the Aliyun/MinIO family. No provider file is touched and no production caller exists yet, so the new unit tests are the first callers.
Refs rustfs/backlog#2040
The audit and notify subsystems each declare their own default KVS table for the same nine delivery targets. For amqp, nats, pulsar, postgres and kafka the two declarations are byte-identical; for redis and mysql they differ only in a single default literal (the pub/sub channel and the destination table). Keeping two copies means every default or key-order change has to be made twice, and a missed edit silently changes what admin config reports for one subsystem only.
Add `config::target_defaults` with one constructor per shared table, taking the diverging literal as a parameter for redis and mysql, plus a small `kv` helper that replaces the repeated `KV { .. }` literals. Key order is reproduced exactly because it drives the order the admin API lists keys in. Unit tests pin the full ordered key/value/hidden_if_empty triple of every table against hard-coded literals, and cover both the audit and the notify literal for the two parameterized tables.
Webhook and mqtt are deliberately left out: audit's webhook table carries extra batching and retry keys, both webhook tables disagree on key order and on the auth-token hidden_if_empty flag, and mqtt disagrees on qos, keep-alive interval and reconnect interval. Those are real behavioral forks, not duplication, so they stay declared in place.
This is the expand step only. Nothing calls the new module yet, so audit.rs and notify.rs are untouched and no default changes; the constructors carry an item-level allow(dead_code) until the migrate step points both files at them.
Refs rustfs/backlog#2044
* test(heal): relative disk target and fail fast on terminal-but-short
The absolute 40 GiB heal target was calibrated to the background scanner
(auto-heal), which is now disabled for determinism; with only the explicit
heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the
success criterion relative: the outage node must reach at least 90% of the
least-used surviving node (absolute HEAL_TARGET_GB floor optional, default
0 = relative only).
Also fail fast when the heal task reaches a terminal success but the disk
target is not met (previously the monitor kept polling until timeout), and
drop the misleading 'progress absent' warning on the final (cleaned) task
response — mid-run progress is reported correctly.
Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB,
vm002=40GB (target 36GB), test PASSED.
* test(heal): gate success on server verdict + data read-back, drop disk GB gate
The per-node disk-usage target (40 GiB / 90% of survivors) is not a
code-level invariant: EC distributes different shards per node, so the
final GB per node depends on the layout, not on heal correctness. Gate the
test on what the server actually verifies:
- Heal task terminal success (finished/completed) with objectsFailed == 0
(the server's per-object scan/repair verdict).
- S3 read-back verification: list the test bucket and GET a sample of
objects, requiring HTTP 200 for every read (end-to-end proof the data is
still reconstructable after repair). The GET uses a discard mode so
binary bodies are not captured (no null-byte warnings / SIGPIPE).
Per-node disk usage stays in the output as observability (with a warning if
the outage node gained no usage), not as the pass/fail gate. Removes the
heal_target_gb input and the relative-target logic.
Validated live: heal summary=finished, 0 failed, 20/20 objects read back,
vm002_used=40GB, PASS.
Add a default-on, size-aware foreground PUT admission policy so large or
unknown-size PutObject requests are backpressured before body ingest and
erasure/RPC fan-out. Preserve the explicit strict gate semantics, including
limit=0 as an opt-out, and keep small PUTs on the legacy fast path.
Closesrustfs/backlog#2038
Co-authored-by: heihutu <heihutu@gmail.com>
* Revert "perf(ecstore): use AHashMap for FileInfo metadata fields (#6738)"
This reverts commit 13a2ae212e.
* fix(filemeta): restore standard HashMap metadata (#6742)
Remove the direct ahash dependency added for FileInfo metadata and revert the affected filemeta/ecstore call sites back to std::collections::HashMap.
Co-authored-by: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
A fixed /tmp path (log file, final heal status, warp log) can be owned by
another user on the shared runner (e.g. a previous root run), which made the
github-runner user fail: tee could not append the test log, the final heal
status write killed step 6 with EACCES, and upload-artifact could not read
stale root-owned warp logs. The heal scenario itself had passed
(summary=finished, vm002 reached the target) before the status-save died.
- Log files fall back to a unique mktemp path when the configured path is not
writable (heal + pool scripts).
- The final heal status is written to a mktemp file (best effort).
- Workflow artifact uploads use globs for the fallback names.
* test(heal): add node-outage heal E2E script and workflow
RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.
Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.
The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.
* ci(pool-test): chain heal regression after the pool test
The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:
1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
(if: always()): a pool failure makes the run red but does not block the
heal regression. Runs the heal script (reset -> install/start 3x4 ->
write/outage -> heal -> verify -> reset).
* test(heal): address review — camelCase progress, fail-closed, workflow hygiene
- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
objectsFailed/progressPercentage); read them with a snake_case fallback and
distinguish null (absent) progress from zero, logging null as evidence
(rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
the write target must be reached, and an unobserved outage or incomplete
write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
an exact successful nightly conclusion; checkout is pinned to the triggering
SHA; comma-separated step args are quoted (actionlint SC2054).
Keep /health/ready from riding the generic internode lock RPC and channel keepalive budgets when a peer host is unreachable. Add a health-specific lock online timeout, route ping failures through the existing remote lock RPC eviction path, cache the static ping payload for readiness fan-out, and cover hanging cached channels with focused tests.
Refs rustfs/backlog#2033
Refs rustfs/rustfs#6286
Co-authored-by: heihutu <heihutu@gmail.com>
Config::new() and the external decode path read the process-global
DEFAULT_KVS OnceLock at call time, and config::tests in the same test
binary register it via crate::config::init() mid-run. Several com.rs
tests asserted on unregistered state (heal section absence, equality
with a later Config::new()), so they could flip depending on thread
scheduling under cargo test -p rustfs-ecstore --lib config::.
Assert on the semantic heal diff instead of section presence, normalize
compared configs with a single DEFAULT_KVS snapshot taken after both
sides exist, and compare the snapshot transaction test against the
persisted baseline bytes.
* test(pool): fix warp log path and retry rebalance start
- warp writes now use a unique mktemp log file instead of a fixed
/tmp/rustfs-warp.log: the runner user could not write the stale
root-owned file, which made the background warp process die instantly
(warp never ran). The workflow uploads /tmp/rustfs-warp.*.log.
- rebalance start is retried (6x, 20s apart): nightly builds gate
rebalance activation on a live cross-pool fence fleet capability proof
that takes ~10-20s to re-establish after a pool joins. Verified live:
attempt 1 fails with 500 'pool activation requires a live fleet
capability proof', attempt 2 succeeds.
* test(pool): annotate known server-side issues in failure output
When a node fails to start, grab the rustfs journal tail and match known
server-side error signatures (e.g. the fleet capability proof cold-start
regression, rustfs/backlog#2031), printing a hint with the tracking issue.
Also annotate the rebalance-start retry exhaustion and the rc.3 decommission
metacache-listing failure with actionable guidance.
* fix(ecstore): defer rebalance activation without fleet proof
---------
Co-authored-by: 马登山 <cxymds@qq.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Preview tags stay as the traceability record for the validated commit, but their GitHub Releases are internal validation state and should not accumulate on the Releases page next to real deliverables.
Add a cleanup-preview-releases job that runs after publish-release succeeds for a release or prerelease tag and deletes every Release whose tag is exactly <target>-preview.<digits>. The tags themselves are kept: the job never passes --cleanup-tag. Tag matching uses jq string operations rather than a regex over the version, so dots in the version cannot widen the match, and the release listing is fetched before filtering so an API failure aborts the job instead of looking like there was nothing to clean up.
Extend the preview release workflow guard with the job condition, the delete invocation, both tag-matching filters, and an absent check for --cleanup-tag.
* test(ecstore): decouple server config snapshot test from global defaults
The final assertion of server_config_snapshot_serializes_read_modify_write_transactions
compared the second snapshot against a fresh Config::new(). Config::new()
reads the process-global DEFAULT_KVS OnceLock, which a sibling test in the
same process can register mid-run (crate::config::init()), so the in-process
run 'cargo test -p rustfs-ecstore --lib config::' failed while nextest's
process-per-test isolation hid the coupling. Assert on the snapshot's raw
bytes against the baseline blob instead, which is deterministic and matches
the invariant under test: the second transaction observes the store unchanged
by the first.
* test: deflake presigned tamper helper and relocated-pool resume staging
tamper_signature only remapped '0' and 'a', so a signature containing
neither (about 1 in 5000) left the URI unchanged and tripped the helper's
own guard assert in CI. Complement every hex digit (15 - v) instead: the
map has no fixed point, so the tamper always changes the value while
keeping length and hex shape.
execute_get_object_resumes_from_relocated_pool_without_splicing_body
staged the relocation by reading xl.meta from every source-pool disk, but
a write-quorum commit legitimately leaves a lagging minority disk without
the object directory (#6701) — the test already tolerates that gap when
normalizing the upload pool, and CI suite IO load hit the same gap in the
staging loop. Skip sourceless disks, carry the staged metadata path
explicitly, and assert a write-quorum majority was staged.
* refactor(site-replication): move business tests next to the service module
backlog#1840 PR5: 79 business-logic tests (plus 12 helpers, 6 of them small fixtures kept on both sides) move from the admin handler file's test module into rustfs/src/site_replication/tests.rs, next to the code they exercise: peer connection/TLS/DNS/egress validation, the peer client cache and payload wire contract, retry-queue classification/settlement/escalation/backoff, the repair state machine, bootstrap-plan construction, lifecycle expiry subsetting, bucket-target reconciliation, endpoint/identity normalization, and state serialization. The 149 tests that exercise the admin handlers, apply/reconcile paths, status/resync builders, and the four include_str! tripwires stay in rustfs/src/admin/handlers/site_replication.rs with their subjects (229 total conserved: 149 + 79 + 1).
The issue's PR5 also called for converting the source-order tripwire at the old file's line 11339 into a behavior test; both adversarial review passes re-derived all four tripwires against the shrunken file and found them non-vacuous and byte-identical in the regions they guard (the handler bodies, which did not move), so they stay as source-text assertions.
Supporting changes: the root facade's site_replication consumer gains cfg(test) re-exports (endpoint types, merge_incoming_replication_config, five lifecycle DTO types) so the relocated tests stay off the direct s3s/admin surfaces — including rewriting the one inline crate::admin BucketMetadata path a moved test carried over (review finding); tests.rs joins the logging-guardrail checked list; the embedded-secrets guard comment follows the validate_peer_connection_inner fixtures to their new file.
Verified: cargo check -p rustfs --all-targets clean; cargo nextest run -p rustfs --lib 3856/3856 passed; relocated tests run under site_replication::tests::; make pre-commit green including the s3s footprint ratchet; logging and embedded-secrets guards green.
Refs rustfs/backlog#1840
* style(site-replication): apply rustfmt import ordering
* test(targets): ship a builder MockTarget testkit and retire the in-crate Target mocks
Adds crates/targets/src/testkit.rs with a builder-style MockTarget implementing Target<E> for every E: PluginEvent, with orthogonal off-by-default knobs: disabled/active override, health delay plus health-started signal plus a drop-guard counter proving a cancelled probe future was dropped, an init failure budget (usize::MAX = always fail) plus blocking init plus an init counter, a close counter/signal/semaphore gate with a runtime block toggle, a save counter plus save failure budget, caller-supplied store and failed-store handles, and a shared final-failure counter. Clones and clone_dyn share all counters, so an observer clone keeps watching a target after it is boxed into a runtime.
The module is gated as #[cfg(any(test, feature = "test-support"))]: in-crate unit tests get it via cfg(test), and downstream test suites opt in through the new off-by-default test-support cargo feature (test-support = [], activating no dependencies).
Migrates the five in-crate duplicate mocks onto it: the plugin.rs registry-factory TestTarget, the runtime/adapter.rs lifecycle TestTarget (init/close/store knobs), the runtime/mod.rs TestTarget plus HealthDropGuard (close gating and health-probe tests), the target/mod.rs MoveTestTarget (folded into the test_support helper constructors used by the NATS JetStream failed-store tests), and the target/mod.rs StoreBackedTarget (the default send_from_store purge test; the mock deliberately does not override send_from_store or handle_terminal_failure). The forced init failure now uses TargetError::Initialization instead of the old adapter mock's Configuration; the adapter derives its redacted failure summary from the target id alone, so the migrated assertions are unchanged.
Leak guard: testkit unit tests assert the crate manifest still declares default = [] and that test-support = [] stays a pure cfg gate, complementing the compile-level cfg gate that keeps the mock out of production builds.
Part of rustfs/backlog#1846 (cluster 3, step 1).
* test(notify,audit): migrate the Target mocks onto the shared testkit
Retires the hand-written Target mocks in crates/notify and crates/audit in favor of rustfs_targets::testkit::MockTarget: notify's notifier.rs TestTarget/DeferredTestTarget/ClosableTestTarget, lifecycle.rs BlockingInitTarget/RetryInitTarget (rebuilt as observed MockTarget templates cloned by their plugin-descriptor factories, sharing the init signal, close counter, and single-failure init budget across generations), runtime_view.rs TestTarget, runtime_facade.rs TestTarget, and audit's pipeline.rs MockTarget, system.rs TestTarget, registry.rs CloseTestTarget, plus TestTarget and FailingTarget in audit/tests/pipeline_layer_test.rs. Removing the two integration-test mocks also removes their respelled PluginEvent bound (E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned), which the plugin-contract rules require to be spelled only via PluginEvent.
lifecycle.rs ReplayTarget stays bespoke on purpose: its generation tags, mpsc observation channels, gated send_raw delivery, and ObservedQueueStore model the replay pipeline itself and would contort a general-purpose mock. The other bespoke mocks named out of scope in PR-3a (ProgrammedTarget, ClassifyingTarget, the ReloadableTargetTls fakes) are likewise untouched.
New testkit knobs, each defaulted off and unit-tested: with_id (rename a clone while keeping the shared counters, for factory templates), with_first_save_gate (the first save notifies entered and waits on release; several mocks may share one pair), with_health_gate (is_active waits on a release handle after notifying health_started), with_delivery_snapshot (fixed snapshot overriding the store-derived default), with_close_failures (close-failure budget, default TargetError::Storage) with with_close_failure_error to shape the variant (audit's registry test pins TargetError::Unknown), and an always-on is_enabled call counter exposed as enabled_call_count (notifier's generation tests count dispatcher selections through it).
Both crates enable the testkit through a dev-dependency on rustfs-targets with the test-support feature; the feature stays out of default and activates no dependencies, so production builds are unchanged.
Part of rustfs/backlog#1846 (cluster 3, step 2).
fix(test): wait for EC write materialization before staging relocated-pool fixture
An erasure-coded write returns once write-quorum disks commit, so a lagging disk can legally still be missing its xl.meta when the relocated-pool resume test starts staging its fixture by iterating every disk of the owning pool. Under CI load this raced into a NotFound panic in the staging loop. Add a bounded readiness poll that waits for xl.meta on every pool disk before the normalization and staging steps.
Fixes#6703
Halil Teyfik