The admin surface had accumulated one near-identical response helper per handler file. This folds the byte-equivalent ones into `rustfs/src/admin/utils.rs` so the wire shape of an admin JSON answer is pinned in one place instead of being re-derived twelve times.
Folded into `crate::admin::utils`:
- `json_response(status, &value)` — 9 local definitions removed: batch_job.rs, kms_backup.rs, oidc.rs, diagnostics.rs (identical signature), object_data_cache.rs and site_replication.rs (hard-coded `StatusCode::OK`, whose call sites now pass `StatusCode::OK` explicitly), ilm_transition.rs (arguments were `(&value, status)` and are swapped at every call site), and kms_key_metadata.rs / kms_key_lifecycle.rs (concrete response types now covered by the generic helper).
- `empty_response(status)` — 2 local definitions removed: site_replication.rs (`Body::empty()`) and table_catalog/mod.rs (`Body::default()`); `Body::empty()` is defined as `Body::default()`, so the two were already the same response.
- `extract_query_params(uri)` — 4 local definitions removed: kms_keys.rs (was `pub(super)`), replication.rs, batch_job.rs, config_admin.rs. All four bodies were behaviourally identical (`form_urlencoded::parse` over `uri.query()`, last-wins on repeated keys, valueless parameters kept as empty strings); they differed only in blank lines. kms_key_lifecycle.rs, which imported the kms_keys copy, now imports the shared one.
Intentionally left alone:
- heal.rs `json_response` — different shape: returns a bare `S3Response` (not `S3Result`) and additionally sets `CONTENT_LENGTH`.
- kms_rekey.rs `json_response` — same divergent shape as heal.rs: bare `S3Response` over already-serialized `Vec<u8>`.
- idp_compat.rs `json_response` — encrypts the payload via `encode_compatible_admin_payload`; it is not a duplicate of the plain JSON helper.
- scanner.rs `json_response` — takes raw `Vec<u8>`, and `ScannerCycleStateResetHandler` genuinely passes a byte literal rather than a serializable value, so the local helper stays.
- oidc.rs `extract_query_param` — singular, returns `Option<String>` for one key, hand-rolls its own splitting via the `urlencoding` crate; a different function, not a variant of the map builder.
Wire behaviour on the success path is byte-identical everywhere: same status, same `Content-Type: application/json` (every local copy spelled the same value, whether via a per-file `JSON_CONTENT_TYPE`/`CONTENT_TYPE_JSON` constant, `HeaderValue::from_static`, or `"application/json".parse()`), same serialized body bytes, and no other header. The only behavioural change is the message text on the serde-serialization-failure arm, which is now uniformly `failed to serialize response: {e}`; that arm is unreachable for these owned response structs and the acceptance criteria pin only status and content type.
No `include_str!` self-grep assertion needed updating: the affected tests in ilm_transition.rs, site_replication.rs, kms_keys.rs, kms_key_metadata.rs, kms_key_lifecycle.rs, object_data_cache.rs, and table_catalog/tests.rs are all bounded by handler `impl Operation` / entry-point markers that sit well after the removed helpers, and none of them assert on a `json_response`, `empty_response`, or `extract_query_params` string.
Tests: `rustfs/src/admin/utils.rs` gains `json_response_carries_status_content_type_and_serialized_body`, `json_response_reports_serialization_failure_as_internal_error`, `empty_response_has_no_body_and_no_headers`, `extract_query_params_decodes_percent_escapes`, and `extract_query_params_keeps_valueless_parameters_and_survives_no_query`. The percent-decoding coverage previously in batch_job's `extract_query_params_decodes_job_id` moves there, and batch_job keeps its own end-to-end coverage as `require_job_id_decodes_and_rejects_missing_and_empty`.
Reference: rustfs/backlog#1829 T6
Remove unused direct dependency declarations found by cargo-shear and delete the unlinked ecstore mimalloc diagnostics file.
Keep feature-forwarding dependencies explicit with package-local cargo-shear ignores so hotpath feature propagation remains intact.
Co-authored-by: heihutu <heihutu@gmail.com>
Every consumer now imports rustfs-heal-contracts / rustfs-scanner-contracts
directly and rg 'rustfs_common::(metrics|heal_channel|last_minute)' reports
zero hits, so the backlog#1843 re-export shims and the transitional
rustfs-common -> contracts dependency edges can go. rustfs-common no longer
recompiles on scanner/heal type changes. Doc references to the moved files
follow the new paths.
* refactor(ecstore): drop the client shim, import rustfs-s3-client directly
Completes the migration window opened by the rustfs-s3-client extraction (rustfs/backlog#1842 PR3): every consumer now imports the client crate directly and the crate::client shim is deleted.
- All in-crate crate::client:: paths (tier warm backends, tier core, lifecycle tier_sweeper, replication storage boundary, set_disk) now import rustfs_s3_client::* directly; crates/ecstore/src/client/mod.rs and the lib.rs mod client declaration are gone.
- The two server-side modules historically misfiled under client/ move to their real homes: object_api_utils.rs to crates/ecstore/src/object_api/ (it builds engine-side object readers/writers), and object_handlers_common.rs to crates/ecstore/src/bucket/lifecycle/ (it is the lifecycle noncurrent-version cleanup helper). The latter now routes its replication calls through the lifecycle replication_sink boundary (schedule_delete wrapper and the sink's ReplicationObjectBridge re-export), as the lifecycle guard requires.
- The ecstore public facade drops api::client: object_api_utils is exposed as api::object_api_utils, and the rustfs crate takes admin_handler_utils (AdminError) from rustfs-s3-client directly (new dependency).
- Guard updates: the migration guard no longer pins mod client in ecstore's lib.rs or the admin_handler_utils facade module (it pins the new api::object_api_utils facade instead), and the module-lint register follows object_api_utils.rs to its new path.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs; cargo fmt --all; tier/transition/lifecycle-focused nextest (626 passed) and the decommission/rebalance/heal families in a filtered run (603 passed; the full-suite parallel run only fails on this machine's known decommission/rebalance baseline flakes, which pass in filtered reruns and fail identically on pristine origin/main); layer/migration/s3s/logging/error-format/doc-path guard scripts all pass.
* docs(architecture): record the S3 client extraction and reword invariant 4 (#6669)
Closes the documentation step of rustfs/backlog#1842. ARCHITECTURE.md invariant 4 now states the serving-vs-consuming distinction the adversarial ruling asked for: ecstore must not serve HTTP/S3 wire types, while consuming remote S3 endpoints is a legitimate engine capability that lives in the extracted rustfs-s3-client crate. The violation note is updated from the pre-extraction snapshot (58 files, embedded client) to the current ratcheted state (shrink-only S3S_ECSTORE_FILES_BASELINE in scripts/check_s3s_footprint.sh, object_lock converted first), and the crate map gains s3-client. ecstore-module-split-plan.md gets the client-directory entry the plan was missing: a Current Shape row and a completed-extraction section describing the pure-move + shim + direct-import sequence and the re-homing of the two misfiled server-side modules.
* refactor(rustfs): carve app/object out of object_usecase.rs — shared, extract, test_support children (backlog#1841 step 1)
Mechanical move-only split of rustfs/src/app/object_usecase.rs (19.7K lines). The file body moves to rustfs/src/app/object/mod.rs, and the first self-contained slices move into children: shared.rs (cross-cutting helpers: quota admission, response checksum injection, object-lock write validation, table-catalog mutation guard, deadlock request guard, proxy passthrough utilities), extract.rs (snowball auto-extract path incl. tar/pax helpers and execute_put_object_extract), and cfg(test) test_support.rs for cross-module test scaffolding. object_usecase.rs stays as a thin pub use facade so every existing crate::app::object_usecase:: path keeps working.
No behavior change: items move verbatim; the only source edits are visibility widenings required by the new module boundaries (private -> pub(super); pub(super) -> pub(crate) for the three helpers multipart_usecase and the app gating tests import). Guard scripts that pinned rustfs/src/app/object_usecase.rs now scan the rustfs/src/app/object tree, and the table_catalog source-text guard test concatenates the split files.
* refactor(rustfs): move the GetObject read path into app/object/get.rs (backlog#1841 step 2)
Move-only continuation of the object_usecase split: cold-fill orchestration, disk-permit admission, streaming readers and resume control, stream-buffer tuning, execute_get_object / execute_get_object_attributes, the GET replication proxy helpers, and their unit tests move from app/object/mod.rs into app/object/get.rs. Items keep their original text; cross-module call sites rely on the visibility widenings introduced in step 1.
* refactor(rustfs): move the PutObject and CopyObject paths into app/object (backlog#1841 step 3)
Move-only continuation: put.rs takes the PUT body admission and timeout readers, zero-copy and eager-commit machinery, execute_put_object, and the PUT unit tests; copy.rs takes the copy namespace/lifecycle lock helpers and execute_copy_object with its tests. Two source edits beyond visibility widenings: PutObjectChecksums fields become pub(super) (read by shared::apply_trailing_checksums across the new module boundary) and one relative super::storage_api call in the copy path becomes crate::app::storage_api since super now resolves to app::object. The table_catalog source-text guard concatenates the new files.
* refactor(rustfs): finish the object_usecase split — delete, head, restore modules (backlog#1841 step 4)
Move-only completion: delete.rs takes the delete helpers, cfg(test) delete hooks, and execute_delete_object/execute_delete_objects; head.rs takes execute_head_object with the HEAD replication proxy helpers; restore.rs takes execute_restore_object. app/object/mod.rs is now just the shared import prelude, module wiring, and the DefaultObjectUsecase struct with its constructors, accessors, and the execute_select_object_content delegation; the emptied tests module is gone. The delete re-export glob is cfg(test)-gated because its only cross-module consumers are the delete test hooks.
The table_catalog source-text guard now isolates the delete entrypoints from app/object/delete.rs, and doc/comment references that pointed at rustfs/src/app/object_usecase.rs internals now point at the per-operation modules.
The object_lock module evaluated WORM state through s3s wire DTOs (ObjectLockRetention, ObjectLockLegalHold, DefaultRetention, Date) and s3s header constants, keeping the storage engine coupled to the serving protocol (rustfs/backlog#1842, ARCHITECTURE.md invariant 4). This PR gives the module its own storage-level vocabulary and pushes the DTO conversions to the boundaries that already speak s3s.
New crates/ecstore/src/bucket/object_lock/types.rs defines RetentionMode, LegalHoldStatus, ObjectRetention, ObjectLegalHold, and DefaultRetention with no s3s dependency. objectlock.rs parses persisted metadata into these types using the rustfs-utils lowercase header constants (the same literal keys as before, pinned by the existing g-key-002 test). objectlock_sys.rs evaluates retention/legal-hold/default-retention from them; the fail-closed error messages and decision logic are unchanged line for line where possible.
Boundary conversions:
- bucket/metadata_sys.rs gains default_retention_from_object_lock_config, converting the persisted s3s configuration into the storage-level DefaultRetention; a rule without a usable GOVERNANCE/COMPLIANCE mode converts to None exactly like the evaluation code always ignored it, and days/years pass through so an invalid period still fails closed at evaluation time.
- check_object_lock_for_deletion_with_config becomes check_object_lock_for_deletion_with_default_retention (it only ever read the default retention); the lifecycle object_lock_boundary keeps the old s3s-typed signature and converts.
- The ObjectLockApi / ObjectLockStatusExt trait impls for the s3s DTOs move next to the persisted configuration owner in bucket/metadata.rs; the traits stay in object_lock/mod.rs.
- check_retention_for_modification now takes Option<RetentionMode>. The serving-layer wrappers (rustfs storage_api, set_disk options path) convert the request string with the new RetentionMode::parse_exact, which accepts only the canonical spelling — preserving the historical literal comparison where a non-canonical requested mode reads as a mode change and stays blocked.
- rustfs app-layer wrappers return the storage types; the replication-overwrite gate in object_usecase.rs uses the typed API (legal_hold.is_on(), RetentionMode::Compliance).
Ratchet: the ecstore-scoped s3s counter drops 42 -> 39 and the repo-wide file counter 211 -> 208 in scripts/check_s3s_footprint.sh.
Verification: cargo check -p rustfs-ecstore --all-targets and -p rustfs (lib+bins); cargo clippy -p rustfs-ecstore --all-targets and -p rustfs --lib --bins (clean); cargo nextest run -p rustfs-ecstore --no-fail-fast (4534/4542; the 8 failures are the same store::rebalance / store::heal machine-baseline set that fails identically on pristine origin/main, plus one fencing flake that passes in isolation); all object_lock/retention/legal-hold tests pass; guard scripts (layer deps, migration rules, s3s footprint, logging, error-format ratchet, doc paths) pass.
* chore(deps): refresh s3s and related dependencies
Update the RustFS s3s git dependency to the requested f4dedc905 revision and keep the resolved dependency refresh from Cargo.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(api): adapt s3s upload stream error mapping
Detect the s3s upload stream SHA256 mismatch through the error chain without relying on the removed crate-root re-export.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(auth): preserve SigV2 S3 compatibility
Keep RustFS S3 service configuration explicit after the s3s default disables SigV2.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Switch IAM QR rendering from qrcode to qrcode-rs 2.0.0 while keeping only the std and svg feature path enabled.
Set the release profile to a single codegen unit and disable release debuginfo as requested.
Verification:
- cargo info qrcode-rs --registry crates-io
- cargo tree -p rustfs-iam -e features
- CARGO_TARGET_DIR=/private/tmp/rustfs-target-qrcode-rs-profile-tuning cargo test -p rustfs-iam --locked
- cargo fmt --all --check
- git diff --check
Co-authored-by: heihutu <heihutu@gmail.com>
* perf(signer): cache signing key to avoid redundant HMAC-SHA256
Cache the AWS4 signing key per (secret, region, date, service_type)
tuple. The signing key is derived from 4 HMAC-SHA256 calls and is
constant for a given user within the same UTC day, so caching it
eliminates ~0.5-1ms of redundant crypto per request.
The cache uses a LazyLock<Mutex<HashMap>> with automatic daily
rotation (cache entries naturally expire when the date component
of the key changes).
Refs: https://github.com/rustfs/backlog/issues/2005
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(signer): bound signing key cache
* fix(signer): satisfy cache lint
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Upgrade rustfs-mimalloc and rustfs-mimalloc-sys to 0.5.1, then call the new safe wrapper from Tokio worker thread startup so mimalloc can treat runtime threads as threadpool workers.
Keep the hint no-op on Windows, matching RustFS allocator platform boundaries.
Co-authored-by: heihutu <heihutu@gmail.com>
The INVENTORY_UID constant is only referenced inside
#[cfg(target_os = "linux")] test functions, so it appears unused on
macOS. Add a cfg_attr to allow dead_code on non-linux targets.
* ci(nightly): persist the nightly deb on Cloudflare R2
Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a
rustfs-nightly-latest.deb alias) through the same R2 channel package.yml
uses, so the nightly package can be downloaded later with a stable URL.
The step is skipped when the R2 secrets are not configured, keeping the
artifact-only mode intact.
* test: add pool expansion / decommission E2E script and workflow
Add the admin-API based pool expansion, rebalance and decommission test
script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch /
nightly workflow that runs it on a self-hosted runner against real nodes.
The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2
package) via the package_url input.
* ci(pool-test): run the pool expansion test on the smoke-testing runner
Backlog#1845 step 8 conclusion. The plan called for folding iam::Error into a policy::Error #[from] wrapper and deleting the hand-written mapping. Measurement rejected the fold: the duplicated variants have ~220 construction/match sites (about 140 in production) across iam and the admin handlers - all auth-critical - and the alias route is blocked by the orphan rule (iam's From<IamStorageError> and io conversions cannot be implemented for a foreign type). Meanwhile the drift risk the fold targeted is already compiler-covered: the From match is exhaustive with no catch-all, so any new policy variant fails the build until mapped.
What remains of the step, delivered: the six dead policy variants are gone (previous commit), the grouped lossy arm is down to the two variants actually produced, a doc comment on the From impl records the verdict with the evidence, and a new totality test constructs one representative of every policy::error::Error variant and asserts the conversion preserves the rendered message - so the mapping is now pinned loss-free in both directions the classifiers care about.
Ref rustfs/backlog#1845
Zhengchao An