Files
rustfs/scripts/prepare_conditional_read_baseline.py
T
dd90bb7488 fix(s3): short-circuit conditional reads at metadata quorum (#8459)
* fix(s3): evaluate conditional reads before opening object bodies

Reuse locked metadata snapshots for conditional GETs and unify HEAD
preconditions and cache headers. Preserve body-cache hooks and validate
encryption metadata and key authorization before returning bodyless errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): unify conditional reads at metadata quorum

Apply scoped metadata conditions to the existing reader pipeline instead
of maintaining parallel prepared-read branches. Preserve quorum checks,
early-stop, coalescing, and cache-hook behavior on condition misses.

Track metadata-only termination through selection and reject body access
or a changed decision. Keep concrete reader futures off request stacks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): strengthen conditional read acceptance evidence

Assert zero shard, decode, cache-fill, and remote-body calls on matching
conditions with live positive controls. Exercise GET and HEAD through
s3s and Hyper with conditional-header and precedence matrices.

Add reproducible paired application-read benchmarks with isolated
metadata controls and strict drift, artifact, and resource gates.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): remove obsolete conditional comparison helper

Keep ETag comparison tests on the canonical read-condition evaluator,
remove unused date-trait imports, and name the HTTP case type.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): validate full conditional etag fields

Reject adjacent tags and malformed trailing list members before accepting a
match. Keep the existing request-scoped metadata read and authorization order,
while removing redundant validation wrappers and unused benchmark overrides.
Replace simulated phase coverage with real 304/412 cache-fill probes and extend
HTTP/1.1 and HTTP/2 ingress regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): cache terminal conditional metadata safely

Preserve quorum-validated conditional metadata when early termination
cancels the remaining fanout. Restrict those entries to terminal GET
readers; body and full-layout consumers still obtain their own metadata.

Serialize cache publication per key so conditional entries cannot replace
complete layouts or renew their TTL. Retain generation fencing and the
final selected-metadata predicate check before any body access.

Cover real four-disk hedging, nonterminal body and range reads, consumer
isolation, generation races, atomic publication and actual cache expiry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(log-analyzer): align heal writer failure rule with current logs

Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 16:59:21 +08:00

75 lines
4.0 KiB
Python

#!/usr/bin/env python3
"""Install only the paired benchmark harness in a clean detached baseline."""
import argparse
from pathlib import Path
import re
import subprocess
def replace_once(source, old, new, path):
if source.count(old) != 1:
raise ValueError(f"baseline anchor changed in {path}; inspect it before measuring")
return source.replace(old, new)
def prepare(root, baseline, revision):
if baseline.resolve() == root.resolve():
raise ValueError("baseline must be a separate worktree")
if not re.fullmatch(r"[0-9a-f]{40}", revision):
raise ValueError("supply the full baseline commit SHA")
def git(*args):
return subprocess.check_output(["git", "-C", str(baseline), *args], text=True).strip()
if git("rev-parse", "HEAD") != revision or git("status", "--porcelain"):
raise ValueError("baseline must be clean at the requested commit")
if subprocess.run(["git", "-C", str(baseline), "symbolic-ref", "-q", "HEAD"],
stdout=subprocess.DEVNULL).returncode == 0:
raise ValueError("use a detached baseline worktree")
edits = {}
def edit(path, old, new):
previous = edits.get(path, (baseline / path).read_text())
edits[path] = replace_once(previous, old, new, path)
edit("rustfs/src/app/object/mod.rs", "mod test_support;",
"mod test_support;\n#[cfg(test)]\nmod conditional_read_bench;")
edit("crates/ecstore/src/set_disk/mod.rs",
'const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";',
'#[cfg(feature = "test-util")]\nmod conditional_read_bench;\n'
'#[cfg(feature = "test-util")]\npub use conditional_read_bench::ConditionalReadBenchmarkMetadataGuard;\n\n'
'const ENV_RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS: &str = "RUSTFS_GET_METADATA_SLOWTAIL_FAULT_DELAY_MS";')
edit("crates/ecstore/src/set_disk/read.rs",
"fn get_object_metadata_cache_request_bypass_reason(bucket: &str, opts: &ObjectOptions, read_data: bool) -> Option<&'static str> {",
"fn get_object_metadata_cache_request_bypass_reason(bucket: &str, opts: &ObjectOptions, read_data: bool) -> Option<&'static str> {\n"
' #[cfg(feature = "test-util")]\n if super::conditional_read_bench::cache_bypass_applies(bucket) {\n'
' return Some("benchmark");\n }')
edit("crates/ecstore/src/api/mod.rs",
"pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;",
"pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;\n"
" pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;")
for path in ("rustfs/src/storage/storage_api.rs", "rustfs/src/app/storage_api.rs"):
# The facades also expose these types elsewhere; target the test-util block.
source = (baseline / path).read_text()
marker = ("pub(crate) use rustfs_ecstore::api::set_disk::test_util::{" if "/storage/" in path else
"pub(crate) use crate::storage::storage_api::ecstore_set_disk::{")
replacement = marker + "\n ConditionalReadBenchmarkMetadataGuard,"
edits[path] = replace_once(source, marker, replacement, path)
for path in ("rustfs/src/app/object/conditional_read_bench.rs",
"crates/ecstore/src/set_disk/conditional_read_bench.rs"):
if (baseline / path).exists():
raise ValueError(f"baseline already contains {path}")
edits[path] = (root / path).read_text()
# Resolve every anchor before writing; never apply a partial adapter.
for path, source in edits.items():
(baseline / path).write_text(source)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline-worktree", type=Path, required=True)
parser.add_argument("--revision", required=True)
args = parser.parse_args()
prepare(Path(__file__).resolve().parent.parent, args.baseline_worktree, args.revision)
if __name__ == "__main__":
main()