rustfs/docs/operations/container-resource-detection.md

Container Resource Detection

RustFS automatically detects container resource limits (CPU and memory) from cgroup v1/v2. This ensures correct resource allocation and accurate metrics in containerized environments (Kubernetes, Docker, etc.).

Problem

When RustFS runs in a container, the underlying system libraries report the host's total CPU cores and memory, not the container's limits. This leads to:

1. Over-provisioned Tokio threads: Too many worker and blocking threads
2. Incorrect memory metrics: rustfs_memory_usage_percent shows host-based percentage
3. Memory budget errors: Object data cache sized to host RAM instead of container limit
4. OOMKills: Container exceeds its memory limit and gets killed

Solution

RustFS now detects cgroup limits directly from the filesystem:

• CPU: /sys/fs/cgroup/cpu.max (v2) or /sys/fs/cgroup/cpu/cpu.cfs_quota_us (v1)
• Memory: /sys/fs/cgroup/memory.max (v2) or /sys/fs/cgroup/memory/memory.limit_in_bytes (v1)

The effective resource limits are the minimum of host and cgroup values.

Detection Logic

CPU Detection

1. Read cgroup v2 /sys/fs/cgroup/cpu.max
   • Format: "$QUOTA $PERIOD" or "max" (unlimited)
   • Calculate: cores = ceil(quota / period)
2. Fallback to cgroup v1 /sys/fs/cgroup/cpu/cpu.cfs_quota_us
   • Calculate: cores = ceil(quota / period)
3. Fallback to host CPU count from sysinfo

Memory Detection

1. Read cgroup v2 /sys/fs/cgroup/memory.max
   • Value in bytes or "max" (unlimited)
2. Fallback to cgroup v1 /sys/fs/cgroup/memory/memory.limit_in_bytes
   • Very large values (≥2^62) indicate unlimited
3. Fallback to host memory from sysinfo

Environment Variables

Disable Cgroup Detection

RUSTFS_DISABLE_CGROUP_DETECTION=1

Disables cgroup detection entirely. Useful for testing or when cgroup filesystem is not accessible.

Override CPU Cores

RUSTFS_OVERRIDE_CPU_CORES=4

Overrides detected CPU cores. Takes precedence over cgroup detection.

Override Memory Limit

RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648

Overrides detected memory limit in bytes. Takes precedence over cgroup detection.

Metrics

New Metrics

Metric	Description
rustfs_memory_effective_total_bytes	Effective memory total (host or cgroup)
rustfs_cgroup_detected	Whether cgroup limits were detected (1=yes, 0=no)
rustfs_cgroup_cpu_cores_limit	Detected CPU cores limit
rustfs_cgroup_memory_limit_bytes	Detected memory limit

Updated Metrics

Metric	Change
rustfs_memory_total_bytes	Now uses effective memory (cgroup-aware)
rustfs_memory_usage_percent	Now calculated against effective memory

Startup Logging

RustFS logs detected container resources at startup:

INFO container resources (detected from cgroup) cpu_cores=2 memory_bytes=1073741824 memory_mib=1024

or

INFO container resources (overridden by environment variables) cpu_cores=4 memory_bytes=2147483648 memory_mib=2048

Examples

Kubernetes with Resource Limits

resources:
  limits:
    cpu: "2"
    memory: "1Gi"
  requests:
    cpu: "500m"
    memory: "512Mi"

RustFS will detect:

• CPU cores: 2
• Memory: 1 GiB (1073741824 bytes)

Docker with CPU and Memory Limits

docker run --cpus=2 --memory=1g rustfs/rustfs:latest

RustFS will detect:

• CPU cores: 2
• Memory: 1 GiB

Manual Override

export RUSTFS_OVERRIDE_CPU_CORES=4
export RUSTFS_OVERRIDE_MEMORY_BYTES=2147483648

RustFS will use:

• CPU cores: 4
• Memory: 2 GiB

Troubleshooting

Cgroup Detection Not Working

1. Check if cgroup filesystem is mounted:

   ls -la /sys/fs/cgroup/
   

2. Check cgroup version:

   stat -fc %T /sys/fs/cgroup/
   

   • cgroup2fs = cgroup v2
   • tmpfs = cgroup v1

3. Check if limits are set:

   # cgroup v2
   cat /sys/fs/cgroup/cpu.max
   cat /sys/fs/cgroup/memory.max
   
   # cgroup v1
   cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
   cat /sys/fs/cgroup/memory/memory.limit_in_bytes
   

Metrics Show Host Values

If rustfs_memory_effective_total_bytes shows host memory instead of cgroup limit:

1. Verify cgroup detection is not disabled:

   echo $RUSTFS_DISABLE_CGROUP_DETECTION
   

2. Check startup logs for cgroup detection:

   grep "container resources" /logs/rustfs.log
   

3. Use environment variable override as workaround:

   export RUSTFS_OVERRIDE_MEMORY_BYTES=1073741824
   

Implementation Details

Files Modified

• rustfs/src/cgroup_resources.rs - Core cgroup detection logic
• rustfs/src/container_config.rs - Container configuration with overrides
• rustfs/src/memory_observability.rs - Updated memory metrics
• rustfs/src/server/runtime.rs - Updated Tokio runtime configuration
• rustfs/src/startup_entrypoint.rs - Startup logging

Performance Impact

• Startup: One-time detection adds ~1ms overhead
• Runtime: Cached values, no repeated filesystem reads
• Memory: Negligible (<1KB for cached values)

Thread Safety

All detection functions are thread-safe and use OnceLock for caching.