OpenShift AI benchmark maps when KV-cache offloading pays off

Red Hat has published a three-model benchmark of tiered KV-cache offloading in OpenShift AI 3.5, measuring when CPU memory, local NVMe and CephFS can recover reusable prompt state faster than a GPU can recompute it. The strongest result was a 5.8× throughput improvement for Nemotron-Ultra-253B, with mean time to first token falling from 59.7 seconds to 9.6 seconds when CPU offload was backed by a secondary storage tier.

The important qualification is that offloading was not uniformly valuable. The result changed with the size of the model’s reusable working set, the capacity of the CPU tier, storage tail latency and concurrency.

The working set decides the tier

The benchmark ran OpenShift AI 3.5 with vLLM 0.24.0 and llm-d 0.8.0 on an OpenShift 4.22.2 cluster with eight-H100 nodes. It replayed a long-context agent trace averaging about 50,600 input tokens against Nemotron-Ultra-253B, Qwen3.6-35B-A3B and Gemma 4 31B.

Nemotron’s reusable KV state exceeded its 256 GiB CPU tier. CPU offload alone raised throughput from 0.13 to 0.44 requests per second, while adding local NVMe or CephFS raised it to 0.76 and 0.77. At concurrency 128, the CPU-only cache hit rate fell to 1.5%, while CephFS retained a 59% hit rate and delivered about twice the throughput of the alternatives.

Qwen’s working set fit inside its 64 GiB CPU tier, so CPU offload captured most of the gain: throughput doubled from 0.58 to 1.17 requests per second. NVMe and CephFS added only about another 10% for a single replica. Gemma improved with offloading, but high GPU-memory use constrained the benefit from adding replicas.

Storage latency is only the first gate

Red Hat’s method starts with model-specific KV block sizes and p99 storage latency, not headline sequential bandwidth. Local NVMe produced the lowest and most stable latency. CephFS was slower per block, but supplied shared capacity, persistence and cache reuse across replicas. The test also used a minimal CephFS replication setting of one copy, meaning the published performance is not automatically a production durability target.

The authors calculate a crossover point where restoring a block becomes faster than recomputing its prefix. Short-context workloads can fall on the wrong side of that boundary, making offloading added overhead rather than an optimization.

What platform teams should test

The reusable decision process is more valuable than the largest multiplier. Teams should benchmark p99 read latency with the actual model block size and mixed I/O pattern, measure uncached prefill time per token, then replay a representative trace across no-offload, CPU-only and secondary-tier configurations.

Local NVMe best fits a single-node workload with stable cache locality. CephFS becomes more attractive when pods move, nodes fail or multiple replicas can reuse prefixes. In either case, the result says to size the cache around the workload’s reusable state and operational topology—not around a generic claim that more storage makes inference faster.