live wire
▸Red Hat ships important MicroShift 4.16.73 security updateRed Hat Product Errata▸AI · Red Hat maps local AI Inference serving and load tests on RHEL (Oct. 15)Red Hat Emerging Technologies▸PLATFORM · IBM guide maps Secure Execution VMs onto OpenShift Virtualization for LinuxONEIBM Community▸AI · vLLM maps Nemotron 3.5 Super VL across Hopper, Blackwell and Red Hat DSpark (Oct. 15)vLLM Recipes▸AI · AWS contributes push-based KV transfer to vLLM, reporting 3x better TTFT under loadAWS Machine Learning Blog▸AI · AWS contributes bidirectional KV transfer to vLLM, cutting multi-turn TTFT up to 3.3xAWS Machine Learning Blog▸AI · vLLM Semantic Router adds d3-turbo with box and video-tracking decisionsvLLM Semantic Router▸IDENTITY · Keycloak DevDay 2027 opens call for deep technical talks through Oct. 24Keycloak▸AI · vLLM Recipes maps Muse Spark Open to single-node Blackwell and AMD deployment (Oct. 16)vLLM▸PLATFORM · Clang 23 bytecode interpreter cuts allocations and expands constant-expression supportRed Hat Developer▸PLATFORM · RHEL becomes first-party on OCI; OpenShift subscriptions reach Oracle MarketplaceRed Hat▸AI · vLLM-Omni merges native MOSS-Image serving with parallelism and offloadvLLM▸AI · vLLM adds System One endpoint for Decision 3.0 multimodal routing models (Oct. 15)vLLM▸SECURITY · Red Hat rates OpenShift AI 3.4.5 update Critical; advisory lists hundreds of CVEsRed Hat Product Security▸Red Hat ships important MicroShift 4.16.73 security updateRed Hat Product Errata▸AI · Red Hat maps local AI Inference serving and load tests on RHEL (Oct. 15)Red Hat Emerging Technologies▸PLATFORM · IBM guide maps Secure Execution VMs onto OpenShift Virtualization for LinuxONEIBM Community▸AI · vLLM maps Nemotron 3.5 Super VL across Hopper, Blackwell and Red Hat DSpark (Oct. 15)vLLM Recipes▸AI · AWS contributes push-based KV transfer to vLLM, reporting 3x better TTFT under loadAWS Machine Learning Blog▸AI · AWS contributes bidirectional KV transfer to vLLM, cutting multi-turn TTFT up to 3.3xAWS Machine Learning Blog▸AI · vLLM Semantic Router adds d3-turbo with box and video-tracking decisionsvLLM Semantic Router▸IDENTITY · Keycloak DevDay 2027 opens call for deep technical talks through Oct. 24Keycloak▸AI · vLLM Recipes maps Muse Spark Open to single-node Blackwell and AMD deployment (Oct. 16)vLLM▸PLATFORM · Clang 23 bytecode interpreter cuts allocations and expands constant-expression supportRed Hat Developer▸PLATFORM · RHEL becomes first-party on OCI; OpenShift subscriptions reach Oracle MarketplaceRed Hat▸AI · vLLM-Omni merges native MOSS-Image serving with parallelism and offloadvLLM▸AI · vLLM adds System One endpoint for Decision 3.0 multimodal routing models (Oct. 15)vLLM▸SECURITY · Red Hat rates OpenShift AI 3.4.5 update Critical; advisory lists hundreds of CVEsRed Hat Product Security
upstreambeat.ai
analysisAI

vLLM pushes KV cache off the decode critical path

An AWS-contributed connector overlaps destination setup with prefill work, trading later routing flexibility for lower latency under load.

By The News Desk· Oct 19, 2026the quick take — two AI hosts go live when you do

Disaggregated model serving splits prompt processing from token generation, but the handoff has a cost: the prefill instance must move its key-value cache to the decode instance before generation can start. vLLM’s default NIXL flow has decode discover and pull those blocks after prefill completes, putting allocation, metadata relay and transfer setup directly on the path to the first token.

AWS has contributed a push-based NIXL connector to vLLM. The design lets decode register its destination blocks while prefill is still computing, then has prefill write the finished cache into those blocks from a background thread. The result is a different scheduling tradeoff rather than simply a faster copy.

What changed

The push protocol has three stages. Decode allocates and registers destination blocks before prefill completes. Prefill matches that registration with the request when its cache is ready. A background worker then prepares descriptors and issues a NIXL write directly into decode memory, with completion notifications telling decode when generation can begin.

AWS says the implementation can map different physical block layouts on each side and accounts for retried or preempted requests when matching registrations. Moving descriptor preparation and transfer submission away from decode’s latency-sensitive engine loop is intended to prevent cache-transfer work from starving token generation.

What the measurements show

AWS tested Qwen3-32B with one prefill and one decode node on EC2 P5en instances connected through Elastic Fabric Adapter. With prefix caching disabled to force a full transfer, the company reports that push mode reached roughly 540 milliseconds time to first token at eight queries per second and a 16,000-token input, versus about 1,744 milliseconds for pull mode. It also reports time per output token near 7 milliseconds in push mode, compared with about 10.4 milliseconds for pull mode at the heaviest tested combinations.

That is up to a threefold TTFT improvement and about a 30% TPOT improvement in AWS’s configuration, not a general guarantee. At low load and short prompts, the two modes were close. The reported gap widened as request rate and context length increased because pull-mode setup continued to run inline on the decode worker.

The routing tradeoff

Push mode must select the decode instance early enough to register its destination memory before prefill finishes. Pull mode can defer that choice, preserving more freedom for load-aware routing and quality-of-service decisions. Pull also remains necessary for vLLM’s new bidirectional KV flow, which returns decode-side state to prefill across conversational turns.

Operators can enable the new path by configuring both sides with NixlPushConnector instead of NixlConnector. The decision should follow workload measurements: push is aimed at high request rates where early decode selection is acceptable, while pull remains the more flexible option when routing decisions or reverse-direction cache reuse matter more.

sources

Filed by The News Desk. Corrections: desk@upstreambeat.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.

    more from the desk