The Next Layer of Inference Efficiency: Cross-Instance KV Cache and Multi-Stage Serving
Two developments in the vLLM ecosystem, LMCache's cross-instance KV cache sharing and vLLM-Omni's multi-stage serving, point at where inference efficiency problems are heading next, and why a one-time configuration decision won't keep up.
Most teams think about inference efficiency as a single-model, single-replica problem: the right GPU tier, the right max-num-seqs, the right batch size. Two recent developments in the vLLM ecosystem suggest that frame is already too narrow.
LMCache: KV cache that outlives a single replica
vLLM's own prefix caching (covered in our KV Cache Pressure post) reuses cached attention state within a single running engine. LMCache extends the same idea across the fleet: it backs the KV cache with a shared store (CPU memory, local disk, or a remote/distributed cache) so a prefix computed by one replica can be reused by a completely different replica serving a later request.
That matters anywhere prefixes repeat across replicas rather than within one: shared system prompts, multi-turn sessions that get load-balanced across pods, RAG pipelines reusing the same retrieved context. Without cross-instance sharing, every replica pays the full prefill cost the first time it sees a prefix, no matter how many times another replica has already seen it.
LMCache also plugs into vLLM's support for prefill/decode disaggregation, running the compute-bound prefill phase and the memory-bandwidth-bound decode phase on separate instances, connected by a KV transfer layer. That lets each phase run on hardware shaped for what it actually needs, instead of forcing one instance to be good at both.
The net effect: lower time-to-first-token from avoided recomputation, more consistent per-token latency from freed-up compute and phase isolation, and the option to size prefill and decode capacity independently.
The catch is the same one that applies to any cache: hit rate depends on how well the cache matches current traffic, and traffic isn't static. A cache that's earning its memory and complexity overhead today can stop earning it the moment the workload mix shifts toward more unique, one-off requests. Sizing and enabling it is a decision made once; whether it's still the right decision is a question that has to be asked continuously.
vLLM-Omni: when "the model" is actually a pipeline of models
vLLM-Omni (vLLM project's own framework for omni-modality serving, which reached its first stable release this year) tackles a different problem: serving models that take and produce more than text. Image, video, audio, and non-autoregressive architectures like Diffusion Transformers all need different execution patterns than a standard autoregressive LLM.
Its core idea is a stage graph: an any-to-any model gets decomposed into a graph of stages (an LLM core for reasoning, a Diffusion Transformer for image or video generation, an audio decoder, an encoder for multimodal input) each of which can run as its own process, on its own hardware, sequenced by an orchestrator and connected by a transfer layer purpose-built to move intermediate state between stages. Qwen3-Omni is a concrete example: it's broken into Thinker, Talker, and Code2wav stages, each a distinct model.
This formalizes something that's already happening informally wherever multimodal and agentic pipelines get built: different stages of one logical request have wildly different resource profiles. A diffusion stage is compute-heavy in a different way than an LLM core; an audio decoder might be lightweight compared to both. Treating the whole pipeline as one workload on one box stops making sense once stages disaggregate, and the right placement for stage one is rarely the right placement for stage three.
The pattern underneath both
LMCache and vLLM-Omni are solving different problems, but they push on the same assumption from two directions: that a request maps cleanly onto one model running on one replica. LMCache breaks that by letting a single phase of one request (prefill) be served by different hardware than another phase (decode) of the same request, with cache state shared across a fleet rather than scoped to a process. vLLM-Omni breaks it further by splitting a single logical request across an arbitrary number of heterogeneous model stages.
Both also introduce something worth tracking that doesn't exist in the simple single-model world: a cache hit rate that's a fleet-wide property, not a per-process one, and a resource profile that varies stage-by-stage within a single request rather than being constant for the life of a deployment.
What this means for operators
If you're adopting either of these, two things are worth doing before you reach for tuning knobs:
- Instrument before you optimize. Know your actual hit rate, eviction rate, and per-stage GPU profile before changing configuration. Guessing at sizing for a cache or a multi-stage pipeline is no more reliable than guessing at GPU tier sizing, and the failure modes (silent latency regressions, unexplained cost increases) look similar.
- Re-check the decision, not just the configuration. A cache or pipeline topology that was correctly sized at launch can become wrong without anyone changing a setting: traffic composition shifts, session patterns change, models get swapped. The decision to enable, size, or restructure either of these isn't a one-time call; it's a question that needs re-asking as the workload evolves.
That second point is the throughline of almost everything we've written in this series. Static configuration was never really the goal; it was a stand-in for "correctly sized for the traffic I have right now." As inference architectures get more sophisticated (cache shared across a fleet, requests split across heterogeneous model stages) the gap between "configured once" and "correct right now" only gets wider.
See how Paralleliq tracks fleet-wide efficiency as inference architectures evolve →