llm-d 1.0 Ships, and the Interesting Problem Moved Up a Layer

llm-d 1.0, announced October 20 at PyTorch Conference North America, argues that open-weight inference has outgrown the engine layer. The hard part is now scheduling, multi-tenancy, and keeping prefix caches warm across an agent's turns.

llm-d 1.0 shipped on October 20, and almost nothing inside it is new.

That is the point. llm-d – the Kubernetes-native distributed inference serving network lead-maintained by Red Hat, IBM and Google, co-founded with CoreWeave and NVIDIA – put a stable tag on a stack whose individual techniques have existed for years. IBM’s own engineering writeup leads with the claim that it is “ready for production inference serving.” That framing is the project’s, and it is the part worth arguing about.

Read the technique list first. Then read what 1.0 actually added.

What llm-d Already Was

llm-d is not an inference engine. It sits above engines like vLLM and SGLang inside a Kubernetes cluster, and decides where requests land and how capacity is shared among the replicas. Think scheduler and control plane, not replacement engine.

Five core techniques carry the project:

Technique What it does
Prefix-cache-aware routing Send a request to the replica already holding that prefix
KV-cache offloading Move cold cache state to slower memory instead of recomputing it
Batch inference optimization Pack requests so more of the GPU does useful work
Prefill/decode disaggregation Split the compute-bound and memory-bound halves of a decode across devices
Wide expert parallelism Spread mixture-of-experts layers wide enough to use them

None of those are new in 1.0. Prefix caching and disaggregated serving were research results that became engine features. What the 1.0 announcement claims is that they are now integrated into one operable network, with the failure modes handled.

That is a smaller claim than “new technology.” It is also a more useful one, because integration is exactly what self-hosters were doing by hand.

What 1.0 Actually Added

Three headline items, and all three are operational rather than algorithmic:

  • Multi-tenant quality of service. Flow control, fairness and prioritization when several workloads share one cluster. Before this, sharing a cluster meant one tenant’s batch job starving everyone else’s interactive traffic.
  • Zero-downtime operations. Rolling changes to models, weights and engine versions without dropping requests.
  • Agent-aware serving. The one that matters most, and the one nobody had built for a workload shape that did not exist three years ago.

Here is the framing signal: “day 2” is platform-engineering vocabulary, not research vocabulary. Day 1 is deploying. Day 2 is everything after – upgrades, scaling, quota fights, mixed tenants, a rolling restart at 2am. The project is explicitly claiming it has crossed from day 1 to day 2. That claim is IBM’s and the maintainers’, not an independent audit result, and no vendor has published a third-party benchmark for the 1.0 stack that I could verify.

Why Agent-Aware Routing Is the Actual Story

An agent conversation is not a series of independent requests. Turn one establishes a system prompt plus task context. Turn two resends all of it plus a little more. Turn five resends all of that plus the tool results. Each turn in a multi-turn agent session recomputes a prefix the previous turn already computed.

So a naive load balancer is actively wasting your hardware. It spreads turns evenly across replicas, which means most of them land on a node that does not hold the prefix and has to recompute it from scratch. Prefix-cache-aware routing sends each turn to the replica that already has that context in its KV cache. Day-2 framing pins the request to appropriate infrastructure across turns specifically to keep those caches warm.

For anyone self-hosting inference, this reframes the optimization target. You have probably been tuning throughput – batch size, quantization, GPU layer offload, speculative decoding. All of that still matters. But if your traffic is agent traffic, cache hit rate on a warm replica may be worth more than raw tokens per second, because recomputing a long shared prefix is pure waste that no engine-level tuning removes.

It is also worth keeping this separate from security. We covered the KV cache as an attack surface on October 19 (Your KV Cache Is Now Part of Your Attack Surface) – untrusted prefixes getting cached and poisoning later requests. That finding is about what gets stored. This is about where requests get sent. Same data structure, opposite question.

The Same Week, Pointing the Same Direction

Within days of the 1.0 announcement, Google shipped TorchTPU backends for vLLM and SGLang, and a preprint titled around prefix-cache soundness (AKV, arXiv:2610.28577) appeared arguing that routing decisions have to account for which prefixes are safe to share. The AKV work is a v1 preprint from its authors and has not been peer-reviewed – treat it as a direction, not a result. Taken together these say the same thing from three directions: inference is becoming a placement problem, and placement is becoming the hard part.

What To Do On Monday

If you run inference yourself, do not rewrite anything this week. Do measure one number you are probably not measuring: what fraction of your requests hit a replica that already holds the prompt prefix. If that number is low and your traffic is multi-turn, you have a routing problem, and it will cost you more than another quantization pass will save.

Second, hold the vendor framing loosely. “Production ready” in a 1.0 announcement is a milestone, not a benchmark. Watch for independent numbers before you migrate a fleet.

Third, and most concrete: the audience for this release is people who already run Kubernetes and already run vLLM or SGLang. If that is you, the release note is worth an hour of your afternoon. If you are running a single model on a single workstation, none of this touches you yet – and that is fine, because it is not aimed at you.

The engine war settled. What replaced it is the boring layer that decides where each request goes, which is where inference cost actually lives now.

Sources