llm-d 1.0 Runs 3,000 Coding Agents at 5 to 10x Lower Token Cost, but Every Benchmark Is a Vendor Demo
llm-d 1.0 adds multi-tenant QoS, zero-downtime upgrades and agent-aware routing. IBM claims 5 to 10x cheaper agent serving, but no independent benchmark exists yet.
On October 20, 2026, the llm-d project released version 1.0 at PyTorch Conference North America in San Jose. The pitch is not another model. It is the plumbing that decides whether an open-weight model is cheap to serve or ruinously expensive.
Version 1.0 adds the operational pieces enterprises keep asking for: multi-tenant fairness, zero-downtime upgrades, and routing that keeps a long agent conversation close to its cached context. IBM, Red Hat, Google Cloud, CoreWeave and NVIDIA founded the project in 2025. It now has more than 500 contributors from 25 organizations and sits under CNCF governance.
The headline numbers are large. IBM says one deployment served up to 3,000 concurrent coding agents, peaked at 6.6 million output tokens per minute, and cost 5 to 10 times less per token than commercial API pricing. Every one of those figures comes from IBM’s own benchmark run. No independent lab has reproduced them.
What llm-d 1.0 actually ships
llm-d is a Kubernetes-native orchestration layer for LLM serving. It does not replace vLLM or SGLang. It decides where requests go, how cached context moves, and what happens when one tenant floods the cluster.
Multi-tenant quality of service. Priority and fairness-aware flow control keeps latency-sensitive requests moving while bursty agent workloads share the same fleet. The goal is to stop a coding agent from starving a customer-facing chatbot.
Zero-downtime operations. Router high availability with a hot standby and rolling updates keeps traffic flowing through upgrades. Prefill and decode replicas are managed as one logical unit, with coordinated rollouts.
A stable v1 configuration API. The project says the request path is hardened and validated across clouds and accelerators. That matters for teams that want to standardize on one serving stack instead of rebuilding it every quarter.
Agent-aware serving. Session-aware routing keeps multi-turn conversations near their cached context. Turn-priority scheduling and native Responses API state support reduce recomputation across agent turns.
Everything from the earlier releases is still there: prefix-cache and load-aware routing, KV cache offloading, prefill and decode disaggregation, wide expert parallelism, and batch or async inference. The code is Apache 2.0 on GitHub.
The benchmark IBM is leading with
IBM published two sets of numbers. One is a serving demo on a 753-billion-parameter open-weight model. The other is an internal research workload. They measure different things, and the press release blends them into one story.
| Claim | Figure | Workload | Source |
|---|---|---|---|
| Peak output throughput | 6.6 million tokens per minute | GLM 5.2, 753B parameters, agentic benchmarks | IBM demo |
| Concurrent agents | 3,000 coding agents, zero preemptions | Same GLM 5.2 deployment | IBM demo |
| Cost vs commercial API | 5 to 10 times lower per token | Same run, current cloud rental rates | IBM estimate |
| Quantum error correction cost | 11.5 times lower | IBM Research workload vs all-proprietary setup | IBM Research |
| Prefix cache hit rate | 98% | Same quantum workload | IBM Research |
| Blended serving cost | About $0.11 per million tokens | Same quantum workload | IBM Research |
Why 3,000 agents is the number that matters
An agent does not send one prompt. It sends a growing conversation, calls a tool, waits, then sends the same growing conversation again. Most of the input tokens are repeats. If the router sends each turn to a different replica, that context is recomputed from scratch.
llm-d keeps session-aware routing and prefix caching at the orchestration layer. IBM says its router captured 97% of available cache reuse in one agentic coding test. The 98% prefix-cache hit rate in the quantum workload is the same idea applied to a different job.
The 753B-parameter GLM 5.2 run matters because it used ordinary NVIDIA GPUs, not exotic hardware. If the numbers hold, enterprises can serve frontier-class open models on fleets they already rent. That is a different claim from “open models are cheaper.” It is the claim that the orchestration overhead no longer eats the savings.
Google is putting it into GKE
Google Cloud is not treating this as a side project. Mark Lohmeyer, vice president and general manager of AI and computing infrastructure, said the company is integrating llm-d 1.0 into Google Kubernetes Engine and enabling it across its TPU and NVIDIA GPU portfolio.
CoreWeave plans to offer it on its Kubernetes service alongside NVIDIA Dynamo. DigitalOcean says it already runs llm-d as the base for Serverless Inference and Managed Agents, serving trillions of tokens. Roblox says it uses the stack for disaggregated serving across cloud and on-premises GPUs.
That list is the strongest evidence in the announcement. Vendors can publish any benchmark they like. Getting a cloud provider, a neocloud, a GPU maker and a large enterprise to describe production use in the same document is harder to fake.
What it costs, and what the 5 to 10x claim leaves out
llm-d itself is free. The bill is the cluster. IBM’s $0.11 blended cost per million tokens came from a workload with a 98% cache hit rate, which is close to a best case. A workload with unique prompts will reuse far less and cost more.
That is why the 5 to 10x claim needs a baseline. IBM compares its deployment with “equivalent commercial API pricing” at current cloud rental rates. It does not publish the API it priced against, the GPU rental rate, or the utilization level. A 5x saving on one workload can be a loss on another.
The same caution applies to DeepSeek’s DeepInfer release, which also promised lower serving costs from a vendor’s own tests. DeepInfer is tied to DeepSeek models. llm-d is engine-agnostic, which makes it easier to adopt and harder to benchmark against a single reference point.
The catch
The release is real. The software exists, the repositories are public, and the feature list is specific. The performance story is thinner.
IBM’s press release says its quantum workload ran at 11.5 times lower inference cost than an all-proprietary configuration. Red Hat’s blog post about the same workload says five to ten times. Both are vendor documents published on the same day. Neither names the proprietary baseline or its price.
Independent verification is the missing piece. A third-party benchmark or a customer with published bills would settle it. Until then, treat the numbers as targets, not guarantees.
Confirmed vs Not Confirmed
| Claim | Status | Why |
|---|---|---|
| llm-d 1.0 released on October 20, 2026 | Confirmed | IBM press release and Google Cloud blog |
| Founded by IBM, Red Hat, Google Cloud, CoreWeave and NVIDIA | Confirmed | IBM press release |
| 500+ contributors across 25+ organizations | Confirmed | IBM press release |
| Multi-tenant QoS, zero-downtime upgrades, v1 config API | Confirmed | IBM Research blog and release notes |
| Google integrating into GKE across TPU and NVIDIA GPUs | Confirmed | Google Cloud executive quote |
| Apache 2.0 license | Confirmed | GitHub repository metadata |
| 5 to 10x lower cost in production | Not confirmed | Vendor estimate, no published bills or baseline |
| 3,000 concurrent agents at that cost | Not confirmed | Single IBM demo, no customer deployment data |
| 11.5x advantage over proprietary serving | Not confirmed | Baseline not named; Red Hat says 5 to 10x |
| Independent throughput or latency results | Not confirmed | No third-party benchmark published |
What it means for Indian developers
For teams in India running open-weight models on Kubernetes, the useful part of 1.0 is operational, not the benchmark. Multi-tenant fairness and rolling upgrades are what platform teams need before they put an internal coding agent in front of 200 developers.
The savings depend on prefix reuse. An internal assistant that shares a long system prompt and repository context will hit cache often. A customer-facing bot with unique prompts will not. Measure the hit rate on your own traffic before you build a cost model.
Managed Kubernetes clusters with NVIDIA GPUs are widely available from Indian cloud providers. Check that the cluster supports the Kubernetes Gateway API inference extension that llm-d uses for routing. The software is free; the engineering time is not.
FAQ
Is llm-d 1.0 free to use?
Yes. The project is open source under Apache 2.0. You pay for the Kubernetes cluster, GPUs or accelerators, and the people who run them.
Does llm-d work only with vLLM?
No. It supports vLLM and SGLang as model servers. It sits above them and handles routing, caching, and scheduling rather than replacing the inference engine.
Do I need NVIDIA GPUs to run it?
The IBM demos used NVIDIA GPUs, and Google says it is enabling llm-d across its TPU portfolio as well. The project targets multiple accelerators, but support varies by engine and cloud.
Is the 5 to 10x cost saving real?
It is an IBM estimate from one benchmark configuration, not an audited result. The saving depends on cache hit rate, GPU rental prices, and how much of your traffic repeats the same context.
How is llm-d different from DeepSeek DeepInfer?
DeepInfer is an inference engine built for DeepSeek’s own models. llm-d is an orchestration layer that can sit in front of vLLM or SGLang and route traffic across many model servers, tenants, and accelerators.