DeepSeek Open-Sources DeepInfer, the Engine Serving 155 Trillion Tokens a Day, but It Leaves Out the KV Cache and Load Balancers and Runs Best on 128 GPUs

DeepSeek has open-sourced the software it uses to serve its own chatbot and API. The company announced DeepInfer, its in-house inference framework, in an X thread on 20 October 2026 that passed 1.7 million views within six hours, and published the code on GitHub under the Apache 2.0 license.

The headline numbers come from DeepSeek’s own deployment notes: a single cluster running DeepInfer serves 155 trillion tokens a day and earns about $38 per GPU-hour (around ₹3,610) even at off-peak prices. That is the engine behind DeepSeek V4.1 Flash, which the company sells for as little as $0.15 per million input tokens.

The thread calls it “fully open source.” The repository’s own README is more careful. Some of the most important production pieces are not in this release, and the setup DeepSeek recommends starts at 128 GPUs. We went through the repo, the deployment Q&A and the price list to separate what you actually get from what DeepSeek kept.

What DeepSeek released

DeepInfer is described in the README as “the high-performance inference framework that powers DeepSeek’s large-scale production services.” This release provides offline inference with what DeepSeek calls the core features needed for serving at scale.

The listed features include optimized kernels from DeepSeek’s earlier open-source libraries (FlashMLA, DeepGEMM, DeepEP and others), attention data parallelism and MoE expert parallelism with load balancing between experts, speculative decoding through a system it calls DSpark, structured outputs for JSON schema, regex and grammar, and a zero-overhead asynchronous batch scheduler.

DeepSeek also released deepseek-recipe, a Rust component with Python bindings under the MIT license. It converts requests in the Chat Completions, Responses and Anthropic Messages formats into model tokens and back, using the same logic as the official DeepSeek API.

The README credits vLLM as the original base of the project, with ideas borrowed from SGLang and NVIDIA’s TensorRT-LLM. So this is not a clean-room engine. It is a heavily tuned fork of the open-source stack many teams already use.

What the release leaves out

The README lists three things that a complete inference service needs and that are “not included in this release”:

• A KV cache system
• Disaggregation of the vision encoder, prefill and decode stages (DeepSeek says most of the engine work for this is already in the code)
• Load balancing across data-parallel replicas for prefill and decode

The deployment Q&A in the repo explains why: the KV cache management and load balancing components “are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced.” Instead, DeepSeek describes how they work in prose and points to a Zhihu blog post.

That matters because the KV cache is a big part of the cost story. DeepSeek says its system keeps cached context on a dedicated 3FS storage cluster of roughly 100 PiB of SSDs, with a three-day lifetime for agent traffic and seven days for chatbot traffic. Cached input is billed at just $0.003 per million tokens off-peak, and this storage layer is what serves those hits. You get the design notes for it, not the code.

A few smaller gaps are also stated openly. The public FlashMLA kernel lacks one hand-tuned optimization that makes DeepSeek’s internal decode kernel 5% to 10% faster, though the company puts the end-to-end gap at 5% or less. The internal speculative-decoding draft model is “somewhat newer,” with what DeepSeek calls a small performance difference.

The hardware bar is high

DeepInfer supports exactly one model today, DeepSeek-V4.1-Flash-1017, a 763-billion-parameter model whose weights are under the MIT license. It runs on two hardware families: NVIDIA SM100 GPUs (the Blackwell data center generation) and Huawei Ascend 950 NPUs.

There is no listed support for Hopper cards such as the H100 or H20, which many cloud providers still rent, and nothing for consumer GPUs. Two features, batch-invariant inference and deterministic sampling, are marked NVIDIA only.

The Q&A is blunt about scale. A deployment with one prefill unit and one decode unit is possible, but because DeepSeek recommends 64-way expert parallelism or more for decode, “the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode.” The example scripts do run offline on a single 8-GPU node, which is enough to test the engine but not to reproduce DeepSeek’s economics.

DeepSeek’s published decode benchmark, run on 64 SM100 GPUs at a 131,072-token context, shows about 26,000 output tokens per second per GPU at 256 requests per rank, with each user getting roughly 100 tokens per second. At 64 requests per rank the chart shows about 15,700 tokens per second per GPU and around 245 tokens per second per user. Note that the chart assumes a fixed 75% draft-token acceptance rate rather than live traffic.

Does the $38 per GPU-hour figure add up?

We ran a rough check using only DeepSeek’s own numbers. At 15,700 output tokens per second, one decode GPU produces about 56.5 million tokens an hour. At the off-peak output price of $0.60 per million tokens on DeepSeek’s pricing page, that is about $33.90 an hour.

That lands in the same range as the $38 DeepSeek reports, before counting any input-token revenue. It is a back-of-envelope estimate, not an audit: it ignores the prefill GPUs, real acceptance rates and the mix of cached and uncached traffic. But it suggests the claim is plausible rather than marketing.

For scale, 155 trillion tokens a day works out to roughly 1.8 billion tokens every second from one cluster.

Confirmed vs not shown

Claim	Status	Source
DeepInfer code released under Apache 2.0	Confirmed	GitHub repo and LICENSE
Supports DeepSeek-V4.1-Flash on NVIDIA SM100 and Ascend 950	Confirmed	README
KV cache, prefill/decode disaggregation, replica load balancing	Not included	README and Q&A
155 trillion tokens a day per cluster	DeepSeek’s claim, not independently verified	Deployment Q&A
$38 revenue per GPU-hour at off-peak prices	DeepSeek’s claim; our rough math is consistent	Deployment Q&A, pricing page
Public kernels within 5% of internal end to end	DeepSeek’s claim	Deployment Q&A
Support for future DeepSeek models	Promised, not shipped	X thread

Who actually benefits

The obvious winners are cloud providers and large companies that already host DeepSeek models. They can now see exactly how DeepSeek tunes expert parallelism, speculative decoding and scheduling, and copy the parts that fit their own stack. The detailed Q&A, with settings like redundant experts per GPU and prefill length buckets, is arguably more useful to them than the code.

For everyone else, deepseek-recipe may be the more practical release. DeepSeek says its purpose is to give community deployments the same API behavior as the official service, which matters if you switch hosts and expect tool calls and reasoning output to parse the same way. The Q&A adds that it stops users from injecting special control tokens through plain text, a useful safety detail for anyone exposing a model to the public.

The Huawei support is also worth noticing. Shipping Ascend 950 as a first-class target, with its own builds of DeepEP and DeepGEMM, suggests DeepSeek wants its production stack to run on Chinese chips, not only on NVIDIA hardware.

What it means for Indian developers

Few teams in India will have 128 Blackwell GPUs to run DeepInfer the way DeepSeek does. If you just want cheap DeepSeek tokens, the hosted API remains the simple path. Our earlier look at DeepSeek V4.1 Flash 1017 covers its benchmarks and per-task cost.

The pricing detail that matters in India is timing. DeepSeek’s peak hours are 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays, which is 6:30 to 9:30 AM and 11:30 AM to 3:30 PM IST. Outside those windows, Flash costs $0.15 per million uncached input tokens and $0.60 per million output tokens (about ₹14 and ₹57), half the peak rate. Batch jobs scheduled for Indian evenings run at the cheaper rate.

Teams building on Indian GPU clouds or serving DeepSeek in-house should read the deployment Q&A even if they never run DeepInfer. And if you are weighing self-hosting a big open model in general, the hardware math looks a lot like what we found with Meta’s Muse Spark open release: open weights and open code do not mean cheap to run.

For a different take on getting more from a model without new hardware, see our breakdown of Google’s open-source Harness2.

The bottom line

DeepInfer is a genuine and unusually detailed release. DeepSeek has published working code for its production engine, a price-relevant claim you can roughly check, and a long document on how the system is run. That is more than most labs share.

But “fully open source” overstates it. The parts that let DeepSeek reuse cached context across days and spread load across hundreds of GPUs are still internal, the engine supports one model, and it wants the newest NVIDIA or Huawei hardware. Treat it as a blueprint plus a strong engine core, not a ready-made copy of DeepSeek’s service.

FAQ

What is DeepSeek DeepInfer?

DeepInfer is DeepSeek’s in-house inference framework, the software that runs DeepSeek V4.1 Flash for its API and apps. It was open-sourced on GitHub on 20 October 2026 under the Apache 2.0 license.

Is DeepInfer fully open source?

The engine code is, but the README says the KV cache system, prefill and decode disaggregation, and load balancing across replicas are not included. DeepSeek says those parts are tied to its internal infrastructure.

What hardware does DeepInfer need?

It supports NVIDIA SM100 Blackwell data center GPUs and Huawei Ascend 950 NPUs. DeepSeek says the smallest efficient deployment is 64 GPUs for prefill plus 64 for decode, though offline tests run on one 8-GPU node.

Which models does DeepInfer support?

Only DeepSeek-V4.1-Flash-1017 for now. DeepSeek says inference code for future models will be added to the same repository.

What is deepseek-recipe?

It is an MIT-licensed Rust library that converts API requests into model tokens and parses outputs back, matching the official DeepSeek API’s behavior so third-party hosts give the same results.