DeepSeek DeepInfer: open-source inference behind V4.1 Flash
On 20 October 2026 DeepSeek put DeepInfer on GitHub under Apache 2.0. The repo is the company's in-house inference framework. It is the stack behind the hosted V4.1 Flash API traffic, not a classroom demo build. The same day they opened deepseek-recipe, a Rust token encoder and decoder whose behavior matches the public DeepSeek API.
This note is for platform leads, ML infra owners, and procurement on programs that already pilot V4.1 Flash or plan on-prem inference. Facts below come from the DeepInfer README, the inference deployment Q&A, and DeepSeek's launch post on X. We have not built DeepInfer in our lab for this article.
What actually shipped
The first release targets offline inference with the features DeepSeek says it needs at scale: fused kernels, MoE expert parallelism with EPLB, speculative decoding (DSpark with confidence-scheduled verification), decoder SWA bounded replay, asynchronous batch scheduling, batch-invariant mode, deterministic sampling on NVIDIA, and structured outputs (regex, JSON schema, EBNF). The default checkpoint in the examples is DeepSeek-V4.1-Flash-1017.
Kernel work pulls in FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and TileLang ops from the same ecosystem DeepSeek has been open-sourcing for months. The README credits vLLM as the primary base, with ideas from SGLang and TensorRT-LLM.
Supported accelerators in this drop are NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Build targets Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2. That bar is intentional. This is datacenter inference, not a laptop experiment.
Production numbers DeepSeek published
In the Q&A, DeepSeek states that one cluster serving V4.1 Flash handles on the order of 155 trillion tokens per day, with overall revenue around US$38 per GPU-hour at valley pricing. Treat that pair as a snapshot of their economics, not a forecast for your rack. Your power, networking, and hit rate will differ.
They also spell out design goals that explain why the open repo looks the way it does: multi-vendor accelerators, prefill on ordinary 8-GPU nodes and decode on 64 to 128 GPU supernodes, and KV retention measured in days (agent traffic around 3 days TTL, chatbot around 7 days in their storage tier). Long cache life cuts repeated prefill when prefixes repeat. Agent loops that re-send the same repo tarball are the obvious OEM parallel.
Benchmark charts in the repo
DeepInfer ships Pareto curves for prefill (throughput vs latency on non-cached input) and decode (aggregate throughput vs per-user speed). Measurements use 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. Reproduction scripts live under examples/sweep/.
Wide expert parallelism matters for decode on MoE models. The Q&A includes EP8 vs EP64 offline data and argues that wide EP cuts per-step memory traffic in the routed experts when EPLB is tuned.
What is not in the tarball yet
The README lists three gaps for a full online service: a KV cache system, prefill/decode (and vision) disaggregation orchestration, and load balancing across data-parallel replicas. DeepSeek says most engine-side functionality for disaggregation is already in the tree, but the KV store and balancers are tied to internal infra. The Q&A document describes how those pieces work in production (3FS-backed block cache, tiered prefill buckets, decode routing that weighs concurrency and KV length). You can read the design even when you cannot clone the ops code.
Practical takeaway: expect weeks of systems work if your goal is parity with api.deepseek.com, not an afternoon docker compose up.
deepseek-recipe and why firmware teams should care
Model weights are only half of a stable agent pipeline. deepseek-recipe normalizes Chat Completions, Responses, and Messages shapes into one conversation format, encodes prompts in Rust with parallel segments around known special tokens, and parses streaming tool calls without letting user text spoof control tokens. That is the same class of bug you worry about when OTA JSON schemas meet LLM-generated configs.
If you self-host DeepInfer but tokenize differently from the API, your eval harness numbers will not transfer. Recipe is DeepSeek's answer: third-party deploys should match hosted token boundaries.
How this compares to vLLM or SGLang for an OEM program
Generic engines chase broad model support. DeepInfer is co-designed with V4.1 Flash: YOCO-style decoder replay env vars, FP4 KV formats, DSpark draft length defaults, and MoE EP widths that assume supernode decode. You gain throughput per dollar when you commit to that model family. You lose portability when your roadmap mixes several checkpoints.
Most connected-device programs we see still route inference through a cloud API and keep on-device ML bounded (wake word, vision front ends, control loops). DeepInfer matters when you operate a private cluster for CI agents, log triage, or supplier document extraction at Flash-scale volume. It does not replace firmware signing, factory test, or certification work.
Suggested next steps
- Run the stock
examples/offline_inference.pyon your target GPU class before you budget a multi-node EP layout. - Pair any self-host path with Recipe-backed tokenization and log hashes of prompts the way you log OTA artifacts.
- Keep a frontier API in the loop for customer-facing or legal review tasks until your harness proves parity on your PDFs and your repos.
- If agents must run on the device with tools and safety bounds, treat model hosting and harness design as one workstream. That is the Physical AI lane.
Sources: deepseek-ai/DeepInfer (README, doc/inference_deployment_qa.md, benchmark images); deepseek-ai/deepseek-recipe; DeepSeek social announcement 20 Oct 2026. We are not affiliated with DeepSeek. Revenue and token-volume figures are quoted from DeepSeek's published Q&A, not independently verified by Techwall.
