Read time: ~11 minutes.

Key facts:

• DeepInfer is the inference framework behind DeepSeek’s own production service, open-sourced on October 20, 2026 under Apache 2.0 (GitHub). The package version is 0.1.0, marked “Development Status :: 4 - Beta” in pyproject.toml.
• It supports one model, DeepSeek-V4.1-Flash, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Hopper (SM90) is not supported in the official tree; a community pull request for H20-3e is open but not merged (PR #92).
• This release does offline (batch) inference. DeepSeek says a KV cache system, prefill/decode disaggregation and data-parallel load balancing are required for a full service and are not included (README).
• DeepSeek says a single cluster running this stack serves 155 trillion tokens per day, with revenue of US$38 per GPU-hour at off-peak (“valley”) pricing (deployment Q&A).
• The default checkpoint, DeepSeek-V4.1-Flash-1017, is about 510 GB in 48 safetensors shards (Hugging Face repo metadata).

What we checked: we read the DeepInfer README, build.sh, scripts/build_vendor.sh, the requirements files, pyproject.toml, examples/README.md, examples/offline_inference.py, examples/_config.py, examples/_launch.py, examples/_structured_output.py, eval/README.md and the deployment Q&A, plus the deepseek-recipe README and the Hugging Face model metadata. We do not have SM100 or Ascend 950 hardware, so we did not run these commands ourselves. Every command, flag and environment variable below comes from those files.

If you run DeepSeek-V4.1-Flash on your own Blackwell nodes, DeepInfer is the first time you can use the exact engine DeepSeek uses in production instead of a third-party port. The practical takeaway: on a single 8-GPU SM100 node you can build it with one script and run batch inference with python examples/offline_inference.py --ep 8. For an online API you still need your own serving layer, because the KV cache store and load balancers stay internal.

This guide walks through hardware checks, the build, your first offline run, picking expert/data/context parallel sizes, the main feature switches (speculative decoding, deterministic sampling, structured outputs), benchmarking, and the errors you are most likely to hit.

If you don’t have SM100 GPUs, this is the wrong starting point. Our DeepSeek V4 Flash API guide covers the hosted route, and the 128 GB Mac GGUF write-up covers the community quantized route. For the checkpoint itself (size, precision, the missing chat template), see our V4.1-Flash-1017 open weights note. More builder tutorials are on the 7minai homepage.

---

1. What DeepInfer is (and what this release leaves out)

DeepSeek describes DeepInfer as “the high-performance inference framework that powers DeepSeek’s large-scale production services.” The README lists these features in the open-source release:

• Optimized kernels, model and sampler runtime, built on DeepSeek’s own libraries: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
• Parallelism: attention data parallel (DP) or balanced context parallel (CP), and MoE expert parallel (EP) with expert-parallel load balancing (EPLB).
• DSpark, DeepSeek’s speculative decoding with confidence-scheduled verification.
• SWA bounded replay, which cuts decoder prefill work by replaying only a short suffix.
• Asynchronous (zero-overhead) batch scheduling, on by default.
• Batch-invariant inference and deterministic sampling (NVIDIA only).
• Structured outputs: regex, JSON schema or EBNF grammar.

The acknowledgments say DeepInfer “was originally built primarily on vLLM,” with inspiration from SGLang and TensorRT-LLM. That explains why the Python API (an LLM class, SamplingParams, llm.generate) will look familiar if you have used vLLM.

What you don’t get in this release, per the README:

Needed for a full service	In this release?
Offline batch inference engine	Yes
KV cache system (DeepSeek uses a 3FS-backed store)	No
Vision-encoder / prefill / decode disaggregation	No (“most required engine functionality is already implemented”)
Load balancing across data-parallel replicas	No

The deployment Q&A says the KV cache management and load balancing components “are tightly coupled to our internal infrastructure, so that code is not yet fully open-sourced.” DeepSeek also says inference code for future DeepSeek models will continue to be released in DeepInfer.

---

2. Check your hardware and software first

From the README’s prerequisites:

Requirement	NVIDIA	Ascend
Accelerator	SM100 GPUs	Ascend 950 NPUs
OS	Linux	Linux
Compiler	GCC 13.0 or newer	GCC 13.0 or newer
Python	3.12	3.12
Toolkit	CUDA 13	CANN 9.2

SM100 is NVIDIA’s Blackwell data-center architecture. DeepSeek’s FlashMLA README, for example, quotes its SM100 kernel numbers on B200. Hopper cards (H100, H200, H20) are SM90 and are not supported by the official tree today. PR #92 reports DeepSeek-V4.1-Flash running end to end on 8x and 4x H20-3e (141 GB). It is a community contribution that is still open, so treat it as experimental.

Quick checks on an NVIDIA node:

nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv
gcc --version | head -1
python3.12 --version
nvcc --version | tail -2

You want compute capability 10.0 (that is SM100), GCC 13+, Python 3.12 and CUDA 13. Other Blackwell variants report different values, and the README lists only SM100.

Disk: the checkpoint is about 510 GB. Make sure your Hugging Face cache (or the directory you download into) has room for it plus the build artifacts.

How many GPUs: offline_inference.py defaults to --ep 8, which means one 8-GPU node. The README’s published benchmarks used 4 SM100 GPUs for prefill and 64 for decode. For production, the Q&A says “the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode,” with prefill at EP2–8 and decode at EP64–128. You do not need that scale to try the engine, only to match DeepSeek’s throughput.

---

3. Build DeepInfer

Start from an empty Python 3.12 environment. The README asks for this explicitly, because build.sh pins exact versions of torch, transformers and the kernel libraries.

git clone https://github.com/deepseek-ai/DeepInfer.git
cd DeepInfer

python3.12 -m venv .venv
source .venv/bin/activate

# Use `ascend` instead on Ascend 950
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Keep DEEPINFER_ACCELERATOR exported in every shell where you run DeepInfer. The evaluation README notes it “selects the platform build of the extension.”

What ./build.sh does, in order (from the script itself):

1. Exits with a usage message unless DEEPINFER_ACCELERATOR is nvidia or ascend.
2. Runs pip install -r requirements-nvidia.txt (or requirements-ascend.txt). On NVIDIA this pins torch==2.13.0+cu132 from the PyTorch CUDA 13.2 index and flash-attn-4==4.0.0b34, on top of the shared requirements.txt (which pins transformers==4.50.0, tilelang==0.1.15, tile-kernels==2.1.0, xgrammar==0.2.2, huggingface_hub==0.36.2 and others).
3. On NVIDIA, force-reinstalls NCCL: pip install --no-deps --force-reinstall nvidia-nccl-cu13==2.32.3.
4. Runs scripts/build_vendor.sh, which initializes the git submodules and builds wheels for DeepSelect, FlashMLA, DeepEP and DeepGEMM (or the -Ascend variants of DeepEP and DeepGEMM), then installs them from a local wheel directory.
5. Runs scripts/build.sh build to compile DeepInfer’s own extension.

It finishes with ✅ DeepInfer is successfully built. You don’t need a recursive clone, because the vendor script runs git submodule update --init for you. It also rewrites git@github.com: URLs to HTTPS, so you don’t need SSH keys for GitHub.

---

4. Get the checkpoint

The examples default to deepseek-ai/DeepSeek-V4.1-Flash-1017. On first run, examples/_config.py calls huggingface_hub.snapshot_download and stores it in your Hugging Face cache.

For a 510 GB checkpoint, downloading ahead of time is usually easier to manage. This uses the same snapshot_download call the examples use, with the huggingface_hub version that build.sh already installed:

python -c "from huggingface_hub import snapshot_download; \
snapshot_download('deepseek-ai/DeepSeek-V4.1-Flash-1017', local_dir='/models/DeepSeek-V4.1-Flash-1017')"

export DEEPINFER_HF_MODEL_DIR=/models/DeepSeek-V4.1-Flash-1017

Avoid upgrading huggingface_hub inside the DeepInfer environment just to get a newer CLI, since requirements.txt pins it to 0.36.2.

How the examples resolve the model, from _config.py and examples/README.md:

• --model_name can be a Hugging Face model id or a local directory containing config.json plus safetensors. A local directory is used directly “with no network access.”
• DEEPINFER_HF_MODEL_DIR only replaces the default model. It “does not override an explicitly passed --model_name.”
• If the directory has no config.json, you get FileNotFoundError: HF checkpoint requires config.json.

The model card ships the prompt encoder as encoding/encoding.py inside the checkpoint instead of a Jinja chat template. The DeepInfer eval scripts load that file at runtime, so keep the checkpoint directory intact.

---

5. Run your first offline inference

On one 8-GPU node:

export DEEPINFER_ACCELERATOR=nvidia
python examples/offline_inference.py

This is the same as --ep 8. The script runs DeepSeek’s built-in sample prompts from examples/_prompts.py. The examples README describes these as text-only and image samples that use the bundled carrots.jpeg and ribs.jpeg, so no external image cache is needed. For each sample it prints one line like:

DP <rank>; INPUT: ...; OUTPUT: ...; output_tokens=...; accept_rate=...; decode_ppl=...

accept_rate is the share of speculative draft tokens that were accepted. decode_ppl is perplexity over the generated tokens.

The engine settings baked into offline_inference.py are useful to know before you adapt it:

Setting	Value in the example
max_model_len	32768
max_num_batched_tokens	32768
max_num_seqs	4
gpu_memory_utilization	0.85
--sampler_force_topk	4096
Speculative draft length (DEEPINFER_USE_NEXTN_DECODE)	5 unless you set it
Extra redundant experts (DEEPINFER_OVERRIDE_NUM_EXTRA_EXPERTS)	32 unless you set it (a lower bound; the engine rounds up so the total is divisible by ep_size)
DEEPINFER_BATCH_INVARIANT	set to 1 on NVIDIA unless you set it

Other flags: --disable_eplb turns off expert load balancing. --long_context switches to 16,389 input tokens and 4,096 output tokens per sample. --check_structured_output runs regex, JSON and EBNF checks after generation (section 7). Run python examples/offline_inference.py --help for the full list.

---

6. Choose EP, DP and CP

The README’s three example layouts:

python examples/offline_inference.py --ep 8
python examples/offline_inference.py --ep 2 --dp 8
python examples/offline_inference.py --ep 2 --cp 8

The rules, from offline_inference.py:

• If you don’t pass --dp, DP defaults to ep // cp, with a minimum of 1.
• World size is dp × cp, and it must be divisible by ep. Otherwise the script fails on assert world_size % ep_size == 0.
• The launcher starts one process per data-parallel rank and gives each rank cp consecutive GPUs, starting from DEVICE_OFFSET (default 0).

In plain terms:

• EP (expert parallel) splits MoE experts across GPUs. DeepSeek’s production decode uses wide EP (64–128) because it “substantially reduces per-step memory traffic in the MoE expert part.”
• DP (attention data parallel) runs independent attention replicas. DeepSeek uses #DP = #EP in production.
• CP (context parallel) splits one long sequence across GPUs, which helps time to first token on long prompts. The production Q&A says “TP and CP are not used for now” in DeepSeek’s own service, so treat CP as an option for long-context experiments.

Multi-node: the launcher reads RANK (node index) and WORLD_SIZE (number of nodes), defaulting to 0 and 1. It assigns each node an equal share of the DP ranks, so DP must be divisible by the node count. The benchmark docs state the parallel scale as ep_size = WORLD_SIZE × len(CUDA_VISIBLE_DEVICES).

---

7. Turn features on and off

All of these are environment variables or SamplingParams fields documented in the README.

DSpark speculative decoding

export DEEPINFER_ENABLE_MTP_VARLEN=1     # confidence-scheduled, variable-length verification
export DEEPINFER_USE_NEXTN_DECODE=5      # max draft length (default 5); 0 disables speculation

DSpark predicts each draft token’s acceptance chance with a confidence head and spends verification compute on the most promising tokens. In DeepSeek’s EP64/DP64 benchmark it verified about 30% fewer draft tokens per step (5.00 down to 3.2–4.4) with almost the same accepted length (3.29 vs 3.20). DeepSeek calls that “+4–7% throughput at the production operating point (~60 concurrent sequences per rank).” At 16 sequences per rank, its own table shows a 1.5% loss, so the gain depends on load.

SWA bounded replay

export DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact
export DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140

These are the README’s default V4.1 settings. The retained suffix “must cover the SWA window plus the maximum backtracking length.” The Q&A explains the idea: decoder layers recompute only the last 128 tokens of the prompt instead of rebuilding the full sliding-window cache, which DeepSeek says is “nearly halving total prefill computation.”

Batch-invariant inference and deterministic sampling (NVIDIA only)

# Same results regardless of how requests are batched together
export DEEPINFER_BATCH_INVARIANT=1

# Fully deterministic sampling: also turn off variable-length verification
export DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0

For deterministic sampling the README also says to pass a fixed seq_seed to SamplingParams. That matters for evals and regression tests where you want identical output for identical input.

Structured outputs

Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams. schema_type is regex, json or ebnf. This is the pattern from examples/_structured_output.py:

import json
from deepinfer.sampling_params import SamplingParams

schema = {
    "type": "object",
    "properties": {
        "color": {"type": "string", "enum": ["red", "green", "blue"]},
        "count": {"type": "integer", "enum": [1, 2, 3]},
    },
    "required": ["color", "count"],
    "additionalProperties": False,
}

params = [
    SamplingParams(temperature=0, max_tokens=128,
                   so_args={"schema_type": "regex", "schema": r"[A-Z]{2}-[0-9]{3}"}),
    SamplingParams(temperature=0, max_tokens=128,
                   so_args={"schema_type": "json", "schema": json.dumps(schema)}),
    SamplingParams(temperature=0, max_tokens=128,
                   so_args={"schema_type": "ebnf",
                            "schema": 'root ::= "answer=" ("yes" | "no") ";level=" [1-3]'}),
]
outputs = llm.generate(prompts=my_prompts, sampling_params=params, use_tqdm=False)

Note that the JSON schema goes in as a string (json.dumps), not a dict. To check your build, run the bundled test, which fails the run if any constraint is violated or generation doesn’t end on EOS:

python examples/offline_inference.py --check_structured_output

Writing your own batch script

The simplest path is to copy examples/offline_inference.py and replace prepare_offline_prompts(...) with your own prompts and SamplingParams. Keep its structure, because _config.py warns that environment setup must happen before import deepinfer. That is why create_llm() imports DeepInfer inside the function, after it sets CUDA_VISIBLE_DEVICES. Keep the launch_dp(main, dp_size) entry point so each data-parallel rank gets its own process. On Ascend, the same file notes that importing torch early initializes ACL and breaks ASCEND_RT_VISIBLE_DEVICES.

To turn real chat requests into V4.1 prompts, use DeepSeek’s deepseek-recipe (MIT, pip install deepseek-recipe, Python 3.10+). It converts Chat Completions, Responses and Messages requests into DeepSeek V4/V4.1 prompts or token IDs, and parses output, including tool calls, back into those formats:

from deepseek_recipe import ChatCompletionRequest, ConversionOptions, DeepseekV41Encoding

request = ChatCompletionRequest({
    "model": "DeepSeek-V4.1-Flash",
    "messages": [{"role": "user", "content": "Hello"}],
})
converted = request.convert(ConversionOptions())
rendered = DeepseekV41Encoding().render_conversation(converted.conversation)
print(rendered.prompt)

DeepSeek says its processing logic “is fully aligned with the official DeepSeek API,” so it is the piece to reach for if you build your own HTTP layer in front of DeepInfer.

---

8. Benchmark your node

Set the devices you actually have, then run the benchmark script:

export WORLD_SIZE=1 RANK=0
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

python examples/benchmark.py                                   # defaults
python examples/benchmark.py --cp 8 --prefill_batch_size 65536 # CP8, bigger prefill chunk
python examples/benchmark.py --decode_num_requests 1           # low-latency, batch 1
python examples/benchmark.py --decode_num_requests 64 --context_length 131072 --yoco_clip

Defaults: decode_num_requests=64, context_length=32768, non_cached_tokens=4096, prefill_batch_size=32768, nextn 5. Set DEEPINFER_USE_NEXTN_DECODE=0 to benchmark without speculative decoding. --prompt_mode text or --prompt_mode image picks the input type.

When it finishes, rank 0 prints three numbers:

• prefill throughput in tokens/s/GPU
• decode avg_SPS in steps/s
• decode throughput in tokens/s/GPU, computed as avg_SPS × decode_num_requests × (1 + nextn) / --cp

Read the third number carefully. It uses “full-acceptance accounting,” meaning it counts 1 + nextn tokens per request per step “regardless of the true acceptance rate,” so it overstates real output speed when drafts are rejected. Add --result_jsonl results.jsonl to save the raw quantities. Use PROFILE=1 to capture a trace, but the docs say step times inside the trace include profiler overhead of “up to nearly 50% inflation at bsz=1.” Use traces to see which kernels run, not to measure speed.

For throughput-versus-latency curves like the ones in the README:

bash examples/sweep/run_sweep.sh decode ep8-decode-128k 131072 3 0.75
bash examples/sweep/run_sweep.sh prefill ep8-prefill-128k 131072

Results land in sweeps/<tag>/results.jsonl and sweeps/<tag>/<mode>_pareto.png (plotting needs matplotlib). The docs suggest putting the parallel scale in the tag name, because it isn’t visible in the command.

---

9. Run the bundled evals

eval/ runs offline Pass@1 evaluation for AIME 2026 (30 tasks, 32 samples each), HLE (2,500 tasks, 342 with images) and GPQA Diamond (198 tasks, 8 samples each). Generation and scoring are separate stages, so you can re-score with a different judge without regenerating. Each benchmark has a run.sh driver under eval/aime/, eval/hle/ and eval/gpqa/. No extra build is needed beyond ./build.sh, but the checkpoint directory must contain encoding/encoding.py.

For quick correctness checks, the repo also stores perplexity baselines in tests/models/offline_decode_ppl.json. The full test run is python -m pytest -v -s tests. The long-context PPL test alone is python -m pytest -v -s tests/models/test_offline_inference_long.py.

---

10. Troubleshooting

Usage: DEEPINFER_ACCELERATOR=nvidia|ascend ./build.sh The variable isn’t exported in this shell. Run export DEEPINFER_ACCELERATOR=nvidia (or ascend) and rerun. Keep it exported when you run examples and evals too.

Dependency conflicts during the build build.sh pins exact versions, including a CUDA 13.2 torch build and a forced NCCL 2.32.3 reinstall. Installing into an existing environment with other torch or transformers versions is the usual cause. Start from the empty Python 3.12 environment the README asks for.

Kernels fail to build or load on H100/H200/H20 Those are SM90 GPUs. The official tree targets SM100 and Ascend 950. PR #92 adds an SM90 path, and related open PRs (#88, #91, #101) cover pieces of Hopper support, but none were merged when we checked.

FileNotFoundError: HF checkpoint requires config.json --model_name or DEEPINFER_HF_MODEL_DIR points at the wrong level of the directory, or the download is incomplete. Point it at the folder that holds config.json and the 48 shards.

DEEPINFER_HF_MODEL_DIR seems ignored It only applies when you use the default model id. If you also pass --model_name, the flag wins.

AssertionError on world_size % ep_size == 0 Your --ep, --dp and --cp don’t fit together. World size is dp × cp and must be divisible by ep. With no --dp, DP is ep // cp.

--cp rejected in the benchmark In benchmark.py, --cp “must divide ep_size,” and ep_size comes from WORLD_SIZE × len(CUDA_VISIBLE_DEVICES). Check both environment variables.

Wrong GPUs or CUDA_VISIBLE_DEVICES has no effect Don’t import torch or deepinfer before create_llm() runs in your own script. The example code sets device visibility first for this reason. Use DEVICE_OFFSET to start from a GPU other than 0.

Saving a candidate PPL baseline fails on multiple nodes --output <path> is supported on a single node only. With multiple nodes, use the default path so each node updates only its own ranks.

Hard-to-read errors on Ascend The examples README suggests export ASCEND_LAUNCH_BLOCKING=1 (run operators synchronously) and export ASCEND_SLOG_PRINT_TO_STDOUT=1 (print CANN logs to stdout).

---

FAQ

Can I serve an OpenAI-compatible API with DeepInfer today? Not out of the box. This release is an offline batch engine. DeepSeek says a KV cache system, disaggregated serving and DP load balancing are needed for a full service and aren’t included. You can pair it with deepseek-recipe and your own HTTP layer, or use the hosted DeepSeek API.

Does it run other models, like DeepSeek-V4-Pro? The README lists only DeepSeek-V4.1-Flash. DeepSeek says inference code for future models will be added to DeepInfer.

Can it run on a single GPU or a Mac? No. It needs SM100 GPUs or Ascend 950 NPUs, and the examples default to 8-way expert parallelism. For smaller machines, see the community GGUF route in our 128 GB Mac write-up.

Is it faster than vLLM or SGLang? DeepSeek publishes its own Pareto curves for DeepInfer (4 SM100 GPUs for prefill, 64 for decode) and a reproduction script, but no head-to-head numbers against other engines. Run examples/benchmark.py and the sweep scripts on your hardware to compare.

How does DeepSeek get US$38 per GPU-hour? That figure is DeepSeek’s reported revenue per GPU-hour for its own service at off-peak pricing, from the deployment Q&A. It reflects DeepSeek’s full stack, including the KV cache store (about 100 PiB of SSD) and load balancers that are not in this release, so don’t expect to reproduce it from the open-source engine alone.

What license is it under? DeepInfer is Apache 2.0. The DeepSeek-V4.1-Flash-1017 weights are MIT, and deepseek-recipe is MIT.

---

Sources

• deepseek-ai/DeepInfer on GitHub: README, build.sh, examples/README.md, offline_inference.py, _config.py, _structured_output.py, eval/README.md
• DeepSeek V4.1 Flash Inference Deployment Q&A
• DeepInfer PR #92: SM90 (Hopper) support
• deepseek-ai/deepseek-recipe and deepseek-recipe on PyPI
• deepseek-ai/DeepSeek-V4.1-Flash-1017 on Hugging Face
• FlashMLA README