NVIDIA Dory1.0-4B is a looped LLM: pick 1, 2 or 4 loops per request, but you need unmerged vLLM or llama.cpp branches to run it
NVIDIA released Nemotron-Labs Dory1.0-4B on October 16, 2026, a 4B looped (latent-recurrent) model that reuses a 20-layer block to trade compute for accuracy without swapping checkpoints. Here is what the loop count changes, how last-loop KV caching keeps memory flat, and the exact vLLM and llama.cpp setups the model card gives while upstream support is still in open PRs.
NVIDIA released NVIDIA-Nemotron-Labs-Dory1.0-4B on October 16, 2026. It is a 4B-parameter “looped” model: instead of a fixed stack of layers, it runs a shared block of layers several times per token, and you choose how many times at serve time. (Source: NVIDIA model card)
Key facts:
- Architecture: a normalized-GPT with interleaved sliding-window and full-attention layers, split into an 8-layer input block, a 20-layer recurrent block and an 8-layer output block. The recurrent block repeats, with the input reinjected on every loop.
- Context: up to 256K tokens, English only.
- Formats: BF16, MXFP8, NVFP4 and a Q5_K_M GGUF, each in its own Hugging Face repo.
- License: OpenMDW-1.1, and the card says the model is ready for commercial or non-commercial use.
- Training: 6T pre-training tokens, the first ~4.5T in NVFP4 and the rest in BF16, then 300B tokens of context extension, SFT and GRPO reinforcement learning.
What the loop count buys you
One checkpoint covers several compute budgets. NVIDIA recommends two loops as the default, one loop when compute per token matters most, and four loops for accuracy-sensitive agentic work. NVIDIA’s own harness shows most of the gain arriving with the second loop:
| Benchmark | 1 loop | 2 loops | 4 loops | Qwen3.5-4B | Qwen3.5-9B |
|---|---|---|---|---|---|
| SWE-bench Verified | 50.3 | 59.5 | 61.8 | 54.7 | 61 |
| Terminal-Bench 2.1 | 12.5 | 18.1 | 22.2 | 15 | 24 |
| GPQA Diamond (no tools) | 69.19 | 72.41 | 72.16 | 77.71 | 80.93 |
| IFBench | 74.29 | 77.94 | 79 | 53.83 | 59 |
These are vendor-measured numbers, so treat them as a starting point. The card also says higher loop counts can reduce the number of generated tokens, so measure end-to-end latency and cost on your workload rather than assuming four loops is four times slower.
The memory trap, and the fix
By default each loop keeps its own KV cache, so cache memory grows with the loop count. Dory also supports last-loop KV caching, where each loop overwrites the previous loop’s entries and the cache stays at the one-loop size for any loop count. NVIDIA reports accuracy close to per-loop mode with nearly unchanged output lengths. If you plan to run four loops at long context or high concurrency, this is the setting to test first.
Running it today
Neither engine supports Dory in a release yet:
- vLLM: support is in vllm-project/vllm#60149, still open. The card installs vLLM from that PR head and sets loops through
--hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}'. Switch the mode to"last_loop"for the flat cache. Tool calling uses--tool-call-parser qwen3_coderand--reasoning-parser nemotron_v3. - llama.cpp: support is in ggml-org/llama.cpp#30306, also open. The card’s instructions pin a specific commit on a fork and target Apple Silicon Macs with Metal. Loops are set with
--override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop", accepting 1 to 4 loops.
The Q5_K_M GGUF is 3,673,288,864 bytes, and the card publishes a SHA-256 to check before launch. Its benchmark table is for BF16, not the quantized file. For coding agents, NVIDIA says to pass chat_template_kwargs with force_nonempty_content: True in each request.
Who should try it
If you want a small local model and can live with an unreleased engine build, Dory is an unusual option: one download that lets you choose per deployment how much compute to spend. If you need a stable release today, wait for the two PRs to merge.
For other recent architectures that needed fresh llama.cpp support, see Kolibri-1 in llama.cpp b11709 and NVIDIA’s Nemotron 3.5 Bolt 24GB recipes. For sliding-window cache behavior in llama-server, see our guide to context checkpoints on hybrid SWA models.