NVIDIA Nemotron 3.5 Bolt 74B-A9B fits in 24GB: which llama.cpp recipe to use on a 32GB or 48GB GPU

NVIDIA announced the Nemotron 3.5 family expansion on October 15, 2026. The release includes the 120B Super VL for servers and two edge models. The one local builders care about is Nemotron 3.5 Bolt 74B-A9B 24GB, which is available now. Its smaller sibling, Bolt E4B 4GB, ships October 23. The question is practical: will it run on my GPU, and with which settings? The model card answers that directly.

What it is

Spec	Value (model card)
Parameters	74B total, 9B active
Architecture	Mamba-2–Transformer hybrid LatentMoE + C-RADIO v4-H vision encoder
Precision	IQ2 GGUF, 2-bit with quantization-aware training
Context	Up to 256K tokens
Inputs / output	Text, images, video / text
License	OpenMDW-1.1; the card says it is ready for commercial use

NVIDIA compressed it from Super VL 120B-A12B with its Star Elastic architecture search. Expert top-K drops from 22 to 9, and the search targeted a footprint under 24 GB at batch size 1 and 128K context. On Hugging Face, the main GGUF is about 22.0 GB and the vision mmproj file about 1.7 GB.

Pick a recipe by memory

The card gives two llama.cpp configurations. Both use a separate DFlash speculative-decoding drafter (the LiLiCorr GGUF):

Your hardware	Recipe	Context	Draft tokens	KV cache
RTX/DGX Spark 128GB or RTX GPU 48GB+	32GB	262,144	up to 7	default
RTX/DGX Spark 64GB or RTX GPU 32GB+	28GB	65,536	up to 6	q8_0 K and V

Both recipes use --parallel 1, so you get one concurrent sequence. If you need to serve more users, read our notes on scaling llama-server to multiple users first.

The 28GB recipe, as published for Windows (the Linux flags are the same):

llama-server --model "$MODEL" --mmproj "$MMPROJ" \
  --batch-size 2048 --ubatch-size 512 --ctx-size 65536 --parallel 1 -ngl 999 \
  --spec-type draft-dflash --spec-draft-n-max 6 \
  --spec-draft-model "$SPEC_DRAFT" --spec-draft-ngl 999 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --temp 1.0 --top-p 0.95 --flash-attn on --jinja

NVIDIA tells you to build the latest llama.cpp from source, with -DGGML_CUDA=ON. One catch: the card’s own Quick Start block uses --ctx-size 98304 with otherwise 28GB-style flags, while its recipe table says 65,536. If you are on a 32GB card and hit out-of-memory errors, use the table value.

Because the backbone is a hybrid Mamba-2 model, llama-server’s prompt-cache behaviour differs from that of a pure-attention model. If you see full prompt re-processing between agent turns, see our guide to llama-server context checkpoints for hybrid and SWA models.

NVIDIA’s blog also lists support in Ollama, LM Studio, Unsloth and vLLM. The card only documents llama.cpp, so the commands above are the only verified path today.

Benchmarks versus Qwen3.6-35B-A3B

These are NVIDIA’s own numbers from the model card, comparing the IQ2 Bolt with BF16 baselines:

Benchmark	Qwen3.6-35B-A3B	Bolt 74B-A9B (IQ2)
Terminal-Bench 2.1	44.38	50.98
GPQA Diamond	83.40	86.43
BrowseComp	48.74	66.75
PinchBench	88.07	87.82
AA-Omniscience non-hallucination	65.85	61.27

Bolt leads on most agentic and reasoning rows. Qwen narrowly keeps PinchBench and does better on the non-hallucination metric. NVIDIA also notes it measured Qwen’s MCP-Atlas score at 51.2 against a reported 62.8, so treat cross-vendor comparisons with care.

Bottom line

If you have a 32GB or 48GB NVIDIA GPU and want a local agent model with vision, Bolt is now a serious option. Pick the 28GB or 32GB recipe by memory, keep the DFlash drafter, and expect one sequence at a time.

Sources: NVIDIA blog · Bolt GGUF model card · NVIDIA technical blog