How to Run Nemotron 3.5 Bolt E4B Locally: llama.cpp, DFlash and Ollama Setup

NVIDIA released Nemotron 3.5 Bolt E4B 4GB on October 23, 2026, and you can run it today: download one 6.18 GB Q4_K_M GGUF, start llama-server with --jinja, and you have an OpenAI-compatible endpoint for a small model tuned for multi-turn tool use, instruction following and short, speech-ready replies. NVIDIA measured 3.1 to 5.0 GB of VRAM and 52 to 133 tokens per second on RTX cards. An optional 737 MB DFlash drafter speeds up generation. Watch out for three things: plain ollama run nemotron-3.5-bolt pulls the 25 GB 74B model, not E4B; Ollama gives you a 4K context by default on GPUs under 24 GiB; and the drafter’s own model card still contains an unverified placeholder command.

This guide covers the official llama.cpp recipe (with and without the drafter), the Ollama route, the chat-template switches for thinking and conversational mode, tool calling, and fixes for the problems you are most likely to hit. Every command and number below comes from NVIDIA’s model card, the DFlash drafter card, NVIDIA’s technical blog, the Ollama library page and the llama-server documentation.

Key facts (from NVIDIA’s model card and blog):

  • What it is: a small language model post-trained by NVIDIA from Google’s Gemma 4 E4B-it. It has 8B total parameters, 4B of them effective.
  • Weights: Q4_K_M GGUF (Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf, 6,180,446,208 bytes). A BF16 safetensors version is also available.
  • Context: up to 64K tokens of input and 64K of output, text in and text out.
  • Controls: enable_thinking on or off, a runtime thinking budget, and conversational_mode for concise spoken-style replies.
  • Recommended sampling: temperature 1.0 and top_p 0.95 in both thinking modes.
  • License: OpenMDW-1.1, which the card says allows commercial and non-commercial use. Gemma 4 itself is under Apache 2.0.
  • Status: GA on Hugging Face and build.nvidia.com on 10/23/2026.

If you want the bigger sibling instead, our Nemotron 3.5 Bolt 74B-A9B guide covers the 24 GB model and its two llama.cpp recipes. If you are coming from plain Gemma 4, the Gemma 4 12B local guide uses the same tooling.

1. Should you use Bolt E4B?

NVIDIA built Bolt E4B for one kind of job: fast local agents that talk to a person and take actions. Examples are voice and device assistants, in-game characters, and robot or IoT front ends. It is not pitched as a coding model. NVIDIA’s own comparison is with other ~4B models, and the results are mixed. That’s useful, because it tells you where it fits.

Here are selected BF16 scores from NVIDIA’s model card. NVIDIA measured all of them with its NeMo Gym harness, so treat them as vendor-reported.

Benchmark (what it tests)Bolt E4BGemma 4 E4B-itQwen3.5 4BNemotron 3 Nano 4B
MultiChallenge (consistency across turns)53.1042.0851.7635.30
IHEval (instruction hierarchy)84.7967.0460.5751.31
IFBench (precise output constraints)63.7942.8458.9160.10
When2Call multi-turn (call, clarify or abstain)63.3662.9037.8527.70
BFCL-v3 (correct tools and arguments)68.5166.3970.9561.10
Tau2-Bench (multi-turn customer tasks)55.1038.5075.0629.20
RULER 64K (long-context retrieval)86.9986.5595.1591.51
LongBench v2 48K50.5049.3459.9331.40

Pick Bolt E4B if your app cares most about following system-prompt rules, staying in persona, and knowing when to call a tool. Qwen3.5 4B still wins on NVIDIA’s own long-context and Tau2-Bench numbers. If your workload is long-document retrieval or long agent tasks, test both before you switch.

NVIDIA also says Bolt E4B produces “up to ~80% fewer tokens” than its peers in reasoning-enabled comparisons. For a voice assistant, shorter output means lower latency, which can matter as much as benchmark points.

2. Hardware: what “4GB” actually means

The name says 4GB, but the GGUF file is 6.18 GB on disk. The “4GB” refers to measured memory use in NVIDIA’s target workloads. The blog describes the model as packing its intelligence “into as little as ~4GB in its target workloads”. Here is what NVIDIA measured with the Q4_K_M file at batch size 1 on llama.cpp (CUDA). “Cache / ISL / OSL” means cached context, new input and output tokens.

HardwareCache / ISL / OSLVRAM (GB)Time to first tokenTokens/s
RTX 5070 (12GB)0 / 3K / 503.5548 ms133.4
RTX 5070 (12GB)32K / 2K / 2564.31,238 ms106.2
RTX Spark (32GB)0 / 3K / 503.1930 ms67.0
RTX Spark (32GB)32K / 2K / 2564.21,988 ms52.0
Jetson Orin Nano (8GB)0 / 3K / 504.25,785 ms15.5
Jetson AGX Orin (32GB)32K / 2K / 2565.04,091 ms20.3

What this means in practice:

  • Disk space: plan for about 6.2 GB for the model, plus 0.74 GB for the optional drafter.
  • GPU: an 8 GB card has comfortable headroom by NVIDIA’s numbers. That is the point of the model: it leaves VRAM for a speech-to-text model, a TTS model or a vision model in the same pipeline.
  • Context costs memory. The table stops at 32K of cached context. If you run the full 64K window (-c 65536, as NVIDIA’s recipe does), expect more memory use than the table shows, and measure on your own card.

3. Download the weights

NVIDIA’s quick start uses the Hugging Face hf CLI. If you don’t have it, install it with pip install -U huggingface_hub, which provides the hf command. Then run the card’s commands as written:

export MODEL_REPO=nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB
export MODEL_FILE=Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf

export DRAFTER_REPO=nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-DFlash
export DRAFTER_FILE=Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf

export MODEL_DIR="$PWD/bolt-models"

hf download "$MODEL_REPO" "$MODEL_FILE" \
  --local-dir "$MODEL_DIR"

hf download "$DRAFTER_REPO" "$DRAFTER_FILE" \
  --local-dir "$MODEL_DIR"

The repos aren’t gated, so you don’t need to accept a license prompt or log in. You can skip the second download if you won’t use speculative decoding.

Check the file names. The DFlash card’s own example uses NVIDIA-Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf and NVIDIA-Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf, with an NVIDIA- prefix. The files actually in the repos have no prefix: Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf and Nemotron-3.5-Bolt-E4B-DFlash-BF16.gguf. If you copy the DFlash card’s command, llama-server will fail with a file-not-found error. Use the names above.

4. Run it with llama.cpp (Docker, no drafter)

This is NVIDIA’s tested recipe. It uses the official llama.cpp CUDA 13 server image pinned to build b11459. That image tag exists on GHCR as of writing.

docker run --rm --gpus all \
  -p 8000:8000 \
  -v "$MODEL_DIR:/models:ro" \
  ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459 \
  -m "/models/$MODEL_FILE" \
  --alias nemotron-bolt \
  --host 0.0.0.0 \
  --port 8000 \
  -c 65536 \
  -ngl 999 \
  --jinja

What each flag does, per the llama-server docs:

  • -c 65536 sets the context to the model’s full 64K. Lower it, for example to -c 16384, if you’re short on VRAM. NVIDIA says typical target workloads use 2.5K to 16K input tokens.
  • -ngl 999 offloads all layers to the GPU.
  • --alias nemotron-bolt is the model name your API calls will use.
  • --jinja uses the model’s own chat template. You need it for enable_thinking, conversational_mode and tool calls. (Jinja is on by default in current builds, but keeping the flag is harmless and matches NVIDIA’s recipe.)

If you build llama.cpp yourself instead of using Docker, the same flags work with a local llama-server binary.

Test the server with the card’s request:

curl --fail-with-body -sS \
  http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nemotron-bolt",
    "messages": [
      {"role": "user", "content": "What is 17 multiplied by 23?"}
    ],
    "temperature": 0.0,
    "max_tokens": 512,
    "chat_template_kwargs": {
      "enable_thinking": true
    }
  }'

If curl returns “connection refused” or a 503 at first, the model is probably still loading. Our llama-server connection refused / 503 / 404 guide walks through each case.

5. Add the DFlash drafter for faster generation

DFlash is a speculative-decoding drafter: a 360M-parameter model with five decoder layers. Each step, it proposes up to 7 tokens at once using hidden states from the main model. The main model verifies them in one pass and keeps the longest valid run. The drafter card says the drafter “does not affect the output distribution of the base model”, so you get the same quality, just faster. NVIDIA’s blog says DFlash delivers “up to 21% higher throughput” than Gemma 4 E4B’s original MTP drafter. (Our Gemma 4 MTP guide covers that drafter.)

NVIDIA’s Docker recipe with the drafter:

docker run --rm --gpus all \
  -p 8000:8000 \
  -v "$MODEL_DIR:/models:ro" \
  ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459 \
  -m "/models/$MODEL_FILE" \
  --spec-draft-model "/models/$DRAFTER_FILE" \
  --spec-type draft-dflash \
  --spec-draft-n-max 7 \
  --spec-draft-ngl 999 \
  --alias nemotron-bolt \
  --host 0.0.0.0 \
  --port 8000 \
  -c 65536 \
  -ngl 999 \
  --jinja

The new flags:

  • --spec-draft-model loads the drafter. Its short form is -md.
  • --spec-type draft-dflash selects the DFlash method.
  • --spec-draft-n-max 7 drafts 7 tokens per step. The llama-server default is 3, so don’t drop this flag.
  • --spec-draft-ngl 999 puts the drafter fully on the GPU.

How much faster? The drafter card reports “acceptance length”, the average number of tokens produced per verification step (8 is the ceiling). These were measured on one RTX 5090 with thinking on and 4,096 output tokens:

Workload (MT-Bench)Greedy (t=0)Sampling (t=1)
Math5.875.67
Extraction4.894.60
Coding4.624.17
Writing2.982.92
Humanities2.722.52
Overall3.883.63

Structured output such as math, extraction and code gets the biggest boost. Free-form chat and roleplay, which are this model’s main jobs, get the smallest. The card adds that “the speedup from the draft model depends on the serving configuration and the quantization of the target model”. Benchmark with and without the drafter on your own prompts before deciding.

The version mismatch. The DFlash card says it supports llama.cpp b11808 (released October 22, 2026). Its sample command is marked [VERIFY: file names, quantization level, and flags after the llama.cpp test], so NVIDIA hadn’t finished checking it at launch. Meanwhile, the main card’s Docker recipe pins b11459. We could not find a server-cuda13-b11808 image tag on GHCR when we checked. If you build from source, b11808’s server docs do list draft-dflash under --spec-type. If you use Docker, start with NVIDIA’s b11459 recipe above. If the drafter fails to load, run without it and check back.

6. Or run it with Ollama

Ollama added E4B the same day. This command is the one most people will get wrong:

# Wrong for E4B: the untagged name pulls the 25GB Bolt 74B-A9B
ollama run nemotron-3.5-bolt

# Right: the E4B tag
ollama run nemotron-3.5-bolt:e4b

On the Ollama library page, nemotron-3.5-bolt:latest is the 25 GB 74B-A9B model from last week. E4B lives under these tags:

TagSizeNotes
nemotron-3.5-bolt:e4b7.2-7.9 GBDefault E4B pick
nemotron-3.5-bolt:e4b-q4_K_M7.9 GBSame quantization as NVIDIA’s GGUF
nemotron-3.5-bolt:e4b-q8_09.8 GBHigher precision
nemotron-3.5-bolt:e4b-bf1617 GBFull precision
nemotron-3.5-bolt:e4b-nvfp47.2 GBMLX
nemotron-3.5-bolt:e4b-mxfp812 GBMLX
nemotron-3.5-bolt:e4b-mlx-bf1617 GBMLX

Ollama’s page says E4B is supported on Windows, Linux and Apple Silicon. That’s notable because NVIDIA’s own card lists only Linux and NVIDIA GPUs. Ollama also lists a 128K context window and image input for these tags, while NVIDIA’s card says 64K and text only. Until those are reconciled, stick to text and stay within 64K, which is what NVIDIA evaluated.

Raise the context. Ollama’s docs say the default context is 4K on GPUs with less than 24 GiB of VRAM. That’s too small for tool definitions plus a conversation. Set it when you start the server:

OLLAMA_CONTEXT_LENGTH=16384 ollama serve

Or set it per request with "options": {"num_ctx": 16384} in the API body. Then run ollama ps and check the CONTEXT and PROCESSOR columns. You want your context size and 100% GPU.

The DFlash drafter is a llama.cpp feature. Ollama’s page mentions the companion drafter, but doesn’t document a way to turn it on. Use llama.cpp if you want speculative decoding.

7. Control thinking, budget and conversational mode

All three switches go through chat_template_kwargs. With the OpenAI Python client against llama-server:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
MODEL = "nemotron-bolt"

Thinking on (the default):

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "Write a haiku about GPUs"}],
    max_tokens=16000,
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "skip_special_tokens": False,
        "chat_template_kwargs": {"enable_thinking": True},
    }
)
print(response.choices[0].message.content)

Thinking off: pass "chat_template_kwargs": {"enable_thinking": False} instead. Use this for quick lookups and most voice turns. It cuts time to first word the most.

Conversational mode: pass "chat_template_kwargs": {"conversational_mode": True}. The card says the model then “responds concisely and without as much textual formatting”, which suits text-to-speech. It works with thinking on or off and with tool calling.

Thinking budget: this is a server flag, not a request field. Add these to the llama-server command:

  --reasoning on \
  --reasoning-budget 300

The llama-server docs define --reasoning-budget N as the token budget for thinking: -1 for unrestricted (the default), 0 to end thinking immediately. NVIDIA’s guidance:

  • A tight budget may cut reasoning off mid-sentence, so evaluate answers at your chosen budget.
  • Size it from real numbers. NVIDIA’s example: if answers average 200 tokens and you need a 500-token ceiling, set a 300-token reasoning budget with max_tokens at 500.
  • To keep the visible answer short, put an explicit limit in the system prompt, such as “limit responses to 1-2 sentences”. Vague requests like “be brief” don’t work as well.

Use NVIDIA’s recommended sampling (temperature 1.0, top_p 0.95) in production. The temperature: 0.0 in the curl test above is only for a repeatable smoke test.

8. Tool calling

Send tools in the standard OpenAI tools array and read message.tool_calls. This is the card’s example:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

response = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content": "What's the weather in Santa Clara?"}],
    tools=tools,
    max_tokens=16000,
    temperature=1.0,
    top_p=0.95,
    extra_body={
        "skip_special_tokens": False,
        "chat_template_kwargs": {"enable_thinking": True},
    },
)
print(response.choices[0].message.tool_calls)

If you serve the BF16 checkpoint with vLLM, the card says to add --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 to the serve command. The card doesn’t print a full vLLM command, so start from your usual vllm serve nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 and append those flags. Note that NVIDIA’s card lists A100, H100 and Blackwell for the PyTorch runtime.

Slow tools need a workaround. The chat template rejects “stray” tool messages, meaning any tool message that doesn’t directly follow the assistant message that made the call. If a tool takes a while (a long report, a robot action), NVIDIA’s pattern is:

  1. Right after the call, send a tool message with a pending status, for example {"status":"pending","task_id":"task_1","function":"build_report"}. The model can keep talking to the user.
  2. When the result arrives later, send it as a user message with a clear prefix, for example TOOL UPDATE: {"status":"completed","task_id":"task_1",...}.

If you send the late result as a second tool message instead, you will hit template errors.

9. Prompting tips from NVIDIA’s card

The card has an unusually detailed prompting section for agent and persona apps. The parts that matter most:

  • Put the role, rules and constraints in the system prompt under clear headings such as ROLE, TASK RULES and PERSONALITY.
  • Write rules as conditions and actions, for example “After crafting succeeds, call the delivery tool.”
  • Delete redundant instructions rather than repeating them in stronger words.
  • Never insert fake assistant turns or tool calls into the history.
  • Label ownership explicitly, for example YOUR INVENTORY versus PLAYER VIEW. Tell the model that unknown state is not the same as empty state.
  • Send state updates as separate user or system messages in the right order. Before the user message they condition, but after the tool result that changed them.

10. Troubleshooting

Ollama downloads 25 GB. You ran the untagged name. Stop it, then run ollama rm nemotron-3.5-bolt and ollama run nemotron-3.5-bolt:e4b.

The model forgets tools or the system prompt in Ollama. The default 4K context is truncating your prompt. Raise OLLAMA_CONTEXT_LENGTH or num_ctx, then check with ollama ps.

enable_thinking or conversational_mode has no effect. Make sure llama-server is using the model’s Jinja template (--jinja), not a built-in template passed with --chat-template. Also send the switches inside chat_template_kwargs, not as top-level fields.

The drafter fails to load. Check the file name (no NVIDIA- prefix) and your llama.cpp build. The drafter card names b11808, while NVIDIA’s Docker recipe uses b11459. Run without the drafter until your build loads it, because the model works fine on its own.

No speedup from the drafter. Expect little gain on chat and roleplay (acceptance length around 2.5 to 3 on MT-Bench). Make sure --spec-draft-n-max 7 is set, since the default is 3, and that the drafter is on the GPU (--spec-draft-ngl 999).

Out of memory at 64K. NVIDIA’s VRAM figures stop at 32K of cached context. Lower -c, or check our llama-server KV cache guide to stop re-processing long prompts on every turn.

FAQ

Is Bolt E4B free for commercial use? NVIDIA’s card says it “is ready for commercial or non-commercial use” under OpenMDW-1.1, with Gemma 4 under Apache 2.0. Read both licenses for your case.

Does it take images? NVIDIA’s card lists text in and text out. Ollama’s page lists image input for its E4B tags. NVIDIA hasn’t documented or evaluated image use for this model, so don’t rely on it yet.

Which languages does it support? The blog says 35 languages, building on a base model pre-trained on more than 140. The main post-training languages were English, French, German, Italian, Japanese, Spanish and Chinese.

Bolt E4B or Bolt 74B-A9B? E4B targets 4-5 GB of VRAM and fast replies for voice, persona and device agents. The 74B-A9B needs a 24 GB+ card and is the better pick for heavier reasoning and vision. See our 74B-A9B guide for its recipes.

For more local model setups, browse the 7minai tutorials.