How to Stop llama-server From Throwing Away Your KV Cache (and Make Local Inference Fast Again)

Read time: ~7 minutes. What you’ll learn: why llama-server re-processes prompts you’ve already run (wasting prefill compute), what it does cache by default, and the exact official flags to fix each failure mode — --cache-reuse for KV shifting across shared chunks, --cache-ram + --cache-idle-slots to keep more prompt cache resident, and --slot-save-path plus the /slots save/restore API for cross-session persistence — plus the SWA gotcha that forces full reprocessing on some models and how to mitigate it.

Sourcing note: every flag, default value, and endpoint below is quoted from the official llama.cpp server README and the KV-cache-reuse tutorial discussion. No speed numbers are invented — the benefit of each flag is described in terms of the prefill work it avoids. Verify flags against your build (llama-server --help), since defaults change between releases. Links at the bottom.

You bought (or rented) the hardware, quantized the model, and got it running in llama-server — and it’s still slow to first token, every single time. The usual culprit isn’t the model or the GPU. It’s that llama-server is re-processing prompt tokens it already computed, throwing away a perfectly good KV cache. The good news: most of the reuse machinery already exists in llama.cpp; it’s just off, capped, or defeated by a model quirk. Here’s how to reclaim it.

1. What llama-server caches by default (and what it doesn’t)

First, the part that does work out of the box. Within a slot, llama-server honors the request parameter cache_prompt, which defaults to true:

cache_prompt: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. — official server docs

So if request B shares a prefix with request A on the same slot, only the new tail is prefilled. That covers the simple chat case: each turn appends to the last, so the growing prefix is reused.

The catch: this only helps when the new prompt shares an exact leading prefix with what’s already cached, on the same slot, in the same server lifetime. The moment any of those breaks — a system prompt changes mid-conversation, a shared chunk isn’t at the very front, the request lands on a different slot, or the connection closes and a new session starts — llama-server falls back to reprocessing. That’s the work we’re going to stop wasting.

(One caveat the docs call out: because logits aren’t guaranteed bit-for-bit identical across batch sizes, cache_prompt can make results slightly nondeterministic. For coding and agentic use that’s almost always a fine trade.)

2. Fix 1 — turn on --cache-reuse for shared chunks that aren’t at the front

The single most impactful flag most people never set is --cache-reuse N. From the server README:

--cache-reuse N — min chunk size to attempt reusing from the cache via KV shifting, requires prompt caching to be enabled (default: 0)

Default 0 means disabled. With it enabled, llama-server can reuse a cached KV slice even when the shared content isn’t a pure leading prefix — it detects a matching chunk of at least N tokens and shifts that cached KV forward into place instead of recomputing it. That’s exactly the case that defeats plain cache_prompt: RAG contexts where the retrieved passage moves, few-shot blocks that shift when you edit the instruction, or a system prompt that grows.

llama-server \
  --model your-model.gguf \
  --cache-reuse 256

256 is a sensible starting chunk size — large enough to avoid churn on tiny fragments, small enough to catch real shared blocks. (You can also set it per request via the n_cache_reuse body parameter, same meaning, same default of 0.) Turn this on first; for many workloads it’s the difference between constant reprefill and near-instant follow-ups.

3. Fix 2 — give the cache room with --cache-ram and --cache-idle-slots

Reuse only helps if the cache is still there. Two flags govern how much prompt cache llama-server keeps around:

  • -cram, --cache-ram N — “set the maximum cache size in MiB (default: 8192, -1 = no limit, 0 = disable).” The default 8 GiB cap means once your cached prompts exceed it, older entries get evicted and have to be reprocessed later. If you have the RAM and juggle several long contexts, raise it (or set -1 for no limit).
  • --cache-idle-slots / --no-cache-idle-slots — “save idle slots to the prompt cache on new task, and clear them when using unified KV (default: enabled, requires cache-ram).” This is what lets an idle conversation’s KV survive while another slot does work, instead of being dropped.
llama-server \
  --model your-model.gguf \
  --cache-reuse 256 \
  --cache-ram -1          # or a large MiB value; keeps more prompt cache resident

If you serve multiple chats or agents against one server and notice old conversations go cold (slow again after a gap), the default 8 GiB --cache-ram cap is the first thing to raise.

4. Fix 3 — survive a restart with --slot-save-path and the /slots API

Here’s the big one for anyone re-launching the server or reconnecting a client: by default, when a session ends the KV cache is gone, and the next session re-processes the entire system prompt and history from scratch. llama.cpp can persist a slot’s cache to disk so you don’t.

Start the server with a save directory:

llama-server \
  --model your-model.gguf \
  --slot-save-path ./kv-cache

Then use the official slot endpoints to save a warmed-up slot and restore it later — for example, after a restart or before a cold client reconnects:

# Save slot 0's prompt cache to a file (written under --slot-save-path)
curl -X POST "http://localhost:8080/slots/0?action=save" \
  -H "Content-Type: application/json" \
  -d '{"filename": "session-a.bin"}'

# Restore it into a slot later — no reprocessing of the cached prefix
curl -X POST "http://localhost:8080/slots/0?action=restore" \
  -H "Content-Type: application/json" \
  -d '{"filename": "session-a.bin"}'

# Erase a slot's cache when you're done with it
curl -X POST "http://localhost:8080/slots/0?action=erase" \
  -H "Content-Type: application/json" \
  -d '{}'

Restoring reads the cached KV back in instead of re-prefilling the whole prompt — so a long, expensive system prompt or document context is paid for once, then reloaded on demand. If you run an agent with a fixed multi-thousand-token system prompt, save that warmed slot once and restore it on every boot.

5. The gotcha: SWA and hybrid-memory models still reprocess

If you’ve done all of the above and a specific model still reprefills the whole prompt every request, you’ve probably hit the Sliding Window Attention (SWA) case. Several models use SWA or hybrid/recurrent memory, and llama-server will log something like “forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory).” It’s not your config — the cache layout for those architectures doesn’t support the same slice reuse.

Two mitigations exist in the server:

  • --swa-full — “use full-size SWA cache (default: false).” Allocating the full-size SWA cache trades more memory for the ability to retain and reuse it, rather than the compact windowed cache that gets invalidated.
  • -ctxcp, --ctx-checkpoints, --swa-checkpoints N — “max number of context checkpoints to create per slot (default: 32).” Context checkpoints let the server roll back to a saved point instead of reprocessing from zero on these models.

If a particular model is your reprefill offender, check whether it’s SWA-based before blaming your flags — and try --swa-full if you can spare the memory.

6. One more default worth knowing: --context-shift

For long, continuous generation, there’s also --context-shift / --no-context-shift — “whether to use context shift on infinite text generation (default: disabled).” When you blow past the context window, context shift drops the oldest tokens and continues instead of erroring or truncating hard. It’s not a cache-reuse feature per se, but it’s part of the same “don’t throw away the whole context” story, and it’s off by default — enable it if you do open-ended long generations.

Putting the reuse flags together for a typical local coding/agent server:

llama-server \
  --model your-model.gguf \
  -c 32768 \
  --cache-reuse 256 \
  --cache-ram -1 \
  --slot-save-path ./kv-cache
# add --swa-full only if your model is SWA-based and reprefills anyway
# add --context-shift if you do open-ended long generations

Then, in your client requests, leave cache_prompt at its default true. That combination gives you: prefix reuse within a slot (default), chunk reuse via KV shifting (--cache-reuse), more resident cache (--cache-ram), and cross-restart persistence (--slot-save-path + /slots).

The takeaway

llama-server isn’t slow because local models are slow — it’s slow because it re-processes prompt tokens you already computed, and most of the machinery to stop that ships off or capped. Turn on --cache-reuse for shared-chunk KV shifting, raise --cache-ram (and keep --cache-idle-slots on) so the cache survives, and use --slot-save-path with the /slots save/restore API to persist warmed contexts across restarts. If one model still reprefills, suspect SWA and try --swa-full. None of this is exotic — it’s all in the official server docs; it’s just not the default. Flip these on and your local inference stops paying the prefill tax twice.

For the models you’ll be serving this way, see GLM-5.2 locally, Kimi K2.7 Code locally, and Qwen 3.6 for local coding.

Sources

  • llama.cpp server README — ggml-org/llama.cpp — --cache-reuse N (default 0, KV shifting, requires prompt caching); cache_prompt request param (default true, prefix reuse, nondeterminism caveat); n_cache_reuse body param (default 0); -cram, --cache-ram N (default 8192 MiB, -1 = no limit, 0 = disable); --cache-idle-slots / --no-cache-idle-slots (default enabled, requires cache-ram); --slot-save-path PATH (default disabled) and POST /slots/{id}?action=save|restore|erase; --swa-full (default false); -ctxcp, --ctx-checkpoints, --swa-checkpoints N (default 32); --context-shift / --no-context-shift (default disabled)
  • Tutorial: KV cache reuse with llama-server — ggml-org/llama.cpp Discussion #13606 — walkthrough of prompt-cache reuse and KV shifting behavior
  • Server forces full prompt re-processing (SWA/recurrent memory) — Issue #21831 — the SWA / hybrid-memory full-reprocessing case
  • Flags and defaults verified against the master server README on July 6, 2026. Defaults change between releases — confirm with llama-server --help on your build before relying on them.