BeeLlama.cpp is a llama.cpp fork that adds KV cache quantization types the upstream project does not have. The headline one is KVarN — variance-normalized KV cache — and it comes with a second feature, the precision tail, that changes how you should think about cache quantization entirely.

Sourcing note: every command, flag, default, and number below is quoted from the official BeeLlama.cpp README and its docs/beellama-args.md, with the KLD benchmark figures from the author’s published benchmark write-up. These are the project’s own measurements, not our own — we did not re-run them (the benchmarks are CUDA-based and our test machine is not). Verify flags against your build with --help, since this is an actively developed fork.

Key facts:

• KVarN offers six widths: kvarn2, kvarn3, kvarn4, kvarn5, kvarn6, and kvarn8. They are set with --cache-type-k and --cache-type-v, the same flags you already use in llama.cpp.
• The precision tail keeps the newest N tokens of the cache exact in F16 or BF16 while the older body stays quantized. It is enabled with --kv-tail-tokens.
• On Qwen 3.6 27B at 64K context, kvarn6 / kvarn6 with a 1024-token tail measures 0.000879 median KLD at 42.6% of BF16 cache size. Standard q8_0 / q8_0 measures 0.000909 at 53.1%. (Source: BeeLlama.cpp README, 2026-08)
• The fork also adds standard low-bit types upstream lacks: q2_0, q2_1, q3_0, q3_1, q6_0, and q6_1.
• Prebuilt binaries exist for macOS arm64 (Metal), Ubuntu (CPU, CUDA 12.4, CUDA 13.1, Vulkan, ROCm 7.2, SYCL), and Windows (CPU, CUDA 12.4, CUDA 13.1, SYCL, HIP/Radeon).
• Docker images ship at ghcr.io/anbeeld/beellama.cpp.
• KVarN is target-cache only. Draft models in speculative decoding must stay on standard cache types.
• KVarN and precision tails are explicitly not recommended for Gemma 4 in this release, because Gemma 4’s 1024-token sliding window means a 1024 tail promotes the entire SWA ring to BF16/F16 and erases the memory saving.

This is a tutorial about a third-party fork, and every number below comes from the maintainer’s own published benchmarks. I have not independently reproduced them, and I say so again at the end where it matters. What I have done is verify every command, flag, default, and constraint against the fork’s own README and argument reference, line by line.

Why KV cache quantization is the lever that actually matters

When you run a model locally, VRAM goes to two places: the weights and the KV cache. The weights are fixed once you pick a quant. The cache grows linearly with context length, and at long context it stops being a rounding error.

Concretely, from the fork’s own benchmark basis: Qwen 3.6 27B with Q5_K_S weights at 64K context has a BF16 KV cache of 4096 MiB. That is four gigabytes of VRAM spent on cache alone, on top of the weights. On a 24 GB card, cutting that cache in half is the difference between 64K context and 128K context, or between offloading layers to CPU and not.

The standard llama.cpp answer is -ctk q8_0 -ctv q8_0 or, when you are tight, q4_0. That works, and it costs quality in a way that is hard to see. The usual failure mode is not gibberish — it is a model that follows a long instruction slightly less reliably, or drops a detail from earlier in the context, in a way you attribute to the model rather than to your cache flags.

KVarN’s claim is that at the same bit width it loses less. The precision tail’s claim is orthogonal and, in practice, more interesting: most of what you lose to cache quantization, you lose on recent tokens, so keep those exact and quantize the rest harder.

What KVarN and the precision tail actually do

KVarN quantizes the cache in groups of 128 tokens, normalizing each group’s variance before quantizing it — that is the “variance-normalized” part of the name. The practical consequence, per the fork’s measurements, is that each KVarN width lands part way up the standard ladder rather than replacing a tier: kvarn4 approaches standard q5 quality, kvarn5 reaches the q6 tier, and kvarn6 reaches the practical q8 median-KLD floor.

The precision tail keeps the most recent tokens of the cache in exact F16 or BF16 while the older body stays quantized. Under FlashAttention, a single global softmax merges the quantized body and the exact tail without materializing the whole cache — so you do not pay a decompression pass for the privilege.

The two compose. The recommended Qwen starting point is not “kvarn6” or “a 1024 tail” — it is both.

One important detail about how the tail is sized: KVarN already keeps an intrinsic 128-token exact suffix even when you pass no tail flag at all. If you omit --kv-tail-tokens or set it to 0, KVarN retains that 128-token suffix; only standard caches fall back to the ordinary cache path at 0. So the question is never “tail or no tail” with KVarN — it is “128, or more.”

Step 1: install BeeLlama.cpp

Option A — prebuilt binaries (start here)

Release binaries are on the releases page. Pick the archive for your platform and backend:

Platform	Backend	Archive
macOS arm64	Metal	bin-macos-arm64.tar.gz
Ubuntu x64	CPU	bin-ubuntu-x64.tar.gz
Ubuntu arm64	CPU	bin-ubuntu-arm64.tar.gz
Ubuntu x64	CUDA 12.4	bin-ubuntu-cuda-12.4-x64.tar.gz
Ubuntu x64	CUDA 13.1	bin-ubuntu-cuda-13.1-x64.tar.gz
Ubuntu x64	Vulkan	bin-ubuntu-vulkan-x64.tar.gz
Ubuntu x64	ROCm 7.2	bin-ubuntu-rocm-7.2-x64.tar.gz
Ubuntu x64	SYCL	bin-ubuntu-sycl-x64.tar.gz
Windows x64	CPU	bin-win-cpu-x64.zip
Windows x64	CUDA 12.4	bin-win-cuda-12.4-x64.zip
Windows x64	CUDA 13.1	bin-win-cuda-13.1-x64.zip
Windows x64	SYCL	bin-win-sycl-x64.zip
Windows x64	HIP/Radeon	bin-win-hip-radeon-x64.zip

Windows CUDA users, read this or you will get a missing-DLL error: the Windows CUDA archives contain a ggml-cuda.dll backend but not the CUDA runtime. Download the matching cudart-win-cuda-*-x64.zip runtime archive and extract it into the same folder. Windows SYCL and HIP archives are standalone — all required runtime DLLs are bundled.

Option B — Docker

docker pull ghcr.io/anbeeld/beellama.cpp:server-cuda13

Available image tags:

Image	Acceleration	Platforms
server, server-cpu	CPU	linux/amd64, linux/arm64
server-cuda, server-cuda12	CUDA 12.4	linux/amd64
server-cuda13	CUDA 13.1	linux/amd64
server-rocm	ROCm	linux/amd64
server-vulkan	Vulkan	linux/amd64
server-sycl	SYCL	linux/amd64

Option C — build from source

The maintainer notes that building with -DGGML_NATIVE=ON may give a tiny bit better performance, so it is worth doing if you plan to use the fork long-term.

# Linux (GCC + CUDA)
cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON \
  -DGGML_CUDA_FA=ON \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

# macOS (Metal)
cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Two build flags matter specifically for this feature:

• The default CUDA FlashAttention build covers 50 standard cache pairs and 15 KVarN fast-decode pairs, including the homogeneous F16 and BF16 pairs that precision tails need. That covers every preset in this article.
• -DGGML_CUDA_FA_ALL_QUANTS=ON compiles all 169 standard and 36 KVarN pairs. You need this only if you want an asymmetric pair outside the default set. It costs a much longer build.
• -DGGML_CUDA_KVARN=OFF builds without KVarN kernels entirely — the opposite of why you are here.

If you are cross-compiling or building in CI without a GPU attached, add -DCMAKE_CUDA_ARCHITECTURES=86 for an RTX 3090 or =89 for an RTX 4090.

Step 2: the one command to start with

For Qwen 3.6, the fork’s documented starting point is:

llama-server -m model.gguf --flash-attn on \
  --cache-type-k kvarn4 --cache-type-v kvarn4 \
  --kv-tail-tokens 1024

--flash-attn on is not optional here. The precision tail merges the quantized body and the exact tail inside a single FlashAttention softmax; that is the whole implementation strategy.

If you have VRAM headroom and want the high-fidelity end instead of the value end, swap the width:

llama-server -m model.gguf --flash-attn on \
  --cache-type-k kvarn6 --cache-type-v kvarn6 \
  --kv-tail-tokens 1024

You can also let the fork pick the tail length:

llama-server -m model.gguf --flash-attn on \
  -ctk kvarn5 -ctv kvarn5 \
  --kv-tail-tokens auto

auto requests 1024 exact tokens per applicable target-cache group, capped by that group’s effective context or attention window. It is the same as writing 1024 for a normal dense model, and it is safer for models where the attention window is smaller than 1024.

Step 3: pick your preset from the ladder

These are the fork’s published Qwen 3.6 27B numbers, at 64K context, Q5_K_S weights, ranked by median KLD against a BF16-cache baseline. “Size vs bf16” is cache size as a percentage of the 4096 MiB BF16 cache.

Profile	K / V	Tail	Size vs bf16	Median KLD
Full baseline	bf16 / bf16	0	100.0%	0.000000
Standard q8	q8_0 / q8_0	0	53.1%	0.000909
KVarN6, no tail	kvarn6 / kvarn6	0	41.4%	0.000889
High fidelity	kvarn6 / kvarn6	1024	42.6%	0.000879
Standard q6	q6_0 / q6_0	0	40.6%	0.000960
KVarN5, no tail	kvarn5 / kvarn5	0	35.2%	0.000927
Balanced	kvarn5 / kvarn5	1024	36.3%	0.000897
Standard q5	q5_0 / q5_0	0	34.4%	0.001154
Standard q5 + tail	q5_0 / q5_0	1024	36.7%	0.000938
KVarN4, no tail	kvarn4 / kvarn4	0	28.9%	0.001111
Value	kvarn4 / kvarn4	1024	30.1%	0.000994
Standard q4	q4_0 / q4_0	0	28.1%	0.001846
Standard + tail	q4_0 / q4_0	1024	30.5%	0.001057
Compact	kvarn3 / kvarn3	1024	23.8%	0.001316

Three things to read out of that table.

The tail is the cheapest win in the whole list. Look at the two standard-cache rows: q4_0 with no tail is 0.001846 median KLD at 28.1% size. Add a 1024 tail and it becomes 0.001057 at 30.5%. That is a 43% reduction in median divergence for 2.4 percentage points of cache size — and it works on standard cache types, so you can take that win without adopting KVarN at all.

KVarN6 with a tail beats standard q8 on both axes. 0.000879 versus 0.000909 median KLD, at 42.6% versus 53.1% of BF16 size. If you are currently running -ctk q8_0 -ctv q8_0 on a Qwen model because it is the safe choice, this is a strict improvement on the fork’s own numbers.

Below kvarn3 you are trading quality for context on purpose. The table stops at kvarn3 at 23.8% for a reason. The narrower KVarN widths exist so you can fit context you otherwise could not, not because they are good.

A note on tail length: use 2048 primarily with q2/q3 caches, or when your workload specifically needs about 2K recent tokens kept exact. At q5 and above the Wikitext result mostly saturates after 1024, while larger tails can cost several percent of throughput.

Step 4: verify it actually applied

This is the step people skip, and it is the one that catches silent misconfiguration.

Read the startup log. BeeLlama’s startup logs show, per cache group: raw, requested, effective, and window lengths; the structural group ID; participating layers; the selected compact-overlay or compact-native-exact representation; actual body and exact types; logical history rows; rollback rows; graph-local body execution rows; and the owner backend. If your requested tail was silently capped or resolved to zero, it says so here.

Measure the cache directly with llama-bench:

llama-bench -m model.gguf --kv-memory \
  -ctk kvarn5 -ctv kvarn5 --kv-tail-tokens 1024

--kv-memory reports cache-owned bytes directly, broken out into kv_k_payload_bytes, kv_v_payload_bytes, kv_exact_history_bytes, kv_rollback_reserve_bytes, kv_staging_bytes, kv_padding_bytes, and kv_resident_bytes for persistent ownership. kv_transient_bytes is the observed reusable CUDA-pool high water mark, and kv_peak_bytes is resident plus transient.

The docs are explicit that these fields are more precise than deriving cache memory from whole-process VRAM — which is exactly what most of us do with nvidia-smi and then wonder why the numbers do not add up. Use the separate CUDA/WDDM fields for reconciliation and spill detection, not as the primary measurement.

Step 5: the flags in full

Flag	Default	What to know
-ctk TYPE, --cache-type-k TYPE	f16	Target K cache. Adds the six kvarnN values plus q6_0, q6_1, q3_0, q3_1, q2_0, q2_1.
-ctv TYPE, --cache-type-v TYPE	f16	Target V cache, same values and rules.
--kv-tail-tokens SPEC	0	For standard caches, 0 = ordinary cache path. For KVarN, omitted or 0 retains the intrinsic 128-token exact suffix.
--kv-tail-type TYPE	bf16 standard / f16 KVarN	Only f16 or bf16 are accepted. Other types are rejected.
--cache-type-k-swa TYPE	same as -ctk	KVarN-only, requires target KVarN, must be paired with the V override.
--cache-type-v-swa TYPE	same as -ctv	KVarN-only, requires target KVarN, must be paired with the K override.

The environment-variable forms are LLAMA_ARG_CACHE_TYPE_K, LLAMA_ARG_CACHE_TYPE_V, LLAMA_ARG_KV_TAIL_TOKENS, and LLAMA_ARG_KV_TAIL_TYPE — useful if you are running the Docker images and would rather not rewrite the command.

--kv-tail-tokens also accepts per-group forms. N0,N1 follows canonical group order. full=N,swa=N accepts role aliases or structural IDs such as full@l0. You need this only on models with mixed attention layer types.

Gotchas

1. A one-sided KVarN setting is silently upgraded. If only K or V is KVarN, the other side is promoted to the same KVarN width — with a warning. So -ctk kvarn6 -ctv q8_0 does not do what it looks like it does; you get kvarn6 / kvarn6. If you want a genuinely asymmetric cache, use two KVarN widths (-ctk kvarn6 -ctv kvarn5), not one of each family.

2. Your tail length gets rounded up, not down. KVarN rounds positive tail values upward to complete 128-token groups. Asking for 1000 gets you 1024. Explicit values are also capped by the group’s effective attention window and context capacity — so on a model with a small attention window, a large tail request quietly becomes a small one. The startup log tells you the effective value; check it rather than assuming.

3. An invalid tail spec fails soft, not loud. Invalid, duplicate, incomplete, or wrong-length specifications resolve additional coverage to zero. On KVarN you still keep the intrinsic 128-token suffix, so the model still works — you just silently lose the feature you thought you enabled. This is the single most likely way to think KVarN + tail is not helping when it was never on.

4. The tail type default differs by cache family. Standard caches default to bf16; KVarN defaults to f16. If the BF16 default lacks a complete Metal or SYCL route but F16 is complete, automatic selection warns and resolves once to F16. An explicit --kv-tail-type bf16 fails in that situation rather than silently downgrading — which is the behaviour you want, but it means an explicit flag can turn a working config into a startup error when you move machines.

5. Draft caches must stay standard. From the fork’s own DFlash example: keep the draft context on a standard cache type — KVarN is target-cache only. A working speculative-decoding invocation looks like this:

llama-server -m target.gguf --spec-type draft-dflash \
  --spec-draft-model drafter.gguf \
  --spec-draft-ngl all \
  --spec-dm-controller profit \
  --flash-attn on --cache-type-k q5_0 --cache-type-v q4_1

Draft and auxiliary contexts also do not inherit the target tail.

6. Vulkan has a hardware requirement. CUDA, ROCm/HIP, Vulkan, and CPU all consume compressed KVarN records directly in native FlashAttention paths — but Vulkan requires shader Int64 and buffer-device-address support for its direct route. Without it you fall back to the materialization route. Pre-Turing NVIDIA GPUs use CUDA’s portable rotated-domain path.

7. Saved state is version-gated. KVarN state version 13 stores logical compressed records and compact exact payloads independently of the ubatch workspace, so state can move between ub=128 and ub=512. Version 12 remains readable where compatible. Version 11 is rejected outright. If you rely on prompt-cache files across a fork upgrade, expect to regenerate them.

8. Increasing -ub does not buy you more exact coverage. KVarN’s physical staging depth is independent of the logical tail policy. A larger ubatch may increase transient work but never increases persistent exact coverage. Do not tune -ub hoping to improve cache fidelity.

The Gemma trap

This is the most important paragraph in the article for anyone running Gemma locally.

KVarN and precision tails are not recommended as a general Gemma optimization in this release. The reason is structural, not a bug: Gemma 4 uses a 1024-token sliding window, so a 1024-token precision tail promotes its entire SWA ring to BF16/F16. You keep the quality, you lose most of the memory advantage, and you have made your config more complicated for nothing.

The maintainer’s separate benchmark run makes the same point from the quality side. On the shared kvarn5-kvarn4 tail-1024 configuration, Gemma 4 31B reads 0.036086 median KLD against Qwen 3.6 27B’s 0.000936, and same-top agreement with its own BF16 baseline lands at 80.5% against Qwen’s 97.976%. KLD is measured against each model’s own baseline, so that is cache damage rather than a statement about which model is better.

The general lesson generalizes past this one fork: KV cache quantization settings do not transfer across model families. If you copied cache flags from a Qwen thread into a Gemma setup, you are running a materially lossier configuration than the person you copied from, and nothing in your logs will tell you.

For Gemma, stay on the standard ladder and stay high — q8_0 / q8_0 or q8_0 / q6_0.

If you do not want KVarN: the standard ladder

The fork also publishes a preset ladder for plain standard quants, useful if you want the fork’s extra types (q6_0 especially) without adopting KVarN. exp(-ΔKLD) here is a derived distribution-similarity proxy, not task accuracy.

K / V	% of bf16 size	99.9% exp(-ΔKLD)	What it is for
bf16 / bf16	100.0	100.00%	Preserving full quality
q8_0 / q8_0	53.1	94.62%	Validation and blame-isolation mode
q8_0 / q6_0	46.9	94.33%	Recommended high-end preset
q8_0 / q5_1	45.3	94.21%	Fallback if q6_0 V is unavailable
q8_0 / q5_0	43.8	93.69%	If the high-end rows miss the fit narrowly
q6_0 / q5_0	37.5	93.29%	Optional headroom tier between q5 and q8 K
q5_0 / q5_0	34.4	93.16%	Normal quality preset
q5_0 / q4_1	32.8	92.65%	Best default if VRAM-constrained
q5_0 / q4_0	31.3	91.39%	If q5_0 / q4_1 misses the fit narrowly
q4_0 / q4_0	28.1	88.87%	Memory saving with visible precision loss

Two practical notes from the type reference. q6_0 is a fork addition at 6.5 bits per value, described as a robust type for high-end presets — it does not exist upstream, and it is the single most useful non-KVarN reason to run this fork. And prefer q5_0 over q4_1 for K: q4_1 is smaller at 5 bpv versus 5.5, but weaker in the tail.

“Validation and blame-isolation mode” for q8_0 / q8_0 is the right way to think about that row. When output quality goes wrong and you do not know why, set both sides to q8_0 first to rule the cache out, then go back down the ladder.

When not to use this

Short contexts. If you run 8K context, your KV cache is a few hundred megabytes and none of this is worth a fork. Cache quantization pays off when the cache is a meaningful fraction of your VRAM budget.

Gemma 4 and other sliding-window models. Covered above. The window structure defeats the tail.

Production systems where a third-party fork is a liability. This is one maintainer’s fork. It tracks upstream, it is open source, and the benchmarks are published in full — but it is not llama.cpp, and your rollback story is “go back to upstream and lose the feature.”

When you have not measured your baseline. If you cannot say what your current cache costs in MiB and what your current output quality looks like on your own eval, changing cache types will just move an unmeasured number to a different unmeasured number. Run llama-bench --kv-memory on your existing config first.

An honesty note on the numbers

Every KLD figure in this article comes from benchmarks run by the fork’s own maintainer: one RTX 3090 on Windows 11, Q5_K_S weights, Wikitext-2 raw, -b 2048 -ub 512, seed 1, FlashAttention, unified KV, no mmap, mlock, no host offload. KLD is calculated against a matching BF16-cache baseline.

Two caveats the maintainer raises, which I am repeating because they are easy to lose:

These are teacher-forced Wikitext scores. That grades the next token at every position, so the newest tokens are always decisive context — and even a 128-token exact suffix gets rewarded far beyond what most real work would give it. In an agent session, the load-bearing tokens are often an earlier instruction or a tool schema sitting well outside the tail. Size the tail to where your important tokens actually live, and treat short-tail rows as optimistic.

The person who implemented KVarN is the person who benchmarked it. That is not an accusation — the method, the exact commands, the artifact hashes, and the full results are published, which is more than most benchmark posts offer. It does mean nobody independent has reproduced this yet. Verify on your own workload before you rewrite your launch flags for a fleet.

Sources

• BeeLlama.cpp on GitHub — README, feature list, install matrix, preset ladders
• BeeLlama argument reference — flag defaults, tail spec syntax, state versioning
• KV Cache Quantization Benchmarks: KVarN, Precision Tail — the 413-configuration sweep and full method
• KV Cache Precision Tail: Implementation and Benchmarks — exact commands, artifact hashes, percentiles

Related: 413 KV cache configs benchmarked · Fix llama-server KV cache reuse · Qwen 3.6 for local coding · How to run Gemma 4 12B locally · Speed up Gemma 4 with MTP