413 KV cache configs benchmarked: kvarn6 beat q8_0 on Qwen 3.6 27B while using 488 MiB less

A single-author KLD sweep across 413 KV cache configurations on Qwen 3.6 27B and Gemma 4 31B, run on BeeLlama.cpp v0.4.0, produced a usable recommendation ladder — and one finding that matters more than the ladder: the same cache quantization costs Gemma 4 31B roughly 40x more divergence than it costs Qwen.

A developer publishing as Anbeeld benchmarked 413 KV cache configurations and posted the results to r/LocalLLaMA on August 6, 2026. (Source: Anbeeld, 2026-08-06)

Key facts:

413 configurations were tested: 238 on Qwen 3.6 27B and 175 on Gemma 4 31B.
Both models ran Q5_K_S weights — Qwen at 64k context, Gemma at 16k.
The runner is BeeLlama.cpp v0.4.0, a fork of llama.cpp by the same author that adds KVarN cache types (kvarn2 through kvarn8, set via --cache-type-k / --cache-type-v), a precision tail (--kv-tail-tokens), and extra standard low-bit types q2_0 through q3_1 plus q6_0/q6_1.
KVarN is described by the author as Variance-Normalized KV-Cache, from Huawei, implemented in BeeLlama. It quantizes in groups of 128 tokens.
The precision tail keeps the most recent N tokens of the cache in (B)F16 while the older body stays quantized.
The metric is KLD against a matching BF16-cache baseline — the same model, same weights, only the cache changes.
At a 128-token tail on Qwen, kvarn6 records 0.000889 median KLD in 1696.00 MiB against q8_0’s 0.000901 in 2184.00 MiB — better divergence for 488 MiB less cache.
kvarn5 comes within 0.000014 of q6_0 (0.000927 vs 0.000913) while using 232.00 MiB less.
Gemma 4 31B is far more cache-fragile than Qwen. On the shared kvarn5-kvarn4 tail-1024 row, Gemma reads 0.036086 median KLD against Qwen’s 0.000936, and same-top agreement with its own BF16 baseline lands at 80.5% versus Qwen’s 97.976%.
Hardware and setup: one RTX 3090 on Windows 11, batch size 2048, micro-batch 512, seed 1, FlashAttention, unified KV, no mmap, mlock, no host offload, Wikitext-2 raw.

Scatter and line chart of total persistent KV cache size in MiB against median KLD versus the BF16 baseline for Qwen 3.6 27B at 64k context, comparing standard cache types at tail 0, 128, 1024 and 2048 with KVarN types at the same tails. — Qwen 3.6 27B, 4 bits and wider. The KVarN curves sit below the same-width standard points across the whole region — less divergence at equal or smaller cache. (Chart: Anbeeld, 2026-08-06)

What this means if you’re running models locally

1. The practical ladder is short and worth memorising. On Qwen at tail 1024: q8_0 at 2176 MiB is the safe standard; kvarn6 at 1744 MiB is the high-end value pick (0.000879 median KLD, actually below q8_0’s 0.000909); kvarn5-kvarn4 at 1360 MiB is the balanced default; kvarn4 at 1232 MiB beats q4_0 at 1248 MiB on both size and divergence. Below kvarn3 the numbers degrade fast — kvarn2 at 720 MiB records 0.003811 median and 0.450496 at the 99.9th percentile. That freed VRAM is the point: cache you don’t spend is context length or a larger weight quant you can.

2. The Gemma result is the actionable one. A ~39x divergence gap on an identical cache configuration means KV cache quantization advice does not transfer between model families. If you copied -ctk q4_0 -ctv q4_0 from a Qwen thread into a Gemma 4 31B setup, you are running a much lossier configuration than the person you copied it from — and nothing in your logs will say so. Re-tune per model, or stay at q8_0 on Gemma.

Chart of KV cache size in MiB against median KLD for Qwen 3.6 27B at 4 bits and narrower, showing curves for different precision tail lengths diverging sharply as cache size drops. — Qwen 3.6 27B, 4 bits and narrower. At these widths the gap between tail settings dwarfs the gap between cache formats — and the curves turn upward fast. (Chart: Anbeeld, 2026-08-06)

3. Read the 99.9th percentile column, not just the median. Gemma’s q8_0 row is 0.0371 median but 16.813929 at 99.9%. Median KLD tells you the typical token is fine; the tail tells you how badly the worst tokens break. For an agent loop where one wrong tool-call token derails a whole run, the tail column is the one that predicts your failure rate.

4. The author’s own caveat limits the precision-tail numbers. These are teacher-forced Wikitext scores, which grade the next token at every position — so the newest tokens are always decisive context, and even a 128-token exact suffix gets rewarded far beyond what real work would give it. In an agent session the load-bearing tokens are often an earlier instruction or a tool schema sitting well outside the tail. Size the tail to where your important tokens actually live, and treat short-tail rows as optimistic.

5. This is one person, one GPU, one dataset. It is a single RTX 3090 on Windows 11, one weight quant per model, Wikitext-2 only, and the KVarN implementation and the benchmark come from the same author. The method is published in full and the fork is open source, so it is checkable — but nobody has reproduced it independently yet. Verify on your own workload before you rewrite your launch flags.

Sources: Anbeeld, “KV Cache Quantization Benchmarks: KVarN, Precision Tail”, 2026-08-06 · BeeLlama.cpp on GitHub · r/LocalLLaMA discussion thread