llama.cpp b11713 speeds up Q2_K and Q3_K decode on CUDA: up to 1.56x on an RTX 3090, little change on a 4090

On October 17, 2026, ggml-org merged PR #30517, “CUDA: faster MMVQ for low-bit types”, by am17an. It is the only change in llama.cpp build b11713, one commit after b11712. If you run Q2_K or Q3_K GGUFs on an older NVIDIA card, this is a free decode speedup. On an RTX 4090 or 5090 it is mostly a rounding error. (Source: llama.cpp PR #30517, merged 2026-10-17)

Key facts:

• What changed: the CUDA mul_mat_vec_q (MMVQ) kernel in mmvq.cu and the dot-product helpers in vecdotq.cuh (+125/−59 lines). The same code also builds for AMD GPUs through HIP.
• When it applies: MMVQ handles matrix multiplies with at most 8 columns (MMVQ_MAX_BATCH_SIZE 8). That covers single-stream token generation and very small micro-batches, not long prompt processing.
• Biggest measured win: Qwen3.5-9B Q3_K_S on an RTX 3090 went from 81.04 to 126.22 tokens/s in tg128 (1.56x).
• Unchanged: Q4_K and Q8_0 kernels stayed at 1.00x, and the PR uses them as its noise floor.

What the patch does

For single-token decode on the generic NVIDIA parameter table, five types now compute two output rows per CUDA block: Q1_0, Q2_K, Q3_K, IQ1_S and IQ1_M. A code comment in the merged commit explains why: low-bit types “load few weight bytes per thread”, so a second row per block shares the q8_1 activation loads and adds memory parallelism. GB10 (DGX Spark) gets the same treatment for IQ2_S and IQ3_S. The PR also discloses that it was “paired with Claude”.

Numbers from the PR

The author’s llama-bench runs (-fa 1 -ngl 99 -p 256 -n 128) show where the gains land:

GPU	Model and quant	tg128 before	tg128 after	Speedup
RTX 3090	Qwen3.5-9B Q3_K_S	81.04	126.22	1.56x
RTX 3090	Qwen3.5-9B Q2_K	102.67	141.02	1.37x
RTX 3090	Qwen3-30B-A3B Q2_K	165.28	191.53	1.16x
A16 (GA107)	Qwen3.5-9B Q3_K_S	14.43	24.95	1.73x
RTX 4090	Qwen3.5-9B Q3_K_S	162.65	174.93	1.08x
RTX 5090	Qwen3.5-9B Q2_K	269.25	279.27	1.04x

IQ2_XXS stayed at 1.00x to 1.01x on every NVIDIA card tested. A DeepSeek-V4-Flash IQ2_XXS + Q2_K mix on GB10 moved from 19.13 to 19.24 tokens/s. In short, Ampere cards running K-quants gain the most, while Ada and Blackwell were already close to their memory-bandwidth limit.

AMD results and one known regression

Reviewers tested the HIP build too. On a Radeon R9700, Qwen3.5-4B Q2_K tg128 rose from 129.6 to 138.0 tokens/s. Early MI50 tests showed Q3_K and Q6_K regressions of up to 16% at some batch sizes. Contributor thelittlefireman wrote a GCN-specific fix that is part of the merged commit. In that contributor’s later test of the patched branch, an RX 6800 (RDNA2) ran a Q2_K multiply with 8 columns about 21% slower (464.68 µs to 587.22 µs). An RDNA2-specific tweak was suggested in the thread but is not in the merged commit, so RX 6000 owners serving several parallel slots should benchmark before upgrading.

How to upgrade and check

Prebuilt archives can lag behind a new tag. If none are listed for b11713 yet, build from source with the documented CUDA flags:

git fetch --tags && git checkout b11713
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j 8
./build/bin/llama-bench -m your-model-Q3_K_S.gguf -fa on -ngl 99 -p 256 -n 128 -ub 1,2,4

Run the same llama-bench line on your previous build and compare tg128. Synthetic benchmarks can miss real-world regressions, as we saw with the b11425 CUDA prompt-processing fix. After that, measure end-to-end streaming in your actual server with our llama-server latency guide. If you squeezed a model into a small card with low-bit weights, the KV cache quantization benchmarks cover the other half of the memory budget.

Sources: llama.cpp PR #30517 · llama.cpp b11713 release · merged commit 2744e48 · llama.cpp build docs · llama-bench README