llama.cpp b11062 enables Qwen4 sparse Flash Attention — the published gain starts with long context

ggml-org published the pre-release llama.cpp b11062 on September 20, 2026. Its single release-note change enables a CUDA sparse Flash Attention path for Qwen4 through PR #28770. (Source: llama.cpp b11062 release notes, 2026-09-20)

Key facts:

• b11062 is a pre-release published on September 20, 2026.
• Its named change is “CUDA: enable sparse fa for qwen4.”
• The upstream implementation describes a 32,768-token Qwen4 context value.
• The author’s longest published prompt-processing case rises from 252.81 to 318.28 tokens/s.
• That is a 25.9% increase, calculated from one upstream DGX Spark result.

What this means if you serve long Qwen contexts

This is a prefill story, not evidence that every Qwen4 completion becomes faster. PR #28770 says the then-current Qwen4 attention path re-scored the entire KV cache, then adds a sparse Flash Attention path that selects a union of used tokens. The author says the code enables sparse attention at an internal long-context threshold; it is not documented as a new llama-server command-line flag. (Source: llama.cpp PR #28770, merged 2026-09-20)

The published matrix uses qwen4exp A3B IQ1_S at 1.5625 bits per weight on a DGX Spark. At a 10,000-token prompt distance, prompt processing rises from 615.70 to 663.17 tokens/s. That is an 7.7% increase. At 100,000, it rises from 252.81 to 318.28 tokens/s. That is an 25.9% increase. (Source: PR #28770 benchmark table, 2026-09-20)

Token generation moves too, but less in that same upstream table: 12.00 to 14.19 tokens/s at the 100,000-distance case. That is an 18.3% increase. Treat all three figures as the PR author’s single-machine measurements, not an independent benchmark or a guarantee for another quantization, GPU, build, or prompt pattern. (Source: PR #28770 benchmark table, 2026-09-20)

For an agent server, the practical question is whether your workload actually sends retrieved repositories, tool schemas, or chat histories into this range. First verify the installed build:

llama-server --version

If you compile locally for NVIDIA, llama.cpp documents -DGGML_CUDA=ON as the CUDA build switch. Upgrade a staging server, rerun the same long-prompt and token-generation workload, and compare time to first token separately from decode speed. (Source: llama.cpp build documentation, retrieved 2026-09-21)

Do not turn a nightly into a production rollout just for this result. It is especially relevant to Qwen4 long-context deployments, while the release does not establish a gain for Qwen3, Apple Metal, Vulkan, or short prompts. For repeated prompts, pair the upgrade with the llama-server KV-cache reuse guide. If several agents share one endpoint, use the llama-server concurrency guide to keep slot pressure visible.

Sources

• llama.cpp b11062 release notes
• PR #28770 — CUDA sparse Flash Attention for Qwen4
• llama.cpp CUDA build documentation