Tools · · 2 min read

A 60-character Metal shim makes llama.cpp 7-16x faster inside macOS VMs — and it works on Muse Glimmer 30B

Cua published benchmarks on August 11, 2026 showing llama.cpp in an Apple Virtualization.framework guest running 11.08x faster prompt processing and 16.36x faster token generation on TinyLlama, and 7.55x/8.87x on Muse Glimmer 30B. The fix is not GPU passthrough — it is changing two capability answers so llama.cpp stops picking the slow Metal kernel.

Cua published benchmarks on August 11, 2026 showing that llama.cpp inside a macOS virtual machine had been leaving 7-16x of Metal performance on the table for a reason that has nothing to do with virtualization overhead. (Source: Cua, “Apple Silicon and macOS VMs: Faster LLM Inference with llama.cpp”, 2026-08-11)

Key facts:

  • TinyLlama 1.1B Chat Q4_K_M: prompt processing goes from stock-guest speed to 4,786.70 tok/s, an 11.08x gain — 98.25% of bare metal. Token generation hits 206.60 tok/s, a 16.36x gain, or 72.06% of bare metal.
  • Gemma 4 12B QAT Q4_0: prompt processing 71.66 → 515.76 tok/s (7.20x, 99.59% of bare metal). Token generation 3.41 → 49.67 tok/s (14.54x, 94.82% of bare metal).
  • Muse Glimmer 30B Q4_K-M in a 64 GiB guest: 194.971 tok/s prompt processing (7.55x) and 21.0823 tok/s generation (8.87x).
  • MLX-LM shows negligible change. The fix is specific to how llama.cpp selects Metal kernels.
  • All numbers are 10-run medians on one host: M1 Ultra, macOS 26.6.1 host, Tahoe guest, Lume 0.5.1, llama.cpp b10359.
Cua benchmark card headlined Apple Silicon and macOS VMs, 11 to 16 times faster LLM inference with llama.cpp. It shows Gemma 4 12B QAT Q4_0 llama.cpp 10-run medians: prompt processing 7.2 times faster, stock VM 71.66 tok/s versus unlocked 515.76 tok/s; token generation 14.5 times faster, stock VM 3.41 tok/s versus unlocked 49.67 tok/s; 99.6 percent of host prompt speed and 94.8 percent of host generation speed.
The Gemma 4 12B result. Note the bottom line: after the fix, the VM reaches 99.6% of host prompt speed. (Card: Cua, 2026-08-11)

The title says passthrough; the mechanism is a lie detector

Nothing is being passed through. The host GPU, Apple’s virtualization bridge, and the guest’s paravirtualized Metal device are all identical before and after. What changes is the answer one guest process gets when it asks the GPU what it can do.

Stock, a macOS guest reports Apple family 5 and a 32 KB maximum threadgroup memory. Cua’s shim answers Apple family 9 and 64 KB for a single process. That flips on SIMD-group matrix operations, SIMD-group reduction, and bfloat16 — and llama.cpp, which branches on exactly these capability answers at kernel-selection time, stops picking its conservative fallback path.

Diagram titled Metal execution path, from conservative capability answers to faster Metal kernels. Host Apple GPU, Virtualization.framework bridge, and guest paravirtualized GPU are all marked unchanged. A capability query for one process branches into two outcomes: the stock guest answer of Apple 5 and 32 KB, with SIMD-group matrix unavailable and conservative feature selection, leading to a slower llama.cpp kernel; and the shim tested profile of Apple 9 and 64 KB, with SIMD-group matrix and bfloat16, leading to a faster llama.cpp kernel.
Everything below the capability query stays the same. Only the answer changes, and only for one process. (Diagram: Cua, 2026-08-11)

That explains the MLX-LM result too. MLX does not gate on those same capability answers, so it gains nothing — which is a useful sanity check that the speedup is real kernel selection and not measurement error.

What this means if you’re building with local models

Sandboxed agents just got cheap on Apple Silicon. The reason to run llama.cpp in a VM is isolation: give a coding agent a disposable macOS guest, snapshot it, let it break things. Until now that cost you an order of magnitude of inference speed, so most people ran the model on the host and accepted the blast radius. At 94-99% of bare metal for prompt processing, that tradeoff largely disappears.

Check whether you have been benchmarking a crippled guest. If you have ever measured a local model inside a macOS VM and concluded Apple Silicon virtualization was hopeless for inference, the number you got was a kernel-selection artifact. Gemma 4 12B generating 3.41 tok/s is not a virtualization tax — it is the wrong kernel. Re-run before you write off the setup. Our Gemma 4 12B local guide and Muse Glimmer 30B guide both assume bare-metal hosts; the VM numbers here now land close enough that the guides transfer.

Prompt processing recovers better than generation. Look at TinyLlama: prompt processing reaches 98.25% of bare metal, but token generation only 72.06%. On the larger models the generation gap narrows (94.82% for Gemma 4 12B), but if your workload is generation-heavy on a small model, expect to keep paying some of the tax.

The caveats are load-bearing

Cua is direct about the limits, and they matter more than usual here:

  • It uses private Metal APIs. This is not a supported Apple configuration and can break on any macOS update.
  • It is per-process only, applied via DYLD_INSERT_LIBRARIES, and does not affect hardened executables — so it will not work on signed apps with hardened runtime.
  • Validation is narrow. One host configuration, one GPU family. There is no evidence yet about M2/M3/M4 hosts or other guest versions.
  • You still pay Virtualization.framework overhead. The gains close the kernel-selection gap, not every gap.

The setup also requires flipping a host-level default (ForceUnrestrictedDeviceFeatureLevel) before the guest boots. Treat it as an experiment on a machine you can reimage, not something to put under a production workload this week. If you want the safe version of local inference, running Muse Glimmer 30B directly on the host remains the boring, supported path.

Sources

Source: Cua