Ling 3.1 Flash (560B MoE) locally: which GGUF fits 256, 384 or 512 GB, and how the four-DGX-Spark llama.cpp RPC setup works
inclusionAI's Ling-3.1-flash is a 560B-total, 25B-active hybrid-linear MoE under MIT. Community GGUFs run from 131 GB (IQ1_S) to 608 GB (Q8_0), so no single 128 GB machine can hold it. Here is the memory math, why two DGX Sparks are tight and four work, and the llama.cpp RPC commands for splitting it across machines.
A post on r/LocalLLaMA on October 18, 2026 reported Ling 3.1 Flash loaded across four NVIDIA DGX Sparks with llama.cpp’s RPC backend and Atomic Chat’s Q4 GGUF. The post had no speed numbers. It still raises a useful question: how much hardware does this model actually need, and what does a multi-machine llama.cpp setup look like?
What Ling-3.1-flash is
The official model card (MIT license) describes a “native hybrid reasoning model” with:
| Spec | Value |
|---|---|
| Parameters | 560B total, 25B active per token |
| Context window | 1M tokens |
| Layers | 63 KDA (linear attention) + 9 Gated MLA, a 7:1 ratio |
| Experts | 512 routed, 1 shared, 8 active |
| Hidden size | 4096 |
inclusionAI’s own recipes target datacenter GPUs: SGLang on 8× H200 (--tp 8 --ep 8) or 4× GB300 (--tp 4 --ep 4), and vLLM with --tool-call-parser ling3 --reasoning-parser ling3. Recommended sampling is temperature=1.0, top_p=0.95, top_k=20 with thinking enabled. If you only want to try it, OpenRouter lists inclusionai/ling-3.1-flash at $0.075 per million input tokens and $0.225 per million output tokens, with a 262,144-token context.
GGUF sizes: nothing fits in 128 GB
File sizes from the Hugging Face repos:
| Quant | AtomicChat | bartowski |
|---|---|---|
| IQ1_S | 141.8 GB | 131.0 GB |
| IQ2_M | 218.1 GB | 213.0 GB |
| IQ3_M | 277.1 GB | 297.3 GB |
| Q4_K_M | 354.4 GB | 358.0 GB |
| Q8_0 | 600.8 GB | 608.4 GB |
Even the smallest 1-bit file is bigger than a single 128 GB DGX Spark, a 128 GB Mac, or a 128 GB RTX Spark laptop. So unlike DeepSeek’s SSD-streaming route in our ds4 128 GB Mac guide, Ling 3.1 Flash needs more memory, either in one box or spread across several.
AtomicChat’s card maps total memory to files: 192 GB for AD-IQ1_M, 256 GB for AD-IQ2_M, 384 GB for AD-Q4_K_M, 512 GB for AD-Q5_K_M, 640 GB for AD-Q8_0. Its “AD” files keep attention, the shared expert, the router and dense layers at q8_0 or higher and push only routed experts low. On its held-out test, AD-Q4_K_M agreed with BF16 on the top-1 token 96.86% of the time. The 1-bit files managed only 86 to 88%, and the card calls them experimental. The card says the files need llama.cpp build b11571 or newer.
Context is cheap. Because only 9 of 72 layers keep an attention cache, the card estimates about 0.5 GB per conversation plus about 10 KB per token. The hybrid layout also means llama-server prompt-cache reuse behaves like other recurrent models. Our context checkpoints guide explains why you may see full prompt re-processing.
Why two Sparks are tight and four work
NVIDIA’s DGX Spark page lists 128 GB of LPDDR5x unified memory with 273 GB/s of bandwidth and a ConnectX-7 NIC at 200 Gbps. NVIDIA rates two Sparks (256 GB) for models up to 400B parameters and four (512 GB) for up to 700B. With two Sparks, your realistic ceiling is a 2-bit file around 213–218 GB, before the OS and runtime take their share. With four, the ~354 GB Q4_K_M splits to roughly 89 GB per node, leaving headroom.
Splitting it with llama.cpp RPC
The RPC README covers the setup. On each worker, build with RPC and CUDA enabled, then start the server bound to the private interface:
cmake -B build -DGGML_CUDA=ON -DGGML_RPC=ON
cmake --build build --config Release
build/bin/ggml-rpc-server -H 10.0.0.2 -p 50052 -c-c turns on a local tensor cache in $HOME/.cache/llama.cpp/rpc, so later loads don’t push hundreds of gigabytes over the network again. On the main host, list the workers with --rpc:
llama-server -m AD-Q4_K_M/Ling-3.1-flash-AD-Q4_K_M-00001-of-00009.gguf \
-ngl 99 -c 32768 --jinja \
--rpc 10.0.0.2:50052,10.0.0.3:50052,10.0.0.4:50052By default, weights and KV cache are spread across devices in proportion to free memory. --tensor-split overrides that. The README also describes an RDMA transport for RoCEv2-capable NICs such as Mellanox ConnectX on Linux, which falls back to TCP when either side can’t use it. Run with GGML_RPC_DEBUG=1 (plus -lv 5 on the client) to see what the handshake negotiated.
Two warnings. The README calls the RPC backend a fragile, insecure proof of concept, and the server prints a warning whenever it binds to anything other than 127.0.0.1. Keep it on an isolated link. Also, nobody has published tokens-per-second numbers for this four-Spark setup yet, so measure your own before you plan around it.