Jul 16, 2026

How to Run Inkling Locally (Thinking Machines' 975B Apache-2.0 Multimodal Model)

Read time: ~8 minutes. Key facts:

Inkling is Thinking Machines’ first open-weights model, released July 15, 2026, under Apache 2.0.

975B total / 41B active Mixture-of-Experts (256 routed experts, 6 active per token, + 2 shared; DeepSeek-V3-style), 66 layers, 1M-token context.

It’s multimodal — text, vision, and audio — and posts strong numbers: 77.6% SWE-bench Verified, 97.1% AIME 2026, 87.2% GPQA Diamond, 91.4% VoiceBench (at effort 0.99).

There’s a lighter Inkling-Small preview: 276B total / 12B active.

The honest hardware math: the smallest usable quant (~1-bit) needs ~290 GB of memory; official inference via vLLM, SGLang, llama.cpp, transformers.

Sourcing note: specs, license, benchmarks, and inference support are from Thinking Machines’ official Inkling announcement and the thinkingmachines/inkling model card; GGUF quant sizes, memory guidance, and the llama.cpp command are from the Unsloth Inkling docs (third-party GGUF distribution, flagged as such). Benchmarks are Thinking Machines’ own. Links at the bottom.

Thinking Machines — Mira Murati’s lab — just shipped its first open-weights model, and it’s a big one in every sense: 975B parameters, Apache 2.0, multimodal, 1M context. Inkling isn’t pitched as the single strongest model; it’s pitched as a strong, permissively-licensed base to customize — which is exactly why builders want it locally. Here’s the honest path to running it, from “you have a 290 GB machine” to “you don’t, use the API.”

1. What Inkling actually is

From the official announcement and model card:

975B total parameters, 41B active — a Mixture-of-Experts model whose design “largely follows DeepSeek-V3”: each MoE layer has 256 routed experts and 2 shared experts, with 6 routed experts active per token, across 66 layers (hidden size 6144, vocab 201,024).
License: Apache 2.0 — fully open, commercially usable, no regional limits.
Multimodal — not just text: vision and audio too, aimed at coding, reasoning, agentic tool use, audio, and vision.
1M-token context (1,048,576), with 64K and 256K options on Thinking Machines’ Tinker fine-tuning platform.
A lighter preview, Inkling-Small: 276B total / 12B active — same family, far smaller footprint.

Thinking Machines is explicit that Inkling is “not the strongest overall model available today” — instead, “a combination of qualities makes it a good open-weights base for customization.” Apache 2.0 + open weights + multimodal + a fine-tuning platform is the pitch: a base you can own and adapt.

The MoE point is the usual one: only 41B of the 975B activate per token, so inference compute is modest — but you still have to hold (nearly) all 975B parameters in memory, which is the crux of §3.

2. What it can do: benchmarks

Thinking Machines’ reported results (their own numbers, at reasoning effort 0.99):

Benchmark	Inkling
SWE-bench Verified (coding)	77.6%
AIME 2026 (math)	97.1%
GPQA Diamond (reasoning)	87.2%
HLE with tools	46.0%
MMMU Pro (multimodal)	73.5%
VoiceBench (audio)	91.4%

Read it in context: 77.6% SWE-bench Verified is genuinely strong for an open-weights model, and the spread across coding, math, multimodal (MMMU Pro), and audio (VoiceBench) is the real story — this is a broad, balanced model, not a narrow specialist. For a customizable Apache-2.0 base, breadth matters more than topping any single leaderboard.

3. The hardware reality check (read before downloading 290 GB)

Here’s the honest math. Even heavily quantized, a 975B model is enormous. From Unsloth’s GGUF builds:

Quant	Disk	Memory (RAM + VRAM)
1-bit (UD-IQ1_S)	270–285 GB	~280–295 GB
2-bit	317 GB	~325 GB
4-bit	—	~600 GB
6/8-bit	—	~870 GB
BF16 (full)	1.9 TB	~1,900 GB

The rule of thumb: your combined RAM + VRAM should roughly cover the quant’s memory figure. So the smallest usable build (1-bit) wants ~290 GB — Unsloth notes it “will need a Mac Studio Ultra or machines with at least RAM+VRAM to be 290GB or so.” That’s a maxed workstation, not a laptop.

Bottom line:

256–512 GB Mac Studio Ultra or a big-RAM + multi-GPU server? §4 (llama.cpp) or §5 (vLLM/SGLang).
Want the model but not 290 GB of RAM? Look at Inkling-Small (276B/12B) in §6, or an API in §7.
A normal machine? §7 — use a hosted endpoint. This guide won’t pretend a 975B model fits in 24 GB of VRAM.

4. Method 1 — llama.cpp + Unsloth GGUF (big-memory local)

If you have the memory, run Unsloth’s GGUF quants through llama.cpp. Note the Unsloth GGUF is a third-party distribution (widely used, referenced from Inkling’s own model card via the llama.cpp PR, but not Thinking Machines’ own upload).

The command straight from the Unsloth docs (auto-download form):

./build/bin/llama-cli -hf unsloth/Inkling-GGUF:UD-IQ1_S \
    --temp 1.0 --top-p 1.0 --min-p 0.0

Or load the shards manually (note 00001-of-00007 — it’s a multi-part GGUF):

./build/bin/llama-cli \
    --model unsloth/Inkling-GGUF/UD-IQ1_S/inkling-UD-IQ1_S-00001-of-00007.gguf \
    --temp 1.0 --top-p 1.0 --min-p 0.0

Things to know:

Recommended sampling: --temp 1.0 --top-p 1.0 --min-p 0.0 (top_p 1.0 = effectively off). Use these; they match Unsloth’s guidance.
Reasoning effort is a dial. Inkling exposes effort levels via the chat template: none (0.0), low (0.2), medium (0.7), high (0.9), xhigh/max (0.99). Set it with:
```
--chat-template-kwargs '{"reasoning_effort":"high"}'
```
Use max/high for hard problems (the headline benchmarks are at 0.99), none/low for quick, cheap answers.
Context: up to 1,048,576 tokens — but don’t max it on a memory-constrained rig; the KV cache adds to memory you already can’t spare.
To serve an OpenAI-compatible endpoint instead of a CLI, use llama-server from the same GGUF.

5. Method 2 — Official vLLM / SGLang (GPU servers)

Inkling ships official inference support for vLLM, SGLang, llama.cpp, and transformers, pulling the real weights (thinkingmachines/inkling, which includes a BF16 and an NVFP4 checkpoint for FP4 inference). There’s an official vLLM recipe and PR. This is the route for a team self-hosting at higher precision on multi-GPU hardware — a 975B MoE at usable precision is a serious GPU footprint (4-bit alone wants ~600 GB), so plan capacity before committing. The NVFP4 checkpoint is the lever for fitting it into less memory on FP4-capable GPUs without hand-rolling a quant.

6. Method 3 — Inkling-Small (the lighter variant)

If 975B is out of reach but you still want to self-host, the Inkling-Small preview (276B total / 12B active) is the same family at roughly a quarter the total size. It won’t match the flagship’s numbers, but 276B/12B-active is far more tractable to serve — and it’s still an Apache-2.0, multimodal, customizable base. Run it through the same official stack (vLLM / SGLang / transformers). (At the time of writing, Unsloth’s GGUF quant table covers the full 975B model; for Small, start from the official weights via vLLM/transformers rather than assuming a GGUF exists yet.)

7. Method 4 — The pragmatic path: hosted APIs

For most builders, this is the right starting point — you don’t stand up 290 GB to evaluate a model. Inkling is available on multiple hosted providers from day one: TogetherAI, Fireworks, Modal, Databricks, and Baseten. Point your OpenAI-compatible tooling at one of them, validate that Inkling fits your workload, and only then decide whether self-hosting (or fine-tuning on Tinker) is worth the hardware.

The honest trade: local/self-host gives you data control and an Apache-2.0 base you fully own; the API gives you full-quality multimodal inference with zero hardware outlay. Given Inkling’s whole pitch is “a base to customize,” the sensible flow is API to evaluate → Tinker or self-host to customize once you’re sure.

8. Who this is for

Self-host (Method 1/2) if you already have a Mac Studio Ultra / big-RAM multi-GPU box, need on-prem control, or run enough volume that per-token cost dominates. The Apache-2.0 license is the point — a frontier-class, multimodal base you can legally build a product on.
Inkling-Small (Method 3) if you want to self-host but not at 290 GB.
Hosted API (Method 4) if you’re evaluating or running normal volumes — which is almost everyone. Full-quality Inkling, no memory bill.

The failure mode to avoid: downloading 285 GB, discovering it swaps to disk at a token per second, and blaming the model — when you just picked the wrong route for your machine. Match the route to your hardware.

The takeaway

Inkling brings a 975B / 41B-active, Apache-2.0, multimodal open-weights base to the table — strong across coding (77.6% SWE-bench Verified), math, multimodal, and audio, with a 1M context. Running it locally is real but heavy: the smallest usable quant is ~290 GB via Unsloth’s GGUF + llama.cpp, or the official vLLM / SGLang path (with an NVFP4 checkpoint for FP4 GPUs). Can’t spare 290 GB? Reach for Inkling-Small (276B/12B) or a hosted API (TogetherAI, Fireworks, Modal, Databricks, Baseten). Start on an API to validate, then self-host or fine-tune on Tinker once you’re committed — the Apache-2.0 license means the customized model is yours.

For other large open-weights models and the same honest-hardware calculus, see running GLM-5.2 locally and Kimi K2.7 Code locally; for a genuinely laptop-friendly option, Ornith-1.0 locally.

Sources

Introducing Inkling — Thinking Machines — 975B total / 41B active MoE (256 routed + 2 shared, 6 active/token, DeepSeek-V3-style), Inkling-Small 276B/12B, multimodal (text/vision/audio), 1M context, benchmarks (SWE-bench Verified 77.6%, AIME 2026 97.1%, GPQA Diamond 87.2%, HLE w/tools 46.0%, MMMU Pro 73.5%, VoiceBench 91.4% at effort 0.99), inference via vLLM/SGLang/llama.cpp/transformers, hosted on TogetherAI/Fireworks/Modal/Databricks/Baseten, fine-tuning on Tinker; released July 15, 2026
thinkingmachines/inkling — Hugging Face model card — Apache 2.0 license, BF16 + NVFP4 checkpoints, 66 layers / hidden 6144 / vocab 201,024, official vLLM + Unsloth deployment recipes, Llama Guard compatibility note
Inkling — Unsloth docs (Run Locally) — GGUF memory figures (1-bit ~280–295 GB, 2-bit ~325 GB, 4-bit ~600 GB, 6/8-bit ~870 GB, BF16 ~1.9 TB), llama.cpp llama-cli command, --temp 1.0 --top-p 1.0 --min-p 0.0, reasoning-effort levels none/low/medium/high/max (0.0–0.99), 1M max context, “Mac Studio Ultra or ~290 GB” for 1-bit. Third-party GGUF distribution.
Benchmark and spec figures are Thinking Machines’ own; hardware figures are Unsloth’s guidance. Verified July 16, 2026 — confirm current details on the official model card before relying on them.