How to Run Kimi K3 Locally (2.8T Open Weights, 104B Active, Native MXFP4)

Read time: ~8 minutes. Key facts:

  • Kimi K3’s open weights are out — moonshotai/Kimi-K3 on Hugging Face, 96 safetensors shards, under a custom “Kimi K3” license (not MIT/Apache — read it).
  • 2.8T total / 104B activated, MoE with 896 experts, 16 selected per token + 2 shared, 93 layers (69 KDA + 24 Gated MLA), 1M context, vocab 160K.
  • Multimodal — text + image, via a MoonViT-V2 vision encoder (401M params).
  • Native MXFP4: quantization-aware training from the SFT stage on, MXFP4 weights + MXFP8 activations.
  • Officially recommended engines: vLLM, SGLang, TokenSpeed. No official GGUF.
  • The gotcha: K3 always thinks, and you must pass reasoning_content back in multi-turn/tool calls or it breaks.

Sourcing note: every spec, engine, and code sample below is from the official moonshotai/Kimi-K3 model card (fetched July 29, 2026) and the HF API. Community GGUF repos exist but are flagged as such — they are not Moonshot-published. Links at the bottom.

We covered K3 on launch day when only the API was live and promised a local guide once the weights shipped. They’ve shipped — so here’s the real one, built from the official model card rather than launch-week speculation.

1. What actually shipped

The official card’s spec table:

ArchitectureMixture-of-Experts
Total parameters2.8T
Activated parameters104B
Experts896, 16 selected per token, 2 shared
Layers93 (1 dense)
Attention69 KDA + 24 Gated MLA, 96 heads, hidden dim 7168
ActivationSiTU-GLU
Context length1,048,576 (1M)
Vocabulary160K
Vision encoderMoonViT-V2 (401M params)
ModalityText + image

Two corrections worth making versus launch-week reporting: the activated-parameter count is 104B (a number that wasn’t public at announcement), and the attention stack is explicitly 69 KDA + 24 Gated MLA layers — Kimi Delta Attention isn’t the whole story.

License: the weights are under a custom “Kimi K3” license (license_name: kimi-k3), not MIT or Apache 2.0. Read the license file before building a commercial product on it — this is the one spec people assume and get wrong.

2. Native MXFP4 — why this matters for local

Most giant models are quantized after training, losing quality. K3 is different: per the card, it applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility.

Practically: the MXFP4 checkpoint is the native format, not a lossy afterthought. If your hardware supports MXFP4/MXFP8, you get the intended model at roughly 4-bit weight footprint — which is the only reason running a 2.8T model is discussable at all.

3. The honest hardware reality

Let’s be blunt: 2.8T parameters is datacenter territory. The repo ships 96 safetensors shards. Even at MXFP4’s ~4-bit-per-weight, the weights alone are hundreds of GB, and MoE cuts compute (only 104B activate per token) — not memory. You still hold the full parameter set.

So:

  • Multi-GPU server (8×H100/H200-class or better)? §4 — vLLM / SGLang / TokenSpeed.
  • A workstation, even a maxed one? Realistically no. Community 1-bit quants exist (§5) but at severe quality cost.
  • Anything smaller? Use the API (§6). That’s not a cop-out — it’s the correct answer for a 2.8T model.

For a sense of scale: Inkling at 975B needs ~290 GB at 1-bit. K3 is nearly 3× the parameters.

4. Method 1 — the officially recommended engines

The model card names exactly three inference engines, each with an official recipe:

EngineOfficial recipe
vLLMrecipes.vllm.ai/moonshotai/Kimi-K3
SGLangdocs.sglang.io cookbook
TokenSpeedlightseek.org/tokenspeed recipes

Follow the per-engine recipe rather than a generic vllm serve line. K3 uses KDA (Kimi Delta Attention) plus MXFP4 weights — both need engine support that landed alongside the release, and the recipes encode the right flags, parallelism, and version floors for your GPU topology. A stock serve command from another model’s guide will not work here.

5. Community GGUF: what’s real and what isn’t

There is no official GGUF from Moonshot. The card lists only vLLM, SGLang, and TokenSpeed.

Community GGUF repos have appeared (including 1-bit IQ1_S builds), and a unsloth/Kimi-K3-GGUF repo exists — but at the time of writing these are early, and none are Moonshot-published. If you go that route:

  • Verify who published it and whether the quant has been validated against the official model’s outputs.
  • Expect real quality loss at 1-bit on a model this sparse — 16-of-896 expert routing is sensitive to weight precision.
  • You’re leaving the native MXFP4 path, which is the format K3 was actually trained for.

This guide won’t print a llama.cpp command line for a build it can’t verify against official docs.

6. Method 2 — the API (the right answer for most)

You can use K3 at platform.kimi.ai by selecting kimi-k3, with OpenAI- and Anthropic-compatible endpoints. For all but a handful of labs, this is the correct way to use a 2.8T model — see our access guide for pricing and setup.

7. The gotcha: preserved thinking history

This is the K3-specific detail that will silently break your integration, self-hosted or API.

K3 always has thinking enabled and returns reasoning_content. Thinking effort is set with the top-level reasoning_effort field: "low", "high", or "max" (default "max").

Critically, K3 was trained in preserved-thinking-history mode. For multi-turn conversations and tool calls, you must pass the complete assistant message back into messages as-is — including reasoning_content and tool_calls, not just content. From the official card:

import openai

def chat_with_preserved_thinking(client: openai.OpenAI, model_name: str):
    messages = [
        {"role": "user", "content": "Tell me three random numbers."},
        {
            "role": "assistant",
            "reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.",
            "content": "473, 921, 235"
        },
        {"role": "user", "content": "What are the other two numbers you have in mind?"}
    ]

    response = client.chat.completions.create(
        model=model_name,
        messages=messages,
        stream=False,
        max_tokens=4096,
        reasoning_effort="max",
    )
    return response.choices[0].message.content

In that example the assistant should recall 215 and 222 — numbers that exist only in the prior reasoning_content. Strip the reasoning and the model loses that state.

Why this bites: most OpenAI-compatible client code stores only content when appending assistant turns. Against K3 that quietly degrades multi-turn coherence and tool-call chains — with no error. Audit your message-append logic before blaming the model.

8. Should you run it locally?

  • Self-host (§4) if you have a multi-GPU server and need data control or volume — use the official per-engine recipe and the native MXFP4 weights.
  • API (§6) for everyone else. A 2.8T model is not a workstation project.
  • Either way, handle reasoning_content (§7) and read the custom license (§1) before shipping.

The takeaway

Kimi K3’s weights are genuinely out — 2.8T total / 104B active, 896 experts (16 + 2 shared), 93 layers of KDA + Gated MLA, 1M context, multimodal via MoonViT-V2, and natively MXFP4 (QAT from SFT onward), in 96 shards under a custom Kimi K3 license. Run it through the officially recommended vLLM / SGLang / TokenSpeed recipes — not a generic serve command — and treat community GGUFs as unvalidated. For almost everyone the kimi-k3 API is the right call. And whichever path you take, pass reasoning_content back on every turn, or multi-turn and tool calls will decay silently.

For the launch-day access guide see How to Use Kimi K3 Today; for comparable giant open models, Inkling locally and Kimi K2.7 Code locally.

Sources

  • moonshotai/Kimi-K3 — Hugging Face model card — full spec table (2.8T total / 104B activated, 896 experts / 16 selected / 2 shared, 93 layers with 69 KDA + 24 Gated MLA, 96 attention heads, hidden dim 7168, SiTU-GLU, 1,048,576 context, 160K vocab, MoonViT-V2 vision encoder 401M, text+image); native MXFP4 quantization (QAT from SFT stage, MXFP4 weights + MXFP8 activations); deployment engines (vLLM, SGLang, TokenSpeed) with official recipe links; platform.kimi.ai API with kimi-k3 and OpenAI/Anthropic-compatible endpoints; preserved-thinking-history requirement and the reasoning_effort field (low/high/max, default max) with the official Python example; custom kimi-k3 license
  • Kimi K3 LICENSE — custom license terms (not MIT/Apache)
  • Kimi K3 Quickstart and Thinking Effort — official guides for vision input, structured output, tool choice, context caching
  • Hugging Face API (July 29, 2026): 96 .safetensors shards, 118 files total. Community GGUF repos are third-party and unvalidated — no official GGUF exists. Verified July 29, 2026.