Models · · 2 min read

Granite Switch 4.1: serve chat, decisions, and speech from one vLLM endpoint — and pin the right vLLM

IBM published Granite Switch 4.1 on October 14–15, 2026: 8B and 30B checkpoints that pack chat, a Decision aLoRA, Guardian Core, and Granite Speech 5.0 TurboCTC into one Apache-2.0 endpoint. Here is how to start the server, flip adapters with chat_template_kwargs, and which vLLM range the docs actually allow.

IBM’s Granite Switch 4.1 blog (October 15, 2026) and the 8B / 30B model cards (release date October 14) answer a concrete serving question: can one OpenAI-compatible endpoint do chat, typed decisions, safety checks, and speech-to-text without spinning up three servers?

Short version: yes, with the precomposed Switch checkpoints. Start one vLLM process, then pick the adapter per request with chat_template_kwargs.adapter_name. Pin vLLM carefully — the model cards and the granite-switch README disagree on the upper bound.

What ships in the 8B and 30B checkpoints

From the 8B model card:

PieceDetail
Baseibm-granite/granite-4.1-8b (8B params, 128K context)
Adaptersdecision (general Decisions aLoRA) and guardian-core (from granitelib-guardian-r1.0)
AudioEmbedded ibm-granite/granite-speech-5.0-470m-turboctc (470M, English ASR)
LicenseApache 2.0
Backends listedHugging Face Transformers, vLLM

The 30B card uses the same adapter + speech layout on ibm-granite/granite-4.1-30b. The blog says adapters plus speech are packed so VRAM stays roughly at base-model size, and aLoRA reuses the base KV cache across switches.

On Decision Index v0.3 (fast subset, 1,068 rows), the 8B card reports hosted-vLLM scores for the decision adapter: overall 42.9, with Tools 59.8 and Arts 18.0. Treat those as the card’s published numbers, not a claim that Switch replaces every hosted decision API.

Start the server

The cards’ recommended path (after pip install -e ".[vllm]" from generative-computing/granite-switch, or pip install "granite-switch[vllm]" per the README):

python -m vllm.entrypoints.openai.api_server \
  --model ibm-granite/granite-switch-4.1-8b \
  --port 8000

The README also documents preview IDs such as ibm-granite/granite-switch-4.1-3b-preview if you want a smaller compose. Requires Python 3.11+ and PyTorch 2.11+ (README); the 8B card lists Transformers >= 5.16, <= 5.17.0.

Flip adapters without a second deployment

Both adapters live in one checkpoint. The card activates either with:

extra_body={"chat_template_kwargs": {"adapter_name": "decision"}}
# or
extra_body={"chat_template_kwargs": {"adapter_name": "guardian-core"}}

For decision, the documented recipe is two user turns (a State: turn, then a ### DECISION turn with single-token labels like A/B/C), max_completion_tokens=1, and vLLM’s logprob_token_ids so you softmax over the option labels — not “top-K by probability,” which can miss a low-probability option. vLLM caps logprob_token_ids at 128 ids; the card says to chunk larger label sets and softmax once over the combined logprobs.

For guardian-core, decode normally and parse {"score": "yes"} / {"score": "no"}. The card warns this checkpoint does not include Guardian’s factuality or policy-guardrails adapters.

Speech uses the same /v1/chat/completions path with an input_audio content part. Audio limits from the card: English only, lowercase/unpunctuated transcripts, and no joint ASR–LLM training yet.

The vLLM pin you can trip over

  • Model cards (8B and 30B dependency tables): vLLM >= 0.26.0, < 0.28.0.
  • Repo README (retrieved 2026-10-18): [vllm] accepts 0.26.x through 0.30.x, with CI groups vllm26 / vllm30.

If you are already on vLLM 0.32.0 for other models (for example Nemotron Super VL’s recommended image), do not assume the same container serves Switch — follow the narrower of the two docs until IBM aligns them. Prefer the version your compose extras actually install.

Source: Hugging Face Blog — Granite Switch 4.1