Apple's RSI-Inference Bench: the best coding agent passes 26.7% of day-0 vLLM and SGLang model-support tasks

Apple ML Research has published RSI-Inference Bench, “a long-horizon benchmark for coding agents that improve the inference stack of large language models.” The repository went public on October 16, 2026, and the announcement post says the team spent “$100K API tokens, 50K GPU hours” building it. If you are deciding whether to let a coding agent touch your vLLM or SGLang deployment, this is the first public scoreboard built on that exact job. (Source: RSI-Inference Bench README)

Key facts:

• 37 tasks in three families: 10 Day-0 Model Support (make a new model serve), 20 Inference Optimization (make it serve faster), and 7 Bug Fix tasks (find and repair a performance regression).
• Real engines, real GPUs: 20 tasks run on vLLM and 17 on SGLang; 17 use H100s and 20 use B200s, with 1 to 8 GPUs per task.
• Offline, long-horizon: the agent gets the engine at a pinned commit and works without internet for up to 8 hours. A separate verifier builds the edited engine, serves the model, and measures it.
• Unsaturated: the top agent scores 26.7% pass@1 on model support.

What an agent actually has to do

Each task is a sealed engine snapshot. The FP8 decode regression task, for example, hands the agent a vLLM tree serving nvidia/DeepSeek-R1-0528-NVFP4 on 8×B200 with data and expert parallelism. Output is correct; only mean time per output token got slower. The instructions forbid consulting upstream history, pull requests, issues or release notes, and a tree that changes output, or reads slower on an unscored guard workload, scores zero. Model-support tasks cover recent releases such as Inkling-Small (from the Inkling family), DiffusionGemma, DeepSeek-V4.1-Flash, MiniMax-M3 and Kimi-Linear. (Source: regression_fp8_decode_throughput task)

The leaderboard

The leaderboard snapshot dated October 12, 2026 covers 12 agents and 444 scored results. Agents are ranked by a weighted Bradley-Terry rating on an Elo scale.

Rank	Agent	Day-0 pass@1	Optimization speed-up
1	Sonnet-5.5	26.7%	1.68x
2	Opus-5.5	13.3%	1.48x
3	Fable-5.1	10.0%	1.41x
4	GPT-6.1-Sol	10.0%	1.37x
5	GPT-6-Astra	10.0%	1.35x
9	DeepSeek-V4.1-Flash	0%	1.24x
12	Gemini-3.7-Flash	0%	1.04x

Sonnet-5.5 fully passed at least once on 5 of the 10 model-support tasks. Nobody passed DiffusionGemma, gpt-oss-120b on SGLang, Qwen3.8-Flash-Next, MiniMax-M3 or Stable Diffusion 3. On regressions alone Sonnet-5.5 averaged 2.19x. The GPT-6.1 Sol versus Astra cost trade-off we covered in OpenAI’s GPT-6 maintenance playbook shows up here too: Sol edges Astra on optimization, and they tie at 10% on model support.

What it means for you

Two practical readings. First, a 26.7% ceiling means day-0 support for a new architecture is still a job to review line by line, not to hand off. If you are standing up a fresh open model like GLM-5.2 on vLLM, expect to finish the port yourself. Second, speed-ups above 1x are common across the top ten, so an agent hunting performance regressions against a benchmark you trust is a reasonable bet.

Run it on your own agent

The harness is Pier, a Harbor-compatible framework vendored in third_party/pier, with adapters for Claude Code, Codex, Cursor CLI, Gemini CLI, mini-SWE-agent and OpenCode. The README’s commands are:

uv sync
uv run pier run -p tasks/<task> --env docker --agent <agent> --model <model>

You need Python 3.12+ and the GPUs each task’s task.toml declares. Report pass@1 for Day-0 tasks and a geometric-mean speed-up for the other 27, where a failed correctness check counts as 1.00x. The paper is listed as “coming soon,” and the code ships under the Apple Sample Code License. (Source: announcement post on X)