Research · · 2 min read

GLM-5.2 edged out Claude on Semgrep's IDOR benchmark — but the harness, not the model, won

On June 22, 2026, Semgrep published an IDOR-detection benchmark where GLM-5.2 with a bare prompt scored 39% F1, beating Claude Code on Opus 4.8/4.7 at 28% — at roughly one-sixth the cost. But Semgrep's own scaffolded harness hit 61%. The real builder lesson is about scaffolding, not picking a model.

On June 22, 2026, security vendor Semgrep published a benchmark in which an open-weight model running a bare prompt outscored a frontier coding agent on a reasoning-heavy security task. (Source: Semgrep blog, 2026-06-22) The headline says “GLM-5.2 beats Claude.” The numbers underneath tell a more useful story for builders.

Key facts:

  • The task was IDOR detection (Insecure Direct Object Reference) on real open-source applications, scored by F1.
  • GLM-5.2, prompt-only via Pydantic AI, scored 39% F1.
  • Claude Code on Opus 4.8/4.7 (Claude Code SDK, IDOR prompt only) scored 28%; Opus 4.6 scored 37%.
  • GLM-5.2 cost roughly $0.17 per vulnerability found — about one-sixth the cost of frontier models.
  • But Semgrep’s own scaffolded “Multimodal” harness scored far higher: 61% with GPT-5.5 and 53% with Opus 4.8.
  • Semgrep held the dataset, evaluation, and system prompt constant, and varied only the model and harness.
Semgrep benchmark chart comparing GLM-5.2, Claude Code, and Semgrep's multimodal harness on IDOR F1 scores
Semgrep's IDOR benchmark summary: a bare-prompt open-weight model (GLM-5.2, 39%) edged out Claude Code on Opus 4.8/4.7 (28%) — but a purpose-built harness more than doubled both. (Source: Semgrep)

What the gap actually measures

It’s tempting to read “GLM-5.2 beats Claude” as a model ranking. Semgrep is careful that it isn’t. The Claude Code runs went through the Claude Code SDK with only the IDOR prompt — no endpoint-discovery scaffolding. The Multimodal pipeline that hit 61% had that scaffolding. So the 28%-vs-61% spread is a harness difference, not a model difference. (Source: Semgrep blog, 2026-06-22)

That also explains the odd-looking result where Opus 4.6 (37%) edged its newer siblings (28%): on a thin prompt with no task structure, small prompt-format and run-to-run variance swamps raw model capability. One task, one vendor’s benchmark — treat the absolute ranks with caution.

What it changes if you’re building security tooling

  • Scaffolding beats model choice on structured tasks. The single biggest lever here was endpoint-discovery scaffolding, which more than doubled F1. Before you swap models, invest in the harness: give the agent the discovery, retrieval, and verification steps the task actually needs.
  • Open-weight is now cost-competitive for scanning loops. At ~$0.17 per finding, GLM-5.2 is cheap enough to run as a first-pass filter, with a frontier model or Opus 4.8 as the expensive second opinion. Put the model ID in a config flag so the cheap/expensive split is one line.
  • Don’t read this as a Claude weakness. Claude Code is built to drive a full agent loop, not to answer a single bare security prompt; Opus 4.8 in Semgrep’s own harness still scored 53%. The benchmark is a lesson about evaluation conditions, not a verdict on the model.
  • Reproduce before you rely. This is a single vendor’s benchmark on one task. If IDOR (or any vuln class) matters to your pipeline, rebuild the eval on your own codebases before changing tools.

For builders wiring agents into a real workflow, the durable takeaway is the same one that keeps showing up: the agent harness and the scaffolding around the model decide outcomes at least as much as which model you pick.

Sources

Source: Semgrep