The Harness Matters More Than the Model — 29pp Pass@1 Swings on SWE-bench Pro From Code-Agent Wrappers Alone

Same model, same benchmark, different scaffolding: 23% to 52% pass@1 on GLM-5.2 SWE-bench Pro depending on which harness wraps it. A 29-point spread driven entirely by the agent framework, not the model. Every coding-agent leaderboard number published in 2026 is partly a harness number, and most of them don't surface that.

The Harness Matters More Than the Model — 29pp Pass@1 Swings on SWE-bench Pro From Code-Agent Wrappers Alone

August 10, 2026 — Take one model. Take one benchmark. Wrap the model in ten different coding-agent harnesses. Watch pass@1 swing by 29 percentage points.

Joel Niklaus ran this experiment across 10 harnesses and the results, published last week, are the most direct evidence yet that the coding-agent leaderboards the industry has been publishing in 2026 are not measuring what they claim to measure. On SWE-bench Pro with GLM-5.2, pass@1 ranged from 23% to 52% depending on which harness wrapped the model. On Gemma 4 26B the same experiment produced a 15% to 36% spread — a 21-point swing driven entirely by scaffolding. (Joel Niklaus on X, 2026-08-07, AINews/Latent Space, 2026-08-08)

The model is the same in every run. The benchmark is the same. The only thing that changes is the agent wrapper — the orchestration code that decides how the model sees the codebase, what tools it gets, how it iterates, and how it parses its own output. That wrapper is producing swings larger than the entire gap between the best open-weight model and the best closed model on the same benchmark.

What the experiment measured

SWE-bench Pro is the harder, more recent benchmark that OpenAI and others have moved to after SWE-bench Verified was effectively retired for credibility reasons (TopClanker, 2026-06-08). It contains 1,865 tasks across public, held-out, and commercial splits, and published scores vary by evaluation scaffold — which is exactly the property the Niklaus experiment isolates.

The setup: pick a single model, run it against SWE-bench Pro under 10 different agent harnesses, measure pass@1 for each. Repeat for a second model. Compare the spread. Two models were tested — GLM-5.2 (the open-weight coding model Z.ai released earlier this year, TopClanker, 2026-04-09) and Gemma 4 26B (Google’s smaller open-weight model, well within reach of local inference on a single high-end consumer GPU). Same benchmark. Same prompts where possible. Different wrapper. (Joel Niklaus on X, 2026-08-07)

The numbers

GLM-5.2 on SWE-bench Pro:

  • Best harness: 52% pass@1
  • Worst harness: 23% pass@1
  • Spread: 29 percentage points
  • Ratio: top harness solves more than 2.2x the problems the bottom harness solves

Gemma 4 26B on SWE-bench Pro:

  • Best harness: 36% pass@1
  • Worst harness: 15% pass@1
  • Spread: 21 percentage points
  • Ratio: top harness solves 2.4x the problems the bottom harness solves

For context: the gap between the leading closed model on SWE-bench Pro (Claude Opus 4.7 at 64.3%) and the leading open-weight model (GLM-5.1 at 58.4%) is roughly 6 percentage points (TopClanker, 2026-04-10, Scale SWE-bench Pro Leaderboard). The harness effect on a single model is 5x larger than the model-tier gap the leaderboard headlines.

If you read a SWE-bench Pro score in a vendor blog post today, you are reading a number that could shift by 29 points in either direction depending on which wrapper the vendor picked. Most vendor posts do not name the wrapper. (Joel Niklaus on X, 2026-08-07, AINews/Latent Space, 2026-08-08)

Harness ranking does not transfer across models

The 29-point GLM-5.2 spread and the 21-point Gemma 4 26B spread are independently large. The finding that should change how anyone reads a leaderboard is what happens when you compare them.

The rank correlation between the two model-specific harness rankings is -0.05 — essentially zero, and slightly negative. Picking the “best” harness for GLM gives you a roughly random harness for Gemma. The harness that puts GLM-5.2 at 52% pass@1 is not the harness that puts Gemma 4 26B at 36% pass@1. (Joel Niklaus on X, 2026-08-07, smol.ai, 2026-08-07)

This kills the standard procurement heuristic. The old heuristic: “find the harness that benchmarks best on a model like ours, deploy that.” The new reality: the best harness is model-specific, the ranking does not transfer, and the harness that wins on the benchmark may be the wrong harness for the model you actually want to deploy. There is no portable winner.

Cost varied 8x for the same model

The same model run under different harnesses produced cost-per-solved-task ranging from $0.84 to $7.05 — an 8.4x spread for identical model weights and identical benchmark tasks. (Joel Niklaus on X, 2026-08-07, AINews/Latent Space, 2026-08-08)

The cost driver is not the model. The cost driver is the harness’s token economy. Niklaus measured that 97% of input tokens in a typical coding-agent run are re-sent conversation prefix — the system prompt, the prior turns, the file contents the agent has already seen. Prompt caching is the dominant cost lever. Harnesses that cache well are dramatically cheaper. Harnesses that don’t, aren’t.

A 6x price gap between two harnesses running the same model is not a model problem. It is a caching, tool-call structure, and prompt-construction problem inside the wrapper. If you’re paying for a hosted coding agent today, the wrapper choice matters more to your bill than the model choice.

Model-agnostic harnesses beat vendor harnesses on the smaller model

The last finding is the one that should change the default for anyone deploying a smaller open-weight model: model-agnostic harnesses outperformed vendor-provided harnesses on Gemma 4 26B. (Joel Niklaus on X, 2026-08-07, AINews/Latent Space, 2026-08-08)

Vendor harnesses — the agent scaffolds a model vendor ships with or recommends — are tuned for that vendor’s flagship, not for smaller models in the same family. When the harness is asked to wrap a smaller open-weight model with different behavioral characteristics, the assumptions baked into the wrapper degrade performance. A model-agnostic harness designed to probe and adapt outperforms a vendor harness that assumes specific tool-call patterns, context lengths, or instruction-following behavior the smaller model does not have.

The “use the vendor harness” default is no longer safe advice for smaller models. For local-LLM deployments running 7B-26B models — exactly the range this audience cares about — the harness choice is the lever, not the model swap.

What this means for SWE-bench Pro leaderboards

The Scale SWE-bench Pro public leaderboard (labs.scale.com/leaderboard/swe_bench_pro_public) publishes model scores. Until now, the implicit assumption was that “model” was the dominant variable and the harness was a fixed evaluation scaffold that anyone could reproduce.

The Niklaus data says the scaffold is at least as variable as the model. A published SWE-bench Pro score is the product of three things: the model, the harness, and the prompt. The harness and prompt together can move the score by more than 20 percentage points on the same model. The leaderboard currently reports one number per model and treats the rest as fixed.

For anyone reading a leaderboard: treat any single SWE-bench Pro number as a lower bound on model capability and an upper bound on what that specific harness can do with that model. The model could be better; the harness could be better. You cannot tell which from the published number.

Practical implications for local-first code agents

For TopClanker’s local-LLM audience running LM Studio, Ollama, llama.cpp, or vLLM with a coding-agent harness on top of an open-weight model, the experiment translates into four concrete points.

1. The harness is your biggest lever, not the next model upgrade. If you are running GLM-5.1 or Gemma 4 locally, swapping the harness — not the model — is likely to produce larger score and cost improvements than swapping to a newer weights release. The community-maintained rankings at RyanAlberts/best-of-Agent-Harnesses are now the most relevant resource for local harness selection, more relevant than any new model release in the same weight class.

2. Cost is a harness property, not a model property. 97% of input tokens are re-sent prefix. If your local stack has a prompt cache layer (llama.cpp’s prompt cache, LM Studio’s session cache, vLLM’s prefix caching), the harness that uses it will be dramatically cheaper than the harness that does not. If you are paying for hosted inference for a code agent, the wrapper choice is the dominant cost driver — bigger than the model.

3. Smaller models need different harnesses. The vendor harness is tuned for the vendor’s flagship. For Gemma 4 26B and similar mid-size open-weight models on a single consumer GPU, the model-agnostic harness is more likely to outperform the vendor-recommended wrapper. Test the harness, not just the model.

4. Benchmark numbers in vendor posts are not portable. A 64% SWE-bench Pro score for a vendor’s flagship is not a number you can reproduce with a different harness on the same model. The 29-point spread is bigger than the entire gap between top-tier and mid-tier models. Ask vendors for the harness name and the prompt template. If they refuse, treat the number as marketing.

The Take

The Niklaus experiment is the most rigorous evidence yet that coding-agent benchmarks in 2026 are measuring the wrapper as much as the model. A 29-percentage-point pass@1 spread on the same model, an 8x cost spread, and a rank correlation of -0.05 across models are not measurement noise. They are the dominant signal.

For the open-weight, local-LLM community this is good news. The biggest performance and cost lever in any coding-agent stack is not the next model release — it is the harness choice and the prompt-cache strategy. Both are local-first, both are reproducible, and both are now the most important things to benchmark.

For the leaderboard-watching community this is a credibility reckoning. Every coding-agent benchmark number published this year is partly a harness number, and most leaderboards do not surface that. The vendors who publish the harness name and the prompt template alongside the score are the only ones whose numbers can be reproduced. The rest are noise.

The harness matters more than the model. That is the new rule.

Sources