Ornith-1.5 Writes Its Own Training Curriculum — and Its Numbers Are Still Unverified

Ornith-1.5 reports 86.0 on SWE-bench Verified and writes its own training curriculum. MIT-licensed, vendor-reported, not yet independently replicated.

Ornith-1.5 reports 86.0 on SWE-bench Verified and 86.1 on Terminal-Bench 2.1, and both numbers come from DeepReinforce itself — as of October 26, 2026, no independent party has published a run of the 397B that reproduces either number.

What changed on August 19 is the shape of the training loop. Ornith-1.0 wrote the harness around tasks that humans supplied; Ornith-1.5 writes the tasks, writes the harness, and grades its own output with reinforcement learning. The curriculum is authored by the model being trained.

We covered the 1.0 release on June 29, which was about a model authoring its own agent scaffolding. This is a different machine, and one word needs care: here “scaffolding” means code the model writes for its own training run, not the AGENTS.md context files that some agent setups bolt onto a repository.

Three things sit under those numbers, and you should hold them while you read the rest. Every Ornith score is vendor-reported. The DeepSWE board run that was requested back in June never happened. And BenchLM — the leaderboard the 86.0 comes from — now excludes SWE-bench Verified from its weighted ranking because it says the tests are contaminated and saturated.

What Ornith-1.5 Actually Changed: The Model Now Writes the Curriculum

Ornith-1.5 ships in three sizes, all MIT-licensed, with weights on Hugging Face: a 397B mixture-of-experts flagship with a 262k context window, a 35B-A3B MoE with roughly 3B active parameters per token, and a 9B dense model.

The loop is the real product. In the 1.0 design, at each reinforcement-learning step the model proposed a per-task scaffold — the orchestration plan for calling tools, retrying, and assembling a trajectory — while the tasks themselves were fixed and human-authored. The 1.5 design extends that: the model generates the tasks, builds the scaffolds to attempt them, and produces the grading signal that reinforces its own attempts.

Task supply is usually the binding constraint on RL: scaling code-agent training means writing more human-authored tasks with checkable rewards, and that pool is finite. If the model can author its own curriculum, the wall moves — which is why this idea deserves more attention than the leaderboard position does.

So does the risk. Writing your own harness is one reward-hacking surface. Choosing your own tasks and writing your own grader is a larger one — the model now decides what to be tested on as well as how to pass. The 1.0 release shipped with three defenses DeepReinforce described against exactly that — an immutable trust boundary, a deterministic monitor, and a frozen judge veto, covered in our June post. A model that also picks its curriculum and grades its own results is a harder case for all three, and nobody outside the company has published an audit of the 1.5 loop.

The Scorecard, With the Provenance Attached

Every figure in the left column comes from DeepReinforce’s launch page, and BenchLM reproduces them from that same page. The 1.0 column is here because it is the only comparison point that shares a lab, a license, and a release lineage.

Benchmark Ornith-1.5-397B Ornith-1.0-397B Change vs 1.0
SWE-bench Verified 86.0 82.4 +3.6
Terminal-Bench 2.1 86.1 77.5 +8.6
SWE-bench Pro 65.1 62.2 +2.9
DeepSWE 56.0 — —
NL2Repo 59.5 — —
GPQA 92.8 — —
HLE (with tools) 56.1 — —
Claw-Eval 81.4 — —
MCP Atlas 80.0 — —
WideResearch 80.8 — —
BrowseComp 86.6 — —
frontierBench (TB3) 13.5 — —

Terminal-Bench 2.1 carries the larger move of the two, and it is the score where the missing conditions cost you the most.

DeepSWE deserves the most suspicion of anything in that table. It is the one entry on a leaderboard run by somebody other than the lab, and BenchLM lists it for Ornith-1.5 while sourcing it back to Ornith’s own launch page. No independent party has run the 397B on DeepSWE, and the board’s row for Ornith-1.5 is the vendor’s own number wearing somebody else’s logo. The row looks independent and is not.

Ornith-1.5 Tops the Open-Weight Board and Sits Below Anthropic’s Best

BenchLM’s SWE-bench Verified board as of 2026-10-25:

Model SWE-bench Verified
Claude Opus 5 96.0
Claude Mythos 5 95.5
Claude Fable 5 95.0
Ember-1 (Fireworks) 92.2
Claude Opus 4.8 88.6
Claude Opus 4.7 (Adaptive) 87.6
Ornith-1.5-397B 86.0
Claude Sonnet 5 85.2
GPT-5.3 Codex 85.0
Ornith-1.0-397B 82.4
Beam 1.0 501B-A23B (Reflection AI) 81.6
Claude Opus 4.5 80.9
Claude Opus 4.6 80.8
DeepSeek V4 Pro 0813 80.6
MiniMax M3 80.5
Qwen3.7 Max 80.4

We covered this same leaderboard in April, when a closed preview model led it and the best open-weight model on it was still roughly 35 percentage points behind.

Two readings of that board are true at once, and a post that gives you only one of them is selling something.

The first: as of 2026-10-25, the 397B Ornith-1.5 is the highest-scoring open-weight model on that board, ahead of its own 1.0, Beam 1.0, DeepSeek V4 Pro, MiniMax M3 and Qwen3.7 Max. The only non-Anthropic model above it is Ember-1 at 92.2, and the sources behind this ranking do not classify Ember-1 as open-weight or closed — if Fireworks ships open weights, the crown moves up a row.

The second: it trails Anthropic’s five models that score higher on the same board — Opus 5, Mythos 5, Fable 5, Opus 4.8 and Opus 4.7 — though it clears Sonnet 5, Opus 4.5 and Opus 4.6. And the board’s operator no longer counts this benchmark in its own ranking at all. The 86.0 is a display number on a leaderboard whose weighted ranking excludes the benchmark entirely.

Why None of These Numbers Has Been Independently Checked

1. The lab says so itself. Ornith’s benchmark policy page states: “All Ornith 1.0 benchmark scores are self-reported by DeepReinforce. Independent verification of Ornith 1.0 results is pending as of June 2026.” The page carries no date of its own beyond that June 2026 stamp, so the wording is about 1.0 and predates the 1.5 launch. The company has not published a verification for 1.5 either. The same notice also carries a line about community members on Reddit and NVIDIA forums reporting positive 1.0 experiences consistent with the published numbers — anecdote about the model, not a benchmark run, and it does not move the scorecard.

2. The independent run was requested, and then the request was closed as moot. On 2026-06-27, an issue on the DataCurve DeepSWE leaderboard repository asked for Ornith to be run on an independent board. On 2026-10-23 that issue was closed as not planned — not because the run happened, but because DeepSWE stopped publishing new leaderboard results on September 3 while its team rebuilds the benchmark. The last entry on its changelog is still September 3. So there is no published run of the 397B, and right now there is no board to publish one to. Every number in the scorecard still traces back to one vendor announcement, and the four outlets covering the launch are covering that same announcement.

3. One outside measurement exists, and it is not a reproduction. In the launch’s Hacker News thread, one commenter ran Ornith-1.5-35B against Qwen3.8-27B and posted the numbers. The 35B landed at 67.8 on Terminal-Bench 2.1 — within a point of the 68.5 the launch page claims for that model — and came out behind Qwen3.8-27B on Terminal-Bench 2.1, SWE-bench Pro and DeepSWE while beating it on NL2Repo. It is one commenter’s run, on a different benchmark mix, with none of the harness conditions the launch page specifies, and it never touches the 397B’s 86.0. It is not a reproduction. It is also the only outside measurement of any Ornith-1.5 number that has been published, and it belongs in the same category as everything else in this section: a data point, not a measurement.

4. A score here is not a property of the model. SWE-bench Verified results are agent-and-scaffold dependent. BenchLM’s own guidance for anyone quoting the number: keep the benchmark version, scaffold, tool budget, and retry policy with the score. A number that arrives without those conditions is not a measurement you can act on. We covered that in August: one model, ten harnesses, and a 29-point swing on SWE-bench Pro with nothing changed but the wrapper (TopClanker, August 10).

BenchLM Excludes SWE-bench Verified From Its Own Ranking

BenchLM’s SWE-bench Verified page, updated 2026-10-25, says the benchmark “is excluded from the weighted scoring formula… because its tests are contaminated and saturated,” and records that it was moved to display-only status in BenchAlign v5.6.

Contamination means some of the benchmark’s tests leaked into training corpora; saturation means the remaining headroom is too small to separate scores cleanly. OpenAI’s audit of the hardest problems found fundamentally flawed test cases, and the lab stopped reporting the benchmark altogether (TopClanker, June 8). Either way, the leaderboard Ornith-1.5 tops is a display surface, and the weighted ranking that would tell you which model is genuinely strongest does not count SWE-bench Verified at all.

Both the leaderboard and that exclusion come from the same page on the same day. This is not a footnote buried three clicks away — it is attached to the board. So when you read the 86.0, and you will, because it is the number in every headline about this release, read it as a number on a display board rather than a ranking input. The gap between those two readings is where the marketing lives.

Verdict: What We Believe, in Order of Confidence

Three things, in order of confidence:

  1. The self-authored curriculum is the real story, and it does not depend on the benchmarks. A model that generates its own tasks, writes its own scaffolding, and produces its own grading signal is running a structurally different training loop from the rest of the field. That is a claim about published architecture, and architecture claims are easier to check than leaderboard claims.

  2. Treat Ornith-1.5 as the open-weight leader in agentic coding, with the asterisk attached. The reported numbers are the strongest open-weight results we could find as of 2026-10-25. They also arrive with no independent run behind them, on a benchmark the leaderboard’s operator has excluded from its own scoring formula, and without the scaffold and retry conditions that would let you reproduce them.

  3. The failure mode nobody has checked is curriculum collapse. If a model picks its own tasks, the training set can drift toward what the model is already good at, or toward tasks whose rewards are easiest to satisfy. No outside lab has audited whether the 1.5 loop avoids that. For anyone planning to fine-tune or serve this family, that is the question to keep open.

MIT licensing and public weights make this unusually cheap to settle. One independent reproduction of the SWE-bench Verified number on a stated harness — or a published DeepSWE run once that benchmark starts publishing again — turns this from a vendor claim into a measurement. Until one of those lands, the open-weight field has a credible new leader and an unverified scorecard, in that order.

Sources