🔍 Read the full analysis: FrontierCode: Is AI’s Pull Request Ready To Join The Project? on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added the repository-focused FrontierCode benchmark on October 12, 2026, and tested 23 AI models. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts, while the benchmark’s results show that even the top model still needs human review. The source reports substantial cost differences and says some major models had not yet been evaluated.
Artificial Analysis tested 23 AI models on FrontierCode, a benchmark that evaluates whether code changes would meet repository maintainers’ acceptance standards, and Claude Opus 5.5 led with a 52.2% score. The leading model passed every blocking check in 57.7% of attempts, according to the results published October 12, a finding that points to continuing limits on handing AI-generated pull requests to a project without human review.
The leaderboard’s second-place model was GPT-6.1 Sol, with a 49.1% score and a reported cost of $0.81 per task. Claude Opus 5.5, run in its max configuration with fallback, cost $10.15 per task. Claude Fable 5.1 placed third at 49.0%. Artificial Analysis ran all models using the same open-source agent harness, Stirrup, with a limit of 250 turns per attempt and up to three attempts per task.
FrontierCode’s score is not the same as its pass rate. The score incorporates quality checks after a model clears all blocking checks; an attempt that fails any blocking check receives zero. Opus’s 57.7% blocking-check pass rate means it did not clear every required check in the other reported attempts. The source also says nine of the 100 tasks in Artificial Analysis’s Main set were not solved by any model on any attempt.
The benchmark compares patches across six areas: hidden tests, command checks for regressions and cleanliness, tests that must fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM review against maintainer criteria. The review judge was GPT-6 Luna (medium). According to the source, an attempt flagged by Cognition’s trajectory scanner for looking up a solution receives zero; oversized diffs, changes to unrelated files and unhelpful tests can also reduce results.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing CI Is Not Enough
FrontierCode targets a practical question for software teams: would maintainers accept the patch? Passing tests alone may not answer that. A change can introduce regressions, touch too much of a repository or include tests that do not demonstrate the behavior they claim to check. By making those concerns part of the scoring, the benchmark measures qualities that affect whether an AI-generated pull request is useful to reviewers.
The results do not establish how models perform across every repository or in day-to-day development. They do show that, under this benchmark’s rules, the leading model cleared all blocking checks in fewer than six in ten attempts. That makes human review relevant even when an AI coding agent completes a task and produces a patch.
The price comparisons add a deployment question. Artificial Analysis lists GPT-6 Luna at $0.09 per task with a 41.9% score, MiMo-V2.6-Pro at $0.18 with 45.4%, GPT-6.1 Sol at $0.81 with 49.1%, and Opus at $10.15 with 52.2%. These are benchmark cost figures, not a guarantee of what teams will pay in other settings. The reported gap between Sol and Opus is 3.1 score points at about 12.5 times the per-task cost.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Patches
FrontierCode was developed by Cognition and open-source maintainers. It was announced in June 2026, followed by version 1.1 in July, which introduced rules against looking up solutions online. More than 20 maintainers created tasks drawn from 36 established repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task.
The full benchmark contains 150 private tasks, while Artificial Analysis ran its 100-task Main set. Keeping tasks private is intended to reduce the chance that models encountered their solutions during training. The benchmark’s six checks combine automated testing and scope limits with a review against criteria written by maintainers. Tomer Nosrati, a Celery maintainer, described the distinction in Cognition’s announcement:
“Where others grade like a CI, FrontierCode grades like a tech lead.”
Results also differ from general model rankings. In the comparison included by the source, Claude Sonnet 5.5 ranked second among the listed models on the Intelligence Index but fifth on FrontierCode. Grok 4.7 ranked 18th on FrontierCode, while DeepSeek V4.1 Flash placed seventh despite a lower Intelligence Index score. General capability rankings and repository-patch results are not interchangeable, though this comparison covers only the models and measures reported.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
Models and Real-World Performance Gaps
The results do not establish how well the tested models would perform on every live pull request. They cover a specific benchmark, task set, harness, turn limit and grading process; performance may differ under other repositories and workflows. The source also does not provide enough detail here to assess how closely benchmark costs match the costs of sustained production use.
Several model results were missing as of October 13, according to the source: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode scores. It is not clear from the reported material when those evaluations will be available. Their absence means the published ranking is not a comparison of every model the source considers relevant.
The source reports that nine tasks were unsolved by every model across all attempts, but does not identify those tasks or explain whether they share particular types of difficulty. It also does not provide task-level results in the material supplied. The headline score should therefore be read as an aggregate benchmark result, not a prediction that a given AI-generated patch will be accepted or rejected.
Awaiting More Model Results
The next development to watch is whether Artificial Analysis adds results for the models missing from its October 13 comparison, particularly the named Gemini, Muse and Mistral systems. The source does not give a publication date for those evaluations, so their timing remains unconfirmed.
Further results may change the leaderboard and make the cost and performance comparisons more complete. For now, the published figures support a narrower conclusion: the leading model did not pass every blocking check on every attempt, and FrontierCode’s scoring treats review quality and patch scope as part of the task. Teams considering these tools still need to assess changes in their own repositories and retain review processes suited to their projects.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition and open-source maintainers. It evaluates repository changes using tests, checks for regressions and scope, and an LLM review against maintainer criteria.
Which model led the published results?
Claude Opus 5.5, in the max configuration with fallback, ranked first with a 52.2% FrontierCode score. It passed all blocking checks in 57.7% of attempts, according to Artificial Analysis.
Does a FrontierCode score equal the pass rate?
No. The score incorporates quality checks for attempts that clear every blocking check. An attempt that fails a blocking check receives zero, while the separately reported pass rate shows how often all blocking checks were cleared.
How much did the leading models cost per task?
Artificial Analysis reported $10.15 per task for Claude Opus 5.5 and $0.81 for GPT-6.1 Sol. These are the source’s benchmark cost figures and may not match costs in other workloads.
Which models did not yet have results?
As of October 13, 2026, the source said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result. It did not state when those results would be published.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
