🔍 Read the full analysis: FrontierCode: How Maintainers Respond To AI-Generated Pull Requests on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis evaluated 23 AI models on FrontierCode, a benchmark that scores whether maintainers would accept a code change, not just whether tests pass. Claude Opus 5.5 ranked first, but passed every blocking check in 57.7% of attempts; the results indicate that human review remains necessary.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark built around whether a repository maintainer would accept a proposed change. Claude Opus 5.5 ranked first with a 52.2% score, but cleared all blocking checks in just 57.7% of attempts, indicating that even the leading model’s pull requests still need close review.
Artificial Analysis added the results on October 12, running each model through the same open-source agent harness, Stirrup. Each model had up to 250 turns per attempt and three attempts per task. The evaluation used the 100-task Main set from FrontierCode; nine of those tasks were not solved by any model on any attempt.
GPT-6.1 Sol placed second with a 49.1% score and a 54.7% pass rate. Claude Fable 5.1 followed at 49.0%, then GPT-6 Astra at 47.0%. The score reflects more than passing tests: a patch must clear all blocking checks to receive a nonzero result, with further quality checks affecting its final score. Artificial Analysis says the review step uses GPT-6 Luna (medium) as the judge.
The benchmark also reports a substantial difference in task cost. Artificial Analysis lists Claude Opus 5.5 at $10.15 per task and GPT-6.1 Sol at $0.81. Xiaomi’s MiMo-V2.6-Pro scored 45.4% at $0.18 per task, making it the highest-scoring open-weights model in the reported results. These figures describe benchmark runs, not a guaranteed cost or outcome in other repositories.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode targets a gap between code that appears to work and a change that is safe and useful to merge. Its reported checks cover hidden-test correctness, command-line regressions and cleanliness, whether an agent’s tests fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM review against maintainers’ criteria.
That approach makes the results relevant to teams considering AI coding agents for pull requests. A passing test suite alone does not establish merge readiness; an overly broad patch, weak tests or changes outside the intended scope can still fail the benchmark. The top model’s 57.7% pass rate is a measure within this test setup, not a prediction that it will succeed at the same rate on every real-world project.
The cost figures also show a trade-off rather than a simple ranking. Opus scored 3.1 percentage points above Sol, while its listed cost per task was about 12.5 times higher. Teams weighing models would need to consider their own review costs, task mix and failure tolerance; the benchmark does not determine which model is best for a particular organization.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Judges Changes
FrontierCode was developed by Cognition with open-source maintainers, after being announced in June 2026. Version 1.1 followed in July with rules against agents looking up solutions online. According to the source report, more than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task.
The complete benchmark contains 150 tasks, while Artificial Analysis ran the 100-task Main set. The tasks are private, which is intended to prevent models from encountering the solutions in training. Cognition’s trajectory scanner can assign a zero to an attempt flagged for solution lookup; failing any blocking check also results in zero. Non-blocking quality criteria affect scores for attempts that clear the required checks.
Results do not track general model rankings exactly. In Artificial Analysis’s comparison, Claude Sonnet 5.5 ranked second among the listed models on its Intelligence Index but fifth on FrontierCode. DeepSeek V4.1 Flash scored 44.0% on FrontierCode despite a lower Intelligence Index result than several models above it. The comparison suggests that general capability rankings and repository-level change quality measure different things.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results do not establish how models would perform across every codebase, task type or team review process. The public account describes the benchmark design and reported scores, but does not provide enough information here to determine how closely its tasks represent the full range of real pull requests or how outcomes vary across individual repositories.
Several models had no reported FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the table is not a complete comparison of all relevant systems. It is also unclear from the supplied material when those models might be tested or whether later runs will change current rankings.
Pass rates and costs are benchmark-specific observations. They do not show the rate at which a model’s code would be accepted without changes in a live project, nor do they account for all costs of human review, agent setup or remediation. The published numbers should not be read as a guarantee of performance or savings.
software development AI assistants
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
More Models and Runs to Watch
The next useful development will be results for models missing from the October 13 leaderboard, particularly those that have performed well on other coding-agent evaluations. Additional runs could alter the ranking and make comparisons across a broader field possible, though no release date for those results is specified in the source material.
Maintainers and engineering teams can also watch for updated benchmark versions and results across further tasks. For now, the reported findings support a limited conclusion: models can produce strong repository changes, but even the top result did not clear every blocking check in more than half of attempts. Human review remains part of the process, and the benchmark does not show that teams can safely merge generated changes without it.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
FrontierCode evaluates whether AI-generated code changes satisfy maintainers’ acceptance criteria, including correctness, regressions, test quality, scope limits and code review. Passing ordinary tests alone is not enough to pass its blocking checks.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in the reported results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the result mean AI pull requests can be merged without review?
No. The benchmark measures performance on its own task set, and the leading model failed to clear all blocking checks in 42.3% of attempts. The results do not establish that generated changes are safe to merge without human review.
How much did the models cost per task?
Artificial Analysis reported costs ranging from $0.09 per task for GPT-6 Luna to $12.00 for Claude Fable 5.1. Listed costs are specific to the benchmark setup and may not match costs in other uses.
Were all major models included?
No. As of October 13, 2026, the source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
