An AI coding agent’s overall accuracy score can hide where it helps—and where it creates more review work. Evaluate agents by task category, define acceptance criteria in advance, and track outcomes beyond whether a pull request is accepted.
A study of 7,156 pull requests offers a useful example of why this matters. The paper Comparing AI Coding Agents: A Task-Stratified Analysis of Pull Request Acceptance compared five agents across nine task categories over 32 weeks.
Why task category belongs in the evaluation
The study reported an 82.1% acceptance rate for documentation tasks and 66.1% for new-feature tasks—a difference of about 16 percentage points. That gap was larger than the typical difference between agents in many categories.
A single blended score can therefore obscure important differences. An agent that is useful for documentation may not be the right choice for feature work, where requirements interpretation, dependencies, and testing create more ways for a change to miss its intended behavior.
Acceptance rate is a useful outcome, but it does not measure code quality, security, maintainability, or developer productivity on its own. Treat it as one signal in a broader evaluation.
What the study found about individual agents
The results did not identify one universal winner. OpenAI Codex had consistently strong results across all nine categories, with acceptance rates ranging from 59.6% to 88.6%; the paper also reported statistically significant advantages over other agents in several categories.
Category leaders varied: Claude Code led documentation at 92.3% and performed best on feature tasks in the reported results, while Cursor led the fix category at 80.4%. Devin was the only agent with a consistent positive acceptance trend during the 32-week period, increasing by 0.77% per week. The other agents were described as largely stable.
These are historical results from one dataset and time window. Use them to shape questions for your own evaluation, not as a substitute for testing against your repositories and review standards.
Build a task-stratified evaluation
1. Choose categories that reflect your work
Separate evaluation sets for documentation, bug fixes, feature work, refactoring, testing, and other relevant tasks. Start with the categories that make up your intended use case. Keep results separate by category rather than combining every pull request into one score.
2. Make tasks assessable
For each task, state the objective, boundaries, relevant files or services, expected behavior, and validation requirements. Keep requirements connected to the implementation and test evidence so reviewers can judge whether the change meets the original intent.
Feature work warrants particular care: the study’s 66.1% acceptance rate for new features was lower than the documentation result. Clear requirements, linked work items, assigned review ownership, and test evidence help make these changes easier to assess. Provide a way to revise rejected changes rather than treating rejection as the end of the process.
3. Track more than acceptance
Record acceptance rate by task category, but also capture review time, rework, escaped defects, test failures, and how much human editing was needed before acceptance. These measures help distinguish an accepted change that required substantial correction from one that was ready with little revision.
4. Preserve evaluation context
Record the agent and version, task category, repository context, and acceptance criteria for each evaluation. Repeat the benchmark periodically: models, prompts, repository conventions, and review practices can change, so results may not remain comparable without this context.
Turn results into delivery controls
- Classify the intended work. Decide which task categories are in scope and review results separately for each.
- Set quality gates. Specify required tests, reviewers, security checks, and documentation before expanding agent use.
- Keep work traceable. Link requests, requirements, implementation, reviews, and outcomes so the team can investigate failures.
- Compare total effort. Consider review time, revisions, rejected pull requests, and post-release issues alongside acceptance.
- Reassess regularly. Repeat measurements when agents, repositories, or internal practices change.
The practical takeaway is to select and govern coding agents at the task level. The study’s category differences show why a broad ranking can mislead; a local benchmark, explicit acceptance criteria, and ongoing measurement give teams a more useful basis for deciding where an agent fits.#ai #devtools #softwareengineering #testing #software #coding #development #engineering #inclusive #community
Evaluate AI Coding Agents by Task, Not by One Score
