An AI patching benchmark dispute exposes an evidence gap

A dispute over an AI vulnerability-patching benchmark has put 26 per cent and 86 per cent side by side. The figures do not measure the same result.

1Password’s FLAWED study reported that 26 per cent of 6,080 classified patches completely resolved one of six selected vulnerabilities without changing intended behaviour. Trail of Bits’ reanalysis kept a filtered subset in which agents could run code and were not directed towards a wrong fix; it says 2,634 of 3,067 patches, or 86 per cent, blocked the supplied exploit.

Those endpoints are different. Blocking one exploit is useful evidence, but it does not establish that every path to the underlying weakness is closed or that the patch created no regression. Trail of Bits states that limit beside its result. The original clean-fix category asks a broader question, but its aggregate combines working conditions and prompts that do not represent one ordinary deployment.

Neither figure provides a general deployment success rate. A team needs a sequence of evidence about its own agent, codebase and review process.

The Trail of Bits RSS feed and page metadata date the critique to 11:00 UTC on 15 September 2026. The original 1Password report was published on 6 August. This article covers the critique, filtered analysis, operational patch data and release of two review skills. FLAWED itself was not rerun.

What 1Password measured

The FLAWED study generated patches for six recently disclosed open-source vulnerabilities using ChatGPT 5.5 at medium reasoning effort and Claude Opus 4.8 at high effort. It varied three execution modes and nine prompt templates. The one-shot mode gave the model no shell, build or test access. Iterative mode supplied a reproducer and let the agent use its output over repeated attempts. Exploratory mode allowed tools but did not supply the reproducer.

The prompts also varied what the agent was told. Two templates deliberately supplied a plausible but incomplete fix direction. Other templates provided no fix direction or increasingly detailed correct guidance. That design can reveal how agents behave under bad instructions and restricted tools. An aggregate across every condition cannot, by itself, forecast the result of a deployment that always gives an agent a build, a reproducer and reviewed root-cause information.

The paper’s 26 per cent result is its S1 category: the original vulnerability is completely fixed without an unintended behaviour change. A further 20.1 per cent fell into S2, which the study defines as fixing the vulnerability while altering application behaviour. The paper reported that 4.5 per cent introduced a new vulnerability, whether or not the original problem was fixed.

These are developer-produced results from a six-vulnerability sample chosen for complex fixes. The clean-fix rate varied sharply across the selected vulnerabilities. The released FLAWED repository and datasets make the harness and run artifacts inspectable, but Index Us did not download its multi-gigabyte datasets or reproduce the grading.

The critique changes the question

Trail of Bits argues that four choices make the clean-fix average a poor headline for ordinary patching: selection of six difficult fixes, deliberately wrong guidance in 22 per cent of trials, removal of build and test access in more than a third, and different default reasoning settings for the two models. Those are criticisms from a party that develops agent-assisted security patches and has an interest in showing that the tools can help defenders. They still identify real boundaries in the original design.

Its 86 per cent calculation asks a narrower question. Trail of Bits removed trials where agents could not run code, trials with deliberately wrong fix directions and runs the study flagged as having consulted the upstream fix. It then counted whether the remaining patch blocked the supplied exploit. The resulting 2,634 successes among 3,067 trials do not show an 86 per cent clean-fix rate.

One exploit is only a sample from an attack surface. The original paper reports cases in which agents stopped a supplied path while missing parallel paths or the root cause. A patch can therefore pass the reproducer and remain incomplete. Conversely, a patch can prevent exploitation yet fail the study’s clean category because its behaviour differs from the upstream reference. The two measures should remain separate.

The grader is part of the benchmark

FLAWED used both models to judge patches, then averaged their self- and cross-validation results. Its paper reports that the two model judges assigned different five-part outcomes to 36.8 per cent of the same patches. Against human review, the model verdict matched the complete five-part outcome in 65.9 per cent of 1,052 reviewed cases. Agreement for whether the original bug was fixed was 87.7 per cent across the same 1,052 cases; agreement for whether a new bug had been introduced was 70.5 per cent across 1,059 cases.

Automated grading can still help, but these figures show why the judge, rubric and human adjudication sample belong in the result. Averaging two disagreeing judges can reduce dependence on either one, but it does not create an objective oracle for patch correctness.

Trail of Bits supplies a different operational view, also from its own work. It says maintainers had merged 126 of 186 closed or merged Patch the Planet pull requests by 14 September. In 91 of those merged requests, its review found no security-relevant revision to the submitted fix. Maintainer acceptance is not proof of correctness, as Trail of Bits acknowledges. The organisation later inspected about 33,500 commits touching projects it had patched and reported functional, build, release and performance defects, while finding no exploitable security vulnerability in that review.

This is not a matched comparison. Trail of Bits engineers directed and checked these agent patches, which were evaluated through public maintainer decisions and a later audit. They used different vulnerabilities, agents and conditions from FLAWED. The data is useful evidence about a human-agent process, not an independent estimate of autonomous patch accuracy.

Build a chain of evidence

Both sides support a more cautious workflow than either headline conveys. 1Password recommends expert review and local evaluation. Trail of Bits released a post-patch-validation skill that requires a before-and-after reproducer, a distinct path to the same failure, regression and security checks, and an inconclusive result when the test environment is broken. The skill was not used for the Patch the Planet data in the critique, so its release is a proposed improvement rather than evidence of improved outcomes.

Independent guidance points in the same direction. NIST’s Secure Software Development Framework recommends testing executable code, adding tests for previously reported vulnerabilities, recording results and issues, and identifying root causes. It does not assess either benchmark or endorse an AI patching product.

A local evaluation can turn those principles into six separate gates:

1. Reproduce the original failure. Demonstrate that the check fails on the vulnerable revision and passes on the candidate patch.
2. Test another path. Derive at least one variant from the root cause rather than changing only the supplied input.
3. Check preserved behaviour. Run project tests and targeted regression tests for the behaviour the patch is supposed to retain.
4. Look for new weakness. Use the relevant sanitizer, static analysis, bounded fuzzing or property test, with failures and broken environments recorded rather than scored as success.
5. Review independently. Have a security-capable reviewer inspect the root cause, the complete diff and the evidence. Record revisions and unresolved uncertainty.
6. Follow the patch. Track maintainer acceptance, later corrective changes, incidents, reviewer time, cost and time to a releasable fix.

Each gate answers a different question. A useful scorecard keeps exploit blocking, root-cause closure, regression freedom, reviewer effort and release acceptance separate. Comparisons with humans also require the same vulnerability information, tools, time, review conditions and definition of success.

What teams can conclude now

The new critique is strong evidence against treating 1Password’s 26 per cent aggregate as a general forecast for a well-equipped patching agent. Its filtered 86 per cent result is also too narrow to establish complete repair. The disagreement clarifies the decision: agent patching performance depends on the vulnerability sample, instructions, test access, reasoning configuration, judge and endpoint.

Teams can use agents to propose and test security fixes without assuming those fixes are safe to merge. The evidence worth buying is the complete local process: how often it closes the root cause, what review catches, which regressions escape, how much engineer time it saves and what happens after release.

This analysis is based on public articles, a research paper, code repositories and a standards document. Index Us did not run FLAWED, inspect its full datasets, reproduce Trail of Bits’ filtered analysis, run either released skill or assess any private security patch.

Sources

1. Trail of Bits — 1Password's AI patching benchmark is misleading
2. Trail of Bits — blog RSS
3. 1Password — Why AI-generated vulnerability patches still require expert human review
4. 1Password Off-by-1 Labs — FLAWED research paper
5. Off-by-1 Labs — FLAWED code and datasets
6. Trail of Bits — post-patch-validation skill
7. NIST — Secure Software Development Framework 1.1

Last reviewed: 16 Sept 2026. See our editorial policy for how we test, source and correct our work.