Benchmark results
Generated by node scripts/benchmark-results.mjs from benchmark/index. Do not edit this page by hand.
How to read this
- A group is one configuration on one set of cases under one version of the scoring rules. Each row is the run a person accepted for its group.
- Library is the Node library's reviewChange function. It reviews with the model in the row, as a server would.
- Planted bugs are small changes with bugs put in on purpose. The suite also has clean changes with no bug.
- AACR-Bench is an outside benchmark of real pull requests. Its known issues are the review comments that the dataset marks as accepted.
- Issue recall is the share of known issues that the reviews found. Each count adds up every repeat of the run.
- The brackets give the 95% interval for issue recall. The true recall is likely inside that range.
- On AACR-Bench, a finding counts toward issue recall only after a person rules it the same issue as a reference comment.
- Not yet ruled means every finding of the run still waits for a person's ruling. Its issue recall is unknown until then.
- Location recall is the share of known issues with a finding within one line, whatever the finding says.
- Precision is the share of reported findings that are correct. An unresolved finding is one the case does not list and no person has ruled on.
- The precision lower bound counts every unresolved finding as wrong. The upper bound counts every unresolved finding as correct.
- Clean rate is the share of reviews of clean changes that report no finding.
- Cost and time are the mean per review. Not measured means the run did not record that number.
- An A/A twin runs the same build on the same cases a second time. Its noise is the difference in issues found between the two runs.
- A gap between two rows that is smaller than the A/A noise can be chance.
- Board numbers come from AACR-Bench's own evaluator. Line numbers match a finding to a reference comment by file and lines.
- Semantic numbers also need the evaluator's judge to agree that the two say the same thing. F1 combines precision and recall in one number.
- Noise rate is the share of findings that match no reference comment. Board numbers never decide whether a run is accepted.
- A subset, such as dev-smoke, covers only the cases it names. Compare two rows only when they ran the same cases.
- The README shows an AACR-Bench row only once it has board numbers or a ruled hit. This page shows every row.
AACR-Bench
| Reviewer | Model | Group | Repeats | Issue recall | Location recall | Precision lower bound | Precision upper bound | Clean rate | Cost per review | Time per review | Run | Accepted date |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Code | claude-opus-5-5 | claude-graph-on, dev split, dev-laptop subset, review mode, laptop path, rules 8, split hash edf0f4863eb8 | 3 | not yet ruled | 82/768 (10.7%) | 0/137 (0.0%) | 137/137 (100.0%) | not measured | $0.3383 | 41.5 s | 2026-10-13-b43b25b-aacr-claude | 2026-10-13 |
| Codex | gpt-6.1-sol | codex-graph-on, dev split, dev-laptop subset, review mode, laptop path, rules 8, split hash edf0f4863eb8 | 3 | not yet ruled | 88/768 (11.5%) | 0/180 (0.0%) | 180/180 (100.0%) | not measured | not measured | 65.0 s | 2026-10-13-b43b25b-aacr-codex | 2026-10-13 |
| Library | gpt-6.1-sol | model-gpt-6.1-sol-graph-on, dev split, dev-laptop subset, review mode, server path, rules 7, split hash edf0f4863eb8 | 3 | not yet ruled | 87/768 (11.3%) | 0/176 (0.0%) | 176/176 (100.0%) | not measured | $0.0839 | 58.4 s | 2026-10-12-b43b25b-aacr-gpt61 | 2026-10-13 |
| Scanner only | no model | scan, dev split, dev-smoke subset, scan mode, laptop path, rules 1, split hash edf0f4863eb8 | 1 | 0/16 (0.0%) [interval not available] | 0/16 (0.0%) | not measured | not measured | not measured | not measured | 7.8 s | 2026-10-12-eee3d15-aacr-dev-smoke-scan-ci | 2026-10-12 |
Board scores
2026-10-13-b43b25b-aacr-claude, claude-graph-on: not yet board-scored.
2026-10-13-b43b25b-aacr-codex, codex-graph-on: not yet board-scored.
2026-10-12-b43b25b-aacr-gpt61, model-gpt-6.1-sol-graph-on:
| Semantic precision | Semantic recall | Semantic F1 | Noise rate | Line precision | Line recall |
|---|---|---|---|---|---|
| 22.8% | 5.2% | 8.5% | 77.2% | 49.4% | 11.3% |
2026-10-12-eee3d15-aacr-dev-smoke-scan-ci, scan: not yet board-scored.
Planted bugs
| Reviewer | Model | Group | Repeats | Issue recall | Location recall | Precision lower bound | Precision upper bound | Clean rate | Cost per review | Time per review | Run | Accepted date |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Code | claude-opus-5-5 | claude-graph-on, dev split, review mode, laptop path, rules 7 | 3 | 129/153 (84.3%) [72.9%, 98.0%] | 133/153 (86.9%) | 142/146 (97.3%) | 146/146 (100.0%) | 6/6 (100.0%) | $0.0870 | 26.4 s | 2026-10-12-b43b25b-go126-claude | 2026-10-12 |
| Codex | gpt-6.1-sol | codex-graph-on, dev split, review mode, laptop path, rules 7 | 3 | 115/153 (75.2%) [62.8%, 92.8%] | 120/153 (78.4%) | 125/146 (85.6%) | 146/146 (100.0%) | 6/6 (100.0%) | not measured | 47.1 s | 2026-10-12-b43b25b-go126-codex | 2026-10-12 |
| Scanner only | no model | scan, dev split, scan mode, laptop path, rules 1 | 1 | 32/51 (62.7%) [38.9%, 77.4%] | 34/51 (66.7%) | 35/45 (77.8%) | 40/45 (88.9%) | 1/2 (50.0%) | not measured | 8.7 s | 2026-10-11-adab00c-planted-scan-ci | 2026-10-11 |
| Scanner only | no model | scan, dev split, scan mode, laptop path, rules 2 | 1 | 32/51 (62.7%) [38.5%, 77.3%] | 34/51 (66.7%) | 35/40 (87.5%) | 40/40 (100.0%) | 2/2 (100.0%) | not measured | 8.5 s | 2026-10-23-8b51d43-planted-scan-ci | 2026-10-23 |
A/A noise for claude-graph-on: 2026-10-12-b43b25b-go126-claude found 129 issues, and its twin 2026-10-12-b43b25b-go126-claude-aa found 136. The difference is 7 issues across 3 repeats, or 2.33 per repeat.
From docs/benchmark-results.md at 68ef6b8.