Perch flags the function holding the known bug in 38 of 50 pull requests in the Greptile benchmark, using the OpenAI Decisions configuration at a 60% cutoff. The saved run used 33.5¢ of model inference across all 50 pull requests. The Jev configuration flagged 22 of 50 and used 14.9¢.
| Recorded Perch configuration | Known bugs located / 50 | Functions flagged | Median flags per pull request | Measured model cost, all 50 |
|---|---|---|---|---|
| OpenAI Decisions, 60% cutoff | 38 (76%) | 490 | 7 | 33.5¢ |
| OpenAI Decisions, 80% cutoff | 28 (56%) | 264 | 4 | 33.5¢ |
| Jev, 60% cutoff | 22 (44%) | 179 | 2 | 14.9¢ |
The pinned Greptile run and scoring scripts identify the code, labels and configuration. Each pull request has one known bug. Perch receives credit when it flags the function containing that bug; Greptile’s published competitor results credit a review comment that identifies and explains it. The scoring units differ, so a function flagged is not equivalent to a correct review comment. Four known bugs sit outside the functions Perch reads.
Choose the detection and review workload together
A higher cutoff reduces the number of findings to inspect. In the same OpenAI Decisions run, moving from 60% to 80% reduced flagged functions from 490 to 264, while known bugs located fell from 38 to 28. These are two ways to score the same saved answers, with the same inference cost.
Greptile’s published comparison, read October 22, reports 82% for Greptile, 58% for Cursor Bugbot, 54% for GitHub Copilot, 44% for CodeRabbit and 6% for Graphite. Those are provider-published comment scores, not fresh competitor runs by Perch. Use the source to inspect their protocol before comparing them with Perch’s function-level results.
Check precision as well as recall on ReviewBench
On the recorded ReviewBench rerun, the OpenAI Decisions configuration at 60% flagged 603 labelled buggy methods, with 1,491 false alerts. Its 40.0% recall means it flagged that share of the 1,506 labelled buggy methods. Its 28.8% precision means that share of its flagged methods was labelled buggy.
| Recorded Perch configuration | Recall | Precision | Correct alerts | False alerts | Measured model cost, 171 scanned PRs |
|---|---|---|---|---|---|
| OpenAI Decisions, 60% cutoff | 40.0% | 28.8% | 603 | 1,491 | 415.69¢ |
| OpenAI Decisions, 80% cutoff | 23.2% | 41.7% | 349 | 487 | 415.69¢ |
| Jev, 60% cutoff | 7.9% | 45.8% | 119 | 141 | 179.82¢ |
The ReviewBench run scores methods, while the ReviewBench leaderboard judges review comments. Its full benchmark has 219 pull requests. The Perch rerun scanned 171; targets it did not read still count as not flagged. Its labels include reference findings and additional judgments of alerts from the earlier frozen run. The reference-only view and development/validation split are published alongside the combined result. These distinctions matter when comparing tools or older configurations.
All prices in these tables are the measured model costs of named saved runs. They are separate from Perch Cloud billing, which charges 10¢ per 1,000 fresh rule checks with cached checks free. They are not a quote for another repository or an estimate of human review time.
Reproduce the scores without another model run
The scripts and saved answers are public. Download the pinned benchmark dataset, then run the scorer from the matching public Perch checkout. Scoring saved answers makes no model requests.
hf download perchscan/benchmark-results \
--repo-type dataset --revision 40285b3a4a91e3aeefd0c8f5544f5f8e94a4c3ac \
--include 'greptile/*' 'reviewbench/*' --local-dir .benchmarks/hf
node .benchmarks/greptile/score.mjs
node .benchmarks/reviewbench/rerun-score.mjs
node .benchmarks/reviewbench/rerun-score.mjs \
--results .benchmarks/hf/reviewbench/results/rerun-jev.jsonl.gz \
--summary .benchmarks/hf/reviewbench/rerun-jev-summary.jsonThe scores above were reproduced from those saved inputs on October 23, 2026. The input revision and scorer commit are linked above.
Check the agent’s next edit
Use the quick start to install and sign in. Run a scan against your base commit, inspect the findings, then recheck after repairing one:
perch setup claude-code # or codex, cursor, pi
perch scan --since <base-commit>
perch issues
perch check <issue-id>To keep checking a project requirement, save a plain-English rule and check the uncommitted file. The CodeRabbit comparison and Greptile comparison cover review scope, saved rules and billing.