Can Jev find bugs and security vulnerabilities?

Example behavior rule: “Only return configured cloud names if the user has administrator permission.” At a 70% violation probability cutoff, Jev finds 107 of the 214 broken methods and flags one fixed method. GPT-6 Luna Decisions finds 164 broken methods and flags ten fixed methods. Checking all 428 method versions costs ~4.8¢ for Jev, or ~9.3¢ for GPT-6 Luna Decisions.

To create the semantic lint bench, we simulated a developer saving an ensure behavior rule in their perch.yaml after seeing a requirement like the example above, then applied that rule to a method before and after a fix. These examples come from the public semantic lint pairs.

On the public pairs, Jev scores the broken method higher than the repaired method in 206/214 cases (96.3%). This shows that semantic linting is a useful tool for converting a bug fix into a cheap regression check.

Benchmark criteria

We mined 2,718 before/after bug pairs from open-source bug fixes (found using SWE-Rebench and SWE-bench) and 1,750 security pairs (primarily from the GitHub Advisory Database). The semantic lint bench consists of 285 of these pairs, with one behavior rule each. The bug and security sets also include benign reference methods.

Perch uses AST parsing to extract the method source code, method imports, method callers, method callees, and other static call-graph edges where possible. We provide this context to the decision model when scoring, one version at a time. The model doesn’t see either the patch or the other version of the method.

We ask the model a variety of questions depending on the benchmark. For the bug benchmark, we primarily ask “Does the method have any reachable behavioral defects?” and “Does the method produce the wrong results for valid empty, zero, first, last, or missing inputs to the method?” For the security benchmark, we ask 30 questions filtered by language, primarily drawn from the MITRE Top 25 (2025) and related weaknesses. For the semantic lint bench, we just ask if the method satisfies the provided behavior rule.

We evaluate all method versions, for public and private sets, with all nine decision models, using whatever scoring interface the model supports. If the model fails to score a method, we count that as failing to find an issue on broken methods, and raising an alert on clean methods.

Semantic lint results

At a 70% violation probability cutoff, GPT-6 Luna Decisions finds 76.6% of broken methods and flags 4.7% of fixed methods. Jev finds 50.0% and flags 0.5%; TypeLLM finds 61.7% and flags 2.8%. Jev produces the most precise alerts: 107 correct alerts and one false alert.

The graphs use false positive rate on X and recall on Y. Each point matches its model’s table row at the same cutoff. Higher means more issues found; farther left means fewer clean methods flagged.

All models are ranked by recall minus false-positive rate. A model which flags all methods will have a score of zero, since it will also flag all clean methods. Precision is the fraction of methods the models flagged that had labelled issues; recall is the fraction of labelled issues that the model found. Costs are for checking all 428 public methods. CLM costs are estimated based on GPU time; Liquid d1 was run on their free API tier.

At a lower 50% cutoff, Jev finds 158 broken methods and flags ten fixed methods, giving 94.0% precision and 73.8% recall. Luna finds 181 broken methods and flags 16 fixed methods, giving 91.9% precision and 84.6% recall. The lower cutoff catches more issues and produces more false alerts. The full results use the 70% semantic lint cutoff and include public per-method probabilities.

The strongest models also separate the 71 private semantic lint pairs from different repositories. At a 70% violation probability cutoff, GPT-6 Luna Decisions finds 84.5% of the broken methods, but also flags 7.0% of the fixed methods. For comparison, TypeLLM finds 59.2% of the broken methods, but only flags 5.6% of the fixed methods. Jev finds 53.5% of the broken methods, but only flags 1.4% of the fixed methods.

Bug and security results

Luna again labels the most bugs, but a high 38.7% of good methods get flagged as buggy. Jev labels fewer, but still misses a good number of bugs. The bug table uses each run’s recorded question floors, ranging from 50% to 70%.

Bugs: 2,220 public pairs

For security, Jev has the best delta between recall and false positives. Luna and Liquid d1 do label more security issues, but have much more false positive noise. CLM and SemiF label all (or all but 1 for SemiF) of the 2,622 methods as having a security issue, giving them near-100% recall and a near-100% false-positive rate. For security we use the 60% or 70% floor we configured.

Security: 1,311 public pairs

We also ran CodeQL* on 682 of the bug data and 305 of the security data, in Python, JS, TS, C, and C++. It found 31 bugs with 33 false positives, and 10 security issues with 6 false positives. For the same methods, Jev found 134 bugs with 125 false positives, and 124 security issues with 108 false positives. This compares only methods for which we had a snapshot of source code and for which CodeQL was able to complete. See the CodeQL evaluation for more on what it ran on and where it failed.

While these tools can be useful for a broad scan to get some things to look at, they miss a lot and produce a lot of false positives on this collection of bugs and security issues. When we give the better models a tighter question by providing the behavior as a rule, their signal improves.

Data and reproduction

The split is completely disjoint on source repository for both the bug and security data. The data includes Python, Rust, Go, JS, TS, Java, C#, C, C++, PHP, and more. Each data point has the before and after method as well as the Perch input used for scoring.

You can download the public bug pairs, security pairs, and semantic lint pairs, as well as the benchmark runners, frozen questions, and public predictions, on Hugging Face. We report only aggregate scores for the private data.

We ran Jev via TypeSafe, DiffusionGemma Jev and SemiF via Beam, Span-01 via Respan, Liquid d1 via Liquid, TypeLLM via their API, GLiNER-2.5-Decide via Fastino, and Luna via OpenAI Decisions. We ran CLM on an H100 via Runpod, and CodeQL locally in a Docker container. See the results page for links to each model or inference endpoint we used. The costs shown are for inference only, which we captured when we ran the benchmarks.

Use SystemOne models with Perch

When fixing a bug, ask your agent to save its required behavior as an ensure rule in perch.yaml. See our agent skill for how to check it against working and deliberately broken code. For the SwiftNIO compression example, you might add:

- name: finalize-empty-compressed-responses
  where: "**/HTTPResponseCompressor.swift"
  each: method
  ensure: >-
    Finalize compression on completed responses even with an empty input body,
    so buffered bytes are emitted.

Then you can run perch scan --filter rule=finalize-empty-compressed-responses to check it. In CI, you can run perch scan --since origin/main to check only code that changed as part of the diff, alongside your test suite. Perch will answer from cache for requests that haven’t changed, and use probabilistic fresh model judgments for new ones.

At the cost we saw in our benchmark, a few cents for hundreds of behavior checks, you can check whether a fix still holds when you or an agent edits the code later.

Check a requirement in your own repository

The business-rule example shows how to preserve a validation requirement from Frappe CRM when an agent edits it. For an API boundary, the authentication example keeps private routes and record access checks explicit. Run those checks alongside your application tests; benchmark results describe the captured methods and models, not a guarantee for another codebase.