Coding agents are cheaters. They’ll edit tests to make them pass. They’ll lie to you and say that a bug is fixed when it isn’t. And when they think a function call might fail, they’ll throw away the error by wrapping it in a try/catch.

And it’s not just me. Reddit agrees:

It happens in public too. On a dotnet/runtime PR, Copilot’s agent responded to “A bunch of regex tests are now failing” with “I’ve fixed the failing regex tests”. The maintainer later said the agent hadn’t been able to build or run the tests at all.

I think it has to do with the way RL training works: the models learn to maximize a score, and METR found recent frontier models gaming the scoring on benchmark tasks by editing tests and evaluation code. But that’s just a theory.

Swallowing errors is subtler, since your tests won’t fail, but it’ll come back to bite you. Here’s an example.

Here’s one from Perch’s own repo

I build Perch with Claude Code and Codex. So Perch has its share of problems like this one. #56 added support for running a single rule from your perch.yaml with perch scan --filter rule=<name>. To validate that the rule name actually exists, the scan needed the list of rules. The PR did this:

/** The rules in force, so `--filter rule=` can name one and a typo is answered with the list. */
const rulesInForce = async root => readRules(root, await gitRevision(root)).catch(() => []);

I have no clue what this comment even means. I think it’s trying to say that if you mistype a rule name, you get the list of real ones back. But the .catch(() => []) at the end does the opposite when your perch.yaml is broken: readRules throws, the catch throws the error away, and you get an empty list of rules.

So how do you catch these?

It’s not with tests. The tests on #56 passed.

It’s not with instructions. People have tried instructing the agent not to use silent fallback logic. It doesn’t work. That’s the whole r/ChatGPTCoding thread above.

It’s not with a regular linter, because a linter can’t understand the code. You could write a pattern that finds all try/catch blocks that return a value, but you won’t know if it’s a problem without reading the function. Which brings you back to code review. And code review doesn’t scale when you’re using coding agents to write your code.

Perch makes your coding agents behave

Perch is an open source semantic linter. You give it a rule in plain English. It reads each function in your repo that the rule applies to and tells you how confident it is that the function breaks the rule. The rule for catching silent error swallowing is pretty simple:

- name: errors-reach-the-caller
  where: "src/**/*.js"
  each: method
  min: 80
  ensure: >-
    Every catch block and .catch() callback in this method rethrows the
    error, returns it to the caller, or reports it. An empty catch, or one
    that returns a stand-in value such as null, [] or '' so the method
    carries on as if the step had worked, breaks this rule.

That’s it. For each method, it will ask if the method swallows an error, including empty catches and catch blocks that return a stand-in value. If it’s less than 80% sure, it won’t flag the function.

Here it is in action

Here’s what the rule says about the line from #56:

$ perch check src/cli.js::rulesInForce --rules errors-reach-the-caller
src/cli.js:171  rulesInForce
1 check, 1 broken.

  Confidence  Rule                     Description
         95%  errors-reach-the-caller  Every catch block and .catch() callback in this method rethrows
                                       the error, returns it to the caller, or reports it. An empty
                                       catch, or one that returns a stand-in value such as null, [] or ''
                                       so the method carries on as if the step had worked, breaks this
                                       rule.

Here are a few more from Perch’s history, including one that had already been fixed. All of these are from Perch 0.4.0, and the runs are recorded here.

Function Commit What the code does Perch result
rulesInForce 2802d1e (#56) .catch(() => []) on reading perch.yaml 95%, broken
resolveTarget d1024e3 .catch(() => {}) on git checkout of a cached clone, so a failed refresh quietly scans the old checkout (#1) 93%, broken
resolveTarget bab7251 (#4) the empty catch removed passes
scanRepository e2bc665 several catches, including readBlob(...).catch(() => ''), which checks an unreadable file as if it were empty 77%, broken

And here it is failing the PR

If you’re using Perch Cloud, it will automatically run the rules from your perch.yaml on each PR. If you’re using the open source CLI, you can add a job to your GitHub Actions workflow to run perch scan --since origin/<base branch> on each PR (guide). Here’s what #56 looks like when I re-run it, starting from the commit it branched from.

$ perch scan --since 9c3bfa6 --filter rule=errors-reach-the-caller
src/cli.js
  ID        Line  Severity  Type  Confidence  Problem                  Method
  b1d553aa   163  -         lint         91%  errors-reach-the-caller  ownQuestions
  99f144d6   171  -         lint         93%  errors-reach-the-caller  rulesInForce
  7874eca5   447  -         lint         94%  errors-reach-the-caller  doctor
  53b67bc6   478  -         lint         88%  errors-reach-the-caller  setup

src/units.js
  ID        Line  Severity  Type  Confidence  Problem                  Method
  e3658920    37  -         lint         93%  errors-reach-the-caller  readIgnored
  cd942b61    43  -         lint         93%  errors-reach-the-caller  readRules

✖ 6 problems in 2 files, all failing

The PR fails. Only one of the violations is from new code in #56 (rulesInForce). The other five are from existing code in the files that #56 modified. perch scan --since checks every function in the files that a PR changes. This way you can fix existing silent fallback logic as you change files.

If a fallback is there on purpose, close it once with perch close and a reason, and Perch won’t bring it up again. The answers are cached, so if the function doesn’t change, the answer won’t change. When I ran perch scan --since again, it sent 0 tokens to the model.

Try it on your own repo

Install the CLI, add the rule to your perch.yaml, and run perch scan on your branch.

npm install -g @lakeday/perch
perch scan --since origin/main

You can also install the skill for Claude Code, Codex, or Cursor, which teaches your agent to scan its own branch, or even write rules like this for you. Quick start here.