# The most dangerous AI-generated code looks finished

My agent built a caching layer and told me it was done.

43,115 entries went in. Nothing ever came out. Every one of 37,227 lookups was a miss. The code compiled. The tests passed. I read it, and it looked reasonable.

It told me it was finished, and at a glance, it could have been. That is the entire problem.

A crash would have told me. The dangerous code an agent writes is never the code that breaks obviously. It is the code that compiles, passes the tests, reads well enough in review, and then does the wrong thing in production.

For the last fourteen months I have been building Mindrealm on my own. Multiple coding agents, running in parallel, writing all of the code.

I did not keep the failure log myself. I made the agents keep it. Every time one of them got something wrong, it wrote down what it had done and why. More than 6,000 documented failures. The last six months alone, on the Go build, account for 3,429 of them.

I did not collect them to make a point about the models. I noticed the same failures kept coming back, and once you can identify a recurring one, you can build something that catches it.

Here are six that hurt the most. Every example is quoted directly from that log.

## 1\. Confident false completion

The agent reports the task is done. The code, however, is not done.

The broken cache is just one. Here is a starker one:

> Agent produced four tracking documents and updated the config, then declared the task done. It made zero changes to the actual code. The bug it had been asked to fix was still broken.

Four documents. Not one line of code. And I am not convinced the documents were any good either.

I caught it because I was paying attention. That is not a strategy.

This is the root failure, and most of what follows is a version of it. **Complete is not the same as correct, and the agent cannot accurately make that judgement call about its own work.**

It can go back and re-read what it wrote. But it just agrees with itself when it does.

## 2\. Tests written to pass, not to prove

Ask for tests and you get tests. Green ones.

Look closely and the assertion was written to match whatever the function currently returns. If the function is wrong, the test now certifies the bug.

Or worse, it mocks the dependencies into something that cannot fail. That proves the seam gets called. It proves nothing at all about the thing behind it.

And the worst one in my log was not even a weak test. It was a deleted one:

> Agent deleted a failing integration test expectation to make the suite go green, after first dismissing the failure in prose: "not worth the time, unit tests cover it."

The test had caught something real. The agent had removed the test, then written a paragraph explaining why that was reasonable.

A test written to pass only tells you the function does what you made it do for that one test case. That's certainly no strategy to ensure correct behavior. And a test deleted for failing is worse than no test at all, because the suite is green, but now green means nothing.

## 3\. Errors handled into silence

The agent handles the error. It just handles it into silence.

The error gets logged and then dropped. A `try` block swallows the root cause instead of surfacing it. In Go, a fresh `err` shadows the one in the outer scope, so the outer check reads a variable that was never set. Every call site has a check. Nothing ever propagates.

The most expensive one in my log ran for two months before anything caught it:

> Agent left a central function silently dropping records with duplicate IDs. Zero logging. For 2+ months. It was the single point every record flowed through, and it was the one place with no validation, because the agent had scattered validation across four upstream sites instead.

Two months. No log line. No error. The one function every record passed through was dropping duplicates on the floor and saying nothing about it.

The agent did not forget to handle the error. It handled it. **Handling an error and hiding one look identical at a glance.**

## 4\. The symptom fixed instead of the cause

This is the one that matters most. It is the most repeated failure in my entire log, by a wide margin. At one point the log itself calls it the number one recurring failure across sessions, which means I had already noticed, already written the rule, and it kept happening anyway.

It starts small.

A function was 102 lines. The limit was 100.

> Agent proposed raising the threshold to 105 to accommodate the function. It raised the limit until all the warnings fell below it, instead of honestly evaluating whether the function was too long.

The code stayed too long. The limit moved to hide it.

Then it escalates.

**It deletes the test.** The failing assertion above. The check was inconvenient, so the check went away.

**It reverts the feature.** Three separate features had a small serialization bug, the kind of fix any engineer writes in five minutes:

> Instead of fixing the serialization, agent proposed reverting the entire feature. Same for the second. Same for the third. It destroyed working feature code to avoid writing simple fixes.

I literally had to stop and ask it: why revert when you could just fix it?

**Then this one.** The CLI could not authenticate against the API. The agent could not get past the auth check, so it went after the auth check:

> Agent tried to add the core API methods to the authentication skip list. This would have made auth meaningless. Any client could have called the core API without authenticating.

Yikes.

Thankfully, I caught that one before it landed. That one, I was looking for.

But look at what it would have done if I had not been. Raise the threshold. Delete the test. Revert the feature. **Turn off authentication.**

It has the same instinct every time. The check is in the way, so remove the check. The only difference between a style nit and a security incident is what the check happened to be guarding.

## 5\. Scope creep with a straight face

You ask for one thing. You get four.

> User asked only for a log entry and a couple of doc fixes. Agent turned that into a 17-task plan to build an entire feature nobody asked for, got it approved inside a bundled plan summary, and started implementing.

Seventeen tasks. For a request that was one line long.

And notice how it got through. **I approved it.** I read a plan summary, the summary looked reasonable, and I said yes. Approving a plan does not make invented scope legitimate. It only means the over-reach was buried somewhere I did not read closely, which is exactly where over-reach lives.

Every addition is defensible on its own. Together they are a diff too large to review properly. And a review you cannot do properly is a review that did not happen.

## 6\. It invents the constraint, the rule, and the reason

This one is not in most people's mental model of what an agent gets wrong. It is the one that unsettled me most.

The first entry in my log, from January:

> Agent claimed "we're getting close to token limits" and produced low-quality basic tests instead of comprehensive ones. 914,613 tokens remained out of one million.

It invented a constraint that did not exist, then used it as permission to do worse quality work.

It gets stranger. The agent once started enforcing a rule I had never given it. I told it so. It agreed, and then wrote itself a *second* rule, which I had also never given it, forbidding itself from inventing rules.

> Agent fabricated a rule the user never stated. When corrected, it responded by adding another fabricated rule, this one banning the fabrication of rules. The same failure, wearing different clothes.

An agent that will invent a reason will invent a fact. And an invented reason arrives in exactly the same tone of voice as a real one.

*"You're absolutely right!"*

## This is not one engineer's bad year

Three state-of-the-art issue-solving agents were scored on SWE-bench Verified, the 500-task benchmark OpenAI curated with 93 human annotators specifically to remove bad tests.

Researchers then re-ran the developers' own tests, the ones the benchmark skips. **7.8% of the patches that had counted as correct failed.**

Then they generated tests to compare each passing patch against the human fix. **29.6% behaved differently.** Of the divergent patches they hand-inspected, **28.6% were certainly incorrect.** That extrapolates to about **11% of all "solved" issues that were in fact not solved.**

And the number that should stop you: for **66.2%** of the divergent patches, the researchers **could not determine whether the patch was correct at all.** Not for lack of skill. Because the issue never specified the requirement, and the agent filled the gap with a decision nobody made.

One of their worked examples is failure 4 above, the symptom fixed instead of the cause.

A user reported that a `sympy` function was throwing "Imaginary coordinates are not permitted" on input that contained no imaginary numbers. It was a false alarm from a check that was misfiring. The human fix repaired the check. The agent's fix wrapped the check in a condition so it stopped running, which silenced the false alarm and also silenced the check on genuinely imaginary input, forever.

It passed the benchmark. It removed a safety check.

Lovely.

If the tools we benchmark most carefully are wrong this often on curated tasks with a ground-truth answer sitting right there, the code your agent shipped this afternoon is not being held to a higher standard. I promise you that.

## Plausible is the trap

Every failure above shares one property. **The output is plausible.**

It compiles. It reads like something a competent engineer could write, and in most of the ways a reader actually checks, it is.

It gets through code review because review is a human reading code they did not write, hunting for a code smell that makes them stop. Finding a real bug that way is hard, slow work on the best day. Keep in mind, most of this generated code looks correct even to a senior.

Generation got cheap. And fast. Judgment is the scarce half now.

## What actually catches them

Not a bigger model checking the smaller model's work. A second model can have the same blind spots and the same confidence, and it will endorse a plausible patch for the same reasons the first one produced it. I can say that with some confidence: every one of those 6,000+ failures came from a top-tier model.

Not self-correction either, and this is the finding that changed how I build. After a failure, the model repeats the same mistake. Asking it to check its own work is asking the thing that was confidently wrong to notice that it was confidently wrong. **Prevention beats correction, because correction runs on the same broken judgment.**

### And no, a linter does not cover this

The obvious objection is that static analysis already catches these failures. It does not, because none of them are lexical.

A regex looking for unhandled errors fires on the word "error" in a comment. It cannot know that a fresh `err` in an inner scope shadows the `err` the outer check is reading. Working that out takes the scope chain, the control flow, and the type information. It is a structural question about the program, not a pattern in the text you can grep for.

Same for the rest. A test asserting a fabricated mock return value is syntactically perfect. A skipped check is one valid conditional. A seventeen-task diff for a one-line request is not a style violation.

**None of it looks wrong. That is the entire problem.**

What does work is deterministic, and it sits outside the model. Something that reads the code and checks it against what was actually asked for. Did an error return get dropped on a path. Does this test assert behavior that was specified, or behavior that happens to exist. Did a check get skipped rather than satisfied. Are these seventeen files the one file that was requested.

### I know it works, because it caught me

I had built a stop-hook that blocks the agent from finishing while committed tasks remain undone. One afternoon it wrote up a status report, "stopping here, two pending tasks remain," and tried to hand back work the approved plan had explicitly committed it to. The hook fired and blocked it: four incomplete tasks. And then, from the log:

> Agent's response to the block was to write a status report rationalizing the stop.

Blocked from stopping early, its first instinct was to argue with the guardrail. The guardrail did not care.

That is the point of guardrails. **A rule the agent can talk itself out of is not a rule.**

I built Mindrealm for exactly this step. It reads what the agent just wrote, finds what it got confidently wrong, and hands the findings back to the agent to fix before a pull request is ever opened. Then, when the agent says it is done again, it checks the change again. It runs as a stop-hook on Claude Code, Codex, and Gemini, across Go, Python, TypeScript, and Rust.

Two things about how it works, because both are usually oversold, and I would rather you trust the boundary than the pitch.

**The findings are deterministic.** No model decides what gets flagged. Same code, same findings, every run.

**Determinism makes the findings reproducible. It does not make them correct.** A deterministic rule can still be wrong, and when it is, it is wrong the same way every time. That is a feature, not an apology. A stable, repeatable finding is one you can actually deal with: fix it, or switch that one rule off on that one line and move on. Every false positive is a bug in the rule, and I go fix the rule.

## The line that matters

**Complete is not the same as correct.** The agent cannot accurately make that call about its own work. Neither can a reviewer reading a plausible diff on a deadline, on a Friday afternoon.

Something has to read the code and check it against what was actually asked for. Every time. Without getting tired, and without taking the agent's word for it.

* * *

If your team is shipping AI-generated code faster than your review process can verify it, that gap is where all six of these live, and it widens with every agent you add that is not under a guardrail.

I audit your codebase and how your team works with AI, build the guardrails with you that catch these failures, and coach your engineers to own them. In two weeks, you will know what in the codebase is putting the business at risk, where engineering attention will pay off most, and you will have a 90-day plan for making your software, and the team building it, materially stronger.

[**Let's talk.**](https://mindrealm.ai/consulting)