Search for a command to run...
Track what agents generate, what engineers accept, and how often senior engineers have to step in.
No comments yet. Be the first to comment.
Turn repeated corrections into rules, hooks, skills, and an independent review loop that improves the next session.
Weak assertions and mocks that verify wiring, not behavior, can still get merged.
Passing tests and clean code do not prove the agent finished the job or followed the process correctly.
Mindrealm caught the second one too. 19 of the 21 findings in a feature the agents wrote were real.
Mindrealm: deterministic review for AI-generated code
8 posts
Field notes from building Mindrealm, an AI code reviewer that catches the slop your agent ships confidently. Posts on why AI agents fail, what it takes to actually trust an agent's output, and my mission to reach zero false positives. For engineers who distrust hype.
Three agents can open three pull requests before lunch, but one senior engineer still has to decide whether any of them should be merged.
All three pull requests can look ready: the tests pass, the diffs look reasonable, and each agent claims it finished every requirement. Those signals help, but the senior engineer still has to verify what the code actually does before feeling confident merging it.
Generating code is cheap, but proving that code deserves to become part of the product is still expensive.
After an agent writes code, two questions still have to be answered: has an engineer or an independent review tool checked it, and has the responsible engineer seen enough evidence to feel confident merging it?
The answers tell you which of three stages the change has reached: generated, reviewed, or accepted.
At the generated stage, the agent has written the code and may have run tests, reviewed its own diff, and reported the task complete.
Those signals are useful, but the agent's self-review carries the same assumptions and blind spots that produced the change. They tell you the code is ready for review, not that it is ready to merge.
At the reviewed stage, an engineer or independent review tool has checked the change against the requirements, its behavior, the tests, and the quality standards that matter in that repository.
A review may leave findings open, show that a test doesn't fully cover the requirements, or show that the implementation technically works while making the next change harder.
A reviewed change is not accepted until the accountable engineer decides the code is ready to merge.
An accountable engineer accepts the change when the team has enough evidence to merge or release it. That means the code behaves as required, any blocking findings are fixed or rejected with a reason, and any changed code has been reviewed again.
The coding agent and an independent review tool can both contribute evidence, but the accountable engineer still decides whether it is enough.
Before merging a change, the responsible engineer still has to answer these questions:
Does the code implement every requirement, or only enough to satisfy the current tests?
Do the tests cover the behavior that changed and the failure paths that matter?
Would those tests fail if the defect were still present, or did the agent weaken an assertion?
Can the changed behavior be tested without brittle setup or hidden dependencies?
Does the code handle the errors, timeouts, and concurrency risks that apply here?
Does the change expose data, weaken authorization, or create another security risk?
Is the code easy to understand and modify, without unnecessary duplication or complexity?
Does it fit the existing architecture, and can it be extended without a rewrite?
Did the change add a dependency or abstraction it does not need?
Does it meet the performance requirements?
If the change triggered a warning, did the agent fix the cause or only suppress the warning?
Whew, that's a lot, and quite possibly even more still needs checking.
Tests, diffs, completion reports, and review findings can help answer those questions, but the engineer responsible for the change still decides whether it is ready to merge.
For work that matters, I want a short record of what the requirements were, what changed, what was reviewed, and why the engineer accepted it.
| Requirement | How it was verified | Review findings | What changed after review | Review of changed code | Decision |
|---|---|---|---|---|---|
| What must be true | Test, trace, reproduction, or other proof | Finding and supporting evidence | Fix or reason for rejection | Result on changed code | Accepted, rejected, or unresolved |
Put the record in the pull request, issue, or whatever already owns the requirement.
The record should answer six questions: What is the requirement? What test, trace, or reproduction shows that the code meets it? What did independent review find? What changed after review? Was the changed code reviewed? Who accepted it, and why?
Without those answers, a green check tells you only that the configured checks passed. It does not prove every requirement was satisfied, and a completion claim tells you only what the agent believes about its own work.
If I were evaluating an agent workflow, I would not start with lines generated, tasks marked done, branches opened, or pull requests created. Those numbers go up before anyone knows whether the work is safe to merge.
I would use:
accepted units / human interventions required to accept them
An accepted unit can be one requirement, one change, or one pull request. Pick the unit your team already uses and keep it stable.
A human intervention is any point where the work requires a person to clarify, correct, judge, or accept it. Explaining the requirement again, rejecting a weakened test, choosing between two architectural options, or sending the agent back to finish missing work all count as interventions.
A study of professional software engineers found that ~80% reported spending less time writing code as their work shifted toward directing, evaluating, and correcting AI output. The authors call this supervisory engineering.
If the team accepts twice as much work but senior engineers intervene twice as often, accepted work per human intervention stays the same, so there is no efficiency gain.
A fix is a code change like any other, which means it can introduce its own bugs. On one feature, my agents went back and fixed all of the findings from an earlier review. Mindrealm reviewed those fixes and found three new bugs the fixes had introduced.
One came from a health check that used an HTTP client with no timeout. Mindrealm flagged it, the agent set the timeout to ten seconds, and it wrote a test to prove the fix worked. Ironically, the handler in that new test waited on a channel with no timeout. If the test returned early, shutdown could wait forever and hang CI. Mindrealm caught the second missing timeout on the next review pass and sent the finding back to the agent, which wrapped the channel receive in a select with its own timeout. Problem solved.
I published the full story here.
That is why Mindrealm's stop hook reviews an agent's changes before a pull request is opened, gives concrete findings back, and reviews the changed code again after the agent fixes them. Once a pull request is opened, the Mindrealm GitHub App uses its deterministic rules to review it.
For the same reviewed code, configuration, and rule version, Mindrealm returns the same findings, giving your team repeatable review results while leaving the merge decision with the accountable engineer.
If you use an LLM to review code, Mindrealm gives the reviewing agent deterministic findings to use alongside its own judgment.
Hand a large pull request to an LLM review tool and watch what it actually reports. An LLM's context window is finite, so the tool samples: some hunks, some files, whatever fits. The bigger the change, the more it skims.
Mindrealm reviews diffs by default. Run mind review --full-scan from the CLI and it reviews every source file in the repository against every rule. The stop hook takes the same setting, so an agent's own review can be a full scan too.
On Mindrealm's own codebase, half a million lines of Go, the cold run finished in just under 12 seconds. The next run, with the cache warm, finished in about 7 seconds.
Mindrealm costs a flat $30 per seat each month, with no usage or token fees, so a full scan of the whole repository costs nothing extra. Review a change as many times as it takes, or put a full scan in CI on every push. The bill does not grow.
.mind/rules/, and they run in every review next to the built-in checks. One rule covers all four languages, and it reports the same thing every time.mind setup command installs the stop hook into Claude Code, Codex, Kimi Code, and Antigravity CLI.golangci-lint, go build, go vet, go mod tidy, go mod verify, ruff, vulture, tsc, eslint, cargo check, and cargo clippy out of the box, and merges what they report into the same output as the rule findings.Code generation will keep getting cheaper, faster, and better, but I care about whether the code reaches a state a senior engineer can accept without becoming the glue holding every agent loop together.