Search for a command to run...
Weak assertions and mocks that verify wiring, not behavior, can still get merged.
No comments yet. Be the first to comment.
Passing tests and clean code do not prove the agent finished the job or followed the process correctly.
Mindrealm caught the second one too. 19 of the 21 findings in a feature the agents wrote were real.
Claude Code marked the goal achieved and tried to stop. Mindrealm's stop hook wouldn't let it because 17 code review findings were still open.
If a completion gate reads a number the agent can change, the agent can pass it by changing the number, and probably will.
Mindrealm: deterministic review for AI-generated code
6 posts
Field notes from building Mindrealm, an AI code reviewer that catches the slop your agent ships confidently. Posts on why AI agents fail, what it takes to actually trust an agent's output, and my mission to reach zero false positives. For engineers who distrust hype.
Agents write all of the code in my repositories and I decide what gets in, so all of the tests I read are tests I did not write. They arrive with the change, already passing, and there are a lot of them.
Then I noticed this in a test:
require.NotNil(t, invoice)
The test stops there. It never checks the invoice total or line items. Every wrong total satisfies that line just as easily as the right one.
The test result is real. The assertion held. But Google's SRE book is clear about the limit: "Passing a test or a series of tests doesn't necessarily prove reliability. However, tests that are failing generally prove the absence of reliability."
That word "generally" matters. A failed test can come from a flaky machine, a defective test, or an expectation that wasn't updated. A passing assertion tells you exactly what it checked. It does not tell you whether that check was strong enough for the requirement.
The test runner answers two questions: did the code run, and did it satisfy the assertions? If the assertion accepts anything non-null, the test only tells you the invoice wasn't null. It does not check the total, currency, or line items.
Ask an agent to add a test and it adds one with assertions in it, and the test suite passes.
None of these patterns are new. Weak assertions, mock abuse, and empty test bodies were in testing books decades before agents. Two things changed. The same agent writes the implementation and the test that is supposed to catch its mistakes. The test can copy the implementation's mistake into its expected value, or use an assertion that passes without checking the required behavior. The agent also writes tests faster than review can keep up, so you read a smaller share of them.
Every language gives you an assertion that only checks whether a value exists:
invoice, err := billing.Invoice(ctx, order.ID)
require.NoError(t, err)
require.NotNil(t, invoice)
Here is what still passes: a zero total, the wrong currency, no line items, or an invoice built from stale order data. The test only catches null values. Every other result on that list still passes.
A harder one to spot looks like this:
const invoice = await buildInvoice(order);
expect(invoice).toEqual({
total: invoice.total,
currency: invoice.currency,
lines: invoice.lines,
});
The expected value is read from the result. A test that expects whatever the result turns out to be is a mirror, not a check. No wrong total can make it fail. An agent writes this when it does not know what the answer should be. The expected value has to come from the requirement, not the result.
mailer = Mock()
ledger = Mock()
service = Billing(mailer, ledger)
service.finalize(order)
mailer.send.assert_called_once()
ledger.record.assert_called_once()
The test proves that send and record were each called once. It does not check the recipient, invoice, or ledger entry.
The test still passes if the service sends the draft invoice instead of the finalized one. It still passes if the service sends the invoice to the wrong address. It still passes if the invoice total is zero. The test verifies that the objects are wired together, but the requirement is about the invoice sent to the customer.
Mocking is not the problem. The mail provider and ledger are exactly the dependencies a unit test should mock. Nobody wants a unit test hitting a live SMTP server. The defect is the assertion. assert_called_once stops at the interaction. It never reads the recipient, the invoice, or the ledger entry.
func TestFinalizeInvoice(t *testing.T) {
order := seedOrder(t, 3)
svc := billing.New(t)
svc.Finalize(context.Background(), order.ID)
}
The test creates an order and calls Finalize. Every executed line counts toward coverage, but the test does not assert anything. If Finalize writes the wrong total, this passes. If it writes no invoice at all, this passes. Work can keep running after the test ends, and this test would still pass.
This one is easy to spot when you are looking for it and nearly invisible in a diff of forty files, because it reads as a finished test.
Five questions, in order, on every test that matters.
Question three is the hard part. The suite already passes. You have to look for what the assertions missed.
Run it on the invoice. The test has to prove that finalized orders return the expected invoice and draft orders return an error.
invoice, err := billing.Invoice(ctx, order.ID)
require.NoError(t, err)
require.Equal(t, int64(4250), invoice.TotalCents)
require.Equal(t, "USD", invoice.Currency)
require.Len(t, invoice.Lines, 2)
_, err = billing.Invoice(ctx, draft.ID)
require.ErrorIs(t, err, billing.ErrOrderNotFinalized)
The last two lines are the ones I argue for hardest. Nothing in "add a test for this" asks for the case where the software has to refuse. NIST's developer verification guidelines put that case in the same list as the specification: black box test cases come from "functional specifications or requirements, negative tests (invalid inputs and testing what the software should not do)... input boundary analysis, and input combinations."
If an assertion cannot tell the required behavior from a plausible wrong result, the test is not finished.
Mutation testing automates the same exercise. Stryker describes it plainly: "Mutation testing introduces changes to your code, then runs your unit tests against the changed code." It changes the implementation, runs the test suite, and reports the changes the tests failed to catch. Gremlins does it for Go, mutmut for Python, Stryker for TypeScript, and cargo-mutants for Rust.
Use mutation testing where you can. It will try more wrong changes than you will by hand. It reruns tests for every mutant, so large mutation sets take time and compute.
A surviving mutant shows that the suite did not catch that code change. Mutation testing can find weak tests, but it cannot tell you what the expected result should be. That has to come from the requirement.
Four of its checks read test files directly. One fires on a test function with no assertion or verification in it at all, and one on an assertion that accepts almost any value. A third fires when a test's assertions are about the mock rather than the behavior under test, and a fourth when a test is skipped with no stated reason. They run on Go, Python, TypeScript, and Rust, and the same code produces the same findings every run.
Mindrealm can tell you that require.NotNil accepts too much. It cannot tell you the total should have been 4,250 when that requirement is missing from the repository. Reading source and executing code answer different questions, and the same NIST guidelines count both as verification: "Verification includes methods such as static analysis and code review, in addition to dynamic analysis or running programs (“testing” in a narrower sense)." Neither one supplies a missing contract.
The check catches the pattern. You still have to know the number.
That is why "does it pass" is not the question. It passed before you opened the file, and it will pass after you approve it.
Ask what wrong result would still pass it, and see how long your list gets.