# A test can pass without proving the required behavior

Agents write all of the code in my repositories and I decide what gets in, so all of the tests I read are tests I did not write. They arrive with the change, already passing, and there are a lot of them.

Then I noticed this in a test:

```go
require.NotNil(t, invoice)
```

The test stops there. It never checks the invoice total or line items. **Every wrong total satisfies that line just as easily as the right one.**

## What did the assertion actually check?

The test result is real. The assertion held. But Google's [SRE book](https://sre.google/sre-book/testing-reliability/) is clear about the limit: "Passing a test or a series of tests doesn't necessarily prove reliability. However, tests that are failing generally prove the absence of reliability."

That word "generally" matters. A failed test can come from a flaky machine, a defective test, or an expectation that wasn't updated. A passing assertion tells you exactly what it checked. It does not tell you whether that check was strong enough for the requirement.

The test runner answers two questions: did the code run, and did it satisfy the assertions? If the assertion accepts anything non-null, the test only tells you the invoice wasn't null. It does not check the total, currency, or line items.

Ask an agent to add a test and it adds one with assertions in it, and the test suite passes.

None of these patterns are new. Weak assertions, mock abuse, and empty test bodies were in testing books decades before agents. Two things changed. The same agent writes the implementation and the test that is supposed to catch its mistakes. The test can copy the implementation's mistake into its expected value, or use an assertion that passes without checking the required behavior. The agent also writes tests faster than review can keep up, so you read a smaller share of them.

## An existence check accepts every wrong answer

Every language gives you an assertion that only checks whether a value exists:

```go
invoice, err := billing.Invoice(ctx, order.ID)
require.NoError(t, err)
require.NotNil(t, invoice)
```

Here is what still passes: a zero total, the wrong currency, no line items, or an invoice built from stale order data. The test only catches null values. Every other result on that list still passes.

A harder one to spot looks like this:

```ts
const invoice = await buildInvoice(order);
expect(invoice).toEqual({
  total: invoice.total,
  currency: invoice.currency,
  lines: invoice.lines,
});
```

The expected value is read from the result. **A test that expects whatever the result turns out to be is a mirror, not a check.** No wrong total can make it fail. An agent writes this when it does not know what the answer should be. The expected value has to come from the requirement, not the result.

## A mock can prove the call and miss the expected outcome

```python
mailer = Mock()
ledger = Mock()
service = Billing(mailer, ledger)
service.finalize(order)

mailer.send.assert_called_once()
ledger.record.assert_called_once()
```

The test proves that `send` and `record` were each called once. It does not check the recipient, invoice, or ledger entry.

The test still passes if the service sends the draft invoice instead of the finalized one. It still passes if the service sends the invoice to the wrong address. It still passes if the invoice total is zero. The test verifies that the objects are wired together, but the requirement is about the invoice sent to the customer.

Mocking is not the problem. The mail provider and ledger are exactly the dependencies a unit test should mock. Nobody wants a unit test hitting a live SMTP server. The defect is the assertion. `assert_called_once` stops at the interaction. It never reads the recipient, the invoice, or the ledger entry.

## A test can execute everything and verify nothing

```go
func TestFinalizeInvoice(t *testing.T) {
    order := seedOrder(t, 3)
    svc := billing.New(t)

    svc.Finalize(context.Background(), order.ID)
}
```

The test creates an order and calls `Finalize`. Every executed line counts toward coverage, but the test does not assert anything. If `Finalize` writes the wrong total, this passes. If it writes no invoice at all, this passes. Work can keep running after the test ends, and this test would still pass.

This one is easy to spot when you are looking for it and nearly invisible in a diff of forty files, because it reads as a finished test.

## Read the assertion literally, then try to break it

Five questions, in order, on every test that matters.

1. **State the required behavior in one sentence.** If you cannot, the test is not the problem yet.
2. **Name the observable outcome that would prove it.** A returned value, a stored row, an emitted event, an error.
3. **Read the assertion literally and ask what wrong result would still pass.** Write those results down. That list shows what the test allows.
4. **Check what the assertions about mocks actually prove.** Isolating a dependency is good. If every assertion stops at a received call, the expected behavior may still be unverified.
5. **Add the smallest negative or boundary case.** Make sure it fails when the behavior is wrong.

**Question three is the hard part.** The suite already passes. You have to look for what the assertions missed.

Run it on the invoice. The test has to prove that finalized orders return the expected invoice and draft orders return an error.

```go
invoice, err := billing.Invoice(ctx, order.ID)
require.NoError(t, err)
require.Equal(t, int64(4250), invoice.TotalCents)
require.Equal(t, "USD", invoice.Currency)
require.Len(t, invoice.Lines, 2)

_, err = billing.Invoice(ctx, draft.ID)
require.ErrorIs(t, err, billing.ErrOrderNotFinalized)
```

The last two lines are the ones I argue for hardest. Nothing in "add a test for this" asks for the case where the software has to refuse. NIST's [developer verification guidelines](https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8397.pdf) put that case in the same list as the specification: black box test cases come from "functional specifications or requirements, negative tests (invalid inputs and testing what the software should not do)... input boundary analysis, and input combinations."

**If an assertion cannot tell the required behavior from a plausible wrong result, the test is not finished.**

## Mutation testing finds what your tests miss

Mutation testing automates the same exercise. [Stryker](https://stryker-mutator.io/docs/) describes it plainly: "Mutation testing introduces changes to your code, then runs your unit tests against the changed code." It changes the implementation, runs the test suite, and reports the changes the tests failed to catch. Gremlins does it for Go, mutmut for Python, Stryker for TypeScript, and cargo-mutants for Rust.

Use mutation testing where you can. It will try more wrong changes than you will by hand. It reruns tests for every mutant, so large mutation sets take time and compute.

A surviving mutant shows that the suite did not catch that code change. Mutation testing can find weak tests, but it cannot tell you what the expected result should be. That has to come from the requirement.

## I built Mindrealm for the part a rule can decide

Four of its checks read test files directly. One fires on a test function with no assertion or verification in it at all, and one on an assertion that accepts almost any value. A third fires when a test's assertions are about the mock rather than the behavior under test, and a fourth when a test is skipped with no stated reason. They run on Go, Python, TypeScript, and Rust, and the same code produces the same findings every run.

Mindrealm can tell you that `require.NotNil` accepts too much. It cannot tell you the total should have been 4,250 when that requirement is missing from the repository. Reading source and executing code answer different questions, and the same NIST guidelines count both as verification: "Verification includes methods such as static analysis and code review, in addition to dynamic analysis or running programs (“testing” in a narrower sense)." Neither one supplies a missing contract.

**The check catches the pattern. You still have to know the number.**

## You already know it passes

That is why "does it pass" is not the question. It passed before you opened the file, and it will pass after you approve it.

Ask what wrong result would still pass it, and see how long your list gets.