Unthought.

Testing

Coverage aimed at what would actually be expensive to get wrong, instead of at a percentage.

A coverage percentage measures how much code a test suite executes, not how much of it the suite would notice being broken. We test against consequence instead of against a number.

The rules that cost money when they are wrong get the most attention: money arithmetic, state transitions, permission boundaries, and anything that decides whether a record is created once or twice. These are the places where a defect is discovered by a customer or an accountant instead of by a developer.

Tests are written against behavior rather than implementation. A test that breaks whenever code is refactored without changing what the system does is a tax on improvement, and teams respond to that tax by refactoring less.

Each test states one thing and its name says what that thing is. A failure should identify the broken behavior from the name alone, without anyone reading the body to work out what was being asserted.

Fixtures represent real data, including the awkward parts. Empty states, a record with a null that has been null for years, a name with an apostrophe, a quantity of zero, a timezone that observes daylight saving. Clean fixtures produce suites that pass reliably and catch nothing.

The boundaries are tested with the outside world stubbed, and the stubs are built from the responses the real service actually returns instead of from the documentation's example.

A bug that reaches production earns a test before it earns a fix, so that the specific failure cannot silently return. This is the cheapest test anyone will ever write, because the reproduction is already known.

A test that fails intermittently is treated as a defect in its own right rather than re-run until it passes. Tolerating one flaky test teaches everyone to ignore a red result, and a suite nobody believes is worse than no suite, because it costs the same to maintain and provides none of the assurance.

The suite runs fast enough to be run constantly. A suite taking twenty minutes is a suite that gets run before merges and not before commits, which removes most of its value as a feedback mechanism.

Exclusions

What this does not cover.

  • Coverage percentage as a target. It measures execution, not detection.
  • Tests asserting implementation details that change whenever code is refactored.
  • Full end-to-end suites for systems small enough that they cost more to maintain than the defects they catch.