Silent failure

The governing rule is that every automation is instrumented so that *not running* is a detectable state. A job reporting success when it succeeds is trivial to build and nearly worthless on its own, because the case that costs money is the one where nothing is reported at all. What has to be watched for is the expected event that never arrived.

The mechanism is a check-in: each scheduled job reports completion to a watcher, and the watcher raises an alarm when a check-in is overdue by more than the job's stated tolerance. The watcher runs somewhere the job does not. A monitor sharing infrastructure with the thing it monitors goes quiet at precisely the moment its silence means the most, and a monitoring system that fails in the same breath as the system it watches has told nobody anything.

Alerting on absence requires knowing what presence looks like, which means the expected cadence is written into configuration, not held in somebody's memory. This job runs hourly. This one runs on weekday mornings. This one fires between thirty and sixty times a day, and zero is a defect even though zero is not an error. None of that is inferable from the code, and none of it survives the departure of the person who knew it.

Volume anomalies deserve the same treatment as outright absence. A synchronization that ordinarily moves several hundred records and today moved four has not failed by any measure available to the code: it ran, it completed, it reported success. Bounds are therefore placed around the ordinary operating range, not only around zero, because the partial failure is both more common than the total one and considerably harder to notice.

Errors are routed somewhere a person actually reads. This sounds too obvious to state and is the most frequently skipped step in this work, because the default destination for an error is a log file, and a log file is a place errors go to be retained rather than seen. The destination is decided during the build, and it is a channel with a human on the other end.

That human is named. An alert delivered to a shared mailbox that belongs to everyone belongs to nobody, and it produces the same outcome as no alert while creating the impression of coverage. Accountability for each automation is recorded with the automation itself.

Alert volume is managed as carefully as alert coverage, because a channel that produces noise gets muted, and a muted channel is strictly worse than an absent one — it is a gap in monitoring that everybody believes is covered. Alerts are reserved for conditions somebody will act on. Everything else belongs on a surface that can be consulted, not pushed at people until they stop looking.

Where an automation touches customer communication, its failures are made visible to the business and not only to whoever built it. The person who will discover that appointment reminders stopped is the one fielding the calls about missed appointments, and giving them a way to see the automation's state converts a week of confusion into a single question.

An alert arrives with an indication of what to do about it. A notification saying only that a job failed obliges whoever receives it to reconstruct, at whatever hour it arrived, what the job does and where to look — and an alert that is expensive to act on gets deferred, which over a few months is indistinguishable from an alert that was never configured. Each one carries the automation's name, what it was doing, and where its record lives.

The most awkward failure here is the automation that succeeds while being wrong. A synchronization runs, reports success, and writes records in which a field the source stopped populating three weeks ago is now empty everywhere downstream. Nothing errored, nothing was absent, and every monitor described above stays quiet. Catching it requires asserting something about the content, not the run — a required field is present, a total falls within range — which is the same discipline the data work applies at its boundaries, arriving here through a different door.

Retry policy is explicit and bounded, and the terminal state is designed rather than defaulted. What becomes of a record that has exhausted its retries is a decision: it goes to a queue somebody works, with the reason attached, rather than into a log line that closes the matter as far as the system is concerned while leaving it entirely open for the customer.

Partial failure is the ordinary case and the one most often left unhandled. A job processing four hundred records and failing on nine has neither succeeded nor failed in any sense the surrounding code usually models, so the outcome is recorded per record and the nine are addressable individually. The alternative — treating the run as failed and re-running it whole — is the reason idempotency stops being optional.

The test that matters is not whether the automation works, which is easy to demonstrate and rarely in doubt. It is whether somebody would find out inside an acceptable window if it stopped. That window is stated per automation and it varies enormously: an hour for appointment reminders, a day for a nightly export, a week for something advisory that nobody acts on immediately.

All of it is verified by breaking things on purpose. In a non-production environment the job is disabled, the credential is revoked, the endpoint is pointed somewhere unreachable, and the question asked is whether an alarm actually reached a person. An untested alarm is a belief about a system, not a property of it, and this is the step that gets skipped — which means it is the step that decides whether everything above was real or merely written down.