Scheduled work
Timezones pinned, clock changes handled deliberately, overlap prevented, and missed runs given a defined behavior.
A job set for two in the morning runs at two in the morning in some timezone, and which one it is has broken more nightly processes than any defect in the work itself. Twice a year the clocks move, and that hour either occurs twice or never occurs at all.
The timezone is pinned explicitly at the process level and committed to configuration, not inherited from whatever the host happens to default to, which is almost always UTC. A report labeled Monday that actually covers Sunday evening in the business's own timezone is wrong in a way that reads as correct for years, because nobody checks the boundary of a number they already believe.
Clock changes are handled, not discovered. A job scheduled inside the hour that daylight saving skips does not run that day; a job inside the hour that repeats runs twice. Both are avoided by scheduling outside those windows, and the residual risk is covered by making the job idempotent so that a repeated run is harmless anyway.
Overlap is prevented explicitly. A run taking longer than its own interval will eventually meet the next one, and two instances of the same job working the same records produce duplicates or deadlocks depending on the storage underneath. A lock held for the duration of the run, with a timeout that releases it if the process dies, is the whole fix.
A missed run has a defined behavior chosen in advance: catch up on what was skipped, or abandon it and process the current period only. Both are legitimate answers for different jobs — a billing export should catch up, a cache warm should not — and the wrong one gets selected by not selecting.
Jobs that process a period name that period explicitly with a start and an end instead of computing it relative to the moment they happen to run. Relative arithmetic evaluated at an unpredictable time is how a run delayed past midnight processes the wrong day and reports success.
Long jobs are checkpointed so they resume rather than restart, because a job that must complete in one attempt will eventually meet a deploy, a restart, or a timeout, and starting over is the response that turns a slow job into one that never finishes at all.
Schedules live in committed configuration, not only in a dashboard. A schedule set through a console is unversioned, unreviewable, and absent when the service is recreated, and the recreation is exactly the moment nobody remembers what the schedule was.
Each job records both its expected cadence and a maximum expected duration, so that a run which hangs is distinguishable from a run that is merely slow. Without the second number, a job stuck indefinitely and a job having a bad afternoon look identical from outside.
What this does not cover.
- Schedules configured only in a hosting dashboard, with nothing in the repository recording them.
- Jobs whose period is computed relative to execution time rather than named explicitly.
- Long-running jobs with no lock, where a slow run can overlap the next one.
Automation & Integration
Silent failure
Instrumenting automations so that not running is a detectable state, and proving the alarm reaches a person by breaking things deliberately.
When work runs twice
Everything in ordinary use delivers at least once, so whether a second delivery is harmless is decided before anything is written.
Event-driven work
A public address carrying somebody else's reliability: verify it, record it, acknowledge fast, do the work afterwards.
Messaging, consent and opt-out
The area where a technically flawless implementation can still be unlawful, so the requirement is part of the specification.
Working with other companies' systems
Built against what another company's system actually does, not what its documentation says, with the differences written down.
The automation register
Six fields per automation, kept with the code, so that what runs is knowable and switching something off is possible.