Pipelines and synchronization

Every synchronization has one direction and, per field, one system of record. Two systems that both accept edits to the same value are a conflict-resolution problem wearing an integration's clothes, and that is entered into deliberately with a written rule for who wins, or it is not entered into at all.

Jobs are safe to repeat. Running the same batch twice produces the same result as running it once, because at some point it will run twice — a retry after a timeout that had actually succeeded, an operator rerunning by hand, a schedule firing during a release. Safe repetition is what makes each of those a non-event instead of a duplicate-records incident.

Work is checkpointed. A job that fails partway resumes from where it stopped instead of starting over or, considerably worse, advancing its cursor and silently skipping the interval it never finished. The skipped interval is the expensive version: nothing errors, and the gap is found weeks later by someone wondering why a Tuesday looks low.

Backfill is designed alongside the pipeline rather than attempted after it. The first question of any historical load is how far back the source will actually serve, and the answer frequently truncates — a source that will only return ninety days is a constraint on what the business can ever report, and it is better known at the start than discovered when someone asks for a year-on-year comparison.

Failures are loud. A pipeline that stops without saying so leaves a reporting surface rendering last week's numbers as though they were current, which is the worst available outcome because the surface still looks healthy. Every job records its last successful run, and the surfaces downstream read that timestamp instead of assuming freshness.

Incoming records are validated at the boundary against the shape they are expected to have. A field that changed type, went missing, or started arriving as a string that used to be a number fails the ingest with a message naming it, instead of propagating a null through six downstream calculations that each handle it differently.

Rate limits, reading in batches, and partial responses are handled as ordinary operating conditions instead of as exceptions. Backoff is implemented once and shared, and a job that hits a limit slows down instead of failing the run.

Raw payloads are retained as received, before any transformation. When a transformation later turns out to be wrong — and the first version of a transformation is frequently wrong — the fix is a reprocess against stored data, not a request to an upstream system that no longer serves that window.