Working with other companies' systems

Observed behavior is written down as it is discovered, in a file that ships with the code. Undocumented nulls, fields that change type, the call that reports success while doing nothing, the header that is required despite being listed as optional — all of it is knowledge acquired at real cost, and none of it survives in anyone's memory. Without the file, the next person pays for it again.

Rate limits are handled with backoff through a shared client rather than per call site, so that the limit is respected in aggregate. Ten places each politely retrying will together exceed a limit none of them individually approaches, and the resulting behavior is intermittent in a way that is genuinely difficult to trace.

Reading through a large, changing set of records uses whatever method the vendor provides for exactly that. Reading by position instead silently skips records whenever anything is inserted mid-read, and the omission is undetectable from the response — the pages all arrive, none of them error, and the total is quietly short.

Differences between sandbox and production are recorded and the consequential ones are tested deliberately against production before launch, in a controlled way. Sandboxes routinely emit fewer event types, enforce weaker validation, and accept payloads that production rejects.

Interface versions are pinned, and retirement notices are routed to a person instead of to an address nobody monitors. Vendors retire versions on their own schedule and communicate through channels that are easy to miss, which is how an integration that has run untouched for a year fails on a date that was announced months earlier.

Credentials are scoped to the narrowest permission that accomplishes the task, held as configuration instead of in code, and rotated on a schedule. The rotation is exercised at least once: a credential nobody has ever rotated is a credential nobody knows how to rotate, and that is discovered during the incident that requires it.

Every outbound call carries a timeout. A call without one will eventually hang against an unresponsive service and take a worker with it, and enough of those take the process — which presents as an unexplained outage in a system whose own code is faultless.

Tests run against the vendor's real system on a schedule, not only against a stand-in. A mock asserts what was believed at the time it was written; only a real call notices that a response shape changed, which is the failure mode that a comprehensive mocked test suite is guaranteed to miss.