Cost and operations

Unit cost is measured during the build, not discovered on an invoice. Cost per operation multiplied by realistic volume, calculated before the architecture is committed to, occasionally returns a number that ends the project — and it is far better to reach that number in week two than in the first month after launch.

Caching is the largest available lever and it is applied deliberately. Identical requests do not need recomputing, stable instructions can be cached by vendors that support it, and a considerable share of production traffic in most deployments is the same handful of questions arriving repeatedly from different people.

Model selection is made per task rather than once per project. Routing every call to the most capable available model is a default nobody actually chose, and a meaningful portion of the work — classification, extraction, routing, short rewrites — runs acceptably on something smaller, cheaper and faster.

Latency has a stated budget. A call taking several seconds inside a path a person is waiting on is a different engineering problem from the same call running in the background, and work that does not need to be synchronous is moved off the request path rather than optimized in place.

Vendor dependency is treated as a dependency and gets the same scrutiny as any other. A business whose core workflow runs through one provider's model carries a concentration risk, and the mitigation is knowing what the alternative would be and what switching would cost, decided in advance instead of during an outage.

Versions are pinned and deprecation dates are tracked. Providers retire models on their own schedule, that schedule is not negotiable, and a deprecation notice arriving with a short window is a planned migration for a business that has been watching and an emergency for one that has not.

Vendor failure is a designed case, not an unhandled one. What the product does when the API is unavailable, rate-limited, or slow enough to be indistinguishable from unavailable is specified — degrading to a defined behavior, queueing, or falling back to the manual path that existed before.

Monitoring covers cost, latency, error rate and refusal rate as one picture. Refusal rate is the one usually missing and frequently the most informative: a system increasingly declining to answer is a product failure that generates no errors, appears in no log as a problem, and shows up first as users quietly giving up.