Warehousing
When a warehouse is genuinely warranted, and the considerably more common case where it is not.
The warehouse is the most over-prescribed artifact in this field. A company with three systems and a few thousand rows does not need one, and provisioning it anyway buys a recurring bill, a transformation layer that must be maintained, and a second place for the truth to live. The default recommendation here is to answer the question from the source systems until that stops working.
The threshold test is whether answering the questions that matter requires joining data held in systems that cannot join to each other. If every question a business actually asks can be answered inside one platform, a warehouse adds a hop and a bill without adding an answer.
Where one is warranted, the structure is conventional and worth stating anyway: a raw layer holding source data as it arrived, never edited in place, and a transformation layer built on top of it. Editing raw data destroys the ability to reprocess, which is the main thing a warehouse is for.
Transformations are code in version control, reviewed and rerunnable, rather than queries saved in an individual editor. A figure produced by a query that exists on one laptop is a figure that leaves the company when the laptop does, and it cannot be checked by anyone in the meantime.
Keys and grain are decided explicitly at the point where sources meet. Two systems describing the same customer under different identifiers is the ordinary case, and the mapping between them is a modeled decision with a rule, not a join written from memory each time somebody needs it.
History is preserved rather than overwritten. A customer's plan changed in March, and a report covering the year has to know which plan applied in January. Overwriting the current value is the fastest way to make every historical calculation quietly wrong while every current one stays right, which is why it survives review.
Cost is a design input, not a monthly surprise. A platform billed by scanned data rewards partitioning, incremental models and a retention policy decided in advance, and punishes the pattern where every dashboard load rescans the full history because that was easiest to write.
Personal data is minimized on the way in rather than scrubbed on the way out. Fields the reporting does not need are not loaded at all, because a warehouse is the easiest place in any business to accumulate a copy of everything and the hardest place to explain later.
The exit is designed with the entrance. The warehouse holds no state that exists nowhere else and can be rebuilt from its sources on demand, so moving off it is a rebuild, not a rescue, and the decision to keep it stays reversible for as long as the business owns its source systems.
What this does not cover.
- Warehouses provisioned for businesses whose reporting is answerable directly from one or two source systems.
- Accumulating everything a business emits against a use that has not been named.
- Transformation layers that cannot be rebuilt from the source systems they were derived from.
Data & Analytics
Metric definitions
Every figure carries what it counts, what it excludes, where it was read from, and when the definition last changed.
Measurement and attribution
Instrumenting what can be observed directly, and stating plainly where observation stops and inference begins.
Pipelines and synchronization
Jobs designed around their failure modes: safe to repeat, able to resume, loud when they stop.
Reporting surfaces
Built to answer a decision instead of to display everything available, with freshness on the face of it.
Data quality
Validation at the boundary where records enter, because everything after that point is cleanup.
Analytics and privacy
Collecting against named questions, from few third parties, under a policy that matches what the code does.