Modern Data Platform Design: Principles Over Products

Separate storage, compute and semantics

Open table formats like Delta Lake let several engines read the same data. Keeping storage independent of compute means you can change or add engines without migrating data, and a semantic layer on top keeps business definitions independent of both.

Treat data as a product

Each gold dataset should have an owner, a documented contract, quality metrics and a support path. Consumers should be able to discover it in a catalog, understand what it means and trust that it will not silently change shape.

• Named owner and steward
• Schema contract and versioning
• Published freshness and quality metrics

Automate everything that repeats

Infrastructure, pipelines, schemas and permissions all belong in source control and are deployed through CI/CD. Manual changes in production are the most common source of drift between environments — and of incidents nobody can explain.