Most data platforms have layers nobody can explain. Ask the data engineer who built the bronze-silver-gold medallion what "silver" actually means for your business, and you'll get a different answer every time. Ask the architect why there are three transformation tools running in parallel and watch them reach for the org chart as an explanation.
This is not a technology problem. It is a clarity problem. And it costs organisations real money — in engineering time, in tooling licenses, in the latency between data being collected and decisions being made.
I've stood up and inherited data platforms at reinsurers, energy companies, and enterprise software businesses. Here is what I actually believe each layer is for, and which ones are being sold to you under false pretences.
What a Data Platform Actually Has to Do
Before talking about layers, let's be clear on the job. A data platform has to do four things:
- Get data from where it is produced to where it can be used
- Represent that data accurately enough to trust it
- Make it accessible to the people and systems that need it
- Do all of this reliably and cheaply enough to justify its existence
Every layer in your platform should be traceable to one of those four jobs. If it isn't, it's overhead. Keep that test in mind as I walk through the standard layers — because several of them fail it.
The end-to-end flow through a well-scoped data platform looks like this:
The Ingestion Layer: Get the Data In
This is the only non-negotiable layer. Something has to move data from production systems, third-party APIs, event streams, and operational databases into the platform. The ingestion layer is that thing.
What it should do: extract data reliably, handle schema drift without breaking downstream pipelines, maintain at-least-once (ideally exactly-once) delivery semantics, and provide observable lineage from source to destination.
What most teams get wrong: they treat ingestion as solved by picking a tool (Fivetran, Airbyte, Kafka, or a custom connector framework) and never revisit it. In practice, the ingestion layer requires ongoing maintenance because sources change. A connector that worked for your Salesforce schema in 2023 will silently drop fields added in 2024 unless someone owns the monitoring.
Tools I've seen work well at scale: Fivetran or Airbyte for SaaS sources, Debezium for CDC from operational databases, Kafka or Kinesis when you need streaming semantics. Avoid building custom connectors unless you have sources no vendor covers — the maintenance burden is permanent.
The Raw Storage Layer: Preserve What You Got
Land the data exactly as it arrived. No transformations. No interpretations. Just fidelity.
This layer is your insurance policy. When a transformation introduces a bug six months from now, or when a source system changes its semantics without telling you, raw storage is what lets you replay from known-good data. It is not an input to your analytics queries. It is a recovery mechanism.
The cost of raw storage is low enough — object storage at €0.02–0.05/GB/month depending on tier — that there is almost never a financial argument for skipping it. The argument for skipping it is usually "we trust the ingestion layer to be lossless," which is the kind of trust that costs you a weekend when it turns out to be wrong.
The one rule for raw storage
Raw storage is append-only and never transformed in place. If anyone on your team proposes "cleaning up" the raw layer, that is a signal that they do not understand what the raw layer is for. Cleaning happens downstream. Raw is the record of truth as received.
The Transformation Layer: Make It Usable
This is where most of the complexity lives — and most of the unnecessary complexity is created. The transformation layer takes raw or lightly-staged data and produces the entities your business actually reasons about: customers, policies, transactions, events, positions.
The medallion architecture — bronze (raw), silver (cleaned), gold (business-ready) — is a reasonable pattern when used with discipline. The failure mode is when "silver" becomes a dumping ground for transformations that never made it to gold, producing a layer that is neither raw nor business-ready and that nobody trusts.
What a Transformation Layer Actually Needs
- A single transformation tool with a defined ownership model (dbt is the default right answer for SQL-based transformation today; change my mind with specifics, not aesthetics)
- Documented lineage from source field to output field
- Tests on every output that a downstream consumer depends on
- A policy on what gets built: transformations should exist because someone has asked for the output, not because the data engineer thought it would be useful
What to Skip
Staging-for-the-sake-of-staging. If your data is clean enough to go from raw to gold in one transformation step, make it one step. The extra layer is not discipline — it's comfort blanket architecture.
| Pattern | When It's Right | When It's Overhead |
|---|---|---|
| Bronze → Silver → Gold | Complex sources needing multi-stage cleaning; regulatory audit trails | Simple SaaS data with well-structured schemas |
| Streaming transformation | Real-time decisioning, fraud detection, IoT anomaly detection | BI dashboards that refresh hourly |
| Feature store | ML models consuming shared, versioned features | Analytics-only platforms with no ML inference |
| Semantic layer | Multiple BI tools needing consistent metric definitions | One tool, one team |
The Serving Layer: Get It to Consumers
The transformation layer produces data. The serving layer makes it accessible. These are different jobs, and conflating them is the source of significant architectural confusion.
Serving means: query interfaces, APIs, access control, and performance optimisation for the consumption pattern of each consumer type.
An analyst running ad-hoc SQL needs a query engine over a columnar format (Snowflake, BigQuery, Databricks SQL, or DuckDB depending on your scale). A microservice doing product personalisation needs low-latency key-value lookups, not a columnar data warehouse. A machine learning training pipeline needs batch-readable parquet files. A regulatory report needs a reproducible, audited query that produces the same result on demand.
These are not the same consumption pattern. Serving them all from the same tool produces a system that is mediocre at everything and excellent at nothing.
The data warehouse is not a serving layer for everything
I see organisations try to serve ML inference, product APIs, and analyst queries from the same Snowflake or BigQuery instance. The cost per query for ML feature lookups in a warehouse is unjustifiable. The latency for product API calls from a warehouse is unacceptable. Build separate serving surfaces for genuinely different consumption patterns. Shared storage underneath; specialised serving on top.
The Orchestration Layer: Make It Run on Schedule
Something has to coordinate when pipelines run, handle dependencies between jobs, retry on failure, and alert when things go wrong. That is orchestration.
Airflow is the incumbent. Prefect and Dagster are the credible modern alternatives. The choice matters less than having exactly one of them and using it consistently. The anti-pattern I encounter most often is bespoke cron jobs for legacy pipelines, Airflow for the "important" pipelines, and a data engineer's personal Prefect instance for the new stuff. You end up with three orchestrators, no single view of pipeline health, and three different alerting conventions.
Orchestration is infrastructure. It should be boring, centrally managed, and have a clear on-call owner.
The Observability Layer: Know When It's Broken
Data observability is still underinvested at most organisations I work with. This layer answers: is my data fresh, complete, and within expected distributions?
It is different from infrastructure observability (is the pipeline running?) and different from data quality testing in the transformation layer (does the output match my assertions?). Observability is the runtime monitoring of data health across the platform, surfacing anomalies before downstream consumers notice them.
Tools like Monte Carlo, Soda, or dbt's built-in tests cover different parts of this. The key is defining a small number of critical quality monitors — freshness, row count drift, null rate on key fields, referential integrity — and ensuring someone is accountable for acting on them.
Most platforms I audit have zero data observability beyond "did the pipeline succeed or fail." That is not enough. A pipeline that succeeds but produces wrong data is worse than a pipeline that fails loudly.
The Metadata and Governance Layer: Know What You Have
This is the layer most organisations build too late and then scramble to retrofit. Metadata — data about your data — includes lineage, ownership, definitions, classification (PII, confidential, public), and access policy.
Without this layer, your platform is a collection of tables that nobody fully understands. Engineers leave. Knowledge walks out with them. A new analyst joins and spends two weeks figuring out which of the three "customer" tables is the canonical one.
Data catalogues (Datahub, Atlan, Collibra at enterprise scale) provide the tooling. But tooling without process produces a catalogue that is outdated within six months. The process requirement is a data stewardship model: named owners for each domain, a definition of what "owned" means, and a lightweight governance review when new datasets are onboarded.
Governance is not compliance theatre
I have seen organisations build governance frameworks that exist to satisfy an auditor and are ignored by every working data engineer. That is not governance — it is documentation written in the wrong incentive structure. Real governance means engineers can find what they need, trust what they find, and know who to call when something is wrong. If your catalogue is not updated within a week of a schema change, it is decoration.
The Layer You Were Probably Sold That You Don't Need
The data lakehouse with all features enabled is the platform decision I see most often misapplied. Delta Lake, Iceberg, or Hudi with full schema evolution, time-travel, merge-on-read, and ACID transactions is powerful technology. It is also significantly more complex to operate than a simple Parquet-on-object-storage model.
If your use case is: batch ingestion, SQL analytics, and nightly ML training jobs — you do not need a lakehouse format. You need well-partitioned Parquet and a query engine. The lakehouse format earns its complexity when you have streaming upserts from CDC pipelines, regulatory requirements for time-travel queries, or concurrent read-write workloads that would produce corruption without ACID guarantees.
Adopting it because it is the modern choice is a category of expensive decision that will cost you 3–6 months of engineering time managing format-specific bugs, compaction jobs, and performance tuning that a simpler architecture would never have required.
A Platform That's Fit for Purpose
The right data platform for your organisation is the smallest set of layers that reliably delivers data that people trust, at a cost that the value justifies.
| Layer | Always Needed | Often Skipped Correctly |
|---|---|---|
| Ingestion | Yes | — |
| Raw storage | Yes | Very rarely |
| Transformation | Yes | — |
| Orchestration | Yes | — |
| Serving (query) | Yes | — |
| Serving (low-latency API) | Only with ML/product use cases | Frequently |
| Observability | Yes | Often (wrongly) |
| Metadata/governance | Yes, eventually | Often built too late |
| Feature store | ML inference only | Most analytics-focused orgs |
| Semantic layer | Multiple tools, large analyst teams | Small orgs, single BI tool |
| Streaming pipeline | Real-time requirements only | Most batch-oriented businesses |
The architecture is not the goal. The data reaching the decision is the goal. Every layer that sits between those two things should justify its existence in operational terms, not aspirational ones.
If you're building or inheriting a data platform and want an honest assessment of what's pulling its weight versus what's adding complexity without value, let's talk. A 30-minute discovery call is enough to identify where the architecture is working against you.