The worst data bug is not the one that crashes. It is the one that quietly reports the wrong number — and every stakeholder downstream builds decisions on top of it, compounding the error, until someone finally notices that reality and the dashboard no longer agree.
We caught ours after six months. By then, an executive team had presented distorted KPIs to a board. A product team had prioritised the wrong roadmap items based on engagement numbers that were double-counted. An ML model had been trained on inputs that were silently corrupted at the join step. The pipeline had never thrown an error. The CI passed clean. The monitors were green. The data was wrong.
This is the post-mortem I wish I had read before it happened.
What We Built and Why We Trusted It
The pipeline in question aggregated customer interaction events from three source systems — a CRM, an event stream, and a legacy batch export — into a unified activity table that fed reporting dashboards and a churn prediction model. The architecture was reasonable: dbt transformations, Airflow orchestration, BigQuery as the warehouse, and a Metabase layer on top. Nothing exotic.
What we had not built adequately was data contracts between the source systems and the pipeline, and data quality checks between the transformation layers. The pipeline tests verified that the DAG ran. They did not verify that the output was correct.
This distinction — "did it run?" versus "is the output right?" — is the gap that swallows data teams whole.
The Failure: A Schema Drift Nobody Announced
Six months before we caught the problem, one of the source systems — the legacy CRM — quietly changed the meaning of an event type field. The field name stayed the same. The data type stayed the same. But the values changed: what had previously been coded interaction_complete now included a second event fired on retry, meaning every interaction that required a retry was counted twice.
No migration note. No change log entry in the data team's channel. No version bump in the schema registry — because the CRM team did not use one. The ingestion pipeline saw valid records with valid values and ingested them without complaint.
The dbt model that consumed those records had been written to trust the source. It joined, aggregated, and passed the doubled counts downstream. Every downstream model inherited the inflation.
The corruption path through the pipeline looked like this:
Schema drift is not a data engineering problem — it is a governance problem
The root cause here was not a missing dbt test. It was the absence of a formal data contract between the CRM team and the data platform team. No agreed schema, no versioning, no change notification process. The technical fix is straightforward. The organisational fix — getting upstream teams to treat their output as an API with consumers — is the harder and more important work.
How We Caught It (Not How We Should Have)
We did not catch it with monitoring. We caught it because a finance analyst was reconciling quarterly numbers against a separate BI export and noticed the aggregate interaction count in the main dashboard was approximately 1.8x the count in the export. She filed a support ticket assuming the export was wrong.
That is the worst possible way to catch a data quality problem: a human, doing manual reconciliation, six months after the fact, by accident.
When I traced it back, the path was clear within an hour. The doubled event type, the ingestion with no validation, the dbt model with no row-count assertions, the downstream models with no statistical plausibility checks. Every failure mode was predictable in hindsight. None of it was monitored.
The Post-Mortem: Five Layered Failures
A single-cause post-mortem would have been dishonest. The corruption persisted for six months because five independent controls failed simultaneously.
| Layer | What Should Have Caught It | What We Had | Gap |
|---|---|---|---|
| Source contract | Versioned schema with change notification | Informal verbal agreement | No formal contract |
| Ingestion validation | Value distribution checks on key fields | Record count check only | No semantic validation |
| Transformation testing | Row-count and aggregate assertions in dbt | Column type tests only | No business-logic tests |
| Output monitoring | Statistical anomaly detection on KPIs | "Pipeline ran successfully" alert | No output quality check |
| Consumer feedback | Automated reconciliation against known sources | Manual ad-hoc analyst check | No systematic reconciliation |
The lesson is not that any one of these should have been perfect. It is that five independent layers of protection all had the same blind spot: they checked execution, not correctness.
What We Fixed — In Order of Impact
Recovery was unglamorous and took eight weeks. This is what we actually did, in the sequence that mattered.
Before any remediation, map every downstream consumer of the corrupted table: reports, dashboards, ML models, data exports, API queries. Assign a severity to each. The instinct is to fix the pipeline immediately. The right move is to understand what you are about to fix first — because reprocessing changes every downstream metric and you need to communicate that before it happens.
Mark the affected dbt models as deprecated. Replace dashboard queries with a clearly-labelled "under review" state. Stakeholders seeing an empty chart ask questions. Stakeholders seeing a wrong chart make decisions. Empty is better.
Negotiate with the CRM team to provide a corrected event export covering the six-month window. Apply the deduplication logic in a new dbt model with a clear lineage note explaining the correction. Backfill the warehouse partitions. Validate the corrected output against the finance analyst's reconciliation figure — that cross-check became our ground truth.
Not just column type tests. Row-count bounds versus the previous run (flag if greater than 20% movement). Aggregate plausibility checks (total interaction count should not exceed registered user count times a reasonable ceiling). Referential integrity tests between the activity table and the user dimension. These tests would have caught the doubling within days.
Deploy Great Expectations assertions on the output tables, configured to send alerts — not just log to a dashboard nobody checks. Set up a weekly automated reconciliation report that compares the primary KPIs against an independent calculation from raw sources. The reconciliation is the canary.
Work with the CRM and event stream teams to agree on a schema contract: field definitions, allowed values, version control, and a mandatory 2-week change notification period before any field semantics change. Not glamorous. Genuinely the most important fix.
What Good Data Pipeline Monitoring Actually Looks Like
"The pipeline ran" is not a health signal. It is a liveness check. Liveness and correctness are different things, and conflating them is how six-month silent corruptions happen.
Here is the monitoring stack I now insist on for any data platform I oversee:
| Signal Type | What to Monitor | Alert Threshold |
|---|---|---|
| Liveness | DAG completion, task duration | Task fails or exceeds 2x p95 runtime |
| Volume | Row counts per model, partition size | Greater than 20% delta vs. prior 7-day average |
| Distribution | Key field value distributions (z-score) | Significant shift in categorical frequencies |
| Plausibility | Business-logic bounds (e.g., events per user) | Exceeds known human maximum |
| Reconciliation | Primary KPIs vs. independent source calculation | Greater than 2% discrepancy |
| Freshness | Time since last successful load | Exceeds agreed SLA |
The volume, distribution, and plausibility checks are the ones most teams skip. They are also the ones that catch the bugs that everything else misses.
The Harder Conversation: Trust in Data
The technical remediation was the easy part. The harder part was rebuilding stakeholder trust in the numbers.
When you tell an executive team that the engagement metrics they presented to the board for two quarters were approximately 80% inflated, the conversation is not comfortable. We ran it with full transparency: here is what was wrong, here is why, here is how long it was wrong, here is the corrected picture, and here is what we have put in place so it cannot happen again.
What saved us was the corrected data actually telling a more coherent story. The inflated engagement had never quite matched the conversion rates. With the corrected numbers, the funnel made sense. The board presentation was uncomfortable, but it was also credible — because we came with evidence of the problem, the root cause, and the fix, not just an apology.
The credibility test for data platform teams
Your data platform team is only trusted to the extent that stakeholders believe the numbers are right. One silent corruption that goes undetected for six months destroys more credibility than a year of on-time delivery builds. The most valuable investment you can make in a data platform is correctness monitoring — not throughput, not latency, not scale. Get the numbers right first.
The Principle I Carry Forward
Every data pipeline I now design or audit starts from the same assumption: the pipeline will eventually receive corrupt, changed, or semantically incorrect input. The question is not whether that input arrives — it will — but whether the system will surface it in hours or in months.
The difference between hours and months is not the quality of the source systems. It is the density and quality of the assertions you put between the source and the consumer.
Treat every transformation model as if the upstream data is lying. Write tests that would catch the lie. Monitor outputs as if nobody is watching. Build the canary reconciliation that catches what everything else misses.
If you are building or scaling a data platform and want an independent assessment of where your correctness gaps are before a stakeholder finds them, let's talk — a 30-minute discovery call is enough to identify the highest-risk gaps in your current architecture.