Plenty of impressive SLAs are backed by manual heroics nobody wants to admit to. I have seen 99.99% uptime commitments maintained by an on-call engineer who woke up at 3am, ran a cron job by hand, and went back to bed — for eighteen consecutive months, without ever documenting it.
The SLA was technically met. The system was operationally fragile. And nobody on the leadership team knew.
This is not a story about incompetent engineers. The engineer who kept that system alive was genuinely talented and deeply committed. It is a story about how reliability theatre gets institutionalised, why it is so hard to see from the outside, and what the real cost is when the human carrying the system eventually leaves or burns out.
The Discovery
I was brought in as Interim CTO to stabilise the platform before a Series B close. The company had a strong commercial story, solid revenue, and an SLA track record that looked excellent on paper. Investors had flagged technology as a risk — the CTO had departed abruptly and the platform was a black box — but the uptime numbers were reassuring.
My first week was standard reconnaissance: architecture review, production access, incident history, monitoring stack, on-call runbooks. The monitoring looked reasonable. The incident history showed five minor incidents in eighteen months, all resolved within the SLA breach threshold. Impressive.
Then I sat down with the senior platform engineer who had been the primary on-call respondent for those eighteen months. I asked him to walk me through how the last incident was resolved.
He pulled up a spreadsheet.
The Spreadsheet
It was a Google Sheet with 47 rows. Each row was a recurring failure pattern: the service that needed to be restarted, the exact SSH command to run, the expected output that confirmed recovery, the downstream services to check afterwards. Some rows had a "time of day" column — because certain failures only occurred after specific batch jobs, which ran at 02:30 UTC.
He had built it over the course of eighteen months because the system had no runbooks, no automated recovery, and no alerting that actually told you what to do — only that something was wrong. He had reverse-engineered each failure pattern the first time it appeared, documented it in the sheet, and then executed from the sheet every subsequent time. Reliably. Accurately. In the middle of the night.
The system's 99.99% uptime record was entirely a function of his personal knowledge, his spreadsheet, and his willingness to be woken up.
He was resigning in four weeks.
The single-point-of-human-failure
Every engineering organisation has bus factor risks in its codebase. Far fewer actively measure bus factor in their operational processes. A system that can only stay within SLA because one specific person is awake and available is not a reliable system. It is a reliable person propping up an unreliable system. The distinction matters enormously when that person leaves.
Why This Happens
This kind of reliability theatre is more common than most engineering leaders want to believe. It does not emerge from laziness or deception — it emerges from a very human set of incentives that compound over time.
The SLA is being met, so the alarm never sounds. Leadership sees green dashboards and a clean SLA track record. There is no visible signal that the mechanism behind the green numbers is fragile. No alert fires. No post-mortem is triggered. The problem is invisible precisely because the heroic intervention works.
The engineer who knows it is broken has no time to fix it. The same person who carries the operational burden is the person who would need to fix the underlying fragility. But fixing it requires uninterrupted engineering time. Uninterrupted engineering time does not exist when you are the person holding the system together manually. The trap is self-reinforcing.
Runbooks feel like an admission of fragility. There is a perverse incentive in some engineering cultures where writing a runbook for a manual process makes the manual process visible and therefore criticisable. So it stays in a personal spreadsheet, invisible to everyone except the person who built it.
Leadership mistakes output for architecture. A clean SLA report is not evidence that the system is reliable. It is evidence that the system has been reliable so far, under the conditions that have existed so far. Those are very different things.
What the True Risk Exposure Was
When I modelled the actual risk, the picture was stark. The client had contractual penalties capped at 15% of monthly contract value per SLA breach, across 23 enterprise contracts. The total penalty exposure for a sustained outage — the kind that would have occurred if the engineer had been unavailable during a failure window — was approximately €400,000 in direct penalties plus the reputational cost of SLA failures during a funding round.
The cost of remediation — proper runbook documentation, automated recovery for the five primary failure patterns, monitoring with actionable alerts, and a second on-call engineer trained to the same level — was estimated at eight weeks of engineering time and approximately €15,000 in tooling.
| Scenario | Probability | Cost Estimate |
|---|---|---|
| Engineer available, SLA met (baseline) | Status quo | €0 additional cost |
| Engineer unavailable, 2-hour outage, penalties across 5 contracts | Moderate if on holiday/sick | €80,000–150,000 |
| Engineer leaves, 2-week knowledge gap, cascading failures | High (notice given) | €250,000–400,000 |
| Remediation: automate recovery, document runbooks, train backup | Certain | €15,000–25,000 |
The remediation cost was less than 6% of the low-end catastrophe estimate. This is the economics of reliability investment: the expected value of prevention is almost always orders of magnitude better than the expected value of reaction.
The Remediation: Eight Weeks to a Real SLA
We ran the remediation in two phases alongside the engineer's notice period, using his knowledge as the primary input. The work progressed like this:
Phase 1 (Weeks 1–4): Knowledge extraction and runbook formalisation.
Every row in the spreadsheet became a structured runbook entry: failure description, detection signal, root cause hypothesis, recovery procedure, validation steps, escalation path. We filmed him walking through each procedure so the institutional knowledge was captured in a format that could be searched and referenced at 3am by someone who had never seen it before.
We identified which failures were recoverable via automation. Three of the five primary patterns were straightforward: a service restart after a memory leak hit a threshold, a database connection pool reset after a batch job overload, and a cache flush when a specific error rate spiked. All three were automatable with existing tooling. We built the automations immediately.
Phase 2 (Weeks 3–8): Monitoring with teeth and second-responder training.
The existing monitoring detected failures but gave no guidance on response. We rewrote the alerting to include a direct link to the relevant runbook in every alert. On-call engineers no longer needed to find the right runbook under pressure — the alert told them where to go.
We ran two other senior engineers through every runbook, with tabletop exercises simulating each failure pattern. By the end of week six, we had three people who could respond to any of the primary failure patterns independently. The single point of human failure was gone.
The three tests for genuine reliability
Before I accept that a system is reliably operating within its SLA, I run three tests. First: if the primary on-call engineer were completely unavailable for 48 hours starting now, would the system still meet its SLA? Second: are the runbooks written by someone other than the engineer who originally created them (proving they transfer knowledge, not just encode it)? Third: has the system successfully self-recovered from at least two of its primary failure patterns without human intervention in the last 90 days? If any of these tests fails, the SLA is backed by heroics, not architecture.
The Governance Question
The harder question, once the immediate remediation was under control, was how this had been allowed to develop for eighteen months without surfacing in any leadership conversation.
Part of the answer was the incentive structure. The engineer was rated highly — correctly — for the reliability he delivered. Nobody looked inside the mechanism that delivered it. There was no engineering leadership layer between the platform team and the C-suite that would have normalised operational review conversations.
The other part was the absence of any reliability engineering culture. There were no error budgets. No SLOs behind the SLA — just the contractual SLA itself, which was either met or not. No blameless post-mortems, because there had been no acknowledged incidents. No regular review of on-call burden or escalation patterns.
An engineering organisation without these practices cannot see its own reliability debt accumulating. It can only see it after the system breaks in a way that the manual heroics cannot contain.
What This Costs in the Long Run
Beyond the specific incident risk, reliability theatre has compounding costs that are easy to underestimate:
| Cost Type | Visibility | Typical Magnitude |
|---|---|---|
| Burnout and attrition | Low — emerges slowly | Loss of 1–2 senior engineers per year |
| Technical debt acceleration | Very low — masked by heroics | 20–40% of platform capacity diverted from product |
| Investor due diligence risk | Moderate — surfaces in diligence | Can block or delay funding rounds |
| SLA penalty exposure | Low until the incident occurs | Can be existential at enterprise contract scale |
| Insurance and compliance | Low — emerges in audits | Non-compliance with availability commitments in regulated sectors |
At the companies I work with in regulated industries — reinsurance, energy, financial services — the compliance dimension alone is enough to make reliability theatre a boardroom-level risk. Operational resilience requirements in financial services are explicit: you cannot rely on undocumented manual processes to meet availability commitments.
The Principle
An SLA you can only meet because one person is awake is not a reliability commitment. It is a dependency on a human who has better options.
The right question to ask your engineering leadership team is not "are we meeting our SLAs?" It is "what happens to our SLAs if our three best on-call engineers are simultaneously unavailable for 72 hours?" If the answer is uncertain, you have reliability debt, regardless of what the dashboard shows.
Real reliability is boring. It is automated recovery, documented runbooks tested by people who did not write them, monitoring that tells you what to do rather than just that something is wrong, and error budgets that surface degradation before it becomes breach. It is the opposite of heroic. It is also the only kind that scales.
If you are scaling a platform under contractual SLA commitments and want to understand whether your reliability is architectural or heroic, let's talk — a 30-minute discovery call is enough to surface where the real risk lives.