On-call is where good engineers quietly decide to leave. Not in a dramatic exit conversation — in a 3am alert on a Saturday that nobody acknowledges on Monday. In a rotation that's four people wide when it should be eight. In a postmortem that produces no action items. In the accumulating, undeniable evidence that leadership has decided their nights and weekends are an acceptable operating cost.
I've fixed on-call rotations at three companies in the past two years. Each time, the same pattern: the team's best engineers — the ones with options — were within one or two bad weekends of updating their CV. The fix was not expensive. The inaction was.
Why On-Call Breaks Good Teams
On-call is not inherently a problem. Shared responsibility for production reliability is a reasonable expectation for engineers who build systems that customers depend on. The problem is almost never the concept — it's the implementation.
The implementations I inherit fall into two failure modes.
Failure mode one: the heroics culture. Two or three senior engineers carry the rotation, explicitly or implicitly. Everyone knows who will actually fix things at 2am. The rotation exists on paper but the load doesn't distribute in practice. The three people who carry it feel indispensable, then irreplaceable, then trapped, then burned out. They leave. The remaining team is left with a production system that now has no one who understands it at midnight.
Failure mode two: the alert noise spiral. The alerting system was set up during a crunch. Every possible failure condition was wired to page someone. Over time, false positives accumulate. Engineers learn that half the alerts can be ignored. They start ignoring them — including, eventually, the ones that matter. Meanwhile, the paging frequency degrades sleep, relationships, and trust in the on-call function itself. The system that was supposed to catch problems becomes a reliable generator of anxiety.
Both failure modes have the same root cause: on-call was never designed. It grew.
The Rotation I Built That Actually Worked
The rotation I've implemented across multiple teams has five components. None of them are novel. The novelty is in implementing all five simultaneously and maintaining them with the same rigour as the production system itself. The five components reinforce each other — removing any one weakens the whole:
Component 1: A Rotation Wide Enough to Be Sustainable
The minimum viable on-call rotation for a team running a production system 24/7 is six to eight engineers. Below six, individuals are on-call too frequently for the rotation to be psychologically sustainable. The math is simple: if you're on-call every third weekend, on-call is not a rotation — it's a permanent part of your life.
At one engagement — a platform team at a Series B insurtech company — the rotation had four engineers covering a 12-service production environment. Each engineer was primary on-call roughly every two weeks. Two of the four had started interviewing elsewhere. We expanded the rotation to seven engineers over six weeks by cross-training three backend engineers on the production systems they were already building. The primary on-call frequency dropped to approximately every six weeks. The two engineers who were interviewing stopped.
| Rotation Size | Primary On-Call Frequency | Psychological Sustainability |
|---|---|---|
| 3 engineers | Every 1–2 weeks | Not sustainable; burnout within 6 months |
| 5 engineers | Every 4–5 weeks | Marginal; tolerable but not comfortable |
| 7–8 engineers | Every 6–8 weeks | Sustainable; comparable to routine work travel |
| 10+ engineers | Every 10–12 weeks | Comfortable; low impact on personal life |
Component 2: An Alert Actionability Standard
Every alert in the system should pass two tests. First: does this alert require a human to take action within 30 minutes, or can it wait until business hours? If the answer is "it can wait," it should not page anyone at 2am. Second: when this alert fires, is there a documented runbook step that the on-call engineer can execute? If not, the alert is a noise generator, not an operational tool.
The alert audit process I run takes about two weeks. I pull 90 days of alert history. For every alert that fired more than twice in that window, I apply both tests. On average, 40 to 60 percent of alerts that currently page engineers outside business hours fail one or both tests. Removing or recategorising them cuts overnight alert volume by three to four times without reducing incident detection meaningfully — because the alerts that fail the tests were not catching real incidents in the first place.
The false safety of comprehensive alerting
The instinct to alert on everything comes from a genuine place: nobody wants to miss a production problem. But alert fatigue produces exactly the failure mode it was designed to prevent. When engineers stop trusting that an alert means something is actually wrong, they respond slower — to everything, including the real incidents. A smaller set of high-confidence, actionable alerts is safer than a comprehensive set of noisy ones.
Component 3: A Real Secondary Rotation
Primary on-call alone is a flawed structure. When the primary gets a complex incident at midnight, they need to be able to escalate to a human — not to a Slack channel where someone might see it in the morning. The secondary rotation provides that safety net. It also provides a learning path: secondary on-call is an excellent way to cross-train engineers on production systems, because they're involved in incidents without carrying the full first-responder weight.
The secondary rotation can be the same engineers as the primary, offset by a week. In a seven-person rotation, engineer A is primary in week one and secondary in week four. This keeps the structure simple without eliminating the safety net.
Component 4: A Postmortem Culture That Produces Action Items
A postmortem that produces no action items is not a postmortem — it's a debrief. The distinction matters. A debrief processes what happened. A postmortem prevents it from happening again. If your postmortems consistently conclude with "the incident was resolved" and nothing on the board changes, you are processing incidents rather than eliminating them.
The postmortem template I use has three mandatory sections beyond the timeline: contributing factors (not root cause — systems rarely have single root causes), owner-assigned action items with due dates, and an alert review question: "Did this incident surface a gap in our alerting, and if so, what change are we making?"
The action item completion rate is as important as the template. I track it in every team I work with. If more than 20 percent of postmortem action items are not closed within 30 days, something is wrong — either with prioritisation or with the team's bandwidth to act on reliability improvements while also shipping product.
Component 5: Explicit Compensation for On-Call Burden
Being on-call outside business hours is not the same as being at work during business hours. Treating them equivalently in how engineers are managed and compensated sends a message — and it is not a message that retains engineers.
The approaches that work vary by company stage and geography. At minimum, I advocate for:
- Time-in-lieu policy for weekends or nights where the engineer was meaningfully interrupted. Not an informal "take Monday morning off." A documented policy.
- On-call supplement for engineers in the rotation — not necessarily large, but explicit. The signal matters as much as the amount.
- On-call acknowledgement in performance conversations. If an engineer had a bad on-call month — high alert volume, two significant incidents — that context should be in the manager's head when they're reviewing output and having 1:1s.
The invisible cost of treating on-call as normal work
When on-call burden is treated as a standard part of the job with no additional recognition or compensation, you create a specific type of resentment: the kind that is quiet, rational, and hard to fix. Engineers don't complain loudly — they calculate. They look at their total experience, including nights and weekends, and they compare it to what they'd experience elsewhere. You will not get a warning. You will get a resignation letter.
The 90-Day Fix
The rotation redesign I run follows a consistent sequence. It's not complicated, but it requires leadership commitment to deprioritise a few feature sprints in favour of reliability infrastructure — which is the actual blocker in most cases.
| Week | Action | Owner |
|---|---|---|
| 1–2 | Alert audit: 90-day history, apply actionability tests, categorise | Engineering lead + on-call rotation |
| 2–4 | Remove / recategorise failing alerts; update runbooks for keepers | On-call rotation |
| 3–5 | Cross-training: pair junior/backend engineers with current on-call | Senior engineers + manager |
| 4–6 | Expand rotation to target size; establish secondary rotation structure | Manager + HR |
| 6–8 | Run first postmortem with new template; track action items | Engineering lead |
| 8–12 | First retrospective on the new rotation; measure alert frequency pre/post | Team |
By week 12, every team I've run this with has had measurably lower alert volume, a rotation that's wide enough to sustain, and engineers who are no longer describing on-call as a reason to leave. That's not because the production systems became more reliable overnight — it's because the rotation became a system that the team could trust.
What Leadership Gets Wrong About On-Call
The most common senior leadership mistake I see is treating on-call as an engineering team problem rather than an organisational design problem. "The engineers need to fix the alerting" is a cop-out that puts the burden of fixing a structural problem on the people most burdened by it. Engineers on a heavy rotation do not have the bandwidth to audit, cross-train, and redesign the rotation while also carrying primary on-call and shipping product.
The rotation redesign requires leadership to make explicit decisions: temporarily reducing feature velocity to invest in reliability, funding the cross-training time, and approving the compensation changes. None of these decisions can be made at the engineering team level. They require a leader who understands that on-call attrition is more expensive than the investment required to prevent it.
The calculation is not subtle. Replacing one mid-senior platform engineer in Berlin costs — conservatively — €35,000 to €50,000 in recruiter fees, severance, onboarding time, and productivity loss. The cost of running the 90-day rotation redesign, including the cross-training time and modest on-call supplements, is typically €15,000 to €25,000 in diverted engineering capacity and direct cost. The ROI exists before you count the harder-to-quantify costs: the knowledge that walked out the door, the team morale impact, the signal it sent to your remaining engineers.
Good engineers leave good companies over operational dysfunction. On-call is often the proximate cause, but the underlying cause is always the same: leadership that could see the problem and chose not to solve it.
That's the choice. The engineering is the easy part.
If your engineering team is carrying a reliability burden that's becoming a retention risk, let's talk — a 30-minute discovery call is enough to understand whether the problem is structural and what it would take to fix it.