One question separates CTOs who have actually shipped from those who have only talked about it.
Most CTO interviews are structurally broken. Candidates are asked about their philosophy on agile, their views on microservices, their approach to technical debt. They are given hypothetical scenarios — "how would you handle a production outage?" — and rewarded for coherent narratives. The interview evaluates how well someone talks about technology leadership. It does not evaluate whether they have actually done it.
I have interviewed, hired, and — on several occasions — replaced CTOs. I have also sat in the chair as an interim CTO brought in after a failed hire. What I see consistently is that the gap between a polished interview performance and the actual capability to lead an engineering organisation is enormous. And most interview processes don't close that gap.
Here is the question that does.
The Question
"Tell me about the last system you shipped that broke in production. What broke, why, and what did you personally do in the first 90 minutes?"
That's it. Everything that follows — the follow-up questions, the silence, the specific details — is signal.
Why this question and not a technical assessment?
Technical assessments measure knowledge. This question measures experience, judgment, and honesty simultaneously. A candidate who has never led a production system to the point of failure has not shipped at production scale. A candidate who cannot be specific about their personal role reveals either that they weren't close enough to the work, or that they're comfortable inflating their involvement. Either is diagnostic.
What a Real Answer Sounds Like
I will give you a composite of the answers I've heard from CTOs who turned out to be genuinely excellent.
They start specific. Not "we had a database issue" — "our primary Postgres replica failed to promote when the primary went down, and we discovered we'd never actually tested the failover." Not "we had a traffic problem" — "we pushed a feature on Friday afternoon that ran an N+1 query against the user session table and took down the API for eleven minutes at 6pm."
They know the timeline. "I was paged at 14:32. By 14:40 I had pulled in the on-call engineer and the backend lead." "I was at a client meeting. I saw the alert, stepped out, and was on a call within eight minutes."
They name their specific actions. Not "we worked to resolve it" — "I made the call to roll back the deploy while the team continued to diagnose, because I was not confident we could fix forward fast enough with customers still hitting errors." "I called the database vendor directly — we had a premium support contract — because our team had never seen this class of failure before."
They distinguish their role from the team's. "I was on comms the entire time, handling the status page and talking to three large customers personally, so the engineering team could focus entirely on the fix."
And then they tell you what changed afterward. "We added a mandatory failover test to our DR runbook. We also started blocking Friday afternoon deploys unless the on-call engineer explicitly approved." "I introduced blameless post-mortems after this. Up until that point we'd been operating without them."
That is what operational experience sounds like. It is specific, personal, and includes the aftermath.
What a Fraud Sounds Like
The patterns on the other side are equally consistent.
The team-as-shield. "We had a major incident where the team had to work through the night." Every pronoun is "we" or "they" — never "I". When you ask what they personally did during those 90 minutes, the answer becomes vague. "I was coordinating with the team." Coordinating how? Doing what specifically? The vagueness is the tell.
The abstraction escape. "We've had several production issues over the years. I generally approach incidents by..." What follows is a description of an incident management process, not an account of a specific event. When you press for a specific example — "give me one incident, one system" — the candidate either pivots to process again or produces an anecdote that feels assembled from parts rather than remembered.
The role inflation. "I was leading the incident response." When you ask who else was involved, it turns out there were 20 engineers in the war room. When you ask what specific decision they made, the answer involves something approved by committee. A CTO in an incident command role makes decisions — routing, escalation, communication, roll-back versus fix-forward. If they can't name a decision they personally owned, they weren't commanding.
The no-incident candidate. "We've been very fortunate — we've had very few production incidents." This is almost never true at the scale a CTO should be operating at. It usually means one of three things: they were not close enough to production to know about incidents; the team handled incidents without involving them; or they are being selective about what they share. None of these is a good sign.
The Follow-Up Questions That Reveal the Depth
| Follow-up | What it reveals |
|---|---|
| "What was your biggest mistake in handling that incident?" | Self-awareness and intellectual honesty |
| "How did the team feel about it afterward?" | Whether they have a read on morale and culture |
| "What would you do differently if it happened again?" | Whether they updated their mental model |
| "How did you communicate to customers or stakeholders?" | Executive presence under pressure |
| "Was the post-mortem blameless? What came out of it?" | Their approach to learning vs. assigning fault |
| "Did the engineer who caused the problem stay on the team?" | Their actual values around psychological safety |
The last one is particularly telling. A CTO who fired or demoted the engineer responsible for a production incident — unless there was gross negligence — has a blame culture. That culture guarantees that future incidents will be hidden longer, escalated later, and resolved more slowly.
The conference speaker trap
Some candidates sound exceptional because they have given talks about this exact type of incident. They have told the story at QCon or KubeCon or a meetup. The talk was polished. The narrative is tight. The lesson was well-packaged. But there is a difference between a story that has been rehearsed to be told and a memory of something you lived through. Press on specifics the talk would not have covered: the name of the on-call engineer, what the rollback took in practice, what the draft of the customer communication looked like. Real experiences survive that pressure. Rehearsed ones don't.
What This Question Is Actually Testing
The incident question is not a test of crisis management skill, though that is part of it. It is a multi-dimensional filter.
Proximity to production. A CTO who has shipped real systems at real scale has been woken up by alerts. They have made roll-back calls they were not sure were right. If they have not experienced this, they have not operated at the level the role requires. I have seen CTOs of large companies so removed from production that they genuinely could not answer this question. That distance is itself the problem.
Honest accounting of their role. Real leaders are clear about what they did versus what the team did. The candidate who says "I made the call to roll back" and then says "the team rebuilt the system to prevent this — I can't take credit for that, they owned it" is showing you something about character, not just capability.
Learning orientation. A CTO who has been through a production failure and came out the other side with nothing changed — no new process, no updated runbook, no cultural shift toward psychological safety — is not a learning organisation. They are a repeat-incident organisation.
Composure under scrutiny. The follow-up questions will feel like pressure if the original answer was not grounded in experience. Watch how the candidate handles that pressure. The response to the follow-up is often more informative than the original answer.
Why This Question Works Better Than Technical Assessments
Most technical CTO assessments focus on architectural knowledge: system design, scalability patterns, security frameworks. These matter. But they test a different thing. A candidate can pass a system design whiteboard exercise and still be unable to lead an engineering team through a crisis. They can name every distributed systems pattern in the textbook and still be absent when production is burning.
The incident question tests something architectural assessments cannot: whether the candidate has actually been there.
There is also a secondary effect. The candidate who gives a rich, specific, honest answer to this question is showing you their relationship with failure. And a CTO's relationship with failure — their own, their team's, the company's — is one of the strongest predictors of how they will build culture. Leaders who treat failure as data build psychologically safe teams. Leaders who treat failure as liability build silence.
Use this question at every stage of the process
This question is strong enough to use twice — once at first interview and once at the final stage with a different incident. The ability to produce multiple concrete examples, each with different context and different lessons, is itself signal. Anyone can rehearse one incident. The candidate with genuine operational depth has a repertoire.
The Scoring Standard
Here is how I score answers:
| Response quality | Verdict |
|---|---|
| Specific system, specific timeline, named their personal actions, articulated the learning | Strong pass — this is a candidate with real operational history |
| Reasonable specificity but roles blurred with the team; learning was vague | Conditional — probe harder with follow-ups before deciding |
| Process-level answer, no specific incident, team-as-shield throughout | Fail — the answer reveals absence of proximity to real operations |
| "We've been very fortunate" — no significant incidents offered | Immediate flag — either naive, detached, or not being honest |
One final note. The best CTOs I have interviewed did not wait for the incident question. They surfaced their failures unprompted — somewhere in the first 20 minutes of conversation — because they have learned more from what went wrong than from what went right. If a candidate only offers successes until you ask them about failure, you have learned something about how they will behave when things go sideways in the role.
Hire the person who tells you about the incident that cost the company money and then explains precisely what they changed because of it. That person has earned the right to lead.
If you are hiring a CTO — or trying to assess whether your current technology leadership is right for the next phase of growth — let's talk. I offer a 30-minute discovery call and can tell you plainly what I think, including whether you need me at all.