When I started using a structured scorecard across every interim and fractional engagement, I expected variation. What I didn't expect was how little correlation there was between organisational confidence and actual score. The teams that believed they were performing well were often not in the top quartile. And the teams that flagged themselves as "struggling" were sometimes further ahead than they realised.
Forty organisations. Same scorecard. The distribution was not what anyone predicted.
What the Scorecard Actually Measures
The scorecard has eight dimensions, each scored 0–15, giving a maximum of 120. I normalise to 100 for easier communication with boards and founders. The dimensions are:
| Dimension | What I'm Testing | Max Points |
|---|---|---|
| Delivery reliability | DORA metrics: deployment frequency, lead time, change failure rate, MTTR | 15 |
| Architecture coherence | Clear ownership, documented decisions, testable modules | 15 |
| Security posture | Secrets management, dependency audit, incident history | 15 |
| Developer experience | Onboarding time, local dev setup, CI feedback loop speed | 15 |
| Observability | Structured logging, tracing, alerting coverage and quality | 15 |
| Technical debt visibility | Estimated carry cost, prioritised backlog, remediation velocity | 15 |
| Hiring and retention | Time-to-hire, offer acceptance rate, 12-month attrition | 10 |
| Alignment to business | Roadmap connects to revenue outcomes, CTO-board communication quality | 10 |
I deliberately weight delivery reliability and architecture coherence highest. You can have exceptional observability and still be a team that ships nothing. The score should reflect operational reality, not tooling ambition.
The Distribution Nobody Expected
I expected a rough bell curve with a slight right skew — some weaker orgs, most in the middle, a few genuinely strong ones. That's not what I found.
The actual distribution is bimodal. There's a cluster between 45 and 60 — functional teams with visible practices but brittle execution. And there's a smaller cluster above 75 — teams with genuine operational maturity. The space between 60 and 75 is thin. That band is where teams are actively improving, usually because something painful forced them to.
Below 45, the patterns are almost uniform: no deployment metrics, architecture maintained in the head of one or two senior engineers, security treated as a compliance event rather than an operational discipline, and onboarding that takes longer than a month to produce any output.
The dangerous middle
The 45–60 band is where I spend the most time. These teams are functional enough that nobody is sounding the alarm, but fragile enough that one key departure, one compliance audit, or one bad quarter can expose the structural weaknesses underneath. The danger isn't catastrophic failure — it's slow degradation that's invisible until it's expensive.
What the Top Quartile Actually Does Differently
The orgs above 75 don't have more engineers. They don't have dramatically better tooling. The consistent differences are operational, not technical.
They measure delivery, not activity
Every high-scoring org tracks deployment frequency and lead time from commit to production. Not sprint velocity. Not story points. Actual time from code complete to users receiving the change. The teams that measure activity — tickets closed, PRs merged, hours logged — systematically overestimate their own performance.
They treat architecture as a living document
The high-scoring orgs have Architecture Decision Records (ADRs). Not as a compliance artifact. As a working tool that engineers actually reference when making new decisions. The quality of an ADR library tells you more about engineering culture than any engineering culture survey.
They have a defined incident response process
Not a complex one. Just a defined one. A rotation. Named first responders. A documented post-incident review template. The low-scoring orgs tend to have informal heroics: the same two people fix every production problem, the incident knowledge lives in Slack threads, the review is a debrief over coffee that produces no written output.
Developer experience is owned
In the top-quartile orgs, someone is specifically responsible for the internal developer experience. That person — whether it's a platform team lead, a VP Engineering with explicit DX mandate, or a rotating engineering committee — has a roadmap for improving onboarding time, CI/CD speed, and local development ergonomics. In the bottom half, developer experience is everyone's problem and therefore no one's problem.
The Surprises That Recalibrated My Priors
Company size is almost irrelevant. I had assumed that larger orgs — with more process, more resources, more hiring budget — would cluster higher. They don't. The highest-scoring organisation I've assessed has 28 engineers. The lowest-scoring has over 200. Beyond about 20–30 engineers, additional headcount without operational maturity becomes a liability, not an asset.
Technology stack is not a predictor. I've seen high-scoring orgs on 10-year-old monoliths and low-scoring orgs on freshly migrated microservices architectures. The stack doesn't score itself. The practices around the stack do.
Founder/CEO involvement is double-edged. Orgs where the CEO came from an engineering background tend to score higher on architecture coherence and lower on business alignment. They've often optimised the technical system and under-invested in the communication of that system to non-technical stakeholders. The inverse is also true: CEOs with pure commercial backgrounds tend to produce orgs that score well on business alignment and poorly on engineering fundamentals.
The self-assessment gap
I ask teams to self-score before I run my own assessment. The average self-score is 67/100. The average actual score is 54/100. That 13-point gap is consistent across org size, industry, and founding story. The teams that most underestimate their score are the ones actively trying to improve. The teams that most overestimate it are the ones that haven't looked closely in a while.
Where Orgs Lose Points They Don't Expect To
Observability is the most overrated dimension. Teams invest significantly in observability tooling — Datadog, Grafana, OpenTelemetry pipelines — and then score poorly on alerting quality. Having dashboards is not observability. Having alerts that page the right people for the right reasons at the right threshold, with runbooks attached, is observability. Most orgs have the former and are building toward the latter.
Technical debt visibility is the most underinvested. Almost no org below 65 has a documented estimate of what their technical debt costs them in engineering capacity per quarter. They know it exists. They experience it. They cannot tell you what it costs. That inability to quantify is also an inability to prioritise — and therefore an inability to remediate.
Hiring metrics are almost universally unmeasured. Time-to-fill, offer acceptance rate, source quality — the vast majority of orgs below the top quartile track none of these. They experience hiring as a continuous difficulty without ever diagnosing where in the funnel the problem lives.
How I Use the Score
The score is not a judgement. I've seen healthy organisations at 58 and troubled organisations at 72. The score is a conversation starter. Here's how I actually use it:
| Score Band | What It Usually Means | Where I Focus First |
|---|---|---|
| Below 40 | Structural risk; likely one incident away from a crisis | Incident response, key-person dependency, deployment pipeline |
| 40–55 | Functional but fragile; limited visibility into real risk | Observability, tech debt quantification, delivery metrics |
| 55–70 | Established practices, execution gaps | Architecture governance, developer experience, hiring process |
| 70–80 | Strong foundations; optimisation opportunity | Alignment to business strategy, senior talent development |
| Above 80 | Genuine engineering maturity | Scaling the culture as the org grows |
The most valuable output of the assessment is not the overall number — it's the dimension-by-dimension breakdown that shows leaders where they're strong and where they're exposed. I've had CTOs tell me the assessment was the first time they had an external frame of reference for what "good" actually looks like in their context.
What This Means for You
If your organisation has never been externally assessed on a structured framework, you almost certainly have a self-score significantly above your actual score. That's not a character flaw — it's a natural consequence of judging your own work by your own standards. The 13-point gap is not specific to bad teams. It's an epistemic problem.
The fix is external calibration. Not to produce a score, but to surface the specific gaps that the internal perspective can't see — because everyone inside has adapted to the way things work, including the things that don't.
If you're a CTO, engineering VP, or founder who wants to understand where your engineering organisation actually sits — not where you think it sits — let's talk. I run structured engineering assessments as part of a broader transformation engagement. Book a 30-minute discovery call and we can scope what's appropriate for your stage and context.