A routine migration became a nine-hour outage because of three checks that take an hour to run. That's the whole story. Everything else is detail — and the detail is worth reading if you're planning any migration of a production system in the next twelve months.
I was brought in as an interim CTO for a mid-size insurer with around €2.5B in gross written premium. They had been running a critical policy administration system on-premises for eleven years, and a board-approved cloud migration programme was finally moving it to Azure. The programme had been well-funded, technically competent, and had taken fourteen months to reach production cutover. The night of the migration, the system went down at 02:17 and came back at 11:09. Nine hours and fifty-two minutes of a system that processes new business applications, renewals, and claims notices — gone.
The business impact was real: delayed policy issuance, regulatory notification obligations triggered, and a trust conversation with a major broker that the commercial team was still managing three months later.
What the Post-Mortem Found
The migration itself was not badly planned. The team had documented the cutover runbook, rehearsed the sequence in staging, and run a dress rehearsal ten days before go-live. The architecture decisions were sound. The Azure landing zone was properly designed.
The outage was not caused by a wrong architectural decision. It was caused by three specific pre-flight checks that were not on the runbook — checks so foundational that experienced engineers assumed they must have already been done.
That assumption is how nine hours happen.
Missed Check 1: DNS TTL Not Reduced Before Cutover
The application used a load balancer DNS entry with a 3,600-second TTL — a default that had sat unchanged for eleven years. During the cutover, the team updated the DNS record to point to the new Azure-hosted environment. Old resolvers — including the broker portal's own infrastructure and several internal claims systems — continued resolving the old IP address for up to an hour after the change.
The insurer's own offices hit the new environment cleanly. The broker integrations did not. Underwriting traffic from two major broker portals continued hitting the decommissioned on-premises load balancer, which had been shut down as part of the cutover sequence, and timed out silently. The monitoring didn't catch this because it was synthetic monitoring from inside the network — same network that was already resolving correctly.
The check: Reduce DNS TTLs to 60–120 seconds at least 48 hours before any cutover. Verify the reduced TTL has propagated using external resolvers (not your own corporate DNS). On cutover night, verify from at least three geographically distributed resolver perspectives before decommissioning the old target.
Time to run this check on cutover night: Ten minutes.
Synthetic monitoring from inside your network is not an external health check
If your pre-cutover monitoring uses agents that sit on your own network or cloud tenant, they will not catch DNS propagation failures for external parties. You need monitoring that resolves your endpoints from external vantage points — the same vantage points your users and integration partners are resolving from.
Missed Check 2: Database Connection String Hardcoding in a Legacy Config Layer
The application had been migrated and retested in staging. What had not been fully mapped was a secondary configuration layer — a set of XML configuration files that had been part of the original implementation and had never been brought into the main deployment pipeline. These files contained hardcoded connection strings pointing to the on-premises SQL Server by IP address.
In staging, this had been masked because a routing rule in the test environment forwarded traffic from the old IP to the new Azure SQL instance. That routing rule was specific to the test environment and was not replicated in production. When the production cutover happened and the on-premises SQL Server was brought offline, several background processes — claims status synchronisation, document generation queues, overnight batch jobs — started failing silently because they were still trying to connect to a server that no longer existed.
These failures didn't surface immediately. The queue depth built up over two hours before alerting fired. By then, the investigation was tangled: engineers were looking for infrastructure failures while the real problem was in a config file nobody had thought to audit.
The check: Run a full string search across every configuration file, environment variable, secrets vault entry, and deployment artifact for the old server names, IP addresses, and database hostnames before decommissioning anything. This includes compiled artifacts — decompile and search if necessary. It also includes any CI/CD pipeline configurations, monitoring agent configs, and integration middleware.
Time to run this check: Twenty to forty minutes, depending on the estate size. A script can do most of it.
Configuration archaeology is not glamorous but it is non-negotiable
In systems older than five years, there will be configuration that nobody on the current team wrote and nobody remembers exists. The correct stance is to assume it exists and go looking for it — not to assume the migration team has found everything because the staging tests passed.
Missed Check 3: Network Security Group Rules Not Validated End-to-End
The Azure landing zone had been correctly designed with Network Security Groups (NSGs) applied at the subnet and NIC level. The team had tested application connectivity from Azure to the external integrations. What they had not tested was connectivity from the external integrations to the Azure environment — specifically, inbound rules for the IP ranges used by three external integration partners.
Two of those partners used legacy IP ranges that had not been included in the NSG allowlist because the documentation provided by the partners listed their "current" ranges, which had been updated six months earlier. The legacy ranges were still in active use for some traffic. When the go-live happened, those traffic streams hit the NSG and were silently dropped.
This was a different failure mode from the DNS issue — it wasn't a propagation delay, it was a permanent block. The affected integration partners couldn't reach the new environment at all, and because the failure was a silent TCP drop rather than an explicit error, the partner-side monitoring showed "service degraded" rather than a clear connectivity failure. The investigation took two hours longer than it should have because both sides were looking at application logs rather than network flow logs.
The check: Run a full connectivity matrix test — not just from Azure outbound, but inbound from every external integration partner IP range, using Azure Network Watcher or equivalent tooling. Get the actual active IP ranges from each partner, not just the documentation, because documentation is always behind reality. Do this check in a dedicated test window before the production cutover night.
Time to run this check: Fifteen to twenty minutes per integration partner, in parallel.
The Runbook Problem
The deeper issue is not that these three checks are obscure. They are not. Any experienced cloud architect would recognise all three on sight. The issue is that they were not on the runbook, and they were not on the runbook because the programme assumed they had been handled during the fourteen months of migration work.
That assumption is exactly backwards. The cutover runbook should be built on the assumption that anything not explicitly verified on cutover night has not been verified for production. Not because the migration team is incompetent — they weren't — but because the gap between "tested in staging" and "confirmed for production cutover" is where outages live.
| Check | Failure Mode | Detection Lag | Fix After Cutover |
|---|---|---|---|
| DNS TTL reduction | External parties resolve old IP | 30–90 min | Re-point DNS, wait for TTL again |
| Config string audit | Background processes connect to dead server | 2–4 hrs | Find all occurrences, redeploy, drain queues |
| NSG inbound validation | External traffic silently dropped | 1–3 hrs | Update NSG rules, verify per partner |
| Combined pre-flight time | — | — | ~1 hour before cutover |
One hour of pre-flight checks. Nine hours and fifty-two minutes of outage. That ratio never gets easier to explain to a board.
What We Changed for the Next Migration
The insurer had two additional systems in the migration programme. Before moving either of them, we made three structural changes to how cutovers were managed.
First, the runbook became a pre-flight checklist with explicit sign-off. Every item required a named engineer to confirm it was verified in the production environment within the 24 hours before cutover. Not "confirmed in staging." Confirmed in production.
Second, external perspective monitoring was added before cutover night. We instrumented synthetic checks from three external vantage points — a SaaS monitoring service, a cloud region outside our primary tenant, and a third-party tool that simulated broker portal resolution behaviour. These ran for 72 hours before cutover and continued through the cutover window.
Third, we ran a "destroy assumptions" session the week before each cutover. Two engineers who had not been working on the migration spent four hours deliberately trying to find configuration, dependencies, and routing that the migration team might have missed. They found things. Every time.
The next two system cutovers went cleanly. Combined downtime: fourteen minutes of planned maintenance window.
The 'destroy assumptions' session is the highest-ROI thing you can do before any major cutover
Fresh eyes with explicit permission to be sceptical find things that migration teams miss — not because the migration team is careless, but because domain familiarity creates blind spots. Budget a half-day for this. It is the cheapest insurance available.
Nine hours is long enough for a major broker to start drafting an email to your competitors. The three checks that caused it take an hour to run. The only reason they weren't run is that no one put them on the runbook.
Put them on the runbook.
If you're managing a cloud migration — particularly in a regulated industry where a nine-hour outage triggers regulatory obligations — let's talk. I've led migrations for insurers, reinsurers, and energy companies where the tolerance for unplanned downtime is effectively zero. Book a 30-minute discovery call and we can assess your cutover readiness before your go-live night.