The real enterprise AI platform looks nothing like the vendor slide. There is no single intelligent system orchestrating everything. There is no seamless pipeline from raw data to business outcome. What there is — at every large insurer, reinsurer, and energy company I have worked with — is a collection of deliberate, largely unglamorous engineering decisions that make the flashy models sit on top of something that won't collapse on a Tuesday afternoon.
This post is the boring diagram. It is what I actually draw on the whiteboard in the first week of an engagement when someone says, "We want to build an AI platform."
Why the Keynote Diagram Is Wrong
Vendor keynotes show a smooth left-to-right flow: data goes in, intelligence comes out, value is created. Three things are always missing from that diagram.
First, the governance layer. Every model prediction that informs a pricing decision, a claims assessment, or an energy dispatch recommendation in a regulated environment needs an audit trail. Who approved this model version? What data was it trained on? What was the confidence threshold? How was drift detected and handled? None of this is intellectually interesting. All of it is non-negotiable.
Second, the feedback loop. Models do not stay correct. A pricing model trained on 2022 data degrades silently on 2025 portfolio composition. The platform needs to detect this, surface it, and either trigger a retraining job or alert a human to make a decision. The feedback loop is usually the last thing built and the first thing that causes a production incident.
Third, the organisational seams. A Fortune-500 AI platform spans data engineering, ML engineering, DevOps or platform engineering, the business domain team, and compliance. Every seam between those groups is a place where things go wrong, handoffs are unclear, and accountability evaporates. The platform design has to encode the organisational model, not just the technical one.
The Actual Layers
Every production AI platform I have seen — at reinsurers running hundreds of pricing models, at energy companies optimising grid dispatch, at insurers automating underwriting workflows — has the same four layers. The tooling varies. The structure does not.
The platform structure looks like this:
Layer 1: Data
This is where 70% of the actual work lives and 70% of the AI budget is spent — not on GPUs, but on pipelines.
The data layer has to solve: unified access to training data across multiple source systems (which, in enterprise environments, are almost always heterogeneous, partially documented, and owned by different teams); data versioning so a model can be reproduced from the exact dataset snapshot it was trained on; feature computation that is consistent between training time and serving time (the single most common source of model degradation I have seen); data quality monitoring with defined SLAs; and lineage tracking that satisfies both the ML team's debugging needs and the compliance team's audit needs.
The tooling here at European enterprises tends to be Databricks or Azure Synapse for compute, Delta Lake or Apache Iceberg for versioned storage, Great Expectations or Soda for data quality, and something custom or Feast for the feature store. The feature store is almost always the last thing prioritised and the first thing teams wish they had built earlier.
Layer 2: Experimentation
Experimentation is where data scientists spend their time. The platform has to make that time reproducible and governable without making it slow.
The minimum viable experimentation layer is: experiment tracking (MLflow is the default; Weights and Biases where teams are more ML-native), a model registry with promotion gates, compute management that can provision GPU instances without a three-day approval process, and version control that ties a model artefact to a specific code commit, data version, and hyperparameter set.
What kills experimentation layer adoption is when it feels like bureaucracy. The design goal is that a data scientist can run an experiment, log it, compare it to the last ten runs, and promote a candidate to staging — all without leaving their notebook or filing a ticket. Anything that requires human intervention for routine steps gets abandoned within three months.
The model registry is the platform's contract with the business
Every model in production should have a named owner, a business purpose, a performance threshold below which it triggers a review, and a documented approval chain for version changes. In regulated industries, this is also what the auditor looks at first. If you build the registry as a technical artefact rather than a governance artefact, you will rebuild it later.
Layer 3: Serving
This is where the platform meets revenue — or, in regulated industries, where it meets the regulator.
Enterprise AI serving is almost never just an inference API. It is an inference API plus: traffic management between model versions (canary deployments and A/B routing), SLA guarantees per model endpoint (a pricing model called during a live underwriting session has very different latency requirements than a batch propensity model run overnight), request logging for audit purposes, authentication and authorisation so that only the right services can call each model, and cost tracking at the per-inference level.
The failure mode I see most often at this layer is teams building the serving infrastructure for the first model and then discovering it does not generalise when they add the second. Standardise the serving contract — request format, response format, error codes, health endpoint, metrics endpoint — before you build the first endpoint, not after.
| Serving Pattern | Latency Target | Throughput | Typical Use Case |
|---|---|---|---|
| Real-time online inference | Less than 200ms | 10–10,000 RPS | Pricing, fraud scoring, UW support |
| Near-real-time streaming | Less than 5s | 1,000–100,000 events/s | Risk monitoring, anomaly detection |
| Micro-batch | Less than 60s | Variable | Enrichment, secondary scoring |
| Overnight batch | Hours window | Millions of records | Portfolio analytics, next-day reports |
Most enterprises need at least two of these patterns. The serving infrastructure should support all of them without requiring separate platforms.
Layer 4: Monitoring
Models degrade. The question is whether you find out before or after the business does.
Model monitoring has three distinct concerns that are often conflated. Infrastructure monitoring (is the endpoint healthy, is latency within SLA, is error rate elevated) is table stakes and uses the same tooling as the rest of the platform. Model performance monitoring (is prediction accuracy tracking against baseline, is the business metric the model was optimised for trending correctly) requires ground truth labels, which means a feedback pipeline from downstream systems back into the platform. Data drift monitoring (has the distribution of input features shifted enough to invalidate the model's assumptions) requires baseline statistics computed at training time and a comparison job running in production.
The tooling I reach for here is Evidently AI or Arize for ML-specific monitoring, integrated into whatever observability stack the platform team is already running — Datadog, Grafana, or Azure Monitor depending on the organisation.
Monitoring without retraining triggers is just alerts nobody acts on
The monitoring layer is only valuable if it feeds into a defined response process. That means: documented thresholds per model, a named person responsible for each model's health, a defined escalation path when thresholds are breached, and an agreed process for retraining or rollback. Without the process, the monitoring is noise.
The Governance Overlay
In regulated industries — which describes every client I have worked with in insurance, reinsurance, and energy — governance is not a layer you add on top. It has to be threaded through all four layers from the start.
| Governance Requirement | Where it lives in the platform |
|---|---|
| Model change approval | Model registry — promotion gates with sign-off |
| Training data lineage | Data layer — versioned datasets with source documentation |
| Prediction audit log | Serving layer — immutable log of every inference with inputs and outputs |
| Drift incident records | Monitoring layer — logged alerts with response actions taken |
| Model performance reporting | Monitoring layer — regular reports to risk and compliance functions |
| Human override capability | Serving layer — mechanism to override model outputs with logged rationale |
At Fortune-500 insurer-scale operations, this governance overlay is what separates a toy ML system from a platform that can be submitted to an internal audit or a regulatory review without significant remediation work.
What the Org Chart Has to Look Like
The platform is only as good as the team running it. The anti-pattern I see most often is a pure data science team trying to own and operate what is fundamentally a distributed systems engineering challenge. Data scientists should be users of the platform, not its operators.
The platform team needs to contain — or have very clear access to — platform engineers who can operate Kubernetes, manage data pipelines, and build and maintain serving infrastructure. The data science team owns models. The platform team owns the infrastructure those models run on. The handoff point is the model registry: a data scientist promotes a model to the registry; the platform team handles everything from there to production.
In a Fortune-500 context, this usually means a dedicated AI/ML platform team of six to twelve engineers, embedded within or closely aligned to the broader platform engineering function, with clearly chartered SLAs to the business units consuming models.
The Honest Roadmap
If you are starting from a position of ad hoc notebooks and manual deployments — which is the starting position of most enterprises despite having large data teams — here is the sequence that works:
Before any model can be reliably trained or retrained, you need a versioned, auditable data access layer. This means a feature store or at minimum a governed Delta Lake with defined schemas and lineage. Everything else depends on this. Starting with the serving layer because it is more visible is the single most expensive sequencing mistake I see.
MLflow or Weights and Biases, a model registry with staging and production environments, and CI/CD that can promote a model artefact through those environments automatically. The goal is that every experiment is logged and every production model is traceable to a specific experiment run.
A single model serving platform that supports online and batch patterns, with authentication, request logging, health endpoints, and cost tracking built in. Resist the temptation to build per-model serving infrastructure. You will have ten models within two years and maintaining ten separate serving setups is not a team, it is a crisis.
Start with infrastructure and latency monitoring. Add data drift detection within 60 days of a model going live. Establish the retraining and rollback runbooks before the first drift alert fires, not after. The first time a production model drifts is not the moment to be designing the response process.
Governance is not a phase. It is a design constraint that applies from the first data pipeline to the last serving endpoint. In regulated industries, the cost of retrofitting governance onto a platform that ignored it during construction is typically larger than the cost of building the platform in the first place.
The Takeaway
Enterprise AI platforms are not won at the model layer. They are won or lost at the data layer, the serving infrastructure, the monitoring feedback loop, and the organisational model that determines who owns what. The keynote shows you the model. The boring diagram shows you everything that makes the model worth running.
The companies I have seen ship AI at scale — in reinsurers' risk-modelling teams, in energy utilities' grid operations, in the underwriting workflows of major European insurers — share one characteristic: they invested in the boring diagram before they invested in the exciting model.
If you are building or inheriting an enterprise AI platform and want an honest assessment of where the gaps are, let's talk — book a 30-minute discovery call. I have been in enough of these rooms to know where the sequencing mistakes happen.