The exciting part of agentic AI — the bit where an LLM autonomously plans a multi-step task, calls tools, evaluates its own output, and iterates — is a rounding error next to the integration work it actually requires. I've been in the room for several enterprise agentic deployments. The agents themselves took weeks to build. The plumbing around them took months.
This isn't a complaint about the technology. It's a calibration. The companies that are winning with agentic AI right now are the ones that went in with eyes open about where the work actually lives. The ones that went in sold on the agent demo are the ones now explaining to their boards why the pilot is still a pilot six months later.
What "Agentic AI" Actually Means in the Enterprise
An agent, in the technical sense, is an LLM that receives a goal, plans a sequence of steps, executes those steps using tools (APIs, databases, browser automation, code execution), evaluates the result, and iterates until the goal is met or it gives up. That is genuinely powerful. It is also a small fraction of what a production agentic system requires.
A production agentic deployment in an enterprise context is more precisely described as: an agent plus an authentication layer, a tool registry, an orchestration framework, a memory and state management layer, a human-in-the-loop override mechanism, an audit log, a cost control system, a retry and error handling layer, a monitoring stack, a security review process, and — in any regulated industry — a governance and explainability framework.
The agent is the interesting bit. Everything else is engineering.
The demo-to-production gap is wider for agents than for any other AI pattern
A RAG application or a classification model has well-understood production requirements. An agentic system is autonomous, stateful, non-deterministic, and capable of taking irreversible actions. That combination means the production requirements are significantly more demanding — and significantly less well-understood by most engineering teams.
The 95%: What the Plumbing Actually Involves
Tool Integration
An agent is only as useful as the tools it can call. In the enterprise, those tools are: internal APIs (often undocumented, poorly maintained, or rate-limited), legacy systems with SOAP interfaces and batch-only access, third-party SaaS platforms with restrictive API terms, databases that weren't designed for programmatic agent access, and file systems, document stores, and collaboration tools that each have their own authentication model.
Building a reliable tool layer for an enterprise agent means: wrapping every tool in a standardised interface, implementing retry logic with exponential backoff, handling partial failures gracefully, building a tool registry so the agent can discover what's available, and — critically — designing idempotency into every tool call so a retried action doesn't double-book a meeting, send a duplicate email, or trigger two purchase orders.
In one engagement at an energy company, the agent's tool layer required integrations into 11 separate internal systems. Four of them had no documented API. Two of them required VPN with certificate-based authentication. One required a formal change request to add any new API consumer. That process took six weeks per system. The agent logic itself took three weeks.
Authentication and Identity
An enterprise agent acts on behalf of a user — or on behalf of the company — using real credentials against real systems. That means you need answers to questions that don't come up in demos:
Whose identity does the agent act under? Does it have a service account with its own credentials? If so, what is the blast radius if those credentials are compromised? If it delegates the user's identity, how does that work across system boundaries?
How are credentials stored and rotated? An agent that hardcodes API keys in its tool definitions is a security incident waiting to happen. Production agentic systems need a secrets management layer — HashiCorp Vault, Azure Key Vault, AWS Secrets Manager — with rotation policies, access logging, and least-privilege scoping for each tool.
How do you audit what the agent did, on whose behalf, and with what authorisation? In regulated industries (insurance, energy, financial services), every action the agent takes needs an immutable audit trail that can be reconstructed months later for a regulator.
Orchestration and State Management
Real agentic workflows are long-running. An agent researching a procurement decision might run for 20 minutes, calling 15 tools, accumulating context, and producing intermediate artefacts before delivering a final output. That means:
State must be persisted. If the agent fails at step 12 of a 20-step task, it shouldn't start over from the beginning. It needs a checkpointing mechanism that can resume from the last successful state.
Timeouts must be designed explicitly. What happens if a tool call doesn't return for 90 seconds? What if the orchestration session exceeds a maximum duration? You need defined behaviour for every failure mode, not just happy-path logic.
Parallel sub-agent calls need coordination. If your orchestrator dispatches three specialist agents simultaneously, you need to handle partial failures, conflicting results, and the ordering of outputs before synthesis. Frameworks like LangGraph, Temporal, or custom orchestration on top of queuing systems (SQS, Azure Service Bus) handle this — but choosing and implementing the right framework is non-trivial.
Human-in-the-Loop
No enterprise should deploy a fully autonomous agent with the ability to take irreversible actions without a human override mechanism. Not for legal reasons. Not for regulatory reasons (though those exist too). Simply because agents make mistakes, and the class of mistakes an autonomous agent makes — confabulation, misinterpreted instructions, tool misuse — can be significantly more consequential than a static model returning a wrong classification.
A human-in-the-loop system requires: a queue where agent actions awaiting approval are surfaced, a UI where an authorised human can review, approve, reject, or edit the proposed action, SLAs for how long an approval can take before the agent times out, and a fallback path if approval is not received in time.
Building this is not trivial. It's an application in itself — with authentication, notifications, mobile responsiveness if approvers are not desk-based, and audit logging of every approval decision.
Observability and Cost Control
An autonomous agent can spend money at a rate that would surprise you. In a production environment, without cost controls, a single misconfigured agent workflow can consume thousands of dollars of LLM API calls before anyone notices. I have seen it happen.
Production agentic systems need: per-run token budgets that cause the agent to stop and escalate if exceeded, per-tool-call cost logging, aggregate cost dashboards by agent type and by user, and alerting when spend is anomalous — not just in aggregate, but at the individual run level.
Observability goes beyond cost. You need trace-level logging of every step the agent took, every tool it called, every intermediate reasoning step, and every output it produced. This is both for debugging (agentic failures are notoriously hard to diagnose from error messages alone) and for the governance and audit requirements mentioned above.
The Reality of Regulated Industries
I've built or overseen agentic deployments at companies operating under insurance regulation, financial services regulation, and energy sector regulation. In every case, the regulatory requirements added a significant layer on top of the already-substantial engineering work:
| Requirement | What it means in practice |
|---|---|
| Explainability | Audit log of every reasoning step, not just the final output |
| Human oversight | Mandatory approval gates for defined action categories |
| Data residency | Agent and all tools must process data within defined geographic boundaries |
| Model governance | Change control for any prompt or model version change, with rollback capability |
| Bias and fairness monitoring | Periodic review of agent outputs for discriminatory patterns |
| Incident reporting | Defined escalation path when the agent takes an unintended action |
None of this is impossible. It is all engineering. But it must be planned from day one, because retrofitting it onto an agent that was designed without these constraints is significantly more expensive than building it in.
The prompt is not the product
The single most common error I see in agentic AI projects is a team that spent 80% of its time on prompt engineering and 20% on everything else. A beautifully crafted system prompt does not constitute a production system. The prompt is roughly the equivalent of writing a job description — necessary but nowhere near sufficient. The production system is the hiring process, the onboarding, the tooling, the oversight, the performance management, and the audit trail. All of that has to be built.
What a Production-Ready Agentic Stack Actually Looks Like
This is the architecture I use as a reference model. Every layer is necessary. Missing any one of them creates a gap that will surface in production.
| Layer | What it includes | Common tools |
|---|---|---|
| Agent core | System prompt, model, reasoning loop | GPT-4o, Claude 3.5, Gemini 1.5 |
| Tool layer | Tool definitions, retry logic, idempotency | Custom wrappers, OpenAPI specs |
| Orchestration | State machine, checkpointing, parallel coordination | LangGraph, Temporal, Prefect |
| Identity / Auth | Service account, secrets rotation, per-tool scoping | Azure Key Vault, HashiCorp Vault |
| Human-in-the-loop | Approval queue, reviewer UI, timeout handling | Custom-built (typically) |
| Observability | Trace logging, cost tracking, anomaly alerting | LangSmith, Arize, custom |
| Governance | Audit log, change control, model versioning | Custom + compliance tooling |
| Security | Prompt injection defence, output filtering, access audit | Guardrails AI, custom |
The agent core — the part people demo — sits at the top. Everything below it is the plumbing. In my experience, the plumbing takes three to five times as long to build as the agent itself.
Every layer feeds into the next, and none can be skipped in a production deployment:
How to Scope an Agentic AI Project Honestly
If you're a CEO or CTO evaluating an agentic AI initiative, here is the honest scoping framework:
Start with the tool inventory. Before you think about the agent, list every system it needs to touch. For each: does it have an API? Is it documented? What is the authentication model? How long does it take to get a new API consumer approved? This alone will tell you whether you're looking at a 3-month project or a 9-month project.
Identify the irreversible actions early. Anything the agent can do that cannot be undone — sending communications, making purchases, modifying records, triggering workflows — needs a human-in-the-loop gate. The number of those gates determines the complexity of your oversight architecture.
Plan for the regulated version, even if you think you don't need it yet. I have never seen an enterprise agentic deployment that turned out to need less governance than initially expected. I have seen many that needed significantly more. Build for the audit trail from day one.
Budget the plumbing explicitly. A rough heuristic: if your agent logic is estimated at 6 weeks, budget 18–24 weeks for the full production stack. If that ratio seems wrong to you, it probably means you haven't scoped the plumbing yet.
Agentic AI is real, it is useful, and enterprises that get it right will have a genuine operational advantage. But getting it right means building the 95%, not just the 5%. The teams that understand that from the start are the ones who ship. The teams that discover it midway through are the ones that explain to their boards why the demo is still a demo.
If you're planning an agentic AI initiative and want to scope it honestly — or if you've already started and the plumbing is the problem — let's talk. I can help you build the architecture that actually gets to production. Book a 30-minute discovery call.