Per-call LLM economics look trivial in a demo and become existential at scale. I have watched this play out at two companies in the last eighteen months — both of which shipped an AI feature without modelling the cost curve, both of which found themselves in an emergency FinOps conversation when the bill arrived.
The math is not complicated. What is complicated is that nobody does it before shipping.
The Demo Problem
In a demo or pilot, LLM costs are genuinely negligible. A thousand test calls at €0.04 each is €40. The product manager signs off, the feature ships, growth happens — and then someone opens the cloud bill.
The problem is the shape of the cost curve. LLM costs scale linearly with usage. Revenue does not — at least not until you have deliberately built pricing that accounts for it. The gap between those two lines is where margin goes to die.
Breaking Down the €0.04
Let me show what €0.04 actually represents, because the composition matters enormously for where you find optimisation leverage.
A typical AI feature — say, a document summarisation function in a B2B SaaS product — might look like this per call:
| Component | Detail | Cost |
|---|---|---|
| System prompt (1,200 tokens) | Context, instructions, format spec | €0.012 |
| User input (800 tokens avg) | Document chunk or query | €0.008 |
| Output (600 tokens avg) | Summary or response | €0.018 |
| Embedding call | For retrieval/RAG step | €0.001 |
| Overhead (retries, logging) | ~5% buffer | €0.001 |
| Total per call | €0.040 |
Now scale it:
| Monthly Call Volume | Monthly Cost | Annual Cost |
|---|---|---|
| 100,000 | €4,000 | €48,000 |
| 500,000 | €20,000 | €240,000 |
| 1,000,000 | €40,000 | €480,000 |
| 2,000,000 | €80,000 | €960,000 |
| 5,000,000 | €200,000 | €2,400,000 |
A product that reaches 5M calls/month — not outlandish for a mid-market SaaS with AI-native features — is spending €2.4M per year on inference alone. If that product charges €50/seat/month and has 2,000 seats, revenue is €1.2M/year. The AI feature costs twice what the product earns.
I am not inventing this scenario. I have seen it.
The freemium acceleration trap
Freemium tiers with AI features are a financial time bomb if the LLM cost per free user exceeds what you can absorb. A free tier that gives users 100 AI calls/month at €0.04 costs you €4 per user per month before they convert — or leave. At 10,000 free users, that is €40k/month in subsidised LLM costs. Model this before you announce the free tier.
Where the Cost Actually Lives
Most AI feature costs are not where teams expect them. After running FinOps reviews on a dozen AI-enabled products, I consistently find the same three surprises.
Surprise 1: The System Prompt Is the Biggest Token Spender
System prompts are sent with every single call. A 1,500-token system prompt repeated across 2M monthly calls costs the same as 3 billion output tokens. Teams iterate on system prompts for quality — adding examples, adding context, adding nuance — without tracking the token budget.
I make it a rule: every token added to a system prompt gets a cost estimate attached. Adding 200 tokens to a prompt running at 1M calls/month adds €2,400/month in perpetuity.
The fix: Use prompt caching. Both Anthropic (Claude) and OpenAI cache system prompts natively when you use the appropriate API parameters. Cached tokens cost 10–25% of the standard input token price. At scale, this single change reduces input costs by 60–80%.
Surprise 2: The Long Tail of Failed Calls
Retry logic without exponential backoff, rate limit handling that retries immediately, and poorly designed error flows can double your call volume without doubling your actual feature usage. I have audited systems where 20–30% of billed LLM calls were retries for errors that were not worth retrying — or that should have been handled without an LLM call at all.
Instrument your retry rate. If more than 5% of LLM calls are retries, you have a problem worth fixing before optimising anything else.
Surprise 3: Using GPT-4 / Claude Opus Where GPT-4o-mini / Haiku Would Work
The most common expensive mistake is using flagship models as the default for everything. Teams start with the best model during development — because quality matters when you are iterating — and never revisit the model selection for production.
The cost difference between model tiers is not incremental. It is 10–50x.
| Task | Appropriate Model | vs. Flagship Cost |
|---|---|---|
| Classification, tagging, routing | Haiku / GPT-4o-mini | 95–98% cheaper |
| Summarisation, structured extraction | Sonnet / GPT-4o | 60–80% cheaper |
| Complex reasoning, multi-step agents | Opus / GPT-4 | Baseline |
| Embeddings | Dedicated embedding model | 99% cheaper than using LLM |
The rule I use with every engineering team: flagship models are for tasks where you have demonstrated that cheaper models fail. Not "might fail" — demonstrated, with evals, that the quality delta is material to the product outcome.
The Unit Economics Model You Need Before You Ship
Before any AI feature goes to production, I require a unit economics model. It does not need to be a spreadsheet with 47 tabs. It needs five numbers:
- Average tokens per call (input + output, broken down)
- Expected call volume at 6 months and 12 months (with growth rate assumption)
- Cost per call at current model selection
- Revenue attribution per call (either directly monetised, or allocated from seat revenue)
- Gross margin impact (revenue attribution minus cost per call)
If number 5 is negative or below your product margin floor, you do not have a cost problem to fix later. You have a pricing problem to fix now.
The right question is margin per call, not cost per call
A €0.10 cost per call is fine if the feature commands €0.50 in revenue attribution. A €0.02 cost per call is a problem if the feature is free and you are subsidising it from elsewhere. Cost per call is a distraction. Margin per call is the metric.
Five Levers That Actually Move the Number
These are the controls I reach for first, ranked by impact-to-effort ratio.
Lever 1: Prompt Caching (Highest impact, lowest effort)
Enable provider-native prompt caching for system prompts. This is an API parameter change, not an architecture change. Impact: 50–80% reduction in input token costs for high-volume endpoints.
Lever 2: Model Routing
Build a simple routing layer that sends tasks to the appropriate model tier based on task type. Start with hardcoded rules — "classification tasks go to Haiku, generation tasks go to Sonnet, escalation tasks go to Opus" — rather than dynamic complexity scoring. You can add sophistication later.
Impact: 40–70% overall cost reduction, depending on your task mix.
Lever 3: Semantic Caching
Cache LLM responses for semantically similar queries. If your AI feature answers questions, a significant proportion of real-world queries will be similar enough to serve from cache. The implementation:
- Embed the incoming query
- Search a vector store for similar past queries (cosine similarity threshold: 0.92–0.95)
- Return cached response if found; call LLM if not
Impact: 20–50% reduction in API calls for customer-facing Q&A features. Lower for generative or personalised features.
Lever 4: Output Length Control
Output tokens cost 2–4x more than input tokens on most models. Instruct the model to be concise and use structured output formats (JSON with defined schemas) rather than free-form prose where the task allows. Setting a max_tokens limit appropriate to the task prevents occasional runaway responses from blowing up your average cost.
Impact: 10–30% reduction in output costs.
Lever 5: Async Batching for Non-Real-Time Workloads
Document processing, report generation, nightly analysis pipelines — these do not need synchronous real-time inference. Batch APIs (available from Anthropic and OpenAI) typically cost 50% less than the synchronous equivalent. Route anything with a latency tolerance above 30 seconds through the batch API.
Impact: 50% cost reduction on batch workloads.
The flow through an LLM gateway that applies these levers in order looks like this:
The Governance Structure That Prevents the Crisis
Cost surprises happen when AI spending is invisible until the cloud bill arrives. The fix is operational, not technical:
Per-feature cost tagging. Every LLM call should be tagged with the feature that triggered it. This requires a metadata field in your LLM gateway or wrapper. Without it, you cannot attribute spend to product decisions.
Budget alerts, not just reports. Set a hard alert at 80% of monthly LLM budget — not at the end of the month when the invoice arrives. For fast-growing products, set a daily spend alert at 2x the trailing 7-day daily average.
Cost as a definition-of-done criterion. Before a new AI feature ships, the engineering spec should include a cost estimate at 100k, 500k, and 1M monthly calls. This is not optional overhead. It is the same category of due diligence as capacity planning.
Quarterly model review. Model pricing changes. New model tiers appear. What cost €0.04 per call in Q1 2025 may cost €0.015 per call in Q4 2025 on a new model tier that did not exist. Review your model selections quarterly against current pricing and newly available options.
The LLM Gateway pattern
If you have more than two or three AI features in production, you need an LLM gateway — a single internal service that all features route through for LLM calls. This gives you: centralised cost tagging, a single point for implementing caching and routing, a circuit breaker for runaway spend, and a log of every LLM call for debugging. Without it, cost controls are applied inconsistently and debugging is painful.
What Good Looks Like at Scale
A well-governed AI product at 2M monthly calls does not cost €80k/month. After applying the levers above:
| Optimisation Applied | Monthly Call Cost | vs. Naive |
|---|---|---|
| Baseline (GPT-4/Opus on everything) | €80,000 | — |
| Prompt caching enabled | €48,000 | -40% |
| Model routing added | €22,000 | -54% vs cached |
| Semantic caching (30% hit rate) | €15,400 | -30% |
| Batch routing for async tasks | €12,000 | -22% |
| Optimised total | €12,000 | -85% vs naive |
The same product, the same feature, the same call volume: €12k/month instead of €80k/month. The difference is not magic. It is one sprint of focused FinOps work applied before the cost becomes a crisis.
€0.04 per call is not a problem. €0.04 per call, multiplied by a number you did not model, compounded by a model tier you did not revisit, served through a retry loop you did not instrument — that is a problem.
Model the cost curve before you ship the feature. Price it before you make it free. Instrument it before it surprises you.
If your AI features are scaling faster than your cost controls, let's talk — book a 30-minute discovery call and we will work through the unit economics together.