You launch an LLM feature.
Users like it. Usage grows. Someone adds conversation history. Someone else adds RAG. Then tools. Then retries. Then an evaluator that uses another LLM.
A few months later, finance asks a question engineering teams increasingly have to answer:
Why is the AI bill growing faster than the product?
The first reaction is often to open the system prompt and start deleting adjectives.
That is usually the wrong place to begin.
Reducing a 1,200-token prompt to 1,050 tokens is useful, but it does not fix an architecture that sends every request to the most expensive model, resends thousands of irrelevant context tokens on every turn, retries silently, generates text nobody displays, and processes overnight jobs through real-time endpoints.
LLM cost optimization is increasingly a systems problem.
The useful question is not:
How do we make this prompt cheaper?
It is:
Why did this request need this model, this much context, this many calls, and this much output in the first place?
That shift changes where you look for savings.
Start With the Cost Equation
Before touching the architecture, write down what you are actually paying for.
Depending on the provider, a simplified request cost looks something like:
request cost =
uncached input tokens
+ cache writes
+ cache reads
+ output tokens
+ tool/API charges
+ retry cost
+ supporting infrastructure
Not every provider exposes or prices those components identically.
That matters.
For example, prompt caching can have separate read and write economics. Current OpenAI models can price cached reads substantially below normal input, while explicit cache writes can carry their own pricing. Anthropic similarly separates normal input, cache writes, and cache reads.
So the first optimization rule is almost boring:
Measure before optimizing.
It is also the rule teams skip most often.
1. Build Cost Attribution Before Cost Optimization
If LLM calls are scattered across services, debugging cost becomes surprisingly difficult.
A classification call in one service triggers a retrieval pipeline in another, which triggers generation, which fails validation, which retries, which invokes an evaluator.
The invoice sees tokens.
You need to see why those tokens existed.
At minimum, record something like:
type LLMUsageEvent = {
requestId: string;
feature: string;
tenantId?: string;
model: string;
route: "small" | "standard" | "frontier";
promptVersion: string;
inputTokens: number;
cachedInputTokens?: number;
outputTokens: number;
purpose:
| "user-response"
| "classification"
| "retrieval"
| "summarization"
| "evaluation"
| "retry"
| "background-job";
latencyMs: number;
retryCount: number;
estimatedCostUsd: number;
};
The exact schema is less important than the attribution.
You want to answer questions such as:
- Which feature spends the most?
- Which tenants generate disproportionate usage?
- Which model handles most requests?
- How much spend comes from retries?
- How much comes from internal LLM calls rather than user-visible responses?
- Which prompt versions increased average context size?
- What percentage of eligible input is actually being served from cache?
- What is the cost per successful business outcome?
That last question matters most.
A support assistant costing $0.08 per conversation may be wonderful economics if it resolves a $6 support interaction.
A summarization feature costing $0.03 per click may be terrible economics if almost nobody uses the summary.
Cheap tokens are not the goal. Efficient product economics are.
2. Route by Difficulty Instead of Sending Everything to Your Best Model
One of the most powerful architecture changes is also conceptually simple:
Not every task deserves your most capable model.
A production workload may contain:
- classification,
- entity extraction,
- formatting,
- summarization,
- straightforward retrieval synthesis,
- code generation,
- ambiguous reasoning,
- multi-step agentic work,
- high-risk decisions.
Those are not the same workload.
They should not automatically have the same inference path.
A basic architecture looks like this:
┌── simple ─────> small / fast model
Request -> Router ───┼── normal ─────> standard model
└── difficult ───> frontier model
↑
validation failure ───┘
The price spread between model tiers can be meaningful, although the exact ratio changes constantly as providers update their lineups and pricing. The important architectural principle is independent of today’s price table: buy only the capability the task requires.
Don’t Build an ML Router on Day One
You probably don’t need a sophisticated learned router initially.
Start with signals you already have:
function chooseModel(request: AIRequest): ModelTier {
if (request.task === "classification") {
return "small";
}
if (request.task === "structured-extraction") {
return "small";
}
if (request.requiresDeepReasoning) {
return "frontier";
}
if (request.risk === "high") {
return "frontier";
}
return "standard";
}
Later, routing can incorporate:
- historical success rates,
- request complexity,
- context length,
- evaluation scores,
- model confidence proxies,
- latency targets,
- tenant SLA,
- task-specific classifiers.
But routing becomes dangerous when cost is the only optimization target.
The correct objective is closer to:
minimize cost
subject to:
quality >= required threshold
latency <= product SLO
safety >= required threshold
That is a very different problem from “use the cheapest model.”
Escalation Makes Routing Safer
Cheap models should not be dead ends.
If a lower-cost model fails schema validation, violates a business rule, produces low-confidence extraction, or fails a task-specific evaluator, escalate the request.
Small model
|
+--- valid ----------> return
|
+--- validation fail -> Standard model
|
+--- fail -> Frontier model
You turn model-routing mistakes into an occasional additional inference rather than a customer-visible failure.
The trade-off is latency, so measure escalation rates rather than pretending routing is free.
3. Put Your Context on a Diet
The next place to look is the request itself.
Modern AI applications often accumulate context without anyone intentionally designing a context budget.
A request gradually becomes:
system instructions
+ user profile
+ conversation history
+ retrieved documents
+ tool schemas
+ examples
+ memory
+ intermediate results
+ current message
Every individual addition seemed reasonable.
The resulting prompt may not be.
The right question is:
What information does the model need to solve this request correctly?
Not:
What information do we happen to have available?
Conversation History
Sending the entire conversation forever is easy to implement and expensive to scale.
A more deliberate strategy is:
stable instructions
+ structured conversation state
+ summary of older history
+ recent verbatim turns
+ current request
But summarization has a quality cost: summaries can discard details.
So don’t blindly compress everything.
Preserve information that must remain exact—identifiers, decisions, constraints, contractual language, code, numerical values—and summarize only history where semantic compression is acceptable.
Retrieval
RAG systems have their own version of context inflation.
A team starts with topK = 10 because ten feels safer than three.
Eventually every query carries ten large chunks into generation whether the model needs them or not.
There is no universal correct k.
k = 3 is not inherently better than k = 10.
Instead, treat retrieval as an evaluated budget:
retrieval quality
vs.
generation quality
vs.
tokens added
vs.
latency
Good systems may combine:
- broader candidate retrieval,
- reranking,
- deduplication,
- relevance thresholds,
- a final context-token budget.
Measure answer quality as you reduce context.
If you can remove 40% of retrieved text with no measurable degradation, that is evidence.
Tool Definitions
Agentic applications can have another hidden source of context: tools.
If the agent knows about dozens of functions, APIs, or MCP tools, you may be repeatedly providing schemas that are irrelevant to the current task.
Where your model/API architecture allows it, expose the smallest useful tool set for the current intent.
A billing question probably does not need the deployment tools.
A calendar request probably does not need your SQL administration schema.
Besides cost, this can simplify the model’s decision surface.
4. Design for Prompt Caching Instead of Merely Enabling It
Caching deserves more nuance than “turn it on.”
Different providers implement caching differently, but the underlying economic idea is similar:
Reusable prompt material should not always require the same computation and price as brand-new context.
Current OpenAI pricing, for example, includes substantially discounted cached-input pricing, while Anthropic exposes separate cache-write and cache-read pricing. Google’s Gemini API also provides context caching for eligible models and workloads.
The architecture implication is important.
Separate stable context from volatile context.
Conceptually:
MOST STABLE
│
├── system / policy instructions
├── stable examples
├── stable tool definitions
├── relatively stable reference context
├── recent conversation
├── retrieved data for this request
└── current user message
│
MOST VOLATILE
Historically, prefix reuse has been especially important for cache-friendly request construction, and modern APIs increasingly expose more explicit cache controls as well. OpenAI’s current API, for example, supports explicit prompt-caching behavior and extended cache-retention controls in addition to automatic behavior.
There is an important catch:
A cache write is not necessarily free.
Depending on provider and configuration, writing reusable context can cost more than ordinary input, while later cache reads are cheaper. Anthropic explicitly uses this model, and current OpenAI explicit caching also distinguishes write economics.
That means caching has a break-even point.
A huge static prompt reused hundreds of times may be a fantastic candidate.
A giant prompt used twice may not be.
Measure:
cache hit rate
cache read tokens
cache write tokens
cache lifetime
requests per cached prefix
net dollars saved
Don’t celebrate “cache enabled.”
Celebrate lower unit cost.
5. Semantic Caching Is Powerful—and Much Easier to Get Wrong
Prompt caching still executes a model request.
A semantic cache tries to avoid the request entirely.
Suppose users repeatedly ask:
How do I reset my password?
I forgot my password.
Where can I change my login password?
How can I create a new password?
For sufficiently stable knowledge, the application can embed the new question, find a semantically equivalent cached query, and reuse a previously generated answer.
The architecture looks roughly like:
Request
|
Embed query
|
Semantic search in response cache
|
+--- strong safe match ---> cached answer
|
+--- no match -----------> LLM
|
+-> cache eligible result
The expensive part is not implementing nearest-neighbor search.
The expensive part is proving that two similar questions can safely share an answer.
Cache Identity Must Include More Than Text
Never assume:
cacheKey = semanticSimilarity(question)
is enough.
Your effective identity may include:
semantic intent
+ tenant
+ permission scope
+ locale
+ document/version
+ product version
+ prompt version
+ model version
+ freshness requirements
Without those boundaries, semantic caching can become a data-isolation problem.
Two users can ask linguistically identical questions and still require different answers.
Some Answers Should Not Be Semantically Cached
Be extremely careful with:
- account-specific information,
- authorization-sensitive results,
- rapidly changing data,
- inventory,
- market data,
- legal or policy status,
- tool-derived live state,
- personalized recommendations.
Semantic caching works best where equivalent questions genuinely have equivalent answers.
6. Move Non-Interactive Work Off the Interactive Path
Ask yourself how much of your AI workload actually requires an answer in a few seconds.
Probably less than you think.
Common background workloads include:
- document enrichment,
- offline extraction,
- nightly summaries,
- evaluation suites,
- synthetic-data generation,
- LLM-as-a-judge jobs,
- bulk classification,
- content tagging,
- migration/backfill work.
Those jobs should be architected differently from chat.
Several major providers currently offer substantial discounts for eligible asynchronous batch processing. OpenAI advertises 50% savings on input and output with its Batch API; Anthropic similarly documents a 50% discount for batch processing; Google’s Gemini API also currently lists a 50% reduction for batch workloads. Exact availability and turnaround requirements vary by provider and model.
So create two lanes.
┌── interactive lane ──> synchronous inference
Request/Job ─────┤
└── background lane ───> queue -> batch processing
This distinction should exist at the platform level rather than being rediscovered by every product team.
Something as simple as:
type LatencyClass =
| "interactive"
| "background"
| "bulk";
can become an important architecture decision.
A nightly evaluation job should not accidentally pay for the same service characteristics as a user waiting on a chat response.
7. Stop Paying for Output Nobody Uses
Context optimization receives a lot of attention because prompts are visible.
Outputs deserve the same scrutiny.
Output tokens can be significantly more expensive per token than input on leading APIs. Current OpenAI GPT-5.6 Sol pricing, for example, lists $5 per million standard input tokens and $30 per million output tokens. Anthropic lists Opus 4.8 starting at $5 per million input and $25 per million output.
Now inspect your UI.
Does the application display the whole response?
Or does it:
- truncate after two paragraphs?
- parse only a JSON field?
- consume only a classification label?
- discard reasoning-like prose?
- throw away five alternatives and keep one?
If the consumer needs:
{
"category": "billing"
}
don’t ask the model for:
Based on my careful analysis of the user's request,
I believe this question is most appropriately categorized
under billing for the following reasons...
Use structured outputs where appropriate.
Set output ceilings based on the task rather than one enormous global maximum.
And measure tokens generated versus tokens actually consumed by the product.
That ratio can be surprisingly revealing.
8. Retries Need a Budget Too
Retries are necessary.
Unbounded retries are an architecture bug.
An LLM call can fail because of:
- transient provider errors,
- timeout configuration,
- tool failure,
- schema-validation failure,
- content validation,
- application bugs.
If every failure retries the entire expensive pipeline, costs multiply quietly.
A sensible retry policy considers:
error type
+ previous attempts
+ model used
+ context size
+ expected business value
A transient HTTP failure may justify retrying the same request.
A schema-validation failure may justify changing strategy.
A bad tool result may require fixing the tool input rather than asking the same model three more times.
For agentic systems, use both:
maximum steps
maximum model calls
maximum tokens
maximum wall-clock time
maximum estimated spend
An agent should have a budget before it starts working.
9. Make Cost a First-Class Platform Constraint
Cost-aware architecture is becoming part of modern system design: model routing, caching, workload isolation, observability, rate limits, and failure budgets all influence both reliability and unit economics.
The most dangerous cost optimization project is the one that succeeds once.
Because six months later:
- a new feature bypasses routing,
- context grows,
- another tool gets added,
- retry limits disappear,
- somebody switches models,
- the cache hit rate falls,
- and the invoice quietly returns.
You need mechanisms that resist that entropy.
Per-Feature Ownership
Every major LLM feature should have an owner who can see:
requests
tokens
model mix
cache hit rate
retry rate
latency
cost
business outcome
Cost that belongs to “the AI platform” tends to become nobody’s problem.
Cost Regression Testing
Prompt changes are code changes.
Treat them that way.
If an eval suite contains representative requests, a CI job can estimate:
old average input tokens
new average input tokens
old average output tokens
new average output tokens
old success rate
new success rate
old estimated cost
new estimated cost
Then a pull request can say:
Quality: +1.2%
Median latency: +4%
Estimated inference cost: +37%
Now you can have the right engineering discussion.
Maybe the quality gain is worth the cost.
Maybe it isn’t.
The point is that the trade-off becomes explicit.
Tenant and Agent Circuit Breakers
For multi-tenant platforms, budget enforcement belongs in the gateway.
Think:
per-user limits
per-tenant limits
per-feature limits
agent step budgets
daily/monthly budget alerts
provider failover rules
This is not just finance governance.
It is reliability engineering.
A runaway loop that happens to call an API with usage-based pricing is both an operational incident and a financial incident.
A Better Order of Operations
If I were designing an LLM cost program from scratch, I would work roughly in this order:
1. Attribute
Know where the money goes.
2. Remove accidental spend
Fix pathological retries, forgotten background jobs, duplicate calls, obsolete fallbacks, and unused generation.
3. Route intelligently
Match model capability to task difficulty.
4. Reduce unnecessary context
Budget conversation history, retrieval, memory, and tools.
5. Improve cache economics
Restructure reusable context and measure hit rates.
6. Separate latency classes
Batch the work that doesn’t need interactive latency.
7. Control outputs
Generate only what downstream consumers use.
8. Add regression gates and budgets
Make the savings survive the next release.
This order matters.
You do not want to spend two weeks shaving 100 tokens from a prompt only to discover that a retry bug was making every request three times.
What I Would Not Do
A few approaches sound attractive but frequently move the problem somewhere else.
“Just use the cheapest model.”
Only if your evals say it works.
Cost optimization without quality constraints is product degradation.
“Let’s summarize all conversation history.”
Only if you can tolerate losing information.
Some history should be compressed. Some state should be structured. Some information should remain verbatim.
“Let’s cache every answer.”
Caching the wrong answer faster is not optimization.
Cache identity, authorization, invalidation, and freshness are part of the design.
“Let’s reduce retrieval to three chunks.”
Three is not magic.
Find the smallest context budget that preserves your required retrieval and answer quality.
“Let’s self-host.”
Maybe.
Self-hosting changes the economics from API metering to infrastructure economics: accelerators, utilization, serving software, observability, scaling, reliability, upgrades, and engineering time.
At sufficiently high and predictable utilization, that can be attractive.
At low or spiky utilization, the supposedly cheaper model may come with a surprisingly expensive platform.
Calculate total cost of ownership rather than comparing only token prices.
What Should You Optimize: Total Spend or Cost Per Outcome?
This is the most important distinction in the article.
Imagine usage doubles.
Your AI bill grows 70%.
Is that bad?
Not necessarily.
If successful customer resolutions grew 120%, your economics improved.
Conversely, your infrastructure team could reduce the total bill by 30% while destroying response quality and user retention.
That is not a successful optimization.
For each meaningful feature, define a unit that connects inference to value:
cost per resolved ticket
cost per generated report
cost per successful workflow
cost per active conversation
cost per accepted code suggestion
cost per qualified lead
Then optimize that number.
A Practical LLM Cost Review Checklist
When reviewing a production AI system, I would ask these questions in order:
- Can we attribute every LLM call to a feature and purpose?
- Are retries visible and bounded?
- Does every task need the model it currently uses?
- Do we have an escalation path when a cheaper model fails?
- How much context do we send per successful request?
- Which context is actually relevant?
- How much retrieved content reaches generation?
- Are irrelevant tool schemas being sent?
- What percentage of reusable context receives cache benefits?
- Are cache writes actually earning enough subsequent reads?
- Which jobs can tolerate asynchronous processing?
- How many generated tokens are actually consumed?
- Can an agent exceed a defined spend or step budget?
- Do prompt changes show cost deltas in CI?
- What business outcome does each dollar of inference buy?
If you cannot answer the first few questions, don’t start by rewriting the prompt.
Start by fixing observability.
Frequently Asked Questions
Does model routing reduce quality?
It can.
That is why routing should be evaluated like any other model change.
Measure task success by route, define escalation conditions, and compare the resulting quality, latency, and cost against your baseline.
A router without evaluation is just a cheaper way to guess.
Is prompt caching always worth using?
No.
Its value depends on your provider’s pricing model, how much context is reusable, how often it is reused, cache retention, and cache-write economics.
High-reuse stable context is usually much more interesting than large one-off prompts.
Should I always summarize old conversation history?
No.
Summarization trades token reduction for information loss.
A robust pattern is often a combination of structured state, summarized older conversation, and a recent verbatim window.
Is batch processing really cheaper?
For eligible workloads, several major API providers currently advertise roughly 50% discounts compared with standard processing, although model support, processing windows, and feature compatibility vary.
If your workload is genuinely asynchronous, it is worth evaluating.
Should AI cost grow slower than usage?
Ideally, unit economics improve as routing, caching, and infrastructure mature.
But there is no law saying total AI spend must grow sublinearly.
A company may deliberately spend more inference per request because a more capable system creates more business value.
The metric to watch is not simply the invoice.
It is cost relative to useful outcome.
Conclusion
The expensive part of an LLM application is rarely one badly worded prompt.
The cost emerges from hundreds of architecture decisions:
Which model receives the request?
What context gets attached?
What gets retrieved?
Which tools are exposed?
How many times can the system retry?
Can the request reuse cached computation?
Does it need real-time processing?
How much output can it generate?
What happens when an agent refuses to stop?
Those decisions eventually become the invoice.
That is good news.
Because it means the bill is not weather.
It is not something that simply happens when an AI product becomes popular.
It is a property of the system you designed.
Instrument it.
Route deliberately.
Budget context.
Cache intentionally.
Batch the slow path.
Bound outputs and retries.
Then put enough governance around those decisions that the next feature cannot casually undo them.
The goal isn’t to make every token cheap.
The goal is to stop paying for tokens that never needed to exist.