Yi Shi is the founder of FlashLabs, pioneering AI agent software. A computer science expert & e/acc proponent shaping transformative tech.
For the past three years, the central question in enterprise AI has been capability: Can the model do the task at all? In 2026, that question is largely settled. Frontier models write production code, parse legal contracts and run multistep agentic workflows.
The strategic question has shifted: How cheaply, how reliably and at what latency can your organization push billions of tokens through these models every week?
This matters because while training costs were the headline number through 2024, inference now dominates operational AI budgets. While inference costs have dropped 280-fold over the last couple of years, according to Deloitte, the increase in AI usage has meant they are still rising dramatically.
From my work building AI infrastructure, I repeatedly see the same handful of mistakes in managing inference costs. Here is what leaders can do differently:
1. Stop pricing AI like a SaaS line item.
Most finance teams still model AI as a flat per-user cost. That model is broken, especially with the rise of agentic AI.
While the cost per chat was once around 4 cents, according to EY, the price rises to $1.20 when you consider orchestration factors like knowledge-base updates, agent evaluation and human-collaboration design. In my experience, reasoning-heavy models can multiply that further because they generate thinking tokens you pay for but never see.
If you are not budgeting at the tokens-per-task level, your forecasts may be off by an order of magnitude. To address this, build a unit economics model where the basic metric is tokens-per-task-per-dollar, and track it monthly—the way SaaS companies track customer acquisition cost (CAC).
2. Multi-provider is not a luxury. It is risk management.
Single-provider deployments are a P0 incident waiting to happen. You do not have to take my word for it: Research from Ookla, which runs Downdetector, found that high-signal disruption days on major AI platforms rose from six days in the first quarter of 2025 to 51 in the same period of 2026.
Beyond reliability, price-performance moves week to week as new models ship. Research from Epoch AI found that the price to reach a fixed performance level has fallen between nine times and 900 times per year, depending on the benchmark, which means the cheapest model for your task in January is rarely the cheapest by April.
Hardcoded provider integrations cost you both money and uptime, which is why it often makes sense to build the ability to swap models easily from day one.
3. Separate the data plane from the control plane.
Over the last couple of years, the data plane—the actual token traffic—has been commoditizing. In 2024, Andreessen Horowitz called the trend “LLMflation,” which is the “rapid increase in tokens you can obtain at a constant price.”
To manage the price, many companies are building a single layer that decides which requests go to which models, enforces policy on what data leaves the building, logs everything for audit, caches what repeats and shows spend per team and per task.
One interesting trend has been a category of inference gateways—OpenRouter, LiteLLM, Portkey, Cloudflare’s AI Gateway and OrcaRouter—that package this layer, increasingly with bring-your-own-key architectures that decouple governance from token markup. (Disclosure: I helped with the ideation for OrcaRouter.)
The goal is that, when routing and governance live in a layer you control, switching models becomes a configuration change instead of a quarter-long migration.
However, you are also adding a dependency and a network hop into your critical path. This means you will need to test the latency budget, rather than assuming it, and scrutinize how any vendor handles your prompts. The alternative—assembling the same layer internally from open-source gateways—trades vendor risk for engineering headcount.
Whether you adopt a vendor or build internally, the principle holds: Pay for control, not for plumbing.
4. Treat caching as a strategic capability, not an optimization.
Most teams treat caching as something to address “later.” However, semantic caching—returning stored answers to near-duplicate queries—and KV-cache reuse on self-hosted deployments can help to stack savings.
For instance, Anthropic discounts cached input tokens by 90% relative to standard rates. OpenAI applies an automatic 50% discount on repeated prompt prefixes. That is a multimillion-dollar deferral at a meaningful scale.
When building your AI infrastructure, make caching a first-class architectural decision before your bills are large enough to force the issue.
One of the challenges, however, is that caching only pays under specific conditions: stable prompt structure, dense traffic and prompts engineered with static prefixes up front.
Cache writes also often cost more than standard tokens, so low-volume, sporadic workloads may actually lose money, and retrofitting cache-friendly prompt architecture into a live system is painful.
That is precisely why it must be an architectural decision made early, as a thoughtful strategy can mitigate these issues.
5. Build observability before you build features.
You cannot manage what you cannot see. Most production deployments cannot answer basic questions: Which prompts are most expensive? Which model handles which task best? What is our cache hit rate? Where are we paying for retries?
These are operational metrics, not research questions. Standing up token-level observability in month one saves you architectural rework in year two.
The Takeaway
By addressing these five points, your team can have a stronger understanding of cost, reliability and latency when pushing billions of tokens:
• Cost: A tokens-per-task model plus caching tells you the marginal cost of every workflow and which lever moves it.
• Reliability: Multi-provider routing can help turn a lab outage from an incident into a failover event measured in basis points of error rate.
• Latency: Observability tells you p95 by task and which model-provider pair is responsible. When your board asks what AI costs and whether it will stay up, you answer with numbers, not adjectives.
The model layer is converging: The open-weight frontier is closing in on the closed frontier, lagging by only four months, according to an Epoch AI analysis. Understanding how this impacts your organization starts with operating discipline—measuring, routing, caching and governing the tokens flowing through your organization—before the inference bill becomes a board-level agenda item.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
