Token & Prompt Cost Management: Why the Old Cost Model Doesn’t Fit AI Workloads
Tokens are becoming the unit of AI cost. How prompt and completion pricing, agentic workloads and multiple providers change cost management.
Tokens are becoming the unit of AI cost. How prompt and completion pricing, agentic workloads and multiple providers change cost management.
Cloud cost management was built around a relatively stable unit: an instance running for an hour costs roughly the same hour over hour. AI workloads break that assumption completely. A single prompt can cost ten times more than another prompt that looks superficially similar, depending on the model it hits, how much context it carries, and how many reasoning steps an agent takes before it produces an answer. Tokens, not hours, are the real unit of cost now, and most organizations are still budgeting as if that transition hasn’t happened.
Every call to a language model consumes tokens in both directions, the prompt going in and the completion coming back, and those two directions are often priced differently. Add an agentic workflow on top, one that reasons across multiple steps, calls tools, and iterates before returning a final answer, and a single user-facing request can trigger dozens of underlying model calls, each consuming its own token budget. Agentic workloads routinely consume many times more tokens than a simple query-response pattern, which means the old mental model of “cost roughly scales with usage” doesn’t hold. Cost scales with how a workload reasons, not just how often it’s called.
A useful view of AI spend has to separate prompt tokens from completion tokens, track cost per million tokens by model, and break spend down by model name, region, account, and even the specific line of code or workflow that triggered the call. An invoice that just shows “OpenAI: $14,200 this month” answers none of the questions that actually matter: which feature drove that, which model, and whether a cheaper model would have produced an acceptably similar result.
Organizations rarely run one model from one provider. A typical estate spans classic cloud AI services, Amazon Bedrock, SageMaker, Google Vertex AI, Azure OpenAI, alongside direct API providers like OpenAI and Anthropic, each with its own pricing structure and its own dimensions. Bedrock, for instance, prices usage across multiple token dimensions including cache-related ones, which a tool built only for simple per-call pricing will misread entirely. Getting this right means tracking each provider’s actual pricing mechanics, not applying one generic per-token rate across everything.
Three things matter most. First, token-level visibility that separates prompt from completion cost and tracks cost per million tokens by model, not just a blended total. Second, model-routing intelligence that can direct a given task to the model that actually fits its complexity, rather than defaulting every request to the most capable, most expensive option available. Third, attribution that traces AI spend back to the specific feature, workflow, or team responsible, the same discipline traditional FinOps applies to cloud spend, extended to a unit of cost that didn’t exist in the traditional model.
Agentry treats token and model-level cost as the first-class unit it was built around, not a variant of traditional compute bolted on afterward. AI Economics tracks spend across both classic cloud AI services and direct API providers, with prompt-versus-completion separation and cost-per-million-token metrics by model, region, and account. Model routing works from that same data, directing requests to the model that fits the task’s actual complexity rather than the most expensive available option by default. This runs on the same underlying data model as the FinOps Platform, so AI spend sits inside the same attribution and governance system as the rest of cloud cost, not in a separate tool that needs to be reconciled against it.
A team running an agentic customer-support workflow might discover that a small fraction of requests, the ones requiring multi-step reasoning across several tools, account for a disproportionate share of total token spend. Without model and workflow-level attribution, that imbalance is invisible: the invoice just shows a number that grew. With it, the fix is specific: route the straightforward requests to a smaller, cheaper model, and reserve the expensive model for the cases that genuinely need it.
For the GPU and infrastructure side of AI cost, training clusters and inference capacity rather than per-token pricing, see AI and GPU cost management. For why standards bodies are moving on token cost accounting ahead of most buyers, see the standards are arriving before the market is ready. For the broader multi-cloud picture, see the cloud cost management guide.
If AI spend currently shows up as one line item on a cloud bill rather than a breakdown by model, workflow, and token type, that’s the specific gap to close first, before trying to optimize anything, since you can’t route intelligently around a cost structure you can’t see.
Is this the same as general cloud cost management applied to AI?
Related, not the same. Traditional cloud cost management is built around relatively stable per-hour or per-resource pricing. Token-based pricing, and the token-volume multiplier effect of agentic reasoning, needs its own attribution logic that a general-purpose cloud cost tool usually doesn’t have natively.
Why do agentic workloads cost so much more than they appear to?
A single user-facing request can trigger many underlying model calls as an agent reasons, calls tools, and iterates before returning a final answer. Each of those calls consumes its own token budget, so the visible request undercounts the actual cost by a wide margin if only the final response is measured.
Does model routing actually produce worse results to save money?
Not when it’s done well. The goal is matching task complexity to model capability, not defaulting to the cheapest option regardless of fit. Many requests genuinely don’t need the most capable available model, and routing them to a smaller one doesn’t meaningfully change the outcome. It just avoids paying a premium the task didn’t require.