Forecasting token spend for next quarter
Why last quarter’s bill is a bad predictor
If you run the finance conversation the way most teams do, you pull last month’s API invoice, multiply by three, and call it a quarterly forecast. That number is almost always wrong, and it’s wrong in a specific, predictable way: it assumes usage grows in a straight line and that the mix of what you’re calling the model doesn’t change. Neither of those things holds once you have an agent workflow, a RAG pipeline, or a coding assistant in the loop, because all three of those change the shape of a request, not just the count of requests.
A forecast built on “spend times three” also hides the two variables that actually move your bill: how many requests you’re sending, and how many tokens each request burns going in and coming out. Those two move independently, and mixing them into one dollar figure throws away the information you need to catch a runaway cost before finance does.
Separate volume from depth before you forecast anything
Pull your usage logs, not your invoice, as the starting point. Every major API provider gives you a usage export with timestamps, model name, input token count, and output token count per call. If you’re going through an orchestration layer like LangChain, LlamaIndex, or a custom agent framework, make sure you’re logging at the call level, not just at the session level, because a single user-facing “chat turn” can fire five or six model calls underneath (a router call, a retrieval call, a synthesis call, sometimes a self-critique pass).
Once you have that data, split it into two series:
- Request volume: how many calls per day, per week.
- Tokens per request: input tokens and output tokens, tracked separately, because input and output are usually priced differently and they grow for different reasons.
Volume growth is usually tied to something you can already forecast, like signups, active users, or ticket volume if this is a support tool. Tokens-per-request growth is tied to something else entirely: longer conversation histories, bigger RAG chunks, more tool calls per agent turn, or a model swap. Treat them as separate curves and you’ll catch the real driver instead of blaming “more usage” for a cost spike that’s actually a context window problem.
Break usage into buckets before you trust a single trend line
A single blended average across your whole product is close to useless for forecasting, because a support chatbot answering one-off questions and a coding agent iterating on a file have completely different token profiles. Bucket your usage by workload type before you fit any trend:
- Simple Q&A or classification calls: short input, short output, usually stable per-call cost.
- RAG-backed answers: input cost scales with chunk size and how many chunks you retrieve, output stays relatively fixed.
- Multi-step agents: cost compounds because each step re-sends prior context, so a five-step agent loop isn’t five times the cost of one call, it’s closer to fifteen times, because steps two through five are each re-reading everything that came before.
- Coding assistant sessions: these tend to have the fattest input tokens of any bucket, because the model is reading whole files or diffs, and that input can grow every time the user asks a follow-up in the same session.
If agent and coding workloads are growing as a share of total volume, your average cost per request will climb even if your total request count stays flat. That’s the shift that a “multiply last month by three” forecast completely misses, and it’s usually where the quarter-over-quarter surprise comes from.
Model mix is the lever that moves the bill the most
Of everything that affects a token spend forecast, nothing swings the number more than which model actually serves the request. Providers ship tiered model families for a reason: a smaller, cheaper model for routing and simple lookups, a mid-tier model for most production traffic, and a frontier model reserved for the calls that need the extra reasoning. If your routing logic defaults everything to the top-tier model because it’s easiest to wire up, your forecast needs to account for that as a policy choice, not a fixed cost of doing business.
The forecast question to ask isn’t “what will tokens cost next quarter,” it’s “what fraction of our calls do we expect to route to which tier, and is that ratio going to hold.” If you’re planning to tighten routing (push simple lookups down to a cheaper model, reserve the expensive one for genuinely hard cases), that’s a cost reduction you can model explicitly as a percentage shift in your bucket mix, separate from raw volume growth. If you’re planning to loosen it (giving the coding assistant more autonomy, letting agents take more steps before returning control), model that as an increase in your average tokens-per-request for that bucket, not as generic volume growth.
Prompt caching and context reuse change the math, not just the discount
If your workload re-sends a lot of the same system prompt, tool definitions, or retrieved context across calls, prompt caching (offered in some form by most major API providers now) can cut the effective input token cost for the repeated portion substantially. But it only helps if your calls actually hit the cache, which depends on request patterns you control: keeping the cached prefix stable, keeping calls close enough together in time that the cache hasn’t expired, and structuring your prompt so the reusable part comes first and the variable part comes last.
When you forecast, don’t just apply a flat discount percentage to your whole input token line. Estimate your cache hit rate per bucket, because it varies a lot: a chat session with a stable system prompt might hit the cache on almost every follow-up turn, while a one-shot classification job with no repeated structure won’t benefit at all. If you’re not currently using prompt caching and you’re planning to adopt it next quarter, that’s a real forecast input, but it’s a separate line item from your baseline growth, and it’s an engineering task with real integration work, not a checkbox.
Agent loops and retrieval depth inflate token count quietly
Two changes that teams make with good intentions routinely blow up a forecast without anyone noticing until the bill arrives. First, increasing the maximum number of steps an agent can take before returning an answer. Each additional step doesn’t just add one more call, it adds one more call that re-reads the growing transcript of every step before it, so the cost curve is closer to quadratic than linear as step count grows. Second, increasing the number of chunks retrieved per RAG query, or increasing chunk size, both of which directly inflate input tokens on every single call that uses retrieval, not just the ones where it mattered.
If either of these is on your roadmap for next quarter (better agent reasoning, more thorough retrieval), quantify it before you ship it: run the change against a fixed sample of representative queries, count the actual token delta per call, and multiply by your expected call volume for that bucket. That’s a small, bounded exercise, and it’s the difference between a forecast with a real number behind it and one built on a hunch that “it’ll probably cost a bit more.”
Put it together as a simple model, not a single number
The forecast that actually holds up is bucket-level, not blended:
For each workload bucket, forecast expected call volume next quarter (tied to your actual growth driver, like users or tickets), expected average input and output tokens per call (tied to your current logs plus any planned changes to context length, retrieval depth, or agent steps), and expected model mix (what share goes to which tier). Multiply volume by tokens by your per-token cost for that tier, sum the buckets, and you have a forecast with a paper trail. When it’s wrong, and it will be somewhat wrong, you can go back to the specific bucket and specific variable that moved instead of staring at one blended number with no explanation.
Set alerts on the leading indicators, not the invoice
A forecast is only useful if you catch the deviation early. Set alerts on tokens-per-request by bucket and on model-tier mix, not just on total daily spend, because total spend is a lagging indicator that shows up after the pattern has already shifted for a week or two. A sudden jump in average input tokens for your agent bucket, or a shift toward the expensive model tier, tells you something changed in the code before the invoice does.
None of this replaces watching the actual bill. It gives you a way to explain it before the next planning cycle asks you to.
If you want more breakdowns like this on the tools and pipelines behind real AI workloads, check out the rest of AI Tool Gazette.