How should a large language model API budget be estimated?
Do not estimate an LLM API budget as a guessed price per chat. The client provides representative tasks, expected users, and an acceptable result. Wavesteam instruments a prototype to measure input, cached input, output, tool calls, failed retries, and peak load. We then apply each candidate supplier's current pricing and present low, expected, and high scenarios, their boundaries, and a recommendation.
A model name is only one line on a price sheet. The bill can also depend on context and output length, cache hits, web or file-retrieval tools, vector storage, voice and image inputs, service tier, concurrency, and retry rate. Two “support answers” can cost very different amounts when one sends a ten-turn transcript and the other sends a concise state summary. A durable estimate therefore uses a recalculable method rather than copying prices that will become stale.
When defining budget, scope, and cost assumptions, also compare How should an annual SMS verification budget be calculated? and Can Wavesteam estimate a project before the requirements are complete?; the linked guidance adds context that should be considered in the same decision.
Map a business task to every billable event
A useful monthly model is:
Monthly cost = Σ[(uncached input tokens × input rate) + (cached input tokens × cached rate) + (output tokens × output rate) + tool charges + storage charges] + retry and fallback cost.
For every rate, record the model version, currency, billing unit, regional or tax treatment, and retrieval date. Daily requests multiplied by 30 is a traffic assumption, not a complete calculation. Include timeout retries, moderation, embeddings, reranking, and fallback calls when they are part of the production chain.
| Strategy | Cost shape | Effect and risk | Suitable evidence |
|---|---|---|---|
| High-capability model for every request | Higher input and output cost; simple engineering | May improve long-tail tasks but wastes capacity on simple work | Early exploration with few samples to establish a quality ceiling |
| Route tasks across models | Small model handles classification or extraction; complex tasks escalate | Routing must be evaluated; misclassification damages quality | Clear task tiers and meaningful volume |
| Cache stable prefixes or results | Repeated input or answers can cost less | Bad keys, freshness, or privacy handling can return stale content | Long stable prompts, versioned knowledge, measured repetition |
| Asynchronous batch work | May use lower batch pricing but adds delay | Must handle partial failure; unsuitable for live interaction | Overnight classification, summaries, or embeddings |
| Self-host an open model | Cost moves to accelerators, serving, operations, and upgrades | Greater deployment control but not “free API” | Hard hosting constraints or stable use proving better total cost |
OpenAI's official API pricing is an example of separate input, cached-input, output, and built-in tool charging. Its Batch API guide describes a separate completion window and discount. If another supplier is selected, replace those assumptions with that supplier's official price sheet, contract, and billing export.
Build three scenarios from real samples
Wavesteam samples short questions, long documents, tool use, refusals, and failure retries from authorized client material. Coverage follows the important task types and risky exceptions, not an arbitrary sample count. For each type we record P50 and P95 input and output tokens, tool calls, success, and human evaluation. Trial logs or the operating plan then produce low, expected, and high monthly scenarios, with peak load and supplier limits shown separately. The client does not need to count tokens or design a concurrency model.
Cost must be paired with quality. A support workflow can measure first-contact resolution, escalation, and factual error. Document extraction can measure field accuracy and human review minutes. Code or report generation can measure acceptance and rework. A cheaper call that creates more manual correction is not a lower-cost service.
The most useful unit is model cost per successful business task: monthly model charges divided by completions that pass the agreed quality threshold. Failed, retried, and useless calls remain visible. Alerts should cover abrupt daily cost, unusual per-user consumption, growth in P95 context, and increasing fallback use.
Remove waste before lowering model capability
First remove irrelevant history and duplicate retrieval, cap unnecessary output, and stop retry loops or abusive traffic. Then evaluate versioned prompt/result caches and move non-urgent jobs to batch. Route a task to a cheaper model only when it still passes the same evaluation set; otherwise the apparent saving moves into review and complaints.
Wavesteam joins token logs, quality evaluation, and supplier charges by task ID. We separate model consumption, engineering service, and cloud resources instead of hiding them in an “AI package.” The budget also states the recommended model and routing, the volume point that would change the recommendation, overspend alerts, and shutdown conditions. For sensitive data, we additionally review the supplier's current API data controls, because caching or background modes can have different retention conditions.