Estimate LLM Cost and Latency Without Fooling Yourself

Model the request you actually expect

A cost estimate begins with a representative request, not a model price screenshot. Measure or estimate input tokens, output tokens, requests per active user, retries, and the proportion of requests that use expensive paths. Include retrieved context and tool output because they often dominate the input budget.

Write down a low, expected, and high workload scenario. A support assistant used ten times per employee has a different risk profile from a public chat interface used unpredictably. The point is not to forecast perfectly. It is to identify which assumption changes the decision.

Separate model cost from system cost

Token charges are only one part of an AI feature. Embeddings, vector storage, reranking, document ingestion, observability, queueing, and human review can be material. Keep the model estimate separate so it remains honest, then list the additional operating costs as a second line item.

Do not hardcode provider prices into internal planning documents without a review date. Enter current input and output rates directly from the provider you intend to use. This keeps the worksheet useful when a provider changes pricing or a team changes models.

Use a p95 latency target

Average response time hides the request that makes a user wait. Choose a p95 target for the user journey, then allocate time to retrieval, reranking, model inference, tool calls, application work, and rendering. The budget is a design constraint, not a promise.

If the planned steps already exceed the target, solve that before implementation. Possible levers include a smaller context, fewer sequential calls, precomputation, a faster model on simple paths, streaming, or a narrower interaction. Adding a loading animation does not fix an overloaded critical path.

Model retries and tail behavior

Retries protect a user from temporary provider failures, but they multiply both cost and tail latency. Include an assumed retry rate in the estimate, then replace it with observed data after launch. Use timeouts and fallback behavior so a retry loop cannot silently consume the whole budget.

For multi-step agents, sum the critical path rather than every possible branch. A parallel retrieval and profile lookup should be represented by the slower branch plus the joining work. A serial tool chain should be represented as serial. That distinction changes where optimization work belongs.

Review estimates after instrumentation exists

Planning numbers should expire. Once the feature has traffic, compare estimated versus observed tokens, cache rates, retries, p50 and p95 latency, and abandonment. Update the budget with measured distributions rather than treating the original spreadsheet as a benchmark.

A cost or latency surprise is often a product-design signal. If a workflow needs a huge context and three tool calls for every question, the right answer may be a more focused feature, not only a cheaper model.

Checklist

• Input and output tokens include retrieval context and tool output.
• Low, expected, and high-volume scenarios are visible.
• Provider prices have a review date.
• The target is p95 end-to-end latency.
• Retries and fallback behavior are part of the model.