Design for a p95 target
Latency Budget Planner
Allocate an end-to-end response target across retrieval, reranking, model inference, tool calls, and rendering.
Most AI latency problems are discovered after the model call is already slow. A budget shows which system boundary must improve first.
How it works
Remaining budget = target p95 latency - sum of planned step latencies.
Limitations
- Use p95 measurements, not only averages, when production data exists.
- Parallel calls should be modeled by their critical path, not summed blindly.
- Network variance and queueing can exceed a lab estimate.