Most articles about AI performance talk in theory.
We put our theory to test
Over the past six weeks, our team has been monitoring our agentic AI system that helps mortgage brokers answer policy questions, look up live rates, and generate client cover letters. The system runs on a single NVIDIA H100 GPU, serving broker queries every day.
Here are some of the metrics we have been measuring
Time to First Token (TTFT) - Lower TTFT makes the application feel more responsive.
End-to-End (E2E) Latency - Determines the overall user wait time.
Queue Wait Time - High queue time indicates overloaded infrastructure or insufficient capacity.
Prefill Time - High prefill slows response start, especially for long prompts.
Decode Time - Usually the biggest contributor to latency for long responses and our focus area.
Inter-Token Latency (ITL) - Lower ITL makes responses stream faster and feel smoother.
Token Throughput - Higher throughput means the system can serve more users efficiently.
Prefix Cache Hit Rate - Higher cache hits reduce prefill time and improve latency while lowering GPU compute (closely monitored).
HTTP Requests/sec -Indicates system load and helps correlate traffic spikes with latency.
One of the most useful lessons when debugging LLM performance is that latency metrics only make sense when viewed together.
Looking at the recent inference dashboard:
The interesting part? Since ITL was flat, the long end-to-end latency wasn't caused by slow decoding. It was caused by longer generations.
The dashboard tells a clear story:
There is a tendency in AI infrastructure to reach for hardware upgrades when performance degrades. Bigger GPU. More memory. More replicas.
Our data suggests the first question should be different: is the system doing work it has already done?
The KV cache is not a niche optimisation. It is the mechanism that determines whether an AI system scales gracefully or collapses under concurrent load. And it fails silently, not with errors, but with unnecessary computation that looks, from the outside, exactly like insufficient hardware.
A good reminder that optimizing LLM systems isn't always about faster GPUs or more replicas. Sometimes the biggest latency win comes from understanding where time is actually being spent.
The system described in this article is built on vLLM, DSPy, OpenSearch, FastAPI, and an NVIDIA H100. Monitoring is via Prometheus and Grafana. All figures cited are from production dashboards captured in July 2026.