Design for AI Latency That Varies Wildly

AI Interfaces · ai, loading, latency

Updated 2026-08-04

Expect the same request to take four hundred milliseconds once and eight seconds the next time. Three separate things move independently: whether the model is already warmed up, how busy the service is, and how much text is being generated.
Never design the waiting state for the fast case. Tuning the animation and the copy to a response you got on a quiet afternoon guarantees it looks broken under load.
Stream the output as it is produced. Generation happens one piece at a time and cannot be parallelized, so the first words are available long before the last ones, and showing them changes the perceived wait completely.
Distinguish the phases in the interface, because they behave differently:

Say when the request is queued rather than slow. High demand right now is information, and an unchanging spinner is not.
Keep the interface usable during a long generation. Scrolling, copying what has arrived, and stopping the response all have to work while it is still running.
Give a stop control that keeps the partial output. Users abandon long responses constantly and the text so far is often enough.
Never time out into an empty state. A generation that failed after producing three paragraphs should keep those paragraphs and offer to continue.
Set expectations by task rather than globally. A one-line rewrite and a full document analysis are different waits and should not share a loading pattern.

AI latency is unpredictable by nature, so the interface has to be built for the slow case and pleasantly surprised by the fast one.