Streaming
Streamed text should be revealed word by word at a steady pace, not rendered in the lumps it arrives in.
The problem
Model output comes over the wire in uneven chunks: three characters, then forty, then a pause. Render it as it lands and the text lurches. Words get cut in half at chunk edges, and the answer feels jittery even when it’s fast.
What I concluded
The text that has arrived and the text that’s shown are two different things. Every word renders right away but starts hidden, and an animation frame loop uncovers them at a steady rate.
The rate follows the backlog. There’s a floor of about 28 words a second, and when more is waiting, it speeds up to four times the backlog per second. So any backlog clears in about a quarter second, and the reveal never falls far behind.
const tick = (now) => {
const backlog = spans.length - revealed.current;
const rate = Math.max(FLOOR, backlog * DRAIN);
budget += ((now - last) / 1000) * rate;
last = now;
while (budget >= 1 && revealed.current < spans.length) {
spans[revealed.current++].removeAttribute("data-pending");
budget -= 1;
}
frame = requestAnimationFrame(tick);
};
The word still being typed is held back until a space arrives. That’s what stops half words from flashing.
The pacing is a visual nicety, so it goes away for anyone who asked for less motion: with reduced motion on, every arrived word shows at once. Screen readers shouldn’t get it either. The answer is marked busy while it streams, so assistive tech waits for the finished text instead of every word.
Rules
- Separate arrived from shown. The network decides what’s arrived. The reveal loop decides what’s visible.
- Pace by backlog. A fixed rate either crawls or falls behind. Scaling with what’s waiting does neither.
- Never show half a word. Hold the last word until it’s complete.
- Respect reduced motion. No pacing, just the text.
- Not everything is prose. A table or code block should appear as one unit once it’s complete. Half a table is worse than a short wait.
What you get
Answers read smoothly no matter how the bytes arrive, and they still finish close to when the stream does. On a very long answer, fold revealed words back into plain text so the loop only walks the pending tail.
Related: Status labels, Event stream