Stateful or stateless
Stateless is simpler until the agent starts doing things. Then the server should hold the conversation.
The problem
Every assistant has to pick who owns the transcript.
Stateless: the client keeps the history and sends all of it on every request. The server is a pure function. That’s how most chat demos work, and it’s a good default: easy to scale, easy to test, nothing to store or expire.
It starts to break when the agent does more than talk:
- The transcript holds things the client never saw. Tool calls, tool results, ids the model used. The client ends up rebuilding a history it only half understands.
- What happened isn’t in the text. The person pressed Stop, ignored a proposed action, or sent from two tabs at once. A transcript the client assembles has no record of any of it.
- The client becomes an editor of the past. Whatever the client sends is what the model believes happened, including history that was trimmed, reordered or tampered with.
- Requests grow every turn. The whole history goes over the wire, every time.
What I concluded
Hold the conversation on the server once the agent has tools. The client sends an id and only what’s new.
stateless stateful
───────── ────────
client ── whole history ──▶ client ── id + new text ──▶
server (pure) server
load ▸ run ▸ append
POST /assistant
{ conversationId: "8f2c",
text: "Which orders are late?",
results: [{ toolUseId: "c1", outcome: "done" }] }
The demo shows what that buys. Three things that usually get lost become facts the next turn can read:
- A client call ends the request. If the agent asks the app to apply a filter, the answer ends there. The result comes with the next request, and a call it leaves out is recorded as declined, so the model never sees a call with no answer.
- Stopping is recorded. The partial answer is saved with a marker saying the person stopped it, so the next turn doesn’t treat a half-sentence as finished.
- Two tabs can’t interleave. Each append is a conditional write against the version it read. The first wins and the second is refused.
The trade-offs
| Stateless | Stateful | |
|---|---|---|
| Server | Pure, nothing stored | A store, keys, expiry |
| Request size | Grows every turn | Small and flat |
| What really happened | Whatever the client says | Exact, including tool calls |
| Branching and editing | Easy on the client | Has to be a server feature |
| Testing | Trivial | Pass in an in-memory store |
When stateless is still right
Single-shot features, like summarizing a page or drafting one paragraph, have no tools to track and no history worth keeping. Stateless is the right call there, and it’s cheaper.
Rules
- Keyed by the verified person. Nobody shares a conversation by guessing an id.
- Narrowed access starts fresh. The old transcript holds things read under wider access.
- It expires. Every stored row carries the same expiry.
- The loop only knows load and append, so the store is swappable.
What you get
Small requests, a transcript that matches what actually happened, and an agent that remembers what it did, not only what it said. The cost is a store to run and less freedom on the client.
Related: Agent loop, Event stream, Queued messages.