Agent loops fail in boring ways. The route timeout cuts a turn off mid-exploration, and the person sees nothing. An answer hits the output limit and stops mid-sentence. One tool throws and takes the request with it. A composer waits on its slowest part. And the tracing that helps you debug all of it quietly becomes a second copy of customer data.

None of these need clever fixes. They need limits that land softly. This is how the loop behind my assistant handles each one.

Every exit is an answer

The outer loop has hard caps, and each one steers toward an answer instead of stopping:

const MAX_ROUNDS = 8;
const FINAL_TURN_MARGIN_MS = 15_000;     // enough for one last model call to write an answer
const MAX_OUTPUT_TOKENS = 8192;          // a full assessment runs ~3k words
const TURN_BUDGET_MS = 180_000;          // inside the request's deadline

for (let round = 1; round <= MAX_ROUNDS; round++) {
  if (deadlineAt - Date.now() < FINAL_TURN_MARGIN_MS && !finalTurnForced) {
    appendUserText(messages, "You are out of time. Give your best final answer now from what you have.");
    finalTurnForced = true;
  }
  const reply = await model(messages, { tools });
  if (reply.stop === "max_tokens" && !continued) {
    appendUserText(messages, "Continue exactly where it stopped, without repeating anything.");
    continued = true; continue;
  }
  // …run this round's tool calls in parallel; a throw becomes { error: message } as its result
}
Limit What the person gets
Time running out a forced final answer from what’s been gathered
Output length one continuation, then a visible “(cut off at the length limit)”
An empty reply one nudge: “call a tool or give your final answer”
A tool throws the model sees { error } and works around it, and the step shows “failed”
  • Nudge, don’t strip. Some model APIs reject a request without tools once the transcript holds tool calls. So the final turn keeps the tools and asks for words.
  • Append, don’t add a turn. Two user messages in a row are invalid, so a nudge goes onto the trailing user message.
  • One of each. At most one continuation and one empty-turn nudge per turn, so recovery can’t loop.
  • Every tool call gets a result, including an unknown tool, bad input or a rejected UI call.
  • Cache the static part. The prompt cache point sits after the fixed system prompt, and the transcript gets two rolling points, within the API’s limit of four.

Count tokens in the gray zone

A prompt over the context window is a 400 that retrying won’t fix. Counting every request with the provider’s counter adds a network call to every turn. So estimate for free, and count only when the estimate is too close to call:

const CHARS_PER_TOKEN = 3;           // conservative: shrink a little early, never late
const SAFETY_MARGIN_TOKENS = 5_000;  // tool specs and tokenizer drift
const LOWER = 0.7, UPPER = 1.3;      // the gray zone, as a share of the budget
const MAX_COUNT_CALLS = 3;

async function fits(prompt: Prompt, budget: number) {
  const estimate = Math.ceil(chars(prompt) / CHARS_PER_TOKEN);
  if (estimate < budget * LOWER) return true;
  if (estimate > budget * UPPER) return false;
  if (counterFailed || calls >= MAX_COUNT_CALLS) return estimate <= budget;
  try { calls++; return (await countTokens(prompt)) <= budget; }
  catch { counterFailed = true; return estimate <= budget; }      // sticky for this request
}

When it doesn’t fit, shrink in a fixed order:

  1. This turn’s page context: verbatim, then compact, then summary.
  2. Page contexts from earlier turns.
  3. The oldest message pairs, keeping at least two.

The system prompt, tools, the latest question and the last answer are never touched. Past the second tier, the trace gets a promptBudget attribute, so a shrinking page shows up on a dashboard before it shows up in bad answers.

Narrow the inner loop

Inside the query generator there’s a smaller loop. With every tool on offer every round, it was slow: a round on the big model took about 4.6s, and three runs in four took 2 to 10 rounds while averaging under 4 tool calls. The latency wasn’t exploring. It was resubmitting.

So the menu narrows as the loop goes:

const MAX_ROUNDS = 5, PROBE_BUDGET = 4, EXPLORATION_ROUNDS = 1;   // +1 after a repair
const FINAL_TURN_MARGIN_MS = 20_000;                              // ~2x the observed p95 tool-turn latency

function menuFor(round: number, s: LoopState): RoundConfig {
  if (round === 1 && s.stronglyGrounded) return { tools: [submitDraft, cantAnswer] };            // draft first
  if (s.forceSubmit || round > EXPLORATION_ROUNDS + s.repairs)
    return { tools: [submitDraft], model: SMALL_MODEL };                                          // only choosing now
  return { tools: [submitDraft, cantAnswer, ...exploreTools] };
}
// every round: toolChoice "any", so the model can't stall in prose

Gates only close, a partly valid submission is done, and the eval scores turn_efficiency = max(0, 1 - (rounds - 1) * 0.2) next to relevance. A change that trades rounds for accuracy shows up in the same experiment.

Carve deadlines from the parent

One slow widget, four rounds and 31.6 seconds, held a whole dashboard. Timeouts were relative, so they added up past the request’s budget, and a timeout was an exception that took its siblings down.

So deadlines are absolute and carved from the parent’s, with a reserve kept back for assembly:

const deadlineAt = Math.min(parent.deadlineAt, Date.now() + workflow.deadlineMs);

const COMPOSE_RESERVE_MS = 15_000;      // compose, validate, maybe replan
const WIDGET_BUDGET_MS = 30_000;        // the median widget takes ~6s
const widgetDeadline = Math.min(Date.now() + WIDGET_BUDGET_MS, deadlineAt - COMPOSE_RESERVE_MS);

const result = await Promise.race([build(spec, widgetDeadline), sleepUntil(widgetDeadline).then(() => "late")]);
if (result === "late") failures.push({ widget: spec.title, failure: "took too long to build" });  // the planner sees it

Each producer’s deadline matches its cost: 5s for one that needs no model, 10s for a small-model call, 60s for a query, 240s for a dashboard. Depth follows the caller too. The same dashboard producer plans up to two rounds when a button asked, and one when a chat turn asked.

Traces keep shape, not values

Tracing is the best debugging tool an AI product has, and it fills up with customer data: every tool result and every transcript the model reads. Regex PII redaction misses business data like order totals, names and notes.

So the data class lives on the tool, and logging follows it:

new Tool({ name: "find-customer", customerData: true, /* … */ });

const KEPT_STRINGS = new Set(["query", "question", "status", "source", "kind", "title", "description", "error"]);

function countsOnly(value: unknown, key?: string): unknown {
  if (Array.isArray(value)) return `[${value.length} withheld]`;
  if (typeof value === "string") return key && KEPT_STRINGS.has(key) ? value : "[withheld]";
  if (value && typeof value === "object")
    return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, countsOnly(v, k)]));
  return value;
}

The model’s span only knows which tool produced a result through the call’s id. So the logger collects the ids of data tools from the transcript and withholds their results there too. If shaping a value throws, the whole value is withheld. “Returned 0 rows” and “the query was X” survive, and those are what you debug with. Grading for correctness happens where the data already lives.

Why it works

None of the limits got softer. They just all end somewhere useful: an answer, a partial answer with a note, a widget marked “took too long”, a trace you can read without leaking anything.

Part of a series: The UI is the engine, Generation you can check, What the assistant knows.