A generated dashboard can look right and be wrong in a dozen ways. The chart’s columns don’t match what the chart reads. The query never ran. “No results” came from loosening the question until it changed. The numbers were true on Tuesday. And the same question gave a different query depending on who asked.

The fix isn’t a smarter model. It’s making each of those failures impossible to ship quietly. This is how the Make side of my assistant does it. The overall shape, an engine per capability with workflows that plan, gather evidence and write, is in tools, workflows, engines.

Born in the consumer’s shape

Whoever renders the output publishes its shape, and that shape goes into generation as a hard constraint:

// published by the dashboard service and fetched live; the generator keeps no copy
type WidgetContract = {
  types: { type: string; dataShape: "table" | "series" | "tree"; queryShape?: { dimension: string; measure: string } }[];
  specSchema: JSONSchema;
};

function outputConstraint(type: WidgetType): string {
  if (type.dataShape === "tree") return "The query MUST end in `RETURN tree`.";
  const { dimension, measure } = type.queryShape!;
  return `Project exactly two columns: \`AS ${dimension}\` and \`AS ${measure}\`.`;
}
  • No type outside the contract. The planner’s preferred chart survives only if the contract has it and the query’s shape fits. Otherwise a compatible type is used, or the widget is dropped.
  • Constrain before, never rewrite after. The shape is in the generator’s prompt. Nothing renames columns afterwards.
  • Validate where you generate. The consumer’s JSON Schema runs before the payload is emitted. Drift is an internal error, never “your request can’t be met”.

Empty needs proof

A generator under pressure ships a query nobody ran, or loosens filters until rows appear and answers a different question. So nothing leaves without proof:

type Run = { query: string; rows: number | null };     // null: it couldn't run

function proofOf(runs: Run[], feedback: Feedback): "rows" | "empty" | null {
  if (runs.some((r) => (r.rows ?? 0) > 0)) return "rows";
  const confirmedEmpty = runs.every((r) => r.rows === 0 && joinsRelatedRecords(r.query) && feedback.clean(r.query));
  return confirmedEmpty ? "empty" : null;
}

No proof gets one repair turn: “Diagnose, do not loosen. Never broaden filters or swap entities just to make rows appear. ‘Could not check’ is not a real zero.”

Then the exit matters. Running out of rounds with nothing held is remembered as unanswerable, so the next ask can say so without trying. Running out of time is remembered as nothing, because a slow afternoon isn’t evidence about the question. A stored answer’s trust moves only on a run that actually ran (see only independent evidence raises trust).

When the writer adds prose and links on top of the evidence, it may only cite what the evidence produced. Every value carries its origin, so a number the AI couldn’t ground never passes as real.

Plan once, read live

A page the assistant lays out for an account, say “orders health” with six chosen cards, takes minutes to plan. It can’t be re-planned on every view. But a stored page of numbers is wrong by tomorrow.

So the plan is stored and the numbers never are:

type Plan = { cards: { title: string; query: string; render: "stat" | "trend" | "table" }[] };   // queries, no values

type BoardState =
  | { status: "planned"; plan: Plan; inputs: string }
  | { status: "planning"; plan?: Plan }               // the old plan still serves
  | { status: "failed"; plan?: Plan; failedAt: number };

const inputs = hash({ v: BOARD_VERSION, objectives, dataFingerprint });   // the data changed → replan

Every view runs each card’s query fresh. When the inputs hash moves, the old plan keeps serving, marked planning, while one deduplicated background replan runs with its progress muted. Pins and hides are applied at read time, so a replan never loses them. It’s the same input-hash idea knowledge uses for derived facts.

A deterministic floor under every model call

A banner like “This feature needs a shipping integration” shouldn’t block the page while a model words it nicely. It shouldn’t be blank when the model is slow either.

async function bannerFor(title: string, readiness: Readiness, deadlineAt: number) {
  const floor = plainBanner(title, readiness);                 // built from what's in hand
  const key = hash({ type: "banner-v1", accountId, title, state: stateOf(readiness), prompt: SYSTEM_PROMPT });
  const cached = await cache.get(key);
  if (cached) return cached;
  const written = smallModel(SYSTEM_PROMPT, { title, readiness }, { temperature: 0, maxTokens: 150 })
    .then((text) => (cache.set(key, text, SEVEN_DAYS), text));  // kept even if we stop waiting
  return (await beforeDeadline(written, deadlineAt)) ?? floor;
}

The prompt text is part of the cache key, so editing the prompt invalidates old answers with no version bump. The same idea scales into a ladder of producers, each using the cheapest thing that can be honest:

Producer Uses
suggested goals stored readings and a score; no model
record summary page stored knowledge; every number read live on render
matching text to a goal embeddings, cosine ≥ 0.45 (unrelated sentences land around 0.2 to 0.35)
starter questions a small model with a forced tool
a dashboard the big model: planning and grounding

Personalize the planner, not the generator

It’s tempting to feed “what this person usually works on” into the query generator. Then the same question produces different queries for different people, and generation drifts toward whatever someone opened last week.

So personalization stops at the planner:

  person's focus ("Refund handling")  ──▶  planner: which cards, which questions      ← personalized
                                                 │ questions
                                                 ▼
                     generator: question + the account's facts + proven answers + output shape   ← never

Focus reaches the planner as activities, not schema names, so it can’t leak into generation through a table name. Queries are a property of the question and the account, and a proven answer is proven for everyone who asks it.

Why it works

Each failure that used to ship quietly now has a place it gets caught: the shape at generation, the proof before return, the staleness at read, the latency at the floor. The model still does the creative part. It just can’t be the last line of defense.

Part of a series: The UI is the engine, An agent loop that always answers, What the assistant knows.