Query reuse
Reuse a proven query only when the new question has the same structure, never just because it sounds similar.
The problem
Caching which query answered which question saves a model call and hands back an answer you already know is good. The obvious way to match questions is embedding similarity, and it’s wrong too often.
“Customers with an order” and “customers without an order” sit almost on top of each other in embedding space. A semantic cache will happily give one the other’s answer.
What I concluded
Compare questions by skeleton. Mask every word that links to something in the data, and keep what’s left:
"open orders" → "* *"
"how many open orders" → "how many * *"
"open orders from Canada" → "* * from *"
"open orders for the spring sale" → "* * for the spring sale"
"customers without an order" → "* without an *"
The words that survive are the ones the data can’t say for you: “how many”, “top 10”, “per”, “without”, a name nothing stores. “Canada” is a value in the data, so it’s masked. “Spring sale” isn’t stored anywhere, so it stays. Two questions with the same skeleton ask for the same shape of answer.
Matching runs in order, cheapest first:
question ─▶ ground words ─▶ skeleton
│
1. known unanswerable? ───────┼─▶ say so, no model
2. same skeleton, covers ─────┼─▶ run the proven query
3. same, one value swapped ───┼─▶ substitute, run it
4. shares labels ─────────────┼─▶ a few examples in prompt
5. nothing ───────────────────┴─▶ generate as usual
Coverage has to hold both ways. Every label, filter and value the stored query uses must be named in the question, and every precise thing the question names must be used by the query. So a broader query never answers a narrower question, and a narrower one never answers a broader question.
Rules
- Similarity never decides. It can rank examples for the prompt. Reuse needs a structural match.
- Check coverage in both directions. Nothing extra, nothing missing.
- Store skeletons, not what people typed.
- Proven answers decay. They lose trust like any fact, and drop out the moment a fact they rest on breaks.
What you get
Repeat questions skip the model and get an answer that already ran. The near-miss cases, the “with” versus “without”, fall through to generation instead of getting a confident wrong answer.
Related: Knowledge store, Checkable output