Skip to content

The correction loop

A wrong answer is an incident, not a ticket that dies in a queue: it should end as an applied fix plus a regression case proving it stays fixed.

A wrong answer becomes a ticket

Two entry points open a review ticket:

  • A caller flags an answer — POST /v1/reviews, or the MCP tool send_for_review(request_id, note?, corrected_sql?). The note and SQL are recorded as a correction, not just a flag.
  • The lens flags itself — the auto_review policy (off | unverified | partial in lens.yaml) opens tickets on low-confidence answers in the background, so the queue watches the lens, not just its callers.

On open (services/reviews/service.py:80): if the caller supplied no correction, one is pre-filled from a failed adversary challenge in the trace; a ≥0.90-similar certified answer's SQL is handed to an AI judge as a reference; and the judge policy runs — a confident approve auto-resolves; everything else escalates to a human. No LLM configured means straight to human. The artifact under review is the full trace — question, SQL, rows, answer, verification — not the conclusion.

Humans rule from the dashboard or the CLI:

dst rule <ticket> --verdict approve|changes|reject [--certify]

--certify promotes the corrected question→SQL pair to the certified library in the same act.

A ruling becomes a patch

POST /mgmt/reviews/{ticket}/draft-patch turns a ruling plus its correction into a patch candidate (kind definition | instruction | certified | reprofile | reindex). The AI drafts; the human approves, never authors — there is no auto-merge. On approval (services/api/reviews.py:240), what happens depends on where the truth lives:

  • Governing state (a certified pair) is applied directly and survives the next apply.
  • Authored truth (a definition, a lens's instructions) is compiled from your files on every dst apply — so the approval returns a proposed file (path, content, diff) with live: false. Commit it and apply; the fix is not in effect until then. Writing it into the server directly would only look applied — the next apply of the same unchanged files would silently revert the ruling.

Either way, the correction is minted as an eval case with status candidate, never auto-approved (services/api/reviews.py:216): a human promoted the fix once, a human promotes it as a test too. Once promoted, it re-runs in the eval gate on every dst apply — the same class of error cannot ship twice unnoticed. The fix's effect is measured by that gate on the lens that actually landed.

Three producers, one rail

The ticket is not the only source of patches. Corpus distillation (mining the request log for recurring shapes worth becoming instructions or definitions) feeds the same candidate → human approval → file rail, and on the paid tier so do continuous-audit findings. One approval discipline, whatever produced the patch — and bootstrapping from history seeds the definitions before the loop even starts.

The payoff is structural, not statistical: a corrected question is served from its certified SQL from then on — deterministic, and cheaper and faster than generation — and its eval case re-runs on every apply, so the same class of error cannot ship twice unnoticed.