Evaluation

There is no separate test corpus to author. Value-shaped eval cases and certified answers were near-isomorphic artifacts, so they merged into one: every active certified answer is a regression test, and certifying an answer and writing its test are the same act. The suite runs on demand (dst test), as a gate inside dst apply, and on a standing cadence.

The corpus is the suite

For each active certified answer, the stored SQL is the oracle (services/evals/certified_suite.py):

1. Execute the certified SQL, read-only, both executions capped at 1000 rows — certified answers are small verified aggregates and fit by construction.
2. Re-ask the question through the real generation path, with certified matching disabled by construction (testing the serve bypass would trivially pass) and the answer under test excluded from its own exemplars — with parity it would match itself at ~1.0 and the suite would grade copying (services/runtime/assembly.py:312).
3. Execute what generation produced and compare executed results: an EXCEPT-style set diff, column-order-insensitive when both sides name the same columns, with a 1e-9 tolerance on single scalars (aggregation order alone can wiggle a DOUBLE's last bits).

Both sides run back-to-back against the live warehouse, so there is no snapshot to drift from — the oracle is the certified SQL run now. A diverging case re-runs generation once before counting (nondeterminism damping; flapping that survives the re-run is signal). A divergence prints both executed values and the choice it forces: fix the definition, or re-certify/retire the answer (services/evals/service.py:301). A green answer is a re-verification: its bindings re-stamp to the current shared-asset hashes.

Generation inputs come from the same assembly seam serving uses — retrieval, skills merge, profile enrichment, exemplar folding, generator tiering — so the suite grades the pipeline production actually runs, and each result records what was assembled, so a leaner-than-production run (say, no embedder configured) is visible, never silent.

Running it

dst test            # every published lens
dst test finance    # one lens
dst test --all      # explicit form of the default

In-process against the configured database, like migrate — no server, token, or URL needed. Always the full active corpus (the apply gate scopes to what a push touches; this verb is the cron/CI sweep). Exit 1 on any divergence or failed expectation (services/cli/main.py:628). A standing scheduler re-runs the same suite per published lens every DST_EVAL_INTERVAL_HOURS (default 24, 0 disables), so the accuracy trend accrues without benchmark days (services/scheduler.py).

Behavioral cases: pinning shape

What remains in lenses/<name>/evals/cases.yaml after the merge is behavioral: expect: clarify | refuse | answer, plus an optional pinned term (services/contracts/eval.py). Each case runs through the real pipeline — deterministic clarify/exclusion pre-checks included, since that is the behavior being pinned — and passes iff the response comes back in the expected shape: a clarification, a refusal with no data served, or a data answer (services/evals/runner.py). Shape is the one thing a value oracle can't express, and expect: answer measures refusal in both directions — a lens that regresses into refusing an answerable question was previously invisible unless that question was certified. Approved behavioral cases run alongside the certified suite in dst test and in the gate. (Legacy value cases with expected SQL are not scored anywhere — certified answers are the regression suite; dst evals migrate converts them into certified answers.)

The gate on publish and apply

eval_gate: off | warn | block per lens, default block — a certified answer gates by default, the way a dbt test blocks by default; a fresh lens with nothing certified publishes with a loud "gate SKIPPED" line, never a refusal (services/contracts/lens_config.py). One shared check runs inside interactive publish and inside dst apply — apply cannot bypass it (services/project/apply.py:312). A blocked publish returns 409 with the score, the previous score, and the failing cases (services/api/mgmt_lenses.py:611); under warn the same findings surface loudly and publish proceeds.

On apply, test selection is the staleness system: only active answers whose stored bindings disagree with the push's asset hashes run — cheap by construction (services/evals/service.py:321). A certified divergence under block is an error in its own right, not merely a score change, and the abort is blue/green and atomic: the whole apply rolls back, prior versions keep serving, and the staged eval run and binding re-stamps roll back with it — a rejected apply can never lower the gate's baseline for the next one (services/evals/service.py:431).

The answers a push itself lands are stamped with fresh bindings, so the staleness selection never picks them — instead, certifying unit-tests itself: an answer the apply creates or SQL-re-authors runs through the same suite in the same apply, generation against the just-stored oracle (services/project/apply.py:695). A divergence here is an alert, never a block: divergence at certification time can be the point — the certified answer overrides generation — so the warning names both executed results and asks you to re-check the oracle if override wasn't the intent. The self-test is unconditional — eval_gate governs the publish gate, never this; applying a certification unit-tests it and alerts, period. Provenance- and status-only edits verify nothing new and are not re-tested; the only answers the self-test cannot reach are those generation cannot serve (no smart-tier model, no connector) — they keep the landed untested nudge toward dst test.

A configured gate that cannot score — no smart-tier model resolves, nothing to run — stands down loudly, with the reason in the publish/apply output (services/evals/service.py:274): a seatbelt that silently stands down is worse than none. One starvation refuses to stand down at all: under eval_gate: block, an apply that empties the active certified corpus — retiring the last active answer, or deleting its file entry — aborts instead of publishing over an emptied corpus; keep at least one active answer, or set eval_gate: warn/off in lens.yaml and re-apply (services/project/apply.py:408). Under warn the same ≥1→0 transition publishes but warns, in the apply row itself — even when approved eval cases keep the gate scoring, the certified gate now has nothing left to test. The scope is exactly the ≥1→0 transition: a lens that never had active answers — a fresh lens doing its first applies under block — is untouched.

Scored three ways

Everything dst measures scores three-way — correct / wrong / declined — with wrong-rate as a co-headline, because a wrong answer is worse than no answer. Declining is the correct response for a question whose data does not exist; a confident number there is a hallucination by construction (services/benchmark/grading.py:54).

The curated layer is not an answer cache: in held-out internal runs it lifted accuracy on questions whose SQL is not in the certified library — the definitions and exemplars generalize beyond the questions they were written for.

The standing rule of the measurement culture: if real numbers break a story beat, the narrative changes, not the numbers.