ecko
Synthetic data and evaluation harnesses
Seeded, deterministic mock mode is a requirement for an eval harness that means anything.
The problem
Model evals that call live APIs are expensive, non-reproducible, and impossible to run in CI. Teams snapshot responses once, then bit-rot sets in when the schema changes. Stochastic sampling makes "accuracy went from 91% to 89%" meaningless noise.
Why Ecko
- Deterministic mock
- Same seed, same typed outputs, every run. Compare prompt versions without confounding provider variance.
- Contracts as metrics
@ensuresclauses become pass/fail criteria beyond exact string match - useful when the right answer is semantic.- ecko test
- Eval cases live next to production code.
ecko testdiscovers them; mock mode is forced in the test runner.
In practice
test "extraction meets contract" {
out = extract(fixture_note)
assert len(out.meds) > 0
}
Try it on your workload.