The eval harness
A golden corpus that runs against the port, so the same cases execute on the mock adapter and on recorded cassettes. A suite that only ever runs against a simulator we wrote ourselves cannot detect our own misunderstanding of a vendor.
#Four levels
| Level | What it can catch |
|---|---|
L1 | Deterministic lookups. Is the payer rule read correctly at all. |
L2 | Tool use. Does the agent call the right things and respect denials. |
L3 | Journey. Does an end-to-end request reach the right outcome. |
L4 | Adversarial. Does it hold when someone is trying to break it. |
L5 | Absent, and declared absent. There is no production traffic to shadow, and a fabricated L5 would be worse than none. |
#Cases are sliced
An aggregate that hides a payer or an urgency is how a regression ships. Results break down by level and by payer, and a slice with a small n is labelled count only rather than being given a percentage that implies precision it does not have.
#Paired gates
Recall and over-escalation are reported together and gated together. Optimising one alone is trivial and useless: a system that escalates everything has perfect recall and is worthless, and one that escalates nothing has a perfect over-escalation rate and is dangerous.
#The harness has found real defects
- No identity reconciliation between the EHR and Practice Management - rated catastrophic, and missed by 236 unit tests. The case was written to inject the fault and found nothing stopping it.
- A PHI leak in the masked list view - patient names and medical record numbers, because the leak test's needles were hand-written guesses. Deriving them from the seed data found it, then immediately found a second leak in
groupNumber. - A defect in itself. An early version shared one world across the corpus, so the first journey's authorization made every later journey look like a duplicate. A harness defect is a P1, not a footnote.
#Determinism
Injected clock, seeded id sequence, no wall time and no randomness. That is what makes the variance report meaningful - any spread it finds is real behaviour rather than scheduler noise. Each case gets its own world.