Evaluating a clinical agent: what a real harness measures

3 min read

Asking whether an AI agent is accurate is not a useful question in this domain. Accurate at what, on which patients, and with what happening when it is wrong. An evaluation harness that answers only the first is how systems reach production looking better than they are.

#The levels

Level	Asks	Why separately
L1	Deterministic lookups. Is the payer rule read correctly.	If the system cannot read a rule reliably there is no point measuring anything built on it. This level should be near perfect and a regression here is serious.
L2	Tool use. Does the agent call the right thing with the right arguments.	Most agent failures are not reasoning failures. They are calling the wrong capability, or the right one with a malformed argument.
L3	Journey. Does an end-to-end request actually reach a decision.	Components can each pass and the whole still fail. This is the level that catches work being silently dropped between stages.
L4	Adversarial. Does it hold when someone is trying to break it.	Cross-practice probes, instructions hidden in untrusted content, and cases engineered to force a confident wrong answer.

L5 is declared absent rather than faked

A fifth level, production shadow evaluation against real outcomes, is the one that would tell you the most. It requires real production volume over time, and until that exists the honest thing is to record it as not present rather than to quietly report four levels as if they were five. An evaluation suite that hides its own gaps is measuring the wrong thing.

#Severity weighting

Not all errors are equal and an aggregate accuracy figure treats them as if they were. Asserting that authorization is not required when it is can mean an unauthorized procedure. Asserting it is required when it is not costs some administrative effort. Both are wrong; one is much worse.

A harness that weights by severity will show a lower headline number than one that does not, and it will be the more useful number. Any evaluation where every case counts once is easy to improve by getting better at the cases that do not matter.

#Paired gates

Single metrics are gameable, and the gaming is usually unintentional. A system tuned to never miss a required authorization can reach that by flagging everything, which is useless. A system tuned to minimise unnecessary escalation can reach that by escalating nothing, which is dangerous.

The fix is pairing: a floor on catching what matters together with a ceiling on over-escalation, and both have to pass. Neither can be satisfied by collapsing the behaviour.

#Slices

An aggregate hides the cases you most need to see. Performance should be reported per payer, per practice and per procedure category, because an agent that is excellent overall and poor on one major payer is, for a practice whose patients are mostly on that payer, simply poor.

#Honest numbers

Every number carries its n: Ninety-four per cent on seventeen cases is not a result, it is an anecdote with a decimal point.
Variance is reported: Model-in-the-loop systems are not deterministic. A single run is a sample. Repeated runs with the spread stated tell you whether a change was real.
The corpus is versioned: A score against an unknown corpus version compares nothing. Improvement is only meaningful against a fixed set.
Failures are enumerable: A scorecard that cannot produce the specific cases it failed is not auditable, and the cases are where the actual information is.