05 of 6 · Requirement violation
The ungraded prediction
The agent decides, the decision is logged, and nobody ever measures what actually happened. It cannot get better.
A decision that cannot be wrong cannot be trusted. Most agent systems log decisions lavishly and grade none of them: no prediction at commit, no horizon, no measurement when the horizon arrives. The counterfeit version is worse — predictions auto-stapled from unrelated metrics, so the ledger looks falsifiable while proving nothing. Ungraded agents don’t learn; they accumulate confidence.
Decision tables with no prediction columns, or predictions defaulted to zero; decisions past their measurement horizon with no grade; predicted metrics the acting lever cannot move; a "measured" count that never converts into hits and misses.
Log a falsifiable prediction — metric, baseline, horizon — at the moment of decision, enforced at commit. Then grade it when the horizon arrives, and publish the hit rate with its sample size.
Part 6: falsifiability-at-commit (a decision without a testable prediction cannot be recorded), horizon-triggered grading, and grading independence. The standard’s stewards apply the rule to themselves: their own instrument caught their grading loop mis-scoring in production, the fix made the number worse, and the record kept both.
Official reference: The accountability loop (Part 6) — related: AP-2, AP-9 — the standard’s normative catalog defines ten anti-patterns; this page is the plain-English door into it.
Ungraded — FAQ
Our agents’ decisions are too qualitative to predict numerically.
Then the decision isn’t ready for autonomy — that’s the discipline, not a limitation. Every lever worth pulling moves something observable: a rate, a count, a duration, a state. A decision whose effect can’t be named in advance can still be made — by a human, or flagged as ungradeable with a reason. What it can’t be is silently autonomous.
What does grading buy us in practice?
A hit rate with a sample size — the only honest basis for giving an agent more rope. It also catches subtle corruption: grading against missing data, coerced baselines, auto-satisfied targets. The standard requires the sample size to travel with the number everywhere, because a hit rate without its n is how this disease markets itself.
Does your fleet have this one?
The free Reality Check probes for this pattern against your own record — read-only, aggregate-only, no signup to read your result.
Run the Reality Check →