| Property | Test |
|---|
| Explicit | Written down before the cases arrive |
| Rule-based | A specific condition fires, not a feeling |
| Checkable | An auditor can see which rule fired on any case |
Anti-pattern: “escalate when unsure” delegates the decision to the component you were unsure about, and nothing can audit it.
| Trigger | Response |
|---|
| A person asks for a human, in any words | Hand off immediately; no negotiation, no one-more-try |
| Policy is ambiguous or silent | Escalate; a model fills gaps with fluent improvisation |
| Proxy | Why it fails |
|---|
| Sentiment | Emotion is poorly correlated with difficulty; rewards the loudest cases, not the hardest |
| Self-reported confidence | One more generated output from the same process; poorly calibrated to real difficulty |
Use instead: observable signals measured by code from outside the model: validation failures, disagreement between passes, missing fields, historically weak document types.
| Rule | Because |
|---|
| Route low-signal and ambiguous cases to people | That is where attention changes outcomes |
| Measure accuracy by segment, not aggregate | 96 percent overall can hide a 71 percent segment |
| Keep a small random sample in strong segments | Calibration decays quietly; the sample is the smoke alarm |
| Never punish escalation in metrics | Punished systems improvise; handoffs go invisible |
| Principle | Practice |
|---|
| The checker is not the doer | A fresh instance with clean context reviews the work |
| The author defends; a stranger reads | Self-review inherits the assumptions that made the errors |
| Generalizes to any generated work | Code diffs, extractions, summaries, reports |
| In Claude Code | A subagent’s clean context is the fresh reviewer |
| When the checker must be human | Agent SDK permission rules plus the approval callback |
| Pass | Catches |
|---|
| Per-item (one document, one job) | Local errors: internal inconsistencies, implausible values |
| Cross-item (relationships only) | Contradictions between documents: dates, sums, stories |
Why not one giant pass: attention dilutes across a large input; page forty-one gets skimmed.
| Line | Meaning |
|---|
| A gate is only as good as the rules that feed it | Veto power needs designed criteria upstream |
| The score has no outside | Self-reported confidence is generated, not measured |
| Escalation is the design working | A visible handoff beats an invisible error |
| The checker is not the doer | Independence is the active ingredient of review |
