Summary: Reliability is a design choice

A triage nurse does not ask patients to rate their own severity and take the answer at face value. The nurse routes on observable criteria, written down in advance by someone who studied the failures. Agentic systems need the same discipline, and most teams skip it, handling “when should this reach a person?” with a vague instruction to escalate when unsure. This lesson replaces that with design: explicit escalation criteria, human review budgeted where it pays, and review structures that catch what a system cannot catch about itself.

Core ideas

• The track’s spine reaches its human end. Architecture is deciding where judgment lives, and there is a third place beyond the model and the code: a person. Some calls (a claim on a policy boundary, a situation the rules never anticipated) belong to someone with authority and accountability that neither the model nor the structure has. The architect designs when those calls get made.
• The AI Agent Teams track showed a gate with veto power, a separate role that reads everything and can overrule. This lesson designs the criteria that feed such a gate, because a gate is only as good as the rules that route work to it.
• “Escalate when unsure” is a blank check. It delegates the escalation decision to the exact component whose judgment you were worried about, and it can never be audited because there is nothing to check it against.
• Working criteria are explicit, rule-based, and checkable. On the lesson’s invented insurance intake system: amount over a threshold, a required document still missing after two requests, dates or amounts conflicting across documents, policy language that does not clearly cover the situation, and a claimant asking for a person. Every rule can be audited after the fact.
• Two escalations are never optional. A person who asks for a human gets one immediately, with no negotiation and no one-more-try. And when the policy is ambiguous or silent, the system escalates rather than improvises, because a model will fill any gap with fluent, confident text, and filling a policy gap is precisely the judgment that belongs to a person.
• Two tempting proxies fail, and this is Clawdemy’s design judgment from engineering experience. Sentiment-based escalation routes attention to the loudest cases instead of the hardest ones. A model’s self-reported confidence score is one more generated output, produced by the same process it claims to measure, and poorly calibrated to real difficulty. Route on observable signals instead: validation failures, disagreement between passes, missing fields, historically weak document types.
• Human attention is a budget. Route the low-signal and ambiguous cases to reviewers; a person confirming ninety-nine clean invoices is being used as decoration. Keep a small random sample even in strong segments, because calibration decays quietly.
• Measure accuracy by segment, never by one aggregate. A 96 percent overall figure can hide a 71 percent segment (handwritten witness statements, in the worked example) because rare segments barely dent the mean. Any system evaluated on one number will hide its weakest segment inside its strongest one.
• A fresh model instance catches what self-review misses. The authoring instance carries its whole reasoning history in context and re-reads its work through the same assumptions that produced the errors, like proofreading your own essay minutes after writing it. A clean-context reviewer reads what is actually on the page. Anthropic’s engineering essay names the workflow shape: in the evaluator-optimizer pattern, one model call generates while another evaluates in a loop, and it works best with clear evaluation criteria.
• Independence generalizes. A fresh instance reviewing a code diff it did not write, a summary against its article, extracted fields against source documents: any generated work, checked by an instance that did not generate it. In Claude Code, a subagent’s clean context is exactly this. When the checker must be a person, the Claude Agent SDK’s permission rules and approval callback put one in the flow.
• Large inputs need multi-pass review. One giant pass dilutes attention; page forty-one gets skimmed. Per-item passes catch local errors, and a separate cross-item pass catches contradictions, which no per-item pass can see because contradictions live between documents, not inside one.
• Escalation is not failure. Punish it in your metrics and the system learns to improvise, converting visible handoffs into invisible errors.

What changes for you

You can now interrogate any product that claims a human safety net: what, specifically, triggers escalation, and can the rules be shown? You ask “by segment?” whenever anyone quotes one accuracy number. And you can use independent review today without building anything: when generated work matters, hand it to a fresh conversation to check, because the author has something to defend and a fresh instance does not. Next, lesson 7 removes the human from the room: agents in CI/CD pipelines, working headless, where judgment written down in advance is the only human presence left.