Practice: Should AI do this task?
Self-check
Section titled “Self-check”Seven short questions. Answer each in your head before opening the collapsible. Active retrieval is where the learning sticks.
1. What are the two filters, and why does passing the first not settle the question?
Show answer
Filter one asks whether the task is the kind of work the machine is built for; filter two asks whether privacy, competing goals, the price of a wrong answer, and the real alternative say you should hand it over anyway. Capability first, consequences second, and the second filter has veto power. That a machine can do something does not mean you should ask it to: New York’s bot sailed through the first filter and crashed on the second.
2. What are the four signs a task suits the machine?
Show answer
First, it needs personalization and interaction, a task shaped like a conversation with one specific person. Second, there is a large body of text to analyze, summarize, or draw from. Third, it benefits from creative variation, the machine’s built-in willingness to put out possibilities beyond the obvious ones. Fourth, demonstration data exists: pairs of a question and an answer someone already judged good.
3. What are the four consequence questions?
Show answer
Privacy: am I comfortable with this information leaving my hands under this tool’s rules? Alignment clarity: do the builder’s goals match the user’s? The cost of false information: what would a confident wrong answer cost, and who would pay? And the real alternative: how well does the tool perform against what actually happens when people do the task, honestly measured?
4. What does “which direction of error hurts more” mean, and why ask it?
Show answer
Most bots can be wrong in two directions. A customer-service bot ruling on discounts can grant one wrongly or deny one wrongly, and the two mistakes usually cost different amounts. Naming the expensive direction tells you what to build against. Air Canada learned this at a tribunal: its chatbot invented a refund policy, and in February 2024 the airline was ordered to pay. Your bot’s words are your words.
5. Why is confidence not evidence?
Show answer
Because machines’ mistakes arrive in the same fluent, assured voice as their correct answers. An early 2024 Stanford study reported that widely used models of that era, tested without grounding, gave false or fabricated answers about a court’s core ruling at least 75 percent of the time. The number belongs to 2024 and behavior has shifted since, but the habit it teaches is permanent: price the wrong answer before the handoff, because the tone will never warn you.
6. Why is the placement of the human check a safety feature?
Show answer
Because a machine drafting for a human reviewer is a different proposition from a machine answering the public unchecked. The course’s policy drafter scored low-risk precisely because its drafts flow into human vetting. The consulting report that Deloitte corrected failed exactly where verification was missing: the document’s whole value was that its claims could be trusted. Remove the check and you have changed the risk, not the workload.
7. What is the good-enough ladder, and who decides where the bar sits?
Show answer
The course’s closing question about the New York bot: acceptable if cheaper but somewhat worse than a human? As accurate as the average city employee? Somewhat better? Much better? Nothing short of one hundred percent correct? The course’s own classroom split on the answer. There is no formula; it is a judgment about values, and your answer, not the vendor’s marketing, should decide adoption.
Try it yourself
Section titled “Try it yourself”Screen the task you brought
Lesson 3 asked you to bring a task from your own week that you are tempted to hand over. This practice is that task’s screening: both filters, all eight questions, in order, ending in a verdict. Clawless, the working environment we use across Clawdemy, is your sparring partner, and a plain conversation is all this needs.
- Set up the sparring partner. Open a conversation in Clawless and give the assistant lesson 2’s interviewer persona, sharpened into a skeptic whose job is to challenge every score you give yourself, demand evidence, and refuse to let a vague answer pass. If you built lesson 3’s tailored assistant, you can reuse it here; its standing instructions already know your work, and you can point it at a document describing the task so it challenges you with specifics.
- State the task in one sentence. Tell the interviewer what the task is, who the result is for, and what finished looks like. If you cannot say it in a sentence, that is the first finding.
- Run filter one, sign by sign. Score your task from one to three on each of the four signs, in order, the way the course’s class did. Does it need personalization and interaction? Is there a large body of text to analyze, summarize, or draw from? Does it benefit from creative variation? Does demonstration data exist, past examples of this task done and judged good? Defend each score against the interviewer.
- Pause at the pivot. A good filter-one score establishes only that the machine can. The second filter decides whether it should, and it has veto power. Say that to the interviewer before moving on; it will hold you to it.
- Run filter two, question by question. Privacy: what would you have to hand over, and are you comfortable with that information leaving your hands under this tool’s rules? Alignment clarity: whose goals would this assistant serve, and where might the builder’s goals and yours part ways? The cost of false information: which direction of error hurts more, what would a confident wrong answer cost, and who would pay? The real alternative: how does the tool compare against what actually happens when a person does this task, honestly measured, not against perfection and not against nothing?
- Name the expensive direction of error out loud. Before the verdict, have the interviewer press you on one thing: of the ways the machine could get this task wrong, which mistake costs more, and where exactly would a human check catch it before the cost lands?
- Deliver the verdict. Three options, no fourth: hand it over, hand over a piece with a human check placed where the cost lives, or keep it. State your verdict and your reasons in three sentences. If the interviewer can puncture the reasoning, revise the verdict, not the reasons.
One last reflection, away from the keyboard: this screening is the one New York City skipped, and it took you minutes. The next time someone proposes handing a task to AI, at work or at home, you can run it again. The most useful question in the room is usually the one you now ask by habit: what does a wrong answer cost, and who pays?
Flashcards
Section titled “Flashcards”Q. What are the two filters between you and a bad AI handoff?
Filter one: is this task the kind of work the machine is built for? Filter two: even if it is, do privacy, competing goals, the price of a wrong answer, and the real alternative say you should hand it over? Capability first, consequences second, and the second filter has veto power.
Q. What are the four signs a task suits the machine?
It needs personalization and interaction; there is a large body of text to analyze, summarize, or draw from; it benefits from creative variation; and demonstration data exists, pairs of a question and an answer someone already judged good.
Q. What does the course mean by creativity, and what dial tunes it?
Not creating the way people do: a willingness to put out possibilities beyond the obvious ones, built in because the machine picks among plausible next words rather than repeating the single most likely one. Many tools expose a dial called temperature: low keeps answers consistent, high reaches for less likely words, too high produces nonsense.
Q. What is demonstration data?
Pairs of a question and an answer someone already judged good, the raw material behind lesson 2’s few-shot move and lesson 3’s fine-tuning. In the idea competition the course cites, the machine’s strongest run came when it was seeded with examples of previously successful ideas.
Q. Why does passing filter one not settle the handoff?
Because a machine can do something does not mean you should ask it to. New York’s business chatbot sailed through the first filter, official text plus interaction, and crashed on the second, giving the public illegal advice unchecked. Most AI failures have that same shape.
Q. What is the privacy question to ask before pasting?
Am I comfortable with this information leaving my hands under this tool’s rules? Whatever you hand the machine, you are handing to whoever runs it. The mitigating layers the course described in 2024 still exist as of mid-2026, but the specifics vary by tool and plan.
Q. What is alignment clarity, in this lesson's practical sense?
Do the builder’s goals match the user’s? A school wants a bot that guides students toward understanding; the student at midnight wants the answers. When goals conflict, users probe until something gives, like the visitor who told a car dealership’s website assistant to agree with anything the customer said, then got it to agree to sell an SUV for one dollar.
Q. What does the direction of error question ask?
A bot can usually be wrong in two directions, like granting a discount wrongly or denying one wrongly, and the two mistakes cost different amounts. Name the direction of error that hurts more, and build against that one.
Q. What did the Air Canada tribunal case establish?
The airline’s chatbot invented a refund policy, and the company argued it was not liable for what its chatbot said. The tribunal wrote that this was in effect suggesting the chatbot was “a separate legal entity that is responsible for its own actions,” and in February 2024 ordered the airline to pay. Your bot’s words are your words.
Q. Why is confidence not evidence?
Machines’ mistakes arrive in the same fluent, assured voice as their correct answers. An early 2024 Stanford study found widely used models of that era, tested without grounding, gave false or fabricated answers about a court’s core ruling at least 75 percent of the time; the number is dated and behavior has shifted since, but the habit stands. Price the wrong answer before the handoff.
Q. What is the fair baseline for judging an AI tool?
The real alternative: what actually happens when people do the task, honestly measured. Not perfection, and not nothing. Humans screening resumes are measurably biased; the machine’s record on the same task still cuts both ways as of mid-2026. Measure both before trusting either.
Q. What is the good-enough ladder?
The course’s closing question about the New York bot: acceptable if cheaper but somewhat worse than a human? As accurate as the average city employee? Somewhat better? Much better? Nothing short of one hundred percent correct? The classroom split; there is no formula. Your answer, not the vendor’s marketing, should decide adoption.
Q. Why is the placement of the human check a safety feature?
A machine drafting for a human reviewer is a different proposition from a machine answering the public unchecked. The policy drafter scored low-risk because its drafts flow into human vetting; the consulting report failed exactly where verification was missing. Put the check where the cost lives.