Should AI do this task? In brief
Lesson 4 continues the track’s first arc, the four lessons on using generative AI well. It adapts the session on when and how to use generative AI from the same Harvard Kennedy School course, and it pays both promises lesson 3 left open: the story of an official chatbot that answered the public confidently and wrongly, and the screening of the task you were asked to bring from your own week.
The lesson asks nothing technical. Its cautionary opening is New York City’s official business chatbot, which journalists caught giving illegal advice in a voice The Markup noted “appears authoritative.” Its framework comes from the course’s two candidate bots, lesson 3’s admissions assistant and a first-draft policy writer for the city of Boston. Its practice is the promised screening, run in Clawless with a skeptical sparring partner.
The capability: after this lesson, you can split “should AI do this task” into two sharper questions, score a task against the four signs it suits the machine, ask the four consequence questions that decide whether to hand it over anyway, name the direction of error that costs more, and land on a verdict for a real task of your own: hand it over, hand over a piece with a human check placed where the cost lives, or keep it.
What the lesson covers. First the cautionary tale: New York’s bot told employers they could pocket part of workers’ tips and told landlords they could turn away tenants with housing vouchers, both illegal there. Nothing malfunctioned; a prediction machine predicted, and the failure was the judgment that placed it. Then the framework. Filter one asks whether the task is the machine’s kind of work, with four signs: personalization and interaction, a large body of text to draw from, benefit from creative variation, and existing demonstration data. Filter two asks whether it should be handed over even so, with four questions. Privacy: whatever you hand the machine, you hand to whoever runs it. Alignment clarity: the builder’s goals versus the user’s, the question a car dealership’s website assistant failed when a visitor talked it into agreeing to sell an SUV for one dollar. The cost of false information: which direction of error hurts more, the question that cost Air Canada a tribunal ruling over an invented refund policy. And performance against the real alternative: not perfection, but what actually happens when people do the task; humans screening resumes are measurably biased too, and the machine’s record on the same task still cuts both ways as of mid-2026. The course’s good-enough ladder closes the framework: how accurate would a public bot have to be before you would run it, a values judgment the course’s own classroom split on. Two endings then test the framework. New York added disclaimers, narrowed its bot’s scope, and in early 2026 a new mayor called it “functionally unusable” and shut it down. And the consulting firm Deloitte corrected a government report containing invented citations and agreed to refund the final installment of its fee, a failure the filters locate exactly where the human check was missing.
Why this order. Lessons 2 and 3 taught capability: asking well, then tailoring. Capability without judgment is how a city ends up telling businesses to break the law, so this lesson adds the judgment layer while the track’s prediction-machine model keeps doing the explaining. The practice in Clawless runs your own task through both filters, all eight questions, with the assistant playing lesson 2’s interviewer persona, tuned skeptical, against every score you give yourself. Lesson 5 takes everything from lessons 2 through 4 and runs it on one real project, end to end.