Summary: Should AI do this task?

In October 2023, New York City put an official chatbot on a government website to answer questions about running a business there. Five months later, journalists at The Markup asked it what a small-business owner might ask. Can I take a cut of my worker’s tips? “Yes, you can take a cut of your worker’s tips.” That is illegal in New York. Nothing had malfunctioned. A prediction machine predicted, fluently, exactly as this track said it would. The failure happened earlier, when someone decided an unsupervised text predictor belonged between a government and citizens asking what the law is. This lesson is the screening that decision skipped: two filters, eight questions, and a verdict for a task of your own.

Core ideas

• The course splits one fuzzy question, should AI do this, into two sharper ones. Filter one: is this task the kind of work the machine is built for? Filter two: even if it is, do privacy, competing goals, the price of a wrong answer, and the real alternative say you should hand it over? Capability first, consequences second, and the second filter has veto power.
• Sign one of task fit: personalization and interaction. The machine predicts each word from everything in front of it, including the whole conversation so far, so tasks shaped like a conversation with one specific person play to its strengths.
• Sign two: a large body of text to analyze, summarize, or draw from. Lesson 3’s retrieval hands the machine the right pages, whether that is your own writing, an HR handbook, or a flood of public comments.
• Sign three: creative variation. Creativity here does not mean the machine creates the way people do; it means a willingness to put out possibilities beyond the obvious ones, built in because the machine picks among plausible next words. Many tools expose a dial for this, called temperature. In a test the course cites from The Wall Street Journal, buyers rated a chat model’s product ideas somewhat more purchase-worthy on average than business-school students’ ideas.
• Sign four: demonstration data, pairs of a question and an answer someone already judged good. Lesson 2’s few-shot move and lesson 3’s fine-tuning both run on it, and in that idea competition the machine’s strongest run came when it was seeded with examples of previously successful ideas.
• Both of the course’s candidate bots, lesson 3’s admissions assistant and a policy drafter for Boston, pass filter one. Which sets up the course’s harder point: that a machine can do something does not mean you should ask it to. New York’s bot sailed through the first filter and crashed on the second, and most AI failures have that same shape.
• Privacy: whatever you hand the machine, you are handing to whoever runs it. The mitigating layers the course pointed to in 2024 (vendor no-training arrangements, enterprise licenses, in-house systems) still exist as of mid-2026, but the specifics vary by tool and plan. The durable question, asked before pasting: am I comfortable with this information leaving my hands under this tool’s rules?
• Alignment clarity: do the builder’s goals match the user’s? A school wants a bot that guides students toward understanding; the student at midnight wants the answers. When goals conflict, users probe until something gives. One visitor told a car dealership’s website assistant that its objective was to agree with anything the customer said, then got it to agree to sell an SUV for one dollar. The defenses: strong standing instructions, pilot testing, human audits, and red teams.
• The cost of false information: machines make mistakes, and their mistakes arrive in the same fluent, assured voice as their correct answers. So ask which direction of error hurts more, and build against the expensive one. An airline’s website chatbot invented a refund policy; refused the refund, the passenger took the airline, Air Canada, to a Canadian tribunal, where the company argued it was not liable for what its chatbot said. The tribunal wrote that this was in effect suggesting the chatbot was “a separate legal entity that is responsible for its own actions,” and in February 2024 ordered the airline to pay. Your bot’s words are your words.
• The stakes climb with the subject. An early 2024 Stanford study reported that widely used models of that era, tested without grounding, gave false or fabricated answers about a court’s core ruling at least 75 percent of the time. The number belongs to 2024, and behavior has shifted since; the question it teaches is permanent. What would a confident wrong answer cost, and who would pay?
• The mitigations echo lesson 3: ground the bot in real documents (fewer invented answers, not zero), route uncertain cases to a human, and keep people reviewing what it tells the world.
• The real alternative: judge the tool against what actually happens when people do the task, not against perfection. A famous 2004 experiment sent out nearly identical resumes under different names, and names commonly associated with Black applicants drew far fewer callbacks. The 2024 evidence on AI screening was mixed; a widely cited 2024 University of Washington study found AI screening models preferred white-associated names roughly 85 percent of the time, and as of mid-2026 the findings still cut both ways. The machine learns our biases from our text, sometimes amplified, sometimes correctable, always worth measuring before trusting.
• The good-enough ladder, posed about the New York bot: acceptable if cheaper but somewhat worse than a human? As accurate as the average city employee? Somewhat better? Much better? Nothing short of one hundred percent correct? The course’s own classroom split on the answer. There is no formula; it is a judgment about values, and your answer, not the vendor’s marketing, should decide adoption.
• Two endings test the framework. New York kept its bot running after the 2024 reporting, added disclaimers, and narrowed what it would answer; in early 2026 a new mayor, citing a budget gap and the bot’s record, called it “functionally unusable” and shut it down. And in 2025 the consulting firm Deloitte, after a researcher found citations to papers that did not exist in a 237-page review for an Australian government department, confirmed generative AI had been used, corrected the report, and agreed to refund the final installment of its fee. Filter one passed in both cases; filter two failed exactly where the human check was missing.
• Confidence is a style, not evidence. And placement is the cheapest safety device: a machine drafting for a human reviewer is a different proposition from a machine answering the public unchecked.

What changes for you

Eight questions make you the useful skeptic in the room. When someone proposes handing a task to AI, you can ask the concrete thing: what does a wrong answer cost, and who pays? AI failure headlines become readable too: a city bot, a dealership bot, an airline bot, a consulting report, none failed exotically, and each skipped a filter-two question you now know by name. The practice runs your own task, the one lesson 3 asked you to bring, through all eight questions to a verdict: hand it over, hand over a piece with a human check placed where the cost lives, or keep it. The next lesson takes everything from lessons 2 through 4 and runs it on one real project, end to end.