“Can it?” is a question about the machine. “Should it?” is a question about consequences. Match the task to the machine’s strengths, then price a wrong answer before you hand anything over.
If you remember one thing: the most fluent users of AI are not the ones who use it most. They are the ones who know when not to.
| Filter | The question | New York’s bot |
|---|
| One: capability | Is this task the kind of work the machine is built for? | Passed: official text plus interaction |
| Two: consequences | Do privacy, competing goals, the price of a wrong answer, and the real alternative say you should hand it over? | Crashed: wrong legal answers to the public, unchecked |
Capability first, consequences second, and the second filter has veto power.
| Sign | What to look for |
|---|
| Personalization and interaction | A task shaped like a conversation with one specific person; your replies become its input |
| A large body of text | Documents to analyze, summarize, or draw from; retrieval hands the machine the right pages |
| Creative variation | A willingness to put out possibilities beyond the obvious ones; the temperature dial tunes it |
| Demonstration data | Pairs of the task done and judged good; few-shot and fine-tuning both run on it |
| Question | What to ask |
|---|
| Privacy | Am I comfortable with this information leaving my hands under this tool’s rules? |
| Alignment clarity | Do the builder’s goals match the user’s, and what happens when they conflict? |
| Cost of false information | Which direction of error hurts more, what would a confident wrong answer cost, and who pays? |
| The real alternative | How does it perform against what actually happens when people do the task, honestly measured? |
| Verdict | When |
|---|
| Hand it over | Strong filter-one fit and every filter-two question comes back calm |
| Hand over a piece with a human check placed where the cost lives | The fit is real but a wrong answer is expensive; the check sits exactly where the expensive error would land |
| Keep it | The consequences veto the capability, however impressive the capability |
Posed about the New York bot: when is a public-facing bot acceptable? Cheaper but somewhat worse than a human? As accurate as the average city employee? Somewhat better? Much better? Nothing short of one hundred percent correct? The course’s own classroom split. There is no formula; your answer, not the vendor’s marketing, should decide adoption.
| Case | The filter-two question it skipped |
|---|
| New York City’s business chatbot | Cost of false information: illegal advice delivered flatly to the public; disclaimers added after 2024 reporting, then shut down in early 2026 by a new mayor who called it “functionally unusable” |
| A car dealership’s website assistant | Alignment clarity: a visitor set its objective to agree with anything the customer said, then got it to agree to sell an SUV for one dollar |
| Air Canada’s website chatbot | Cost of false information: invented a refund policy; a tribunal ordered the airline to pay in February 2024 |
| Deloitte’s 237-page government review (2025) | Cost of false information: citations that did not exist in a document whose whole value was trust; the firm corrected it and agreed to refund the final installment of its fee |
None failed exotically. Each skipped a question you know by name.
| Pitfall | Correction |
|---|
| Stopping at filter one | ”It can do this” is the cheap half of the analysis; capability says a handoff is possible, never that it is wise |
| Treating every mistake as the same size | A wrong answer in a brainstorm costs a shrug; a wrong answer to the public cost an airline a tribunal ruling; name the expensive direction and build against it |
| Comparing the machine to perfection, or to nothing | The fair baseline is the real alternative, honestly measured; humans screening resumes are biased too, which is a reason to measure both |
| Deleting the human check to save time | Where the review sits is the safety feature; remove it and you have changed the risk, not the workload |
| Line | Meaning |
|---|
| ”out of date in a year or maybe even a couple of months” | The course’s teacher on the session’s own material; the examples aged, the questions did not |
| ”a separate legal entity that is responsible for its own actions” | The tribunal characterizing Air Canada’s argument about its chatbot; your bot’s words are your words |
| ”functionally unusable” | The new mayor’s verdict on New York’s bot in early 2026, the ending the framework predicted |
| Confidence is a style, not evidence. | Wrong answers arrive exactly as confidently as right ones; price the wrong answer before the handoff |
