Reading the results: the confusion matrix, precision, recall, and ROC

What you’ll learn

This is lesson 15 of Track 10, the close of Phase 4 (Knowing whether your model is any good) and the close of the track. By the end you will be able to pick the right metric for an imbalanced or asymmetric-cost problem and read a confusion matrix and a ROC curve. The one capability to walk away with: given a classifier’s confusion matrix, compute the standard metrics by hand, say which matters for the problem (and why), and recognize when “99% accurate” is hiding a useless model.

The track structurally mirrors StatQuest’s intuition-first machine learning videos, with Microsoft’s “ML For Beginners” as the hands-on companion for readers who want to build the models in code. Full attribution is in this lesson’s references.

Where this fits

This is the final lesson of T10. The previous two lessons covered how to evaluate honestly (bias-variance diagnostic from train/test error, and cross-validation for stable estimates). This lesson covers what to evaluate with, the metric you actually compute on the held-out data. Together the three lessons make evaluation rigorous, and with that, the classical-machine-learning toolbox the track set out to build is complete: every workhorse model from regression through ensembles to clustering, plus the framework to judge whether any of them is doing the job.

Before you start

Prerequisite: Lesson 14, Train, test, and cross-validation. You need the idea of evaluating a model on held-out data, because this lesson is about which metric to compute on that held-out evaluation. The lesson also calls back to the decision threshold introduced way back in lesson 4 (logistic regression); having that one in mind helps but is not strictly required. No math beyond addition and division.

By the end, you’ll be able to

Explain why accuracy lies on imbalanced data
Read a confusion matrix and compute precision, recall, specificity, F1
Explain the precision-recall tradeoff via the decision threshold
Read a ROC curve and AUC; recognize when to prefer the precision-recall curve
Pick the right metric for an imbalanced or asymmetric-cost problem

Time and difficulty

Read time: about 13 minutes
Practice time: about 15 minutes (a compute-from-a-confusion-matrix exercise, a pick-the-metric scenario, and flashcards)
Difficulty: standard