How sure are we? confidence intervals

What you’ll learn

This is lesson 12 of Track 9 (Statistics & Probability for AI) and the second lesson of Phase 4 (From sample to truth). The previous lesson established that any sample estimate has a standard error; this lesson turns that wobble into an honest report, a range of plausible values for the truth. You will learn to build a confidence interval, how its width responds to sample size and confidence level, and the correct interpretation, taking care to dismantle the misreading nearly everyone falls into. The source curriculum is Khan Academy’s Statistics & Probability course, by Sal Khan and the Khan Academy team, freely available and cited as further study.

The lesson moves from point estimate to interval (estimate plus or minus a margin of error), gives the rule of thumb that a 95% interval is about two standard errors wide on each side, shows the two dials that set the width (data narrows, confidence widens), and spends real care on interpretation: a 95% interval is a statement about the procedure’s long-run reliability, not a 95% probability for this one range. It closes on reading AI metrics with intervals and the overlapping-intervals signal that two results are indistinguishable.

Where this fits

This is lesson 12 of 14, the middle of Phase 4. It is built directly on the standard error and the central limit theorem from the previous lesson. It also sets up the next lesson: overlapping confidence intervals hint that a difference may be noise, and Hypothesis testing makes that judgment formal. The capstone then folds confidence intervals into the larger picture of evaluating models honestly.

Before you start

Prerequisites: Sampling and the central limit theorem (lesson 11), since the margin of error is built from the standard error and the central limit theorem is what makes the two-standard-error rule work. Comfort with multiplication and addition is the only arithmetic required.

About the math

The arithmetic is light: multiply a standard error by about 2 (for 95% confidence), then add and subtract from the estimate. The harder work is conceptual, holding the correct interpretation of the confidence level, which the lesson and practice drill directly. No new formulas beyond the margin of error.

By the end, you’ll be able to

Explain why a point estimate alone hides uncertainty and an interval shows it
Build a confidence interval as estimate plus or minus a margin of error (about two standard errors for 95%)
Describe how sample size and confidence level trade off against interval width
State the correct interpretation of a confidence interval and reject the common probability misreading
Use confidence intervals to read AI metrics honestly and see when two results are indistinguishable

Time and difficulty

Read time: about 12 minutes
Practice time: about 15 minutes (a self-check, a build-the-interval computation across confidence levels and sample sizes, a true-or-false interpretation drill, and flashcards)
Difficulty: standard (easy arithmetic; the challenge is the interpretation, which the practice targets)