Cheatsheet: How sure are we? confidence intervals

The one idea

A confidence interval reports an estimate with its uncertainty: estimate plus or minus a margin of error. Its meaning is about the procedure’s long-run hit rate, not a probability for this one interval.

Building the interval

confidence interval = point estimate  +/-  margin of error
margin of error      = multiplier x standard error

95% confidence -> multiplier about 2 (1.96)   =>  estimate +/- 2 standard errors
99% confidence -> multiplier about 2.6

Example: 90% accuracy, standard error 2 points:
  95% CI = 90% +/- 2 x 2% = [86%, 94%]

What moves the width

More data       -> smaller standard error (sigma/sqrt(n)) -> NARROWER interval
Higher confidence -> bigger multiplier                     -> WIDER interval
  90% acc, SE 2 -> 95% CI [86,94];  SE 1 -> [88,92] (more data)
                   99% CI [84.8,95.2] (more confidence, wider)
A tight high-confidence interval comes from MORE DATA, not the confidence dial.

Interpretation: right vs wrong

RIGHT: "If we repeated the sampling many times, ~95% of the intervals built
        would contain the true parameter." (a statement about the procedure)
WRONG: "There's a 95% probability the truth is in THIS interval."
        (the parameter is fixed; the interval is what varies)
WRONG: "95% of the data falls in this interval." (it's about the parameter, not data points)

In machine learning

• Report metrics WITH intervals: “90%, 95% CI [86,94]” not bare “90%”.
• Overlapping intervals => two results are indistinguishable (cannot call a winner).
• Small test set => wide interval => honest signal of how little you know.

Pitfalls to dodge

• The probability misreading (95% chance for THIS interval).
• Thinking the interval covers 95% of the data.
• Forgetting higher confidence = wider interval.
• Comparing point estimates while ignoring overlapping intervals.

Words to use precisely

• Point estimate: a single best-guess number for the parameter.
• Margin of error: multiplier x standard error; the interval’s half-width.
• Confidence level: the procedure’s long-run capture rate (e.g. 95%).
• Confidence interval: estimate +/- margin of error; a range of plausible parameter values.