Practice: From sample to population: sampling and the central limit theorem

The skills: keeping statistic and parameter straight, computing a standard error and feeling the square-root law, and reading what the central limit theorem buys you. Keep a scratchpad.

Self-check

Six short questions. Answer each in your head before opening the collapsible.

1. What is the difference between a parameter and a statistic?

Show answer

A parameter is a true number about the whole population (the true average, the true accuracy), usually unknown. A statistic is the matching number computed from your sample (the sample average, the measured accuracy), known but only an estimate of the parameter. Inference uses the statistic to say something about the parameter.

2. Why is a sample statistic considered a random variable?

Show answer

Because a different random sample produces a different value, just by the luck of who landed in the sample. That sample-to-sample variation is sampling variability, and the distribution of the statistic across all possible samples is its sampling distribution.

3. What is the standard error, and what is its formula for a sample mean?

Show answer

The standard error is the standard deviation of the sampling distribution: how much the estimate bounces around from sample to sample. For a sample mean it is sigma over the square root of n (the population standard deviation divided by the square root of the sample size). Smaller standard error means a more precise estimate.

4. To cut your standard error in half, what must you do to the sample size?

Show answer

Quadruple it. The standard error shrinks with the square root of n, so halving the error requires four times the data. This is why early data helps a lot and later data helps less (diminishing returns).

5. State the central limit theorem in your own words.

Show answer

For a large enough sample, the sampling distribution of the mean is approximately normal, regardless of the shape of the original population. Even a wildly skewed population produces sample means that pile up into a bell curve, centered on the true mean with spread sigma over root n.

6. In what sense is a model’s test-set accuracy a sample estimate?

Show answer

The test set is a sample; the accuracy on it is a statistic estimating the model’s true accuracy on all future data (the parameter). A different test set would give a slightly different number, so the reported accuracy has a standard error, larger for smaller test sets.

Try it yourself: standard error and the square-root law

A population has a standard deviation of 30. Compute the standard error of the sample mean for these sample sizes, then answer the follow-up.

1. n = 25
2. n = 100
3. n = 900
4. How large a sample would you need for a standard error of 1.5?

Show answer

1. SE = 30 / sqrt(25)  = 30 / 5  = 6
2. SE = 30 / sqrt(100) = 30 / 10 = 3
3. SE = 30 / sqrt(900) = 30 / 30 = 1

4. Want 30 / sqrt(n) = 1.5  ->  sqrt(n) = 30 / 1.5 = 20  ->  n = 400.

Notice the square-root law in action: going from n = 25 to n = 100 (4x the data) cut the standard error from 6 to 3 (in half), and n = 100 to n = 900 (9x) cut it from 3 to 1 (a third). More data always helps, but with diminishing returns.

Try it yourself: which estimate do you trust?

Model A reports 92% accuracy, measured on a test set of 100 examples.
Model B reports 90% accuracy, measured on a test set of 10,000 examples.

1. Which accuracy estimate is more trustworthy, and why?
2. Model A's number is higher. Does that settle that A is the better model?
3. Does Model B's larger test set make its individual predictions more
   consistent, or its accuracy estimate more precise?

Show answer

• 1: Model B’s. Its accuracy is measured on 100 times more data, so its standard error is about ten times smaller (square-root law). Model A’s 92% on only 100 examples could swing by several percentage points on another test set.
• 2: No. A’s point estimate is higher, but it is a noisy estimate from a small sample. The two estimates may well overlap once you account for their standard errors; deciding whether A is really better is a hypothesis test (two lessons from now). A higher number on a small sample is not a verdict.
• 3: Its accuracy estimate more precise. A larger test set does not change how the model behaves on individual examples; it shrinks the standard error of the measured accuracy, making the estimate of true accuracy tighter.

Flashcards

Eight cards. Click any card to reveal the answer. Use the Print flashcards button to lay out the full set as one card per page for offline review.

Print flashcards

Q. Parameter vs statistic: which is which?

A.

A parameter is a true (usually unknown) number about the whole population. A statistic is computed from a sample and estimates the parameter. Inference uses the statistic to learn about the parameter.

Q. Why is a sample statistic a random variable?

A.

Because a different random sample gives a different value. That sample-to-sample variation is sampling variability; the distribution of the statistic over all samples is its sampling distribution.

Q. What is the standard error of the mean, and its formula?

A.

The standard deviation of the sampling distribution of the mean: how much the estimate varies sample to sample. Formula: sigma / sqrt(n) (population sd over root sample size). Smaller = more precise.

Q. To halve the standard error, what must you do to the sample size?

A.

Quadruple it. The standard error shrinks with the square root of n, so cutting it in half needs 4x the data. Diminishing returns on more data.

Q. State the central limit theorem.

A.

For a large enough sample, the sampling distribution of the mean is approximately normal, regardless of the population’s shape. Even skewed populations give bell-shaped sample means.

Q. Why does the central limit theorem matter for the rest of inference?

A.

Because sample means are approximately normal, you can apply z-scores and the 68-95-99.7 rule to them. That is exactly what confidence intervals and hypothesis tests do.

Q. How is a model's test-set accuracy a sample estimate?

A.

The test set is a sample; the measured accuracy estimates the true accuracy on all future data. A different test set gives a slightly different number, so accuracy has a standard error (bigger for smaller test sets).

Q. Does a bigger sample make the data less spread out or the estimate more precise?

A.

The estimate more precise. It does not change the spread of the data (sigma); it shrinks the standard error of the statistic. More data means more certainty about the average, not more similar individuals.