Cheatsheet: Updating beliefs with evidence: Bayes' theorem

The one idea

Bayes’ theorem updates a belief with evidence: posterior = (likelihood x prior) / evidence. It converts P(evidence | hypothesis) into P(hypothesis | evidence), and it keeps the base rate in the calculation.

The formula and its parts

              P(E | H) x P(H)
  P(H | E) = -----------------
                   P(E)

  P(H)      prior       base rate before the evidence
  P(E | H)  likelihood  how well the evidence fits H (e.g. test hit rate)
  P(E)      evidence    total chance of E = P(E|H)P(H) + P(E|not H)P(not H)
  P(H | E)  posterior   the updated belief (what you want)

Natural frequencies (usually easier)

Disease 1 in 100, test 99% accurate both ways, 10,000 people:
                 positive   negative   total
  has disease       99          1        100
  healthy           99       9,801      9,900
  total            198       9,802     10,000
P(disease | positive) = 99 / 198 = 0.50   (rare base rate -> as many false as true positives)

Same answer by formula

prior 0.01, likelihood 0.99, false-positive 0.01 on healthy 0.99
P(E) = 0.99*0.01 + 0.01*0.99 = 0.0198
P(H|E) = 0.0099 / 0.0198 = 0.50

Updating twice (posterior becomes the next prior)

2nd independent positive, new prior 0.50:
P(E) = 0.99*0.50 + 0.01*0.50 = 0.50
P(H|E) = 0.495 / 0.50 = 0.99
1%  ->  50%  ->  99% as evidence accumulates.

In machine learning

• Naive Bayes spam filter: prior spam rate x word likelihoods (assumes words independent given class).
• Combine a detector’s hit rate with the base rate to get the real probability of a flag.
• Bayesian updating: hold a prior, sharpen it as data arrives.

Pitfalls to dodge

• Ignoring the prior (base-rate neglect): the headline error.
• Confusing likelihood P(E|H) with posterior P(H|E): the flipped-bar error.
• Forgetting false positives in P(E): overstates the posterior.
• Expecting certainty from one piece of evidence: low base rates need accumulating evidence.

Words to use precisely

• Prior: probability of the hypothesis before the evidence (the base rate).
• Likelihood: P(evidence | hypothesis); how well the evidence fits.
• Posterior: P(hypothesis | evidence); the updated belief.
• Base-rate neglect: ignoring the prior when interpreting evidence.