Type I and type II errors

Every hypothesis test reaches one of two conclusions — reject the null hypothesis or fail to reject it — and the truth is one of two states, null true or null false. Two of the four resulting combinations are correct decisions and two are errors. Rejecting a true null hypothesis is a type I error; failing to reject a false null hypothesis is a type II error.

The probabilities have standard names. The probability of a type I error equals the significance level α chosen before the test, so testing at α = 0.05 accepts a 5% chance of a false positive when the null is true. The probability of a type II error is called β, and the complement 1 − β is the power of the test — the probability of correctly detecting a real effect. Power increases with a larger sample size, a larger true effect, less variability in the data, and a larger α.

That last item is the trade-off: for a fixed sample size, lowering α to guard against false positives raises β and makes false negatives more likely. Which error is worse depends on context. In a courtroom framing where the null is "the defendant is innocent," a type I error convicts an innocent person and a type II error acquits a guilty one. In a medical screening test where the null is "no disease," a type I error means a healthy patient is told they are sick and a type II error means a sick patient is told they are fine. Increasing the sample size is the one adjustment that escapes the trade-off: α stays wherever you set it, but a larger sample drives β down at that same α, so both error rates can be held low at once.

AP Statistics tests this directly, asking students to state each error in the context of a given study and describe its practical consequence — a context-free definition earns no credit. The FE Mechanical and FE Civil exams cover the same concepts within hypothesis testing, alongside z-tests and t-tests.