Linear regression
Also known as: least squares regression, line of best fit
Linear regression is a statistical method that fits a straight line through paired data to describe how one variable changes with another. The fitted line, written ŷ = a + bx, is chosen to make the sum of the squared vertical distances between the data points and the line as small as possible.
Linear regression models the relationship between an explanatory variable x and a response variable y as a straight line. The fitted equation is written ŷ = a + bx, where a is the y-intercept, b is the slope, and ŷ is the predicted response. The hat on ŷ is a reminder that the line produces predictions, not observed values.
The line is chosen by the least squares criterion. For each data point, the residual is the observed value minus the predicted value, y − ŷ — the vertical gap between the point and the line. The least squares line is the one that minimizes the sum of those squared residuals. Its slope can be computed from the correlation coefficient as b = r(sy / sx), and the line always passes through the point of averages (x̄, ȳ), so the intercept follows as a = ȳ − bx̄.
Interpretation matters as much as computation. The slope says that for each one-unit increase in x, the predicted value of y changes by b units — on average, not exactly. The intercept is the predicted y when x = 0, which is only meaningful when x = 0 falls within the range of the data. The coefficient of determination, r², gives the proportion of the variation in y explained by the linear relationship with x. A residual plot showing no pattern supports a linear model; curvature or fanning suggests the model is wrong. And a strong fit never establishes causation, nor does it license extrapolation far outside the observed range of x.
Regression appears across several exams. AP Statistics covers least squares regression, residual analysis, and confidence intervals for the slope; the CMA Part 1 exam treats regression analysis as a forecasting technique for estimating cost behavior; and the FE exams include it within probability and statistics. The fundamentals are the same in each: fit the line, interpret the slope, and check whether a linear model is appropriate.
Key takeaways
- Linear regression fits the line ŷ = a + bx through paired data by minimizing the sum of squared residuals.
- A residual is the observed value minus the predicted value: y − ŷ.
- The slope equals r(sy / sx), and the line always passes through (x̄, ȳ).
- r² is the proportion of variation in the response explained by the linear relationship.
- A strong linear fit does not prove causation, and predictions should not be extrapolated beyond the observed data.