Linear Regression and the Least Squares Method: A Comprehensive Guide

Introduction to Linear Regression

Linear regression is a cornerstone of supervised machine learning. At its core, it is a method for modelling the relationship between a scalar response variable and one or more explanatory variables. For any undergraduate student, understanding this concept is vital, as it forms the foundation for more complex algorithms like neural networks and support vector machines.

In this article, we will explore how to define a linear model, the mathematical derivation of the least squares method, and how we use optimisation techniques like gradient descent to find the best parameters. By the end, you will have a robust understanding of how to minimise error and improve predictive accuracy in your models.

Defining the Linear Model

A simple linear regression model assumes a linear relationship between the input $x$ and the output $y$. We express this as:

$$y = \beta_0 + \beta_1 x + \epsilon$$

Where $\beta_0$ is the intercept, $\beta_1$ is the slope (or weight), and $\epsilon$ represents the irreducible error or noise. Our goal in machine learning is to find the estimates $\hat{\beta}_0$ and $\hat{\beta}_1$ that minimise the difference between our predicted values $\hat{y}$ and the actual observed values $y$.

The Least Squares Method

The least squares method is an optimisation approach that seeks to minimise the sum of the squared residuals (RSS). A residual is defined as the difference between the observed value and the predicted value: $e_i = y_i - \hat{y}_i$. To avoid negative residuals cancelling out positive ones, we square them:

$$RSS = \sum_{i=1}^{n} (y_i - (\hat{\beta}_0 + \hat{\beta}_1 x_i))^2$$

By minimising this function, we ensure that the line of best fit is as close as possible to all data points simultaneously.

Worked Example 1: Simple Linear Regression

Consider a small dataset: $(1, 2), (2, 3), (3, 5)$. We want to fit $y = \beta_0 + \beta_1 x$.

1. Calculate the means: $\bar{x} = 2$, $\bar{y} = 3.33$.
2. Calculate the slope $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$.
   • Numerator: $(1-2)(2-3.33) + (2-2)(3-3.33) + (3-2)(5-3.33) = 1.33 + 0 + 1.67 = 3$.
   • Denominator: $(1-2)^2 + (2-2)^2 + (3-2)^2 = 1 + 0 + 1 = 2$.
   • $\hat{\beta}_1 = 3 / 2 = 1.5$.
3. Calculate the intercept $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} = 3.33 - (1.5 \times 2) = 0.33$.

The resulting model is $y = 0.33 + 1.5x$.

Gradient Descent for Optimisation

While we can solve for parameters analytically using matrix algebra (the normal equation), large datasets often require iterative optimisation. Gradient descent updates parameters by moving in the direction of the steepest descent of the cost function:

$$\beta_j := \beta_j - \alpha \frac{\partial}{\partial \beta_j} J(\beta)$$

Where $\alpha$ is the learning rate. By calculating the partial derivative of the cost function with respect to each weight, we iteratively nudge the parameters toward the global minimum.

Worked Example 2: Gradient Descent Step

Given a cost function $J(\beta) = \frac{1}{2n} \sum (\hat{y}_i - y_i)^2$, the derivative with respect to $\beta_1$ is $\frac{1}{n} \sum (\hat{y}_i - y_i)x_i$. If our current prediction $\hat{y}_i = 2$ and actual $y_i = 3$ with $x_i = 1$, the gradient is $(2-3) \times 1 = -1$. If $\alpha = 0.1$, the new weight becomes $\beta_1^{new} = \beta_1^{old} - 0.1(-1) = \beta_1^{old} + 0.1$. This increases the weight to reduce the error.

Common Mistakes

• Confusing Correlation with Causation: A high $R^2$ value does not imply that $x$ causes $y$.
• Ignoring Outliers: Least squares is sensitive to outliers because squaring large residuals disproportionately increases the cost.
• Overfitting: Adding too many features without regularisation can lead to a model that performs well on training data but fails on unseen data.
• Incorrect Learning Rate: If $\alpha$ is too high, the gradient descent may overshoot the minimum; if too low, convergence will be painfully slow.

Frequently Asked Questions

What is the difference between OLS and Gradient Descent? Ordinary Least Squares (OLS) provides a closed-form analytical solution, whereas Gradient Descent is an iterative numerical approach suitable for very large datasets.

Why do we square the residuals? Squaring ensures that all residuals are positive and penalises larger errors more heavily than smaller ones, which is mathematically convenient for differentiation.

Can linear regression handle non-linear data? Yes, by using polynomial features or basis functions, you can model non-linear relationships while keeping the model linear in its parameters.

Conclusion

Linear regression remains a fundamental tool in the machine learning toolkit. By mastering the least squares method and understanding the mechanics of gradient descent, you are well-equipped to tackle more advanced topics. For a visual breakdown of these concepts, visit MathInstructor AI to generate a free animated lesson on this topic.