Mastering Logistic Regression and Classification in Machine Learning

In the field of machine learning, classification is a fundamental task where we aim to predict discrete labels for input data. While linear regression is excellent for predicting continuous values, it fails when we need to categorise data into distinct groups, such as identifying spam emails or diagnosing medical conditions. This is where logistic regression becomes an essential tool in your toolkit.

In this article, we will explore why logistic regression is the standard approach for binary classification. You will learn how to transform linear outputs into probabilities using the sigmoid function, understand the concept of log-odds, and see how to apply these principles to real-world data. Mastering these concepts is vital for your undergraduate machine learning modules and provides the foundation for more complex neural network architectures.

The Limitation of Linear Regression for Classification

If we attempt to use standard linear regression for a binary classification task where labels are $y \in {0, 1}$, we model the relationship as $z = \beta_0 + \beta_1 x$. However, linear regression can produce outputs far outside the $[0, 1]$ range, which is mathematically inconsistent with the definition of a probability. A probability must be bounded between 0 and 1. Furthermore, a linear model assumes a constant rate of change, which is inappropriate when we are interested in the likelihood of a categorical outcome.

The Sigmoid Function: Mapping to Probability

To constrain our output to the range $(0, 1)$, we pass our linear combination $z$ through the sigmoid function (also known as the logistic function). The sigmoid function is defined as:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

As $z$ approaches positive infinity, $\sigma(z)$ approaches 1. As $z$ approaches negative infinity, $\sigma(z)$ approaches 0. This "S-shaped" curve allows us to interpret the output as the probability $P(Y=1|X)$.

Worked Example 1: Calculating Probability

Suppose we have a model with a bias $\beta_0 = -2$ and a weight $\beta_1 = 0.5$. For an input $x = 6$, calculate the predicted probability.

1. Calculate the linear combination: $z = \beta_0 + \beta_1 x = -2 + (0.5 \times 6) = -2 + 3 = 1$.
2. Apply the sigmoid function: $\sigma(1) = \frac{1}{1 + e^{-1}} \approx \frac{1}{1 + 0.3679} \approx 0.731$.

The model predicts a 73.1% probability that the input belongs to the positive class.

Log-Odds and the Logit Function

Logistic regression is often described as modelling the log-odds of an event. The odds are defined as the ratio of the probability of an event occurring to the probability of it not occurring: $\text{Odds} = \frac{p}{1-p}$.

By taking the natural logarithm of the odds, we get the logit function:

$$\ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x$$

This transformation is powerful because it maps the probability space $(0, 1)$ back to the entire real number line $(-\infty, \infty)$, allowing us to use linear techniques to solve classification problems.

Decision Boundaries

In binary classification, we typically use a threshold of 0.5 to make a final decision. If $\sigma(z) \ge 0.5$, we classify the input as 1; otherwise, we classify it as 0. Since $\sigma(z) = 0.5$ occurs when $z = 0$, the decision boundary is defined by the equation $\beta_0 + \beta_1 x = 0$.

Worked Example 2: Finding the Decision Boundary

Given the model $z = -4 + 2x$, find the value of $x$ that acts as the decision boundary.

1. Set $z = 0$: $-4 + 2x = 0$.
2. Solve for $x$: $2x = 4$, so $x = 2$.

Any input $x > 2$ will result in a probability greater than 0.5, classifying it as the positive class.

Common Mistakes

• Confusing Regression with Classification: Remember that despite the name, logistic regression is a classification algorithm. It predicts the probability of class membership, not a continuous quantity.
• Ignoring the Threshold: Always define your decision threshold clearly. While 0.5 is standard, some applications (like fraud detection) may require different thresholds to balance precision and recall.
• Assuming Linearity in Probability: The relationship between the input $x$ and the probability $p$ is non-linear (sigmoid), but the relationship between $x$ and the log-odds is linear.

Frequently Asked Questions

Why is it called logistic regression? It is a historical misnomer. It uses the logistic function to perform classification, but the underlying mathematical structure is derived from linear regression techniques.

Can logistic regression handle more than two classes? Yes, by using techniques like "one-vs-all" or by extending the model to multinomial logistic regression (often using the Softmax function).

What is the loss function for logistic regression? We typically use Binary Cross-Entropy loss (or Log Loss), which penalises confident but incorrect predictions heavily.

Conclusion

Logistic regression remains a cornerstone of machine learning, providing a robust and interpretable method for binary classification. By understanding the sigmoid function and the log-odds transformation, you are well-equipped to tackle more advanced topics like neural networks. To see these concepts in action with interactive visualisations, generate a free animated lesson on this topic at MathInstructor AI.