Mastering Eigenvectors, Eigenvalues and Diagonalisation for Machine Learning

In the landscape of machine learning, linear algebra is the engine under the bonnet. Whether you are working on Principal Component Analysis (PCA), understanding the stability of gradient descent, or optimising neural network architectures, you will inevitably encounter the concepts of eigenvectors and eigenvalues. These tools allow us to decompose complex linear transformations into simpler, interpretable components.

This article provides a rigorous yet accessible guide to these concepts. By the end, you will understand how to compute these values, why diagonalisation is a powerful tool for simplifying matrix operations, and how these mathematical structures underpin the algorithms you use every day.

Defining Eigenvalues and Eigenvectors

For a square matrix $A \in \mathbb{R}^{n \times n}$, an eigenvector is a non-zero vector $v$ that does not change its direction when multiplied by $A$. Instead, it is simply scaled by a factor $\lambda$. This relationship is defined by the equation:

$$Av = \lambda v$$

Here, $\lambda$ is the eigenvalue associated with the eigenvector $v$. If we rearrange this, we get $(A - \lambda I)v = 0$. For a non-zero solution $v$ to exist, the matrix $(A - \lambda I)$ must be singular, meaning its determinant must be zero. This leads us to the characteristic equation:

$$\det(A - \lambda I) = 0$$

Calculating Eigenvalues: A Worked Example

To find the eigenvalues of a matrix, we solve the characteristic polynomial. Let us consider the matrix:

$$A = \begin{pmatrix} 4 & 1 \ 2 & 3 \end{pmatrix}$$

1. Subtract $\lambda$ from the diagonal: $A - \lambda I = \begin{pmatrix} 4-\lambda & 1 \ 2 & 3-\lambda \end{pmatrix}$.
2. Calculate the determinant: $(4-\lambda)(3-\lambda) - (1)(2) = 0$.
3. Expand the polynomial: $12 - 7\lambda + \lambda^2 - 2 = \lambda^2 - 7\lambda + 10 = 0$.
4. Factorise: $(\lambda - 5)(\lambda - 2) = 0$.

The eigenvalues are $\lambda_1 = 5$ and $\lambda_2 = 2$.

Finding Eigenvectors

Once you have the eigenvalues, you find the corresponding eigenvectors by solving the homogeneous system $(A - \lambda I)v = 0$ for each $\lambda$.

For $\lambda_1 = 5$:

$$(A - 5I)v = \begin{pmatrix} 4-5 & 1 \ 2 & 3-5 \end{pmatrix} \begin{pmatrix} x \ y \end{pmatrix} = \begin{pmatrix} -1 & 1 \ 2 & -2 \end{pmatrix} \begin{pmatrix} x \ y \end{pmatrix} = \begin{pmatrix} 0 \ 0 \end{pmatrix}$$

This gives $-x + y = 0$, so $x = y$. An eigenvector is $v_1 = \begin{pmatrix} 1 \ 1 \end{pmatrix}$.

For $\lambda_2 = 2$:

$$(A - 2I)v = \begin{pmatrix} 4-2 & 1 \ 2 & 3-2 \end{pmatrix} \begin{pmatrix} x \ y \end{pmatrix} = \begin{pmatrix} 2 & 1 \ 2 & 1 \end{pmatrix} \begin{pmatrix} x \ y \end{pmatrix} = \begin{pmatrix} 0 \ 0 \end{pmatrix}$$

This gives $2x + y = 0$, so $y = -2x$. An eigenvector is $v_2 = \begin{pmatrix} 1 \ -2 \end{pmatrix}$.

The Power of Diagonalisation

A matrix $A$ is diagonalisable if there exists an invertible matrix $P$ and a diagonal matrix $D$ such that $A = PDP^{-1}$. The columns of $P$ are the eigenvectors of $A$, and the diagonal entries of $D$ are the corresponding eigenvalues.

Diagonalisation is crucial in machine learning because it allows for efficient matrix exponentiation. If $A = PDP^{-1}$, then $A^k = PD^kP^{-1}$. Since $D^k$ is simply the diagonal elements raised to the power of $k$, this drastically reduces the computational cost of iterative processes like Markov chains or power iteration methods.

Geometric and Algebraic Multiplicity

The algebraic multiplicity of an eigenvalue is the number of times it appears as a root of the characteristic polynomial. The geometric multiplicity is the dimension of the eigenspace associated with that eigenvalue. A matrix is diagonalisable if and only if the geometric multiplicity equals the algebraic multiplicity for every eigenvalue.

Common Mistakes

1. Forgetting the Identity Matrix: When calculating $\det(A - \lambda I)$, students often subtract $\lambda$ from every element of $A$ instead of just the diagonal. Always remember to subtract $\lambda I$.
2. Normalisation Errors: While any scalar multiple of an eigenvector is also an eigenvector, machine learning applications often require unit vectors. Always normalise your vectors if the algorithm demands it.
3. Assuming All Matrices are Diagonalisable: Not every matrix can be diagonalised. If a matrix is 'defective' (geometric multiplicity < algebraic multiplicity), it cannot be diagonalised.

Frequently Asked Questions

What is the difference between algebraic and geometric multiplicity? Algebraic multiplicity is the count of the root in the characteristic polynomial; geometric multiplicity is the number of linearly independent eigenvectors for that eigenvalue.

Why is diagonalisation useful in machine learning? It simplifies matrix powers and allows us to decouple systems of linear differential equations, which is essential for analysing the convergence of optimisation algorithms.

Can eigenvalues be complex? Yes, even for real matrices, eigenvalues can be complex. However, for real symmetric matrices, which are common in machine learning (e.g., covariance matrices), all eigenvalues are guaranteed to be real.

Conclusion

Understanding eigenvectors, eigenvalues, and diagonalisation is a rite of passage for any serious machine learning student. These concepts provide the mathematical foundation for dimensionality reduction and model stability analysis. To see these concepts in action with interactive visualisations, generate a free animated lesson on this topic at MathInstructor AI today.