Mastering Singular Value Decomposition: A Guide for Undergraduates

Introduction to Singular Value Decomposition

In your undergraduate linear algebra modules, you have likely encountered the eigendecomposition of square, symmetric matrices. However, what happens when a matrix is non-square or non-diagonalisable? This is where the Singular Value Decomposition (SVD) becomes an essential tool. SVD is a fundamental matrix factorisation that decomposes any $m \times n$ matrix into three distinct components, revealing the underlying geometric structure of the linear transformation.

Understanding SVD is not just an academic exercise; it is the backbone of modern data science, signal processing, and numerical analysis. For your exams, mastering SVD allows you to solve problems involving rank, pseudo-inverses, and low-rank approximations with confidence. This article will guide you through the mechanics of SVD, ensuring you can perform the decomposition manually and interpret the results effectively.

The Definition of SVD

For any real $m \times n$ matrix $A$, the Singular Value Decomposition is defined as:

$$A = U \Sigma V^T$$

Where:

• $U$ is an $m \times m$ orthogonal matrix whose columns are the left singular vectors of $A$.
• $\Sigma$ is an $m \times n$ diagonal matrix containing the singular values $\sigma_i$ in descending order.
• $V^T$ is the transpose of an $n \times n$ orthogonal matrix $V$, whose columns are the right singular vectors of $A$.

Geometrically, this represents a rotation ($V^T$), followed by a scaling ($\Sigma$), and a final rotation ($U$). Because $U$ and $V$ are orthogonal, they preserve lengths and angles, meaning the entire transformation's distortion is captured solely by the diagonal entries of $\Sigma$.

Step-by-Step Calculation Procedure

To compute the SVD of a matrix $A$ manually, follow these steps:

1. Compute $A^T A$: This is a symmetric $n \times n$ matrix.
2. Find Eigenvalues and Eigenvectors of $A^T A$: The eigenvalues $\lambda_i$ are related to the singular values by $\sigma_i = \sqrt{\lambda_i}$. The eigenvectors form the columns of $V$.
3. Construct $\Sigma$: Place the singular values $\sigma_i$ on the diagonal of an $m \times n$ matrix.
4. Find $U$: Use the relation $u_i = \frac{1}{\sigma_i} A v_i$ for non-zero singular values. If $m > n$, use the Gram-Schmidt process to complete the basis for $U$.

Worked Example 1: A Simple 2x2 Matrix

Let $A = \begin{pmatrix} 3 & 0 \ 4 & 5 \end{pmatrix}$.

Step 1: $A^T A$ $$A^T A = \begin{pmatrix} 3 & 4 \ 0 & 5 \end{pmatrix} \begin{pmatrix} 3 & 0 \ 4 & 5 \end{pmatrix} = \begin{pmatrix} 25 & 20 \ 20 & 25 \end{pmatrix}$$

Step 2: Eigenvalues of $A^T A$ Characteristic equation: $\det(A^T A - \lambda I) = (25-\lambda)^2 - 400 = 0$. $(25-\lambda)^2 = 400 \implies 25-\lambda = \pm 20$. $\lambda_1 = 45, \lambda_2 = 5$. Thus, $\sigma_1 = \sqrt{45} = 3\sqrt{5}$ and $\sigma_2 = \sqrt{5}$.

Step 3: Eigenvectors ($V$) For $\lambda_1 = 45$: $\begin{pmatrix} -20 & 20 \ 20 & -20 \end{pmatrix} v_1 = 0 \implies v_1 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \ 1 \end{pmatrix}$. For $\lambda_2 = 5$: $\begin{pmatrix} 20 & 20 \ 20 & 20 \end{pmatrix} v_2 = 0 \implies v_2 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \ -1 \end{pmatrix}$.

Step 4: Construct $U$ $u_1 = \frac{1}{\sigma_1} A v_1 = \frac{1}{3\sqrt{5}} \begin{pmatrix} 3 & 0 \ 4 & 5 \end{pmatrix} \begin{pmatrix} 1/\sqrt{2} \ 1/\sqrt{2} \end{pmatrix} = \frac{1}{3\sqrt{10}} \begin{pmatrix} 3 \ 9 \end{pmatrix} = \frac{1}{\sqrt{10}} \begin{pmatrix} 1 \ 3 \end{pmatrix}$. $u_2$ is orthogonal to $u_1$, so $u_2 = \frac{1}{\sqrt{10}} \begin{pmatrix} 3 \ -1 \end{pmatrix}$.

Worked Example 2: Rank-One Matrix

Let $A = \begin{pmatrix} 1 & 1 \ 1 & 1 \end{pmatrix}$.

$A^T A = \begin{pmatrix} 2 & 2 \ 2 & 2 \end{pmatrix}$. Eigenvalues: $\det(A^T A - \lambda I) = (2-\lambda)^2 - 4 = 0 \implies \lambda_1 = 4, \lambda_2 = 0$. Singular values: $\sigma_1 = 2, \sigma_2 = 0$. $v_1 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \ 1 \end{pmatrix}, v_2 = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \ -1 \end{pmatrix}$. $u_1 = \frac{1}{2} \begin{pmatrix} 1 & 1 \ 1 & 1 \end{pmatrix} \begin{pmatrix} 1/\sqrt{2} \ 1/\sqrt{2} \end{pmatrix} = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 \ 1 \end{pmatrix}$. This demonstrates that for rank-deficient matrices, some singular values are zero.

Common Mistakes

1. Forgetting to normalise: Always ensure your eigenvectors in $V$ and $U$ are unit vectors (length 1). Failing to do so violates the definition of orthogonal matrices.
2. Incorrect ordering: Singular values must be placed in $\Sigma$ in descending order ($\sigma_1 \ge \sigma_2 \ge \dots$). If you swap them, you must also swap the corresponding columns in $U$ and $V$.
3. Sign errors in $U$: When calculating $u_i = \frac{1}{\sigma_i} A v_i$, ensure the sign of $u_i$ is consistent with the transformation. If you choose a different sign for an eigenvector, it must be reflected in the corresponding column of $U$.

Frequently Asked Questions

Is SVD unique? The singular values are unique, but the singular vectors are not; they can be multiplied by a phase factor (or sign) as long as the product $U \Sigma V^T$ remains unchanged.

How does SVD relate to eigendecomposition? Eigendecomposition only applies to square matrices and requires a basis of eigenvectors. SVD applies to any matrix and always uses an orthonormal basis.

Why is SVD used in data compression? By keeping only the largest singular values and setting others to zero, we can create a low-rank approximation of a matrix, significantly reducing storage while retaining the most important features.

Conclusion

Singular Value Decomposition is a powerful, versatile tool that every mathematics undergraduate should master. By breaking down complex transformations into rotation and scaling, it provides deep insights into the nature of linear systems. To see these concepts in motion, visit MathInstructor AI to generate a free, narrated animated lesson on SVD tailored to your specific study needs.