Singular value decomposition

Every matrix is a rotation, a stretch and a rotation

Eigendecomposition needs a square matrix, and even then it can fail. The singular value decomposition works for every matrix, rectangular or not: $A=U\Sigma V^{T}$ with $U$ and $V$ orthogonal and $\Sigma$ diagonal with non-negative entries $\sigma_1\ge\sigma_2\ge\cdots\ge0$, the singular values. Geometrically $A$ first rotates by $V^{T}$, then stretches each axis $i$ by $\sigma_i$, then rotates by $U$. A unit circle becomes an ellipse whose semi-axes are the singular values.

The columns of $V$ are the eigenvectors of $A^{T}A$ and the columns of $U$ are the eigenvectors of $AA^{T}$, with eigenvalues $\sigma_i^2$. So the SVD is the spectral theorem applied to $A^{T}A$, but computed without ever forming that product, which would square the condition number and lose half the digits. Reading off facts is easy: the rank is the number of non-zero singular values, $\lVert A\rVert_2=\sigma_1$ is the largest stretch, the Frobenius norm squared is the sum of the squared singular values, $|\det A|=\prod\sigma_i$ for a square matrix, and the condition number $\sigma_1/\sigma_{\min}$ measures how much $A$ can amplify errors. For a $m\times n$ data matrix with $m\ge n$, the full $U$ is $m\times m$, $\Sigma$ is $m\times n$ and $V$ is $n\times n$.

• SVD: every matrix is $U\Sigma V^{T}$ with orthogonal $U$, $V$ and non-negative sorted singular values.
• Facts: rank is the count of non-zero $\sigma_i$, $\lVert A\rVert_F^2=\sum\sigma_i^2$, $|\det A|=\prod\sigma_i$.
• Condition number: $\sigma_1/\sigma_{\min}$; a large value means unstable solves and unreliable inverses.

Lab · what's graded

• The largest singular value20 pts
• The Frobenius identity10 pts
• The product of singular values10 pts
• Shapes of a full SVD15 pts
• Singular values and eigenvalues15 pts
• A condition number15 pts
• Why not eigendecompose AᵀA15 pts