Orthogonality and projections

Right angles and shadows

Two vectors are orthogonal when their dot product is zero, $\mathbf{u}\cdot\mathbf{v}=0$: they meet at a right angle. The length of $\mathbf{u}$ is $\|\mathbf{u}\|=\sqrt{\mathbf{u}\cdot\mathbf{u}}$. Orthogonality is what makes geometry computable: Pythagoras holds, $\|\mathbf{u}+\mathbf{v}\|^2=\|\mathbf{u}\|^2+\|\mathbf{v}\|^2$ when $\mathbf{u}\perp\mathbf{v}$, and uncorrelated returns are orthogonal once their means are removed.

The projection of $\mathbf{b}$ onto the line through $\mathbf{a}$ is its shadow $\mathbf{p}=c\,\mathbf{a}$, the closest point of the line to $\mathbf{b}$. The leftover $\mathbf{b}-\mathbf{p}$ is orthogonal to $\mathbf{a}$, which fixes $c=\frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{a}\cdot\mathbf{a}}$. A projection matrix $P$ does this for every $\mathbf{b}$: $\mathbf{p}=P\mathbf{b}$, with $P^{T}=P$ and $P^2=P$ (projecting twice changes nothing). Its eigenvalues are only 0 and 1.

• Orthogonal: $\mathbf{u}\cdot\mathbf{v}=0$. Orthonormal: orthogonal and each of length 1.
• Projection: the closest point of a subspace; the leftover is orthogonal to the subspace.
• Orthonormal columns ($Q^{T}Q=I$) keep lengths and angles, so they never amplify errors.

Lab · what's graded

• The projection coefficient15 pts
• A projection matrix20 pts
• The size of the leftover15 pts
• Properties of projection matrices15 pts
• Spot the orthogonal pair10 pts
• Residuals and regressors25 pts