Orthogonality and Projection

Orthogonality

Two vectors $\vec{u}, \vec{v} \in \mathbb{R}^n$ are orthogonal if their dot product is zero:

$$\vec{u} \perp \vec{v} \iff \vec{u} \cdot \vec{v} = \sum_{i=1}^n u_i v_i = 0$$

The norm (length) of a vector is $\|\vec{v}\| = \sqrt{\vec{v} \cdot \vec{v}}$. A unit vector has $\|\hat{v}\| = 1$; any nonzero vector can be normalized: $\hat{v} = \vec{v}/\|\vec{v}\|$.

A set $\{\vec{q}_1, \ldots, \vec{q}_k\}$ is orthonormal if $\vec{q}_i \cdot \vec{q}_j = \delta_{ij}$ (1 if $i=j$, 0 otherwise). Orthonormal sets are automatically linearly independent.

Orthogonal Projection onto a Line

Given a line through the origin spanned by $\vec{v}$, the orthogonal projection of $\vec{u}$ onto this line is the point on the line closest to $\vec{u}$:

$$\text{proj}_{\vec{v}}(\vec{u}) = \frac{\vec{u} \cdot \vec{v}}{\vec{v} \cdot \vec{v}}\,\vec{v}$$

The scalar $\frac{\vec{u} \cdot \vec{v}}{\|\vec{v}\|^2}$ is the signed component of $\vec{u}$ along $\vec{v}$. The error (residual) $\vec{u} - \text{proj}_{\vec{v}}(\vec{u})$ is orthogonal to $\vec{v}$ — this is the defining geometric property of orthogonal projection.

As a matrix: for a unit vector $\hat{v}$, the projection is $\hat{v}(\hat{v}'\vec{u}) = (\hat{v}\hat{v}')\vec{u}$. The matrix $P = \hat{v}\hat{v}'$ is the rank-1 projection matrix onto the line. It satisfies $P^2 = P$ (idempotent) and $P = P'$ (symmetric).

Gram-Schmidt Orthogonalization

Gram-Schmidt converts any basis $\{\vec{v}_1, \ldots, \vec{v}_n\}$ into an orthonormal basis $\{\vec{q}_1, \ldots, \vec{q}_n\}$ spanning the same space:

Step 1: $\vec{u}_1 = \vec{v}_1$, then $\vec{q}_1 = \vec{u}_1/\|\vec{u}_1\|$

Step $k$ (for $k = 2, 3, \ldots, n$): subtract the projections onto all previous $\vec{q}_i$:

$$\vec{u}_k = \vec{v}_k - \sum_{i=1}^{k-1} (\vec{v}_k \cdot \vec{q}_i)\,\vec{q}_i, \qquad \vec{q}_k = \vec{u}_k / \|\vec{u}_k\|$$

Each $\vec{u}_k$ is the component of $\vec{v}_k$ orthogonal to the subspace spanned by $\vec{v}_1, \ldots, \vec{v}_{k-1}$.

QR decomposition: Gram-Schmidt applied to the columns of $A$ produces $A = QR$ where $Q$ has orthonormal columns and $R$ is upper triangular. This is used in least-squares solvers and eigenvalue algorithms.

Projection into a Subspace

Let $W$ be a subspace of $\mathbb{R}^n$ with orthonormal basis $\{\vec{q}_1, \ldots, \vec{q}_k\}$. The orthogonal projection of $\vec{u}$ onto $W$ is:

$$\text{proj}_W(\vec{u}) = \sum_{i=1}^k (\vec{u} \cdot \vec{q}_i)\,\vec{q}_i = QQ'\vec{u}$$

where $Q = [\vec{q}_1 \mid \cdots \mid \vec{q}_k]$. The projection matrix $P_W = QQ'$ satisfies:

• $P_W^2 = P_W$ (projecting twice does nothing)
• $P_W' = P_W$ (it is symmetric)
• $\text{rank}(P_W) = k = \dim(W)$

The complement $\vec{u} - P_W \vec{u}$ is orthogonal to every vector in $W$.

Least Squares: The Geometry

When the system $A\vec{x} = \vec{b}$ has no solution (as happens in overdetermined systems with more equations than unknowns), the best we can do is find $\hat{x}$ minimizing $\|A\hat{x} - \vec{b}\|^2$.

Geometrically: $A\hat{x}$ is the projection of $\vec{b}$ onto the column space of $A$. The residual $\vec{b} - A\hat{x}$ must be orthogonal to all columns of $A$:

$$A'(\vec{b} - A\hat{x}) = \vec{0} \implies A'A\hat{x} = A'\vec{b}$$

These are the normal equations. When $A$ has full column rank, $A'A$ is invertible and the unique least-squares solution is:

$$\hat{x} = (A'A)^{-1}A'\vec{b}$$

The matrix $P = A(A'A)^{-1}A'$ is the projection matrix onto the column space of $A$ — the hat matrix familiar from regression. This is the linear algebra foundation of ordinary least squares, and by extension of any optimization problem whose solution is a projection.