Vector Spaces and Geometry

Geometric Thinking About Matrices

The previous readings treated matrices as arrays of numbers obeying algebraic rules. This reading builds the geometric picture — what matrices do to space — which is essential for understanding covariance, projections, and transformations in ML.

Vector Spaces

A vector space $V$ is a set of objects (vectors) that can be added together and scaled by scalars, with results that stay in the set. The formal axioms are satisfied by all the vector types we use: column vectors in $\mathbb{R}^n$, matrices, functions.

The standard example: $\mathbb{R}^n$, the set of all $n$-dimensional column vectors with real entries.

Linear Combinations

A linear combination of vectors $\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_K$ is any vector of the form:

$$c_1 \mathbf{a}_1 + c_2 \mathbf{a}_2 + \cdots + c_K \mathbf{a}_K$$

where $c_1, \ldots, c_K$ are scalars. This is the fundamental building block of linear algebra.

From the previous reading: the product $A\mathbf{b}$ is a linear combination of the columns of $A$ with coefficients from $\mathbf{b}$. Every matrix-vector product is a linear combination.

Span

The span of a set of vectors $\{\mathbf{a}_1, \ldots, \mathbf{a}_K\}$ is the set of all linear combinations of those vectors:

$$\text{span}\{\mathbf{a}_1, \ldots, \mathbf{a}_K\} = \left\{ \sum_{k=1}^K c_k \mathbf{a}_k \;:\; c_k \in \mathbb{R} \right\}$$

Geometrically:

• Span of one nonzero vector = a line through the origin
• Span of two linearly independent vectors in $\mathbb{R}^3$ = a plane through the origin
• Span of $n$ linearly independent vectors in $\mathbb{R}^n$ = all of $\mathbb{R}^n$

The span of the columns of a matrix $A$ is called the column space of $A$, written $\mathcal{C}(A)$. The system $A\mathbf{x} = \mathbf{b}$ has a solution if and only if $\mathbf{b} \in \mathcal{C}(A)$.

Linear Independence

Vectors $\mathbf{a}_1, \ldots, \mathbf{a}_K$ are linearly independent if the only solution to:

$$c_1 \mathbf{a}_1 + c_2 \mathbf{a}_2 + \cdots + c_K \mathbf{a}_K = \mathbf{0}$$

is $c_1 = c_2 = \cdots = c_K = 0$. Otherwise they are linearly dependent — at least one vector is a linear combination of the others and contributes no new direction.

Why this matters for ML: a covariance matrix whose columns are linearly dependent is singular (non-invertible). PCA exploits linear dependence to find lower-dimensional representations.

Basis and Dimension

A basis for a vector space $V$ is a set of vectors that:

1. Spans $V$ — every vector in $V$ is a linear combination of the basis vectors
2. Is linearly independent — no basis vector is redundant

The number of vectors in any basis for $V$ is the dimension of $V$, written $\dim(V)$.

The standard basis for $\mathbb{R}^n$ is $\{\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_n\}$, where $\mathbf{e}_k$ has a 1 in position $k$ and zeros elsewhere. Any vector $\mathbf{x} \in \mathbb{R}^n$ is uniquely expressed as $\mathbf{x} = x_1 \mathbf{e}_1 + \cdots + x_n \mathbf{e}_n$ — the coordinates are the coefficients.

Key fact: if a set of $n$ vectors in $\mathbb{R}^n$ is linearly independent, it automatically spans $\mathbb{R}^n$ and is therefore a basis.

Column Space, Row Space, and Null Space

Three fundamental subspaces are associated with any $n \times K$ matrix $A$:

The null space and row space are orthogonal complements in $\mathbb{R}^K$: every vector in the null space is perpendicular to every vector in the row space, and their dimensions sum to $K$.

Matrices as Linear Transformations

Multiplying by an $n \times K$ matrix $A$ defines a linear map $T: \mathbb{R}^K \to \mathbb{R}^n$, written $T(\mathbf{x}) = A\mathbf{x}$. The map is linear because:

$$A(c\mathbf{x} + d\mathbf{y}) = c A\mathbf{x} + d A\mathbf{y}$$

Every linear map between finite-dimensional spaces can be represented as a matrix. This correspondence — matrix ↔ linear transformation — is the central insight of linear algebra.

Geometric operations as matrices:

• Rotation by angle $\theta$ in $\mathbb{R}^2$: $\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$
• Scaling axes independently: diagonal matrix with scale factors on the diagonal
• Projection onto a subspace: an idempotent matrix ($P^2 = P$)
• Reflection: orthogonal matrix with determinant $-1$

In 3D Gaussian Splatting, the covariance matrix $\Sigma = RSS'R'$ decomposes the Gaussian's shape into a rotation ($R$) applied to an independent scaling ($S$) — a direct application of these geometric operations.

Orthogonality

Two vectors are orthogonal if their inner product is zero: $\mathbf{a}'\mathbf{b} = 0$. Orthogonality generalizes perpendicularity to any dimension.

A set of vectors is orthonormal if they are mutually orthogonal and each has unit length ($\|\mathbf{a}_k\| = 1$). An orthonormal basis is maximally convenient: coordinates are computed by simple inner products, and lengths are preserved under transformation.

A square matrix $Q$ is orthogonal if its columns form an orthonormal set:

$$Q'Q = QQ' = I$$

Orthogonal matrices represent pure rotations (and reflections) — they preserve lengths and angles. The eigendecomposition of a symmetric matrix (Reading 5) is built entirely from orthogonal matrices.