Determinants and Matrix Rank

The Determinant

The determinant of a square matrix $A$, written $\det(A)$ or $|A|$, is a scalar that encodes whether $A$ is invertible and by how much it stretches or compresses space.

2×2 Case

For a $2 \times 2$ matrix:

$$\det\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc$$

Geometrically: if the two rows are interpreted as vectors in $\mathbb{R}^2$, the determinant equals the signed area of the parallelogram they span. If the rows are parallel (linearly dependent), the parallelogram collapses to a line — area zero, determinant zero.

3×3 Case (Cofactor Expansion)

For a $3 \times 3$ matrix, expand along the first row:

$$\det(A) = a_{11}(a_{22}a_{33} - a_{23}a_{32}) - a_{12}(a_{21}a_{33} - a_{23}a_{31}) + a_{13}(a_{21}a_{32} - a_{22}a_{31})$$

This pattern — alternating signs, each term multiplying a row element by the determinant of the submatrix formed by deleting that element's row and column — is cofactor expansion and generalizes to any $n \times n$ matrix.

Properties

• $\det(I) = 1$
• $\det(A') = \det(A)$
• $\det(AB) = \det(A)\det(B)$
• $\det(cA) = c^n \det(A)$ for an $n \times n$ matrix
• Swapping two rows negates the determinant
• Adding a multiple of one row to another leaves the determinant unchanged
• A matrix with two identical rows has determinant zero

Geometric interpretation in $n$ dimensions

For an $n \times n$ matrix $A$, $|\det(A)|$ is the $n$-dimensional volume of the parallelepiped formed by the rows (or columns) of $A$. The sign encodes orientation. A linear transformation $\mathbf{x} \mapsto A\mathbf{x}$ scales all volumes by $|\det(A)|$.

Invertibility and the Determinant

A square matrix $A$ is invertible (also called non-singular) if and only if $\det(A) \neq 0$.

When $\det(A) = 0$:

• $A$ is singular — it cannot be inverted
• The columns of $A$ are linearly dependent
• The transformation $A\mathbf{x}$ maps all of $\mathbb{R}^n$ into a lower-dimensional subspace
• The system $A\mathbf{x} = \mathbf{b}$ either has no solution or infinitely many

In ML: a singular covariance matrix signals that your data lives in a lower-dimensional subspace than assumed — some features are exact linear combinations of others.

Matrix Rank

The rank of a matrix $A$, written $\text{rank}(A)$, is the dimension of its column space — equivalently, the number of linearly independent columns (which always equals the number of linearly independent rows).

For an $n \times K$ matrix:

• $\text{rank}(A) \leq \min(n, K)$
• If $\text{rank}(A) = \min(n, K)$, the matrix has full rank
• For a square $n \times n$ matrix: $A$ is invertible $\iff \text{rank}(A) = n$

Rank-Nullity Theorem

For an $n \times K$ matrix $A$:

$$\text{rank}(A) + \text{nullity}(A) = K$$

where $\text{nullity}(A) = \dim(\mathcal{N}(A))$ is the dimension of the null space. Every column "direction" either contributes to the output (rank) or gets killed to zero (null space) — the two together always account for all $K$ input dimensions.

Idempotent Matrices

A matrix $M$ is idempotent if $M^2 = MM = M$. Idempotent matrices represent projections: applying the transformation twice gives the same result as applying it once, because the output is already in the target subspace.

The mean-deviation matrix

$$M^0 = I - \frac{1}{n}\mathbf{i}\mathbf{i}'$$

(where $\mathbf{i}$ is the $n\times 1$ vector of ones) is a symmetric idempotent matrix. Pre-multiplying a data vector $\mathbf{x}$ by $M^0$ produces the mean-deviation form $\mathbf{x} - \bar{x}\mathbf{i}$. This matrix appears throughout regression and ANOVA.

For any symmetric idempotent matrix $M$:

• $\text{rank}(M) = \text{tr}(M)$ (trace equals rank)
• Its eigenvalues are all 0 or 1
• It represents an orthogonal projection onto its column space

Trace

The trace of a square matrix is the sum of its diagonal elements:

$$\text{tr}(A) = \sum_{i=1}^n a_{ii}$$

Key properties:

• $\text{tr}(A + B) = \text{tr}(A) + \text{tr}(B)$
• $\text{tr}(AB) = \text{tr}(BA)$ (cyclic property — even when $AB \neq BA$)
• $\text{tr}(A) = \sum_i \lambda_i$ where $\lambda_i$ are the eigenvalues of $A$

The cyclic property makes trace useful for simplifying quadratic forms: $\mathbf{x}'A\mathbf{x} = \text{tr}(A\mathbf{x}\mathbf{x}')$, which can be easier to differentiate.