Eigenvalues, Eigenvectors, and Decompositions

Eigenvalues and Eigenvectors

When a matrix $A$ multiplies most vectors, both the direction and magnitude change. An eigenvector is a special nonzero vector whose direction is preserved — it is only scaled:

$$A\mathbf{q} = \lambda \mathbf{q}$$

The scalar $\lambda$ is the corresponding eigenvalue. The eigenvector gets stretched ($|\lambda| > 1$), compressed ($|\lambda| < 1$), reversed ($\lambda < 0$), or left unchanged ($\lambda = 1$) — but never rotated off its line.

Finding Eigenvalues: Characteristic Equation

Rearranging $A\mathbf{q} = \lambda\mathbf{q}$ gives $(A - \lambda I)\mathbf{q} = \mathbf{0}$. For a nonzero solution $\mathbf{q}$ to exist, the matrix $A - \lambda I$ must be singular:

$$\det(A - \lambda I) = 0$$

This is the characteristic equation. For an $n \times n$ matrix it produces a degree-$n$ polynomial in $\lambda$ — the characteristic polynomial — whose roots are the $n$ eigenvalues (counted with multiplicity, possibly complex).

Example: 2×2 matrix

For $A = \begin{bmatrix} 3 & 1 \\ 0 & 2 \end{bmatrix}$:

$$\det(A - \lambda I) = (3-\lambda)(2-\lambda) - 0 = \lambda^2 - 5\lambda + 6 = (\lambda-2)(\lambda-3) = 0$$

Eigenvalues: $\lambda_1 = 2$, $\lambda_2 = 3$. For each, substitute back to find the eigenvector by solving $(A - \lambda I)\mathbf{q} = \mathbf{0}$.

Properties

• $\text{tr}(A) = \sum_i \lambda_i$ — trace equals sum of eigenvalues
• $\det(A) = \prod_i \lambda_i$ — determinant equals product of eigenvalues
• If $\lambda = 0$ is an eigenvalue, $A$ is singular (verifies the determinant connection)
• Eigenvalues of $A^{-1}$ are $1/\lambda_i$
• Eigenvalues of $A^k$ are $\lambda_i^k$

Symmetric Matrices: Spectral Theorem

For symmetric matrices ($A = A'$), the eigenstructure is especially clean. The spectral theorem guarantees:

1. All eigenvalues are real
2. Eigenvectors corresponding to distinct eigenvalues are orthogonal
3. There exists a complete orthonormal basis of eigenvectors

This means any symmetric $n \times n$ matrix can be decomposed as:

$$A = Q \Lambda Q'$$

where $Q$ is an orthogonal matrix whose columns are the eigenvectors, and $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ is the diagonal matrix of eigenvalues. This is the eigendecomposition (or spectral decomposition).

Since $Q$ is orthogonal ($Q'Q = I$), this can also be written:

$$A = \sum_{i=1}^n \lambda_i \mathbf{q}_i \mathbf{q}_i'$$

Each term $\lambda_i \mathbf{q}_i \mathbf{q}_i'$ is a rank-1 matrix — a scaled outer product. The full matrix is a sum of $n$ rank-1 pieces along orthogonal directions.

Positive Definite Matrices

A symmetric matrix $A$ is positive definite (PD) if:

$$\mathbf{x}'A\mathbf{x} > 0 \quad \text{for all nonzero } \mathbf{x}$$

Equivalently: all eigenvalues are strictly positive. It is positive semi-definite (PSD) if $\mathbf{x}'A\mathbf{x} \geq 0$ (eigenvalues $\geq 0$).

Covariance matrices are always PSD (PD when the data has full rank). The 3DGS covariance $\Sigma = RSS'R'$ is PSD by construction — $S$ is diagonal with non-negative entries, so $SS'$ has non-negative eigenvalues, and orthogonal $R$ preserves the sign.

Singular Value Decomposition (SVD)

The eigendecomposition requires a square matrix. The singular value decomposition generalizes it to any $n \times K$ matrix:

$$A = U \Sigma V'$$

where:

• $U$ is $n \times n$ orthogonal — left singular vectors (columns span the column space of $A$)
• $\Sigma$ is $n \times K$ diagonal — singular values $\sigma_1 \geq \sigma_2 \geq \cdots \geq 0$ on the diagonal
• $V$ is $K \times K$ orthogonal — right singular vectors (columns span the row space of $A$)

The singular values are the square roots of the eigenvalues of $A'A$ (equivalently $AA'$). The number of nonzero singular values equals the rank of $A$.

Truncated SVD and Low-Rank Approximation

Keeping only the top $r$ singular values gives the best rank-$r$ approximation of $A$ in the Frobenius norm:

$$A_r = \sum_{i=1}^r \sigma_i \mathbf{u}_i \mathbf{v}_i'$$

This is the mathematical foundation of PCA, latent semantic analysis, and matrix factorization methods. Throwing away small singular values discards the directions of least variance while retaining the most informative structure.

Application: PCA via Eigendecomposition

Given a data matrix $X$ ($n$ observations, $K$ features, mean-centered), the sample covariance matrix is:

$$S = \frac{1}{n-1} X'X$$

$S$ is $K \times K$, symmetric, and positive semi-definite. Its eigendecomposition $S = Q\Lambda Q'$ gives:

• Principal components: the eigenvectors $\mathbf{q}_1, \ldots, \mathbf{q}_K$ (columns of $Q$) — orthogonal directions of maximum variance
• Explained variance: the eigenvalue $\lambda_i$ is the variance of the data projected onto $\mathbf{q}_i$
• Projection: $Z = XQ_r$ (where $Q_r$ keeps the top $r$ eigenvectors) gives the $r$-dimensional PCA embedding

In 3DGS, each Gaussian's scale parameters $(s_x, s_y, s_z)$ define the square roots of the eigenvalues of the Gaussian's covariance — they directly encode the variance along the three principal axes of the ellipsoid.

Cholesky Decomposition

For a positive definite matrix $A$, the Cholesky decomposition is:

$$A = LL'$$

where $L$ is a lower triangular matrix with positive diagonal entries. It is the matrix analogue of the square root. Cholesky is numerically preferred over full eigendecomposition when you only need a factored form — it runs in $O(n^3/3)$ rather than $O(n^3)$ and is more numerically stable. It is used in multivariate normal sampling and as a preconditioner in optimization.