Eigenvalues, Diagonalization, and Jordan Form

Similarity and Diagonal Representations

A transformation $t: V \to V$ can look complicated in one basis and simple in another. The best possible representation is diagonal — where the matrix has nonzero entries only on the main diagonal. A diagonal transformation just scales each basis vector independently:

$$t(\vec{\beta}_i) = \lambda_i \vec{\beta}_i$$

When can we find such a basis? The vectors $\vec{\beta}_i$ must be sent to scalar multiples of themselves — they are eigenvectors, and $\lambda_i$ are eigenvalues.

Eigenvalues and Eigenvectors

For a transformation $t: V \to V$, a nonzero vector $\vec{v}$ is an eigenvector with eigenvalue $\lambda$ if:

$$t(\vec{v}) = \lambda\vec{v}$$

In matrix form: $A\vec{v} = \lambda\vec{v}$, or equivalently $(A - \lambda I)\vec{v} = \vec{0}$. For a nonzero $\vec{v}$ to exist, $A - \lambda I$ must be singular:

$$\det(A - \lambda I) = 0$$

This is the characteristic equation. Expanding the determinant gives the characteristic polynomial $p(\lambda)$ — a degree-$n$ polynomial whose roots are the eigenvalues.

Finding Eigenvalues and Eigenvectors

1. Compute $p(\lambda) = \det(A - \lambda I)$ and find its roots $\lambda_1, \ldots, \lambda_k$
2. For each eigenvalue $\lambda_i$, solve $(A - \lambda_i I)\vec{v} = \vec{0}$ by row reduction. The null space $\mathcal{N}(A - \lambda_i I)$ is the eigenspace of $\lambda_i$ — all eigenvectors for that eigenvalue plus $\vec{0}$.

Properties

• $\text{tr}(A) = \sum_i \lambda_i$ (sum of eigenvalues = trace)
• $\det(A) = \prod_i \lambda_i$ (product of eigenvalues = determinant)
• Eigenvectors for distinct eigenvalues are linearly independent
• Real symmetric matrices always have real eigenvalues and orthogonal eigenvectors

Diagonalizability

An $n \times n$ matrix $A$ is diagonalizable if it is similar to a diagonal matrix:

$$A = P\Lambda P^{-1}$$

where $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_n)$ and the columns of $P$ are the corresponding eigenvectors.

When is $A$ diagonalizable?

• $A$ has $n$ linearly independent eigenvectors (equivalently: the eigenvectors span $\mathbb{R}^n$)
• $A$ has $n$ distinct eigenvalues → automatically diagonalizable
• For repeated eigenvalues: $A$ is diagonalizable iff the geometric multiplicity (dimension of the eigenspace) equals the algebraic multiplicity (multiplicity of the root in $p(\lambda)$) for every eigenvalue

Powers and Functions of a Matrix

Diagonalization makes computing powers trivial:

$$A^k = P\Lambda^k P^{-1}, \qquad \Lambda^k = \text{diag}(\lambda_1^k, \ldots, \lambda_n^k)$$

Matrix exponential: $e^A = P\,\text{diag}(e^{\lambda_1}, \ldots, e^{\lambda_n})\,P^{-1}$. This underlies the solution of linear differential equations $\dot{\vec{x}} = A\vec{x}$.

The Jordan Canonical Form

Not all matrices are diagonalizable. The next best thing is the Jordan canonical form — a block-diagonal matrix where each block corresponds to one eigenvalue:

$$J = \begin{bmatrix} J_{n_1}(\lambda_1) & & \\ & \ddots & \\ & & J_{n_k}(\lambda_k) \end{bmatrix}$$

A Jordan block $J_m(\lambda)$ of size $m$ is:

$$J_m(\lambda) = \begin{bmatrix} \lambda & 1 & 0 & \cdots & 0 \\ 0 & \lambda & 1 & \cdots & 0 \\ & & \ddots & \ddots & \\ 0 & 0 & 0 & \lambda & 1 \\ 0 & 0 & 0 & 0 & \lambda \end{bmatrix}$$

Diagonal blocks of size 1 are just the eigenvalue — a diagonalizable matrix has all $1 \times 1$ Jordan blocks. A block of size $> 1$ arises when there are fewer independent eigenvectors than the algebraic multiplicity of the eigenvalue.

Theorem: every square matrix over $\mathbb{C}$ (or over $\mathbb{R}$ if we allow complex eigenvectors) is similar to a Jordan canonical form. The Jordan form is unique up to reordering of blocks.

Nilpotent Matrices

A matrix $N$ is nilpotent if $N^k = 0$ for some $k$. The only eigenvalue of a nilpotent matrix is 0. Every Jordan block with eigenvalue 0 is a nilpotent block; any nilpotent matrix is similar to a block of such blocks.

The Jordan form decomposes any transformation as:

$$A = D + N, \quad DN = ND$$

where $D$ is diagonalizable (semisimple part) and $N$ is nilpotent. This is the Jordan-Chevalley decomposition — the foundation of the theory of matrix functions and linear ODEs.

Connection to Spectral Theory in ML

For symmetric matrices (which covariance matrices always are), the Jordan form is always diagonal with real entries — this is the spectral theorem. The eigendecomposition $A = Q\Lambda Q'$ always exists, with $Q$ orthogonal.

This is why PCA is tractable: the covariance matrix $X'X/(n-1)$ is symmetric positive semi-definite, so it always diagonalizes with non-negative real eigenvalues and orthogonal eigenvectors. The Jordan form (with its potential for non-diagonal blocks) never arises in practice for covariance matrices.