Matrices and Vectors

What Is a Matrix?

A matrix is a rectangular array of numbers arranged in rows and columns. We denote a matrix by a bold capital letter, and its typical element by a subscripted lowercase:

$$A = [a_{ij}] = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1K} \\ a_{21} & a_{22} & \cdots & a_{2K} \\ \vdots & & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nK} \end{bmatrix}$$

The subscript always reads row, then column — so $a_{ij}$ is the element in row $i$, column $j$. This convention is universal and worth internalizing early.

Dimensions

The dimensions of a matrix are its row count and column count, written $n \times K$ (read "n by K"). We say $A$ is an $n \times K$ matrix if it has $n$ rows and $K$ columns.

• When $n = K$, the matrix is square.
• When $n = 1$, the matrix is a row vector.
• When $K = 1$, the matrix is a column vector.

Special Matrix Types

Several structured matrices appear constantly in ML and statistics. Recognizing them on sight saves significant algebra.

Symmetric Matrix

A matrix is symmetric if it equals its own transpose:

$$A \text{ is symmetric} \iff a_{ij} = a_{ji} \text{ for all } i, j$$

Covariance matrices and kernel matrices are always symmetric. This makes them diagonalizable with real eigenvalues — a key fact for PCA and Gaussian geometry.

Diagonal Matrix

A diagonal matrix is a square matrix whose only nonzero entries lie on the main diagonal (top-left to bottom-right):

$$D = \begin{bmatrix} d_1 & 0 & \cdots & 0 \\ 0 & d_2 & \cdots & 0 \\ \vdots & & \ddots & \vdots \\ 0 & 0 & \cdots & d_n \end{bmatrix}$$

Diagonal matrices represent independent scaling along each coordinate axis. In 3D Gaussian Splatting, isotropic Gaussians use diagonal scale matrices.

Scalar Matrix

A scalar matrix is a diagonal matrix with the same value $c$ in every diagonal position: $D = cI$. It scales all directions equally.

Identity Matrix

The identity matrix $I$ (or $I_n$ to specify order $n$) is a scalar matrix with ones on the diagonal:

$$I_3 = \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

It is the multiplicative identity for matrices: $AI = IA = A$ for any conformable $A$.

Triangular Matrices

A lower triangular matrix has zeros everywhere above the main diagonal; an upper triangular matrix has zeros below it:

$$L = \begin{bmatrix} l_{11} & 0 & 0 \\ l_{21} & l_{22} & 0 \\ l_{31} & l_{32} & l_{33} \end{bmatrix} \qquad U = \begin{bmatrix} u_{11} & u_{12} & u_{13} \\ 0 & u_{22} & u_{23} \\ 0 & 0 & u_{33} \end{bmatrix}$$

Triangular matrices are important in LU decomposition and Cholesky factorization of covariance matrices.

Zero (Null) Matrix

The zero matrix $\mathbf{0}$ has every element equal to zero. It is the additive identity: $A + \mathbf{0} = A$.

Vectors

A vector is a matrix with exactly one row or one column.

• A column vector $\mathbf{a}$ is $n \times 1$. Unless otherwise stated, a vector is assumed to be a column vector.
• A row vector $\mathbf{a}'$ is $1 \times n$ — the transpose of the column vector.

$$\mathbf{a} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix}, \qquad \mathbf{a}' = \begin{bmatrix} a_1 & a_2 & \cdots & a_n \end{bmatrix}$$

Vectors carry geometric meaning: you can think of $\mathbf{a}$ as a point in $n$-dimensional space, or as an arrow from the origin to that point.

Indexing Conventions for Rows and Columns

We need consistent notation to refer to individual rows and columns of a matrix $A$:

• $\mathbf{a}_{\cdot k}$ or $\mathbf{a}_k$ denotes column $k$ of $A$ (a column vector)
• $\mathbf{a}_{i \cdot}$ or $\mathbf{a}_i'$ denotes the column vector formed by transposing row $i$ of $A$

So when we write $\mathbf{x}_i$, we typically mean the $i$-th observation as a column vector — the transpose of the $i$-th row of the data matrix $X$. This subscript-based convention eliminates most ambiguity in econometric and ML notation.