Matrix Operations

Equality

Two matrices $A$ and $B$ are equal if and only if they have the same dimensions and every corresponding element is equal:

$$A = B \iff a_{ij} = b_{ij} \text{ for all } i, j$$

Dimension mismatch means not-equal regardless of element values. This seems obvious but matters when checking whether a transformation preserves matrix structure.

Transposition

The transpose of $A$, written $A'$ or $A^\top$, is formed by swapping rows and columns — the $k$-th row of $A$ becomes the $k$-th column of $A'$:

$$B = A' \iff b_{ij} = a_{ji}$$

If $A$ is $n \times K$, then $A'$ is $K \times n$.

Properties of the Transpose

• $(A')' = A$ — transposing twice recovers the original
• $(A + B)' = A' + B'$ — transpose distributes over addition
• $(AB)' = B'A'$ — transpose reverses the order of multiplication
• $(ABC)' = C'B'A'$ — reversal extends to any product length
• $A$ is symmetric $\iff A = A'$

The reversal rule $(AB)' = B'A'$ catches many beginners off guard. It arises from the way inner products work: transposing a product swaps which matrix is pre- and post-multiplied.

Matrix Addition and Subtraction

Addition and subtraction are element-wise operations:

$$C = A + B = [a_{ij} + b_{ij}] \qquad C = A - B = [a_{ij} - b_{ij}]$$

Matrices must have identical dimensions to be added — they are then said to be conformable for addition.

Properties

• Commutative: $A + B = B + A$
• Associative: $(A + B) + C = A + (B + C)$
• Additive identity: $A + \mathbf{0} = A$

Scalar Multiplication

Multiplying a matrix by a scalar $c$ scales every element:

$$cA = [c \cdot a_{ij}]$$

Scalar multiplication commutes: $cA = Ac$. It distributes over addition: $c(A+B) = cA + cB$.

Vector Inner Product

The inner product (dot product) of two column vectors $\mathbf{a}$ and $\mathbf{b}$ of the same length is a scalar:

$$\mathbf{a}'\mathbf{b} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n = \sum_{i=1}^n a_i b_i$$

Note the form: $\mathbf{a}'\mathbf{b}$ is a row vector times a column vector, yielding a $1 \times 1$ scalar. Since each term $a_i b_i = b_i a_i$, the inner product is symmetric:

$$\mathbf{a}'\mathbf{b} = \mathbf{b}'\mathbf{a}$$

Geometrically, $\mathbf{a}'\mathbf{b} = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta$ where $\theta$ is the angle between the vectors. The inner product is zero when vectors are orthogonal — a fundamental concept in least squares and PCA.

Matrix Multiplication

For an $n \times K$ matrix $A$ and a $K \times M$ matrix $B$, the product $C = AB$ is $n \times M$, where element $c_{ik}$ is the inner product of row $i$ of $A$ with column $k$ of $B$:

$$C = AB, \qquad c_{ik} = \mathbf{a}_i' \mathbf{b}_{\cdot k} = \sum_{j=1}^K a_{ij}\, b_{jk}$$

Conformability

The matrices must be conformable for multiplication: the number of columns in $A$ must equal the number of rows in $B$. A useful check: write the dimensions in sequence — $(n \times K)(K \times M)$ — the inner dimensions must match, and the outer dimensions give the result.

Non-Commutativity

Matrix multiplication is generally not commutative:

$$AB \neq BA \text{ in general}$$

In fact, $BA$ may not even be defined (if dimensions don't conform), or it may have different dimensions than $AB$, or it may be defined with the same dimensions but unequal to $AB$. This is why we distinguish pre-multiplication ($B$ is premultiplied by $A$) from post-multiplication.

Properties

• Associative: $(AB)C = A(BC)$
• Distributive: $A(B + C) = AB + AC$
• Identity: $AI = IA = A$
• Zero matrix: $A\mathbf{0} = \mathbf{0}$
• Transpose of product: $(AB)' = B'A'$

Column Interpretation

Each column of $C = AB$ is a linear combination of the columns of $A$, with coefficients drawn from the corresponding column of $B$:

$$\mathbf{c}_{\cdot k} = A\, \mathbf{b}_{\cdot k} = b_{1k}\,\mathbf{a}_{\cdot 1} + b_{2k}\,\mathbf{a}_{\cdot 2} + \cdots + b_{Kk}\,\mathbf{a}_{\cdot K}$$

This column interpretation is one of the most useful ways to think about what a matrix multiplication does: it expresses the columns of the result as combinations of the columns of the left factor.

Sums via Vectors

A column of ones $\mathbf{i} = [1, 1, \ldots, 1]'$ (length $n$) provides compact notation for sums:

$$\sum_{i=1}^n x_i = \mathbf{i}'\mathbf{x}$$

The arithmetic mean follows immediately:

$$\bar{x} = \frac{1}{n}\mathbf{i}'\mathbf{x} = \frac{\mathbf{i}'\mathbf{x}}{n}$$

The sum of squares and cross-products of a data matrix $X$ ($n \times K$) take the matrix form $X'X$, where element $[X'X]_{jk}$ is the inner product of columns $j$ and $k$ of $X$. This $K \times K$ matrix is the workhorse of ordinary least squares.