Vector Spaces and Subspaces

Why Axiomatize?

In the previous readings, vectors were $n$-tuples of reals. But "addition" and "scalar multiplication" appear in many other settings — polynomials, functions, matrices. The concept of a vector space abstracts the shared structure, letting us prove theorems once that apply everywhere.

Definition of a Vector Space

A vector space over $\mathbb{R}$ is a set $V$ equipped with two operations — addition $\vec{v} + \vec{w}$ and scalar multiplication $c\vec{v}$ — satisfying ten axioms:

Addition axioms:

1. Closure: $\vec{v} + \vec{w} \in V$
2. Commutativity: $\vec{v} + \vec{w} = \vec{w} + \vec{v}$
3. Associativity: $(\vec{v} + \vec{w}) + \vec{u} = \vec{v} + (\vec{w} + \vec{u})$
4. Zero element: there exists $\vec{0} \in V$ such that $\vec{v} + \vec{0} = \vec{v}$
5. Additive inverse: for each $\vec{v}$ there exists $-\vec{v}$ such that $\vec{v} + (-\vec{v}) = \vec{0}$

Scalar multiplication axioms:

6. Closure: $c\vec{v} \in V$
7. Distributivity over vector addition: $c(\vec{v} + \vec{w}) = c\vec{v} + c\vec{w}$
8. Distributivity over scalar addition: $(c + d)\vec{v} = c\vec{v} + d\vec{v}$
9. Associativity: $(cd)\vec{v} = c(d\vec{v})$
10. Identity: $1\vec{v} = \vec{v}$

The axioms capture exactly what it means for a set to behave "like" $\mathbb{R}^n$.

Canonical Examples

$\mathbb{R}^n$: column vectors with component-wise operations. The archetypal vector space.

$\mathcal{M}_{m \times n}$: $m \times n$ matrices with component-wise addition and scalar scaling. The zero vector is the zero matrix.

$\mathcal{P}_n$: polynomials of degree $\leq n$, with coefficient-wise addition. E.g., $(1 + 2x) + (3 - x) = 4 + x$. The zero vector is the zero polynomial.

Function spaces: the set of all functions $f: \mathbb{R} \to \mathbb{R}$ with $(f+g)(x) = f(x)+g(x)$ and $(cf)(x) = c\cdot f(x)$. This is an infinite-dimensional vector space. Fourier analysis lives here.

Trivial space: $\{\vec{0}\}$ with the only possible operations. Dimension zero.

Subspaces

A subspace of $V$ is a subset $W \subseteq V$ that is itself a vector space under the same operations. Rather than checking all ten axioms, a subset $W$ is a subspace if and only if:

1. $\vec{0} \in W$
2. Closed under addition: $\vec{u}, \vec{v} \in W \Rightarrow \vec{u} + \vec{v} \in W$
3. Closed under scalar multiplication: $\vec{v} \in W, c \in \mathbb{R} \Rightarrow c\vec{v} \in W$

Conditions 2 and 3 can be combined: $W$ is a subspace iff for all $\vec{u}, \vec{v} \in W$ and scalars $a, b$: $a\vec{u} + b\vec{v} \in W$.

Examples of subspaces of $\mathbb{R}^3$:

• $\{\vec{0}\}$ (trivial)
• Any line through the origin: $\{t\vec{v} : t \in \mathbb{R}\}$
• Any plane through the origin: $\{s\vec{u} + t\vec{v} : s, t \in \mathbb{R}\}$
• All of $\mathbb{R}^3$

Non-examples: a line not through the origin is not a subspace (it doesn't contain $\vec{0}$). The set of vectors with non-negative entries is not a subspace (not closed under negation).

Spanning Sets

The span of a set $S = \{\vec{v}_1, \ldots, \vec{v}_k\}$ is the set of all linear combinations:

$$[S] = \text{span}(S) = \{c_1 \vec{v}_1 + c_2 \vec{v}_2 + \cdots + c_k \vec{v}_k : c_i \in \mathbb{R}\}$$

The span is always a subspace. We say $S$ spans $W$ if $[S] = W$.

When looking for a spanning set for a space arising from a linear system, write the solution set in parametric form — the vectors multiplied by the free parameters form a spanning set for the null space.

Checking Span Membership with Gauss's Method

To test whether $\vec{v}$ lies in $\text{span}\{\vec{u}_1, \ldots, \vec{u}_k\}$, ask: do scalars $a_1, \ldots, a_k$ exist such that

$$a_1 \vec{u}_1 + \cdots + a_k \vec{u}_k = \vec{v}\,?$$

Assemble $A = [\vec{u}_1 \mid \cdots \mid \vec{u}_k]$ (columns are the spanning vectors) and apply Gauss's method to $[A \mid \vec{v}]$. If the system is consistent, $\vec{v}$ is in the span; if it is inconsistent, it is not.

Example. Is $\vec{v} = (5,4)$ in $\text{span}\{\vec{u}_1=(1,1),\,\vec{u}_2=(2,1)\}$?

$$a\begin{pmatrix}1\\1\end{pmatrix}+b\begin{pmatrix}2\\1\end{pmatrix}=\begin{pmatrix}5\\4\end{pmatrix}\quad\Longrightarrow\quad\left[\begin{array}{rr|r}1&2&5\\1&1&4\end{array}\right]$$

The reduction $\rho_2 \leftarrow \rho_2 - \rho_1$ gives $b=1$, then back-substitution gives $a=3$, so $(5,4) = 3\vec{u}_1 + 1\vec{u}_2$. Replacing the right-hand side with a general $(x,y)$ shows a solution always exists, confirming $\text{span}\{\vec{u}_1,\vec{u}_2\} = \mathbb{R}^2$.

Computing Span Membership with einsum

Once you have the coefficients, einsum is the idiomatic way to verify a linear combination in both PyTorch and TensorFlow. The subscript 'ij,j->i' means: multiply column $j$ of $A$ by scalar $c_j$ and sum over $j$ to produce output index $i$.

import numpy as np

# Spanning vectors as *columns* of A
A = np.array([[1., 2.],
              [1., 1.]])
v = np.array([5., 4.])

# Solve A @ c = v  (find coefficients)
c = np.linalg.solve(A, v)
print('coefficients:', c)          # [3. 1.]

# Verify: einsum 'ij,j->i' means sum_j A_ij * c_j
v_hat = np.einsum('ij,j->i', A, c)
print('reconstructed v:', v_hat)    # [5. 4.]

# Does the span cover all of R^2?
# Yes iff det(A) != 0
print('det(A):', np.linalg.det(A))  # -1.0  (nonzero -> span = R^2)

The einsum string 'ij,j->i' is identical in NumPy, PyTorch (torch.einsum), and TensorFlow (tf.einsum). For over-determined systems (more equations than spanning vectors) where an exact solution may not exist, use np.linalg.lstsq — a non-zero residual means $\vec{v}$ is not in the span.

The Null Space and Column Space Revisited

For $A \in \mathcal{M}_{m \times n}$:

• Null space $\mathcal{N}(A) = \{\vec{x} \in \mathbb{R}^n : A\vec{x} = \vec{0}\}$ is a subspace of $\mathbb{R}^n$.
• Column space $\mathcal{C}(A) = \{A\vec{x} : \vec{x} \in \mathbb{R}^n\}$ is a subspace of $\mathbb{R}^m$ (the span of the columns of $A$).

Both are subspaces — they are closed under the linear combinations that define those spaces. This is why the General = Particular + Homogeneous theorem works: $\mathcal{N}(A)$ is a genuine subspace, so we can form the coset $\vec{p} + \mathcal{N}(A)$.