Gaussian Mathematics: Attributes, Covariance, and Spherical Harmonics

The 59-Parameter Gaussian

Every 3D Gaussian splat carries a fixed set of attributes. Understanding what each encodes and how many parameters it contributes is essential for any compression discussion.

The full parameter budget per Gaussian:

At float32 precision: 59 × 4 = 236 bytes per Gaussian. A typical scene with 5 million Gaussians is ~1.2 GB.

Covariance Factorization: Why $\Sigma = RSS^\top R^\top$

A covariance matrix must be positive semi-definite (PSD) — this is a mathematical requirement for a valid Gaussian. Storing and optimizing a full 3×3 symmetric matrix is possible, but gradient descent will almost certainly produce non-PSD matrices at intermediate steps, breaking the Gaussian interpretation.

3DGS avoids this by factoring:

$$\Sigma = R S S^\top R^\top$$

where:

• $R \in SO(3)$ is a rotation matrix (encoded as a quaternion $q \in \mathbb{R}^4$, $\|q\|=1$)
• $S = \text{diag}(s_1, s_2, s_3)$ is a diagonal scale matrix with $s_i > 0$

Why is $\Sigma$ always PSD? For any vector $v$: $$v^\top \Sigma v = v^\top R S S^\top R^\top v = \|S^\top R^\top v\|^2 \geq 0$$

This is always non-negative by construction. The optimization landscape is unconstrained — you can apply gradient descent to $q$ and $\log s_i$ (log to keep scale positive) without special projection steps.

The geometric interpretation: $S$ stretches a unit sphere along three principal axes, and $R$ rotates the result. The Gaussian ellipsoid is the image of this transformation.

Spherical Harmonics for View-Dependent Color

Real surfaces exhibit view-dependent appearance: specular highlights, reflections, refractions, and subsurface scattering all change color based on the viewing direction. A single RGB color per Gaussian is insufficient.

Spherical harmonics (SH) are an orthonormal basis of functions on the unit sphere. They decompose a directional function $f(\theta, \phi)$ into frequency bands:

$$f(\theta,\phi) = \sum_{l=0}^{L} \sum_{m=-l}^{l} c_l^m Y_l^m(\theta,\phi)$$

where $Y_l^m$ are the SH basis functions and $c_l^m$ are learnable coefficients.

The number of coefficients per band:

• Order 0 (DC): $(0+1)^2 = 1$ coefficient → view-independent color
• Order 1: $(1+1)^2 = 4$ total → 3 new coefficients (linear variation)
• Order 2: 9 total → 5 new coefficients
• Order 3: $(3+1)^2 = 16$ total → 7 new coefficients

Since 3DGS uses order 3 and stores one SH expansion per color channel, that's $16 \times 3 = 48$ SH parameters. Subtracting the DC term (already counted under "DC color"): 45 SH AC coefficients.

Why SH Dominates the Bit Budget

The 45 SH coefficients represent $45/59 \approx 76\%$ of all parameters per Gaussian. This immediately suggests SH degree reduction as the highest-leverage compaction strategy — a topic we will explore in Module 3.

The trade-off: dropping from order 3 to order 0 (DC only) halves the parameter count but eliminates all view-dependent effects. Intermediate degrees offer a smooth quality–size curve.

The Gaussian Density Function in Context

Putting it together, the full density contribution of Gaussian $i$ at point $\mathbf{p}$ (a multivariate Gaussian density):

$$f_i(\mathbf{p}) = \sigma_\alpha \exp\!\left(-\frac{1}{2}(\mathbf{p}-\mu_i)^\top \Sigma_i^{-1}(\mathbf{p}-\mu_i)\right)$$

The color seen along direction $\mathbf{d}$ is then:

$$c_i(\mathbf{d}) = \text{sigmoid}\!\left(\sum_{l,m} c_l^m Y_l^m(\mathbf{d})\right)_{\text{per channel}}$$

The sigmoid maps the linear SH sum into $[0,1]$ for each of the three RGB channels.