Implicit Regularization — Initialization, SGD Noise, and Spectral Normalization

Explicit vs. Implicit Regularization

All regularizers seen so far are explicit — they add a penalty term, drop units, or modify training data. But many regularizing effects arise from choices that seem unrelated to regularization: initialization, optimizer, batch size, and architecture.

Weight Initialization

Poor initialization is a training problem that acts like regularization failure: weights start at the wrong scale, leading to vanishing or exploding activations before a single gradient step.

The Variance Preservation Principle

For a layer $\mathbf{h} = W\mathbf{x}$, if $\text{Var}(x_i) = v$ and weights are i.i.d. with variance $\text{Var}(w_{ij}) = \sigma^2$, then $\text{Var}(h_j) = n_{\text{in}} \cdot \sigma^2 \cdot v$.

To preserve variance across layers ($\text{Var}(\mathbf{h}) = \text{Var}(\mathbf{x})$), set:

$$\sigma^2 = \frac{1}{n_{\text{in}}}$$

Xavier / Glorot Initialization (2010)

For sigmoid or tanh activations, balances the forward and backward pass variances:

$$\text{Var}(w) = \frac{2}{n_{\text{in}} + n_{\text{out}}}$$

Equivalent: uniform distribution $\mathcal{U}[-a, a]$ where $a = \sqrt{6/(n_{\text{in}} + n_{\text{out}})}$.

Used for: linear layers, tanh, sigmoid — activations that are approximately linear near zero.

He / Kaiming Initialization (2015)

ReLU kills approximately half the variance (zeros out negative activations). He initialization compensates with a factor of 2:

$$\text{Var}(w) = \frac{2}{n_{\text{in}}}$$

Derivation: $\mathbb{E}[\text{ReLU}(z)^2] = \frac{1}{2}\mathbb{E}[z^2]$ for zero-mean Gaussian $z$ — ReLU halves the expected squared magnitude, so we double the initial variance to compensate.

Used for: ReLU, Leaky ReLU, GELU — any rectifying activation that clips negative values.

Why Initialization Is Regularization

Initialization constrains the initial function class: networks initialized close to zero compute near-identity functions. Training from this point implicitly prefers solutions that remain close to initialization — an implicit prior analogous to L2 regularization toward zero weights.

SGD Noise and Flat Minima

Mini-batch SGD does not follow the exact gradient — it follows a noisy estimate. This noise has a regularizing effect:

Noise temperature: the gradient noise covariance is proportional to $\eta/B$ where $\eta$ is the learning rate and $B$ is the batch size. Larger $\eta$ or smaller $B$ → more noise.

Flat vs. sharp minima: loss landscapes have both broad, flat minima and narrow, sharp ones. Both have near-zero training loss, but they generalize differently:

• Flat minimum: a small perturbation to the parameters changes the loss little. The function in a neighborhood of the flat minimum generalizes similarly to the minimum itself.
• Sharp minimum: even tiny parameter perturbations cause large loss increases. Such solutions are sensitive to distribution shift.

SGD with small batches and large learning rates tends to escape sharp minima (high gradient noise makes them unstable) and settle in flat ones. Large-batch SGD has less noise and converges to sharper minima — this is the sharp minimum / large batch problem and is why simply scaling up batch size does not maintain accuracy without adjusting the learning rate (linear scaling rule: $\eta \propto B$) or adding noise corrections.

Spectral Normalization

The spectral norm of a matrix $W$ is its largest singular value $\sigma_1(W) = \|W\|_2$. It bounds the Lipschitz constant of the linear map $\mathbf{x} \mapsto W\mathbf{x}$:

$$\|W\mathbf{x} - W\mathbf{y}\|_2 \leq \sigma_1(W) \|\mathbf{x} - \mathbf{y}\|_2$$

Spectral normalization (Miyato et al., 2018) divides each weight matrix by its spectral norm at each step:

$$\hat{W} = W / \sigma_1(W)$$

This constrains $\sigma_1(\hat{W}) = 1$, bounding the per-layer Lipschitz constant to 1. The network's overall Lipschitz constant is bounded by the product across layers.

Why it matters for GANs: In WGAN and its variants, the discriminator must be a Lipschitz-1 function. Spectral normalization enforces this without the gradient penalty of WGAN-GP. It dramatically stabilizes GAN training and has become standard in image generation architectures.

Why it is regularization: Constraining the Lipschitz constant limits how much the network output can change in response to input perturbations — the model cannot fit high-frequency noise in the training data.

Gradient Clipping

Gradient clipping caps the norm of the gradient before the optimizer step:

$$\mathbf{g} \leftarrow \begin{cases} \mathbf{g} & \|\mathbf{g}\| \leq \tau \\ \tau \cdot \mathbf{g}/\|\mathbf{g}\| & \|\mathbf{g}\| > \tau \end{cases}$$

Primary use: prevents exploding gradients in RNNs and deep transformers. Covered in detail in the architectures supplement (RNN reading) from a stability perspective.

Regularization angle: clipping is equivalent to limiting the step size of gradient descent at each iteration. This prevents the optimizer from jumping to extreme parameter values when encountering an unusually high-loss example — a form of robustness to outliers in the training batch.

PyTorch and TensorFlow

PyTorch — initialization, gradient clipping, spectral norm:

import torch
import torch.nn as nn
from torch.nn.utils import spectral_norm

# Xavier / Glorot initialization (for tanh/sigmoid activations)
linear = nn.Linear(128, 64)
nn.init.xavier_uniform_(linear.weight)    # Var(w) = 2 / (fan_in + fan_out)
nn.init.zeros_(linear.bias)

# He / Kaiming initialization (for ReLU activations)
conv = nn.Conv2d(64, 128, 3)
nn.init.kaiming_normal_(conv.weight, mode='fan_out', nonlinearity='relu')
# mode='fan_out' preserves variance in the forward pass
# mode='fan_in'  preserves variance in the backward pass

# PyTorch default: Kaiming uniform for Linear and Conv layers
# Most architectures work fine without manual init

# Gradient clipping: prevent exploding gradients in RNNs and deep transformers
optimizer = torch.optim.AdamW(linear.parameters(), lr=1e-3)
loss = linear(torch.randn(8, 128)).sum()
loss.backward()

torch.nn.utils.clip_grad_norm_(linear.parameters(), max_norm=1.0)
# clips the global gradient norm to 1.0 before stepping
# use clip_grad_value_ to clip by absolute value instead
torch.nn.utils.clip_grad_value_(linear.parameters(), clip_value=0.5)

optimizer.step()
optimizer.zero_grad()

# Spectral normalization (see Normalization supplement for full treatment)
disc_layer = spectral_norm(nn.Linear(512, 1))

TensorFlow / Keras:

import tensorflow as tf

# Initialization via initializer argument on any layer
dense_xavier = tf.keras.layers.Dense(
    64, kernel_initializer='glorot_uniform'   # Xavier uniform — Keras default
)
dense_he = tf.keras.layers.Dense(
    64, activation='relu',
    kernel_initializer='he_normal'            # He normal for ReLU
)

# Gradient clipping on the optimizer
optimizer_clip_norm  = tf.keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)
optimizer_clip_value = tf.keras.optimizers.Adam(learning_rate=1e-3, clipvalue=0.5)
# clipnorm  clips the global gradient norm to max_norm
# clipvalue clips each gradient component independently

# Spectral normalization wrapper (TF 2.6+)
sn_layer = tf.keras.layers.SpectralNormalization(tf.keras.layers.Dense(64))