Catalog
Supplement
Normalization in Deep Learning
A comprehensive treatment of normalization techniques — from why they work to how to choose between them. Covers the covariate shift and loss landscape views, BatchNorm internals (running stats, train/eval modes, SyncBN), the LayerNorm family (RMSNorm, DeepNorm, pre/post-norm), weight and spectral normalization, small-batch alternatives (GroupNorm, InstanceNorm), and adaptive/conditional normalization (AdaIN, SPADE, FiLM, adaLN-Zero in DiT).
Readings
1
Why Normalization? Covariate Shift and the Loss Landscape
Internal covariate shift, Santurkar et al. loss landscape smoothing, β-smoothness, and a unified taxonomy of all normalization techniques across four axes
13 min
2
Batch Normalization — Algorithm, Placement, and Multi-GPU
Full BN algorithm with running statistics, the model.eval() bug, spatial BN for CNNs, pre-activation vs. post-activation placement, and Synchronized BatchNorm
17 min
3
Layer Norm, RMSNorm, and the Pre-Norm / Post-Norm Debate
LayerNorm formula and properties, pre-norm vs. post-norm stability analysis, RMSNorm's mean-centering removal and 15–20% speedup, DeepNorm's scaled residual for 1000+ layer transformers
15 min
4
Weight Normalization and Spectral Normalization
Weight norm reparameterization w = g·v/‖v‖, orthogonalized gradient updates, weight norm vs. BN comparison, spectral norm via power iteration, and WGAN Lipschitz constraint
15 min
5
Group Normalization, Instance Normalization, and Spatial Variants
Why BN degrades at small batch sizes, InstanceNorm for style transfer, GroupNorm unifying framework (GN as generalization of LN and IN), and when to use each normalization
14 min
6
Adaptive and Conditional Normalization: AdaIN, SPADE, and DiT
AdaIN style transfer via channel statistics replacement, Conditional BN / FiLM for class and language conditioning, SPADE spatially-adaptive maps for semantic image synthesis, adaLN-Zero in Diffusion Transformers
14 min
Quizzes
Covariate Shift, BatchNorm & LayerNorm
6 questions · 70% to pass
Weight Norm, GroupNorm & Adaptive Conditioning
6 questions · 70% to pass
Labs
Practice