Catalog
Supplement
Weight Initialization
A ground-up treatment of all PyTorch and TensorFlow/Keras weight initializers — from constant and random baselines to variance-scaling methods (Xavier/Glorot, He/Kaiming, LeCun) and orthogonal initialization. Covers the variance-propagation derivations, default layer behaviors, and a practical selection guide by architecture and activation.
Readings
1
Why Initialization Matters
The symmetry-breaking problem, variance propagation through layers, and why the wrong starting point causes signals and gradients to explode or collapse.
12 min
2
Constant & Identity Initializations
zeros, ones, constant, eye, identity, and dirac — what each does, when it is appropriate, and why all-zeros weights break symmetry in practice.
11 min
3
Random Distribution Initializations
Uniform, normal, truncated normal, and sparse initializations — the role of scale, why naïve N(0,1) fails, and how to pick the right standard deviation.
12 min
4
Xavier / Glorot Initialization
The variance-propagation derivation, fan-in/fan-out trade-off, uniform vs normal variants, the gain parameter, and when to prefer Xavier over He.
15 min
5
He / Kaiming & LeCun Initialization
Why ReLU invalidates Xavier, the corrected variance formula for ReLU/LeakyReLU, fan modes, the a parameter, and LeCun initialization for SELU.
15 min
6
Orthogonal & Structured Initializations
Orthogonal initialization via SVD, why it prevents exploding/vanishing gradients in RNNs, dirac initialization for convolutional layers, and sparse initialization.
12 min
7
Practical Guide & Default Behaviors
PyTorch and Keras layer defaults, how to override initialization, the gain table, and a decision guide for every activation-architecture combination.
13 min
Quizzes
Initialization Fundamentals
6 questions · 70% to pass
Variance-Scaling & Structured Initializations
6 questions · 70% to pass
Labs
Weight Initialization in PyTorch
Apply every torch.nn.init function, run variance-propagation experiments, compare convergence across initializers on a deep MLP, and verify orthogonality numerically.
50 min
Weight Initialization in TensorFlow
Use all tf.keras.initializers, verify parity with PyTorch equivalents, and compare training stability of Xavier vs He on CIFAR-10 with different activations.
40 min
Practice