Catalog
Supplement
Regularization
A unified treatment of regularization in deep learning — from the bias-variance tradeoff through explicit penalties (L1, L2, weight decay), dropout and its variants (DropConnect, MC Dropout, Stochastic Depth), normalization layers (BatchNorm, LayerNorm, RMSNorm), early stopping, data augmentation (Mixup, CutOut, CutMix, RandAugment), output regularizers (label smoothing, confidence penalty), and implicit regularization from initialization, SGD noise, and spectral normalization.
Readings
1
Overfitting and the Bias-Variance Tradeoff
Underfitting vs. overfitting, bias-variance decomposition, the definition of regularization, and the double-descent phenomenon in overparameterized models.
13 min
2
L1 and L2 Weight Penalties
L2 weight decay derivation, L1 sparsity geometry, elastic net, MAP estimation interpretation, and the AdamW decoupling fix.
15 min
3
Dropout and Stochastic Regularization
Inverted dropout, ensemble interpretation, co-adaptation prevention, DropConnect, MC Dropout for uncertainty, and Stochastic Depth / DropPath.
15 min
4
Normalization Layers
Internal covariate shift, Batch Normalization algorithm (train vs. inference modes), implicit regularization, Layer Norm, Group Norm, Instance Norm, and RMSNorm.
16 min
5
Early Stopping and Data Augmentation
Early stopping algorithm, connection to L2 regularization, standard geometric augmentations, Mixup, CutOut, CutMix, and RandAugment.
15 min
6
Label Smoothing and Output Regularizers
Label smoothing derivation, overconfidence and logit bounds, representation effects, temperature scaling, and the confidence penalty in supervised and RL settings.
12 min
7
Implicit Regularization — Initialization, SGD Noise, and Spectral Normalization
Xavier and He initialization via variance preservation, flat vs. sharp minima, SGD noise as regularization, the large-batch problem, spectral normalization, and gradient clipping.
14 min
Quizzes
Overfitting, Penalties, Dropout & Normalization
6 questions · 70% to pass
Augmentation, Output Regularizers & Implicit Regularization
6 questions · 70% to pass
Labs
Practice