Supplement · Weight Initialization
Weight Initialization in TensorFlow
Google Colab Notebook
Weight Initialization in TensorFlow
Lab Objectives
1
Apply all tf.keras.initializers classes and verify resulting tensors match their documented formulas
2
Verify parity between PyTorch and TensorFlow initializers: show that kaiming_normal_(a=0) and HeNormal() produce the same variance
3
Use VarianceScaling to reconstruct GlorotUniform, HeNormal, and LecunNormal from first principles
4
Train a 10-layer network on CIFAR-10 with Tanh+Xavier, ReLU+Xavier, and ReLU+He — compare training curves and activation statistics
5
Apply the kernel_initializer parameter at scale: build a ResNet-style block and confirm each layer uses the intended initializer
Lab Overview
This notebook mirrors the PyTorch lab in TensorFlow/Keras, with additional focus on the VarianceScaling abstraction and Keras's kernel_initializer API. You will verify framework parity and explore the practical difference between Xavier and He on a convolutional task.
Sections
| Section | Content |
|---|---|
| 1 | All tf.keras.initializers — instantiate each class, generate tensors, verify mean, variance, and clip behavior for TruncatedNormal |
| 2 | VarianceScaling deep-dive — reconstruct GlorotUniform, HeNormal, and LecunNormal using VarianceScaling with explicit scale, mode, and distribution parameters |
| 3 | PyTorch parity check — for each initializer pair (e.g. kaiming_normal_ vs HeNormal), generate 10,000 samples from each and compare histograms and variance |
| 4 | Initialization on CIFAR-10 — train a small Conv network under three conditions: (a) Tanh + GlorotUniform, (b) ReLU + GlorotUniform, (c) ReLU + HeNormal; plot training loss, validation accuracy, and per-layer activation variance at epoch 1 |
| 5 | kernel_initializer API — build a ResNet-style block using the Functional API; apply he_normal to all Conv layers, glorot_uniform to the projection shortcut, and orthogonal to a custom RNN layer; verify each layer's kernel init with layer.kernel_initializer |
| 6 | Keras defaults audit — build Dense, Conv2D, LSTM, and BatchNormalization layers with default settings; inspect layer.kernel_initializer and layer.recurrent_initializer to confirm defaults |