Upskilled Consulting & Training

Core linear algebra for practitioners — matrix notation, operations, determinants, inverses, and eigendecomposition. Required background for any course involving optimization or geometric transformations.

A rigorous treatment following Hefferon's acclaimed open textbook — Gaussian elimination, vector spaces, linear maps, matrix representations, orthogonal projection, determinants, and eigendecomposition. 11 readings covering Chapters 1–5.

The probability and statistics prerequisites for modern ML — random variables, expectation, Bayes' theorem, key distributions (Gaussian, Bernoulli, Categorical, Dirichlet), importance sampling, and KL divergence. Directly addresses the Spinning Up background requirements for deep reinforcement learning.

Single-variable calculus for ML practitioners who know algebra and trigonometry. Covers the derivative as a limit, differentiation rules, exponentials and logarithms, optimization and gradient descent, and integration with the Fundamental Theorem. Every concept is anchored to its role in training neural networks.

A bottom-up tour of the core neural network architectures — MLPs, convolutional layers and ResNets, vanilla RNNs, LSTMs, GRUs, scaled dot-product attention, multi-head attention, and the transformer. Builds the architectural vocabulary assumed by every other supplement.

A comprehensive guide to all 31 PyTorch activation functions — ReLU variants, saturating activations, smooth modern activations, gating mechanisms, shrinkage functions, and advanced NLP activations — with equation breakdowns, gradient analysis, and side-by-side PyTorch and TensorFlow implementations.

A ground-up tour of all 20 PyTorch loss functions — regression, classification, distribution, ranking, embedding, and metric learning — with equation breakdowns, derivations from probability theory, and side-by-side PyTorch and TensorFlow implementations.

A comprehensive guide to all 13 PyTorch optimizers and 15 learning rate schedulers — SGD, Adam, AdamW, adaptive methods, quasi-Newton, and the full lr_scheduler suite — with update-rule derivations, hyperparameter intuition, and side-by-side PyTorch and TensorFlow implementations.

A ground-up treatment of all PyTorch and TensorFlow/Keras weight initializers — from constant and random baselines to variance-scaling methods (Xavier/Glorot, He/Kaiming, LeCun) and orthogonal initialization. Covers variance-propagation derivations, default layer behaviors, and a practical selection guide by architecture and activation.

A comprehensive treatment of normalization techniques — from why they work to how to choose between them. Covers BatchNorm internals (running stats, train/eval modes, SyncBN), the LayerNorm family (RMSNorm, DeepNorm, pre/post-norm), weight and spectral normalization, small-batch alternatives (GroupNorm, InstanceNorm), and adaptive/conditional normalization (AdaIN, SPADE, FiLM, adaLN-Zero in DiT).

A unified treatment of regularization in deep learning — bias-variance tradeoff, L1/L2 weight penalties, dropout and its variants (MC Dropout, Stochastic Depth), normalization layers (BatchNorm, LayerNorm, RMSNorm), early stopping, data augmentation (Mixup, CutOut, CutMix), label smoothing, and implicit regularization from initialization and SGD noise.