Prerequisite · Calculus Foundations

Derivatives in Practice

Colab Notebook · ~45 min
Google Colab Notebook
Derivatives in Practice
Python · ~45 min
Open in Colab
Lab Objectives
1
Implement a finite difference approximation and observe the tradeoff between step size h and numerical accuracy.
2
Verify the power, product, and chain rules by comparing analytic derivatives to numerical estimates for several functions.
3
Visualize the secant line converging to the tangent line as h → 0 for f(x) = x².
4
Plot σ(x) and σ'(x) = σ(x)(1−σ(x)) and identify the x-range where the gradient is effectively zero.
5
Confirm all analytic derivatives against PyTorch autograd, observing that autograd computes exact symbolic derivatives rather than finite differences.

Lab 1: Derivatives in Practice

You have seen the limit definition of the derivative and the rules for computing it analytically. This lab grounds those ideas in code: you will implement derivative approximations from first principles, verify the differentiation rules numerically, and use PyTorch's autograd to see what a framework actually computes under the hood.

What You'll Build

  • A finite difference function that approximates f′(x)f'(x) using [f(x+h)−f(x)]/h[f(x+h) - f(x)] / h for decreasing values of hh, and a plot showing how accuracy improves then degrades with floating-point noise
  • A rule verifier that checks the power rule, product rule, and chain rule by comparing your analytic derivative to the numerical estimate
  • A tangent line visualizer that animates the secant line converging to the tangent as h→0h \to 0
  • A sigmoid derivative explorer that plots σ(x)\sigma(x) and σ′(x)=σ(x)(1−σ(x))\sigma'(x) = \sigma(x)(1-\sigma(x)) side-by-side, highlighting the vanishing gradient zones
  • A PyTorch autograd checker that confirms all your analytic results match .backward() to machine precision

Key Concepts Practiced

By the end you will understand why the limit definition is useful conceptually but numerical differentiation is impractical at scale; why autograd computes symbolic (not numerical) derivatives; and how the vanishing gradient problem appears directly from the sigmoid's derivative formula.