Semester 2 · Core
Optimisation & Training at Scale
https://epochschool.com/subjects/optimisation-training-at-scale
Why training works, why it fails and what it costs. You derive how gradient descent, SGD, momentum and Adam behave from the shape of the loss, then learn the craft of learning-rate schedules, warm-up, curvature measurement and reproducible, budgeted tuning.
The second half is training at scale: number formats and mixed precision, FLOP and memory accounting, checkpointing and accumulation, and data, sharded, tensor and pipeline parallelism. Every idea is computed by hand first, then built and tested on one CPU core, with large-scale systems simulated faithfully. Taken alongside Neural Networks & Deep Learning, a co-requisite.
- Prerequisites
- Required: Finish this subject before you start. The subject builds directly on it.Corequisite: Take this subject before or alongside, in the same semester. Chapters lean on it as you go.Recommended: Helpful background. You can start without it if you are willing to fill the gaps.
What you’ll be able to do
- LO1Analyse a loss's convexity, smoothness and conditioning, and predict gradient descent's stable step sizes and convergence rate from them.
- LO2Quantify the noise in mini-batch gradients and use it to choose batch size, learning rate and decay schedule together.
- LO3Derive, implement and compare momentum, Nesterov, AdaGrad, RMSProp, Adam and AdamW, and justify an optimiser choice for a given model.
- LO4Design learning-rate schedules, warm-up and budgeted hyperparameter searches, and run experiments that resume exactly and are compared with sound statistics.
- LO5Measure curvature with Hessian-vector products and power iteration, relate sharpness to the learning rate and training stability (the edge of stability), and judge claims that link flat or connected minima to generalisation, including their limits.
- LO6Diagnose and prevent numerical failures, and train in reduced precision (bf16, fp16 with loss scaling) without losing accuracy.
- LO7Estimate the FLOPs, memory and time of a training run, and use gradient checkpointing and accumulation to fit a memory budget.
- LO8Explain, cost and simulate data, sharded (ZeRO/FSDP), tensor and pipeline parallelism, including their communication.
Chapters
- 01Convexity, conditioning and the shape of a loss
- 02Stochastic gradient descent and its noise
- 03Momentum, Nesterov and adaptive methods
- 04Schedules, warm-up, sharpness and reproducible tuning
- 05Floating point, numerical stability and mixed precision
- 06Memory, compute and parallel training
Graded labs
- L1
Gradient descent and SGD under the microscope
After chapter 2 · about 5 hours · Synthetic quadratics and least-squares problems with set condition numbers · Course own work, CC BY 4.0 · Runs on a CPU
- L2
Optimisers from scratch: momentum, Nesterov, RMSProp, Adam and AdamW
After chapter 3 · about 5 hours · Synthetic diagonal and rotated quadratics with set condition numbers · Course own work, CC BY 4.0 · Runs on a CPU
- L3
Schedules, reproducible runs and the edge of stability
After chapter 4 · about 6 hours · Optical recognition of handwritten digits · BSD-3-Clause (scikit-learn distribution) · Runs on a CPU
- L4
Mixed precision on a CPU
After chapter 5 · about 5 hours · Optical recognition of handwritten digits · BSD-3-Clause (scikit-learn) · Runs on a CPU
- L5
Memory accounting and parallel training on one core
After chapter 6 · about 6 hours · Teacher–student regression · Course own work, CC BY 4.0 · Runs on a CPU
Before you start
Required
Finish this subject before you start. The subject builds directly on it.
Corequisite
Take this subject before or alongside, in the same semester. Chapters lean on it as you go.
Neural Networks & Deep Learning
Recommended
Helpful background. You can start without it if you are willing to fill the gaps.
Probability & Statistics for ML
Foundations of Machine Learning
Where it leads
NLP & Language Model Foundations
Building and Training LLMs from Scratch
Deep Visual Recognition
Probabilistic ML & Bayesian Methods
Reinforcement Learning Foundations
Capstone Project
How it’s taught
You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.
No videos. No live classes. No AI tutor.