Course Pass 2: Foundations

the bigram retrained by their autograd (L0.5 takes over bigram.py), gradcheck everywhere, the token-stream reader; the engine unchanged (same checkpoint contract). About 7.5 weeks at 10 to 12 hours a week.

Part of: the course. Needs: Pass 1.

Gate: MS-P2 = MS-L0.

Stages, chapter links, and check columns are in path.tsv. Use practice/bin/ol learn course-p02-p02-foundations to see progress.

Stages

Generated from path.tsv. Track progress locally with practice/bin/ol learn.

Stage	Read	Done when
1	Solve set: functions, exp/log, trig and Euler, series, polynomials, inequalities Solve set: functions, exp/log, trig and Euler, series, polynomials, inequalities	You complete the S-M00 solve set and its check passes.
2	Exponents, logs, change of base, units of information Exponents, logs, change of base, units of information	Your implementation of M00.1 passes its course checks.
3	Trig, unit circle, 2D rotations, complex numbers, Euler’s formula Trig, unit circle, 2D rotations, complex numbers, Euler’s formula	Your implementation of M00.2 passes its course checks.
4	Sequences, geometric series, frequency ladders Sequences, geometric series, frequency ladders	Your implementation of M00.3 passes its course checks.
5	Polynomials, Horner, stable quadratic roots Polynomials, Horner, stable quadratic roots	Your implementation of M00.4 passes its course checks.
6	Derivative as a limit, finite differences, step-size choice Derivative as a limit, finite differences, step-size choice	Your implementation of M01.1 passes its course checks.
7	Newton’s method Newton’s method	Your implementation of M01.2 passes its course checks.
8	Taylor series, remainder bounds, range reduction Taylor series, remainder bounds, range reduction	Your implementation of M02.1 passes its course checks.
9	Activation functions and their derivatives Activation functions and their derivatives	Your implementation of M01.3 passes its course checks.
10	Calculus 1 problem set: limits, derivatives, rates, optimization, integrals, L’Hôpital Calculus 1 problem set: limits, derivatives, rates, optimization, integrals, L’Hôpital	You complete the S-M01 solve set and its check passes.
11	Series convergence, EMA as a geometric series, bias correction Series convergence, EMA as a geometric series, bias correction	Your implementation of M02.2 passes its course checks.
12	Calculus 2 problem set: integration techniques, the Gaussian, series, Taylor, polar, ODEs Calculus 2 problem set: integration techniques, the Gaussian, series, Taylor, polar, ODEs	You complete the S-M02 solve set and its check passes.
13	Partial derivatives, gradients, gradcheck Partial derivatives, gradients, gradcheck	Your implementation of M04.1 passes its course checks.
14	Jacobians, multivariable chain rule, numeric JVP/VJP Jacobians, multivariable chain rule, numeric JVP/VJP	Your implementation of M04.2 passes its course checks.
15	Calculus 3 problem set: vectors, gradients, the chain rule, Hessians, multiple integrals, Lagrange Calculus 3 problem set: vectors, gradients, the chain rule, Hessians, multiple integrals, Lagrange	You complete the S-M04 solve set and its check passes.
16	Discrete math 1 problem set: sets, counting, proofs, induction, relations, bijections Discrete math 1 problem set: sets, counting, proofs, induction, relations, bijections	You complete the S-M05 solve set and its check passes.
17	Graphs, DAGs, and an iterative topological sort Graphs, DAGs, and an iterative topological sort	Your implementation of M06.1 passes its course checks.
18	Discrete math 2 problem set, part a: DAGs, modular arithmetic, hashing Discrete math 2 problem set, part a: DAGs, modular arithmetic, hashing	You complete the S-M06a solve set and its check passes.
19	Modular arithmetic, hashing, and PCG32 in Python Modular arithmetic, hashing, and PCG32 in Python	Your implementation of M06.3 passes its course checks.
20	Gaussian elimination and LU with partial pivoting Gaussian elimination and LU with partial pivoting	Your implementation of M03.2 passes its course checks.
21	Random variables, expectation, variance, and normal draws by Box-Muller Random variables, expectation, variance, and normal draws by Box-Muller	Your implementation of M07.0 passes its course checks.
22	Orthogonality, Householder QR, and orthogonal initialization Orthogonality, Householder QR, and orthogonal initialization	Your implementation of M03.3 passes its course checks.
23	Eigenvalues, power iteration, and the spectral radius Eigenvalues, power iteration, and the spectral radius	Your implementation of M03.4 passes its course checks.
24	Linear algebra problem set, part a: elimination, LU, bases, rank, determinants, eigenvalues, QR Linear algebra problem set, part a: elimination, LU, bases, rank, determinants, eigenvalues, QR	You complete the S-M03a solve set and its check passes.
25	Probability problem set, part a: axioms, Bayes, random variables, Box-Muller Probability problem set, part a: axioms, Bayes, random variables, Box-Muller	You complete the S-M07a solve set and its check passes.
26	IEEE 754 anatomy, ulp, round to nearest even, and bf16 and fp16 emulation IEEE 754 anatomy, ulp, round to nearest even, and bf16 and fp16 emulation	Your implementation of M09.1 passes its course checks.
27	Stable numerics: logsumexp, softmax, compensated sums Stable numerics: logsumexp, softmax, compensated sums	Your implementation of M09.2 passes its course checks.
28	Floating point problem set, part a: representation, rounding, cancellation Floating point problem set, part a: representation, rounding, cancellation	You complete the S-M09a solve set and its check passes.
29	Entropy, cross-entropy, KL, JS, and the k3 estimator Entropy, cross-entropy, KL, JS, and the k3 estimator	Your implementation of M11.1 passes its course checks.
30	Information theory problem set, part a: entropy, chain rules, KL and cross-entropy Information theory problem set, part a: entropy, chain rules, KL and cross-entropy	You complete the S-M11a solve set and its check passes.
31	Dual numbers and forward-mode autodiff Dual numbers and forward-mode autodiff	Your implementation of M08.1 passes its course checks.
32	Scalar reverse mode: the Value graph Scalar reverse mode: the Value graph	Your implementation of M08.2 passes its course checks.
33	Matrix differentials, the trace trick, and closed-form VJPs Matrix differentials, the trace trick, and closed-form VJPs	Your implementation of M08.3 passes its course checks.
34	Matrix calculus problem set: differentials, trace trick, VJPs, SDPA Jacobian Matrix calculus problem set: differentials, trace trick, VJPs, SDPA Jacobian	You complete the S-M08 solve set and its check passes.
35	Convexity, smoothness, gradient descent, and Armijo line search Convexity, smoothness, gradient descent, and Armijo line search	Your implementation of M10.1 passes its course checks.
36	The Optimizer protocol: SGD, momentum, Nesterov, weight decay The Optimizer protocol: SGD, momentum, Nesterov, weight decay	Your implementation of M10.2 passes its course checks.
37	Adam and AdamW: bias correction and decoupled weight decay Adam and AdamW: bias correction and decoupled weight decay	Your implementation of M10.3 passes its course checks.
38	Learning-rate schedules (cosine, WSD, Noam) and gradient clipping Learning-rate schedules (cosine, WSD, Noam) and gradient clipping	Your implementation of M10.4 passes its course checks.
39	Optimization problem set, part a: convexity, GD rates, momentum, Adam Optimization problem set, part a: convexity, GD rates, momentum, Adam	You complete the S-M10a solve set and its check passes.
40	Variance propagation and initialization Variance propagation and initialization	Your implementation of M07.3 passes its course checks.
41	TDD, unit tests, and how your tests are graded TDD, unit tests, and how your tests are graded	Your implementation of craft.03 passes its course checks.
42	Tensor, broadcasting backward, and no_grad Tensor, broadcasting backward, and nograd	Your implementation of L0.1 passes its course checks.
43	The op library and gradcheck_all The op library and gradcheckall	Your implementation of L0.2 passes its course checks.
44	Losses with a fused backward Losses with a fused backward	Your implementation of L0.3 passes its course checks.
45	Module system and basic layers Module system and basic layers	Your implementation of L0.4 passes its course checks.
46	Training loop and the autograd bigram Training loop and the autograd bigram	Your implementation of L0.5 passes its course checks.
47	Safetensors for every dtype, atomic checkpoints, and the token stream Safetensors for every dtype, atomic checkpoints, and the token stream	Your implementation of L0.6 passes its course checks.
48	Milestone L0 Pass 2 milestones	The MS-L0 gate passes for your system.
49	Milestone P2 Pass 2 milestones	The MS-P2 gate passes for your system.