Course Pass 5: Transformer

transformer, GPT/BERT/ELECTRA/LoRA, the model zoo, modern block; SmolLM2-135M loads and matches HF. About 6 weeks at 10 to 12 hours a week.

Part of: the course. Needs: Pass 4.

Gate: MS-P5 = MS-L5 + MS-L6 + MS-L7.

Stages, chapter links, and check columns are in path.tsv. Use practice/bin/ol learn course-p05-p05-transformer to see progress.

Stages

Generated from path.tsv. Track progress locally with practice/bin/ol learn.

Stage	Read	Done when
1	Definite integrals, trapezoid, Simpson Definite integrals, trapezoid, Simpson	Your implementation of M01.4 passes its course checks.
2	Hypothesis tests: paired and two-sample permutation, McNemar, Holm Hypothesis tests: paired and two-sample permutation, McNemar, Holm	Your implementation of M07.5 passes its course checks.
3	Logistic regression (IRLS), ROC-AUC, calibration (ECE) Logistic regression (IRLS), ROC-AUC, calibration (ECE)	Your implementation of M07.7 passes its course checks.
4	Solve set: hypothesis tests, regression, queueing and Little’s law, balls into bins Solve set: hypothesis tests, regression, queueing and Little’s law, balls into bins	You complete the S-M07d solve set and its check passes.
5	Scaled dot-product attention (forward and backward) Scaled dot-product attention (forward and backward)	Your implementation of L5.1 passes its course checks.
6	Masks: causal, padding, sliding window, additive Masks: causal, padding, sliding window, additive	Your implementation of L5.2 passes its course checks.
7	Multi-head attention Multi-head attention	Your implementation of L5.3 passes its course checks.
8	Positional encodings: sinusoidal, learned Positional encodings: sinusoidal, learned	Your implementation of L5.4 passes its course checks.
9	Encoder-decoder Transformer: post-LN and pre-LN, Noam, label smoothing Encoder-decoder Transformer: post-LN and pre-LN, Noam, label smoothing	Your implementation of L5.5 passes its course checks.
10	Milestone L5 Pass 5 milestones	The MS-L5 gate passes for your system.
11	GPT decoder-only, causal LM loss, GPT-2 weight loading GPT decoder-only, causal LM loss, GPT-2 weight loading	Your implementation of L6.1 passes its course checks.
12	BERT encoder and MLM masking BERT encoder and MLM masking	Your implementation of L6.2 passes its course checks.
13	ELECTRA replaced-token detection ELECTRA replaced-token detection	Your implementation of L6.3 passes its course checks.
14	LoRA with PiSSA init and merge LoRA with PiSSA init and merge	Your implementation of L6.6 passes its course checks.
15	Fine-tuning heads and the linear-head export Fine-tuning heads and the linear-head export	Your implementation of L6.5 passes its course checks.
16	LM evaluation harness and the model zoo LM evaluation harness and the model zoo	Your implementation of L6.7 passes its course checks.
17	Milestone L6 Pass 5 milestones	The MS-L6 gate passes for your system.
18	Counting: params, FLOPs, KV bytes, memory plans Counting: params, FLOPs, KV bytes, memory plans	Your implementation of M05.1 passes its course checks.
19	Pre-LN, RMSNorm Pre-LN, RMSNorm	Your implementation of L7.1 passes its course checks.
20	Gated MLPs: SwiGLU, GeGLU Gated MLPs: SwiGLU, GeGLU	Your implementation of L7.2 passes its course checks.
21	RoPE (half and interleaved layouts, partial rotary) RoPE (half and interleaved layouts, partial rotary)	Your implementation of L7.3 passes its course checks.
22	Context extension: PI, NTK, YaRN, Llama-3 scaling; ALiBi (optional part) Context extension: PI, NTK, YaRN, Llama-3 scaling; ALiBi (optional part)	Your implementation of L7.4 passes its course checks.
23	MQA/GQA attention with cache hook, window, learned sinks MQA/GQA attention with cache hook, window, learned sinks	Your implementation of L7.5 passes its course checks.
24	Multi-head latent attention (DeepSeek-V2/V3), weight absorption Multi-head latent attention (DeepSeek-V2/V3), weight absorption	Your implementation of L7.6 passes its course checks.
25	Sliding window, StreamingLLM sinks, learned sinks Sliding window, StreamingLLM sinks, learned sinks	Your implementation of L7.7 passes its course checks.
26	Mixture of Experts: routing, sorted dispatch, Switch aux loss, aux-free bias Mixture of Experts: routing, sorted dispatch, Switch aux loss, aux-free bias	Your implementation of L7.8 passes its course checks.
27	Llama-family model, HF config, weight loading, downloader Llama-family model, HF config, weight loading, downloader	Your implementation of L7.9 passes its course checks.
28	Oracles, golden and differential tests, gradcheck as a test (R5) Oracles, golden and differential tests, gradcheck as a test (R5)	Your implementation of craft.05 passes its course checks.
29	Milestone L7 Pass 5 milestones	The MS-L7 gate passes for your system.
30	Milestone P5 Pass 5 milestones	The MS-P5 gate passes for your system.
31	T5 span corruption and relative position buckets T5 span corruption and relative position buckets	Your implementation of L6.4 passes its course checks.