Course Pass 9: Capstone training
about 10M model trained on TinyStories (mixed precision, accumulation, resume) with core ablations and the zoo table, safety evals, released through ModelRelease, and served. About 5 weeks at 10 to 12 hours a week.
Part of: the course. Needs: Pass 8.
Gate: MS-P9 = MS-L11 + MS-C1.
Stages, chapter links, and check columns are in path.tsv. Use practice/bin/ol learn course-p09-p09-capstone-training to see progress.
Stages
Section titled “Stages”Generated from path.tsv. Track progress locally with practice/bin/ol learn.
| Stage | Read | Done when |
|---|---|---|
| 1 | Hessian-vector products and the recompute-versus-memory schedule Hessian-vector products and the recompute-versus-memory schedule | Your implementation of M08.4 passes its course checks. |
| 2 | Mixed precision, loss scaling, gradient accumulation, and activation checkpointing Mixed precision, loss scaling, gradient accumulation, and activation checkpointing | Your implementation of L11.1 passes its course checks. |
| 3 | Datasheet and model card Datasheet and model card | Your implementation of ethics.03 passes its course checks. |
| 4 | Bias and safety evals Bias and safety evals | Your implementation of ethics.04 passes its course checks. |
| 5 | ModelRelease: gates, canary, PromQL burn check, rollback ModelRelease: gates, canary, PromQL burn check, rollback | Your implementation of dur.12 passes its course checks. |
| 6 | Model evals as tests (R8) Model evals as tests (R8) | Your implementation of craft.22 passes its course checks. |
| 7 | Capstone: TinyStories, owned end to end Capstone: TinyStories, owned end to end | Your implementation of C1 passes its course checks. |
| 8 | Milestone L11 Pass 9 milestones | The MS-L11 gate passes for your system. |
| 9 | Milestone C1 Pass 9 milestones | The MS-C1 gate passes for your system. |
| 10 | Milestone P9 Pass 9 milestones | The MS-P9 gate passes for your system. |
| 11 | Collectives over processes: ring all-reduce Collectives over processes: ring all-reduce | Your implementation of L11.2 passes its course checks. |
| 12 | Data parallelism and ZeRO stages 1 to 3 Data parallelism and ZeRO stages 1 to 3 | Your implementation of L11.3 passes its course checks. |