Course Pass 6: Inference and kernels
Python inference stack (sampler, KV, paged KV, quant, spec decoding, constrained decoding), plus optional standalone C exercises. About 7 weeks at 10 to 12 hours a week.
Part of: the course. Needs: Pass 5.
Gate: MS-P6 = MS-L8; optional C exercises are not pass-gate dependencies.
Stages, chapter links, and check columns are in path.tsv. Use practice/bin/ol learn course-p06-p06-inference-and-kernels to see progress.
Stages
Section titled “Stages”Generated from path.tsv. Track progress locally with practice/bin/ol learn.
| Stage | Read | Done when |
|---|---|---|
| 1 | Rejection sampling and residual distributions Rejection sampling and residual distributions | Your implementation of M07.6 passes its course checks. |
| 2 | Error analysis, condition numbers, tolerance budgets Error analysis, condition numbers, tolerance budgets | Your implementation of M09.3 passes its course checks. |
| 3 | FP8 E4M3/E5M2, MXFP4/MXFP8 with E8M0 scales FP8 E4M3/E5M2, MXFP4/MXFP8 with E8M0 scales | Your implementation of M09.4 passes its course checks. |
| 4 | Floating point problem set, part b: condition numbers, error bounds, Newton, polynomial approximation, low precision Floating point problem set, part b: condition numbers, error bounds, Newton, polynomial approximation, low precision | You complete the S-M09b solve set and its check passes. |
| 5 | Sampling and logit processors Sampling and logit processors | Your implementation of L8.1 passes its course checks. |
| 6 | KV cache, incremental decode, incremental UTF-8 detokenizer, generate KV cache, incremental decode, incremental UTF-8 detokenizer, generate | Your implementation of L8.2 passes its course checks. |
| 7 | Paged KV cache in Python Paged KV cache in Python | Your implementation of L8.3 passes its course checks. |
| 8 | Radix tree over token ids with index-linked LRU leaf list Radix tree over token ids with index-linked LRU leaf list | Your implementation of ds.07 passes its course checks. |
| 9 | Radix prefix cache (block-granular, LRU leaf eviction, locks) Radix prefix cache (block-granular, LRU leaf eviction, locks) | Your implementation of L8.4 passes its course checks. |
| 10 | Quantization: int8, int4 group (packed), fp8, KV quant Quantization: int8, int4 group (packed), fp8, KV quant | Your implementation of L8.5 passes its course checks. |
| 11 | Speculative decoding: n-gram, prompt-lookup, and model drafts Speculative decoding: n-gram, prompt-lookup, and model drafts | Your implementation of L8.6 passes its course checks. |
| 12 | Constrained decoding: regex to DFA to token masks, JSON-schema subset Constrained decoding: regex to DFA to token masks, JSON-schema subset | Your implementation of L8.7 passes its course checks. |
| 13 | Benchmarks and perf gates (R7) Benchmarks and perf gates (R7) | Your implementation of craft.06 passes its course checks. |
| 14 | Milestone L8 Pass 6 milestones | The MS-L8 gate passes for your system. |
| 15 | Milestone P6 Pass 6 milestones | The MS-P6 gate passes for your system. |
| 16 | C runtime support and arena allocator with marks (optional) C runtime support and arena allocator with marks (optional) | Your implementation of rt.02 passes its course checks. |
| 17 | Growable array tl_vec (type-erased, optional C) Growable array tlvec (type-erased, optional C) | Your implementation of ds.01 passes its course checks. |
| 18 | Swiss table tl_map (u64 to u64, optional C) Swiss table tlmap (u64 to u64, optional C) | Your implementation of ds.02 passes its course checks. |
| 19 | Intrusive list + LRU (optional C) Intrusive list + LRU (optional C) | Your implementation of ds.03 passes its course checks. |
| 20 | Binary heap top-k in C (optional) Binary heap top-k in C (optional) | Your implementation of ds.04 passes its course checks. |
| 21 | Thread pool and tl_parallel_for (optional C) Thread pool and tlparallelfor (optional C) | Your implementation of rt.03 passes its course checks. |
| 22 | Paged KV block pool (format v1, optional C) Paged KV block pool (format v1, optional C) | Your implementation of rt.04 passes its course checks. |
| 23 | Fast inverse square root in standalone C (optional) Fast inverse square root in standalone C (optional) | Your implementation of M09.5 passes its course checks. |
| 24 | Polynomial approximation and range reduction: expf in C (optional) Polynomial approximation and range reduction: expf in C (optional) | Your implementation of M09.6 passes its course checks. |
| 25 | Low-precision conversions in standalone C (optional) Low-precision conversions in standalone C (optional) | Your implementation of M09.7 passes its course checks. |
| 26 | Cache-blocked, packed, batch-invariant matmul in C (optional) Cache-blocked, packed, batch-invariant matmul in C (optional) | Your implementation of L9.1 passes its course checks. |
| 27 | Softmax in C: three-pass and online two-pass Softmax in C: three-pass and online two-pass | Your implementation of L9.2 passes its course checks. |
| 28 | FlashAttention forward in C FlashAttention forward in C | Your implementation of L9.3 passes its course checks. |
| 29 | Paged attention for decode in C Paged attention for decode in C | Your implementation of L9.4 passes its course checks. |
| 30 | Fused int4 and int8 dequantize-matmul in C Fused int4 and int8 dequantize-matmul in C | Your implementation of L9.5 passes its course checks. |
| 31 | Elementwise kernels in C: RMSNorm, RoPE, SiLU-mul, embedding, add, argmax Elementwise kernels in C: RMSNorm, RoPE, SiLU-mul, embedding, add, argmax | Your implementation of L9.6 passes its course checks. |
| 32 | Milestone L9 Pass 6 milestones | The MS-L9 gate passes for your system. |