Course Pass 3: Tokens and data

corpus pipeline v1 with ledger and decontamination, BPE, WordPiece, and Unigram in Python, BPE in Rust, n-gram/NPLM/word2vec run through {tinyllm}; the engine keeps serving the bigram. About 7.5 weeks at 10 to 12 hours a week.

Part of: the course. Needs: Pass 2.

Gate: MS-P3 = MS-L1 + MS-corpus + MS-L2.

Stages, chapter links, and check columns are in path.tsv. Use practice/bin/ol learn course-p03-p03-tokens-and-data to see progress.

Stages

Generated from path.tsv. Track progress locally with practice/bin/ol learn.

Stage	Read	Done when
1	Injective, surjective, bijective; the GPT-2 byte map Injective, surjective, bijective; the GPT-2 byte map	Your implementation of M05.2 passes its course checks.
2	Trees and tries, longest-prefix match Trees and tries, longest-prefix match	Your implementation of M06.2 passes its course checks.
3	Solve set: trees, birthday bounds, Jaccard/MinHash/LSH S-curve, Bloom FP rate, recurrences Solve set: trees, birthday bounds, Jaccard/MinHash/LSH S-curve, Bloom FP rate, recurrences	You complete the S-M06b solve set and its check passes.
4	Perplexity, bits per byte, NLL accumulator Perplexity, bits per byte, NLL accumulator	Your implementation of M11.2 passes its course checks.
5	Categorical sampling: inverse CDF, Gumbel-max, alias method Categorical sampling: inverse CDF, Gumbel-max, alias method	Your implementation of M07.1 passes its course checks.
6	MLE, Laplace, absolute discounting MLE, Laplace, absolute discounting	Your implementation of M07.2 passes its course checks.
7	Solve set: joint and covariance, MLE Solve set: joint and covariance, MLE	You complete the S-M07b solve set and its check passes.
8	Tokenizer protocol, char tokenizer Tokenizer protocol, char tokenizer	Your implementation of L1.1 passes its course checks.
9	Byte-level BPE (GPT-2 compatible): pre-tokenizer, trainer, HF loader Byte-level BPE (GPT-2 compatible): pre-tokenizer, trainer, HF loader	Your implementation of L1.2 passes its course checks.
10	WordPiece (BERT basic tokenizer + greedy longest match) WordPiece (BERT basic tokenizer + greedy longest match)	Your implementation of L1.3 passes its course checks.
11	Unigram LM tokenizer (EM, Viterbi, subword sampling) Unigram LM tokenizer (EM, Viterbi, subword sampling)	Your implementation of L1.4 passes its course checks.
12	Tokenizer metrics Tokenizer metrics	Your implementation of L1.6 passes its course checks.
13	Robin Hood hash map with backward-shift deletion Robin Hood hash map with backward-shift deletion	Your implementation of ds.05 passes its course checks.
14	Binary heap with lazy deletion Binary heap with lazy deletion	Your implementation of ds.06 passes its course checks.
15	Rust fast BPE (tl-tok), byte tokenizer, streaming decoder Rust fast BPE (tl-tok), byte tokenizer, streaming decoder	Your implementation of L1.5 passes its course checks.
16	Property-based tests (R4) with Hypothesis, proptest, rapid, and ol_prop.h Property-based tests (R4) with Hypothesis, proptest, rapid, and olprop.h	Your implementation of craft.04 passes its course checks.
17	Milestone L1 Pass 3 milestones	The MS-L1 gate passes for your system.
18	Data licensing and the ledger Data licensing and the ledger	Your implementation of ethics.01 passes its course checks.
19	Privacy and PII policy Privacy and PII policy	Your implementation of ethics.02 passes its course checks.
20	Bloom filter Bloom filter	Your implementation of ds.08 passes its course checks.
21	Python asyncio: event loop, tasks, cancellation, bounded concurrency Python asyncio: event loop, tasks, cancellation, bounded concurrency	Your implementation of lang.08 passes its course checks.
22	Async fetch with resume, checksums, license capture Async fetch with resume, checksums, license capture	Your implementation of data.01 passes its course checks.
23	Extract, normalize, quality filters (generator stages) Extract, normalize, quality filters (generator stages)	Your implementation of data.02 passes its course checks.
24	Exact dedup: paragraph hashes, Bloom screen, sort-merge confirm Exact dedup: paragraph hashes, Bloom screen, sort-merge confirm	Your implementation of data.03 passes its course checks.
25	Near-duplicate dedup: MinHash, LSH, union-find, and decontamination Near-duplicate dedup: MinHash, LSH, union-find, and decontamination	Your implementation of data.04 passes its course checks.
26	PII scrub with typed placeholders and audit spans PII scrub with typed placeholders and audit spans	Your implementation of data.05 passes its course checks.
27	Parquet shards, manifest, and the document-hash split Parquet shards, manifest, and the document-hash split	Your implementation of data.06 passes its course checks.
28	Tokenize and pack to llm.c .bin Tokenize and pack to llm.c .bin	Your implementation of data.07 passes its course checks.
29	Licensing ledger verification and the datasheet Licensing ledger verification and the datasheet	Your implementation of data.08 passes its course checks.
30	Milestone corpus Pass 3 milestones	The MS-corpus gate passes for your system.
31	SVD, Eckart-Young low rank, and least squares SVD, Eckart-Young low rank, and least squares	Your implementation of M03.5 passes its course checks.
32	Inner products, projections, cosine similarity, and top-k Inner products, projections, cosine similarity, and top-k	Your implementation of M03.6 passes its course checks.
33	Linear algebra problem set, part b: inner products, SVD and low rank, matmul accounting and the roofline Linear algebra problem set, part b: inner products, SVD and low rank, matmul accounting and the roofline	You complete the S-M03b solve set and its check passes.
34	Mutual information, PMI, and PPMI Mutual information, PMI, and PPMI	Your implementation of M11.4 passes its course checks.
35	Information theory problem set, part b: coding and Huffman, mutual information, maximum entropy, rate-distortion Information theory problem set, part b: coding and Huffman, mutual information, maximum entropy, rate-distortion	You complete the S-M11b solve set and its check passes.
36	n-gram language model with interpolated modified Kneser-Ney n-gram language model with interpolated modified Kneser-Ney	Your implementation of L2.1 passes its course checks.
37	Bengio’s neural probabilistic language model Bengio’s neural probabilistic language model	Your implementation of L2.2 passes its course checks.
38	word2vec skip-gram with negative sampling, and the PPMI-SVD baseline word2vec skip-gram with negative sampling, and the PPMI-SVD baseline	Your implementation of L2.3 passes its course checks.
39	Milestone L2 Pass 3 milestones	The MS-L2 gate passes for your system.
40	Milestone P3 Pass 3 milestones	The MS-P3 gate passes for your system.