Course Pass 7: Serving platform
async Rust, gRPC, and SQL primers; candle-based Rust engine (continuous batching, chunked prefill, disaggregation, speculative decoding, tool calls), gateway (auth, limits, routing, cascades, cache, ledger), loadgen, gateway and engine charts, Tilt, Prometheus, Tempo, Grafana, SLOs. About 8 weeks at 10 to 12 hours a week.
Part of: the course. Needs: Pass 6.
Gate: MS-P7 = MS-L10 + MS-gateway + MS-prod.
Stages, chapter links, and check columns are in path.tsv. Use practice/bin/ol learn course-p07-p07-serving-platform to see progress.
Stages
Section titled “Stages”Generated from path.tsv. Track progress locally with practice/bin/ol learn.
| Stage | Read | Done when |
|---|---|---|
| 1 | Candle model runner (Llama + bigram, int4), Rust KV cache, sampler and PCG32 Candle model runner (Llama + bigram, int4), Rust KV cache, sampler and PCG32 | Your implementation of L10.1 passes its course checks. |
| 2 | Continuous batching scheduler with priority and preemption Continuous batching scheduler with priority and preemption | Your implementation of L10.2 passes its course checks. |
| 3 | Chunked prefill (mixed prefill + decode batches) Chunked prefill (mixed prefill + decode batches) | Your implementation of L10.3 passes its course checks. |
| 4 | Block manager with prefix cache (none, hash, radix) Block manager with prefix cache (none, hash, radix) | Your implementation of L10.4 passes its course checks. |
| 5 | Async Rust and tokio: futures, tasks, channels, cancellation, hyper Async Rust and tokio: futures, tasks, channels, cancellation, hyper | Your implementation of lang.09 passes its course checks. |
| 6 | Speculative decoding in the engine Speculative decoding in the engine | Your implementation of L10.8 passes its course checks. |
| 7 | tl-serve: OpenAI-compatible HTTP + SSE on tokio/hyper tl-serve: OpenAI-compatible HTTP + SSE on tokio/hyper | Your implementation of L10.5 passes its course checks. |
| 8 | Protocol Buffers and gRPC: schema evolution, unary and streaming RPCs, stubs Protocol Buffers and gRPC: schema evolution, unary and streaming RPCs, stubs | Your implementation of lang.10 passes its course checks. |
| 9 | Disaggregated prefill/decode, KV transfer, heartbeat client Disaggregated prefill/decode, KV transfer, heartbeat client | Your implementation of L10.6 passes its course checks. |
| 10 | Serving metrics, SLO histograms, OTel spans and propagation Serving metrics, SLO histograms, OTel spans and propagation | Your implementation of L10.7 passes its course checks. |
| 11 | Tool calls and constrained JSON decoding in the engine Tool calls and constrained JSON decoding in the engine | Your implementation of L10.9 passes its course checks. |
| 12 | Open-loop load generator, log-linear histogram, Go PCG32 port Open-loop load generator, log-linear histogram, Go PCG32 port | Your implementation of load.01 passes its course checks. |
| 13 | Run comparison and regression gate Run comparison and regression gate | Your implementation of load.02 passes its course checks. |
| 14 | Milestone L10 Pass 7 milestones | The MS-L10 gate passes for your system. |
| 15 | Server skeleton, composition root, go/config, graceful shutdown Server skeleton, composition root, go/config, graceful shutdown | Your implementation of gw.01 passes its course checks. |
| 16 | AuthN (API keys) and AuthZ (scopes, model allowlist, tenants) AuthN (API keys) and AuthZ (scopes, model allowlist, tenants) | Your implementation of gw.02 passes its course checks. |
| 17 | Rate limiting: RPM token bucket + TPM reserve/settle Rate limiting: RPM token bucket + TPM reserve/settle | Your implementation of gw.03 passes its course checks. |
| 18 | SSE streaming proxy (upgrades gw.00) SSE streaming proxy (upgrades gw.00) | Your implementation of gw.04 passes its course checks. |
| 19 | Consistent hash ring with bounded loads Consistent hash ring with bounded loads | Your implementation of ds.09 passes its course checks. |
| 20 | Worker registry, routing, cascades, failover, disaggregated orchestration Worker registry, routing, cascades, failover, disaggregated orchestration | Your implementation of gw.05 passes its course checks. |
| 21 | Response cache: LRU + TTL, singleflight, tenant-scoped keys Response cache: LRU + TTL, singleflight, tenant-scoped keys | Your implementation of gw.06 passes its course checks. |
| 22 | SQL and SQLite: schema, transactions, WAL mode, indexes, aggregates SQL and SQLite: schema, transactions, WAL mode, indexes, aggregates | Your implementation of lang.11 passes its course checks. |
| 23 | Usage ledger, metering, admin API Usage ledger, metering, admin API | Your implementation of gw.07 passes its course checks. |
| 24 | Contract tests (R6): consumer-driven tests for gateway to engine Contract tests (R6): consumer-driven tests for gateway to engine | Your implementation of craft.20 passes its course checks. |
| 25 | Milestone gateway Pass 7 milestones | The MS-gateway gate passes for your system. |
| 26 | Dockerfiles for the gateway and the engine Dockerfiles for the gateway and the engine | Your implementation of dep.01 passes its course checks. |
| 27 | kind cluster + local registry + port mappings kind cluster + local registry + port mappings | Your implementation of dep.02 passes its course checks. |
| 28 | Helm charts for the gateway and the engine, plus the observability stack Helm charts for the gateway and the engine, plus the observability stack | Your implementation of dep.03 passes its course checks. |
| 29 | Tilt dev loop Tilt dev loop | Your implementation of dep.04 passes its course checks. |
| 30 | CI for the learner repo CI for the learner repo | Your implementation of dep.05 passes its course checks. |
| 31 | Tracing for the serving path Tracing for the serving path | Your implementation of obs.01 passes its course checks. |
| 32 | Metrics and log correlation Metrics and log correlation | Your implementation of obs.02 passes its course checks. |
| 33 | SLOs and multi-window burn-rate alerts SLOs and multi-window burn-rate alerts | Your implementation of obs.03 passes its course checks. |
| 34 | Dashboards as code for the serving path Dashboards as code for the serving path | Your implementation of obs.04 passes its course checks. |
| 35 | A decode worker dies mid-stream A decode worker dies mid-stream | You complete the drill and its resolve check passes. |
| 36 | Code review: review a seeded PR against your system Code review: review a seeded PR against your system | Your implementation of craft.08 passes its course checks. |
| 37 | Design review: tracer and serving platform Design review: tracer and serving platform | Your implementation of review.01 passes its course checks. |
| 38 | Milestone prod Pass 7 milestones | The MS-prod gate passes for your system. |
| 39 | Milestone P7 Pass 7 milestones | The MS-P7 gate passes for your system. |