Performance engineering for LLM inference

Each project builds on the same engine, with a standalone entry lab if you want to explore one mechanism first.

1. 01Reconstruct Qwen3Weeks 1–2Draft

   Trace the forward pass, load real weights, and make tensor shapes and memory use explicit.

   You’ll work towardA validated model implementation and a tensor / memory inventory.

2. 02Build the runtime & KV cacheWeeks 3–4Draft

   Implement request scheduling, continuous batching, and bounded KV allocation.

   You’ll work towardAn explanation of latency, throughput, and memory tradeoffs.

3. 03Predict inference performanceWeeks 5–6Draft

   Calibrate a performance model, identify bottlenecks, and test predictions on held-out workloads.

   You’ll work towardPrefill and decode predictions you can compare with measurements.

4. 04Optimize GEMM & attentionWeeks 7–8Draft

   Study a GEMM and an attention kernel, then measure their effect inside the engine.

   You’ll work towardA kernel investigation that connects local changes to model execution.

5. 05Explore speculative decodingWeeks 9–10Draft

   Use 8B as the draft and 32B as the target. Implement verification, cache rollback, and correct sampling.

   You’ll work towardA measured break-even analysis, whether speculation helps or hurts.

6. 06Quantize & evaluate qualityWeeks 11–12Draft

   Evaluate a real quantization path across quality, memory, latency, and serving capacity.

   You’ll work towardAn evidence-based comparison, including interactions with speculation.

7. 07Scale with tensor parallelismWeeks 13–14Draft

   Implement sharding and collectives, then compare tensor parallelism with independent replicas.

   You’ll work towardA justified choice for serving a fixed workload on two GPUs.

8. 08Separate prefill & decodeWeeks 15–16Draft

   Transfer KV state between workers and evaluate the costs of disaggregated serving.

   You’ll work towardA final serving design defended with reproducible evidence.