Learning Path

A suggested order for working through the AI playground, from LLM internals to GPU optimization. Phase 0 is an optional refresher β€” skip it if MLPs, backprop, and cross-entropy are already second nature.

Phase 0: DNN Refresher (optional)

Neural-network fundamentals rebuilt from scratch, at a scale where every number is checkable by hand. Everything later assumes these.

0.1. Neurons, Layers, and the Forward Pass β€” notebooks/00_dnn_refresher/00_neurons_and_mlps.ipynb - A neuron is wΒ·x + b + nonlinearity; a layer is a matmul - Why stacked linear layers collapse; XOR by hand; universal approximation - Code: src/ai_playground/fundamentals/nn.py, datasets.py

0.2. Backprop from Scratch: a Tiny Autograd Engine β€” notebooks/00_dnn_refresher/01_backprop_micrograd.ipynb - Chain rule on a computation graph; a micrograd-style Value class - grad += accumulation, topological order, the PyTorch bridge - Code: src/ai_playground/fundamentals/autograd.py - Paper: Rumelhart, Hinton & Williams (1986)

0.3. The Training Loop: Gradient Descent to Minibatches β€” notebooks/00_dnn_refresher/02_training_loop.ipynb - forward β†’ zero_grad β†’ backward β†’ step; learning-rate regimes - Minibatch SGD; the same loop in PyTorch, with the wall-clock gap measured

0.4. Softmax, Cross-Entropy, and Your First Classifier β€” notebooks/00_dnn_refresher/03_softmax_crossentropy.ipynb - Temperature, subtract-max stability, the p βˆ’ y gradient - Why cross-entropy (not MSE) trains classifiers β€” and LLMs

Milestone: You can derive and implement backprop by hand, and β€œtrain a network” means a specific five-line loop to you.

πŸ”¨ From-scratch project: p0 β€” Autograd from a blank file β€” rebuild the engine with no reference open: scalar autograd, an MLP, and a training loop that separates the moons. uv run pytest projects/p0_grad_engine/

Phase 1: Transformer Internals

Understand every component of a modern LLM before optimizing anything.

  1. Attention Mechanisms β€” notebooks/01_transformer_internals/01_attention_mechanisms.ipynb
    • Multi-Head, Grouped-Query, Multi-Query attention
    • KV cache memory implications
    • Code: src/ai_playground/models/attention.py
  2. Positional Encodings β€” notebooks/01_transformer_internals/02_positional_encodings.ipynb
    • Sinusoidal, RoPE, ALiBi
    • Why RoPE won (relative positions, length extrapolation)
    • Code: src/ai_playground/models/layers.py (precompute_rope_frequencies, apply_rope)
  3. Normalization & Activations β€” notebooks/01_transformer_internals/03_activation_functions.ipynb, 04_normalization.ipynb
    • LayerNorm vs RMSNorm (why skip the mean?)
    • GELU vs SwiGLU (gated activations)
    • Code: src/ai_playground/models/layers.py (RMSNorm, SwiGLU)
  4. Tokenizers β€” notebooks/01_transformer_internals/05_tokenizers_deep_dive.ipynb
    • BPE algorithm internals
    • tiktoken vs SentencePiece

Milestone: You can read any LLM architecture paper and understand every component.

πŸ”¨ From-scratch project: p1 β€” A tiny GPT from a blank file β€” a working causal transformer in one file, no ai_playground.models, no nn.MultiheadAttention. uv run pytest projects/p1_tiny_gpt/

Phase 2: Training Optimization

Make training fast and stable on a single GPU.

  1. Mixed Precision β€” notebooks/02_training_optimization/01_mixed_precision.ipynb
    • FP32 vs FP16 vs BF16 vs FP8
    • Loss scaling and why BF16 doesn’t need it
    • Code: src/ai_playground/training/trainer.py (AMP setup)
  2. Gradient Accumulation β€” notebooks/02_training_optimization/02_gradient_accumulation.ipynb
    • Simulating large batches on small GPUs
    • Code: src/ai_playground/training/trainer.py (accumulation loop)
  3. Optimizers β€” notebooks/02_training_optimization/04_optimizer_internals.ipynb
    • AdamW from scratch (every step explained)
    • Code: src/ai_playground/training/optimizers.py
  4. Learning Rate Schedules β€” notebooks/02_training_optimization/03_lr_schedules.ipynb
    • Cosine decay with warmup
    • WSD (Warmup-Stable-Decay)
    • Code: src/ai_playground/training/trainer.py (get_lr)

Milestone: You can train a model efficiently on a single GPU and understand every training hyperparameter.

πŸ”¨ From-scratch project: p2 β€” The training stack from a blank file β€” your own AdamW (matched to torch’s to 1e-5), warmup+cosine schedule, and gradient accumulation proven exactly equivalent to the big batch. uv run pytest projects/p2_trainer/

Phase 3: Inference Optimization

Make inference fast and memory-efficient.

  1. KV Cache β€” notebooks/04_inference_optimization/01_kv_cache.ipynb
    • Why generation without cache is O(n^2)
    • Prefill vs decode phases
    • Code: src/ai_playground/inference/generate.py, models/attention.py (cache logic)
  2. Quantization β€” notebooks/04_inference_optimization/02_quantization.ipynb
    • INT8 absmax, GPTQ, AWQ, GGUF
    • Code: src/ai_playground/inference/quantize.py
  3. Flash Attention β€” notebooks/04_inference_optimization/05_flash_attention.ipynb
    • IO-aware attention (memory hierarchy)
    • Code: models/attention.py (SDPA path)
  4. Speculative Decoding β€” notebooks/04_inference_optimization/03_speculative_decoding.ipynb
    • Draft + verify for faster generation

Milestone: You understand every optimization used in vLLM, TGI, and other serving frameworks.

πŸ”¨ From-scratch project: p3 β€” A KV-cached decoder from a blank file β€” a given model, your inference engine: full-recompute greedy, KV-cached greedy (token-for-token identical), top-p sampling. uv run pytest projects/p3_kv_serve/

Phase 4: Distributed Training

Scale to multiple GPUs and nodes.

  1. Data Parallelism β€” notebooks/03_distributed_training/01_data_parallel.ipynb
    • DDP: replicate model, split data, all-reduce gradients
    • FSDP: shard everything (ZeRO Stage 3)
    • Code: src/ai_playground/training/distributed.py
  2. Tensor Parallelism β€” notebooks/03_distributed_training/02_tensor_parallel.ipynb
    • Megatron-style column/row parallel Linear layers
  3. Pipeline Parallelism β€” notebooks/03_distributed_training/03_pipeline_parallel.ipynb
    • GPipe, 1F1B schedules, micro-batching

Milestone: You can design a parallelism strategy for training any model on any cluster.

πŸ”¨ From-scratch project: p4 β€” Data parallelism from a blank file β€” hand-rolled DDP over raw collectives, 2 CPU processes, proven numerically invisible vs a single-process run. uv run pytest projects/p4_ddp/

Phase 5: GPU & NVIDIA Tools

Squeeze maximum performance from hardware.

  1. CUDA Basics β€” notebooks/05_gpu_nvidia_tools/01_cuda_basics.ipynb
    • Memory hierarchy, warps, occupancy
    • Writing kernels with Numba/CuPy
  2. Triton Kernels β€” notebooks/05_gpu_nvidia_tools/02_triton_kernels.ipynb
    • Fused operations (why fusing matters for memory bandwidth)
    • Writing custom kernels in Triton
  3. Profiling β€” notebooks/05_gpu_nvidia_tools/03_nsight_profiling.ipynb
    • Nsight Systems (timeline), Nsight Compute (kernel-level)
    • Code: src/ai_playground/profiling/nsight.py
  4. MFU β€” Use profiling/flops.py to measure and optimize Model FLOP Utilization
    • Target: 40-60% on modern GPUs

Milestone: You can profile, diagnose, and fix GPU performance bottlenecks.

πŸ”¨ From-scratch project: p5 β€” A decode roofline model from a blank file β€” FLOPs, bytes, intensity, and tokens/sec predictions for any config on any GPU spec, no GPU required. uv run pytest projects/p5_roofline/

Phase 6: Building AI Agents

Move up the stack: instead of training and serving a model, use one in a loop with tools, memory, and planning. This is the layer most β€œAI products” actually live at.

  1. What is an agent? β€” notebooks/06_agents/00_what_is_an_agent.ipynb
    • The ReAct loop in <20 lines from scratch
    • Why tool calling is just structured text + parser
    • Code: src/ai_playground/agents/agent.py
    • Paper: ReAct (Yao et al., 2022)
  2. Tool use β€” notebooks/06_agents/01_tool_use.ipynb
    • Tool anatomy (name + description + JSON schema + callable)
    • Tool registry and dispatch with graceful error reporting
    • Trade-offs: tool granularity, security, idempotency
    • Code: src/ai_playground/agents/tools.py
  3. Memory systems β€” notebooks/06_agents/02_memory.ipynb
    • Working memory (bounded conversation + summarization)
    • Long-term memory (vector store + similarity retrieval)
    • Code: src/ai_playground/agents/memory.py
    • Paper: MemGPT (Packer et al., 2023)
  4. Planning & multi-agent β€” src/ai_playground/agents/planner.py, multi_agent.py

Milestone: You can design and build an agent for a real task (e.g. a research assistant, a code reviewer, a triage bot) and know where the failure modes live.

πŸ”¨ From-scratch project: p6 β€” An agent from a blank file β€” your own ReAct loop with tool dispatch, error observations, and a step budget, tested against a scripted LLM (no API key). uv run pytest projects/p6_agent/

Every phase’s from-scratch project is indexed in projects/README.md, including the rules of the game (what each one bans you from importing) and how the acceptance tests skip-until-attempted.