Video Resources

Curated videos matched to this repo’s modules. Watching one before working through the corresponding notebook gives you the intuition; the notebook then makes it concrete in code. All links verified.

DNN Refresher (notebooks/00_dnn_refresher/)

  • But what is a neural network? | Deep Learning Chapter 1 (3Blue1Brown) Neurons, layers, and weights as a function machine — the visual version of the forward pass. Watch before 00_neurons_and_mlps.ipynb. Watch

  • Gradient descent, how neural networks learn | Deep Learning Chapter 2 (3Blue1Brown) The loss landscape and downhill steps, animated. Watch before 02_training_loop.ipynb. Watch

  • Backpropagation, intuitively | Deep Learning Chapter 3 (3Blue1Brown) What the backward pass computes and why, with no calculus notation. Watch before 01_backprop_micrograd.ipynb. Watch

  • Backpropagation calculus | Deep Learning Chapter 4 (3Blue1Brown) The chain-rule bookkeeping behind chapter 3, including the cross-entropy derivative that 03_softmax_crossentropy.ipynb re-derives in code. Watch

  • The spelled-out intro to neural networks and backpropagation: building micrograd (Andrej Karpathy) Builds the exact engine of 01_backprop_micrograd.ipynb in 2.5 hours of live code — src/ai_playground/fundamentals/autograd.py follows its design. Watch

Transformer Internals (notebooks/01_transformer_internals/)

  • Transformers, the tech behind LLMs | Deep Learning Chapter 5 (3Blue1Brown) The best visual intuition for what a transformer actually computes: embeddings as directions in space, the residual stream, and next-token prediction. Watch before 00_transformer_overview.ipynb. Watch

  • Attention in transformers, step-by-step | Deep Learning Chapter 6 (3Blue1Brown) Queries, keys, and values animated — why attention is a soft, differentiable lookup. Pairs directly with 01_attention_mechanisms.ipynb and src/ai_playground/models/attention.py. Watch

  • How might LLMs store facts | Deep Learning Chapter 7 (3Blue1Brown) What the FFN/MLP blocks are for — the part of the architecture attention videos usually skip. Relevant to the SwiGLU implementation in layers.py. Watch

  • Let’s build GPT: from scratch, in code, spelled out (Andrej Karpathy) Builds a GPT in ~2 hours of live coding, from bigram counts to multi-head attention. The single most effective transformer tutorial; our from-scratch model follows the same spirit with LLaMA-style components. Watch

  • Let’s build the GPT Tokenizer (Andrej Karpathy) BPE from scratch — merges, regex splitting, and the weird failure modes tokenization causes (spelling, arithmetic, trailing whitespace). Background for the vocab_size config choice; see also PAPERS.md § Tokenization. Watch

  • A Visual Guide to Mixture of Experts (MoE) in LLMs (Maarten Grootendorst) Dense vs sparse MoE, top-k routing, expert capacity, and load balancing — the concepts implemented in 02_mixture_of_experts.ipynb. Watch

Training Optimization (notebooks/02_training_optimization/)

  • Let’s reproduce GPT-2 (124M) (Andrej Karpathy) A ~4-hour end-to-end training run that hits every topic in module 02: mixed precision, gradient accumulation, fused AdamW, LR warmup + cosine decay, torch.compile, and DDP. The closest thing to this repo in video form. Watch

  • Neural Networks: Zero to Hero (Andrej Karpathy, playlist) The prerequisite series: backprop by hand, micrograd, makemore. If gradients feel shaky, start here before the optimizer notebooks (04_adamw_from_scratch.ipynb). Watch

  • Deep Dive into LLMs like ChatGPT (Andrej Karpathy) The full lifecycle — pretraining, SFT, RLHF — at a conceptual level. Good for placing this repo’s pretraining focus in the bigger picture. Watch

Inference Optimization (notebooks/04_inference_optimization/)

  • Fast LLM Serving with vLLM and PagedAttention (Anyscale) The vLLM authors present PagedAttention and continuous batching — the two ideas built from scratch in 00_mini_vllm.ipynb. See PAPERS.md for the SOSP 2023 paper. Watch

  • The KV Cache: Memory Usage in Transformers (Efficient NLP) The memory math that makes serving a cache-management problem — the right primer before the mini-vLLM chapter. Watch

Distributed Training (notebooks/03_distributed_training/)

  • Distributed Training with PyTorch: complete tutorial with cloud infrastructure and code (Umar Jamil) DDP from the ground up: gradient synchronization, all-reduce, torchrun, and a real multi-node cloud setup. Directly applicable to src/ai_playground/training/distributed.py. Watch

  • Distributed Data Parallel in PyTorch — official video series (PyTorch) Short official tutorials going from single-GPU to multi-node DDP, including a minGPT training example. Watch

GPU & NVIDIA Tools (notebooks/05_gpu_nvidia_tools/)

  • How CUDA Programming Works (Stephen Jones, GTC 2022) A CUDA architect explains the hardware truths behind the programming model: memory bandwidth as the real constraint, warps, occupancy. Pairs with 01_cuda_basics.ipynb, and is the mental model behind the SGEMM notebook’s roofline analysis. Watch

  • Flash Attention derived and coded from first principles with Triton (Umar Jamil) Derives online softmax and tiling from scratch, then implements Flash Attention in Triton. Long but self-contained — no prior CUDA/Triton assumed. See PAPERS.md for the Flash Attention papers. Watch

  • GPU MODE lecture series (GPU MODE) Ongoing lecture series on CUDA, Triton, and GPU performance by practitioners. Lecture 50 recounts a CUDA → Triton → Flash Attention learning journey similar to this repo’s module 05. Channel