Inference Performance Exercises

Serving-side companions to the Performance Debugging Exercises: the same scenario → diagnose → fix format, focused on what determines inference cost — KV-cache memory, decode bandwidth, batching, and quantization.

These come after the inference chapter, not before it. Each exercise’s Needs line names the Learning Path phase it assumes; most build directly on the mini-vLLM chapter and work as retention checks right after its sessions.

# Exercise Needs (do after) Sessions
1 The KV Cache Serving Budget Phase 3 — Inference Optimization; M2’s cache math scaled up S4.1
2 Decode Is Memory-Bound, Not Compute-Bound Phase 3 + Phase 5’s roofline (M9; see also project p5) S5.1–S5.2, S4.1
3 TTFT vs Throughput — the Batching Tradeoff Phase 3 — Inference Optimization, through continuous batching S4.2
4 The Quantization Quality–Speed Tradeoff Phase 3 — Inference Optimization S4.1–S4.2

Exercises 1 and 3 run fine on CPU/MPS (Exercise 1 is mostly arithmetic); 2 and 4 want a GPU for representative numbers (see CLOUD_SETUP.md) but degrade gracefully to the tiny config.

Exercise 1: The KV Cache Serving Budget

Needs: Phase 3 — Inference Optimization (S4.1). This is M2’s bytes-per-token calculation, scaled from one model to a serving fleet — and the reason the mini-vLLM chapter pages the cache at all.

Scenario: You’re putting the SMALL model (~125M params) behind an API on a 24 GB GPU. Product asks: “how many concurrent chats can one GPU hold at a 2048-token context?” Your answer decides the fleet size.

Task: Compute the serving budget from the config alone, then find where the memory actually goes at realistic request lengths.

from ai_playground.models.config import SMALL

cfg = SMALL
DTYPE_BYTES = 2  # BF16

kv_per_token = 2 * cfg.n_layers * cfg.kv_heads * cfg.head_dim * DTYPE_BYTES
kv_per_request = kv_per_token * cfg.max_seq_len
weights = cfg.num_params() * DTYPE_BYTES

print(f"KV per token:   {kv_per_token / 1024:.1f} KB")
print(f"KV per request: {kv_per_request / 1024**2:.1f} MB at {cfg.max_seq_len} ctx")
print(f"Weights:        {weights / 1024**2:.0f} MB")

Questions: 1. With ~2 GB reserved for weights, activations, and CUDA context, how many concurrent 2048-context requests fit in 24 GB? How does the answer change for MHA (n_kv_heads=12) and MQA (n_kv_heads=1)? 2. Real requests average ~300 tokens of context, but you must reserve for the worst case. What fraction of your reserved KV memory is actually used if you preallocate max_seq_len per request? 3. With paged allocation (16-token blocks, as in the mini-vLLM chapter), what’s the worst-case waste per request? 4. At what context length does one request’s KV cache exceed the model’s own weights?

Hints
  • The formula is the M2 one: 2 (K and V) × n_layers × n_kv_heads × head_dim × bytes per token. For SMALL with GQA-4 in BF16 that’s 12 KB/token — 24 MB per 2048-token request.
  • Concurrency = (VRAM − weights − overhead) / KV-per-request. The n_kv_heads factor passes straight through: MHA costs 3× GQA-4, MQA 1/4 of it.
  • Preallocating worst-case for average-case traffic is the fragmentation the chapter opens with: 300/2048 ≈ 15% utilization — 85% of your “full” GPU is reserved-but-unused.
  • Paged waste is bounded by the last partially-filled block: at most block_size − 1 tokens (~180 KB), independent of max_seq_len.
Tool to use

TransformerConfig (kv_heads, head_dim, n_layers, num_params()) — the whole exercise is derivable from the config; that’s the point. Cross-check the per-token number against your own M2 entry in checkpoints/myllm/config.json, scaled to SMALL’s dimensions.

To verify empirically on a GPU: run ai_playground.inference.benchmark.benchmark_generation at growing batch_size and watch peak_memory_mb — its slope per extra request should match your KV-per-request figure (plus activation noise).

For the paging side, the mini-vLLM chapter block-table cells are the reference implementation.

Solution 
from ai_playground.models.config import SMALL, TransformerConfig

DTYPE_BYTES = 2
VRAM = 24 * 1024**3
OVERHEAD = 2 * 1024**3  # weights live inside this for SMALL (~200 MB) + activations + context

def kv_per_token(cfg):
    return 2 * cfg.n_layers * cfg.kv_heads * cfg.head_dim * DTYPE_BYTES

variants = {
    "MHA (12 KV heads)": TransformerConfig(dim=768, n_layers=12, n_heads=12, n_kv_heads=12),
    "GQA (4 KV heads)":  SMALL,
    "MQA (1 KV head)":   TransformerConfig(dim=768, n_layers=12, n_heads=12, n_kv_heads=1),
}

print(f"{'Variant':<20} {'KB/token':>9} {'MB/request':>11} {'Concurrent @24GB':>17}")
for name, cfg in variants.items():
    per_tok = kv_per_token(cfg)
    per_req = per_tok * 2048
    fits = (VRAM - OVERHEAD) // per_req
    print(f"{name:<20} {per_tok/1024:>9.1f} {per_req/1024**2:>11.1f} {fits:>17,}")

# Q2: utilization with worst-case preallocation
avg_ctx, max_ctx = 300, 2048
print(f"\nPreallocation utilization: {avg_ctx/max_ctx:.0%} "
      f"({1 - avg_ctx/max_ctx:.0%} of reserved KV memory never used)")

# Q3: paged waste bound
block = 16
waste = (block - 1) * kv_per_token(SMALL)
print(f"Paged worst-case waste/request: {waste/1024:.0f} KB (vs "
      f"{(max_ctx-avg_ctx)*kv_per_token(SMALL)/1024**2:.1f} MB wasted by preallocation)")

# Q4: context where KV > weights
weights = SMALL.num_params() * DTYPE_BYTES
crossover = weights / kv_per_token(SMALL)
print(f"KV cache exceeds weights beyond {crossover:,.0f} tokens of context")

Expected results (SMALL, BF16):

Variant KB/token MB/request @2048 Concurrent @24 GB
MHA (12 KV heads) 36 72 ~312
GQA (4 KV heads) 12 24 ~938
MQA (1 KV head) 3 6 ~3,750
  • Q2: 15% utilization — preallocating worst-case for 300-token average traffic wastes 85% of the KV reservation. This, not raw capacity, is why naive serving runs out of memory at a fraction of the theoretical concurrency.
  • Q3: Paged allocation bounds waste at block_size − 1 tokens ≈ 180 KB per request — a ~100× improvement over the ~21 MB preallocation waste, at the cost of the block-table indirection the chapter builds.
  • Q4: ~17k tokens for SMALL. For production-size models the crossover is far lower relative to capacity — which is why PagedAttention (Kwon et al., 2023) (see docs/PAPERS.md) frames serving as a memory-management problem, not a compute problem.

The scaling law to remember: weights are paid once per GPU; KV is paid per request × per token. Every serving optimization in this phase (GQA, paging, quantized KV) attacks the second term.

Exercise 2: Decode Is Memory-Bound, Not Compute-Bound

Needs: Phase 3 — Inference Optimization (S4.1) and Phase 5’s roofline (S5.1–S5.2, M9). This is your M9 conclusion measured empirically — and the from-scratch project p5 builds the general calculator.

Scenario: During autoregressive generation, your GPU utilization drops to 5% even though you’re processing tokens as fast as possible.

Task: Understand why decode is fundamentally different from prefill and what determines decode speed.

import torch
from ai_playground.inference.benchmark import benchmark_generation, print_benchmark
from ai_playground.models import Transformer, TransformerConfig

config = TransformerConfig.SMALL
model = Transformer(config).cuda().eval()

# Compare prefill vs decode rates
results = benchmark_generation(model, prompt_len=512, gen_len=512,
                                batch_size=1, dtype=torch.bfloat16)
print_benchmark(results)
print(f"Prefill/Decode ratio: {results['prefill_tokens_per_sec'] / results['decode_tokens_per_sec']:.1f}x")

Questions: 1. Why is prefill so much faster per token than decode? 2. What’s the arithmetic intensity (FLOPs per byte loaded) of a decode step vs a prefill step? 3. How does batching multiple requests help decode throughput? 4. Why do serving systems like vLLM use continuous batching?

Hints
  • Prefill processes all prompt tokens in one big matrix multiplication (e.g., [batch, seq_len, dim] @ [dim, dim]). This is compute-bound — the GPU’s tensor cores are fully utilized.
  • Decode processes one token at a time ([batch, 1, dim] @ [dim, dim]). The matrix multiply is tiny, but you still need to load the entire weight matrix from GPU memory. This is memory-bandwidth-bound.
  • Arithmetic intensity: prefill with seq_len=512 does 512x more FLOPs per weight byte loaded. Decode does ~2 FLOPs per byte (one multiply, one add) — far below the GPU’s compute-to-bandwidth ratio.
  • Batching helps because you load weights once and multiply against multiple sequences. Going from batch=1 to batch=32 is nearly free in latency but 32x the throughput.
Tool to use

ai_playground.inference.benchmark.benchmark_generation — measures prefill and decode separately, reporting tokens/sec and latency for each phase. This is the primary diagnostic tool.

For understanding why decode is slow, compute the roofline model analytically: - Arithmetic intensity = FLOPs / bytes loaded from memory - Compare to GPU’s ops:byte ratio (A100: 312 TFLOPS / 2 TB/s = 156 FLOP/byte) - If your operation’s arithmetic intensity < 156, it’s memory-bound on A100

Use ai_playground.profiling.flops.estimate_flops with batch_size=1, seq_len=1 for decode vs seq_len=512 for prefill to see the compute difference analytically.

To verify empirically, vary batch size and measure decode tokens/sec — if throughput scales linearly with batch size, you’re memory-bound (loading weights once, doing more compute per load).

Solution 
import torch
from ai_playground.models import Transformer, TransformerConfig
from ai_playground.inference.benchmark import benchmark_generation, print_benchmark
from ai_playground.profiling.flops import estimate_flops

config = TransformerConfig.SMALL
model = Transformer(config).cuda().eval()

# Part 1: Measure prefill vs decode at different batch sizes
print("=== Prefill vs Decode ===\n")
for batch_size in [1, 4, 16, 32]:
    try:
        results = benchmark_generation(
            model, prompt_len=512, gen_len=128,
            batch_size=batch_size, dtype=torch.bfloat16
        )
        ratio = results['prefill_tokens_per_sec'] / results['decode_tokens_per_sec']
        print(f"batch={batch_size:>2}: prefill={results['prefill_tokens_per_sec']:>8.0f} tok/s, "
              f"decode={results['decode_tokens_per_sec']:>8.0f} tok/s, "
              f"ratio={ratio:.0f}x")
    except torch.cuda.OutOfMemoryError:
        print(f"batch={batch_size:>2}: OOM")
        break

# Part 2: Roofline analysis
print(f"\n=== Roofline Analysis ===\n")
params = config.num_params()
weight_bytes_bf16 = params * 2  # BF16

# A100 specs
a100_bandwidth_bytes = 2e12  # 2 TB/s
a100_peak_tflops = 312e12   # 312 TFLOPS BF16

ops_byte_ratio = a100_peak_tflops / a100_bandwidth_bytes
print(f"A100 ops:byte ratio: {ops_byte_ratio:.0f} FLOP/byte")
print(f"(Operations below this ratio are memory-bound)\n")

# Decode: [1, 1, 768] @ [768, 768] per linear layer
decode_flops = estimate_flops(config, seq_len=1, batch_size=1)
decode_intensity = decode_flops['forward_tflops'] * 1e12 / weight_bytes_bf16
print(f"Decode (batch=1):  {decode_intensity:.1f} FLOP/byte  → {'MEMORY-BOUND' if decode_intensity < ops_byte_ratio else 'COMPUTE-BOUND'}")

# Prefill: [1, 512, 768] @ [768, 768] per linear layer
prefill_flops = estimate_flops(config, seq_len=512, batch_size=1)
prefill_intensity = prefill_flops['forward_tflops'] * 1e12 / weight_bytes_bf16
print(f"Prefill (seq=512): {prefill_intensity:.1f} FLOP/byte → {'MEMORY-BOUND' if prefill_intensity < ops_byte_ratio else 'COMPUTE-BOUND'}")

# Batched decode
for batch_size in [1, 8, 32, 128]:
    decode_flops = estimate_flops(config, seq_len=1, batch_size=batch_size)
    intensity = decode_flops['forward_tflops'] * 1e12 / weight_bytes_bf16
    bound = 'MEMORY' if intensity < ops_byte_ratio else 'COMPUTE'
    print(f"Decode (batch={batch_size:>3}): {intensity:>6.1f} FLOP/byte → {bound}-BOUND")

# Part 3: Theoretical minimum decode latency
min_latency_ms = weight_bytes_bf16 / a100_bandwidth_bytes * 1000
print(f"\nTheoretical minimum per-token latency (A100): {min_latency_ms:.2f} ms")
print(f"  = time to load all {params/1e6:.0f}M params from HBM")
print(f"  = {1000/min_latency_ms:.0f} tokens/sec max (batch=1)")

Expected results (small.yaml on A100):

A100 ops:byte ratio: 156 FLOP/byte

Decode (batch=1):    2.0 FLOP/byte  → MEMORY-BOUND    (78x below roofline!)
Prefill (seq=512): 512.0 FLOP/byte  → COMPUTE-BOUND
Decode (batch=128): 256.0 FLOP/byte → COMPUTE-BOUND

Theoretical minimum per-token latency: 0.12 ms = 8,000 tokens/sec max (batch=1)

Key takeaways: 1. Decode at batch=1 wastes 98.7% of the GPU’s compute. You load the entire model just to compute one vector-matrix multiply. 2. Batching is the primary fix. At batch=128, you’re doing 128x more FLOPs per weight byte loaded, crossing into compute-bound territory. 3. Continuous batching (vLLM, TGI) keeps the batch full by injecting new requests as old ones finish. Static batching wastes GPU time when sequences finish at different times. 4. Speculative decoding is another fix: use a small draft model to predict N tokens, then verify all N in one batched forward pass of the large model.

Exercise 3: TTFT vs Throughput — the Batching Tradeoff

Needs: Phase 3 — Inference Optimization through continuous batching (S4.2).

Scenario: Your API has a latency SLO: time-to-first-token under 200 ms. Batching more requests together raises throughput (Exercise 2 says decode is memory-bound, so batching is nearly free) — but prefill work grows with every request in the batch, and TTFT with it. Where’s the ceiling?

Task: Measure both sides of the tradeoff and find the largest SLO-compliant batch.

import torch
from ai_playground.models import Transformer
from ai_playground.models.config import SMALL
from ai_playground.inference.benchmark import benchmark_generation

model = Transformer(SMALL).cuda().eval()

for batch_size in [1, 2, 4, 8, 16, 32]:
    r = benchmark_generation(model, prompt_len=512, gen_len=128,
                             batch_size=batch_size, dtype=torch.bfloat16)
    print(f"batch={batch_size:>2}: TTFT={r['time_to_first_token_ms']:>7.1f} ms, "
          f"decode={r['decode_tokens_per_sec']:>8.0f} tok/s")

Questions: 1. How does TTFT scale with batch size? Is it linear? Why would it be? 2. How does decode throughput scale? Where does it stop scaling linearly (and what does that say about Exercise 2’s roofline)? 3. What’s the largest batch meeting a 200 ms TTFT SLO — and what throughput do you give up versus the largest batch you measured? 4. Static batching admits requests in fixed groups: a request arriving just after a batch launches waits a full batch cycle, and every batch member waits for the slowest to finish. How does continuous batching (the mini-vLLM scheduler) decouple these?

Hints
  • Prefill is compute-bound (Exercise 2): batching multiplies the work, so TTFT grows roughly linearly once the GPU is saturated — batch 32 × 512-token prompts is 16k tokens of prefill in one shot.
  • Decode is bandwidth-bound: weights are read once per step regardless of batch, so tokens/sec grows near-linearly with batch until the KV-cache reads (which do scale with batch) start to dominate — reconnect this to your Exercise 2 intensity numbers.
  • The tension is structural: the batch size that maximizes throughput and the one that honors TTFT are different. A fixed batch size is the wrong knob.
  • Continuous batching resolves it by scheduling at token granularity: new requests prefill in the gaps while resident requests keep decoding, so admission latency stops being a multiple of batch-completion time.
Tool to use

ai_playground.inference.benchmark.benchmark_generation — reports time_to_first_token_ms (the prefill side) and decode_tokens_per_sec (the throughput side) per batch size; peak_memory_mb tells you when the KV budget from Exercise 1 becomes the binding constraint instead of the SLO.

The mini-vLLM chapter scheduler — after measuring, reread its prefill-vs-decode admission logic: the code answers question 4. Your own M13c serve() is the same logic on your model.

On CPU/MPS use the tiny config and scale the SLO up (~2 s); the shape of both curves — linear TTFT, saturating throughput — is hardware-independent.

Solution 
import torch
from ai_playground.models import Transformer
from ai_playground.models.config import SMALL
from ai_playground.inference.benchmark import benchmark_generation

model = Transformer(SMALL).cuda().eval()
SLO_MS = 200

rows = []
for batch_size in [1, 2, 4, 8, 16, 32]:
    r = benchmark_generation(model, prompt_len=512, gen_len=128,
                             batch_size=batch_size, dtype=torch.bfloat16)
    rows.append(r)

best = None
print(f"{'batch':>5} {'TTFT ms':>9} {'decode tok/s':>13} {'tok/s per req':>14} {'SLO':>5}")
for r in rows:
    ok = r["time_to_first_token_ms"] <= SLO_MS
    if ok:
        best = r
    print(f"{r['batch_size']:>5} {r['time_to_first_token_ms']:>9.1f} "
          f"{r['decode_tokens_per_sec']:>13.0f} "
          f"{r['decode_tokens_per_sec']/r['batch_size']:>14.0f} "
          f"{'✓' if ok else '✗':>5}")

print(f"\nLargest SLO-compliant batch: {best['batch_size']} "
      f"({best['decode_tokens_per_sec']:.0f} tok/s)")
print(f"Throughput ceiling at batch {rows[-1]['batch_size']}: "
      f"{rows[-1]['decode_tokens_per_sec']:.0f} tok/s "
      f"({rows[-1]['decode_tokens_per_sec']/best['decode_tokens_per_sec']:.1f}x more, "
      f"but TTFT {rows[-1]['time_to_first_token_ms']:.0f} ms)")

Expected shape (A100, SMALL, 512-token prompts):

batch   TTFT ms  decode tok/s  tok/s per req   SLO
    1      ~40         ~1,900          1,900     ✓
    4     ~120         ~7,200          1,800     ✓
    8     ~230        ~13,500          1,690     ✗
   32     ~880        ~38,000          1,190     ✗
  1. TTFT grows ~linearly with batch once prefill saturates the GPU — it’s compute-bound work, and batching multiplies the work. (At tiny batches it’s sublinear: the GPU wasn’t saturated yet.)
  2. Decode throughput grows near-linearly, then bends — per-request tok/s decays as batched KV reads eat the bandwidth that weight-amortization freed. This is Exercise 2’s arithmetic-intensity curve read off real hardware.
  3. On the numbers above, batch 4 is the largest SLO-compliant point: you serve at ~7.2k tok/s and leave ~5x throughput on the table — that gap is the price of the SLO under static batching.
  4. Continuous batching removes the coupling, not the tradeoff. With token-level scheduling (Orca, Yu et al., OSDI 2022; vLLM, Kwon et al., 2023 — see docs/PAPERS.md), a new request’s TTFT is one prefill slice, not “wait for the current batch”; finished requests free their slots immediately instead of padding out the batch. The resident batch stays large (throughput) while admission stays fast (TTFT). That’s exactly the scheduler you built at M13c — rerun your serve() and watch it interleave.

Exercise 4: The Quantization Quality–Speed Tradeoff

Needs: Phase 3 — Inference Optimization (S4.1–S4.2).

Scenario: You need to serve the model and want to reduce memory and latency using INT8 quantization. But how much quality do you lose?

Task: Quantize the model, measure the speedup, and check if outputs diverge.

import torch
from ai_playground.models import Transformer, TransformerConfig
from ai_playground.inference.quantize import quantize_model_weights
from ai_playground.inference.benchmark import benchmark_generation, print_benchmark

config = TransformerConfig.SMALL
model = Transformer(config).cuda().eval()

# Baseline benchmark
results_fp32 = benchmark_generation(model, prompt_len=128, gen_len=256)
print_benchmark(results_fp32)

# Quantize and check compression
stats = quantize_model_weights(model)
print(f"Original:    {stats['original_mb']:.1f} MB")
print(f"Quantized:   {stats['quantized_mb']:.1f} MB")
print(f"Compression: {stats['compression_ratio']:.2f}x")

# Compare outputs before and after quantization
prompt = torch.randint(0, config.vocab_size, (1, 64)).cuda()
with torch.no_grad():
    logits_fp32 = model(prompt)
    # TODO: load quantized weights back and compare logits

Questions: 1. What compression ratio do you get with INT8 absmax? 2. How does the max absolute error in logits compare to the mean? 3. Is absmax quantization per-tensor or per-channel? Why does that matter? 4. At what point does quantization error actually affect generation quality?

Hints
  • INT8 absmax on FP32 weights gives ~4x compression (4 bytes → 1 byte per weight).
  • Per-tensor quantization (what quantize.py implements) uses one scale for the entire tensor. Outlier values waste dynamic range for the rest. Per-channel quantization uses one scale per output channel — much better quality.
  • Check for outliers: weight.abs().max() / weight.abs().mean(). Ratios above 10-20 indicate outliers that hurt per-tensor quantization.
  • In practice, absmax INT8 works well for weights >100M params. For smaller models, the quantization noise is proportionally larger.
Tool to use

ai_playground.inference.quantize.quantize_tensor_absmax and dequantize_tensor — quantize individual tensors and measure round-trip error. Use these to inspect specific layers.

ai_playground.inference.quantize.quantize_model_weights — quantize all linear layers and get compression stats.

ai_playground.inference.benchmark.benchmark_generation — measure end-to-end inference speed before and after quantization.

For analyzing outliers, use plain PyTorch tensor operations — .abs().max(), .abs().mean(), histograms with torch.histc().

Solution 
import torch
from ai_playground.models import Transformer, TransformerConfig
from ai_playground.inference.quantize import (
    quantize_tensor_absmax, dequantize_tensor, quantize_model_weights
)

config = TransformerConfig.SMALL
model = Transformer(config).cuda().eval()

# Step 1: Understand per-tensor quantization error
print("=== Per-layer quantization analysis ===\n")
for name, param in model.named_parameters():
    if param.dim() < 2:
        continue  # skip norms

    q, scale = quantize_tensor_absmax(param.data, bits=8)
    reconstructed = dequantize_tensor(q, scale)
    error = (param.data - reconstructed).abs()

    outlier_ratio = param.data.abs().max().item() / param.data.abs().mean().item()

    print(f"{name:>50}: "
          f"max_err={error.max().item():.6f}, "
          f"mean_err={error.mean().item():.6f}, "
          f"outlier_ratio={outlier_ratio:.1f}")

# Step 2: Compare model outputs before and after quantization
prompt = torch.randint(0, config.vocab_size, (1, 128)).cuda()

with torch.no_grad():
    logits_original = model(prompt).clone()

# Quantize and dequantize all weights (simulating INT8 inference)
for name, param in model.named_parameters():
    if param.dim() >= 2:
        q, scale = quantize_tensor_absmax(param.data, bits=8)
        param.data = dequantize_tensor(q, scale)

with torch.no_grad():
    logits_quantized = model(prompt)

# Compare logits
logit_error = (logits_original - logits_quantized).abs()
print(f"\n=== Logit comparison ===")
print(f"Max absolute error:  {logit_error.max().item():.4f}")
print(f"Mean absolute error: {logit_error.mean().item():.4f}")
print(f"Logit range:         [{logits_original.min().item():.2f}, {logits_original.max().item():.2f}]")

# Do the top-1 predictions change?
top1_original = logits_original.argmax(dim=-1)
top1_quantized = logits_quantized.argmax(dim=-1)
agreement = (top1_original == top1_quantized).float().mean()
print(f"Top-1 agreement:     {agreement.item():.1%}")

# Step 3: Check compression
stats = quantize_model_weights(model)
print(f"\n=== Compression stats ===")
print(f"Original:    {stats['original_mb']:.1f} MB")
print(f"Quantized:   {stats['quantized_mb']:.1f} MB")
print(f"Compression: {stats['compression_ratio']:.2f}x")

Expected results:

  • Compression: ~3.8x (close to theoretical 4x; the scale factor per tensor adds a tiny overhead)
  • Outlier ratios: Layers with ratios >20 will have worse quantization quality. Attention output projections and the first/last layers often have the worst outliers.
  • Top-1 agreement: Typically 90-98% for a 125M model. Errors accumulate across layers, so deeper models show more divergence.
  • Max logit error: Can be 10-100x larger than mean error due to outliers. This is the fundamental weakness of per-tensor quantization.

Why per-channel is better:

# Per-tensor: one scale for entire [out, in] matrix
# If one output channel has a value of 100 and others are <1,
# the scale is set by the 100, wasting 7 bits of range for all other channels.

# Per-channel: one scale per output row
# Each channel uses its full INT8 range independently.
# Outliers in one channel don't affect others.

When quality degrades noticeably: - Models <50M params: too few parameters to absorb the quantization noise - Per-tensor on models with activation outliers (common in LLMs at >1B scale) - When cascading: small errors in early layers amplify through the network