Module 10 · before
Training like a pro 6 questions on what this module teaches. Guessing is fine: this is your starting point, not a test. Take the same check again at the end to see what you learned.
1. Predict: Adam (β₁ 0.85, β₂ 0.99, lr 0.1) sees a gradient of 4, then a gradient of 0. How big is the second step? m, v = 0.0, 0.0
for t, g in [(1, 4.0), (2, 0.0)]:
m = 0.85 * m + 0.15 * g
v = 0.99 * v + 0.01 * g ** 2
m_hat = m / (1 - 0.85 ** t)
v_hat = v / (1 - 0.99 ** t)
step = 0.1 * m_hat / (v_hat ** 0.5 + 1e-8)
print(round(step, 3)) 0.065 0.0 0.1 0.128
2. After many training steps, knob A’s gradient is steadily about 50 and knob B’s is steadily about 0.05. With Adam, which one takes the bigger steps? Neither moves, because m and v cancel out B, because Adam always boosts small gradients above lr About the same size: each step is divided by that knob’s own √v A, about 1,000 times bigger
3. Predict: with microgpt’s schedule, how many of its 1,000 steps use a learning rate above 0.005? lr, num_steps = 0.01, 1000
fast = [step for step in range(num_steps) if lr * (1 - step / num_steps) > 0.005]
print(len(fast)) 1000 500 250 750
4. Your training run’s moving average (window 100) sits at about 3.30 from start to finish, while single steps bounce between 3.2 and 3.4. What does it tell you? The window is too small to show the trend It is not learning: 3.30 is the know-nothing score, so check the update step It is learning normally; the bouncing is just noise from single names It has beaten counting, which scores 2.45
5. A friend moves p.grad = 0 to the top of the Adam loop, before the m[i] line, instead of after the update. What happens? Gradients pile up from step to step Nothing changes: the reset still happens once per step The program crashes when it divides by √v The knobs never move: m and v only ever see a gradient of 0
6. Predict: at which step does the smoothed curve (window 3, full windows only) first dip below 2.5? losses = [3.3, 2.1, 3.6, 2.4, 2.2, 2.6, 2.0] # step 1 is losses[0]
k = 3
below = [i + 1 for i in range(k - 1, len(losses)) if sum(losses[i - k + 1:i + 1]) / k < 2.5]
print(below[0]) 3 6 5 2
Answer all 6 to see your result Your answers are saved with a random id, not your name, so we can see which lessons work. Privacy