Module 10 · after

Training like a pro

The same 6 questions you saw before the module. No running the code: predict, then compare with your first score.

1. Predict: Adam (β₁ 0.85, β₂ 0.99, lr 0.1) sees a gradient of 4, then a gradient of 0. How big is the second step?

m, v = 0.0, 0.0
for t, g in [(1, 4.0), (2, 0.0)]:
    m = 0.85 * m + 0.15 * g
    v = 0.99 * v + 0.01 * g ** 2
    m_hat = m / (1 - 0.85 ** t)
    v_hat = v / (1 - 0.99 ** t)
    step = 0.1 * m_hat / (v_hat ** 0.5 + 1e-8)
print(round(step, 3))

2. After many training steps, knob A’s gradient is steadily about 50 and knob B’s is steadily about 0.05. With Adam, which one takes the bigger steps?

3. Predict: with microgpt’s schedule, how many of its 1,000 steps use a learning rate above 0.005?

lr, num_steps = 0.01, 1000
fast = [step for step in range(num_steps) if lr * (1 - step / num_steps) > 0.005]
print(len(fast))

4. Your training run’s moving average (window 100) sits at about 3.30 from start to finish, while single steps bounce between 3.2 and 3.4. What does it tell you?

5. A friend moves p.grad = 0 to the top of the Adam loop, before the m[i] line, instead of after the update. What happens?

6. Predict: at which step does the smoothed curve (window 3, full windows only) first dip below 2.5?

losses = [3.3, 2.1, 3.6, 2.4, 2.2, 2.6, 2.0]   # step 1 is losses[0]
k = 3
below = [i + 1 for i in range(k - 1, len(losses)) if sum(losses[i - k + 1:i + 1]) / k < 2.5]
print(below[0])

Your answers are saved with a random id, not your name, so we can see which lessons work. Privacy