Module 6 · after

Learning = walking downhill

The same 6 questions you saw before the module. No running the code: predict, then compare with your first score.

1. Predict: what does this print?

def f(x):
    return x * x * x

h = 0.001
print(round((f(2 + h) - f(2 - h)) / (2 * h), 2))

2. Predict: the valley (x − 3)² has its bottom at x = 3. Starting at 0 with a learning rate of 1.0, where is the knob after 3 steps?

def slope(x):
    return 2 * (x - 3)   # the slope of (x - 3) ** 2

x = 0.0
for step in range(3):
    x = x - 1.0 * slope(x)
print(x)

3. Gradient descent stops moving a knob once it reaches the bottom of the valley, even though nobody tells it to stop. Why?

4. Predict: what does this print?

import math

scores = [0.0, math.log(3)]
exps = [math.exp(s) for s in scores]
total = sum(exps)
print([round(e / total, 2) for e in exps])

5. A model has 100 knobs, and you measure its gradient with two-sided nudges, (f(x + h) − f(x − h)) / 2h, for each knob. How many loss evaluations does one step cost?

6. A friend’s embedding model gives each of the 27 symbols 8 numbers in wte, and scores the 27 possible next symbols with lm_head. How many knobs does it have?

Your answers are saved with a random id, not your name, so we can see which lessons work. Privacy