6 questions on what this module teaches. Guessing is fine: this is your starting point, not a test. Take the same check again at the end to see what you learned.
1. Predict: two heads of 2 numbers each, reading a cache of two values. What does this print? def blend(weights, values):
return [sum(w * v[j] for w, v in zip(weights, values)) for j in range(len(values[0]))]
values = [[1, 2, 3, 4], [5, 6, 7, 8]]
out = []
for h, weights in [(0, [1, 0]), (1, [0, 1])]:
v_h = [v[h * 2:h * 2 + 2] for v in values]
out.extend(blend(weights, v_h))
print(out) [[1, 2], [7, 8]] [1, 2, 5, 6] [1, 2, 3, 4] [1, 2, 7, 8]
2. Predict: a tiny MLP that widens 2 numbers to 4, applies ReLU, then squeezes back to 2. What does it print? def linear(x, w):
return [sum(wi * xi for wi, xi in zip(row, x)) for row in w]
fc1 = [[1, 1], [1, -1], [-1, 1], [-1, -1]]
fc2 = [[1, 1, 1, 1], [1, -1, 0, 0]]
x = [3, 1]
hidden = [max(0, v) for v in linear(x, fc1)]
print(linear(hidden, fc2)) [6, 2] [4, 2, 0, 0] [12, 2] [0, 2]
3. You change microgpt to n_head = 8, keeping n_embd = 16. What happens inside attention? Each head works on 16 numbers, and scores are divided by 4 Each head works on 2 numbers, and scores are divided by √16 Each head works on 8 numbers, and scores are divided by √8 Each head works on 2 numbers, and scores are divided by √2
4. microgpt has 4,192 knobs with n_layer = 1. Each layer holds 4 attention tables of 16 × 16 plus mlp_fc1 (64 × 16) and mlp_fc2 (16 × 64). How many knobs with n_layer = 2? 4,192 7,264 6,144 8,384
5. Which order does gpt() run for one token? wte + wpe → rmsnorm → [rmsnorm → MLP → add residual] → [rmsnorm → attention → add residual] → lm_head wte + wpe → attention → MLP → lm_head → rmsnorm wte + wpe → rmsnorm → [rmsnorm → attention → add residual] → [rmsnorm → MLP → add residual] → lm_head wte + wpe → rmsnorm → attention → MLP → add one residual → lm_head
6. Predict: a stand-in block inside a 2-layer loop with residuals. What does it print? def block(x):
return [max(0, v - 1) for v in x]
x = [0.5, 3.0]
for li in range(2):
x = [a + b for a, b in zip(block(x), x)]
print(x) [0.5, 3.0, 0, 2.0] [0.0, 1.0] [0.5, 9.0] [0.5, 5.0]
Answer all 6 to see your result Your answers are saved with a random id, not your name, so we can see which lessons work. Privacy