Module 8 · after

Attention

The same 6 questions you saw before the module. No running the code: predict, then compare with your first score.

1. Predict: one head’s match score between a query and a key. What does this print?

import math

q = [1, 2, 0, 1]
k = [2, 1, 1, 2]
score = sum(a * b for a, b in zip(q, k)) / math.sqrt(len(q))
print(score)

2. Predict: the query only cares about the first number. What blend does this print?

import math

def softmax(z):
    e = [math.exp(v - max(z)) for v in z]
    return [x / sum(e) for x in e]

q = [1.0, 0.0]
keys = [[3.0, 5.0], [3.0, -5.0]]
values = [[4.0, 0.0], [0.0, 8.0]]
w = softmax([sum(a * b for a, b in zip(q, k)) / math.sqrt(2) for k in keys])
print([sum(wt * v[j] for wt, v in zip(w, values)) for j in range(2)])

3. Predict: what does this print?

def rmsnorm(x):
    ms = sum(xi * xi for xi in x) / len(x)
    scale = (ms + 1e-5) ** -0.5
    return [xi * scale for xi in x]

print([round(v, 3) for v in rmsnorm([0.0, 3.0, 0.0, -3.0])])

4. microgpt is generating a name. It has produced “mar” after the start marker and is now working out the next letter. How many times has it computed the key for the “m” so far?

5. A stack has 3 layers, and each layer’s own rate is −0.5. Every layer is wrapped in a residual connection (x = x + layer(x)). What rate reaches the input?

6. microgpt reads “abba” (start marker at position 0). What can you say about the vectors x for the two b’s, at positions 2 and 3, right after line 111?

Your answers are saved with a random id, not your name, so we can see which lessons work. Privacy