Module 8 · Attention · train3.py · ~25 min

8.3 Scaled dot-product attention

The counting model saw one letter back. The embedding model saw one letter back. Today a letter gets to look at every letter before it and decide for itself which ones matter. This is the idea that made GPT possible.

1Watch

One head of attention, for the newest letter, takes three steps:

Score its query against every key so far, its own included (scaled dot product, 8.2).
Softmax the scores (6.4). Now they’re attention weights: positive, adding up to 100%.
Blend the values, each weighted by its attention. A letter with 60% attention contributes 60% of the result.

A worked blend

In the lab’s tiny example the weights come out 0.576, 0.14 and 0.284, and the three values are [10, 0], [0, 10] and [5, 5]. The blend is 0.576 × [10, 0] + 0.14 × [0, 10] + 0.284 × [5, 5], done one number at a time:

first number: 0.576 × 10 + 0.14 × 0 + 0.284 × 5 = 5.76 + 0 + 1.42 = 7.18
second number: 0.576 × 0 + 0.14 × 10 + 0.284 × 5 = 0 + 1.4 + 1.42 = 2.82

So the output is [7.18, 2.82]: mostly the first value, because its key matched best. Like mixing paint in those proportions.

What the blend is for

Here is the whole trip one letter makes through microgpt: embed (8.1) → normalise (8.5) → attention → add back (8.6) → MLP (9.2) → lm_head turns the vector into 27 scores, one per possible next letter (6.5). Attention’s job is to pack what the earlier letters say into this letter’s vector before that guess. Without it, the guess after “emm” could only see the last “m”.

Nothing about “which letters matter” is hand-coded: the query, key and value matrices are knobs, and training tunes them.

The heatmap shows the real trained microgpt reading a name, one head at a time (each head is its own search on its own 4 numbers; 9.1 shows all four side by side). Honest note: in a model this small (1 layer, see 9.3; 1,000 steps), the heads are fuzzy rather than neatly specialised. Each spreads its attention over recent letters, the start marker and itself. Bigger models grow much sharper patterns.

2Explore

Try it: with “kaylee” and head 2, look at the last row. Where does the final “e” look most? (Answer: the “l”, 43%.) Tap a square to read its percentage. Then type your own name.

Loading the trained model…

3Build

Two blanks in attend(q, keys, values):

Softmax the scores into weights: softmax(scores).
Blend: one sum per output position j, exactly like the worked example. The shape is [sum(… for w, v in zip(weights, values)) for j in range(len(values[0]))], and the … uses v[j]. (You can’t multiply a whole list by 0.576 in Python, so w * v on its own fails.)

import json, math

# The real microgpt after 1,000 training steps (same weights the widgets use).
M = json.load(open("microgpt-trained.json"))
W, UCHARS = M["weights"], M["uchars"]
BOS = len(UCHARS)

def linear(x, w):  # microgpt line 94
    return [sum(wi * xi for wi, xi in zip(row, x)) for row in w]

def softmax(z):  # lesson 6.4
    m = max(z)
    e = [math.exp(v - m) for v in z]
    t = sum(e)
    return [v / t for v in e]


# One head of attention, for the newest token:
#   1. score every key against my query (scaled dot product)
#   2. softmax the scores into attention weights (they add up to 1)
#   3. blend the values, each weighted by its attention
def attend(q, keys, values):
    scores = [sum(qi * ki for qi, ki in zip(q, k)) / math.sqrt(len(q)) for k in keys]
    weights = [1 / len(keys)] * len(keys)  # TODO: softmax the scores instead
    blended = values[-1]  # TODO: for each position j, sum weight * value[j] over all values
    return blended, weights


# A tiny hand-made example: the query looks for "vowel-ish" keys.
q = [1.0, 0.0]
keys = [[2.0, 0.0], [0.0, 2.0], [1.0, 1.0]]      # vowel, consonant, a bit of both
values = [[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]]
# try runs the lines under it. If one fails with a TypeError, Python jumps to
# the matching except block and prints a hint instead of stopping with an error.
try:
    out, w = attend(q, keys, values)
    print('attention weights:', [round(x, 3) for x in w])
    print('blended value:   ', [round(x, 3) for x in out])
except TypeError as err:
    print('TypeError:', err)
    print('Hint: Python cannot multiply a whole list by a number (0.576 * [10, 0] fails).')
    print('Blend one output position j at a time: one sum per j, using v[j].')
except IndexError as err:
    print('IndexError:', err)
    print('Hint: j counts the numbers inside ONE value, so use range(len(values[0])).')

4Check yourself

1. Predict: the weights are [0.5, 0.5] and the values are [2, 0] and [0, 4]. The blend is…

2. Predict: when there is only one key in the cache, the attention on it is…

3. Who decides which letters a head attends to?

5Unlocked in microgpt

Line 123 starts collecting the head outputs, and lines 129 to 131 are your attend(): scores, softmax, blended values. Lines 124 to 128 split the work across 4 heads, which is lesson 9.1.

Progress is saved in this browser. Sign in to keep it across devices.

microgpt.py105 / 175 lines learned

1"""

2The most atomic way to train and run inference for a GPT in pure, dependency-free Python.

3This file is the complete algorithm.

4Everything else is just efficiency.

6@karpathy

7"""

9import os # os.path.exists

10import math # math.log, math.exp

11import random # random.seed, random.choices, random.gauss, random.shuffle

12random.seed(42) # Let there be order among chaos

14# Let there be a Dataset `docs`: list[str] of documents (e.g. a list of names)

15if not os.path.exists('input.txt'):

16 import urllib.request

17 names_url = 'https://raw.githubusercontent.com/karpathy/makemore/988aa59/names.txt'

18 urllib.request.urlretrieve(names_url, 'input.txt')

19docs = [line.strip() for line in open('input.txt') if line.strip()]

20random.shuffle(docs)

21print(f"num docs: {len(docs)}")

23# Let there be a Tokenizer to translate strings to sequences of integers ("tokens") and back

24uchars = sorted(set(''.join(docs))) # unique characters in the dataset become token ids 0..n-1

25BOS = len(uchars) # token id for a special Beginning of Sequence (BOS) token

26vocab_size = len(uchars) + 1 # total number of unique tokens, +1 is for BOS

27print(f"vocab size: {vocab_size}")

29# Let there be Autograd to recursively apply the chain rule through a computation graph

30class Value:

31 __slots__ = ('data', 'grad', '_children', '_local_grads') # Python optimization for memory usage

33 def __init__(self, data, children=(), local_grads=()):

34 self.data = data # scalar value of this node calculated during forward pass

35 self.grad = 0 # derivative of the loss w.r.t. this node, calculated in backward pass

36 self._children = children # children of this node in the computation graph

37 self._local_grads = local_grads # local derivative of this node w.r.t. its children

39 def __add__(self, other):

40 other = other if isinstance(other, Value) else Value(other)

41 return Value(self.data + other.data, (self, other), (1, 1))

43 def __mul__(self, other):

44 other = other if isinstance(other, Value) else Value(other)

45 return Value(self.data * other.data, (self, other), (other.data, self.data))

47 def __pow__(self, other): return Value(self.data**other, (self,), (other * self.data**(other-1),))

48 def log(self): return Value(math.log(self.data), (self,), (1/self.data,))

49 def exp(self): return Value(math.exp(self.data), (self,), (math.exp(self.data),))

50 def relu(self): return Value(max(0, self.data), (self,), (float(self.data > 0),))

51 def __neg__(self): return self * -1

52 def __radd__(self, other): return self + other

53 def __sub__(self, other): return self + (-other)

54 def __rsub__(self, other): return other + (-self)

55 def __rmul__(self, other): return self * other

56 def __truediv__(self, other): return self * other**-1

57 def __rtruediv__(self, other): return other * self**-1

59 def backward(self):

60 topo = []

61 visited = set()

62 def build_topo(v):

63 if v not in visited:

64 visited.add(v)

65 for child in v._children:

66 build_topo(child)

67 topo.append(v)

68 build_topo(self)

69 self.grad = 1

70 for v in reversed(topo):

71 for child, local_grad in zip(v._children, v._local_grads):

72 child.grad += local_grad * v.grad

74# Initialize the parameters, to store the knowledge of the model

75n_layer = 1 # depth of the transformer neural network (number of layers)

76n_embd = 16 # width of the network (embedding dimension)

77block_size = 16 # maximum context length of the attention window (note: the longest name is 15 characters)

78n_head = 4 # number of attention heads

79head_dim = n_embd // n_head # derived dimension of each head

80matrix = lambda nout, nin, std=0.08: [[Value(random.gauss(0, std)) for _ in range(nin)] for _ in range(nout)]

81state_dict = {'wte': matrix(vocab_size, n_embd), 'wpe': matrix(block_size, n_embd), 'lm_head': matrix(vocab_size, n_embd)}

82for i in range(n_layer):

83 state_dict[f'layer{i}.attn_wq'] = matrix(n_embd, n_embd)

84 state_dict[f'layer{i}.attn_wk'] = matrix(n_embd, n_embd)

85 state_dict[f'layer{i}.attn_wv'] = matrix(n_embd, n_embd)

86 state_dict[f'layer{i}.attn_wo'] = matrix(n_embd, n_embd)

87 state_dict[f'layer{i}.mlp_fc1'] = matrix(4 * n_embd, n_embd)

88 state_dict[f'layer{i}.mlp_fc2'] = matrix(n_embd, 4 * n_embd)

89params = [p for mat in state_dict.values() for row in mat for p in row] # flatten params into a single list[Value]

90print(f"num params: {len(params)}")

92# Define the model architecture: a function mapping tokens and parameters to logits over what comes next

93# Follow GPT-2, blessed among the GPTs, with minor differences: layernorm -> rmsnorm, no biases, GeLU -> ReLU

94def linear(x, w):

95 return [sum(wi * xi for wi, xi in zip(wo, x)) for wo in w]

97def softmax(logits):

98 max_val = max(val.data for val in logits)

99 exps = [(val - max_val).exp() for val in logits]

100 total = sum(exps)

101 return [e / total for e in exps]

102

103def rmsnorm(x):

104 ms = sum(xi * xi for xi in x) / len(x)

105 scale = (ms + 1e-5) ** -0.5

106 return [xi * scale for xi in x]

107

108def gpt(token_id, pos_id, keys, values):

109 tok_emb = state_dict['wte'][token_id] # token embedding

110 pos_emb = state_dict['wpe'][pos_id] # position embedding

111 x = [t + p for t, p in zip(tok_emb, pos_emb)] # joint token and position embedding

112 x = rmsnorm(x) # note: not redundant due to backward pass via the residual connection

113

114 for li in range(n_layer):

115 # 1) Multi-head Attention block

116 x_residual = x

117 x = rmsnorm(x)

118 q = linear(x, state_dict[f'layer{li}.attn_wq'])

119 k = linear(x, state_dict[f'layer{li}.attn_wk'])

120 v = linear(x, state_dict[f'layer{li}.attn_wv'])

121 keys[li].append(k)

122 values[li].append(v)

123 x_attn = []

124 for h in range(n_head):

125 hs = h * head_dim

126 q_h = q[hs:hs+head_dim]

127 k_h = [ki[hs:hs+head_dim] for ki in keys[li]]

128 v_h = [vi[hs:hs+head_dim] for vi in values[li]]

129 attn_logits = [sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim**0.5 for t in range(len(k_h))]

130 attn_weights = softmax(attn_logits)

131 head_out = [sum(attn_weights[t] * v_h[t][j] for t in range(len(v_h))) for j in range(head_dim)]

132 x_attn.extend(head_out)

133 x = linear(x_attn, state_dict[f'layer{li}.attn_wo'])

134 x = [a + b for a, b in zip(x, x_residual)]

135 # 2) MLP block

136 x_residual = x

137 x = rmsnorm(x)

138 x = linear(x, state_dict[f'layer{li}.mlp_fc1'])

139 x = [xi.relu() for xi in x]

140 x = linear(x, state_dict[f'layer{li}.mlp_fc2'])

141 x = [a + b for a, b in zip(x, x_residual)]

142

143 logits = linear(x, state_dict['lm_head'])

144 return logits

145

146# Let there be Adam, the blessed optimizer and its buffers

147learning_rate, beta1, beta2, eps_adam = 0.01, 0.85, 0.99, 1e-8

148m = [0.0] * len(params) # first moment buffer

149v = [0.0] * len(params) # second moment buffer

150

151# Repeat in sequence

152num_steps = 1000 # number of training steps

153for step in range(num_steps):

154

155 # Take single document, tokenize it, surround it with BOS special token on both sides

156 doc = docs[step % len(docs)]

157 tokens = [BOS] + [uchars.index(ch) for ch in doc] + [BOS]

158 n = min(block_size, len(tokens) - 1)

159

160 # Forward the token sequence through the model, building up the computation graph all the way to the loss

161 keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]

162 losses = []

163 for pos_id in range(n):

164 token_id, target_id = tokens[pos_id], tokens[pos_id + 1]

165 logits = gpt(token_id, pos_id, keys, values)

166 probs = softmax(logits)

167 loss_t = -probs[target_id].log()

168 losses.append(loss_t)

169 loss = (1 / n) * sum(losses) # final average loss over the document sequence. May yours be low.

170

171 # Backward the loss, calculating the gradients with respect to all model parameters

172 loss.backward()

173

174 # Adam optimizer update: update the model parameters based on the corresponding gradients

175 lr_t = learning_rate * (1 - step / num_steps) # linear learning rate decay

176 for i, p in enumerate(params):

177 m[i] = beta1 * m[i] + (1 - beta1) * p.grad

178 v[i] = beta2 * v[i] + (1 - beta2) * p.grad ** 2

179 m_hat = m[i] / (1 - beta1 ** (step + 1))

180 v_hat = v[i] / (1 - beta2 ** (step + 1))

181 p.data -= lr_t * m_hat / (v_hat ** 0.5 + eps_adam)

182 p.grad = 0

183

184 print(f"step {step+1:4d} / {num_steps:4d} | loss {loss.data:.4f}", end='\r')

185

186# Inference: may the model babble back to us

187temperature = 0.5 # in (0, 1], control the "creativity" of generated text, low to high

188print("\n--- inference (new, hallucinated names) ---")

189for sample_idx in range(20):

190 keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]

191 token_id = BOS

192 sample = []

193 for pos_id in range(block_size):

194 logits = gpt(token_id, pos_id, keys, values)

195 probs = softmax([l / temperature for l in logits])

196 token_id = random.choices(range(vocab_size), weights=[p.data for p in probs])[0]

197 if token_id == BOS:

198 break

199 sample.append(uchars[token_id])

200 print(f"sample {sample_idx+1:2d}: {''.join(sample)}")

this lessonlearnedKarpathy’s original

Was this lesson clear?

Next available lesson → 8.4 Causal masking & the KV cache