Module 11 · Capstone · ~180 min

11.1 microgpt from a blank file

This is it: an empty file and nine section headings. You have built every piece that goes under them, lesson by lesson; a few fixed choices that tie them together are listed below. Write it once more, start to finish, and your microgpt will match Karpathy’s training run number for number.

1Watch

Allow 2 to 4 hours, over several sittings. Every earlier lab had one or two blanks; this one is about 150 lines. Your code saves automatically, so stop whenever you like.

The starter file keeps only Karpathy’s header and his section comments. Fill each section in order. If you get stuck, the lesson that taught it is listed below. Re-open it rather than copying: the full program is in the lesson 0.1 and 11.2 labs and in the sidebar, and you could copy it. Nothing stops you. But the point of this lesson is to find that you can write it yourself.

Conventions: choices your file must make to match Karpathy exactly

Tables, in this orderwte (27×16), wpe (16×16), lm_head (27×16), then for each layer attn_wq, attn_wk, attn_wv, attn_wo (16×16 each), mlp_fc1 (64×16), mlp_fc2 (16×64). Every knob is random.gauss(0, 0.08), drawn row by row; params lists them in the same order.
Which name each step usesdoc = docs[step % len(docs)], not random.choice: the names were already shuffled once.
TokensBOS on both ends: [BOS] + letters + [BOS]. Then n = min(block_size, len(tokens) - 1) positions.
softmax on ValuesLesson 6.4’s softmax used plain numbers. Here the scores are Values: subtract the max using .data (max(v.data for v in logits)), then call .exp() on each Value.
Residuals on listsx = [a + b for a, b in zip(x, x_residual)]; x + x_residual would glue the lists end to end.
Adamlearning_rate 0.01 (the 10.1 lab used 0.1), beta1 0.85, beta2 0.99, eps 1e-8, bias correction with step + 1.
Learning-rate decaylr_t = learning_rate * (1 - step / num_steps). It depends on num_steps, so a 20-step run is not the first 20 steps of a 1,000-step run.
After the updatep.grad = 0 for every knob. Gradients add up (+=), so without the reset each step would also carry the old ones.
Python you may wanta if cond else b picks a or b in one expression (Value wraps plain numbers with it). [p for mat in tables for row in mat for p in row] reads its fors left to right, like nested loops: it flattens the tables into one list. break leaves a loop at once (stop sampling at BOS). .index and enumerate are from 8.4.
Two lines for the checksparams_at_start = [p.data for p in params] right after building params, and step_losses.append(loss.data) at the end of every step (the starter already has step_losses = []).

Other harmless differences still match: sum(losses) / n or (1 / n) * sum(losses), head_dim ** 0.5 or math.sqrt(head_dim).

The check is strict on purpose. microgpt is deterministic: random.seed(42) fixes the shuffle, the starting knobs and the samples. So a correct microgpt with num_steps = 20 lands on Karpathy’s exact loss after step 20 (2.7749), to 9 decimal places. One wrong sign anywhere and the numbers drift.

The checks go in four stages, in the order of the file: the shuffled names, the starting knobs, the step-1 loss, then all 20 losses. Step 1 (3.3660) runs on the starting knobs, before any update. So if step 1 is wrong, look at the setup and the model; if step 1 matches and a later step differs, look at the training loop and Adam. Press Check after each section to see how far you have got.

When it passes, set num_steps = 1000 and press Run for the real thing. It takes a few minutes in the browser. Then read the names it invents: every line that made them is yours.

2Explore

Sections, and where you learned them

Datasetlessons 1.5, 3.4
Tokenizerlessons 2.2, 5.1
Autograd: Value and backward()lessons 3.3, 7.4, 7.5
Parameterslessons 2.3, then 9.1 to 9.3 for the layer tables
Model: linear, softmax, rmsnormlessons 4.5, 6.4, 8.5
gpt(): attention, MLP, layerslessons 8.2 to 9.3
Adamlessons 10.1
Training looplessons 7.5 (backward), 10.1 to 10.3
Inferencelessons 5.2, 6.4 (logits ÷ temperature)

3Build

Write microgpt under the section comments, following the conventions card. Keep num_steps = 20 while checking. Reading names.txt replaces Karpathy’s download (lines 15 to 18), since the file is already in your browser.

Milestones as you go: num docs: 32033, vocab size: 27, num params: 4192, then step 1 / 20 | loss 3.3660 and step 20 / 20 | loss 2.7749.

"""
YOUR microgpt, from a blank file. Every section below is one you have built
in an earlier lesson. Write them in order, following the conventions card on
the page. With num_steps = 20 the checks compare your run with Karpathy's own
20-step run in four stages: the shuffled names, the starting knobs, the step-1
loss, then every step's loss. Check after each section to see how far you got.

The most atomic way to train and run inference for a GPT in pure, dependency-free Python.
This file is the complete algorithm.
Everything else is just efficiency.

@karpathy
"""

import os       # (Karpathy uses os.path.exists to download names; not needed here)
import math     # math.log, math.exp
import random   # random.seed, random.choices, random.gauss, random.shuffle
random.seed(42) # Let there be order among chaos

# Let there be a Dataset `docs`: list[str] of documents (e.g. a list of names)
# TODO

# Let there be a Tokenizer to translate strings to sequences of integers ("tokens") and back
# TODO

# Let there be Autograd to recursively apply the chain rule through a computation graph
# TODO

# Initialize the parameters, to store the knowledge of the model
# TODO
# For the checks, end this section with: params_at_start = [p.data for p in params]

# Define the model architecture: a function mapping tokens and parameters to logits over what comes next
# TODO

# Follow GPT-2, blessed among the GPTs, with minor differences: layernorm -> rmsnorm, no biases, GeLU -> ReLU
# TODO

# Let there be Adam, the blessed optimizer and its buffers
# TODO

# Repeat in sequence
num_steps = 20 # 20 while checking; set 1000 for the real run
step_losses = [] # for the checks: end every step with step_losses.append(loss.data)
# TODO

# Inference: may the model babble back to us
# TODO

4Check yourself

1. Why can the check demand an exact match with Karpathy’s loss?

2. Predict: your step-1 loss is 3.3660, but step 3 differs from Karpathy’s. Where is the bug most likely?

3. You create lm_head after the layer tables instead of third. What happens?

5Unlocked in microgpt

The header (lines 1 to 8) and the rest of the inference section (lines 186, 188 to 194 and 197 to 200): start at BOS, run gpt(), divide by the temperature, roll, stop at BOS. With these, every line on the map is lit.

Progress is saved in this browser. Sign in to keep it across devices.

microgpt.py175 / 175 lines learned

The code is hidden so you can write it yourself. Try first; peek only when you’re stuck.

1"""

2The most atomic way to train and run inference for a GPT in pure, dependency-free Python.

3This file is the complete algorithm.

4Everything else is just efficiency.

6@karpathy

7"""

9import os # os.path.exists

10import math # math.log, math.exp

11import random # random.seed, random.choices, random.gauss, random.shuffle

12random.seed(42) # Let there be order among chaos

14# Let there be a Dataset `docs`: list[str] of documents (e.g. a list of names)

15if not os.path.exists('input.txt'):

16 import urllib.request

17 names_url = 'https://raw.githubusercontent.com/karpathy/makemore/988aa59/names.txt'

18 urllib.request.urlretrieve(names_url, 'input.txt')

19docs = [line.strip() for line in open('input.txt') if line.strip()]

20random.shuffle(docs)

21print(f"num docs: {len(docs)}")

23# Let there be a Tokenizer to translate strings to sequences of integers ("tokens") and back

24uchars = sorted(set(''.join(docs))) # unique characters in the dataset become token ids 0..n-1

25BOS = len(uchars) # token id for a special Beginning of Sequence (BOS) token

26vocab_size = len(uchars) + 1 # total number of unique tokens, +1 is for BOS

27print(f"vocab size: {vocab_size}")

29# Let there be Autograd to recursively apply the chain rule through a computation graph

30class Value:

31 __slots__ = ('data', 'grad', '_children', '_local_grads') # Python optimization for memory usage

33 def __init__(self, data, children=(), local_grads=()):

34 self.data = data # scalar value of this node calculated during forward pass

35 self.grad = 0 # derivative of the loss w.r.t. this node, calculated in backward pass

36 self._children = children # children of this node in the computation graph

37 self._local_grads = local_grads # local derivative of this node w.r.t. its children

39 def __add__(self, other):

40 other = other if isinstance(other, Value) else Value(other)

41 return Value(self.data + other.data, (self, other), (1, 1))

43 def __mul__(self, other):

44 other = other if isinstance(other, Value) else Value(other)

45 return Value(self.data * other.data, (self, other), (other.data, self.data))

47 def __pow__(self, other): return Value(self.data**other, (self,), (other * self.data**(other-1),))

48 def log(self): return Value(math.log(self.data), (self,), (1/self.data,))

49 def exp(self): return Value(math.exp(self.data), (self,), (math.exp(self.data),))

50 def relu(self): return Value(max(0, self.data), (self,), (float(self.data > 0),))

51 def __neg__(self): return self * -1

52 def __radd__(self, other): return self + other

53 def __sub__(self, other): return self + (-other)

54 def __rsub__(self, other): return other + (-self)

55 def __rmul__(self, other): return self * other

56 def __truediv__(self, other): return self * other**-1

57 def __rtruediv__(self, other): return other * self**-1

59 def backward(self):

60 topo = []

61 visited = set()

62 def build_topo(v):

63 if v not in visited:

64 visited.add(v)

65 for child in v._children:

66 build_topo(child)

67 topo.append(v)

68 build_topo(self)

69 self.grad = 1

70 for v in reversed(topo):

71 for child, local_grad in zip(v._children, v._local_grads):

72 child.grad += local_grad * v.grad

74# Initialize the parameters, to store the knowledge of the model

75n_layer = 1 # depth of the transformer neural network (number of layers)

76n_embd = 16 # width of the network (embedding dimension)

77block_size = 16 # maximum context length of the attention window (note: the longest name is 15 characters)

78n_head = 4 # number of attention heads

79head_dim = n_embd // n_head # derived dimension of each head

80matrix = lambda nout, nin, std=0.08: [[Value(random.gauss(0, std)) for _ in range(nin)] for _ in range(nout)]

81state_dict = {'wte': matrix(vocab_size, n_embd), 'wpe': matrix(block_size, n_embd), 'lm_head': matrix(vocab_size, n_embd)}

82for i in range(n_layer):

83 state_dict[f'layer{i}.attn_wq'] = matrix(n_embd, n_embd)

84 state_dict[f'layer{i}.attn_wk'] = matrix(n_embd, n_embd)

85 state_dict[f'layer{i}.attn_wv'] = matrix(n_embd, n_embd)

86 state_dict[f'layer{i}.attn_wo'] = matrix(n_embd, n_embd)

87 state_dict[f'layer{i}.mlp_fc1'] = matrix(4 * n_embd, n_embd)

88 state_dict[f'layer{i}.mlp_fc2'] = matrix(n_embd, 4 * n_embd)

89params = [p for mat in state_dict.values() for row in mat for p in row] # flatten params into a single list[Value]

90print(f"num params: {len(params)}")

92# Define the model architecture: a function mapping tokens and parameters to logits over what comes next

93# Follow GPT-2, blessed among the GPTs, with minor differences: layernorm -> rmsnorm, no biases, GeLU -> ReLU

94def linear(x, w):

95 return [sum(wi * xi for wi, xi in zip(wo, x)) for wo in w]

97def softmax(logits):

98 max_val = max(val.data for val in logits)

99 exps = [(val - max_val).exp() for val in logits]

100 total = sum(exps)

101 return [e / total for e in exps]

102

103def rmsnorm(x):

104 ms = sum(xi * xi for xi in x) / len(x)

105 scale = (ms + 1e-5) ** -0.5

106 return [xi * scale for xi in x]

107

108def gpt(token_id, pos_id, keys, values):

109 tok_emb = state_dict['wte'][token_id] # token embedding

110 pos_emb = state_dict['wpe'][pos_id] # position embedding

111 x = [t + p for t, p in zip(tok_emb, pos_emb)] # joint token and position embedding

112 x = rmsnorm(x) # note: not redundant due to backward pass via the residual connection

113

114 for li in range(n_layer):

115 # 1) Multi-head Attention block

116 x_residual = x

117 x = rmsnorm(x)

118 q = linear(x, state_dict[f'layer{li}.attn_wq'])

119 k = linear(x, state_dict[f'layer{li}.attn_wk'])

120 v = linear(x, state_dict[f'layer{li}.attn_wv'])

121 keys[li].append(k)

122 values[li].append(v)

123 x_attn = []

124 for h in range(n_head):

125 hs = h * head_dim

126 q_h = q[hs:hs+head_dim]

127 k_h = [ki[hs:hs+head_dim] for ki in keys[li]]

128 v_h = [vi[hs:hs+head_dim] for vi in values[li]]

129 attn_logits = [sum(q_h[j] * k_h[t][j] for j in range(head_dim)) / head_dim**0.5 for t in range(len(k_h))]

130 attn_weights = softmax(attn_logits)

131 head_out = [sum(attn_weights[t] * v_h[t][j] for t in range(len(v_h))) for j in range(head_dim)]

132 x_attn.extend(head_out)

133 x = linear(x_attn, state_dict[f'layer{li}.attn_wo'])

134 x = [a + b for a, b in zip(x, x_residual)]

135 # 2) MLP block

136 x_residual = x

137 x = rmsnorm(x)

138 x = linear(x, state_dict[f'layer{li}.mlp_fc1'])

139 x = [xi.relu() for xi in x]

140 x = linear(x, state_dict[f'layer{li}.mlp_fc2'])

141 x = [a + b for a, b in zip(x, x_residual)]

142

143 logits = linear(x, state_dict['lm_head'])

144 return logits

145

146# Let there be Adam, the blessed optimizer and its buffers

147learning_rate, beta1, beta2, eps_adam = 0.01, 0.85, 0.99, 1e-8

148m = [0.0] * len(params) # first moment buffer

149v = [0.0] * len(params) # second moment buffer

150

151# Repeat in sequence

152num_steps = 1000 # number of training steps

153for step in range(num_steps):

154

155 # Take single document, tokenize it, surround it with BOS special token on both sides

156 doc = docs[step % len(docs)]

157 tokens = [BOS] + [uchars.index(ch) for ch in doc] + [BOS]

158 n = min(block_size, len(tokens) - 1)

159

160 # Forward the token sequence through the model, building up the computation graph all the way to the loss

161 keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]

162 losses = []

163 for pos_id in range(n):

164 token_id, target_id = tokens[pos_id], tokens[pos_id + 1]

165 logits = gpt(token_id, pos_id, keys, values)

166 probs = softmax(logits)

167 loss_t = -probs[target_id].log()

168 losses.append(loss_t)

169 loss = (1 / n) * sum(losses) # final average loss over the document sequence. May yours be low.

170

171 # Backward the loss, calculating the gradients with respect to all model parameters

172 loss.backward()

173

174 # Adam optimizer update: update the model parameters based on the corresponding gradients

175 lr_t = learning_rate * (1 - step / num_steps) # linear learning rate decay

176 for i, p in enumerate(params):

177 m[i] = beta1 * m[i] + (1 - beta1) * p.grad

178 v[i] = beta2 * v[i] + (1 - beta2) * p.grad ** 2

179 m_hat = m[i] / (1 - beta1 ** (step + 1))

180 v_hat = v[i] / (1 - beta2 ** (step + 1))

181 p.data -= lr_t * m_hat / (v_hat ** 0.5 + eps_adam)

182 p.grad = 0

183

184 print(f"step {step+1:4d} / {num_steps:4d} | loss {loss.data:.4f}", end='\r')

185

186# Inference: may the model babble back to us

187temperature = 0.5 # in (0, 1], control the "creativity" of generated text, low to high

188print("\n--- inference (new, hallucinated names) ---")

189for sample_idx in range(20):

190 keys, values = [[] for _ in range(n_layer)], [[] for _ in range(n_layer)]

191 token_id = BOS

192 sample = []

193 for pos_id in range(block_size):

194 logits = gpt(token_id, pos_id, keys, values)

195 probs = softmax([l / temperature for l in logits])

196 token_id = random.choices(range(vocab_size), weights=[p.data for p in probs])[0]

197 if token_id == BOS:

198 break

199 sample.append(uchars[token_id])

200 print(f"sample {sample_idx+1:2d}: {''.join(sample)}")

this lessonlearnedKarpathy’s original

Was this lesson clear?

Next available lesson → 11.2 Your own dataset