Module 5 · before

Language as probability

6 questions on what this module teaches. Guessing is fine: this is your starting point, not a test. Take the same check again at the end to see what you learned.

1. What does this print?

counts = {}
for name in ['ann', 'anna']:
    chars = '.' + name + '.'
    for a, b in zip(chars, chars[1:]):
        counts[(a, b)] = counts.get((a, b), 0) + 1
print(counts[('n', 'n')], counts[('a', 'n')], len(counts))

2. What does this print?

counts = [3, 1]
T = 0.5
weights = [c ** (1 / T) for c in counts]
total = sum(weights)
print(round(weights[0] / total, 2))

3. What does this print?

import math

def surprise(p):
    return -math.log(p)

all_s = []
for chances in [[0.5, 0.5], [0.25]]:
    all_s.extend([surprise(p) for p in chances])
print(len(all_s), round(sum(all_s) / len(all_s), 2))

4. A tiny language uses only 3 letters plus the start/end marker. What does the know-nothing model score?

import math
VOCAB = 3 + 1
print(round(-math.log(1 / VOCAB), 2))

5. The row for 'q' has a total of 272, and the pair ('q', 'x') was never seen. With +1 smoothing over 27 symbols, what chance does the model give ('q', 'x')?

6. Two models guess the next letter. Model A gives the right letter a chance of 0.5 every time. Model B gives it 0.9 half the time and 0.1 the other half. Which has the lower loss?

Your answers are saved with a random id, not your name, so we can see which lessons work. Privacy