AI Doesn't Read Words. It Reads LEGO Bricks With Numbers
A token is a LEGO brick with a number on it, and almost none of the math that moves it is new.
By Geordie Everitt
You hear the word token every day now. AI is sold by the token, prices are quoted per million of them, and headlines keep count. Most people know a token is some kind of word, and that is about as far as it goes.
A token is a LEGO brick with a number on it. Here is what that means.
Why you keep hearing about them
AI companies charge by the token, the way a power company charges by the kilowatt-hour. In English, a token averages about four-fifths of a word, so a million tokens is roughly 800,000 words. I wrote about the money side in Tokens Are the New Electrons. This piece is about the thing being sold.
The brick
A language model does not read words. It reads tokens, and a token is a lot like a LEGO brick.
Picture an orange 2-by-4 brick with a number printed on it. That is the brick for the word "cat." In GPT-2, an open model anyone can download, its number is 3797, one of 50,257 bricks in the box. The number is only a label, like a seat number. It says nothing about whiskers.
The brick is just the letters c, a, t. It does not know whether it means a pet, a command, or slang. The same brick turns up in "My cat is asleep on the sofa," in "Type the cat command to show a file," and in "He is a cool cat." The machine tells them apart by the bricks next to it.
What happens to a number
The number is whole because it only says which brick this is. The decimals come next. When the machine picks up a brick, it looks up that brick's fingerprint, a list of 768 small decimal numbers, about half of them negative. Related bricks have similar fingerprints. Scored from 0 to 1, the fingerprint for "cat" is closest to other forms of cat, then to "dog" (0.55), "kitten" (0.50), and "pet" (0.45), and far from "car" (0.38), "the" (0.21), and "justice" (0.19).
Numbers between 0 and 1 show up at the very end. To choose the next brick, the machine gives every brick in the box a chance between 0 and 1, and the chances add up to 100 percent. After "The cat sat on the," its top picks are "floor," "bed," "couch," and "ground." "Mat," the word from the old reading-primer sentence, comes in 31st.
The math is old
Here is the surprise: almost none of the math inside a language model is new.
It starts with Isaac Newton. In the late 1600s he developed calculus, a way of measuring slopes: if you nudge one thing a hair, how much does something else change? A language model uses slopes to ask one question about each of its dials: if I turn this one a hair, does the model get less wrong or more?
The rest was added over the next three centuries. The machine multiplies lists of numbers, an old trick from the 1800s. It learns by walking downhill, nudging its dials a little each time, an idea from 1847. Text came into it early: in 1913 a Russian mathematician counted the vowels and consonants in a poem, and in 1948 Claude Shannon built English-looking text from the odds of one letter following another.
The newest piece is a shortcut called backpropagation, which is Newton's slope-finding done for every dial at once. A model has millions of dials, and the shortcut tells you which way to turn each of them. It was cheap by 1970, famous by 1986, and fast enough to be useful by 2012, when a network with 60 million dials learned to recognize images on two gaming graphics cards.
The GPT-2 I ran for this article has about 124 million dials, twice as many, and it runs on a laptop. The recipe was ready decades ago. What we were waiting for was machines fast enough and text plentiful enough.
A token, then, is a number standing in for a piece of text, and the machinery around it is mostly old math doing a very large amount of arithmetic. The frightening and thrilling things people say about AI are claims about what a machine does with those bricks, which is a separate question, and the next piece in this series takes it up.