AI mechanics

Token

The chunk of text a model actually reads, instead of a word.

Text is cut into tokens before a model sees it. Sometimes a whole word, sometimes part of one, sometimes a single punctuation mark. Each token becomes a number, and the numbers are all the machine ever handles.

Every limit you hit and every price you pay is counted in tokens.

Also called: tokens, tokenizer, tokenisation.

Badge not earned yet

Finish the AI 101 card that teaches it and the badge is yours. Reading this page earns nothing.

Learn this in AI 101

0.1A token is the unit it actually sees