Welcome to the Tokenizer
AI models like ChatGPT don't read English directly. They read tokens. Tokens are small chunks of text (letters, syllables, or whole words). Let's see how a model physically breaks sentences apart.
Sandbox: Break Your Own Words
Type a short sentence below. We'll run a simulated tokenizer to show how AI might chop it up into smaller chunks.
Notice: This is a simplified simulation. Real AI tokenizers use massive dictionaries to decide these splits.
Surprising Splits
Long words aren't always broken down logically. Click each word to see how a real AI tokenizer might snap it apart based on statistical frequency.
Notice how the capital 'S' in 'Summarize' changes how the word is split!
Context Matters
Select an example to see how different data types (like code or emojis) are handled.
Language Inequity
Because tokenizers are trained mostly on English text, non-English languages are often chopped into far more pieces. Hover over the bars to see inside.
More tokens mean slower processing and higher costs for non-English users.