How AI Reads Your Words

Before AI can understand your words, it breaks them into pieces -- like cracking a chocolate bar along its molded lines. But the pieces aren't always whole words, and some breaks will surprise you.

How AI Reads Your Words

Before AI can understand your words, it breaks them into pieces -- like cracking a chocolate bar along its molded lines. But the pieces aren't always whole words, and some breaks will surprise you.

The Split

Write a lesson plan for 8th grade algebra covering linear equations

Surprising Splits

Click each word to see how AI breaks it apart.

understanding
Summarize
unbelievable

Example Selector

Multilingual Comparison

What is artificial intelligence?
5
¿Qué es la inteligencia artificial?
10
人工知能とは何ですか?
8

How Tokenizers Are Built

Takeaway

Shorter, simpler prompts cost fewer tokens. The language you write in changes your token count. And that emoji? It might cost more than the word it replaced.

AI has broken your text into tokens. But tokens are just labels. How does the model know what they mean? →
c:\Users\bhava\OneDrive\Desktop\ai-simulations\ai-simulations\tokenization.html