How AI Understands Meaning

The Analogy

Think of a starting embedding as an initial location in a city of meaning. The model then updates that representation using the surrounding context.

Use the arrows below, the dots above, or your keyboard arrow keys to move through the stages.

Stage 1 -- Token ID

First, the Word Becomes a Token ID

cat

Token cat

Example token ID 9827

A token ID is a lookup number created by a tokenizer. Here, 9827 is an illustrative value; another tokenizer may assign a different ID.

Stage 2 -- Vectorization

Then, the Token ID Selects a Starting Embedding

Example token ID 9827

Embedding table row 9827

The token ID selects one row from the model's embedding table. That list of numbers is the token's starting embedding, before context updates it.

Stage 3 -- Visualize the Embedding

Visualize the Starting Embedding

Simplified example: the numbers place a token in a meaning space before the model uses the surrounding context.

Similar starting embeddings can appear near one another. This 2D view uses illustrative values; real embeddings have many more dimensions.

Stage 4 -- The Neighborhood

Starting Embeddings Form Neighborhoods

Click a word to inspect it

Sample embedding preview

This shows 6 sample dimensions out of a much larger embedding.

Nearest neighbors

Click any word to lock the lines, see distances, and inspect a clear sample embedding. Click the background to reset.

In this simplified projection, 'cat' and 'dog' are closer than 'cat' and 'airplane'. Real relationships are more complex.

Stage 5 -- The Bias Reveal

Watch the Pattern Form

Stage 6 -- Context Changes Everything

Same Token, Different Contextual Representation

The token starts with the same embedding, then context changes its internal representation. That helps the model distinguish fruit from a company name.