Issue library

comicMemori / Issue 02 of 37

Tokens, the sticky notes of text

In a small office, an intern named Lex sits at a desk reading sticky notes. Each note holds a piece of text: sometimes a full word, sometimes just part of one.

Fernando Torres4 min

In a small office, an intern named Lex sits at a desk reading sticky notes. Each note holds a piece of text: sometimes a full word, sometimes just part of one. This is how an AI reads your messages.

Lex stands for the LLM, an AI that reads assembled input and writes a response. His desk stands for the context window, the bounded workspace assembled for one model response.

Ava walks in and sees the desk covered in colorful sticky notes spread out in rows. The desk meter is healthy. Lex is processing them in order. Emotional beat: curiosity.
Panel 1 transcript: Ava: "I sent you one paragraph. Why are there so many sticky notes on the desk?"
Close-up of the same short sentence shown in two rows. A card labeled "Tokenizer A" divides it into five colored groups. A card labeled "Tokenizer B" divides it into seven. A small label reads "illustrative boundaries." Emotional beat: discovery.
Panel 2 transcript: Lex: "A tokenizer turns text into pieces. Different tokenizers can cut the same sentence differently."
Ava holds two count cards side by side: "Tokenizer A: 5" and "Tokenizer B: 7." The original sentence above them is identical. Emotional beat: understanding dawning.
Panel 3 transcript: Ava: "So word count cannot tell me the token count?" Lex: "Right. The tokenizer for this model decides the boundaries."
Lex arranges the notes into a neat reading order on the desk. He reads them left to right, top to bottom. The desk has a moderate spread of notes but is not crowded. Emotional beat: things clicking into place.
Panel 4 transcript: Lex: "The ordered tokens become the input I process."
Ava looks at the spread of notes, then at the desk edges. She starts counting notes with her finger. Emotional beat: a new question forming.
Panel 5 transcript: Ava: "That is a lot of notes for one paragraph. How many can the desk hold?"

Problem: Ava's word count does not tell her how much model context the paragraph uses.

Resolution: The reader sees that language models process tokenizer-defined tokens. The exact pieces and count depend on the tokenizer used by the model.

Module A variant: Desk close-up. Show a top-down view of the desk with one short sentence at the top and two rows of colored token boundaries below it.

A language model processes tokens rather than a human word count, and the exact boundaries depend on its tokenizer.

Section 1: How text becomes sticky notes

When you type a message and hit send, a language model does not receive a human word count. A tokenizer converts the text into a sequence of pieces represented by numbers. In the office analogy, each piece becomes a sticky note on Lex's desk.

This step usually happens behind the scenes before the model processes the input. Text models operate on tokens, but different model families can use different tokenizers. Multimodal systems also have their own ways of representing images, audio, and other inputs, so the sticky-note analogy here is specifically about text.

The process of breaking text into pieces is called tokenization. Think of it as the step between you typing a message and Lex reading it. You write a sentence. The sentence becomes a row of sticky notes. Lex reads the sticky notes. That is the pipeline.

Section 2: Not every word is one note

Here is the part that surprises people: words and tokens do not map one-to-one. A tokenizer may keep a familiar word together, split a word into smaller pieces, or bundle whitespace or punctuation with a neighboring piece. Another tokenizer may divide the same sentence differently.

Code, URLs, formatting, non-English text, and unusual character sequences can produce token counts that differ sharply from their visible length. That does not mean they always cost more than prose. It means visual length and word count are unreliable measures of token use.

This means there is no universal conversion such as "one word equals one token." If an exact count matters, use the tokenizer or token-counting tool for the specific model. An example from a different model may not match.

Section 3: Why this matters for the desk

Every sticky note takes up space on Lex's desk. The desk, the context window, has a fixed size. So the real question is not "how many words can I send?" It is "how many tokens will those words produce?"

A paragraph that looks short may still use more tokens than its word count suggests, while another tokenizer may encode it more compactly. Counting with the right tokenizer is more reliable than guessing from visible length.

Token count first matters as a capacity measure: every input token uses part of the context window. Whether the model uses the right detail is a separate attention and retrieval question that the next phase will explore.

How many tokens can the desk actually hold? That question has a specific, measurable answer, and it is the subject of the next episode.

Glossary

Tokenization
The model-specific process that converts text into a sequence of token IDs before processing. | Office analogy: A tokenizer prepares ordered sticky notes for Lex. | Example: Two model tokenizers can divide the same sentence into different numbers of pieces.
Token types
The kinds of text a token can represent, such as a whole word, part of a word, punctuation, or whitespace bundled with nearby text. Boundaries depend on the tokenizer. | Office analogy: Sticky notes can hold different-sized text fragments. | Example: Check the tokenizer for the model you use rather than assuming one word equals one token.
LLM
An AI that reads input and writes a response, like Lex the intern at his desk. | (origin: P01-E01)
Context window
The bounded workspace assembled for one model response, like Lex's desk. | (origin: P01-E01)

Try it yourself

  1. Count the words in a short message, then inspect it with a tokenizer for a specific model. Record the model or tokenizer name with the result.
  2. Compare the same sentence in two model-specific tokenizer tools. Notice whether the boundaries or counts differ.
  3. Compare prose, code, a URL, and non-English text of similar visible length. Treat the result as tokenizer-specific, not a universal rule.

Next episode

Lex keeps adding sticky notes to the desk. Twelve. Twenty. Thirty-five. Ava asks: "Is there a limit?" Lex pauses. Next: the desk has a fixed size, and we are about to find its edges.