What happens between your prompt and the next word?
You paste a paragraph into an AI assistant and ask for a summary. To you, it is a collection of words and sentences. Before a language model can process it, software divides the text into smaller pieces and gives each piece an identifier. Those pieces are called tokens, and the software that makes them is the tokenizer.
A token is not necessarily a whole word. It might be a word, part of a word, punctuation or a sequence of bytes—the small units used to store digital information. For example, a hypothetical vocabulary could contain play, ground and ing, allowing playing and playground to be built from smaller pieces. This is the idea behind subword tokenization: reusing a limited set of pieces to represent many different expressions. It is not a claim that a particular production model splits those words that way.
Different tokenizers choose different pieces. Algorithms such as byte pair encoding, WordPiece and Unigram build and use vocabularies in different ways. Spaces, capitalization and normalization also matter. Normalization means changing text into a consistent form, such as using a standard representation for accented characters. Because those choices vary, a token count from one tokenizer is not automatically the right count for another model.
Loading the interactive example.
Try two simple segmentation rules
Try entering a short fictional message below. The first rule groups letters and numbers into runs, keeps runs of spaces together, and separates punctuation. The second breaks the text into Unicode code points, the numerical units used to represent characters in computing. Both rules preserve the original text. They make splitting visible, but neither uses the learned vocabulary of a GPT or other production language model.
Switch between the two rules and compare the count and displayed pieces. A word can become several code points. A visible character can also contain more than one code point: an accented letter may combine a letter and an accent, and some emoji join several parts. The numbers beside the pieces say where they occur in your entered text; they are not IDs from a model’s vocabulary.
Code points are not necessarily visible characters: a combining accent or a joined emoji can span several. These counts are not a way to estimate a provider's token bill.
Use fictional text in the controls. The calculations happen in your browser. Existing site analytics remain enabled, and the editable text and displayed results are masked for Clarity.
In a real application, token counts matter partly because a model has a context window: a limit on the sequence it can work with at one time. That sequence can include the system instructions, previous conversation, retrieved documents and descriptions of available tools, as well as the message the user just typed. Leave space for the answer too. When the request is too large, decide which material the task needs instead of simply cutting off the end. Losing one important exception can change a summary more than losing several paragraphs of background.
Once the text has been converted to token identifiers, a model maps them to numerical representations. A representation is a collection of numbers the model can process. In a transformer language model, layers of computation combine information from the available context, so a word’s representation can depend on the words around it. The model then produces scores for possible next tokens. Those scores tell the generation process which continuations the model considers more likely; they do not verify the truth of a continuation.
The generation process chooses a next token, adds it to the sequence and repeats. Choosing tokens from the scores is called decoding, and converting the resulting token sequence back into text is also commonly described as decoding. The context tells us which meaning is intended. Generation can stop when an end marker is selected, a length limit is reached or the application applies its own stopping rule.
There are several ways to make each choice. One always takes the highest-scoring token. Another samples a choice from a probability distribution, so the same prompt can lead to different continuations. Temperature is a setting that changes how concentrated that distribution is: lower values tend to favor the strongest choices, while higher values give alternatives more opportunity. Other settings can restrict which candidates are considered. None of these settings gives the model new facts or checks whether its answer is correct.
Try a count-based next-token example
To see how a prediction can be added to a sequence, we use a much smaller system than a modern language model. It counts word pairs in eight invented sentences, which you can open below. A bigram means a pair of neighboring tokens. Our bigram model uses only the last word to decide what might follow it. It ignores all earlier words and learns no neural-network weights. Here the tokens are lowercase English words, and END represents the end of a sentence.
The starting prompt is the order. In the small collection of sentences, order is followed by is twice, by has once and by the end of a sentence once. Click Append most likely token to add is, then notice that the probabilities are recalculated using is as the last word. You are seeing the same basic repeat-and-append pattern used in text generation, but with a deliberately simple counting rule.
Inspect the complete fictional corpus
We add 0.5 to the count for every possible next token before turning counts into probabilities. This adjustment is called smoothing. It avoids assigning an absolute zero just because a word pair is absent from this tiny collection, but it also gives some unlikely choices a small probability. The rule always appends the choice with the highest probability, a method called greedy selection. It illustrates that choice without reproducing the computations of a transformer language model.
For the starting prompt, the displayed probability of is is 25%; END and has each receive 15%. These numbers come from the disclosed sentences and the smoothing rule, not from an LLM. Try unknown as the prompt. The collection contains no examples of what follows that word, so every candidate receives the same probability. The calculator displays that uniform fallback but disables appending for unknown contexts. For supported contexts, equal-probability choices use token order.
A modern language model uses a much richer representation of the sequence than this bigram example. Even so, a fluent continuation is not proof that an answer is true. A plausible order number is still unverified unless the application has checked the relevant record. For an integration, use the tokenizer and request format of the actual model, record their versions, and test representative text—including uncommon names, numbers, punctuation and multiple languages. Then check the complete task, such as whether the resulting summary preserves the information the reader needs.