How Large Language Models Generate Text

Quick answer

Large language models (LLMs) like GPT-4, Claude, or Llama do not possess thoughts, awareness, or intentionality. They do not maintain a mental model of the world, nor do they query an internal search engine when you ask them a question.

At its core, a large language model is a massive mathematical function trained on trillions of words to perform a single recurring task: given an input sequence of text, calculate the probability distribution of what token should come next.

The generation process runs in an automated loop:

1. Tokenization: The software chops your text prompt into standardized numeric chunks called tokens.
2. Vector Embedding: Each token ID is mapped to a vector—a list of thousands of precise coordinates in a high-dimensional geometric space.
3. Transformer Self-Attention: Dozens of stacked neural network layers allow every token to inspect and weigh every other token in the prompt, updating its numeric meaning based on grammatical and semantic context.
4. Logit Projection & Softmax: The final hidden state of the last token is projected across the entire vocabulary (often 32,000 to 128,000 possible tokens), producing a percentage probability for every possible next token.
5. Sampling: A decoding algorithm (such as temperature or top-p nucleus sampling) selects a single winning token.
6. Autoregressive Loop: The winning token is appended to the prompt, and the entire neural network evaluates the enlarged sequence all over again to choose the token after that.

Every paragraph, poem, explanation, or block of Python code produced by an AI model is the cumulative output of this mechanical token-by-token loop.

---

The simple mental model: The Auto-Regressive Pipeline

Think of an LLM as the world's most sophisticated autocomplete engine. When your smartphone keyboard suggests three words above the keypad based on the last word you typed, it uses a tiny statistical model. An LLM performs that same conceptual task, but it conditions its prediction on thousands of words of prior context simultaneously, calculated through hundreds of billions of interconnected mathematical weights:

---

Phase 1: Breaking words into numbers (Tokenization)

Computers cannot perform matrix multiplications on letters or words directly; they operate exclusively on numbers. Before any neural network calculation begins, the text prompt must be transformed into a sequence of integer IDs. Computers cannot perform matrix multiplications on letters or words directly; they operate exclusively on numbers. (At the silicon level, these calculations are executed by billions of physical semiconductor gates and clocked execution pipelines; see How Binary and Logic Gates Became Computation and How a CPU Executes an Instruction.) Before any neural network calculation begins, the text prompt must be transformed into a sequence of integer IDs.

Why not simply use individual letters or whole words?

Early computational linguistics tried two extremes:

• Character-level models: Processing one letter at a time (c, a, t). While this requires a tiny vocabulary (around 100 characters for English), it forces the neural network to spend enormous computational resources merely learning how to spell words, and it inflates sequence lengths by a factor of five.
• Word-level models: Assigning a unique number to every English word. This creates an unmanageably vast vocabulary (millions of entries) that breaks when encountering typos, slang, compound terms, or foreign languages.

Modern LLMs use Subword Tokenization, most commonly an algorithm called Byte-Pair Encoding (BPE):

• Common words remain single, intact tokens: "the" $\to$ 464, "capital" $\to$ 3139.
• Rare or compound words are broken down into familiar syllables or morphemes: "unbelievable" $\to$ ["un", "believ", "able"].
• Code and whitespace are tokenized efficiently: four spaces of indentation often compress into a single token.

The famous tokenization quirks

Because humans communicate in concepts and letters while models process tokens, several well-known AI quirks emerge directly from this layer:

Prompt: "How many letters 'r' are in the word strawberry?"

To a human, the word contains nine letters (s-t-r-a-w-b-e-r-r-y) with three rs. But to a tokenizer, "strawberry" is typically chunked into two discrete tokens: ["straw", "berry"]. The underlying neural network never actually inspects the individual letters unless prompted to spell it out character by character.

Similarly, when performing multi-digit math, numbers like 95821 may be split arbitrarily into ["958", "21"]. Because the model does not "see" the raw digits arranged in vertical decimal columns, arithmetic requires explicit step-by-step reasoning tokens (often called "scratchpad" or "chain-of-thought") to track carries.

---

Phase 2: From numbers to geometry (Vector Embeddings)

Once the prompt is a list of token IDs, each integer is translated into a vector embedding—a continuous coordinate in an extremely high-dimensional space.

If an LLM uses an embedding dimension of $d_{\text{model}} = 4,096$, each token ID points to a row in a lookup table containing 4,096 floating-point numbers:

$$\vec{v}_{\text{Paris}} = [0.0241, -0.8120, 0.4501, \dots, 0.1192]$$

Semantic geometry

During pre-training, backpropagation forces semantically related tokens to gravitate toward similar neighborhoods in this 4,096-dimensional universe:

• Words with similar grammatical roles (such as "apple", "banana", and "orange") align closely in subspace clusters.
• Geometric directions represent conceptual relationships: moving along a specific vector offset shifts a concept from singular to plural, or from a country to its capital city ($v_{\text{Paris}} - v_{\text{France}} \approx v_{\text{Rome}} - v_{\text{Italy}}$).

Adding position: Rotary Position Embeddings (RoPE)

Unlike legacy Recurrent Neural Networks (RNNs) that read text sequentially from left to right, a Transformer processes all tokens in a prompt simultaneously in parallel.

Without extra information, the model cannot distinguish between:

• "The dog bit the man"
• "The man bit the dog"

To preserve word order, models apply positional encodings. Modern open-weights and frontier models utilize Rotary Position Embeddings (RoPE). RoPE rotates pairs of coordinates in the embedding vector by an angle proportional to the token's absolute position in the sentence. When two tokens interact during attention, the inner product naturally captures the relative distance between them, allowing the model to distinguish adjacent words from words separated by fifty paragraphs.

---

Phase 3: The Transformer Core (Self-Attention & MLPs)

Once tokens are converted into position-aware vectors, they pass through a stack of identical Transformer decoder layers (typically 32 layers in smaller 7-billion parameter models, and over 80 to 120 layers in frontier architectures).

Each layer contains two primary sub-blocks:

1. The Multi-Head Self-Attention Mechanism (which moves information between tokens).
2. The Feed-Forward Network / MLP (which processes information within each token).

┌────────────────────────────────────────────────────────┐
│ Transformer Layer N                                   │
│                                                        │
│   Input Vectors                                        │
│         │                                              │
│         ▼                                              │
│   ┌──────────────────────────────────────────────┐     │
│   │ Multi-Head Self-Attention                     │     │
│   │ (Tokens query and exchange context)          │     │
│   └──────────────────────────────────────────────┘     │
│         │  + Residual Connection                       │
│         ▼                                              │
│   ┌──────────────────────────────────────────────┐     │
│   │ Feed-Forward MLP (Dense Layers)              │     │
│   │ (Associative factual memory / transformation)│     │
│   └──────────────────────────────────────────────┘     │
│         │  + Residual Connection                       │
│         ▼                                              │
│   Output to Layer N + 1                                │
└────────────────────────────────────────────────────────┘

The Query, Key, and Value intuition

Self-attention allows every token in the prompt to ask: "Which other tokens in this sequence are relevant to my meaning?"

To compute this, every token generates three separate vectors via learned weight matrices:

• Query ($Q$): "What kind of information am I looking for?"
• Key ($K$): "What kind of information do I contain?"
• Value ($V$): "If someone needs my information, what content do I hand over?"

Consider the sentence:

"The bank of the river was muddy because it had rained for three days."

When the model processes the ambiguous token "bank", its Query vector reaches out across the sentence:

• It computes dot products against the Keys of all other tokens.
• The dot product against "river" and "muddy" yields high positive numbers (strong attention weights).
• The dot product against unrelated words yields numbers near zero.

The model calculates a weighted sum of the Value vectors based on these attention scores. Through this calculation, the vector representation of "bank" is updated in real time to mean geological riverbank rather than financial institution.

Mathematically, this is expressed by the famous attention equation:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

The scaling factor $\sqrt{d_k}$ (where $d_k$ is the dimension of the key vectors) prevents dot products from growing excessively large, which would otherwise push the softmax function into regions with near-zero gradients.

Feed-Forward Networks: Where facts reside

After tokens exchange context via attention, each token passes independently through a Feed-Forward Network (a Multi-Layer Perceptron, or MLP).

While self-attention acts as the routing switchboard connecting tokens, research shows that the billions of parameters in the MLP layers act as key-value associative memories. It is within these dense matrix weights that static world facts—such as "Paris" being associated with "France" or "boiling point of water" being associated with "100°C"—are largely encoded.

---

Phase 4: Output Logits and Softmax

After traversing dozens of Transformer layers, the model produces a final hidden vector for the very last token in the sequence.

If the prompt was "The capital of France is", the final token "is" now holds an enriched 4,096-dimensional vector that has absorbed the context of "capital", "France", and "is".

The Unembedding Projection

The model multiplies this final vector by an unembedding matrix ($W_u$). This matrix projects the 4,096-dimensional hidden vector back into a gigantic array matching the size of the vocabulary (e.g., 100,000 items).

This raw array is called the logits:

Token in Vocabulary	Raw Logit ($z_i$)
" Paris"	18.42
" the"	12.10
" situated"	10.85
" located"	9.40
" a"	8.15
" banana"	-6.20

Normalizing into Probabilities with Softmax

Raw logits are unconstrained real numbers. To turn them into valid probabilities that sum to 100% ($1.0$), the model applies the Softmax function:

$$P(w_i) = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}$$

Where $T$ is the temperature parameter. Now every candidate word has an exact mathematical probability:

• " Paris": 88.4%
• " the": 5.2%
• " situated": 2.8%
• " located": 1.9%
• " a": 0.9%
• " banana": 0.000001%

---

Phase 5: Sampling and Temperature

Now that the model has calculated probabilities for every token, how does it pick the winner?

This choice is governed by decoding parameters.

Probability Distribution over Vocabulary:

[" Paris"]     ████████████████████████████████████ 88.4%
[" the"]       ██ 5.2%
[" situated"]  █ 2.8%
[" located"]   █ 1.9%
[" a"]         ░ 0.9%

1. Greedy Decoding ($T = 0$)

The computer strictly picks the token with the highest probability (" Paris").

• Advantage: Deterministic, fast, and mathematically optimal for factual recall and code generation.
• Disadvantage: Often leads to repetitive, stiff prose and catastrophic looping when writing long narrative passages.

2. Temperature ($T > 0$)

Temperature scales the logits before the softmax calculation:

• Low Temperature ($T = 0.2$): The highest logits are exaggerated, making the distribution steep and peaked. The model sticks almost exclusively to high-confidence tokens.
• High Temperature ($T = 1.2$): The logit differences are flattened out. Low-probability tokens receive a much higher chance of being rolled, producing creative, unexpected, or bizarre phrasing.

3. Top-k and Top-p (Nucleus) Sampling

To prevent high-temperature generation from picking nonsensical tokens like " banana" in the middle of a serious history essay, modern runtimes apply mathematical filters:

• Top-k: Truncates the candidate pool to only the top $k$ most likely tokens (e.g., $k = 40$).
• Top-p (Nucleus Sampling): Dynamically sorts tokens by probability and keeps only the smallest subset whose cumulative probability reaches threshold $p$ (e.g., $p = 0.90$). If the model is 99% confident in a single word, only that one word is considered. If the model is uncertain, the pool expands naturally.

---

The Autoregressive Loop: Why inference is sequential

Once a single token is sampled (e.g., " Paris"), the model does not stop. It enters the autoregressive loop:

1. " Paris" is appended to the original prompt:

   "The capital of France is Paris"

2. The entire updated prompt is passed back into the Transformer.
3. The model calculates the next set of logits based on the new final token.
4. The winning token might be "." (period).
5. The prompt becomes "The capital of France is Paris."
6. The next winning token might be the special end-of-sequence token (<|im_end|> or <|endoftext|>).
7. When the model emits the end-of-sequence token, generation terminates.

The KV Cache: Memory vs. Compute

Because each new token requires a pass through the neural network, generating 100 words (roughly 130 tokens) requires running the entire multi-billion-parameter model 130 separate times in sequence.

If the model had to re-calculate Query, Key, and Value vectors for all past tokens on every single step, inference time would scale quadratically with length.

To avoid this, runtimes maintain a KV Cache in GPU memory (VRAM). Keys and Values for historical tokens are preserved in high-speed memory; the neural network only computes calculations for the single new incoming token.

This creates a physical hardware reality:

• Pre-fill phase (reading your initial prompt): Highly parallelized, compute-bound (saturates GPU tensor cores).
• Generation phase (streaming output tokens one by one): Highly sequential, memory-bandwidth bound. The GPU must stream gigabytes of model weights from VRAM into compute registers just to generate a single token.

---

Failure Modes and Architectural Boundaries

Understanding the mathematical mechanics of generation reveals why LLMs fail in predictable ways:

1. No retroactive correction

Humans plan sentences with internal foresight and can pause mid-sentence to revise an idea before vocalizing it. An autoregressive LLM generates strictly forward in time. If a model selects a slightly suboptimal token at step 4, it cannot backtrack; it is forced to condition all subsequent tokens on that earlier misstep, frequently doubling down on flawed logic.

2. The quadratic context cost

In standard attention, every token attends to every other token. A prompt of length $N$ requires $N \times N$ attention calculations ($O(N^2)$). While innovations like FlashAttention and multi-query attention make long contexts feasible, extremely long context windows place heavy burdens on GPU memory and memory bandwidth.

3. Context dilution ("Lost in the Middle")

Even with 1-million-token context windows, attention distributions are normalized via softmax. When thousands of tokens compete for attention weights, subtle facts buried in the middle of a massive document receive diluted attention scores compared to tokens near the immediate beginning or end of the prompt.

---

Why this matters

When you interact with a modern AI assistant, it is easy to succumb to anthropomorphism—to feel that an entity is listening, comprehending, and answering your thoughts.

In reality, the entire experience is an extraordinary triumph of statistical approximation and hardware engineering:

• There is no repository of stored truth.
• There is no consciousness reflecting on your intent.
• There is only an exquisitely tuned mathematical function projecting vectors across high-dimensional manifolds, balancing attention weights, and sampling the next most probable sequence of syllables.

When the statistical distribution aligns with historical reality, the output appears brilliantly insightful. But when probability diverges from fact, the exact same mathematical machinery produces fluent, confident nonsense.

To understand how and why this divergence occurs, explore the companion explainer on Why AI Chatbots Sometimes Make Things Up or see how models are chained into autonomous workflows in How AI Agents Work.