How Large Language Models Work
A first-principles guide to transformer attention, next-token prediction, sampling mathematics, and why AI models hallucinate.
Pieces in this series
How Large Language Models Generate Text
From raw prompt strings to token embeddings, self-attention calculations, and next-token probability sampling
Large language models do not think, reason, or look up answers in a hidden database. They are statistical machines that predict the single most probable next token in a sequence, repeating that calculation billions of times per minute.
Why AI Chatbots Sometimes Make Things Up
The mechanics of hallucination, probability vs. truth, and the engineering behind grounded AI responses
AI hallucinations are not software bugs or accidental glitches. They are the direct, natural consequence of training a neural network to maximize statistical plausibility rather than factual truth.