How Large Language Models Actually Work

Large language models feel like magic, but the mechanism is surprisingly compact. Understanding it is the first step to using them reliably.

Text becomes tokens

An LLM never sees words. A tokenizer splits text into tokens: whole words, word fragments or punctuation. "Unbelievable" might become un, believ, able. Each token maps to an integer ID, and every API call is billed and limited by the number of tokens in and out.

Tokens become vectors

Each token ID is looked up in an embedding table, producing a long vector of numbers. Tokens with related meanings end up near each other in this space, which is why the model can generalise across synonyms and paraphrases.

Attention decides what matters

The core of a transformer is self-attention. For every token, the model computes how strongly it should look at every other token in the context. In the sentence "The trophy didn't fit in the suitcase because it was too big", attention is what lets the model link "it" to "trophy". Dozens of stacked layers, each with many attention heads, refine these relationships.

Predicting the next token

After the final layer, the model outputs a probability distribution over its entire vocabulary for the next token. A sampling strategy (greedy, temperature, top-p) picks one, appends it, and repeats. The whole answer is generated one token at a time.

How the model learns

• Pre-training: the model reads a vast text corpus and learns to predict missing next tokens. This is where world knowledge and language skill come from.
• Instruction tuning: examples of questions with good answers teach it to follow instructions.
• Preference tuning (RLHF and similar): human or AI feedback pushes the model towards helpful, safe responses.

Why LLMs hallucinate

The model optimises for plausible continuations, not verified ones. If its training data lacks a fact, it will still produce fluent text that fits the pattern. It has no built-in notion of a source of truth, and its knowledge is frozen at training time.

Making them dependable

Production systems wrap the model with the missing pieces:

1. Retrieval: fetch relevant, current documents and put them in the prompt.
2. Structure: give the model a schema or semantic layer so it works from defined concepts.
3. Tools: let it call calculators, databases and APIs instead of guessing.
4. Evaluation: measure accuracy on your own test set before and after every change.

The model is the reasoning engine; the surrounding system is what makes it trustworthy.