How Large Language Models Work: Tokens, Context, Training, and Inference

Large language models (LLMs) are next-token predictors. Given a sequence of text, a trained model computes a probability distribution over what should come next, picks one option, appends it, and repeats. Chatbots, code assistants, summarizers, and translation tools all run on that same loop. Once you understand the loop, the practical questions — why a prompt gets truncated, why long documents slow down generation, why the same question produces different answers, why a small model fits on a laptop — stop being mysteries and start being engineering tradeoffs.

This guide is the central reference for that mental model. It covers four connected ideas: tokens (the units a model actually reads and writes), context (the working memory available at generation time), training (how the model’s weights acquire their behavior), and inference (how a trained model produces output). It is written for technically curious beginners who are moving toward practical implementation — running a local model, calling an API, or wiring an LLM into a workflow — and it assumes no prior machine-learning background. Advanced material appears after the foundations, and each section ends with the decision it should help you make.

A boundary note: this article explains general mechanisms shared across most modern LLMs. Specific numbers — context lengths, parameter counts, prices, license terms — change frequently and vary by model. Where a number matters, check the model card or provider documentation for the model you are actually using.

The core mechanism: prediction, not retrieval

An LLM does not store a database of facts and look things up. It stores parameters — numeric weights learned during training — and uses them to compute probabilities over the next token. Everything that looks like knowledge, reasoning, or style is a consequence of that statistical objective applied at very large scale.

This distinction explains several behaviors that otherwise seem contradictory. A model can produce a fluent, confident sentence that is factually wrong, because fluency and factual accuracy are different properties of the same probability distribution. A model can answer a question about a document you just pasted, because the document is in its context, not because it learned it. A model can appear to remember your earlier conversation within a session and then forget it entirely in a new session, because conversation history is context, not training.

The prediction loop runs in three conceptual stages: tokenization turns text into a sequence of integer IDs; the transformer processes those IDs through many layers of attention and feed-forward computation; and sampling converts the resulting probability distribution into a chosen token. The rest of this guide expands each stage.

Tokens: the units a model actually reads

What a token is

A token is a chunk of text mapped to an integer ID in a fixed vocabulary. Tokens are not words and not characters. Depending on the tokenizer, a token might be a whole common word (“the”), a word fragment (“token” + “ization”), a single punctuation mark, a number, or a byte sequence representing an uncommon character. The vocabulary is fixed when the model is trained and does not change afterward.

Most modern LLMs use subword tokenization, built with algorithms such as byte-pair encoding (BPE), WordPiece, or SentencePiece unigram models. These algorithms start from characters or bytes and repeatedly merge the most frequent adjacent pairs until the vocabulary reaches a target size. The result is a vocabulary, typically tens of thousands of entries, that balances two goals: covering common words with single tokens (efficient) and still being able to represent any string by falling back to smaller pieces (robust).

It is worth contrasting this with human vocabulary. Estimates of the vocabulary size of young adult speakers of American English range from about 30,000 to 100,000 words, depending on how “knowing a word” is defined, and children appear to acquire roughly 7 to 10 words per day to reach those levels (Jurafsky & Martin, Speech and Language Processing, draft of August 20, 2024). An LLM’s token vocabulary is a different kind of object: it is a fixed compression scheme chosen for coverage and efficiency, not a set of meanings. The model learns meaning later, from the statistical relationships between token IDs.

Why tokenization matters in practice

Almost every practical constraint on an LLM is measured in tokens rather than words or characters:

• Cost. Hosted APIs typically bill per input token and per output token, with output usually priced higher.
• Context limits. The maximum input the model can process is expressed in tokens, and the model’s own output counts toward that budget too.
• Speed. Generation speed is measured in tokens per second, and prompt processing time scales with prompt token count.
• Quality. Text that tokenizes inefficiently — unusual identifiers, long numbers, some non-Latin scripts — consumes more of the budget and can be harder for the model to handle.

A common rule of thumb for English prose is roughly four characters per token, or about 0.75 words per token. Treat this as an approximation, not a guarantee: it varies by tokenizer, by language, and by content type. Code, URLs, JSON, and numeric data usually tokenize less efficiently than ordinary prose. Before you rely on a number for budgeting, measure it with the tokenizer that matches your model.

The following snippet is illustrative. It shows the shape of a token-counting check using the Hugging Face transformers library; replace the model identifier with the model you actually use and confirm the library version in your environment.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('<model-id>')
text = 'Tokenization is not the same as splitting on spaces.'
ids = tokenizer.encode(text)
print('token count:', len(ids))
print('tokens:', tokenizer.convert_ids_to_tokens(ids))

Run this on a paragraph you care about — a support ticket, a product description, a code file — and compare the result to your word count. The gap is the practical difference between how humans and models segment text.

Inside the transformer: how tokens become predictions

Embeddings

Each token ID is mapped to a vector — a list of numbers — through an embedding table. Positional information is added or encoded alongside the embedding so the model can distinguish order; without it, a bag of tokens would carry no sequence information. The embedding table is one of the largest single components in many models because it has one row per vocabulary entry.

Self-attention

For each position, the model computes a query, a key, and a value vector. The query at one position is compared against the keys at all positions to produce attention weights, and those weights are used to combine the values. This is how information moves between positions in the context. Multi-head attention runs several of these operations in parallel with different learned projections, letting the model track different kinds of relationships at once. Grouped-query attention shares key and value projections across heads, which shrinks the KV cache at inference time.

Feed-forward layers and depth

After attention, each position passes through a feed-forward network independently. Stacking these blocks lets the model build increasingly abstract representations: early layers tend to capture local patterns, while later layers encode more meaning- and task-level structure. The final hidden state is projected onto the vocabulary to produce logits — raw scores for each possible next token — which a softmax function converts into probabilities that sum to one.

Two implications for practice. First, the model has no separate reasoning module; computation is distributed across layers and attention heads, which is part of why behavior can be hard to attribute to a single component. Second, the vocabulary projection is the same at every position, so the model can only ever output tokens that exist in its vocabulary. Anything outside that vocabulary must be assembled from smaller pieces.

Context: the model’s working memory

What the context window contains

The context window is the maximum number of tokens the model can attend to at one time. Everything the model can condition on during generation lives in that window: the system prompt, the conversation history, any retrieved documents or pasted text, the current user message, and every token the model has already generated in the current response.

This has a counterintuitive consequence. If you paste a 20-page contract and ask a question, the model is not searching a database — it is conditioning on tokens that are present in the window. If the contract exceeds the window, the earliest parts are dropped or the request fails, depending on the implementation. If a relevant detail sits in the middle of a very long context, models do not always weight it as strongly as information near the beginning or end. Long context is a capability, not a guarantee of uniform attention.

Prefill and decode: the two phases of inference

Inference happens in two distinct phases, and understanding them explains most performance complaints.

Prefill processes the entire prompt at once. Because the prompt tokens are all known in advance, the model can compute them in parallel, which is efficient on modern accelerators. Prefill produces the first output token and builds the key-value (KV) cache — a stored summary of the attention state for every token processed so far. The time this takes is the time to first token.

Decode generates the response one token at a time. Each new token depends on all previous tokens, so the model cannot parallelize across output positions. It reuses the KV cache instead of recomputing the prompt, which is what makes generation feasible at all. The speed of this phase is measured in tokens per second.

This is why a long prompt can feel slow to start but fast once it begins, while a long response feels steady but adds up. It is also why batching multiple requests together improves throughput: during decode, the accelerator is often underutilized by a single sequence.

The KV cache and memory

The KV cache is usually the reason a model that “fits” in memory still runs out of it. Model weights are a fixed cost; the KV cache grows with sequence length and batch size. A rough planning formula, stated as an illustration with explicit assumptions, is:

KV cache bytes ≈ 2 × n_layers × n_kv_heads × head_dim
                 × sequence_length × batch_size × bytes_per_element

The factor of 2 accounts for keys and values. If the model uses grouped-query attention, the number of KV heads is smaller than the number of attention heads, which reduces the cache substantially. Quantizing the cache to 8-bit or 4-bit elements reduces it further, at some cost to output quality. The exact values for a given model come from its configuration file — check it rather than guessing.

Practical implication: when you increase context length, you are not only increasing the model’s reach, you are increasing memory use linearly and attention computation superlinearly. A long-context configuration that works for one request may fail under concurrent load.

Training: how the model acquires its behavior

Training is the process that turns a randomly initialized network into a useful predictor. It happens in stages, and each stage contributes something different to the final behavior. The overview below follows the training pipeline described in Understanding LLMs: A Comprehensive Overview from Training to Inference, which covers data preprocessing, training architecture, pre-training tasks, parallel training, and fine-tuning, and pairs it with inference-side concerns such as model compression, parallel computation, memory scheduling, and structural optimization (arXiv:2401.02038v2).

Stage 1: data collection and preprocessing

Pretraining data typically comes from large web crawls, books, code repositories, and curated datasets. Raw data is not used as-is. Pipelines deduplicate near-identical documents, filter low-quality or harmful content, remove boilerplate, normalize formatting, and tokenize the result. Data quality and composition have a large effect on the final model: a model trained mostly on code behaves differently from one trained mostly on prose, even at the same parameter count.

Because this stage is expensive and largely invisible in the final product, it is also where many differences between models originate. Two models with similar architectures can behave very differently because of what they were trained on.

Stage 2: architecture

Most current LLMs are decoder-only transformers. The architecture stacks repeated blocks, each containing a self-attention layer and a feed-forward network, with residual connections and normalization. Self-attention lets each position weigh information from other positions in the context; the feed-forward network transforms each position’s representation. Parameters are the learned weights inside these blocks, plus the embedding table that maps token IDs to vectors and the output layer that maps final representations back to vocabulary scores.

Parameter count is a rough proxy for capacity, not a direct measure of quality. A well-trained smaller model can outperform a poorly trained larger one on a specific task. Architecture choices — attention variants, activation functions, normalization placement — also affect efficiency and behavior.

Stage 3: pretraining objective

The core pretraining task is next-token prediction, also called causal language modeling. The training corpus is split into token sequences; the model predicts each next token given the preceding tokens; the loss measures how far the predicted distribution is from the actual token; and gradients update the weights. Because the “label” for each position is simply the next token in the text, the process is self-supervised and can scale to enormous corpora.

This objective is why the model is good at continuation and completion, and why it has no inherent notion of truth. It is trained to produce likely text, not verified text. Instruction-following, refusal behavior, and conversational tone come later.

Stage 4: parallel training

A modern LLM does not fit on a single accelerator, and training it requires distributing computation across many devices. The main strategies are data parallelism (replicate the model, split the batch), tensor parallelism (split individual layers across devices), pipeline parallelism (split layer groups into stages), and sharding of optimizer state and parameters to reduce memory per device. These techniques are combined in practice, and they are the reason training runs can involve thousands of accelerators.

For readers working locally, this stage matters indirectly: the same constraints — memory per device, communication overhead, batch size — reappear at inference time in smaller form. The training overview notes that cost-efficient training and deployment are an increasing focus as LLM use grows (arXiv:2401.02038v2).

Stage 5: fine-tuning and alignment

A pretrained model is a text completer, not an assistant. Fine-tuning adapts it:

• Supervised fine-tuning (SFT) trains on curated instruction–response pairs so the model learns to follow directions rather than merely continue text.
• Preference optimization — including reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) — adjusts the model toward responses that human raters or preference data rank more highly. This is where much of the helpful-assistant behavior comes from.
• Parameter-efficient fine-tuning methods such as LoRA and QLoRA train a small number of additional parameters instead of updating the whole model, which makes adaptation feasible on modest hardware.

Fine-tuning changes behavior and style; it is a poor tool for injecting a large, frequently updated body of facts. For that, retrieval-augmented generation (RAG) — putting relevant documents into the context window at request time — is usually the better fit. The tradeoff is covered in the next-steps section.

What training does not do

Training ends. After it ends, the weights are frozen. When you chat with a model, it does not learn from your conversation in any persistent way, and it cannot update its knowledge with new events. The model’s knowledge is bounded by its training data, often described as a knowledge cutoff. Any appearance of learning within a session is context, not weight change. This is a common source of confusion and a key reason to verify time-sensitive information independently.

Inference: from weights to words

The generation loop

Inference is mechanical once the components are clear. The prompt is tokenized; prefill computes the first next-token distribution and fills the KV cache; a sampling rule selects a token; that token is appended; decode computes the next distribution using the cache; and the loop repeats until a stop condition is met — an end-of-sequence token, a maximum length, or a stop string.

The sampling rule is a policy choice, not part of the model:

• Greedy decoding always picks the highest-probability token. It is deterministic and often repetitive.
• Temperature rescales the distribution. Low temperature (near 0) makes output more predictable; higher temperature flattens the distribution and increases variety.
• Top-k restricts sampling to the k highest-probability tokens.
• Top-p (nucleus sampling) restricts sampling to the smallest set of tokens whose cumulative probability exceeds p.
• Repetition penalties reduce the probability of tokens already used, which helps with loops but can harm accuracy if overdone.
• Beam search explores several candidate sequences in parallel and picks the best-scoring one. It is more common in translation-style tasks than in open-ended chat.

These settings explain why the same prompt can produce different answers on different runs. If you need reproducibility, set temperature to 0 and fix the seed where the runtime supports it — and expect that even then, different hardware or library versions can produce small differences.

Inference optimization

Because inference cost dominates real-world LLM usage, a large body of engineering work targets it. The training-to-inference overview groups these efforts into model compression, parallel computation, memory scheduling, and structural optimization (arXiv:2401.02038v2). In practical terms:

• Quantization stores weights (and sometimes activations or the KV cache) in lower precision — commonly 8-bit or 4-bit — reducing memory and often increasing speed. Quality loss varies by method and model; test on your own task rather than assuming.
• Distillation and pruning produce smaller models that approximate a larger teacher or remove redundant weights.
• Batching and continuous batching group requests to keep the accelerator busy during decode.
• Paged attention manages the KV cache in fixed-size blocks, reducing fragmentation and enabling higher concurrency. This is the technique behind several high-throughput serving engines.
• Speculative decoding uses a small draft model to propose several tokens that the larger model verifies in one pass, which can increase throughput without changing the output distribution.
• Memory scheduling decides what stays resident, what is offloaded, and when to evict cache entries — a major factor in serving cost.

Local inference: what changes

Running a model on your own hardware changes the constraints but not the mechanics. The same prefill/decode loop runs; the difference is that you own the memory budget and the latency profile.

Plan memory in three parts:

1. Weights. Approximately parameters × bytes per parameter. A 7-billion-parameter model at 4-bit is roughly 3.5 GB of weights before overhead; at 16-bit it is roughly 14 GB. These are approximations — the actual figure depends on the quantization scheme and the runtime.
2. KV cache. Grows with context length and batch size, as described above. Long context can exceed the weight footprint.
3. Runtime overhead. Framework buffers, temporary tensors, and the operating system’s own usage.

Practical starting rules: choose a quantized model that fits comfortably rather than one that barely fits; start with a modest context length and increase it only when a task requires it; measure time to first token and tokens per second on your own hardware instead of trusting published numbers; and keep an eye on memory as context grows. If generation is slow, check whether the bottleneck is prefill (long prompt), decode (large model), or memory pressure (swapping or offloading).

A worked example: one prompt, end to end

The following is an illustrative walkthrough, not a record of a specific test. Numbers are simplified to show the mechanism.

Suppose you send this request to a chat model:

System: You are a support assistant. Answer in one sentence.
User: A customer reports that their export job fails with a timeout
      after 30 minutes. Summarize the issue for the engineering team.

1. Tokenization. The full prompt — system message, user message, and any chat template tokens the runtime adds — is converted to token IDs. Suppose it totals 60 tokens. That count, not the 35 or so words, is what the model processes.
2. Prefill. All 60 tokens are processed in parallel through the transformer. The model produces a probability distribution over the vocabulary for the next token and stores attention state in the KV cache. This is the time-to-first-token.
3. Sampling. The runtime applies its sampling settings. At temperature 0, it picks the highest-probability token; at a higher temperature, it may pick a lower-ranked one.
4. Decode. The chosen token is appended, and the model computes the next distribution using the cache. This repeats token by token until the model emits an end-of-sequence token or hits the output limit.
5. Detokenization. The generated token IDs are converted back to text and returned. Chat templates may strip special tokens before display.
6. Human review. A one-sentence summary is a draft, not a verified report. The reviewer checks that the timeout duration, the failing component, and the requested action are accurate — the three details most likely to be distorted or invented.

Every practical lever in this guide maps onto one of these steps. Cost and context limits map to step 1. Latency maps to steps 2 and 4. Output variability maps to step 3. Truncation maps to the output limit in step 4. Quality control maps to step 6.

Choosing an approach: hosted API or local model

There is no universal winner. The right choice depends on privacy requirements, budget structure, hardware, and how much control you need.

Two cautions. First, open weights is not the same as open source, and neither automatically grants commercial-use rights; read the specific license. Second, a hybrid pattern is often best: use a hosted model for tasks that need the strongest reasoning, and a local model for high-volume, privacy-sensitive, or latency-critical work.

Evaluation, limitations, and responsible operation

LLMs are probabilistic text generators. Their failure modes are systematic, not random, and they can be tested for.

• Hallucination. The model produces fluent content that is not grounded in fact or in the provided context. Mitigation: require citations to supplied sources, ask for uncertainty, verify claims independently, and use retrieval so the relevant facts are in context.
• Staleness. Knowledge is bounded by training data. Mitigation: supply current information in the prompt or via retrieval rather than relying on the model’s memory.
• Context limits. Important information can be dropped or underweighted in long inputs. Mitigation: keep contexts focused, place critical instructions clearly, and test whether the model actually uses the middle of a long document.
• Tokenization artifacts. Tasks that depend on characters — counting letters, reversing strings, precise formatting — are unreliable because the model sees tokens, not characters. Mitigation: use tools or code for exact string operations.
• Sampling variance. Non-deterministic output complicates testing and auditing. Mitigation: fix sampling settings for evaluation runs and report them alongside results.
• Privacy and data handling. Prompts sent to a hosted API leave your environment. Mitigation: review provider terms, redact sensitive data, or run locally where the requirement demands it.

To evaluate a model or configuration for your own use, build a small fixed test set rather than relying on published benchmarks. Include representative inputs, a clear pass/fail criterion for each, and a record of the exact model version, quantization, context length, and sampling settings. Run the same set across candidates and compare failure patterns, not just scores. Published benchmarks measure specific tasks under specific conditions and rarely match your workload.

Troubleshooting common problems

When a problem is not listed here, isolate the variable. Change one setting at a time — context length, model, quantization, sampling — and record the result. Most LLM troubleshooting is controlled comparison, and community threads are most useful when they include the environment, exact steps, observed behavior, and logs.

A practical learning path

You do not need to train a model to understand one. A short sequence of hands-on steps builds the intuition this article describes.

1. Count tokens. Take a paragraph you use regularly and run it through a tokenizer for a model you are considering. Compare token count to word count. This calibrates every cost and context estimate you will make later.
2. Run something. Either call a hosted API with a simple prompt or run a small quantized model locally. The goal is a working loop, not a benchmark.
3. Test sampling. Send the same prompt at temperature 0 and at a higher temperature. Note the difference in consistency and creativity. This is the cheapest way to see the sampling policy at work.
4. Measure performance. Record time to first token and tokens per second for a short prompt and a long prompt. Watch memory as context grows. You now have the data to decide whether your hardware is adequate.
5. Add grounding. Build a minimal retrieval step: put relevant text into the prompt and require the model to answer only from it. Compare grounded and ungrounded answers on the same question.
6. Evaluate systematically. Turn your representative cases into a fixed test set with pass/fail criteria and record the configuration for each run.

From there, the natural next steps branch by goal. If cost or privacy is the driver, go deeper on quantization, KV cache management, and local serving. If accuracy on domain content is the driver, go deeper on retrieval-augmented generation and evaluation. If behavior and style are the driver, go deeper on fine-tuning methods and their tradeoffs against retrieval.

If you are working through a specific setup — a particular model, runtime, or hardware configuration — bring the details to the community: your goal, your environment, the exact steps you took, what you observed, and any logs or error messages. That level of detail is what makes a troubleshooting thread useful to the next person who hits the same wall.