The practical answer is that VRAM demand follows a fairly predictable structure: model weights scale with parameter count and quantization level, and everything else — context, KV cache, and runtime overhead — is added on top. At 4-bit quantization, a 3-billion-parameter model needs around 2 GB, a 7-to-8 billion model around 5 GB, a 13-to-14 billion model around 8 GB, and a 70 billion model around 40 GB. Doubling to 8-bit roughly doubles those figures, and full 16-bit precision roughly quadruples them (Textbook of Linux).
This guide is for anyone sizing a local inference setup — choosing a model for a card you already own, or deciding what to buy. It covers the arithmetic behind the estimates, the parts of memory use that the simple rule hides, and how to pick the largest model that fits comfortably rather than exactly.
The arithmetic behind every VRAM estimate
Weights are the easiest component to predict. Weight memory is approximately (parameters in billions) × (bits per weight) ÷ 8 gigabytes. A 70B model at Q4 therefore stores about 70 × 4 ÷ 8 = 35 GB of weights (llamaperf).
This is why quantization dominates sizing conversations. Moving from 4-bit to 8-bit roughly doubles the weight footprint, and moving to full 16-bit precision roughly quadruples it relative to 4-bit (Textbook of Linux). A 13B model that fits comfortably at 4-bit becomes a poor fit at 16-bit, where it needs on the order of 26 GB.
A second, more conservative formula comes from the Ohio Supercomputer Center, which assumes fp16 and adds an explicit context term: estimated GPU VRAM in GB = 2 × parameters (in billions) + 1 × context length (in thousands). For StableCode with 3 billion parameters and a 16k context, that yields 6 GB of weights plus 16 GB of overhead, or 22 GB total (OSC).
The two approaches are not really in conflict; they serve different purposes. The OSC version is a resource-request estimate for shared clusters, where over-requesting wastes an allocation but under-requesting kills a job. For a desktop card running a quantized model, the weight formula plus a context allowance is much closer to what you will actually observe.
Context and KV cache: the part that grows while you work
Weights are static. The KV cache is not. It grows with the length of the conversation, which means memory use at the end of a long session is higher than at load time. The practical consequence is stated bluntly in the Textbook of Linux guide: a model that exactly fills your VRAM at load time will run out part way through a long session (Textbook of Linux).
The llamaperf tables bake in roughly 2–4 GB of headroom for typical 4K–8K contexts and warn that long contexts of 32K or more, along with a full-precision KV cache, add meaningfully more (llamaperf). If you routinely paste long documents into a chat, or run an agent that accumulates tool output, size for the context you actually use — not the default.
An illustrative example: an 8 GB card
Consider a machine with an 8 GB GPU and a desktop environment that also consumes some VRAM. A 7-to-8B model at 4-bit needs about 5 GB for weights, leaving roughly 3 GB for context, KV cache, and runtime overhead. That is a good fit, and the Textbook of Linux guide names it as exactly that: on an 8 GB card, a 7-to-8B model at 4-bit is a good fit (Textbook of Linux).
The same guide notes that a 13B model at 16-bit is not a fit at that capacity and will fall back to CPU. This is the single most common failure mode reported by people new to local inference: the model loads, generation is dramatically slower than expected, and the cause is partial CPU offloading rather than a bad model.
This scenario is illustrative arithmetic, not a measurement. Your own headroom depends on your runtime, your context setting, and what else is using the GPU.
VRAM by model size and quantization
The table below comes from llamaperf’s requirements guide. Values include weights plus roughly 2–4 GB of headroom for typical context lengths and assume Q4_K_M, described there as the most common quantization (llamaperf).
| Model size | Q4_K_M | Q5_K_M | Q6_K | Q8_0 | FP16 |
|---|---|---|---|---|---|
| 7B | 5 GB | 6 GB | 7 GB | 8 GB | 14 GB |
| 13B | 8 GB | 10 GB | 12 GB | 14 GB | 26 GB |
| 30B | 18 GB | 22 GB | 26 GB | 32 GB | 60 GB |
| 70B | 40 GB | 50 GB | 58 GB | 70 GB | 140 GB |
| 120B (MoE) | 60 GB | 75 GB | 88 GB | 110 GB | 240 GB |
Two details matter when reading this table. First, the 70B Q4_K_M figure of 40 GB sits above the raw 35 GB weight calculation because of context and KV cache — a reminder that the weight formula is a floor, not a target. Second, MoE (mixture-of-experts) models are listed by total parameters rather than active parameters, which is why a 120B MoE entry appears so large even though only a fraction of the weights are used per token (llamaperf).
What fits on each VRAM tier
Working from the hardware side produces the same information from the opposite direction.
| VRAM | Common examples | What fits |
|---|---|---|
| 8 GB | RTX 3060 8GB, RTX 4060 8GB, 8 GB Apple Silicon Macs | 7B at any quant; 13B at Q3 with short context |
| 12 GB | RTX 3060 12GB, RTX 4070, 16 GB Apple Silicon Macs | 13B at Q4; 30B at Q3 with short context |
| 16 GB | RTX 4060 Ti 16GB, RTX 4070 Ti Super 16GB, RTX 5060 Ti 16GB | 13B at Q8; 30B at Q3/Q4 |
| 24 GB | RTX 3090, RTX 4090, RTX 5090 (32 GB) | 30B at Q4–Q6; 70B at Q2 with offloading |
| 32 GB+ | Apple Silicon Pro/Max at 32–64 GB, dual 3090/4090 | 30B at Q8; 70B at Q4 (single 32 GB+ card or dual 24 GB) |
| 48 GB+ | RTX A6000, dual 3090/4090, Apple Silicon Max 48 GB+ | 70B at Q4 with full context; 120B-class models |
Source: llamaperf’s VRAM requirements guide (llamaperf).
Where these numbers mislead
Offloading is not the same as fitting. The 24 GB row lists “70B at Q2 with offloading.” That configuration runs, but part of the model executes on the CPU, and generation speed reflects it. Treat offloaded configurations as a way to try a model, not as a way to run one comfortably.
Quantization level is a quality tradeoff, not a free dial. Q4_K_M appears throughout these tables because it is the common default, but the guide that produced them also maintains a separate quantization reference for what the codes mean (llamaperf). Dropping to Q3 or Q2 to squeeze a larger model onto a card is a real option, but you are trading model capability for capacity.
Capacity beats throughput. For local inference, VRAM capacity matters more than raw compute throughput: a slower card with more memory beats a faster card with less (Textbook of Linux). Buying decisions that optimize for clocks rather than memory often produce disappointing local-model performance.
Inference and training are different regimes. The OSC guide estimates training memory at roughly 40× the parameter count in billions, so a 7B model implies a minimum of about 280 GB — beyond a single H100 and requiring distributed training. Activation checkpointing can reduce that at the cost of longer runtime (OSC). None of the inference tables above apply to fine-tuning.
How to size your own setup without guessing
- Identify usable VRAM. Note that on a desktop system the display compositor and other applications consume some GPU memory, so the number in the specification is an upper bound rather than a budget.
- Pick a quantization first. Q4_K_M is the common starting point. It sets the weight footprint more than any other single choice.
- Compute weights. Parameters (billions) × bits ÷ 8. A 13B model at 4-bit is roughly 6.5 GB of raw weights, which rounds up to the 8 GB table entry once context is included.
- Reserve context headroom. Decide the context length you actually need. If you run long documents or agent loops, reserve more than the 2–4 GB that typical 4K–8K contexts imply.
- Test with a long session, not a short prompt. Memory grows as the conversation does. A model that survives a one-line prompt can still fail twenty turns in.
- If you hit an out-of-memory error, change one variable at a time. Reduce context length first, then lower the quantization, then step down in model size. Changing all three at once makes it impossible to learn what your card can actually hold.
Choosing a model for the card you have
If you have 8 GB, plan around 7–8B at 4-bit and accept that 13B means a lower quantization and a short context. At 12 GB, 13B at Q4 becomes comfortable. At 16 GB, 13B at Q8 or 30B at Q3/Q4 are both viable depending on how much you value quality versus size. At 24 GB, 30B at Q4–Q6 is the sweet spot, with 70B at Q2 available only with offloading. Above 32 GB, 70B at Q4 becomes realistic, and dual 24 GB cards open the same door (llamaperf).
The next action is straightforward: pick the largest model and quantization combination that leaves measurable headroom after a long session, not the largest one that merely loads. If you want to compare notes with other practitioners, share your GPU, model, quantization, context length, and observed memory use in the community — concrete configurations are far more useful to the next reader than a general recommendation.
FAQ
Can a 70B model run on a single 24 GB card?
Only in a limited configuration. The llamaperf tier table lists 70B at Q2 with offloading for 24 GB cards, which means part of the model runs on the CPU and generation is substantially slower than a fully resident model.
Does a faster GPU beat a GPU with more VRAM for local LLMs?
No. For this workload, VRAM capacity matters more than raw compute throughput — a slower card with more memory beats a faster card with less, because a model that does not fit falls back to CPU execution.
How much VRAM does fine-tuning or training need compared with inference?
Far more. The Ohio Supercomputer Center estimates training memory at roughly 40× the parameter count in billions, so a 7B model implies a minimum of about 280 GB and requires distributed training. Inference-focused VRAM tables do not apply.
Why does my model run out of memory after a long conversation?
The KV cache grows with context length. A model that exactly fills VRAM at load time will run out part way through a long session, so the practical rule is to choose the largest model that fits comfortably rather than exactly.