Prompt Caching and Prefix Caching in Local LLMs: How They Reduce Memory and Latency
Prompt caching and prefix caching both reuse the key-value (KV) state a model already computed for an unchanged prompt prefix, so a new request can skip that…
TAGGED GUIDES
#KV cache
Explore the guides, ideas, and practical explanations in this topic.
7 published articlesHow Context Length Affects Local AI Memory Usage
Raising a local model's context window from 2K to 32K changed generation speed by essentially nothing in a published llama.cpp sweep, but added about 1.7 GB of…
What Quantization Changes in a Language Model (and What It Leaves Alone)
Quantization rewrites a model's weights in lower precision while leaving its architecture, tokenizer, and parameter count intact. The memory savings are predictable; the quality cost is not,…
How to Choose a Local AI Model for Your Computer
Choosing a local AI model is a hardware-fit problem before it is a model-quality problem. Define the job, find your real memory budget, estimate weights plus KV…
RAM vs VRAM for Local AI: What Each One Does
When you run a local LLM, VRAM is where inference happens on the GPU, while system RAM provides capacity for loading models, CPU inference, and offloading. Both…
How Large Language Models Work: Tokens, Context, Training, and Inference
Large language models are next-token predictors, and almost every practical constraint they have follows from four ideas: how text becomes tokens, how the context window limits what…
How to Record a Reproducible Local Model Benchmark
A reproducible local model benchmark is a record, not a number. It needs the exact model bytes and hashes, the runtime version, the hardware and power state,…