GGUF vs Safetensors vs GPTQ: Local Model File Formats Explained
GGUF, Safetensors and GPTQ are not three versions of the same thing. GGUF and Safetensors are container formats, while GPTQ is a quantisation method whose weights are…
TAGGED GUIDES
#quantization
Explore the guides, ideas, and practical explanations in this topic.
13 published articlesHow to Choose Quantization Settings for Local Models Without Breaking Output Quality
Start with Q4_K_M, then let the model repository override that default. This guide explains what GGUF quantization actually changes, how to read the documented quality, size, and…
How to Choose a GPU for Local AI: VRAM, Quantization, and Model Size
For local LLMs, dedicated VRAM capacity decides which models you can load at all — compute speed only decides how fast they answer. This guide explains how…
Local AI: A Complete Guide to Running Language Models on Your Computer
Local AI means running a language model on hardware you control instead of sending prompts to a hosted service. This guide explains how local inference works, how…
How Context Length Affects Local AI Memory Usage
Raising a local model's context window from 2K to 32K changed generation speed by essentially nothing in a published llama.cpp sweep, but added about 1.7 GB of…
What Quantization Changes in a Language Model (and What It Leaves Alone)
Quantization rewrites a model's weights in lower precision while leaving its architecture, tokenizer, and parameter count intact. The memory savings are predictable; the quality cost is not,…
How to Choose a Local AI Model for Your Computer
Choosing a local AI model is a hardware-fit problem before it is a model-quality problem. Define the job, find your real memory budget, estimate weights plus KV…
Why a Local AI Model Runs Slowly: A Troubleshooting Checklist
A slow local AI model is usually not one problem. Separate model loading, prompt processing, and token generation first, then check GPU offload, context length, quantization, and…
RAM vs VRAM for Local AI: What Each One Does
When you run a local LLM, VRAM is where inference happens on the GPU, while system RAM provides capacity for loading models, CPU inference, and offloading. Both…
How to Compare Local Language Models with Reproducible Tests
A reproducible local model comparison pins the model, runtime, hardware, prompt template, context settings, and sampling parameters before measuring quality, latency, memory, and failure rates. This guide…
AI Hardware Explained: RAM, VRAM, Quantization, and Model Size
Running a local language model is mostly a memory-planning problem. This guide explains how parameter count, quantization, VRAM, RAM, context length, and runtime overhead determine whether a…
How Large Language Models Work: Tokens, Context, Training, and Inference
Large language models are next-token predictors, and almost every practical constraint they have follows from four ideas: how text becomes tokens, how the context window limits what…