EXPLORE THE TOPIC
Running local models with attention to memory, speed, and quantization tradeoffs.
22 published articlesA single-GPU RAG stack works once you decide which process owns the GPU. The most predictable pattern reserves the GPU for generation and keeps embeddings and retrieval…
Exact reproduction of an Ollama run requires pinning five layers at once: the model artifact, the runtime version, the sampling parameters, the prompt and context, and the…
GGUF, Safetensors and GPTQ are not three versions of the same thing. GGUF and Safetensors are container formats, while GPTQ is a quantisation method whose weights are…
Start with Q4_K_M, then let the model repository override that default. This guide explains what GGUF quantization actually changes, how to read the documented quality, size, and…
Local LLM endpoints time out for reasons cloud APIs rarely do: cold model loads, KV cache allocation, prompt evaluation, and queueing behind other requests. This guide explains…
For local LLMs, dedicated VRAM capacity decides which models you can load at all — compute speed only decides how fast they answer. This guide explains how…
Local AI means running a language model on hardware you control instead of sending prompts to a hosted service. This guide explains how local inference works, how…
Raising a local model's context window from 2K to 32K changed generation speed by essentially nothing in a published llama.cpp sweep, but added about 1.7 GB of…
Quantization rewrites a model's weights in lower precision while leaving its architecture, tokenizer, and parameter count intact. The memory savings are predictable; the quality cost is not,…
Choosing a local AI model is a hardware-fit problem before it is a model-quality problem. Define the job, find your real memory budget, estimate weights plus KV…
Ollama installs with a single command on macOS, Linux, and Windows, and a first local model can be running within minutes. This guide covers prerequisites, installation per…
A slow local AI model is usually not one problem. Separate model loading, prompt processing, and token generation first, then check GPU offload, context length, quantization, and…