Local AI: A Complete Guide to Running Language Models on Your Computer

Local AI means running a language model on hardware you control — a laptop, a desktop, or a workstation — rather than sending each prompt to a hosted service. The appeal is concrete: prompts and documents need not leave your machine, there is no per-token bill, and you decide which model version you run and when to replace it. The cost is equally concrete: you supply the memory, the runtime, the model files, and the licence compliance.

This guide is written for readers who can install software and follow command-line instructions but have not yet run a language model themselves. By the end you should be able to explain how local inference works, estimate whether a given model fits your hardware, install a runtime, run a first model, verify the output rather than assume it, and recognise the failure modes that account for most beginner frustration. The scope is deliberately bounded: this is a guide to running existing open-weight models locally, not to training models from scratch or to building a production inference cluster.

What Local AI Is, and What It Is Not

A large language model is a large file of numbers — the weights — plus a program that performs arithmetic on those numbers. Local AI is the practice of running that arithmetic on your own hardware. The model itself is inert data; nothing happens until an inference engine loads it into memory and feeds it your input, one token at a time.

Three distinctions prevent most confusion.

Local inference is not the same as open source. Many models you can download carry open weights: the trained parameters are published, but the licence may restrict commercial use, redistribution, or derivative models. Others are released under permissive licences such as Apache 2.0 or MIT. A model being downloadable tells you nothing about what you may legally do with it. The distinction matters commercially: an Apache 2.0 release and a custom community licence can look identical on a download page and differ completely in what a business may ship.

Local inference is not training. Running a model — inference — needs enough memory to hold the weights and the working state, and enough compute to generate tokens at a tolerable speed. Training or fine-tuning needs orders of magnitude more. A machine that runs a 27-billion-parameter model comfortably may be entirely unable to fine-tune it.

Local does not automatically mean private. Keeping inference on your machine removes the model provider from your data path, which is a genuine gain. It does not remove every other party: an editor that syncs files to a cloud account, a runtime that sends anonymous telemetry, a plugin that calls a hosted API for embeddings, or a chat history stored in a synced folder can all move data off the machine. Privacy is a property of the whole pipeline, not of the model’s location.

How a Local Model Generates Text

Understanding the generation loop explains most of the performance behaviour you will observe.

Tokenisation. Text is split into tokens — roughly word fragments — by the model’s tokeniser. Token counts, not word counts, determine context usage and memory pressure. A page of dense technical prose can consume considerably more tokens than a page of simple dialogue.

Prefill. The engine processes the entire prompt at once, computing the key-value cache for every input token. This phase is compute-heavy and parallel, and for long documents it dominates the time before the first token appears.

Decode. The model then produces one token at a time, each depending on all previous tokens. This phase is limited by memory bandwidth more than raw compute, which is why generation speed depends heavily on how fast the hardware can move weights and cache.

Sampling. The model outputs a probability distribution over the next token; settings such as temperature and top-p decide how that distribution is sampled. Deterministic settings make comparisons reproducible; creative settings make them not.

Two practical consequences follow. First, time to first token and tokens per second are different measurements that respond to different bottlenecks — long prompts stress the first, weak hardware stresses the second. Second, the key-value cache is a first-class memory consumer, not an implementation detail, which is why context length and memory are inseparable.

The Four Components of a Local Setup

Every local deployment, from a laptop demonstration to a workstation serving a small team, is assembled from the same four parts.

• Model weights. The trained parameters, usually distributed in a quantised format such as GGUF for CPU and mixed CPU/GPU inference, or in safetensors for full-precision work. The file size is the single largest determinant of whether the model fits.
• Inference engine (runtime). The software that loads the weights, manages memory, applies the chat template, and generates tokens. llama.cpp, Ollama, and LM Studio are the common choices for local desktop use; RunLocal lists Ollama, LM Studio, and llama.cpp among the free software it recommends for getting started (runlocal.blog).
• Hardware. Memory is the binding constraint. GPU VRAM is fastest, system RAM is the fallback, and unified-memory machines sit between the two. Storage matters for holding several multi-gigabyte model files.
• Interface. The chat window, API endpoint, or script that sends prompts and displays results. This is the layer most users actually touch, and it is the easiest to replace without changing anything below it.

A useful mental model is a pipeline: your text is tokenised, the engine loads weights and a key-value cache into memory, the model produces one token, the token is appended to the context, and the process repeats. Every performance and memory symptom you encounter will trace back to one of those stages.

Where Local AI Sits Among Related Approaches

Local inference is one option in a family of approaches, and knowing the boundaries prevents wasted effort.

• Hosted APIs. You send prompts to a provider and pay per token. The provider handles hardware, scaling, and updates. The trade is data flow, recurring cost, and dependence on a service that can change or disappear.
• Self-hosting on a server. The same runtimes used on a desktop can serve a team from a machine you administer. This adds availability, access control, and backup obligations that a single-user desktop setup does not have.
• Retrieval-augmented generation (RAG). Instead of relying on the model’s internal knowledge, you retrieve relevant passages from your own documents and place them in the prompt. This is the standard way to work over material larger than the context window, and it is usually a better first investment than fine-tuning.
• Agents and tool use. The model calls functions — search, file access, calculators — to complete multi-step tasks. Some models are explicitly built for this; local-ai.net describes Muse Glimmer 30B as designed for agentic work including tool use, multi-step reasoning, and recovery from failures (local-ai.net).
• Fine-tuning. Further training on your own examples to change behaviour or style. It requires substantially more hardware than inference and should follow, not precede, a working RAG setup.
• Embeddings. A separate model converts text into vectors for search and clustering. Embedding models are small and fast compared with generative models, and they are often the first local model a beginner should run.

Why Memory, Not Raw Speed, Decides What You Can Run

Beginners usually ask which model is best. The operative question is which model fits, at a quality level you can accept, on the machine you have. Three quantities answer it.

Weight size and quantisation

Quantisation stores each weight in fewer bits. Full precision uses two bytes per parameter; an 8-bit quantisation uses roughly one; 4-bit quantisations use roughly half a byte, plus overhead. The trade is quality against size, and the trade is not linear — the drop from full precision to 8-bit is usually small, the drop from 8-bit to 4-bit is noticeable but often acceptable, and below about 4 bits quality degrades quickly for many models.

These figures describe the quantisation scheme, not a promise about any particular model. Always check the actual file size on the model page, because architectures and quantisation implementations differ.

Context and the key-value cache

Model weights are not the only thing in memory. Every token in the conversation occupies space in a key-value cache, and that cache grows with context length and with the number of concurrent requests. A model that fits comfortably at a 4,000-token context can fail at 32,000 tokens on the same machine.

Long-context models make this vivid. DeepSeek V4 Flash is described as supporting a 1M-token context, with 65k native and a 16-times extension via YaRN, which is a capability claim about the model rather than a statement that any desktop can serve a million tokens (runlocal.blog).

Runtime overhead

The engine, the operating system, the display, and any other application all compete for the same memory. Practical planning therefore reserves headroom rather than filling memory to the last gigabyte. A common rule of thumb is to keep 10 to 20 percent free, and more when you intend to raise context length.

Put together, the estimate looks like this:

required_memory ~= weights + kv_cache + runtime_overhead + headroom

For a 27B model at 4-bit, the weights alone land in the mid-teens of gigabytes, which is why a 24 GB graphics card is repeatedly described as the comfortable target for that class of model. RunLocal calls the Qwen 3.8 27B release the current default for a 24 GB card, and local-ai.net describes Gemma 3 27B as designed to run on a single 24 GB card at 4-bit (runlocal.blog; local-ai.net). Those are statements about fit, not guarantees of speed.

Dense and mixture-of-experts architectures

Not every parameter is used for every token. A dense model activates all of its weights on each token. A mixture-of-experts (MoE) model routes each token through a subset of specialised sub-networks, so total parameter count and per-token compute diverge sharply.

DeepSeek V4 Flash is described as a 304B-parameter MoE with 256 routed experts and 6 active per token, meaning roughly 13B parameters fire on any given token; the source notes that this makes it answer far faster than its total size suggests (runlocal.blog). The catch is memory: the whole model still has to be resident even though only a fraction is used at a time, so MoE improves speed per token far more than it reduces the memory requirement.

Choosing a Model: Size, Task, and Licence

Model choice is a three-way negotiation between what the task needs, what the hardware allows, and what the licence permits.

Task fit comes first

General chat, code assistance, summarisation, multilingual work, and agentic tool use stress different capabilities. A model that is excellent at conversational writing may be mediocre at structured tool calls, and vice versa. The only reliable way to know is to test on your own prompts, which is why the evaluation protocol later in this guide matters more than any recommendation list.

Size and architecture

Bigger is not automatically better for a given job. A well-quantised mid-size model that fits in fast memory frequently beats a larger model that spills into system RAM and generates at a crawl. MoE models complicate the comparison further, because a large total parameter count can come with a modest per-token cost.

Chat templates and prompt format

Instruction-tuned models expect a specific prompt structure, defined by their chat template. Using the wrong template is one of the most common causes of a model that seems unintelligent: the problem is not the model but the wrapper. Runtimes usually apply the template automatically when a model’s metadata declares one; when you call the underlying API directly, you are responsible for it.

Multimodality

Some models accept images as well as text. Qwen 3.8 27B is described as natively multimodal, and both Gemma 3 27B and Muse Glimmer 30B accept images (runlocal.blog; local-ai.net). Multimodal input adds memory pressure, because image tokens occupy context just as text tokens do.

Licence

Read the actual licence for the specific checkpoint. Some families mix terms across sizes — RunLocal notes that Qwen 3.8 carries Apache 2.0 on the 27B but a custom Qwen licence on the 2.4T MoE sibling, while DeepSeek V4 Flash is MIT-licensed (runlocal.blog). local-ai.net marks Gemma 3 27B and Llama 3.3 70B Instruct as permitted with conditions (local-ai.net). Permitted with conditions is not the same as unrestricted, and the conditions are the part that matters if you intend to publish or sell the output.

Some models worth knowing about, as catalogued by the sources consulted for this guide:

Model catalogues move quickly. local-ai.net, which tracks 48 models, 55 tools, 28 GPUs and machines, and 32 guides as of its current listing, warns that its fit figures are approximate and that fits does not always mean runs well, and that its prices are indicative rather than live (local-ai.net). Treat any catalogue, including this one, as a starting point that must be re-checked against the model card and licence before you commit.

Runtimes and Tools: Ollama, LM Studio, and llama.cpp

The runtime is where most beginners spend their first hour, and the choice is less consequential than it appears: these tools wrap much of the same underlying inference work.

• Ollama runs as a background service with a command-line interface and a local HTTP API. It handles model downloads and can keep models loaded between requests. It suits users who want a stable local endpoint that scripts and applications can call.
• LM Studio is a graphical application for downloading, configuring, and chatting with models, with a built-in server mode. It suits users who prefer to inspect settings visually before touching a configuration file.
• llama.cpp is the lower-level engine that much of the ecosystem builds on. It offers the most control over quantisation, offloading, and context settings, and it is the layer to reach for when a model requires a specific fork or a setting the friendlier tools do not expose.

The GGUF file format is the common currency for CPU and mixed CPU/GPU inference, and it is why a model released on one day can be running on a laptop the next: community quantisers publish GGUF builds quickly. RunLocal notes that GGUF builds for Qwen 3.8 27B appeared from unsloth and bartowski within days of release (runlocal.blog).

Forks matter more than they first appear. A fork can change the memory floor for a given model — RunLocal describes Prism ML’s Bonsai 2 build squeezing the same 27B model into about 6 GB, on the condition that you run it on Prism ML’s llama.cpp fork (runlocal.blog). That is a specific, verifiable claim about a specific build, not a general property of 27B models. Reproducing it requires the fork, and the quality trade-off should be evaluated on your own tasks rather than assumed.

Managing Models on Disk

A local setup accumulates files quickly, and a little discipline prevents confusion later.

• Keep one quantisation per model at first. Downloading four variants of the same model consumes disk and makes it easy to test the wrong one.
• Record filenames exactly. Quantisation suffixes are not interchangeable, and a comparison between two builds is meaningless if the filenames are not written down.
• Delete deliberately. Model files are large and otherwise invisible. Review storage periodically rather than discovering a full disk mid-experiment.
• Prefer official or well-known sources. A model file from an unverified mirror is an unverified artefact.
• Version your prompts alongside your models. The prompt template is part of the experiment, not an afterthought.

A Practical First Run: From Hardware Check to Verified Output

The following path is a suggested sequence, not a record of a test performed for this article. Each step ends with an observable check so you can tell whether it worked before moving on.

1. Inventory your hardware. Note GPU model and VRAM, total system RAM, free disk space, and operating system. Check whether the runtime you intend to use supports your GPU acceleration path. Check: you can state your VRAM and free RAM in gigabytes without looking them up again.
2. Pick a target model class from your memory budget. Subtract headroom from your usable memory, then match against published file sizes. If you have about 8 GB of VRAM, look at 7 to 9B models at 4-bit; around 12 to 16 GB, 12 to 14B; around 24 GB, the 27 to 32B class. Check: the model’s quantised file size plus an estimated key-value cache fits inside your budget with room to spare.
3. Install one runtime. Choose Ollama, LM Studio, or llama.cpp and stop there. Installing three in parallel makes every later problem harder to diagnose. Check: the runtime reports its version and lists no models yet.
4. Download a single quantisation. Start with a mid-range 4-bit quantisation rather than the largest available. Check: the file downloads completely and the runtime lists the model by name.
5. Run a short prompt with a small context. Ask something with a verifiable answer, such as a short arithmetic or factual question, and keep the context window modest. Check: tokens appear steadily and the response is coherent. Note the generation rate if the interface reports it.
6. Test the real task on a small input. Move from a toy prompt to one realistic document or question. Check: you can describe specifically what the model got right and wrong, rather than judging it good or bad overall.
7. Raise context deliberately. Increase the context setting in steps and watch memory. Check: performance degrades gracefully; if the process is killed or the machine swaps heavily, you have found your practical ceiling.
8. Record what you ran. Note the model name and quantisation, runtime version, context setting, hardware, and observed speed. This record is what makes later comparisons meaningful.

The ten-minute framing that appears on getting-started hubs refers to installation and a first prompt, not to a tuned setup (runlocal.blog). Expect the first working prompt to be quick and the first genuinely useful configuration to take longer.

Worked Example: A Local Summarise-and-Draft Workflow

The following is an illustrative workflow, not a tested case study. It shows how the components interact on a realistic task: turning a long source document into a structured draft.

Input. A 6,000-word document — a report, a transcript, or a set of research notes — plus a short instruction describing the desired output shape.

Process. The document is split into overlapping chunks small enough to fit the context window alongside the instruction. Each chunk is summarised by the local model, with the chunk index and source offsets preserved. The chunk summaries are then combined in a second pass into a single outline. A third pass expands the outline into a draft. Retrieval — fetching only the chunks relevant to a question — is the alternative to summarising everything, and is the basis of retrieval-augmented generation.

Output. A draft with citations back to chunk identifiers, so any claim can be traced to the passage it came from.

Evaluation and human review. Check three things separately: whether each summary is faithful to its chunk, whether the outline reflects the document’s actual structure, and whether the draft’s claims survive a spot-check against the source. The failure mode to watch for is fluent invention — a confident sentence that no chunk supports. Because the chunk identifiers are preserved, verification is a lookup rather than a re-read of the whole document.

Where the human belongs. Before publication or any irreversible action. A local model can draft, classify, and extract at volume; a person should decide what is true, what is publishable, and what is attributable. This division is also what makes local inference attractive in regulated or confidential settings, where the alternative is not no AI but AI with the data leaving the building.

Evaluating Quality, Speed, and Fit

Subjective impressions after three prompts are unreliable. A short, repeatable protocol produces evidence you can act on.

• Fix the task set. Write 10 to 20 prompts that represent your actual work, with a known-good answer or a clear pass/fail criterion for each. Reuse the same set for every model you compare.
• Fix the sampling settings. Temperature, top-p, and the system prompt change results more than most users expect. Record them alongside your results; otherwise a comparison measures the settings, not the model.
• Measure speed in tokens per second, not seconds per answer. Answers differ in length, so wall-clock time is misleading. Note prompt-processing speed separately from generation speed, because long inputs are dominated by the former.
• Watch memory at the peak, not at idle. Observe memory during the longest-context request, since that is where the key-value cache peaks.
• Score quality on a rubric. For each prompt, record correctness, adherence to format, and whether the answer is supported by the input. A three-point scale per dimension is enough to reveal differences.
• Test the failure boundary. Deliberately exceed the context window, ask a question the input cannot answer, and request a format the model has not seen. How a model fails is often more informative than how it succeeds.

Record the environment with the results: model name and quantisation, runtime and version, context length, hardware, and date. Without those, a result cannot be reproduced — including by you, three weeks later.

Limitations, Privacy, and Responsible Operation

Licensing is a real constraint. Open weights, open source, and permission for commercial use are three different things. Some models are released under permissive licences such as Apache 2.0 or MIT; others are permitted with conditions or carry custom terms that vary by model size within the same family (runlocal.blog; local-ai.net). Read the licence attached to the exact checkpoint you download, and re-read it when you upgrade.

Accuracy does not improve because the model is local. A local model can produce confident, unsupported statements in the same way a hosted one can. Local operation changes where the computation happens, not how reliable it is.

Privacy is a pipeline property. Local inference removes the model provider from your data path. It does not remove cloud-synced chat histories, telemetry, third-party plugins, or an embedding step that calls a hosted service. If confidentiality is the reason you are going local, audit the whole chain rather than the model alone.

Model files are code-adjacent. Prefer well-known formats and reputable sources. Files that require executing arbitrary code to load are a different risk category from a plain quantised weight file, and a model downloaded from an unofficial mirror is an unverified artefact.

Resource use is physical. Sustained inference draws power, produces heat, and wears fans. A workstation that runs a large model all day has a running cost that belongs in any comparison against an API bill. local-ai.net provides a cost calculator for exactly this break-even question, and states its assumptions explicitly, including the ones that count against local (local-ai.net).

Estimates are estimates. Fit calculators and memory rules of thumb are approximations. The same source notes that fits does not always mean runs well (local-ai.net). Treat every number as a hypothesis to test on your own hardware.

Connecting a Local Model to a Publishing Workflow

Local models are a natural fit for content operations where drafts are sensitive, volume is predictable, and a human editor is already in the loop. A workable architecture separates four concerns.

1. Input. A queue of jobs — a brief, a set of notes, a source document — with a stable identifier so the same job is never processed twice.
2. Generation. A script or service that calls the local runtime’s HTTP API, applies a fixed prompt template, and writes the result to a staging location rather than the live site.
3. Review. A human step that reads the draft, checks claims against sources, and decides whether to publish. Nothing reaches the public site without this gate.
4. Logging and recovery. Record the model, prompt, parameters, and output for every job. When a draft is wrong, the log is how you find out whether the cause was the prompt, the model, or the input.

Two cautions belong here. First, credentials and server access should be scoped to the minimum the job needs, and the local model should never hold publishing rights directly. Second, a local model’s output is not automatically safe to publish: attribution, licensing of the source material, and factual accuracy are editorial responsibilities that do not transfer to the tool. The value of the local setup is that it makes high-volume drafting affordable and keeps confidential source material on your own infrastructure — not that it removes the editor.

Troubleshooting Common Problems

Diagnose one variable at a time. Changing the model, the quantisation, the runtime, and the prompt together produces a result you cannot interpret.

Learning Path and Next Steps

Immediate next action. Install one runtime, download one 4-bit quantisation of a model that fits your memory budget, and run a single verifiable prompt. Record the model name, quantisation, runtime version, context setting, and observed speed. That record is the baseline every later comparison depends on.

Beginner to intermediate. Move from a toy prompt to your real task. Build a fixed prompt set of 10 to 20 items and score two models against it. Learn what your machine’s practical context ceiling is by raising it until performance degrades.

Intermediate to advanced. Add retrieval so the model can work over documents larger than its context window; add structured output so results can be parsed by other software; and separate the model endpoint from the interface so you can swap either without rebuilding the workflow.

Where to look things up. RunLocal maintains a plain-language glossary, a hardware picker, and an opinionated model directory aimed at readers choosing their first setup (runlocal.blog). local-ai.net provides a hardware capability matrix, a machine speccer for a chosen model and quality level, and a cost calculator for the break-even question between buying hardware and paying per token (local-ai.net). Both are catalogue-style resources; verify model licences and current file sizes at the source before committing.

If you are working through this for the first time, the most useful thing you can contribute back is a reproducible record: your hardware, the model and quantisation, the runtime version, the settings, and what actually happened. That is the evidence other readers need, and it is the kind of report that turns a personal experiment into shared knowledge.