Local embedding models let you build retrieval-augmented generation (RAG) in which your documents are converted to vectors, stored, and searched entirely on hardware you control. The practical payoff is that the text of your notes, contracts, support tickets, or internal wiki never has to leave the machine — provided the model that writes the final answer also runs locally.
This guide is for technically curious beginners and intermediate practitioners who already understand roughly what RAG does and want a working mental model plus a concrete local stack. It covers what “private” really means in a RAG pipeline, how to choose an embedding model, how the components fit together, and where local embeddings quietly fail.
What “private RAG” actually means
Privacy in RAG is a boundary question, not a model question. A RAG pipeline has three distinct stages that can each run in a different place:
- Embedding — turning text chunks and the user’s question into numeric vectors.
- Vector storage and search — keeping those vectors in an index and finding the nearest ones.
- Generation — feeding the retrieved chunks to a language model that writes the answer.
A system is fully local only when all three run on your hardware. A common half-measure is local embeddings with a cloud LLM. That keeps your entire corpus off the vendor’s servers, but every retrieved chunk — often the most relevant few paragraphs in the document — is sent to the API along with the question. If that tradeoff is acceptable, make it deliberately; do not assume that “local embeddings” automatically means “private system”.
The reverse half-measure matters too: a fully local pipeline still leaks if your application logs full prompts to a remote observability service or syncs the vector database to a managed cloud instance.
The pipeline in five stages
- Ingest and chunk. Split source documents into passages small enough to be semantically focused but large enough to carry context.
- Embed. Run each chunk through the embedding model to produce a fixed-length vector.
- Store. Write the vector plus payload metadata (source file, page, heading) into a vector index.
- Retrieve. Embed the user’s question with the same model and search for the nearest vectors.
- Generate. Pass the retrieved chunks and the question to a local language model to produce a grounded answer.
Steps 2 and 4 must use an identical model and identical preprocessing. This is the single most common source of silently broken retrieval.
Choosing a local embedding model
Mozilla Builders published a shortlist of embedding models packaged as llamafiles for local RAG, selected for performance on RAG-relevant English-language tasks. Their comparison table is a useful starting point because it pairs capability with licensing and footprint:
| Model | License | Reported fp32 memory | Embedding size |
|---|---|---|---|
| Salesforce/SFR-Embedding-Mistral (best overall) | CC-by-NC | 26.49 | 4,096 |
| intfloat/e5-mistral-7b-instruct (best overall, commercial-friendly) | MIT | 26.49 | 4,096 |
| mixedbread-ai/mxbai-embed-large-v1 (best small) | Apache 2 | 1.25 | 1,024 |
Three caveats apply to that table. First, the memory figures are the source’s reported fp32 weight footprints — they are not total runtime memory, which also includes the KV cache, context buffers, and framework overhead. Second, the evaluation was explicitly scoped to English; the authors note that some models may be multilingual but that non-English performance was not assessed. Third, license is not the same as availability: SFR-Embedding-Mistral is distributed under CC-by-NC, which restricts commercial use, while e5-mistral-7b-instruct (MIT) and mxbai-embed-large-v1 (Apache 2) are permissive. Treat “open weights” and “commercial-use permitted” as separate questions and check the model card for the version you actually download.
For a first local build, the small model is usually the right call. A 1.25-unit fp32 footprint fits comfortably on modest hardware and leaves room for the generator, whereas the 7B embedding models compete with your LLM for the same memory.
A minimal local stack
The Gemma cookbook includes a notebook titled “Local Agentic RAG without APIs” that demonstrates a fully local pipeline built from three components: FastEmbed for generating embeddings, Ollama running Gemma 3 as the local generator, and Qdrant as the vector database. That combination is a reasonable template because each part is replaceable — swap the embedding library, the vector store, or the generator without changing the overall shape of the pipeline.
Component responsibilities break down cleanly:
- FastEmbed handles chunk-to-vector conversion on CPU or GPU and manages model download and caching.
- Qdrant stores vectors with payload metadata and performs nearest-neighbour search, including filtered search when you want to restrict results to a source or date range.
- Ollama serves the generation model locally over a simple HTTP interface, so the retrieved context never leaves the host.
An illustrative walkthrough
The following is a design sketch, not a report of a tested run. Exact library APIs change between versions; verify commands against the current documentation for each tool.
- Prepare a small corpus. Start with 10–50 documents you can read yourself. A corpus you cannot manually verify makes retrieval bugs impossible to diagnose.
- Chunk with metadata. Split on headings or paragraphs rather than fixed character counts where possible, and attach source filename, heading path, and chunk index to each piece. Target a few hundred words per chunk with modest overlap.
- Embed the chunks. Load the embedding model once and embed in batches. Record the model name and the resulting vector dimension alongside the index.
- Index in Qdrant. Create a collection whose vector size matches the embedding dimension exactly, and store the chunk text and metadata as payload.
- Query with the same model. Embed the user’s question using the identical model instance and configuration, then request the top k results.
- Generate locally. Assemble a prompt containing the retrieved chunks plus an instruction to answer only from the provided context, and send it to the Ollama-hosted model.
A minimal pipeline in pseudocode looks like this:
chunks = chunk_documents(docs, max_words=300, overlap=50)
vectors = embed_model.embed([c.text for c in chunks])
collection.upsert(vectors, payloads=[c.metadata for c in chunks])
q_vector = embed_model.embed([question])[0]
hits = collection.search(q_vector, limit=5)
answer = local_llm.generate(context=hits, question=question)Observable checks after each step: the number of vectors should equal the number of chunks; every vector’s length should equal the collection’s configured dimension; and a query for a phrase you know exists in one document should return that document in the top results. If it does not, stop and fix retrieval before touching the generator.
Chunking and retrieval quality
Embedding model choice gets most of the attention, but chunking usually decides whether retrieval works. Practical guidance that holds across models:
- Respect document structure. Splitting mid-sentence or mid-table produces vectors that represent nothing coherent.
- Keep overlap. A 10–20% overlap prevents answers that straddle a boundary from becoming unreachable.
- Store enough payload to cite. Source file, heading, and position let you show the reader where an answer came from and make debugging far easier.
- Consider hybrid search. Dense embeddings handle paraphrase well but can miss exact identifiers, error codes, and rare names. Combining vector search with keyword matching often fixes those cases.
- Retrieve more than you generate with. Fetching a wider candidate set and then trimming it gives better results than retrieving exactly the number of chunks you plan to pass to the model.
Memory, hardware, and licensing tradeoffs
The reported 26.49 fp32 footprint for the 7B embedding models is a weights figure. At runtime you must also budget for the context and KV cache, the vector index itself, and the generator model if it shares the machine. Quantizing the embedding model reduces memory but can shift retrieval quality, so re-evaluate your own queries rather than assuming a quantized model behaves like the full-precision one.
Throughput matters as much as capacity. Embedding a large corpus is a batch job, and on CPU-only hardware a small model can be dramatically faster to index than a 7B one. Since the embedding step is usually run once per document while generation runs once per query, favouring a smaller embedding model and a larger generator is often the better allocation of limited hardware.
Licensing deserves the same scrutiny as performance. If the RAG system supports commercial work, CC-by-NC models are off the table regardless of how well they score.
Common failure modes
- Dimension mismatch. The collection was created for one model’s vector size and the query model produces another. Symptom: index errors or nonsensical neighbours.
- Model drift between indexing and querying. A library upgrade silently changes the default model. Symptom: retrieval quality collapses after an update that touched nothing else.
- Missing normalization. Some models expect normalized vectors for cosine similarity. Symptom: results cluster around one document.
- Silent truncation. Embedding models have a maximum input length; longer chunks are cut off without an error. Symptom: the tail of every long document is unsearchable.
- Privacy leakage at the edges. Local inference with remote logging. Symptom: none visible in the application — you have to inspect outbound traffic and logging configuration to find it.
Where to go next
Build the smallest possible version first: one embedding model, one collection, one locally hosted generator, and a corpus you can read end to end. Verify retrieval with a handful of questions whose answers you already know. Once that works, the improvements that pay off most are usually better chunking and hybrid search, not a larger embedding model.
If you are choosing a model for a specific language or domain, test it on your own queries before committing to an index — published benchmarks are a filter, not a verdict. Readers working through a local RAG build are welcome to share their hardware, model choice, corpus size, and any retrieval errors in the community so others can compare notes.
FAQ
Do I need a GPU to run local embedding models?
Not necessarily. The small model on Mozilla's shortlist is listed with a 1.25 fp32 weight footprint, which fits on modest hardware. CPU throughput becomes the constraint when indexing large corpora, so measure your indexing time rather than assuming a GPU is required.
Can I use these embedding models in a commercial product?
It depends on the license. SFR-Embedding-Mistral is CC-by-NC, which restricts commercial use, while e5-mistral-7b-instruct is MIT and mxbai-embed-large-v1 is Apache 2. Check the model card for the exact version you download.
Why did retrieval quality drop after I changed models?
The query and the indexed chunks must be embedded by the same model with the same preprocessing, including vector normalization. If the model changes, the existing vectors are no longer comparable and the collection must be re-indexed.
Is local embedding enough to make my RAG system private?
No. Local embeddings only keep the embedding step on your hardware. If the generator is a cloud API, the retrieved chunks are still transmitted, and remote logging or a managed vector database can leak content even in an otherwise local pipeline.