A local retrieval-augmented generation (RAG) stack is three cooperating parts: an ingestion pipeline that parses documents into chunks and converts them into vectors, a vector database that stores and searches those vectors, and a local model server that supplies both the embedding model and the answering model. Two documented reference builds illustrate the pattern — a LlamaIndex cookbook that pairs ChromaDB with Ollama, and a Node.js project that pairs Qdrant with Ollama. Neither requires an API key: embedding generation, vector search, and answer generation all run on the machine.
This article explains what each component does, walks through the documented setup of both stacks, and covers the failure modes that most often make a local RAG pipeline return something useless.
The three planes of a local RAG stack
Running a model locally is not the same as running RAG locally. A plain local model answers from its training weights. A RAG stack adds a retrieval plane in front of it, so the answer is grounded in documents the model never saw during training.
- Ingest plane. Documents are read, parsed into text, split into chunks, converted into vectors by an embedding model, and written to a vector store together with metadata such as the source file.
- Retrieval plane. At query time the question is embedded with the same embedding model, the vector store returns the nearest chunks, and those chunks become the context.
- Generation plane. The retrieved chunks plus the question are passed to the local language model, which produces the answer.
The division matters because each plane fails in a different way. When a local RAG system gives a wrong or empty answer, the cause is more often retrieval than the model. The LlamaIndex cookbook makes this explicit by including a dedicated evaluation step and a set of named failure modes rather than treating the pipeline as finished once it returns text.
Component 1: Ollama as the local model server
Ollama fills two distinct roles in a RAG stack, and conflating them is a common source of confusion. It serves the embedding model that turns text into vectors, and it serves the generation model that writes the final answer. These are separate models with separate jobs.
The LlamaIndex cookbook uses nomic-embed-text for embeddings and llama3.2:3b for generation. Its prerequisites are that Ollama is installed and running, and that both models have been pulled:
ollama pull llama3.2:3b
ollama pull nomic-embed-textThe Node.js Qdrant project uses the same embedding model, nomic-embed-text, but a different generator: llama3.1, run at zero temperature so that answers are deterministic. Deterministic generation is a deliberate choice when citation discipline matters, because a model that paraphrases freely is harder to audit against the retrieved sources.
One practical constraint follows from this split: the embedding model determines the dimensionality and meaning of every vector in the store. It must be identical at ingest time and at query time. Swapping the embedding model later does not upgrade the index — it invalidates it, and the corpus must be re-embedded.
Component 2: the vector database
The vector store holds each chunk’s embedding alongside its text and metadata. Given a query vector, it returns the nearest chunks, which become the model’s context. The two documented builds make different architectural choices here.
The LlamaIndex cookbook uses ChromaDB as an embedded, locally persisted store. It is created and used inside the same process as the pipeline, which keeps the setup small: there is no separate service to start, and persistence is a local directory.
The Node.js project uses Qdrant, run as a service (Docker in that project), with cosine similarity search. A separate service adds a moving part but also decouples storage from the application process, which is useful when several tools need to read the same index or when the corpus outgrows a single script.
The tradeoff is not about which database is better in the abstract. An embedded store is the shortest path to a working prototype; a served store is the shorter path to a shared, long-lived index. Both approaches are documented as fully local, with no external API calls at ingest or query time.
Component 3: document ingestion
Ingestion is where local RAG stacks are usually won or lost, because it determines what the retriever can ever find. Three decisions matter most.
Parsing and format coverage
The Node.js project routes mixed formats through a parser router: a text and code reader for .js/.jsx, .yml, and .md, officeparser for .docx and .pptx, and pdf-parse for .pdf. The LlamaIndex cookbook takes a simpler route, ingesting a sample document with a file reader and then splitting it. Format coverage is a real constraint: a PDF that parses into empty or garbled text produces chunks that no query will ever match, and the failure is silent unless retrieval is measured.
Chunking
The cookbook uses SentenceSplitter from LlamaIndex to chunk the source document. The Node.js project uses a block-aware chunker of roughly 6000 characters, which keeps related blocks together rather than cutting at an arbitrary offset. Chunk size is a balance: chunks that are too small lose the surrounding context the model needs to interpret them, while chunks that are too large dilute the embedding so that a precise query matches weakly.
Metadata, attribution, and injection framing
Both builds treat the source as part of the answer. The Node.js project produces source-tagged, citation-enforced answers, and wraps retrieved documents in XML tags before they reach the model — a prompt-injection resistant framing that keeps ingested content visually separate from instructions. It also ingests from a manifest, so the knowledge base is a controlled, predictable list rather than whatever happens to be in a folder.
A documented minimal pipeline: LlamaIndex + ChromaDB + Ollama
The LlamaIndex cookbook is the smallest complete reference for this stack. It is published as a notebook that runs entirely on the local machine, with no API key. The documented flow has six steps: configure the pipeline, ingest and chunk a sample document with SentenceSplitter, embed the chunks with OllamaEmbedding and persist them in ChromaDB, query the index with a local LLM, evaluate retrieval quality against a gold question-and-answer set, and explore failure modes including empty context, long queries, and a hallucination guard.
Its dependency list is installed in one command:
pip install -q llama-index-core llama-index-llms-ollama llama-index-embeddings-ollama llama-index-vector-stores-chroma llama-index-readers-file chromadbThe notebook’s import block establishes the components directly:
import chromadb
from llama_index.core import Document, StorageContext, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import OllamaThe Chroma vector store binding is then imported from llama_index.vector_stores.chroma. From there the pattern is the general one: build Document objects, split them into nodes, embed the nodes, hand them to a VectorStoreIndex backed by the Chroma store, and query that index with the Ollama LLM. This section describes the published notebook; it has not been independently re-run for this article.
A multi-format variant: Node.js + Qdrant + Ollama
The Qdrant project covers the case where the corpus is not a single clean text file. It is manifest-driven, exposes a two-command CLI (ingest and ask), and enforces citations in the answer. Its most notable behavioural choice is a hard refusal: when the retrieved context does not contain the answer, the system says so instead of improvising. For a local knowledge base used for factual questions, that refusal is a feature, not a limitation — it is the difference between a system that is wrong quietly and one that is wrong loudly.
Where local RAG stacks break
The cookbook names three failure modes worth internalising because they generalise beyond that notebook.
- Empty context. The retriever returns nothing usable, and the model either answers from its weights or produces something vague. Check the retriever’s output before reading the answer.
- Long queries. Verbose, multi-part questions embed poorly and match weakly. Splitting a compound question into single-intent queries usually improves recall more than changing the model.
- Hallucination with thin context. When retrieved chunks are only tangentially related, the model fills the gap. A refusal rule or a hallucination guard is the mitigation.
Two further failure modes are worth adding from the ingestion side. A silent parse failure produces chunks of noise, which looks like a retrieval problem but is a parsing problem. And an embedding-model change at query time, without re-embedding the corpus, produces a store whose vectors no longer share a space with the query vectors — search results degrade without any error being raised.
Choosing between ChromaDB and Qdrant
Neither documented build declares a universal winner, and the choice follows from the situation. ChromaDB fits a single-process prototype, a personal document set, and a pipeline that should start with one command. Qdrant fits a corpus that several tools read, a workflow that already runs Docker, and a knowledge base that is expected to persist and grow. Both are documented as fully local. If the corpus is small and the goal is to learn the pattern, start embedded; move to a served store when sharing or scale becomes the binding constraint.
Verify retrieval before blaming the model
The most useful habit in this stack is measuring retrieval separately from generation. The LlamaIndex cookbook does this with a gold question-and-answer set and two metrics: hit-rate, which asks whether the correct source chunk appeared in the retrieved results at all, and mean reciprocal rank (MRR), which rewards ranking the correct chunk near the top. A pipeline with a high hit-rate and poor answers has a generation problem. A pipeline with a low hit-rate has an ingestion or embedding problem, and no amount of prompt tuning will fix it.
A reasonable next step is to build the smallest possible version of this stack: pull nomic-embed-text and a small generator through Ollama, ingest a handful of documents you can verify by hand, and run the retrieval evaluation before adding more formats or a larger corpus. Once hit-rate is acceptable, the remaining work — citation enforcement, refusal behaviour, and multi-format parsing — becomes a series of incremental additions rather than a debugging exercise.
If you build this stack and hit an unexpected error, share your environment, the models you pulled, and the retriever’s output in the community so others can compare notes.
FAQ
Do I need a GPU to run a local RAG stack?
The supplied sources do not specify hardware requirements. The LlamaIndex cookbook's default generation model is llama3.2:3b, which is small, but actual memory use also depends on context length, KV cache, and runtime overhead rather than model weights alone. Treat any hardware estimate as something to verify on your own machine.
Can I use a vector database other than ChromaDB or Qdrant?
Yes. The two documented builds simply show two different architectures: an embedded store (ChromaDB, persisted locally inside the pipeline process) and a served store (Qdrant, run separately with cosine similarity search). The rest of the pipeline — embedding model, chunking, retrieval, generation — stays the same.
Why does my local RAG system refuse to answer?
The Node.js Qdrant project deliberately refuses when the retrieved context does not contain the answer, rather than improvising. If refusals are frequent rather than occasional, the likely cause is retrieval: check whether the correct chunk is being returned at all before adjusting the prompt.
What is the fastest way to tell whether retrieval or generation is the problem?
Evaluate retrieval on its own. The LlamaIndex cookbook measures hit-rate and MRR against a gold question-and-answer set. A low hit-rate points to ingestion, chunking, or embedding; a high hit-rate with poor answers points to the generation step.