How to Run a Local RAG Stack on a Single GPU: Hardware Requirements and Bottlenecks

A local retrieval-augmented generation (RAG) stack can run on one GPU, but only after you decide which process owns that GPU. The most predictable arrangement reserves the GPU for generation and keeps document parsing, embeddings, PostgreSQL/pgvector retrieval, and orchestration on the CPU, so generation memory stays available for model weights and KV cache. A second, common arrangement runs embedding and inference together on the GPU through Ollama, which is simpler to operate but caps throughput at the runtime’s thread pool rather than at either model.

This guide is for practitioners who already understand what embeddings and vector search do and now need to size hardware and locate the real bottleneck. It compares the two architectures, maps the constraints that actually limit throughput, and gives a measurement protocol you can run on your own machine.

What the single GPU actually has to hold

VRAM is not a single number to match against a model file. On one GPU you are budgeting for at least four things at once:

• Model weights, at whatever quantization you chose.
• KV cache, which grows with context length, batch size, and the number of concurrent requests.
• Runtime overhead: CUDA context, framework buffers, and any embedding model you place on the GPU.
• Headroom for the largest prompt your retrieval step can produce.

This is why a model that fits in 16 GB can still fail under load. Weight size is a floor, not the total. The KV cache and concurrent sequences are what usually push a working configuration over the edge, and retrieval makes prompts longer by construction.

The architecture described by Stanley Yang avoids most of this arithmetic by moving embeddings off the GPU entirely. In that stack, PostgreSQL with pgvector stores canonical chunk text, source metadata, access fields, and embeddings; an ingestion worker parses, chunks, embeds, and versions data; a query API authenticates, retrieves, builds context, enforces budgets, and validates citations; and a vLLM server owns the GPU and exposes generation only on a protected internal interface.

Two reference architectures for one GPU

Pattern A: the GPU is reserved for generation

The component boundary is explicit and runs in two phases:

offline ingestion
document reader -> normalized sections -> chunks -> CPU embedding -> Postgres + pgvector

online query
authenticated API -> CPU query embedding -> tenant-filtered retrieval
-> prompt with numbered evidence -> vLLM on one GPU -> cited answer + validation

The trade-off is stated plainly by the source: this design favors operational clarity over maximum ingest speed. CPU embedding is slower during bulk ingest, but if you ingest ahead of time, that cost is paid once and never competes with user-facing latency. The GPU stays dedicated to the workload where latency is most visible.

Pattern B: embedding and inference share the GPU

A second documented stack puts Ollama in charge of both embedding and LLM inference, uses LangChain for orchestration and caching, and stores vectors in Chroma, all on a single GPU node behind FastAPI and Nginx. The reported result is up to 500 queries/min on a single NVIDIA T4 with p50 latency around 800 ms, tested with Ollama 0.6.2, LangChain 0.3.14, and Chroma 0.5.23 on an AWS EC2 G4dn.xlarge (NVIDIA T4, 16 GB) with a RHEL 9 base.

The same source names the defining constraint: because embedding and inference share one GPU, throughput is capped by Ollama’s thread pool rather than by either model individually. Treat those figures as a reference point for that specific stack, not as a general guarantee for a T4 or for other models.

Hardware requirements by role, not by tier

There is no universal VRAM number. A 16 GB card is a documented working point for the shared-GPU stack above. For the GPU-reserved pattern, the same card is more comfortable because the embedding model is not competing for it. The honest answer is that you size the GPU around the largest model and longest context you intend to serve, then measure.

The bottleneck map

VRAM capacity limits concurrency and context, not correctness

When VRAM runs out, the failure is usually a rejected request, an out-of-memory error, or a silent drop in batch size, not a wrong answer. Reduce context length, reduce concurrent sequences, lower quantization, or move embeddings to CPU. Keeping embeddings on CPU is effectively a preemptive fix for this class of problem.

Embedding throughput is the ingest bottleneck

In Pattern A, embedding is CPU work, so ingest speed scales with cores and with the embedding model’s size. In Pattern B, embedding and generation contend for the same GPU, so a large ingest job can degrade query latency for everyone. Separate the two workloads in time: run bulk ingestion when interactive traffic is low, or keep embeddings on CPU.

There is also a correctness-adjacent rule that is easy to miss: pin one embedding model for both documents and queries. Mixing models between ingest and query produces vectors in different spaces and silently degrades retrieval quality.

Retrieval latency depends on index design and filtering

Vector search cost depends on the index type, the number of vectors, and how selective your filters are. Tenant filtering is the common trap: a filter that removes most rows still requires the search to consider candidates, and an index tuned for unfiltered search may not help. Requiring tenant-filtered retrieval with stable source IDs is both an authorization control and a performance decision.

Thread pool and orchestration caps

The shared-GPU pattern gives the clearest example: throughput is bounded by Ollama’s thread pool, not by the models. If throughput goes flat while GPU utilization sits below saturation, inspect the serving runtime’s concurrency settings before buying a bigger card.

A reproducible way to find your own bottleneck

What follows is a protocol, not a set of measured results. Run it on your hardware and record your own numbers.

1. Define one fixed workload: a single prompt template, a fixed top-k, a fixed context budget, and 20-50 questions drawn from your real corpus.
2. Record the environment: GPU model and VRAM, CPU, RAM, OS, runtime versions, model name and quantization, context length, and concurrency.
3. Measure retrieval alone. Time the query embedding plus the vector search with the generator disabled. If this is already slow, the GPU is not your problem.
4. Measure generation alone. Send the same prompts with retrieval bypassed and record time-to-first-token and tokens per second.
5. Measure the full path at concurrency 1, then 2, 4, and 8. Watch for the point where p50 latency bends upward.
6. Watch GPU memory and utilization throughout the run.

nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu 
  --format=csv -l 1

Read the results like this: if GPU utilization stays low while latency climbs, the cap is orchestration or CPU work. If memory.used approaches memory.total, you are capacity-bound and should cut context or concurrency. If retrieval time dominates, fix the index or the filter before touching the model.

What a single-GPU RAG stack does not fix

Retrieval supplies current evidence; it does not guarantee a factual answer. The controls that still have to exist are retrieval evaluation, source attribution, abstention, and authorization. Requiring the generator to cite stable chunk IDs makes those controls checkable, because an answer without a valid citation can be rejected automatically.

There is also a model-quality trade-off. Local models trail closed frontier models, and llama3.1:8b is not a drop-in replacement for GPT-4-class quality on nuanced, multi-step reasoning. If your queries require that level of reasoning, a single local GPU may be the wrong constraint, not because the hardware is too small but because the model class is.

Choosing your next step

If ingestion is batched and query latency matters most, start with the GPU-reserved pattern: GPU for generation, CPU for embeddings, pgvector for storage, and citations enforced at the API layer. If your corpus is small and you want the fewest moving parts, the shared-GPU Ollama stack is a documented working configuration, but plan for the thread-pool ceiling and measure before you scale users.

Either way, run the protocol above before you buy hardware. In a single-GPU RAG stack, the bottleneck is frequently not the GPU.

If you have run a similar stack, share your hardware, runtime versions, corpus size, and measured p50 latency in the community so others can compare configurations.