AIForum.

How to Benchmark Local Embedding Models for RAG

On this page

THE SHORT ANSWER

Choosing an embedding model for retrieval-augmented generation from a public leaderboard is unreliable, because rankings on generic datasets rarely transfer to your own corpus. This guide lays out a reproducible protocol: freeze the corpus and chunking, build a labeled query set, run every candidate model through the same harness, and compare recall@k, latency, and memory before deciding.
Quick facts

You do not need a leaderboard to choose an embedding model for retrieval-augmented generation (RAG). You need a small, labeled retrieval set built from your own documents, a fixed chunking configuration, and a harness that runs every candidate model through the same queries. The model that wins is the one with the best recall at your chosen cutoff — measured on your corpus, on your hardware, at the latency you can afford.

This guide is for practitioners who already run a RAG pipeline and want to compare local embedding models — sentence-transformers checkpoints, quantized variants, or self-hosted weights — against each other, and possibly against a hosted API model as a reference point. It describes a reproducible protocol. No benchmark numbers are reported here, because none were measured for this article; the goal is to give you a procedure whose output you can trust.

Why a general leaderboard is not enough

MTEB and similar public benchmarks rank embedding models on generic, public datasets. The premise behind tooling such as embench is that this ranking does not transfer cleanly to a specific domain: the model at the top of a public leaderboard is, in the project’s own words, “often not the best one for your domain — your legal contracts, your support tickets, your medical notes, your code” (embench project description).

There are concrete reasons for that gap. Public benchmarks use their own corpora, their own chunk lengths, and their own definition of relevance. Your corpus has different vocabulary, different document lengths, and different queries. A model trained heavily on web text may underperform a smaller model on dense technical prose. Chunking interacts with model behaviour too: a model with a short effective context window can lose information if your chunks are long. None of these effects are visible in a leaderboard score.

What you need before you start

  • A frozen corpus snapshot. Export the documents you will actually retrieve from, and record the date and size. If the corpus changes mid-benchmark, the comparison is void.
  • A chunking configuration you will hold constant. Chunk size, overlap, and the splitter used are part of the experiment, not incidental details.
  • A query set with relevance judgments. A list of realistic queries plus, for each one, the identifiers of the chunks or documents that count as correct answers.
  • A hardware and runtime record. CPU or GPU model, RAM or VRAM, operating system, Python version, and the exact library versions.
  • A decision rule. Decide in advance what “good enough” means — for example, a minimum recall@10 and a maximum p95 latency — so the result tells you something actionable.

The metrics that matter for RAG retrieval

Embedding quality and retrieval quality are not the same thing. Measure retrieval directly:

  • Recall@k — the fraction of queries for which at least one relevant chunk appears in the top k. This is usually the most decision-relevant number, because a retriever that misses the answer cannot be rescued by a better generator.
  • MRR (mean reciprocal rank) — the average of 1/rank of the first relevant result. Useful when position matters, such as when you truncate context aggressively.
  • nDCG@k — a graded measure that rewards putting the most relevant chunks highest. Worth adding when you have multiple relevant chunks per query with different degrees of usefulness.
  • Latency — report embedding throughput (chunks per second) and query latency separately. Index build time and query time have different budgets.
  • Footprint — index size on disk and peak memory during embedding. For local models, this often decides the choice as much as quality does.

Report each metric with the number of queries it was computed over. A recall figure without n is not interpretable.

A reproducible benchmark protocol

1. Freeze the corpus and the chunking

Chunk once, save the chunks with stable IDs, and reuse that same chunk file for every model. If you re-chunk per model, you are measuring chunking as much as embedding. Keep a small smoke subset — a few hundred chunks — for debugging the harness before you run the full set.

2. Build a query set with relevance judgments

This is the part most teams skip, and it is the part that determines whether the benchmark means anything. Two practical routes:

  1. Mine real queries. Pull actual user questions from search logs, support tickets, or internal chat. Have a human mark which chunks answer each one. This is slow but reflects the distribution you care about.
  2. Generate synthetic queries and verify them. Ragas documents a test-generation workflow for exactly this situation: it can “generate a wide variety of challenging test cases from your document corpus” and produce a test set tailored to retrieval and generation tasks (Ragas: Compare Embeddings for retriever). In the documented example, the generator is configured with distributions over three question evolutions — simple, multi_context, and reasoning — so the resulting set is not uniformly easy.

Synthetic queries are a starting point, not ground truth. A generated question can be answerable in ways the generator did not anticipate, and the relevant chunk it implies may be wrong. Spot-check a sample by hand, discard the bad ones, and record how many you discarded. If you mix synthetic and real queries, report them separately at least once.

3. Select candidate models

Keep the candidate list short and defensible. A useful spread for a local RAG pipeline:

  • One small, fast local model as the baseline.
  • One or two larger local models in the same family, to see whether size buys accuracy on your data.
  • If memory is tight, a quantized or distilled variant of a model you already like.
  • One hosted API model as an external reference point, if your privacy and cost constraints allow it.

embench is designed around this mix. Its documented example wraps a local sentence-transformers model and a hosted OpenAI model in the same interface and caches both:

import embench as eb

models = [
    eb.CachedModel(eb.SentenceTransformerModel("all-MiniLM-L6-v2")),
    eb.CachedModel(eb.OpenAIModel("text-embedding-3-small")),
]

retrieval = eb.RetrievalDataset.from_json("my_data.json")

The project description continues from there with a results step; check the repository for the current results API rather than assuming the call signature. Note that sentence-transformers is an optional dependency of embench, listed under the local and all extras, so a local-only install needs that extra.

4. Run the same harness for every model

For each candidate: embed the frozen chunks, build the index, embed the query set, retrieve top k (use the same k for all models, and at least the k your production pipeline uses), and compute the metrics against the same judgments. Cache embeddings so a rerun does not re-embed tens of thousands of chunks — this is what CachedModel is for. Store raw per-query results, not just the aggregate, so you can inspect which queries a model fails and why.

5. Record the environment

Write down: operating system and version; CPU and GPU model; RAM and VRAM; Python version; the versions of the embedding library and the model weights (including a commit hash or revision where the hub exposes one); the pooling and normalization settings; the maximum sequence length; and whether any quantization was applied. A benchmark you cannot reproduce next quarter is a one-off measurement, not evidence.

6. Report variance, not single numbers

Run the evaluation more than once where the pipeline has any nondeterminism (batching, approximate nearest-neighbour indexes, GPU kernels). If two models differ by less than the run-to-run spread, treat them as tied and decide on latency, memory, or licence instead. Report the spread alongside the mean.

Where local embedding benchmarks go wrong

  • Small query sets. A few dozen queries produce noisy estimates; a two-point recall difference may be chance. Be explicit about n and about the confidence you actually have.
  • Tuning on the test set. If you try twenty models and pick the winner on the same queries, you have partly fitted the queries. Hold back a set of queries you never look at until the final decision.
  • Confounding chunking with the model. Re-chunking between runs invalidates the comparison, even if the new chunking is better.
  • Ignoring quantization effects. Quantized weights reduce memory and can change embedding geometry. Measure the quantized model directly instead of assuming the loss is negligible.
  • Comparing latency across different hardware. A local model benchmarked on your laptop is not comparable to a hosted API model measured over the network. State where each measurement was taken.
  • Assuming open weights means unrestricted use. Model licences differ, and open weights are not the same as an open-source or commercial-use licence. Check the licence attached to each checkpoint before it enters a production pipeline.

Turning the result into a decision

Apply your pre-declared rule. If the best local model meets your recall target and your latency and memory budgets, stop there — a hosted model that is marginally better on recall but worse on cost, privacy, or operational complexity is not automatically the right answer. If no local model meets the target, the useful next experiment is usually not “try more models” but “change the chunking” or “add a reranker”, because those address a different failure mode.

Keep the harness. The value of a benchmark is that you can rerun it when you change the corpus, upgrade the model, or move to new hardware. A saved query set, a frozen chunk file, and a recorded environment turn a one-time comparison into a regression test for your retrieval stack.

If you run this protocol, the community is interested in the shape of your result rather than the winner: the corpus type, the query-set size, the hardware, and whether the local model held up. Share your setup and your per-query failures in the forum so others can compare notes.

FAQ

Can I just pick the top model on a public leaderboard?

Not reliably. Public benchmarks rank models on generic datasets, and the embench project description states plainly that the leaderboard leader is often not the best model for your domain, whether that is legal contracts, support tickets, medical notes, or code. Use leaderboards to shortlist candidates, then measure those candidates on your own corpus.

How many queries do I need in my evaluation set?

There is no universal number, and the sources here do not specify one. The practical rule is that small sets produce noisy estimates: if two models differ by less than the run-to-run variation of your harness, treat them as tied and decide on latency, memory, or licence. Report the query count alongside every metric.

What if I have no labeled queries at all?

Ragas documents a synthetic test-generation workflow that builds test cases from your document corpus, including distributions over simple, multi-context, and reasoning question types. Treat generated queries as a draft: spot-check them by hand, discard the bad ones, and record how many you removed.

Should I benchmark the quantized version of a model separately?

Yes. Quantization reduces memory use and can change embedding geometry, so the quantized checkpoint is effectively a different model. Measure it on the same frozen chunks and query set instead of assuming the quality loss is negligible.

References

  1. https://pypi.org/project/embench/
  2. https://docs.ragas.io/en/latest/howtos/applications/compare_embeddings/

Related terms

MTEB
A public benchmark that ranks embedding models on generic datasets; useful for shortlisting, but not a substitute for testing on your own corpus.
Recall@k
The share of queries for which at least one relevant chunk appears among the top k retrieved results.
MRR
Mean reciprocal rank: the average of 1 divided by the rank of the first relevant result across queries.
nDCG@k
Normalised discounted cumulative gain at k; a graded ranking metric that rewards placing the most relevant chunks highest.
Chunking
Splitting documents into embeddable passages. Chunk size and overlap change retrieval results, so they must be held constant during a model comparison.
Quantization
Storing model weights at reduced precision to cut memory use; it can alter embedding geometry, so quantized models should be measured rather than assumed equivalent.

Leave a Reply

Your email address will not be published. Required fields are marked *