A blind A/B test compares two AI tools on tasks you actually do, with every output stripped of the clues that reveal which tool made it. You score the work first and unblind the labels afterwards. The method is cheap to run and unusually informative: in one published example, a model that looked like the obvious winner on reputation scored lower than its rival while costing far more for the same run.
This guide is for practitioners who already work with at least two models or tools and want a defensible reason to prefer one on their own jobs. It covers test design, blind scoring, cost tracking, and the failure modes that quietly invalidate results. Nothing here requires a research budget; the discipline sits in the blinding and the record-keeping.
What a blind A/B test can and cannot tell you
A blind A/B test measures quality and preference on a small, specific sample of your work. It answers questions such as: which model needs fewer edits on my client emails, which one produces a usable landing-page draft, and which one costs less per finished job.
It does not produce a universal ranking. A single run of a few tasks cannot establish statistical significance, and results decay as vendors ship new versions. Treat the output as a decision aid for the next few weeks, not a permanent verdict.
The blinding matters because model names carry expectations. The LAB 0017 field notes from Lenny’s AI Builders describe the author’s guess that Claude Haiku 5.5 would be better than GPT-6 Luna but much more expensive. The blind scores contradicted that guess: on his own benchmark the model cost more and did not score better. That is the kind of gap only a blind test surfaces, because a labelled comparison invites you to rationalise the answer you expected. LAB 0017 field notes
Prerequisites and what to prepare
- Two tools or models you can run on the same inputs. These can be hosted APIs, chat interfaces, or local runtimes such as Ollama or llama.cpp.
- Three to seven tasks you genuinely perform. The AI Workflows guide on blind taste testing recommends identifying 3-7 core tasks, for example drafting documents, coding, summarising notes, or triaging an inbox. AI Workflows blind taste test
- A scoring sheet — a spreadsheet with one row per output and one column per criterion.
- A cost log that records cost per job, not just cost per token.
- A label holder — a colleague, a script, or a file you do not open until scoring is finished.
Step-by-step procedure
1. Define the benchmark tasks
Choose jobs that appear in a normal week and that you can judge without outside expertise. LAB 0017 puts it plainly: pick three jobs you really do. Keep the set small enough to finish in one sitting, because fatigue changes how you score.
2. Freeze one brief per task
Write the prompt, the input material, and the constraints once, then reuse that exact text for both tools. Save each brief to a file so it cannot drift between runs. If you find yourself editing the prompt for the second tool, you are no longer running a comparison.
3. Run both tools under comparable settings
Match reasoning effort, temperature, context length, and tool access. In the LAB 0017 run, both models ran at extra high for four hours. If one tool runs at maximum effort and the other at a default, you are testing settings rather than models, and the result will not transfer to your daily use.
4. Strip identifying metadata and label the outputs
Paste each result into plain text, remove tool-specific formatting, signatures, and any phrase that gives away the source, then assign labels A and B. Flip a coin for the mapping and store it in a separate file. If your tasks involve WordPress content, keep drafts on a staging site and clear plugin or theme markup before scoring, since generated wrappers are a common source of accidental identification.
5. Score blind, before you look at the labels
Rate every output for one task in a single pass. The AI Workflows guide suggests a personal vibe scale from 1 to 5 with notes on style, structure, and quality; a slightly more structured rubric works well too:
- Correctness — factual errors, broken code, invented details.
- Brief adherence — did it do what was asked, at the requested length and format?
- Completeness — what is missing that you would have to add?
- Edit effort — how many minutes until the output is usable?
- Style — voice, structure, and readability for your audience.
Score all criteria for every output before unblinding anything. Partial unblinding is the most common way a careful test turns into a confirmation exercise.
6. Record cost per job
Log what each run actually consumed: input and output tokens, subscription-equivalent usage, or local runtime plus hardware time. A model can be cheap per token and hungry per task, so the useful figure is the cost of producing one finished job. The LAB 0017 notes make this point directly and pair each score with its run cost.
7. Unblind and aggregate
Open the mapping and add the model names to your sheet. Compute an average per criterion and per task, then look for patterns. A model that wins on summarising may lose on code, and a model that wins on average may lose on the two tasks you care about most.
8. Assign models per task
The AI Workflows guide recommends ending with an assignment rather than a single winner: give specific models to specific tasks in your daily workflow. This is also the most durable output of the test, because it survives a new model release better than a global ranking does.
A worked example: BuilderBench in LAB 0017
The clearest published illustration of the method is BuilderBench, the author’s own ten-task benchmark covering writing, a landing page, a lead magnet, video editing, motion, thumbnails and titles, browser and computer use, research, automation, and founder conversations. It was run on 8 October 2026, with both models at extra high for four hours.
| Model | BuilderBench score (out of 100) | Estimated run cost |
|---|---|---|
| GPT-6 Luna | 44.58 | $1.47 |
| Claude Haiku 5.5 | 39.18 | $37.39 |
Luna led by 5.4 points while costing dramatically less, and the author states plainly that the result made him appreciate GPT-6 Luna far more than he had. Two caveats belong with those numbers. First, they are cost estimates, not bills. Second, they come from a single run on one person’s task set: the same notes record that, over time on that benchmark, the best model has been Opus 5.5. A blind test tells you what happened on your tasks in one window, not which model is best in general.
Reproducibility: what to write down
A result you cannot repeat is an anecdote. Record at least the following alongside the scores:
- Model names and versions, plus the date of the run.
- Reasoning effort, temperature, and sampling settings.
- Context length and any system prompt used.
- For local models: runtime, hardware, and quantisation. Memory use is not just the weight file — the KV cache and runtime overhead matter, so a model that fits on paper can still fail with a long context.
- Number of repetitions per task. One run per task is a starting point, not a measurement.
- Cost basis and the assumptions behind any estimate.
If you automate the harness, have the script write each output to a neutrally named file and keep the mapping in a separate directory. That single design choice removes almost all opportunity for accidental unblinding.
Common failure modes
- Label leakage. Tool-specific phrasing, formatting quirks, or timestamps survive the copy-paste and give the game away.
- Prompt drift. The second run gets a slightly improved brief, so the comparison measures your prompt writing.
- Scoring after a peek. Even a glance at the mapping biases the remaining ratings.
- Length bias. Longer answers often feel more thorough; score edit effort and correctness, not volume.
- Mismatched effort. Comparing a maximum-effort run against a default one.
- Ignoring edit time. A cheap model that needs twenty minutes of cleanup is not cheap.
- Over-reading one run. Small samples produce noisy averages; repeat before you change your whole workflow.
Limitations and when to repeat the test
Blind scoring measures your judgement, and your judgement is a legitimate criterion — but it is personal. A colleague running the same test may prefer the other output, which is why the AI Workflows method includes notes on what you liked and disliked rather than a bare number. Re-run the test when a model version changes, when your task mix shifts, or when your cost basis changes, such as moving a workload from a hosted API to a local runtime. For local deployments, remember that the tradeoff is different in kind: you are exchanging per-token spend for hardware, electricity, and maintenance time, and those costs belong in the same column as the API bills.
A useful next action
Pick three jobs you will do this week, freeze one brief for each, and run both tools before you decide anything. Score the anonymous outputs first, then unblind and write the cost per job next to each score. If the result surprises you, that is the point of the exercise. Readers who want to compare notes can share their task set, environment, and outcome in the community discussion.
FAQ
How many tasks do I need before the result means anything?
Three to seven tasks is the practical starting point recommended by the AI Workflows guide, and LAB 0017 suggests picking three jobs you really do. That is enough to guide a near-term decision but not enough to establish statistical significance, so repeat the test before making a large workflow change.
Should both tools run at the same reasoning effort?
Yes. If one tool runs at maximum effort and the other at a default, the comparison measures settings rather than models. In the LAB 0017 run, both models ran at extra high for four hours.
Can I include a local model in the same test?
Yes, and it is often worth doing. Record the runtime, hardware, quantisation, and context length alongside the score, and account for memory beyond the weight file, since the KV cache and runtime overhead also consume memory. Local runs trade per-token spend for hardware, electricity, and maintenance time.
Why not just check public leaderboards?
Leaderboards measure other people's tasks. A model can be cheap per token and hungry per task, and the LAB 0017 result is a concrete case where the expected winner cost more and scored lower on the author's own benchmark. Your own task set is the only sample that matches your work.