How to Test AI Integrations: Mocking, Golden Datasets, and Regression Tests

Testing an AI integration is not the same problem as testing a database query or a REST endpoint. A query returns the same rows; an LLM asked to summarise the same paragraph will produce slightly different wording on every call. The practical response is not to skip testing, and not to write brittle assertions on exact strings, but to test in layers: mock the model call for deterministic logic, replay recorded fixtures and a golden dataset for the integration path, score output quality with evaluation tests, and run regression checks as a quality gate before deployment.

This guide is written for developers and automation builders who already have an LLM call somewhere in their stack — a WordPress plugin that drafts content, a Node.js service that classifies support tickets, a local model behind an OpenAI-compatible endpoint — and who now need that call to stop being the untested part of the system.

Why exact-match assertions break on LLM output

Traditional unit tests follow a simple contract: given input X, expect output Y. LLM output breaks that contract in three ways that the testing literature consistently highlights. First, non-determinism: with temperature above zero, identical input can produce different output on every call. Second, semantics instead of syntax: “You have 14 days” and “The return window is two weeks” mean the same thing but share almost no characters, so assertEquals is the wrong tool. Third, the ground can move underneath you — model updates, API version changes, and prompt edits can all alter behaviour without any change to your application code.

Teams tend to fall into one of two traps. The first is not testing at all, on the reasoning that “the output changes.” The second is writing tests that assert on exact strings and then breaking them every time the model hiccups. Both are avoidable. Setting temperature to 0 reduces variance and is a sensible default for test runs, but it is not a guarantee: model updates and infrastructure differences can still shift outputs.

The four test layers and what each one is for

The layered model described in production write-ups separates concerns by what is actually being verified. Each layer is deterministic or non-deterministic in a different way, and each catches a different class of failure.

Keeping the first two layers free of live API calls matters more than it first appears: it means the bulk of your suite runs on every commit, on any machine, without cost or rate limits. The expensive, non-deterministic evaluation layer then runs on a schedule or on pull requests that touch prompts and model configuration.

Layer 1: Mocking the model call

Mocking an LLM does not mean mocking the provider’s SDK internals. It means introducing one narrow seam you own — a client wrapper with a single method — and stubbing that seam in tests. Everything above the seam (prompt construction, response parsing, retry logic, validation) is ordinary deterministic code and should be tested as such.

A minimal Node.js example using Mocha, Chai, and Sinon:

// llmClient.js — the seam you own
async function summarize(client, text) {
  const res = await client.complete({
    model: process.env.MODEL_ID,
    messages: [{ role: 'user', content: `Summarize: ${text}` }],
    temperature: 0
  });
  return res.choices[0].message.content.trim();
}

module.exports = { summarize };

// test/summarize.test.js
const { expect } = require('chai');
const sinon = require('sinon');
const { summarize } = require('../llmClient');

describe('summarize', () => {
  it('passes the prompt through and trims the reply', async () => {
    const client = {
      complete: sinon.stub().resolves({
        choices: [{ message: { content: '  Short summary.  ' } }]
      })
    };

    const out = await summarize(client, 'long text');

    expect(out).to.equal('Short summary.');
    expect(client.complete.firstCall.args[0].temperature).to.equal(0);
  });
});

For code paths that talk HTTP directly rather than through a wrapper, an interception library such as Nock can stand in for the provider endpoint. The tradeoff is the same either way: a mock proves that your plumbing works, and proves nothing about model behaviour. That is exactly what the next layer is for.

Layer 2: Golden datasets and recorded fixtures

A golden dataset is a version-controlled collection of representative inputs paired with the criteria a correct response must satisfy. It is not a list of expected strings. Each case typically carries a schema for the output shape plus semantic requirements — facts that must appear, phrases that must not appear, a maximum length, a required language.

An illustrative fixture shape (not a real recorded record):

{
  "id": "summary-014",
  "prompt_version": "summarize-v3",
  "model": "provider/model-id",
  "recorded_at": "YYYY-MM-DD",
  "input": "...",
  "expected": {
    "schema": { "type": "string", "maxLength": 400 },
    "must_include": ["refund window"],
    "must_not_include": ["I cannot"],
    "language": "en"
  }
}

Two practices keep this layer useful over time. First, record the model identifier and prompt version alongside every fixture, so that when a test fails you can tell whether the prompt changed, the model changed, or both. Second, treat the dataset as a living artefact: when a real bug reaches production, add the input that exposed it. That is how a golden dataset turns into a regression suite rather than a static snapshot.

Contract and schema tests belong here too. Verifying that the response parses as valid JSON, contains the required keys, and respects length or enum constraints catches a large share of integration failures without needing to judge meaning at all.

Layer 3: Evaluation scoring instead of equality

Evaluation tests score output quality rather than asserting equality. A common pattern is LLM-as-judge: a second model call receives the input, the candidate output, and a rubric, and returns a score or a pass/fail verdict against each criterion. The suite then asserts that the aggregate score stays above a threshold you chose deliberately.

This layer is where the honest caveats live. A judge model is itself non-deterministic, may carry its own biases, and costs money on every run. Calibrate it against a small set of human-labelled examples before you trust its threshold, and re-calibrate when you switch judge models. If the judge has never been checked against human judgement, a passing score is not evidence of quality — it is evidence that two models agree.

Layer 4: Regression tests as a CI quality gate

The scenario that motivates the whole structure is the quiet regression: someone changes a prompt from “reply in Polish” to “always respond in the Polish language,” the tests written against the old wording still pass, and three weeks later users report English responses on edge cases after a model update. Regression tests exist to catch that class of change.

A GitHub Actions job can run the layers in cost order, so cheap deterministic tests fail fast and the expensive evaluation layer runs last:

name: llm-tests
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm run test:unit      # mocked, no API key
      - run: npm run test:golden    # replays fixtures, no API key
      - run: npm run test:eval      # scores quality, needs a key
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Where the provider allows it, pin the model version in configuration rather than tracking a floating alias. Pinning does not make the model deterministic, but it makes changes to it deliberate and reviewable, which is what a regression gate needs.

Limitations, cost, and what these tests cannot prove

No combination of these layers proves that an AI feature is correct. Mocked tests only validate your own code. Golden datasets only cover the cases someone thought to include, and they drift out of date as the product changes. Evaluation thresholds are policy choices, not measurements of truth. A passing suite means the behaviour you decided to check has not changed in a way your checks can detect — which is genuinely valuable, and also less than it sounds.

Cost and latency are real constraints. Evaluation runs consume tokens on every execution, so most teams schedule them rather than running them on every commit. Fixture replay is cheap, but fixtures need maintenance: re-record them when prompts or model versions change, and review the diffs rather than accepting them blindly.

A practical starting sequence

If you are adding tests to an existing AI integration, the lowest-effort path with the highest return is roughly this order. Extract a single client seam and write mocked unit tests for prompt assembly and response parsing. Then write ten golden cases covering the happy path, two edge cases, and one historical bug. Add a schema assertion so malformed output fails loudly. Only after those are green in CI should you add a scored evaluation run, and when you do, start with a low threshold you can raise as you calibrate the judge.

Readers building this out are welcome to share their setup in the community: the runtime and model version you are testing against, how many golden cases you started with, and which layer caught your first real regression.