Most local model failures can be diagnosed without ever reading the user’s text. The practical approach is to log structure instead of content: request IDs, model and runtime versions, token counts, latency, retry and fallback counts, error categories, and hashes of prompts and retrieved chunks. That metadata is usually enough to tell you whether a problem lives in retrieval, in a tool schema, in a context limit, or in a model that changed under you.
This guide is for practitioners running self-hosted or locally served models behind a chat feature, a RAG pipeline, or an agent loop, and who need to debug those systems under a privacy constraint. The triage order below follows the general LLM debugging playbook published by Modern AI Guide, combined with a community discussion on logging LLM API calls without storing user prompts. Neither source is specific to one runtime, so treat the tool names as examples rather than requirements.
Start with the symptom, not the prompt
The core argument of the debugging playbook is that most production LLM bugs fall into a small number of recognizable shapes — “JSON breaks intermittently,” “the model picks the wrong tool,” “RAG misses an obvious chunk,” “the agent loops” — and each shape has a near-deterministic shortlist of root causes if you check the right thing first. The same author explicitly warns against “prompt roulette,” the habit of re-running until the output looks acceptable, and against prompt tweaking before reading traces.
That framing matters for privacy work, because the first diagnostic checks rarely require the prompt at all. They require knowing which stage failed.
| Symptom | First check | Likely cause |
|---|---|---|
| JSON breaks intermittently | Is schema enforcement (response_format or equivalent) actually on? | Unconstrained decoding for a structured-output task |
| Model picks the wrong tool | Tool names, descriptions, and parameter schemas | Ambiguous or overlapping function definitions, or a hallucinated function that does not exist |
| RAG misses an obvious chunk | Chunk boundaries, reranker, hybrid search configuration | Retrieval failure, not generation failure |
| Agent runs forever | Whether a max-iterations cap exists | Missing loop termination |
| Quality drops after a model swap | Canary or A/B comparison before full cutover | Model regression |
| Streaming starts but never completes | Completion status and timeout configuration | Truncation or timeout, not a bad answer |
The symptom list and the corresponding fixes — temperature=0 with a fixed seed, a max-iterations cap, chunking and reranking checks, canary deploys — come from the playbook. The table above is a reorganization of that material for triage purposes, not a new set of claims.
The metadata layer that replaces prompt logging
The community discussion on privacy-safe logging makes a point that is easy to miss: a request can technically succeed while still being problematic. Latency degrades, retries happen silently, token usage creeps upward over time, fallback models handle more traffic than expected, streaming stalls, and long prompts fail more often than short ones — none of which shows up in a status code.
The compromise proposed there is to log metadata rather than content. The fields suggested are:
- Model and provider or runtime — so you can separate failures by engine and version.
- Request ID — the join key that lets you correlate a user report with a log line without storing what they typed.
- Feature or product area — so a regression in one surface does not look like a global outage.
- Latency — including time-to-first-token where streaming is involved.
- Input and output token counts — the cheapest early warning for context growth and truncation.
- Retry count — silent retries hide instability.
- Fallback source and target — so a fallback model quietly absorbing traffic becomes visible.
- Error category — a small enumerated set, not a free-text dump.
- Prompt hash — discussed below.
None of these fields reconstruct a user message. Together they answer most of the questions you would otherwise open the prompt to answer: Did this request get long? Did it retry? Did it fall back? Did it finish?
Prompt hashing, done carefully
A hash lets you group identical or near-identical requests without storing their text. It is genuinely useful: two users hitting the same template with the same input produce the same hash, so you can count frequency, correlate with failures, and detect template changes.
Two cautions apply. First, an unsalted hash of a short, predictable prompt is guessable — an attacker with the hash can test candidate strings offline. If the prompt could be short or low-entropy, use a keyed hash (an HMAC with a secret key held outside the log store) rather than a bare digest. Second, hashing the prompt alone is often less informative than hashing the inputs to the pipeline: the template version, the retrieval query, and the identifiers of the retrieved chunks. Hashing retrieved chunk IDs tells you which document was selected without storing the document, and that is frequently the fact you actually need.
Pin randomness before changing anything
The playbook’s advice is to make the system deterministic while investigating: set temperature=0 and fix the seed. This is a debugging discipline, not a fix. It removes sampling variance from the equation so that when a failure does or does not reproduce, you learn something.
One limitation is worth stating plainly: reduced sampling variance is not the same as guaranteed determinism. Batching, kernel selection, quantization, and hardware differences can still produce variation between runs on local hardware. Treat temperature=0 as narrowing the search space, not eliminating it.
Build a reproduction from a synthetic prompt
Once you know the failing stage, reproduce it without the user’s content. Construct a minimal harness that sends a synthetic prompt of comparable length and structure through the same pipeline, with the same retrieval results (referenced by ID), the same tool schemas, and the same sampling settings.
For example, if metadata shows that failures cluster above a certain input-token count, the reproduction does not need the original text — it needs a fixture of that length. If failures cluster on requests where a particular chunk was retrieved, the fixture needs that chunk ID, not the user’s question.
This is where an eval set earns its place. The playbook distinguishes an eval set or regression evals from unit tests: unit tests check that code behaves as written, while regression evals catch quality changes that no assertion would flag. A synthetic eval set built from your own fixtures is the mechanism that lets you confirm a fix without ever retaining production text.
Traces without payloads
A trace is the log of one request — prompt in, model out, tools called. The playbook treats “pull the trace” as the standard first move when something breaks. You can keep the trace and drop the payloads: preserve the span structure, timings, token counts, tool names, whether tool arguments validated against their schema, and retrieval ranks and scores, while replacing prompt and completion text with hashes or redaction markers.
That structure answers the questions that matter during triage. A trace showing three tool calls with invalid arguments points at the schema. A trace showing a retrieval span that returned nothing points at the index. A trace showing a completion that stopped early points at the output limit. In each case, the text is not the evidence — the shape is.
When you do need to see content
There are genuine cases where metadata is insufficient: a safety incident, a user-reported wrong answer that no fixture reproduces, or a suspected prompt-injection path through retrieved documents. In those cases, the practical controls are scope and time, not prohibition.
- Make content capture opt-in per request, not a global switch, and record who enabled it.
- Set a short retention window and delete on a schedule rather than relying on manual cleanup.
- Keep captured content in a separate store from operational logs, with tighter access control.
- Redact known-sensitive spans before storage where the pipeline can identify them.
- Never place raw prompt text inside exception messages or stack traces — those propagate into log aggregators and error trackers that were never designed to hold them.
These are general operational practices rather than claims from the sources above; the sources establish the risk and the metadata alternative, not the specific controls.
Verify the fix with metadata, not vibes
After a change, re-run the synthetic eval set and then watch the same metadata fields you used to find the bug. Retry count, fallback rate, input-token distribution, and the failure rate for long prompts are the signals the community discussion identifies as the ones that reveal a request that “succeeds” while degrading. If those trends return to baseline and the eval set passes, you have evidence. If the eval set passes but fallback rate keeps climbing, you have found a second problem.
The useful next step is small: add the metadata fields to your logging path first, before you need them. A request ID, a prompt hash, a token count, and an error category take little effort to emit and turn an unanswerable “the model is being weird” into a triage you can run in minutes — with the user’s text never leaving the request.
FAQ
Can I debug a RAG miss without reading the retrieved chunks?
Often yes. Log the retrieved chunk identifiers, their ranks and scores, and the retrieval query hash. If the correct chunk never appears in the results, the problem is in chunking, the reranker, or hybrid search configuration rather than in the model's answer.
Does setting temperature=0 make a local model fully deterministic?
It reduces sampling variance and is recommended while investigating, but it is not a guarantee. Batching, kernel selection, quantization, and hardware differences can still cause variation between runs.
Is hashing a prompt enough to protect it?
Not always. Short or predictable prompts can be recovered by testing candidate strings against an unsalted hash. Use a keyed hash with a secret key, or hash pipeline inputs such as the template version and retrieved chunk identifiers instead.
What is the difference between an eval set and unit tests?
Unit tests verify that code behaves as written. Eval sets and regression evals catch changes in model output quality that no code assertion would flag, which is why the debugging playbook treats them as separate tools.