How to Document a Prompt Experiment for Other Users

To document a prompt experiment for other users, record everything a second person would need to reproduce the result: the exact prompt revision, the variables and message roles it expects, the model route and settings, the frozen evaluation cases, the scoring rules, and the raw outputs. A saved prompt text is only one input to an experiment — it is not the experiment record.

This guide is written for practitioners who run prompt experiments in a local runtime, a hosted platform, or inside an application, and who need to hand the result to a teammate, a community thread, or their own future self. The emphasis is on records that survive label changes, editor rewrites, and staff turnover.

Why a prompt revision is not an experiment record

Prompt versioning is the practice of retaining identifiable prompt revisions so a team can explain which instructions produced an output, according to the PromptLens guide to prompt versioning. The important detail is that useful versioning includes the execution context that changes behavior, not only the latest text in an editor. A revision history becomes operational evidence when it connects a change to frozen evaluation cases and the exact configuration used to run them.

AWS Prescriptive Guidance makes the same point from a lifecycle perspective: prompts are critical software artifacts that demand a rigorous, disciplined lifecycle management process, and versioning a prompt goes beyond tracking changes to the text. Each version’s identity is defined by its performance, so a version number must be intrinsically linked to the comprehensive evaluation results it produced. When a team discusses a name such as customer-support-summarisation-v2.1.0, they are referring to more than the template — they are also referring to its known set of performance metrics on a specific evaluation dataset.

That is the core distinction. A prompt file answers “what instructions did we write?” An experiment record answers “what did we run, against what, and what happened?”

The fields a reproducible record needs

Based on the practices described in both sources, a handoff-ready record contains the following elements. Not every experiment needs all of them at full depth, but omitting one should be a deliberate choice rather than an oversight.

• Immutable prompt version identifier. Revisions should be immutable, because an incident must still resolve to the same prompt after labels move.
• Message templates and roles. The system and user message templates, recorded as they were executed.
• Variable contract. Variable names and expected types. Variables and message roles are part of behavior, so they belong in the version, not in a separate note.
• Output contract. The shape the prompt is expected to produce.
• Model route and settings. The model route and settings used for the evaluation.
• Evaluation dataset version. The frozen cases the prompt was scored against.
• Evaluator version. The scoring rules or evaluator configuration.
• Raw evidence. Outputs, not just aggregate scores.
• Previous release and decision. What the change replaced and why it was accepted or rejected.

If the application adds retrieval context or tools, retain the relevant application configuration separately — the prompt version alone does not capture that behavior.

Step-by-step: writing the record

1. Freeze the prompt revision before you run anything

Assign a version identifier and stop editing that revision. Treat version identifiers as evidence pointers: a mutable production label tells you where new requests should go, while the resolved immutable version tells you what an individual request actually used. If you keep editing during the run, your results describe a prompt that no longer exists.

2. Version the contract, not just the text

Record the system and user message templates together with variable names, expected types, and the output contract. A change to a variable name or a message role is a behavioral change even when the prose is untouched, so it belongs in the revision history.

3. Pin the execution context

Write down the model route and settings used for the run. A model route is a pointer, and pointers move; the record should name the route as it was resolved at run time, plus any sampling or decoding settings the platform exposes. Where the environment matters — runtime, quantization, context length, hardware — state it, because those details change what a local run produces.

4. Freeze the cases and the scoring rules

A release decision needs pinned cases and scoring rules. Save the exact evaluation cases as a dataset version and save the evaluator configuration as its own version. If you revise a scoring rule after seeing results, that is a new evaluator version, and the earlier comparison must be re-run rather than reinterpreted.

5. Attach the evidence

Store the raw outputs alongside the scores. Aggregate numbers hide the failure that matters: a prompt can improve an average while breaking one specific case. Keeping the outputs also lets another user re-score them under a different evaluator without re-running the model.

6. Record the behavioral diff and the decision

A text diff shows where wording changed; a behavioral review asks which requirement the edit changes. Removing a line such as “escalate active outages” is a small text change and a large behavioral one. Write the behavioral diff in the record, then state the decision — accepted, rejected, or pending — and the reason.

An illustrative experiment record

The following structure is illustrative, adapted from the release record example in the PromptLens guide. Replace the placeholder values with your own identifiers; the field names are a convention, not a standard.

{
  "promptVersion": "support-triage-v4",
  "variables": ["ticket", "policy"],
  "modelRoute": "your-pinned-model-route",
  "datasetVersion": "support-cases-v2",
  "evaluatorVersion": "escalation-check-v1",
  "previousRelease": "support-triage-v3"
}

The record is short on purpose. It is an index into evidence — the full outputs, dataset, and evaluator live in their own versioned stores — but it is enough for another user to find every artifact the experiment depended on.

Where to keep the record

AWS Prescriptive Guidance recommends decoupling prompts from application code and storing them in a centralized prompt store or registry. That can range from a version-controlled Git repository containing YAML or JSON configuration files to a dedicated prompt management tool such as Langfuse. The separation lets prompt engineers, or even non-technical domain experts, update and experiment with prompts without a full application redeployment.

Two documentation habits make a registry usable by other people. First, adopt a structured, consistent labeling convention so the purpose of each prompt is immediately obvious from its name. Second, maintain structured documentation for each version rather than scattering notes across chat threads. A Git repository satisfies both if you commit the record next to the prompt and reference dataset and evaluator versions by tag or commit hash.

Worked example: a support-triage change (illustrative)

Suppose a team edits a support-triage prompt to stop asking the model to escalate active outages, because a separate routing rule now handles them. A handoff note that another user can act on would read roughly as follows.

• Change: removed the escalation instruction for active outages from the system message.
• Behavioral diff: the model no longer produces an escalation flag; downstream routing must supply it.
• Version: support-triage-v4, replacing support-triage-v3.
• Execution context: pinned model route and settings recorded at run time.
• Evidence: dataset support-cases-v2, evaluator escalation-check-v1, raw outputs retained.
• Decision: accepted, with the caveat that any consumer reading the escalation flag must be updated first.

This is an illustrative scenario, not a reported result. Its value is structural: it shows how a wording change becomes a documented behavioral change with evidence attached.

Failure modes that break reproducibility

• Documenting a label instead of a version. A production label is mutable; the resolved immutable version is what a request actually used.
• Omitting the variable contract. A renamed variable or changed message role can alter behavior without any prose change.
• Unpinned model routes. A route that silently updates invalidates the comparison even when the prompt is unchanged.
• Dataset drift. Adding or editing evaluation cases after a run changes the baseline; version the dataset.
• Re-scoring without versioning the evaluator. Old scores and new rules are not comparable.
• Dropping the retrieval or tool configuration. Application configuration that supplies context or tools changes output and belongs in the record, kept separately from the prompt version.

A handoff checklist

Before you publish an experiment to other users, confirm that a second person can answer each of these from the record alone:

1. Which immutable prompt version was executed?
2. What variables, roles, and output contract did it expect?
3. Which model route and settings were resolved at run time?
4. Which dataset version and evaluator version produced the scores?
5. Where are the raw outputs, and can they be re-scored?
6. What behavioral requirement changed, and what was decided?

If any answer is “it’s in the chat history”, the record is not yet portable.

Limitations

The sources describe practices and record structures, not a single universal standard. Field names, storage formats, and labeling conventions differ between Git-based workflows and dedicated prompt platforms, and neither source prescribes a fixed schema. Treat the structure above as a starting convention and verify it against the export and versioning features your own tooling actually provides. Where a platform does not expose an immutable resolved version, record the timestamp, route, and settings manually and note the gap.

Next step

Pick one prompt you have changed recently and write its record from memory. The fields you cannot fill in are the ones your next experiment should capture automatically. If you want a second opinion on your structure, share your prompt version, dataset version, evaluator version, and the environment you ran in with the community — those four identifiers are usually enough for someone else to judge whether your experiment is reproducible.