Designing an AI Content Review Rubric with Quality Scores and Rejection Reasons

A content review rubric is a fixed, written definition of what “good enough to publish” means, expressed as values an editor agent can compute and reasons it can name. Without one, an AI writing pipeline behaves like a coin flip: the same brief produces a publishable draft on Monday and an unusable one on Tuesday, and nobody can explain the difference. This guide is for practitioners who already generate drafts with a model and now need a review step that is consistent, auditable, and cheap to run. It covers what the rubric must output, how to choose and score its axes, how to write rejection reasons a writer can act on, and how to tell whether the rubric is working.

Two outputs: a score and a named reason

The most common design mistake is building a rubric that returns only a verdict. A bare “fail” gives the writer agent nothing to fix and gives the operator nothing to trend. A workable rubric returns two distinct things on every pass:

• A score per axis, usually on a 0–100 scale, so quality can be tracked over time and compared between briefs, models, and prompt versions.
• A named rejection reason when a draft misses a threshold, so the rewrite instruction is specific rather than “make it better.”

Keeping these separate matters because they answer different questions. The score answers “how good is this draft?” The rejection reason answers “what exactly must change before it ships?” A pipeline that only produces scores tends to stall at the human review queue, because a reviewer still has to diagnose the problem by hand.

Choosing the axes

Published rubric designs converge on a small set of independent axes. The Oleno QA framework describes five quality dimensions and notes that each measures a different aspect of quality and requires different validation logic — an important constraint, because overlapping axes double-count the same defect and distort the total score. That source details two of the five:

• Structure (0–100, passing threshold ≥85): heading hierarchy, logical section progression, paragraph length, presence of required sections, and section balance. The stated reason it matters is that structure determines how easily search engines and LLMs can parse and index content.
• Readability (0–100): sentence complexity, paragraph density, transition clarity, use of concrete examples, and avoidance of unexplained jargon.

A second, independently published design uses a six-axis rubric: entity coverage, evidence density, banned words, tone match, structural compliance, and answer-first opening. The overlap with the list above is instructive. Structural compliance and structure measure the same property; readability sits close to answer-first opening and tone match. When merging sources into one rubric, collapse duplicates rather than stacking them, and keep each remaining axis measurable by a single check.

Two practical rules follow. First, mix scored axes with binary gates. Entity coverage, evidence density, and readability are naturally continuous, so 0–100 scores suit them. Banned words, tone match, and answer-first opening are closer to yes/no checks, and forcing a number onto them adds noise. Second, define each axis in terms the reviewer can point at. “Evidence density” is only useful if the rubric states what counts as evidence — a cited source, a measurement, a named example — and what does not.

Thresholds, gates, and what happens on failure

A score without a threshold is decoration. Every axis needs a line that determines pass or fail, and the line should be justified rather than arbitrary. The ≥85 structure threshold in the Oleno framework is a concrete example of a published cutoff. Thresholds for other axes are a local decision: set them from your own archive by scoring drafts you already consider acceptable and drafts you already rejected, then placing the line between the two distributions.

Decide the failure behavior before you need it. A typical loop returns a failed draft to the writer with its rejection reasons attached, allows a bounded number of rewrites, and escalates to a human when the budget is exhausted. The rewrite cap matters: an unbounded loop can burn tokens indefinitely on a draft that is failing for a reason the writer model cannot fix, such as a missing source the brief never supplied.

Writing rejection reasons that a writer can act on

Rejection reasons are the part of the rubric that most often gets under-designed. A good reason is short, stable, and maps to one corrective action. Treat them as an enumerated vocabulary rather than free text, so they can be counted, filtered, and compared across months.

Weak reasons describe a feeling: “reads poorly,” “not engaging,” “quality below standard.” Strong reasons name the defect and its location. Compare:

• THIN_EVIDENCE — the section makes a quantitative claim with no source attached.
• BURIED_ANSWER — the direct answer appears after the second heading instead of in the opening.
• STRUCTURE_VIOLATION — a third-level heading appears without a parent second-level heading.
• BANNED_TERM — the draft contains a word from the project’s exclusion list.
• TONE_DRIFT — the draft shifts into promotional language the brief did not authorize.
• MISSING_ENTITY — a required product, platform, or concept named in the brief never appears.

Each code should carry a one-line repair instruction in the rubric file, so the rewrite prompt is generated rather than written by hand. This is where a rubric stops being a checklist and starts being infrastructure.

An illustrative rubric schema

The following structure is an illustrative design, not a tested configuration. Only the ≥85 structure threshold is drawn from the cited source; the remaining thresholds are placeholders that each team should calibrate against its own accepted and rejected drafts.

{
  "rubric_version": "1.0",
  "axes": [
    { "id": "entity_coverage",       "type": "score", "range": [0, 100], "threshold": 80 },
    { "id": "evidence_density",      "type": "score", "range": [0, 100], "threshold": 75 },
    { "id": "structure",             "type": "score", "range": [0, 100], "threshold": 85 },
    { "id": "readability",           "type": "score", "range": [0, 100], "threshold": 80 },
    { "id": "tone_match",            "type": "gate",  "values": ["pass", "fail"] },
    { "id": "banned_words",          "type": "gate",  "values": ["pass", "fail"] },
    { "id": "answer_first_opening",  "type": "gate",  "values": ["pass", "fail"] }
  ],
  "rejection_reasons": {
    "MISSING_ENTITY":     "Add the required entity named in the brief, with a concrete example.",
    "THIN_EVIDENCE":      "Attach a source, measurement, or named example to the unsupported claim.",
    "STRUCTURE_VIOLATION":"Fix the heading hierarchy so no level is skipped.",
    "BANNED_TERM":        "Remove the flagged term and rewrite the sentence in plain language.",
    "TONE_DRIFT":         "Return to the requested register; remove promotional phrasing.",
    "BURIED_ANSWER":      "Move the direct answer into the opening paragraphs."
  },
  "on_fail": "return_to_writer_with_reasons",
  "max_rewrites": 2,
  "on_max_rewrites": "escalate_to_human"
}

Storing the rubric as data rather than as prose inside a prompt has a direct operational benefit: the version number travels with every logged evaluation, so a change in pass rates can be attributed to a rubric revision rather than guessed at.

Separate the editor model from the writer model

One design decision has an outsized effect on whether the loop catches anything. The editor should not be the same model as the writer. As the eval-loop literature puts it plainly: same model, same blind spots. A writer and an editor sharing weights will agree on the same mistakes, and the review step degrades into a rubber stamp.

This is also where a local deployment can be useful. If drafts contain unpublished client material or internal notes, running the editor model on local hardware keeps the review step inside the same trust boundary as the draft itself, while a hosted model handles generation. The tradeoff is real: local editor models are typically smaller, so the rubric should lean on mechanical checks — heading hierarchy, banned-word lists, entity presence — rather than subtle stylistic judgment. Verify any specific model’s behavior on your own drafts before relying on it as a gate.

First-pass rate is the health metric

The single number that tells you whether the rubric is doing its job is the first-pass rate: the share of drafts that clear every axis without a rewrite. It should start low. In one six-month run of a single pipeline, the first 30 drafts had a 38% first-pass rate; pass rates generally begin in the 30–45% band and climb above 80% as the brief library tightens.

That trajectory is the point. A rubric that passes almost everything on the first attempt is not strict — it is probably not measuring anything. A rubric that passes almost nothing is either miscalibrated or being fed briefs that lack the inputs the writer needs. Watching the rate over time, alongside the distribution of rejection codes, separates the two cases: if most failures cluster on one code, fix the brief template; if failures are spread evenly, the thresholds are likely too tight.

Log every evaluation. Three months of logs become the dataset for the next rubric version, and the rejection-code histogram is usually more informative than any single score.

What the rubric cannot do

A rubric is a filter, not an authority. Several things sit outside its reach and need to be handled explicitly:

• Factual accuracy. A rubric can require that a claim carries a source, but it cannot confirm the source says what the draft claims. That check needs retrieval against a trusted corpus or a human reader.
• Novelty and usefulness. A draft can score well on every axis and still add nothing a reader could not find elsewhere.
• Taste. Voice and judgment resist enumeration. Where they matter, keep a human in the loop rather than pretending a score covers them.
• Gaming. Once a writer model is optimized against a rubric, it will satisfy the letter of the checks. Rotate axes, add new ones, and re-read passing drafts periodically.

For anything published under a real byline, the sensible default is that a passing rubric score moves a draft to “ready for human review,” not to “published.” In a WordPress workflow this maps cleanly onto post statuses: failing drafts return to a revision state with their rejection codes attached, passing drafts enter an editorial queue, and publication remains a human action.

A reasonable next step

Start with four axes and no more: one structural score, one readability score, one evidence check, and a banned-word gate. Write the rejection codes before you write the thresholds, because the codes force you to decide what you actually care about. Run the rubric in shadow mode on twenty drafts you have already published or rejected, and compare its verdicts against your own. Only then wire it into the pipeline as a gate. If you build one, share the axes you chose, the thresholds you calibrated, and the rejection codes that fired most often — the community’s failure histograms are more useful than any single team’s rubric.