The Day My LLM Judge Drifted — and How I Caught It

In June 2026, the LLM judge that scores my content pipeline got generous. Not broken, generous. Nothing errored, no test failed, and every score it produced looked plausible on its own. That’s what makes judge drift dangerous: it’s a quality problem that presents as good news.

If you own an AI feature whose quality gate is an LLM judge, this essay is about one decision: how much to trust that judge’s scores, and what has to be in place before you let them route real outcomes. Mine route everything. I run an eval-gated content operation where a judge scores every page on a seven-dimension rubric before anything ships, and the score picks the action: 85+ publishes with confidence, 82+ publishes, 78+ gets one repair pass, and below that a page is rejected. When the judge inflates, bad pages ship. When it deflates, good pages die in review. Either way, the leaderboard I use to choose between models and prompts (28 pages across 12 configurations at the time) quietly stops meaning anything.

What drift actually looked like

There was no alarm. What I noticed, reading the run reports, was that one dimension, uniqueness, had crept upward across recent batches. Pages that I knew shared structure with their siblings were scoring in the high 70s and 80s on the one dimension designed to punish exactly that.

That fit a suspicion I already had. Uniqueness is the most relative of my seven dimensions. A human reviewer judges it by comparison (“this reads like the other city pages”), and a model judge slides the same way unless it’s pinned to absolute anchors. The dimensions grounded in checkable facts held steady. The one that needed an absolute reference frame was the one that walked.

The fix: freeze, archive, re-anchor, rescore

The repair took four steps, and the order matters:

Freeze and archive. Before touching anything, I archived the entire leaderboard state into a timestamped directory. If you rescore without archiving, you haven’t corrected drift. You’ve changed your mood and deleted the evidence.
Re-anchor the rubric. I tightened the anchor descriptions, the fixed definitions of what a 50, 70, 85, and 95 mean on each dimension, and bumped the rubric version.
Rescore everything. Unchanged pages are normally never re-judged, so the version bump is what forces a full pass under the new rubric.
Compare. The archive made the drift measurable instead of anecdotal.

The before/after data

Six pages existed in both snapshots. Here’s what re-anchoring did to them:

Page · config	Old total	New total	Uniqueness before → after
kolkata · claudev3_xhigh	87.6	85.7	88 → 78
pune · codexv3	84.8	81.6	78 → 66
gulbarga · cowork4.7	83.7	85.0	79 → 71
indore · cowork4.7	83.1	83.0	78 → 67
jamnagar · cowork4.6	82.0	85.5	78 → 68
bhubaneswar · cowork4.6	78.6	84.5	70 → 64

The signature is unambiguous: uniqueness dropped on all six pages, by between 6 and 12 points and a mean of −10.2, while the grounded dimensions barely moved. The old judge had been inflating precisely the dimension that most needed an absolute anchor.

And the totals didn’t just shift; they re-ranked. The #1 page changed. One page jumped from the reject band (78.6) to a publishable 84.5, which means the drifted judge hadn’t only been letting weak pages through. It had been burying a good one. Drift doesn’t just soften your gate; it scrambles every comparison you made while it was active.

What I changed about the system, not just the scores

Judge scores are only comparable under a pinned rubric version. Every score in my leaderboard carries the rubric version that produced it, and cross-version comparisons are treated as invalid by construction.
Absolute anchors, scored independently. The judge prompt forbids comparing or ranking pages within a batch. Each page is scored alone against fixed anchor descriptions, so a score means the same thing regardless of what else was scored that day.
Deterministic where possible. Drift can’t touch code. The dimensions that can be computed are blended in as deterministic signals alongside the judge, and template similarity is scored by code alone.
Archive before every rescore. Storage is free. Unrecoverable calibration history is not.

One thing I’ll be honest about: my rubric has a designed mechanism for locked benchmark pages, frozen reference pages with pinned scores that would act as canaries and flag drift automatically instead of waiting for me to smell it. The mechanism is built. I haven’t populated it yet. This incident is the argument for finishing that work, and it’s next on the list.

What this means if you’re the PM

Every team running LLM-as-a-judge evaluation will meet some version of this. The judge is a model, and models move: a provider update, a prompt tweak, a change in context. Before I let a judge’s score gate anything now, I want four things in place:

A rubric version on every score, so everyone knows which numbers can be compared.
An archive of every scoring state, so “the scores feel generous” can become a measurement instead of an argument.
A drift check that doesn’t depend on someone noticing. Locked reference examples rescored on a schedule are the cheapest canary.
A named owner for the bar. Someone decides when the judge gets re-anchored and signs off on the rescore. That call belongs to whoever owns the product’s quality, not to whoever last edited the prompt.

Without those, you won’t detect drift when it happens. You’ll slowly stop being able to trust your own dashboard, and you won’t know when the trust expired. The broader lesson I took: an eval system is a product surface, not a script. It needs versioning, regression protection, and an audit trail, exactly like the product it guards. I’ve collected the full set of practices in LLM evals for product managers.

The numbers above come from my pipeline’s own archived leaderboard snapshots. The full system is documented in the motomotar case study.