LLM Evals for Product Managers: A Field Guide

Read ten AI-product job descriptions and a pattern jumps out: they ask for PMs who can define evaluations: golden datasets, LLM-as-judge pipelines, quality gates. Not “familiarity with AI.” Evals, specifically. That’s because the eval system is where an AI product’s quality bar actually lives, and someone has to own the bar.

This guide is about which parts of that system the PM should own outright and which can be delegated, and it’s what I wish I’d read before building my own. It’s grounded in a system I operate in production, an eval-gated content pipeline where nothing publishes without passing deterministic gates and an anchored LLM judge, including the time that judge drifted and what it cost to fix. Nothing here is theoretical.

What an eval actually is

Strip the vocabulary away and an eval is two things: a score you can trust, attached to a decision you’ll actually make. Ship or block. Pick config A over config B. Roll back or roll forward. If either half is missing (scores nobody acts on, or decisions made on vibes), you don’t have an eval system yet. You have a dashboard or a habit.

That framing puts the PM at the center, because both halves are product decisions: what “good” means for your users, and what happens at each score.

Start from the decision, not the metric

The most common failure is starting with “what can we measure?” Start instead with: what will we do differently at different quality levels?

In my pipeline that’s explicit publish bands: 85+ ships confidently, 82+ ships, 78+ gets one repair pass, below 78 is rejected. The exact numbers matter less than their existence — they turn a score into a routing rule, and they tell you how much calibration precision you actually need. (If nothing changes between a 74 and an 82, you didn’t need that resolution.)

Decide the actions first. Then design the smallest measurement that reliably picks between them.

Define the dimensions like a product surface

A single “quality” score hides everything you need to know. Break quality into the dimensions your users would recognize (mine include customer value, uniqueness, and readability) and keep the list short enough that every dimension changes a decision. For each one, write down what a 50, a 70, an 85, and a 95 mean, in concrete language. These anchor descriptions are the most underrated artifact in the whole system: they’re what makes scores comparable across time, raters, and models.

Golden sets: your quality bar, made portable

A golden set is a fixed collection of examples with known-correct verdicts: the concrete form of your quality bar. Three practical rules:

Include failures on purpose. A golden set of only good examples teaches your eval nothing about where the floor is.
Cover your slices. If your product spans languages, formats, or user segments, your golden set must too. Aggregate scores hide slice-level collapse, and the slice failing is usually the one your team can’t read.
Freeze it, version it. The set’s whole value is being a fixed reference. When it changes, its version changes, and scores across versions are not comparable.

LLM-as-a-judge: powerful, and guilty until calibrated

Using a model to grade outputs is the only economical way to judge qualities like usefulness at scale. It’s also a model, which means it flatters, it drifts, and it will happily agree with whatever framing your prompt leaks. The design rules I now consider non-negotiable, each learned the hard way:

Score against absolute anchors, never comparatively. My judge prompt explicitly forbids ranking pages within a batch. Comparative judging means a score depends on what else was scored that day.
Pin a rubric version to every score. When you re-anchor the rubric, the version bumps and everything is re-scored. Cross-version comparison is invalid by construction.
Archive before every recalibration. When my judge drifted, the archived snapshot turned “the scores feel generous” into a measurable fact: a mean 10-point inflation on one dimension, which had silently re-ranked my configs and buried a publishable page in the reject band.
Watch for drift as a standing assumption, not an incident. Judges move when models update, prompts change, or context shifts. Locked benchmark examples re-scored on a schedule are the cheap canary.

Blend the judge with signals that can’t drift

Models are bad at counting; code has no taste. So don’t choose. Blend them per dimension. Everything countable (overlap, structure, banned patterns, factual traps) goes to deterministic checks that cost nothing and never drift. Everything that requires reading like a human goes to the judge, constrained as above. And add hard caps for the non-negotiables: in my system, a confirmed factual error caps the total score in the reject band no matter how good the prose is. If a property is non-negotiable, don’t weight it — gate it.

Protect the eval like it’s production code, because it is

The eval system guards your product, so it needs what production needs: regression tests around the pipeline and its tools, so a “small prompt tweak” that breaks a gate turns a test red instead of shipping bad output. My toolchain runs its assertions in CI; the one time I trusted a change without the suite is the origin story of half the suite.

What the PM owns, and what to delegate

You don’t have to write the judge prompt or the CI config. You do have to own the parts that encode a product judgment, because nobody else is positioned to make those calls:

Own the decisions and thresholds. Which actions the eval drives, and where each line sits. Moving a threshold is a product change, and it should be reviewed like one.
Own the definition of good. The dimensions, the anchor descriptions, and the hard caps. If engineering is guessing what an 85 means, the bar belongs to whoever guessed last.
Own the golden set’s coverage. Engineers can build the harness; the PM knows which user slices, languages, and failure cases it has to represent.
Own the re-anchor call. When the judge drifts, deciding to re-anchor and rescore changes every comparison you’ve made. That sign-off sits with the owner of the quality bar.
Delegate the mechanics, review the outputs. Prompt engineering, deterministic checks, and test wiring can sit with the team. Read the run reports anyway. That’s where drift shows up first.

A starting checklist

List the decisions your eval will drive, and the action at each threshold.
Break “good” into 4–7 dimensions your users would recognize; write anchor descriptions for each.
Build a small golden set, including failures, that covers your real slices. Version it.
Stand up deterministic checks for everything countable; add hard caps for non-negotiables.
Add an LLM judge for the judgment dimensions: absolute anchors, independent scoring, pinned rubric versions.
Archive every scoring state; schedule a drift check.
Wire the scores to the actual gates, and only then start tuning.

None of this requires a research background. It requires muscles PMs already have: defining “good,” designing decisions, and protecting a bar under pressure. Point them at a new surface and you have the thing the job descriptions are asking for.

This guide reflects the system documented in the motomotar case study, including its failures. I update it as the system teaches me more.