· updated
Hybrid LLM Evaluation: Let Code Count, Let Models Judge
Which parts of 'good' to give code and which to an LLM judge: per-dimension weights, hard caps, and publish bands as product decisions.
Ask an LLM to count how much two pages overlap and it will flatter you; models are famously bad at counting. Ask a regex whether a paragraph would actually help a reader and you’ll get silence, because code has no taste. Every hybrid LLM evaluation system I’ve built comes down to one design question: which failures do you want caught by arithmetic, and which by judgment?
That question lands on the PM’s desk whether or not anyone says so, because the answer defines what “good” means for the product. I run a content pipeline where nothing publishes without a quality score, and the score that gates publication is neither a pure LLM verdict nor a pure metric. It’s a per-dimension blend, and the blend ratios are product decisions I own.
The split of labor
My rubric scores seven dimensions. For each one, the final score is a weighted mix of a deterministic signal (computed by code) and an LLM judge’s assessment:
| Dimension | Deterministic | LLM judge | Why this ratio |
|---|---|---|---|
| Template similarity | 100% | 0% | Pure counting: n-gram overlap against sibling pages. A model has no business here. |
| Human readability | 60% | 40% | Sentence mechanics are measurable; whether prose breathes is judgment. |
| Uniqueness | 50% | 50% | Overlap is countable, but “says something its siblings don’t” is a judgment call. |
| India relevance | 50% | 50% | Lexicon checks catch the surface; the judge catches whether it’s used meaningfully. |
| SEO quality | 40% | 60% | Structure is checkable; search intent satisfaction isn’t. |
(The remaining dimensions, customer value and perceived quality, lean on the judge, because no metric I trust exists for “would a reader be glad they landed here.”)
The principle behind every ratio: push weight toward code exactly as far as a trustworthy metric exists, and not one step further. A deterministic signal can’t drift, can’t be sweet-talked, and costs nothing to re-run. My judge, meanwhile, has drifted before. But a deterministic signal also can’t tell you a page is useless. Both failure modes are real, and the blend insures against each with the other.
Caps: where blending stops and rules take over
Weighted averages have a flaw: a page can be bad in one non-negotiable way and average its way to a pass. So on top of the blend, the system has hard caps that no blend can override:
- A confirmed factual error caps the total score at 60, deep in the reject band, no matter how well the page reads. Facts aren’t a dimension to trade off; they’re a precondition.
- Near-duplicate similarity (≥ 0.85 against a sibling) forces ceilings on uniqueness and perceived quality. Past a threshold, “how well-written is this copy of another page” is not a question worth scoring.
If a property of the output is non-negotiable, don’t weight it. Gate it.
Scores are routing decisions, not grades
The blended score exists to make a decision, and the decision boundaries are explicit: 85+ publishes with confidence, 82+ publishes, 78+ earns one repair pass, and below 78 the page is rejected. Pages scoring near a boundary get routed to the judge for a closer look rather than auto-decided by the deterministic layer, because that’s where a wrong call is most likely and most expensive.
A score nobody acts on is a vibe. The bands are what turn evaluation into a quality gate, and they’re also what make calibration matter, because a two-point drift near a boundary changes real publish decisions.
How to decide what goes where
The heuristic I use, which generalizes past content:
Give code everything that is countable, thresholdable, or regression-testable. Overlap ratios, banned phrases, structural requirements, factual-trap scans. Code is incorruptible and free.
Give the model everything that requires reading as a human reads. Usefulness, register, whether an “explanation” explains. Then constrain the model hard: absolute anchor definitions, independent scoring (never comparative within a batch), and a pinned rubric version on every score.
Give neither side final authority. The blend catches what each misses; the caps catch what neither should be allowed to forgive.
What this means if you’re the PM
The ratios, caps, and bands look like implementation details an engineer can pick. They aren’t. Each one is a product decision with consequences you can measure, and each needs an owner:
- A ratio is a statement of trust. Moving weight toward the judge on a dimension says you trust its taste more than your metric there. Write down why, version it, and revisit it when either side changes.
- The cap list is your list of things the product must never do. If the team can’t write that list, it hasn’t agreed on the bar yet, and no amount of weighting will cover the gap.
- Calibration effort belongs near the lines. Precision in the middle of a band changes nothing. Precision at 77.9 versus 78.1 decides what ships.
A hybrid evaluator is a small product with real users: the people and pipelines that act on its scores. Design it deliberately, version it, and keep an audit trail. For the wider practice (golden sets, judge calibration, regression protection), see LLM evals for product managers.
The weights, caps, and bands above are the live configuration of my production pipeline, documented in the motomotar case study.