live
motomotar: an eval-gated AI content operation
The product decisions behind a one-person AI content business: who it serves, where the quality bar sits, what I killed, and the honest numbers.
The user and the job
An Indian car buyer close to a purchase has one question the brochure won’t answer: what will this car actually cost me, in my city? The ex-showroom price is only the start. Road tax changes by state, insurance changes by zone, and the dealer’s quote is hard to check. motomotar.com exists to answer that question with numbers a buyer can trust, plus the buying guides and calculators around it.
I’ve run it since January 2025, alone, on evenings and weekends, next to a full-time product job. A conventional operation of this size needs writers, editors and fact-checkers. I have a pipeline and a quality bar.
The bet
AI can produce pages at this volume; that was never in doubt. My bet was narrower: that I could build a system strict enough about what is allowed to publish that a buyer never has to wonder whether a price was made up. If the gate held, scale would be cheap. If it didn’t, every extra page would be a liability with my name on it.
Early generation showed me what failure looked like, and each failure became a requirement:
- Confident nonsense. Plausible prices, tax rates and specs that were simply wrong.
- Template smell. Hundreds of city pages that were obviously one page with the city name swapped.
- AI dialect. Stock phrasing that readers, and increasingly search engines, discount on sight.
- Silent drift. Quality sliding across hundreds of pages with no single moment where anything broke.
The quality bar is a product decision
Every page is scored on a seven-dimension rubric (customer value, uniqueness, Google-perceived quality, India relevance, SEO quality, human readability, template similarity) that rolls up to a 0–100 total. The total matters only because it maps to an action:
| Score | What happens |
|---|---|
| 85+ | Publish, confident |
| 82+ | Publish, medium confidence |
| 78+ | One repair pass |
| Below 78 | Reject |
I set those lines by asking what each kind of mistake costs. A false publish puts a wrong price or a templated page in front of a buyer under my name, and on a young domain every thin page makes the whole site look less worth crawling. A false reject costs a repair pass and a day. The costs aren’t symmetric, so the bands aren’t either. The publish line sits high, and the band just below it gets a second chance instead of a hard cut, so a nearly-good page is repaired rather than thrown away.
Some properties aren’t traded off at all. A confirmed factual error caps the total deep in the reject band, however well the page reads, and a near-duplicate of a sibling page hits a ceiling on uniqueness. If a property is non-negotiable, I gate it instead of weighting it.
Each dimension blends a deterministic signal with an LLM judge’s read. Template similarity is scored by code alone, uniqueness is split 50/50, and SEO quality leans 60/40 toward the judge, because models are good at judgment and bad at counting (the full design). The judge scores each page alone against fixed anchor descriptions, never ranking within a batch, so a score means the same thing whatever else ran that day.
What I chose not to automate
The cheapest way to make an LLM pipeline trustworthy is to keep it away from the jobs it is worst at. Automation handles research, grounded drafting, a humanization pass, and QA gates that block on structure, factual traps, banned phrases and link rules. These stayed out of its reach:
- Live numbers never come from generation. City fuel prices, refreshed daily for 658 cities, and the 36-state road-tax dataset behind the calculators come from maintained data pipelines. The model writes around the numbers. It never produces them.
- Publishing is my action. The pipeline stops at a WordPress draft. After I publish, the live page is re-fetched and checked against what was supposed to ship, because “it published” and “it published correctly” are different claims.
- Local knowledge is human-approved. Locality notes on dealer pages go live only after I approve them.
- Editorial bets stay with me. What to build next, which clusters to pursue, when to noindex. When I noindexed ~139 orphaned news posts, the system supplied the evidence and I made the call.
Choosing a config with evidence, not taste
I compare generation configs (models, prompts and pipelines) on a leaderboard of 28 pages across 12 configs, with 13 dated run reports. The decision metric is the rubric total under a pinned rubric version, because that is the number the publish gate acts on. A config that reads beautifully in a demo but clears the publish line less often is not the better config.
The day the judge drifted
In June 2026 the scores got generous, most visibly on uniqueness. I froze the state, archived the entire leaderboard, re-locked the rubric’s anchor definitions under a new version, and rescored everything from scratch. On the six pages present in both snapshots, uniqueness dropped on every one (mean −10.2 points) while the grounded dimensions held. Rankings inverted: the #1 page changed, and one page moved from the reject band to publish-medium.
That last result is the product finding. The drifted judge hadn’t only been letting weak pages through; it had been burying a good one, so the cost landed on both sides of the gate. Since then, scores are comparable only under a pinned rubric version, and nothing is rescored without an archive. The incident and its before/after table are in The day my LLM judge drifted.
Keeping the bar from eroding
Quality rules rot when they live in two places. The banned-phrase list lives in one file; the draft QA gate and the post-publication verifier both load it, and a regression test asserts that they do, so nobody can relax one side and silently fork the standard. The humanization stage keeps its own machine-checked list of 228 phrases across 11 categories.
The keyword-research toolchain runs 145 assertions per green run in CI, and the full local suite across price pipelines, tax math and page rendering is 3,858 assertions green before a pipeline change ships. When a data source or prompt changes, a red test tells me before a wrong price reaches a live page.
What I killed
The question-page cluster. I built pages targeting “People also ask” questions, on the theory that low-competition questions would earn topical authority and pass it to the price pages. Google deindexed the cluster wholesale. Over the same months, all nine state road-tax rule pages indexed at positions 6–12. My read: on a young domain, a page with a hard data payload earns a crawl, and a page answering a question that’s answered everywhere else doesn’t. I stopped adding to the cluster.
The keyword ladder. Through late summer, new content went through a strict program: candidate terms passed a series of vetting gates, launched one cluster at a time, and waited weeks for a ranking verdict before the next cluster could start. It did its job, and the state rule pages that indexed came out of it. But a one-cluster-at-a-time gate was the wrong shape for scaling what it had found. I closed the program in October 2026, kept its verification steps, and dropped the selection gates.
Where it stands
From Search Console, March to August 2026: 204,649 impressions across 17,160 distinct queries, with 4,278 queries and 86 pages at a weighted average position of 10 or better, including top-two positions on real buyer queries. Clicks were still early, at 338 in that window. The site is ~1,040 URLs: 618 on-road model×city pages across six car models, 104 city hubs, 62 long-form articles published from a 75-article topical map, and 135 city-page rewrites pushed through a QA-tracked batch process.
The honest bottleneck isn’t content quality any more. It’s indexation. In August only 49 of the 618 on-road pages were earning impressions. A controlled request-indexing test indexed 16 of 20 treated pages while a 10-page holdout stayed at zero.
Update, October 2026. Indexation moved, slowly: 89 of the 618 on-road pages are now indexed (14%). The site also started carrying ads in October, so there is finally a revenue line to set against the token line.
I’m publishing these numbers because a case study that only contains victories is marketing. The system produces publishable pages at one-person cost. Getting a young domain crawled at scale is a different problem, and it isn’t solved yet.
Decisions I’d defend
- Set the bar by the cost of each mistake. A false publish costs trust; a false reject costs a day.
- Numbers come from data pipelines, never from the model.
- The publish button stays human. The pipeline’s last step is a draft.
- Pin a rubric version to every score. Scores across versions are never compared.
- Kill on evidence, keep the method. The question pages went when Google’s verdict came in; the ladder went once it had shown what worked.
What I’d do differently
- Start with the data-heavy pages. The pages that index carry a payload nobody else has. I’d build the state-level road-tax layer before the city grid, not after it.
- Lock calibration anchors before scaling the judge. The rubric supports frozen reference pages that would flag drift automatically. The mechanism is built, but I haven’t locked a benchmark set yet. June is the argument for finishing it, and it’s next.
- Keep a cost ledger from day one. I track tokens, not yet a per-page cost model against a human-writer baseline, so “one-person cost” is a claim I can describe but can’t yet price. That analysis is coming to this page.
This case study is written from the system’s own records: manifests, changelogs, test suites, and a local Search Console mirror. Every number above is traceable to a file.