LLM-as-a-Judge vs Deterministic Heuristics: A Guide

Q: What is LLM-as-a-judge and when should you use it?

LLM-as-a-judge is the practice of using a language model to grade another model's output against a rubric — scoring faithfulness to sources, helpfulness, tone, reasoning quality, or comparing two answers pairwise. It exists because those qualities cannot be expressed as code assertions, and human grading does not scale past a few hundred examples. Use it exactly there: for the semantic remainder that survives your deterministic checks. Everything code can express — schema validity, required fields, citation presence and resolution, exact-fact matches, executable tests, latency budgets, banned-content patterns — should be checked by code, which costs microseconds, behaves identically on every run, and never hallucinates a grade. The judge then grades what assertions cannot see, under three conditions that separate evaluation from decoration: a rubric with binary or three-point verdicts anchored by examples (not 1-to-10 scales nobody can agree on), a judge model from a different family than the candidate (self-preference bias is documented and systematic), and measured agreement with human labels — Cohen's kappa against 100 to 200 human-graded examples, at or above roughly 0.6, before the judge's scores gate anything. The original MT-Bench research showed strong judges can agree with humans about as often as humans agree with each other; the caveat teams skip is that this agreement must be measured in your domain with your rubric, not assumed because the judge is a frontier model.

Q: What are the main biases in LLM-as-a-judge evaluation?

Four dominate, and all are systematic rather than random, which means they skew every score in the same direction rather than averaging out. Position bias: in pairwise comparisons, judges prefer whichever answer appears first, so any A/B evaluation run in one ordering is measuring position as much as quality; the mitigation is running both orderings and averaging, flagging cases where the verdict flips. Verbosity bias: judges reward longer answers — research across frontier judges measured 15 to 30 points of inflated preference for verbose output — so a model that got wordier will look like a model that got better; mitigate with length normalization and explicit rubric language that longer is not better. Self-preference bias: judges favor text generated by their own model family, which is why the convenient choice (judging with the same model your app already uses) is the biased choice; the cardinal rule is a different family for judge and candidate, and a small cross-family panel for anything load-bearing. Leniency drift: judge scores inflate slowly over time as prompts, models, and traffic shift under a fixed judge configuration, producing dashboards that improve while user experience does not; the mitigation is monthly recalibration against a fixed human-labeled set, plus recalibration on every judge-model or rubric change. None of these are exotic — they appear in essentially every production judge deployment we have audited, and none of them are visible without a calibration measurement.

Q: How do you calibrate an LLM judge against human labels?

With a fixed procedure that takes about a day and invalidates more dashboards than any other single exercise. Sample 100 to 200 real outputs from your system, stratified across the traffic you care about — not cherry-picked successes. Have humans grade them against the exact rubric the judge will use, with the same verdict options; if two humans disagree substantially on what the rubric means, fix the rubric before involving the judge, because a rubric humans cannot apply consistently is not going to be applied consistently by a model. Run the judge on the same set and measure agreement with Cohen's kappa, which corrects for chance agreement — raw percent agreement flatters heavily imbalanced label distributions. A kappa at or above roughly 0.6 is a defensible floor for letting the judge gate releases; below that, the judge's verdicts are too noisy to act on. When kappa is low, the fix is almost never a larger judge model: tighten the rubric language, split a mushy dimension into two precise ones, add few-shot pass and fail examples that anchor each verdict, or switch to binary from a scale. Then make calibration a ritual rather than an event — recheck monthly and on every judge or rubric change, because leniency drift means the judge that agreed with humans in March is quietly more generous by July, and the drift reads as product improvement on every dashboard that nobody validated.

Q: What should be checked with deterministic assertions instead of a judge?

Everything that has a right answer code can verify, which in most production systems is close to half of what teams currently pay judges to grade. The catalogue: structural validity (JSON parses, schema conformance, required fields present and typed correctly); referential checks (the answer contains a citation, the citation resolves to a real document, the linked ID exists); factual lookups with ground truth (dates, prices, entity names checked against the authoritative store); executable verification (generated SQL runs against the schema, generated code compiles and passes its tests — the strongest oracle available anywhere in LLM evaluation); safety patterns (no PII regexes match, no banned competitor names, refusal template present when policy required refusal); and operational budgets (latency, token count, cost per request). These assertions cost microseconds, run on every request rather than a sample, produce identical verdicts on run one and run one million, and never suffer position bias or leniency drift. The design consequence is the cascade: assertions run first and fail fast for free, which means the judge never wastes a token on output that a schema check would have rejected, and the judge's budget concentrates on the genuinely semantic questions — faithfulness, completeness, tone — where it is the only scalable option. Paying a judge to confirm that JSON parses is the purest form of burning money to add noise.

Q: How much does LLM-as-a-judge evaluation cost compared to deterministic checks?

The gap is roughly four orders of magnitude, which is why architecture matters more than model choice. Deterministic assertions cost effectively zero: they run in microseconds on your own compute, on every request in production and every case in CI, with no per-call fee. A judge call at around 1,000 tokens per evaluation lands in the range of a few tenths of a cent to a cent per verdict on frontier models — trivial for a single case, meaningful at volume. Concretely: a 300-case golden set graded by a single-dimension judge costs roughly $1 to $3 per full run and a few minutes of wall clock, which is cheap enough to run on every pull request and every prompt change without thinking about it. The place costs explode is production: judging every live request at even modest scale — say a million requests a month across three rubric dimensions — is three million judge calls and a five-figure annual line item for scores nobody reads individually. The standard design is sampling: judge 1 to 5 percent of production traffic, which preserves statistical visibility into quality trends at a fiftieth of the cost, and spend a fraction of the savings on the thing that actually improves trust — a larger, fresher human-labeled calibration set. Multi-dimension judging multiplies cost linearly (three dimensions is three calls), which is another argument for keeping every dimension that can be an assertion out of the judge's workload entirely.

Q: Why should the judge be a different model than the one being evaluated?

Because self-preference bias is documented, systematic, and invisible without a controlled measurement. Research on judge behavior shows models rate text generated by their own family measurably higher than equivalent text from other families — the judge recognizes, at the level of style and token distribution, output that resembles its own, and grades it up. The practical trap is that violating this rule is the path of least resistance: the model already integrated into your application is the cheapest and easiest thing to also use as the grader, so the default architecture in most teams has the candidate grading itself, wearing a rubric as a costume. The consequences compound at exactly the wrong moment: when you run a model comparison to decide whether to migrate — the highest-stakes use of evaluation — a same-family judge tilts the comparison toward its relatives, and the bias is largest precisely where the true quality gap is smallest, which is where decisions are hardest. The mitigations, in increasing order of rigor: use a judge from a different model family than any candidate being graded; for pairwise comparisons, run both orderings to also neutralize position bias; for load-bearing decisions like model migrations or vendor selection, use a small panel of judges across two or three families and treat disagreement between them as a signal to escalate that case to a human rather than averaging it away. And regardless of judge choice, the calibration measurement against human labels remains the only proof that any judge, from any family, is measuring what you think it measures.

Q: How do you write a good rubric for an LLM judge?

Write it like a spec for a junior grader who will take every word literally, because that is what a judge is. The patterns that survive production use: binary or three-point verdicts instead of numeric scales — pass/fail or pass/borderline/fail — because neither humans nor judges apply a 1-to-10 scale consistently, and a score of 7 carries no actionable meaning; one dimension per judge call, because a single "rate the overall quality" prompt collapses faithfulness, completeness, and tone into a mush that cannot be debugged — separate calls let you see that faithfulness dropped while tone held; reasoning before verdict, instructing the judge to explain its assessment before emitting the grade, which both improves verdict quality and gives you an audit trail for disagreements; and few-shot anchors — two or three worked examples of each verdict, drawn from your real traffic, so "fail" is defined by demonstration rather than adjective. Add explicit counter-bias language where the known biases bite: state that longer answers are not better, that confident tone does not indicate correctness, that refusing to answer is correct when the source material lacks the answer. Then validate the rubric on humans first: if two human graders disagree on what it means, the judge will too, and no amount of prompt polish fixes a dimension that was never precisely defined. The rubric is a versioned artifact — change it and every historical score becomes incomparable, so recalibrate on every edit and keep the old version pinned to old results.

Q: Where do judges and assertions fit in a CI/CD pipeline for LLM features?

In a cascade with different jobs at different stages, and the discipline is keeping each grader where its properties fit. On every pull request that touches a prompt, a retrieval parameter, or a model version: the deterministic assertion suite runs first as a hard gate — schema checks, executable tests, safety patterns over the full golden set, in seconds, for free — and any failure blocks the merge with a verdict that never flakes. Behind it, the judge pass runs on the same golden set for the semantic dimensions, a dollar or three per run, producing scores that gate the merge only if the judge's kappa against human labels has been measured and holds above the floor; an uncalibrated judge in CI is a random merge-blocker that teaches engineers to ignore the gate. On deploy, canary rollouts meter the change onto a traffic slice while sampled judge scores and assertion failure rates are compared against the baseline — a drop triggers rollback before full exposure. In steady-state production, assertions run on every request (they are effectively free), judges on a 1-to-5 percent sample feeding quality dashboards and drift alarms, and humans grade the monthly calibration set plus any case where panel judges disagree. The failure mode to design against is inversion: teams that put the judge first and assertions nowhere end up with expensive, flaky gates and no cheap, exact ones — the precise opposite of what each grader is good at.