Most AI evals now have a model doing the marking. It is quick and cheap, and almost nobody checks the marker. At Assertif.AI I built an evaluation engine that had a model mark people's work, so this was a question I had already had to answer once. This time I measured it.
Three Claude models answered 120 requests. I picked the better answer in every pair by hand. Then the same three models judged every pair in three prompt formats, both ways round, with one answer padded and without: 4,200 judge calls in all.
What it says
The prompt format mattered more than the model. Giving the judge the task criteria and asking for reasoning before the verdict beat a bare "which is better?" for every judge (Opus 58% to 65%, Sonnet 50% to 56%, Haiku 44% to 56%). Scoring each answer from 1 to 10 on its own and comparing the scores was the worst format for every judge, from 25% to 46%. On short QA it fell to 9% to 18%: two answers scored separately often land on the same number, so a clear winner comes out as a tie. Pointwise scoring is popular because it has no order to be biased by. Here it paid for that by losing the comparison.
The judges pick a winner far more often than I did. I called 72 of 120 pairs about the same - usually two correct answers to a question with one right answer. On those pairs the judges also called a tie only 14% to 46% of the time. Opus with the plain prompt picked a side on 86% of them. Part of that is my rubric prompt, which says to use a tie only if neither answer is better in any way that matters. My guess for the rest is that judges treat small differences in wording as a reason to choose. Either way, a judge that never says "about the same" will make small differences look like real ones.
The judges agree with each other more than with me. Opus and Sonnet with the plain prompt gave the same final verdict on 80% of pairs. Agreement with my labels beyond chance (Cohen's kappa) was 0.01 to 0.19 across the nine setups. Most of that gap is the ties above, so this is less "the judges are wrong" than "the judges and I mean different things by better". That is worth knowing before a judge decides what ships.
No position bias. Every pairwise judge picked whichever answer it saw first between 46% to 50% of the time, against a fair 50% (smallest p = 0.29). Swapping the order changed the verdict about as often as just asking the same question again, and these models no longer accept a temperature setting, so asking again is a real source of noise.
Every judge marked padded answers down hard (-0.93 to -0.81 on a -2 to +2 scale), in every format. There is a catch I built in by accident: every request had a word limit, so padding an answer to 1.7x its length also broke the user's instruction. Marking it down may be the judge doing its job, not a dislike of long answers. A clean test would re-judge the padded pairs with the word limit removed.
Opus favours its own answers. Compared with judges whose model wrote neither answer, Opus leaned towards its own by +0.23 to +0.26 on the same -2 to +2 scale, with every interval clear of zero, in all three formats. Sonnet and Haiku showed nothing similar. If Opus is the judge, do not let it mark Opus.
Agreement with my labels
Accuracy is the share of the 48 pairs I called one way or the other where the judge made the same call. A judge tie, or a verdict that changed when the order was swapped, counts as a miss. With 48 pairs the intervals are wide, so read the pattern rather than the third digit.
claude-opus-5-5claude-sonnet-5-5claude-haiku-5-5By task
| Judge | Short QA (11) | Summaries (16) | Code review (21) | Kappa |
|---|---|---|---|---|
| Opus · Plain | 46% | 63% | 62% | 0.10 |
| Opus · Rubric | 46% | 69% | 71% | 0.12 |
| Opus · Score 1-10 | 18% | 56% | 52% | 0.09 |
| Sonnet · Plain | 55% | 50% | 48% | 0.07 |
| Sonnet · Rubric | 55% | 44% | 67% | 0.14 |
| Sonnet · Score 1-10 | 9% | 44% | 43% | 0.08 |
| Haiku · Plain | 27% | 50% | 48% | 0.12 |
| Haiku · Rubric | 46% | 56% | 62% | 0.19 |
| Haiku · Score 1-10 | 9% | 31% | 29% | 0.01 |
Ties
| Judge | A | B | Tie | Tie where I said tie (72) |
|---|---|---|---|---|
| Opus · Plain | 49 | 58 | 13 | 14% |
| Opus · Rubric | 42 | 58 | 20 | 17% |
| Opus · Score 1-10 | 34 | 46 | 40 | 35% |
| Sonnet · Plain | 45 | 53 | 22 | 21% |
| Sonnet · Rubric | 46 | 48 | 26 | 26% |
| Sonnet · Score 1-10 | 32 | 38 | 50 | 46% |
| Haiku · Plain | 35 | 48 | 37 | 39% |
| Haiku · Rubric | 38 | 46 | 36 | 38% |
| Haiku · Score 1-10 | 38 | 34 | 48 | 44% |
Bias checks
These three do not use my labels at all, so they hold whatever you think of my taste.
Position
| Judge | Picks first | Same both ways | Same, asked again |
|---|---|---|---|
| Opus · Plain | 50% | 92% | 90% |
| Opus · Rubric | 46% | 86% | 78% |
| Sonnet · Plain | 47% | 86% | 90% |
| Sonnet · Rubric | 47% | 83% | 85% |
| Haiku · Plain | 47% | 77% | 85% |
| Haiku · Rubric | 50% | 75% | 70% |
Length (padded answer)
| Judge | Swing to padded | 95% interval |
|---|---|---|
| Opus · Plain | -0.89 | -1.07 to -0.72 |
| Opus · Rubric | -0.85 | -1.00 to -0.68 |
| Opus · Score 1-10 | -0.83 | -0.97 to -0.69 |
| Sonnet · Plain | -0.83 | -1.00 to -0.67 |
| Sonnet · Rubric | -0.90 | -1.06 to -0.75 |
| Sonnet · Score 1-10 | -0.82 | -0.96 to -0.69 |
| Haiku · Plain | -0.87 | -1.02 to -0.72 |
| Haiku · Rubric | -0.81 | -0.96 to -0.67 |
| Haiku · Score 1-10 | -0.93 | -1.07 to -0.80 |
Self-preference
| Judge | Lean to own answer | 95% interval |
|---|---|---|
| Opus · Plain | +0.26 | +0.13 to +0.40 |
| Opus · Rubric | +0.23 | +0.06 to +0.40 |
| Opus · Score 1-10 | +0.25 | +0.10 to +0.40 |
| Sonnet · Plain | -0.16 | -0.28 to -0.04 |
| Sonnet · Rubric | -0.11 | -0.26 to +0.04 |
| Sonnet · Score 1-10 | +0.05 | -0.10 to +0.23 |
| Haiku · Plain | +0.03 | -0.12 to +0.18 |
| Haiku · Rubric | -0.04 | -0.21 to +0.13 |
| Haiku · Score 1-10 | -0.05 | -0.24 to +0.14 |
How it was set up
120 requests, 40 each of short factual questions, summaries for a busy manager (with traps in the dates and conditions) and code reviews. Every request has a reference answer that I saw while labelling and the judges never did. Each request was answered by two of claude-opus-5-5, claude-sonnet-5-5, claude-haiku-5-5, rotating so every matchup comes up equally often, and which answer is A is a seeded coin flip.
Three judge formats. Plain: which is better, answer A, B or TIE. Rubric: the task's criteria, a note that order and length do not matter, reasoning first, then the verdict. Score: each answer scored 1 to 10 on its own, higher score wins. Pairwise judges saw every pair both ways round, and a verdict only counts if it holds both ways. A second copy of 40 pairs was asked again in the same order, to measure how much changes by chance. For the length test, one answer per pair was padded by Haiku to about twice the length with nothing new added (0 of 20 spot-checked added content). Judges ran at API defaults, which is how most people use them.
How I labelled, and the limits
The labelling page is blind: no model names, and left or right is a coin flip per pair. I labelled in two passes. The first I threw out: I did not yet have a clear sense of what I was judging and hardly used "about the same", and those labels matched the judges' consensus no better than chance. For the second pass I knew the questions and used another chat to work out what to look for in each type of pair, then made every call myself. Only the second pass is used here.
I re-labelled 20 pairs blind with the sides swapped and made the same call 100% of the time. That flatters me: by then I had seen every pair twice, and 11 of the 20 were ties. One labeller is one person's taste. Three models from one family judging each other is a narrow panel. And 48 decided pairs is a small sample, which is why every number has an interval.