← articles

Image judging, version 2: four evaluators on the same pictures

Thirty-one pictures for eight new stories, each scored by four LLM families’ models playing four prompt roles, with the evidence behind every score. A second version of the image model evaluation gallery

Every number on this page comes from the run folders named in Method; nothing is quoted from memory.

What this page is

The first version of this gallery compared image models on about a thousand pictures with one AI judge, and its most quoted result, a twenty-point gain that turned out to live in a step production already ran, came from comparing every change against an unchanged baseline. This version keeps the baseline habit and changes the judge. Thirty-one pictures for eight new stories, one story per grade from 1 to 8, were drawn by Stable Image Core 1.1 from prompts the product's own prompt builder wrote for each paragraph, and every set was scored by four evaluators from four LLM families, each playing four prompt roles in turn, against the illustration rubric with a quoted piece of evidence behind every score.

On the illustration rubric, three of the four evaluators failed every set of pictures and the fourth passed 13% of them. The scale scores agree more than the pass rates do: Nova Pro's mean picture scale is 3.1, the other three sit between 2.2 and 2.4 on a 4-point scale, and Krippendorff's alpha across the four on the per-story picture scale is 0.54 (bootstrap 0.38 to 0.64), better than the text scales managed and below the 0.67 that Krippendorff treats as the floor for even tentative conclusions. By the same marginal-means comparison as the quiz page, the picture score differed by 0.88 points between evaluators within a role (0.67 to 1.16) and by 0.23 between roles within a evaluator (0.18 to 0.39). The pictures are on the page so every score can be checked against the thing it describes.

How to read this page

Gallery shows every picture next to the paragraph it was drawn for, and under each set a table with every evaluator's score on I1 to I9 and its evidence on hover. Results is the roll-up. Method names the models, the prompts and the run folders. Limits says what thirty-one pictures can and cannot support.

What changed since the first version

One vision judge became four.
The first gallery's judge was Claude Opus 4.7 with vision. This version seats Opus 4.7, GPT-6 Sol, Nova Pro and Gemini 3.8 Flash, and all four receive the pictures as JPEG bytes with the paragraph each one belongs to. Whether four vision models read a picture with comparable acuity is not something this run tests; a evaluator that misses small text in an image, or sees an object that is not there, fails the rubric in its own way, and the evidence column is where that shows. The reason is the same as on the quiz page: one judge has one set of blind spots, and a judge from the same LLM family as the generator has a conflict nobody can see from inside (Verga et al. 2024; Panickssery, Bowman and Feng 2024). Here the generator is an image model from a fifth LLM family, so the same-family question falls on the stories and quizzes rather than the pictures; the Anthropic seat is dashed on the talk slide for the text, not for the pictures.
Rubric v2 for illustrations.
Nine items. Two gates (I1, I2) and two checks (I3, I4) scored per image, with the evaluator required to name the failing image numbers; five scales (I5 to I9) scored on the set. A fail or a score under 3 must name the image, so a verdict can be traced to a picture a person can look at.
Roles and the Latin square.
Same four roles, same rotation, same four runs as the quiz page, because the pictures were scored in the same calls as the text and quiz for each story. A children's book editor and an assessment item writer do not look at a picture the same way; the square exists so that difference can be measured rather than assumed. On this batch the role moved picture scores less than the model did.
Prompts from the product's builder, not hand-written.
The first gallery's experiments varied the prompt by hand to find what the image model could and could not do. These pictures come from the harness's port of the production image-prompt builder, fed by the same prompt-block rows the product reads from its database, for stories the product's next text model wrote. They are what the production pipeline produces, not what a prompt engineer can produce with a hand-tuned prompt.
A refused prompt is recorded, not dropped.
One of the thirty-two prompts was refused by the image model's own filter. It is shown as a gap in the gallery and recorded as a note on every judgment of that story, not scored and not removed from the count, because a failure that leaves the average quietly improves it.

Method

Material. Eight stories, grades 1 to 8, generated in one batch (run folder haiku-batch-2026-10-06T19-16-45). Text by Claude Haiku 4.5 on Bedrock through the ported production paragraph prompt. One image prompt per paragraph from the ported production builder (image-prompt-builder.mjs), with the image_prompt_text_blocks rows read from the dev database. Pictures by Stable Image Core 1.1 (stability.stable-image-core-v1:1, Bedrock, us-west-2), one per paragraph, seed derived from the story id and page so a re-run reproduces the picture. 32 prompts, 31 pictures.

Evaluators, roles, design, counts and spend are the same run as the quiz page and are described there: four evaluators, four roles rotated by Latin square over four runs, 123 judgments written, 120 clean, three grade 8 cells lost to expired session keys, five Gemini cells unwritten. The pictures were sent to each evaluator as JPEG bytes at their generated size, 1536 by 1536, with the story text and the paragraph each picture belongs to. On this page they are shown at 640 pixels to keep the file under five megabytes.

Agreement. Krippendorff's alpha (interval) over the per-story, per-evaluator mean picture scale is 0.54, against 0.45 on the text scales. Computed on the cells that exist; the missing ones are left missing.

Roll-up. A seat's picture scale is the mean of I5 to I9. The illustration verdict is the rubric's own rule over the gates and checks. The grade 8 story's fourth picture is absent for every evaluator, so its I1 to I4 are scored on three pictures and say so in the evidence.

Rubric items scored on every set of pictures

itemwhat it asks
I1Species and type
I2Safe for the grade
I3Readable image
I4Same character throughout
I5Match to the paragraph
I6Descriptor coverage
I7Style fidelity
I8Focus
I9Representation

Results

The gates are where the pictures fail. The scale items average between 2.2 and 3.1 across evaluators, which is "partly" territory, but the per-image gates (the right characters, no invented text, the paragraph's moment) fail on at least one of four pictures in almost every set, and a set with one failed gate fails the rubric. Nova Pro is the generous evaluator on pictures as it is on text: a 3.1 mean against 2.2 to 2.4, and the only seat to pass any set. Whether that is leniency or a different reading of the anchors is a question for the human scores and for the Gallery tab, where each of its passes can be looked at.

By model

seatjudgmentspictures passpicture scalestext passtext scales
Anthropic Claude Opus 4.732 (2 err)0%2.2462%3.19
OpenAI GPT-6 Sol320%2.400%2.98
Amazon Nova Pro31 (1 err)13%3.1294%3.63
Google Gemini 3.8 Flash280%2.4354%3.60

By role

seatjudgmentspictures passpicture scalestext passtext scales
children's book editor32 (1 err)3%2.5656%3.33
educational-software reviewer284%2.5350%3.26
literacy teacher32 (1 err)0%2.4547%3.32
assessment item writer31 (1 err)7%2.6555%3.42

Every cell of the square

seatjudgmentspictures passpicture scalestext passtext scales
Anthropic Claude Opus 4.7 as children's book editor8 (1 err)0%2.3462%3.12
Anthropic Claude Opus 4.7 as educational-software reviewer80%2.2362%3.19
Anthropic Claude Opus 4.7 as literacy teacher80%2.0862%3.17
Anthropic Claude Opus 4.7 as assessment item writer8 (1 err)0%2.3462%3.27
OpenAI GPT-6 Sol as children's book editor80%2.330%2.90
OpenAI GPT-6 Sol as educational-software reviewer80%2.420%2.96
OpenAI GPT-6 Sol as literacy teacher80%2.350%2.98
OpenAI GPT-6 Sol as assessment item writer80%2.500%3.08
Amazon Nova Pro as children's book editor812%3.08100%3.62
Amazon Nova Pro as educational-software reviewer812%2.9888%3.54
Amazon Nova Pro as literacy teacher8 (1 err)0%3.1488%3.67
Amazon Nova Pro as assessment item writer729%3.31100%3.69
Google Gemini 3.8 Flash as children's book editor80%2.4862%3.65
Google Gemini 3.8 Flash as educational-software reviewer40%2.4550%3.46
Google Gemini 3.8 Flash as literacy teacher80%2.3338%3.52
Google Gemini 3.8 Flash as assessment item writer80%2.4862%3.69

Limits

Sources