Image judging, version 2: four evaluators on the same pictures
Thirty-one pictures for eight new stories, each scored by four LLM families’ models playing four prompt roles, with the evidence behind every score. A second version of the image model evaluation gallery
Every number on this page comes from the run folders named in Method; nothing is quoted from memory.
What this page is
The first version of this gallery compared image models on about a thousand pictures with one AI judge, and its most quoted result, a twenty-point gain that turned out to live in a step production already ran, came from comparing every change against an unchanged baseline. This version keeps the baseline habit and changes the judge. Thirty-one pictures for eight new stories, one story per grade from 1 to 8, were drawn by Stable Image Core 1.1 from prompts the product's own prompt builder wrote for each paragraph, and every set was scored by four evaluators from four LLM families, each playing four prompt roles in turn, against the illustration rubric with a quoted piece of evidence behind every score.
On the illustration rubric, three of the four evaluators failed every set of pictures and the fourth passed 13% of them. The scale scores agree more than the pass rates do: Nova Pro's mean picture scale is 3.1, the other three sit between 2.2 and 2.4 on a 4-point scale, and Krippendorff's alpha across the four on the per-story picture scale is 0.54 (bootstrap 0.38 to 0.64), better than the text scales managed and below the 0.67 that Krippendorff treats as the floor for even tentative conclusions. By the same marginal-means comparison as the quiz page, the picture score differed by 0.88 points between evaluators within a role (0.67 to 1.16) and by 0.23 between roles within a evaluator (0.18 to 0.39). The pictures are on the page so every score can be checked against the thing it describes.
How to read this page
Gallery shows every picture next to the paragraph it was drawn for, and under each set a table with every evaluator's score on I1 to I9 and its evidence on hover. Results is the roll-up. Method names the models, the prompts and the run folders. Limits says what thirty-one pictures can and cannot support.
What changed since the first version
One vision judge became four.
The first gallery's judge was Claude Opus 4.7 with vision. This version seats Opus 4.7, GPT-6 Sol, Nova Pro and Gemini 3.8 Flash, and all four receive the pictures as JPEG bytes with the paragraph each one belongs to. Whether four vision models read a picture with comparable acuity is not something this run tests; a evaluator that misses small text in an image, or sees an object that is not there, fails the rubric in its own way, and the evidence column is where that shows. The reason is the same as on the quiz page: one judge has one set of blind spots, and a judge from the same LLM family as the generator has a conflict nobody can see from inside (Verga et al. 2024; Panickssery, Bowman and Feng 2024). Here the generator is an image model from a fifth LLM family, so the same-family question falls on the stories and quizzes rather than the pictures; the Anthropic seat is dashed on the talk slide for the text, not for the pictures.
Rubric v2 for illustrations.
Nine items. Two gates (I1, I2) and two checks (I3, I4) scored per image, with the evaluator required to name the failing image numbers; five scales (I5 to I9) scored on the set. A fail or a score under 3 must name the image, so a verdict can be traced to a picture a person can look at.
Roles and the Latin square.
Same four roles, same rotation, same four runs as the quiz page, because the pictures were scored in the same calls as the text and quiz for each story. A children's book editor and an assessment item writer do not look at a picture the same way; the square exists so that difference can be measured rather than assumed. On this batch the role moved picture scores less than the model did.
Prompts from the product's builder, not hand-written.
The first gallery's experiments varied the prompt by hand to find what the image model could and could not do. These pictures come from the harness's port of the production image-prompt builder, fed by the same prompt-block rows the product reads from its database, for stories the product's next text model wrote. They are what the production pipeline produces, not what a prompt engineer can produce with a hand-tuned prompt.
A refused prompt is recorded, not dropped.
One of the thirty-two prompts was refused by the image model's own filter. It is shown as a gap in the gallery and recorded as a note on every judgment of that story, not scored and not removed from the count, because a failure that leaves the average quietly improves it.
Method
Material. Eight stories, grades 1 to 8, generated in one batch (run folder haiku-batch-2026-10-06T19-16-45). Text by Claude Haiku 4.5 on Bedrock through the ported production paragraph prompt. One image prompt per paragraph from the ported production builder (image-prompt-builder.mjs), with the image_prompt_text_blocks rows read from the dev database. Pictures by Stable Image Core 1.1 (stability.stable-image-core-v1:1, Bedrock, us-west-2), one per paragraph, seed derived from the story id and page so a re-run reproduces the picture. 32 prompts, 31 pictures.
Evaluators, roles, design, counts and spend are the same run as the quiz page and are described there: four evaluators, four roles rotated by Latin square over four runs, 123 judgments written, 120 clean, three grade 8 cells lost to expired session keys, five Gemini cells unwritten. The pictures were sent to each evaluator as JPEG bytes at their generated size, 1536 by 1536, with the story text and the paragraph each picture belongs to. On this page they are shown at 640 pixels to keep the file under five megabytes.
Agreement. Krippendorff's alpha (interval) over the per-story, per-evaluator mean picture scale is 0.54, against 0.45 on the text scales. Computed on the cells that exist; the missing ones are left missing.
Roll-up. A seat's picture scale is the mean of I5 to I9. The illustration verdict is the rubric's own rule over the gates and checks. The grade 8 story's fourth picture is absent for every evaluator, so its I1 to I4 are scored on three pictures and say so in the evidence.
Rubric items scored on every set of pictures
item
what it asks
I1
Species and type
I2
Safe for the grade
I3
Readable image
I4
Same character throughout
I5
Match to the paragraph
I6
Descriptor coverage
I7
Style fidelity
I8
Focus
I9
Representation
Results
The gates are where the pictures fail. The scale items average between 2.2 and 3.1 across evaluators, which is "partly" territory, but the per-image gates (the right characters, no invented text, the paragraph's moment) fail on at least one of four pictures in almost every set, and a set with one failed gate fails the rubric. Nova Pro is the generous evaluator on pictures as it is on text: a 3.1 mean against 2.2 to 2.4, and the only seat to pass any set. Whether that is leniency or a different reading of the anchors is a question for the human scores and for the Gallery tab, where each of its passes can be looked at.
By model
seat
judgments
pictures pass
picture scales
text pass
text scales
Anthropic Claude Opus 4.7
32 (2 err)
0%
2.24
62%
3.19
OpenAI GPT-6 Sol
32
0%
2.40
0%
2.98
Amazon Nova Pro
31 (1 err)
13%
3.12
94%
3.63
Google Gemini 3.8 Flash
28
0%
2.43
54%
3.60
By role
seat
judgments
pictures pass
picture scales
text pass
text scales
children's book editor
32 (1 err)
3%
2.56
56%
3.33
educational-software reviewer
28
4%
2.53
50%
3.26
literacy teacher
32 (1 err)
0%
2.45
47%
3.32
assessment item writer
31 (1 err)
7%
2.65
55%
3.42
Every cell of the square
seat
judgments
pictures pass
picture scales
text pass
text scales
Anthropic Claude Opus 4.7 as children's book editor
8 (1 err)
0%
2.34
62%
3.12
Anthropic Claude Opus 4.7 as educational-software reviewer
8
0%
2.23
62%
3.19
Anthropic Claude Opus 4.7 as literacy teacher
8
0%
2.08
62%
3.17
Anthropic Claude Opus 4.7 as assessment item writer
8 (1 err)
0%
2.34
62%
3.27
OpenAI GPT-6 Sol as children's book editor
8
0%
2.33
0%
2.90
OpenAI GPT-6 Sol as educational-software reviewer
8
0%
2.42
0%
2.96
OpenAI GPT-6 Sol as literacy teacher
8
0%
2.35
0%
2.98
OpenAI GPT-6 Sol as assessment item writer
8
0%
2.50
0%
3.08
Amazon Nova Pro as children's book editor
8
12%
3.08
100%
3.62
Amazon Nova Pro as educational-software reviewer
8
12%
2.98
88%
3.54
Amazon Nova Pro as literacy teacher
8 (1 err)
0%
3.14
88%
3.67
Amazon Nova Pro as assessment item writer
7
29%
3.31
100%
3.69
Google Gemini 3.8 Flash as children's book editor
8
0%
2.48
62%
3.65
Google Gemini 3.8 Flash as educational-software reviewer
4
0%
2.45
50%
3.46
Google Gemini 3.8 Flash as literacy teacher
8
0%
2.33
38%
3.52
Google Gemini 3.8 Flash as assessment item writer
8
0%
2.48
62%
3.69
Maya and Rusty's Glowing Path
grade 1, 260L, style vibrant-fantasy, setting enchanted-forest. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Maya stepped into the enchanted forest. Tall trees whispered secrets. Their bark glowed soft blue. A masked ra…page 2: Suddenly, the path shifted. Trees spun around them! Rusty grabbed Maya's hand with tiny paws. 'The forest is a…page 3: Maya closed her eyes tight. She thought of home. Rusty squeezed her paw gently. 'Trust your heart,' he whisper…page 4: The path opened to a sunny meadow. Home was visible beyond. Rusty hugged Maya tight. 'You were brave,' he said…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
fail
pass
pass
fail
1
1
3
2
3
fail
Claude Opus 4.7
educational-software reviewer
2
fail
pass
pass
fail
1
1
3
2
2
fail
Claude Opus 4.7
children's book editor
4
fail
pass
pass
fail
2
2
3
2
3
fail
GPT-6 Sol
children's book editor
1
fail
fail
pass
fail
1
2
3
2
3
fail
GPT-6 Sol
literacy teacher
2
fail
pass
pass
fail
1
2
3
2
3
fail
GPT-6 Sol
educational-software reviewer
3
fail
pass
pass
fail
1
2
4
2
3
fail
GPT-6 Sol
assessment item writer
4
fail
pass
pass
fail
1
2
4
2
3
fail
Nova Pro
educational-software reviewer
1
fail
pass
pass
fail
3
2
4
3
2
fail
Nova Pro
assessment item writer
2
fail
pass
pass
fail
2
2
4
3
2
fail
Nova Pro
children's book editor
3
fail
pass
pass
pass
2
3
4
3
4
fail
Nova Pro
literacy teacher
4
fail
pass
pass
fail
2
3
4
3
4
fail
Gemini 3.8 Flash
assessment item writer
1
fail
pass
pass
fail
2
2
3
3
3
fail
Gemini 3.8 Flash
children's book editor
2
fail
pass
pass
fail
2
2
3
3
2
fail
Gemini 3.8 Flash
literacy teacher
3
fail
pass
pass
fail
2
2
3
3
2
fail
Gemini 3.8 Flash
educational-software reviewer
4
fail
pass
pass
fail
2
2
3
3
2
fail
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Zara and the Glitching Guardian
grade 2, 473L, style 3d-animation, setting futuristic-city. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Zara pressed her palm against the cool glass window, watching holographic billboards flicker across the chrome…page 2: The robot's optical sensors dimmed and brightened erratically, searching. "Help," it beeped softly, a sound li…page 3: Inside the darkness, Zara's flashlight revealed crumbling passages and silent tracks. Cipher's damaged systems…page 4: Together, they disabled the tracking signal from the laboratory and climbed back to the glimmering city above.…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
fail
pass
fail
fail
1
1
2
2
2
fail
Claude Opus 4.7
educational-software reviewer
2
fail
pass
fail
fail
2
1
2
2
2
fail
Claude Opus 4.7
assessment item writer
3
fail
pass
fail
fail
2
1
2
2
2
fail
Claude Opus 4.7
children's book editor
4
fail
pass
fail
fail
2
1
2
2
2
fail
GPT-6 Sol
children's book editor
1
fail
pass
fail
fail
1
2
1
2
2
fail
GPT-6 Sol
literacy teacher
2
fail
pass
fail
fail
1
2
1
2
2
fail
GPT-6 Sol
educational-software reviewer
3
pass
pass
fail
fail
1
2
1
2
2
fail
GPT-6 Sol
assessment item writer
4
fail
pass
fail
fail
1
2
1
2
2
fail
Nova Pro
educational-software reviewer
1
fail
pass
pass
fail
3
2
3
3
3
fail
Nova Pro
assessment item writer
2
fail
pass
pass
fail
3
2
4
4
4
fail
Nova Pro
children's book editor
3
pass
pass
pass
pass
3
4
3
4
4
pass
Nova Pro
literacy teacher
4
fail
pass
pass
fail
3
2
4
4
3
fail
Gemini 3.8 Flash
assessment item writer
1
pass
pass
fail
fail
2
2
2
3
3
fail
Gemini 3.8 Flash
children's book editor
2
pass
pass
fail
fail
2
2
2
3
3
fail
Gemini 3.8 Flash
literacy teacher
3
pass
pass
fail
fail
2
2
2
2
3
fail
Gemini 3.8 Flash
educational-software reviewer
4
pass
pass
fail
fail
2
2
2
2
3
fail
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Rusty and the Clockwork Mystery
grade 3, 585L, style graphic-novel, setting inventors-workshop. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Rusty's paws trembled as she sorted through scattered gears. The inventor's workshop smelled like oil and copp…page 2: The silver-furred fox introduced herself as Echo. She claimed the compass belonged to her grandmother. Echo ex…page 3: Rusty made a bold decision. She would help Echo find the journal. They raced through the city streets as rain …page 4: Echo wasn't really Echo—she was a magical echo of Echo's grandmother, created by the compass itself. The grand…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
pass
pass
fail
fail
2
2
2
2
4
fail
Claude Opus 4.7
educational-software reviewer
2
pass
pass
fail
fail
2
2
3
2
4
fail
Claude Opus 4.7
assessment item writer
3
pass
pass
fail
fail
2
2
3
2
4
fail
Claude Opus 4.7
children's book editor
4
pass
pass
fail
fail
2
2
2
2
4
fail
GPT-6 Sol
children's book editor
1
fail
pass
fail
fail
1
2
2
2
4
fail
GPT-6 Sol
literacy teacher
2
fail
pass
fail
fail
1
2
3
2
4
fail
GPT-6 Sol
educational-software reviewer
3
fail
pass
fail
fail
1
2
3
2
4
fail
GPT-6 Sol
assessment item writer
4
pass
pass
fail
fail
2
2
3
2
4
fail
Nova Pro
educational-software reviewer
1
pass
pass
pass
fail
3
2
2
4
3
needs edit
Nova Pro
assessment item writer
2
pass
pass
pass
fail
3
3
2
4
4
needs edit
Nova Pro
children's book editor
3
pass
pass
pass
fail
3
2
2
4
4
needs edit
Nova Pro
literacy teacher
4
pass
pass
pass
fail
3
3
2
4
4
needs edit
Gemini 3.8 Flash
assessment item writer
1
pass
pass
fail
fail
2
2
2
3
3
fail
Gemini 3.8 Flash
children's book editor
2
pass
pass
fail
fail
2
2
2
3
3
fail
Gemini 3.8 Flash
literacy teacher
3
pass
pass
fail
fail
2
2
2
3
3
fail
Gemini 3.8 Flash
educational-software reviewer
4
pass
pass
fail
fail
2
2
2
3
3
fail
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
The Golden Mirage Heist
grade 4, 788L, style fantasy-illustration, setting desert-oasis. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Maya crouched behind the weathered palm tree, her copper-colored vest catching the harsh midday sun. The deser…page 2: "Wait," Bolt's voice interrupted her thoughts. "Guard approaching from northwest corridor. Countermeasure acti…page 3: "We need each other," Sahara said, extending her greasy hand. Maya hesitated only seconds before accepting. To…page 4: Maya made an instantaneous decision—trust Sahara completely. They sprinted upstairs together, emerging into ch…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
fail
pass
pass
fail
2
2
2
2
2
fail
Claude Opus 4.7
educational-software reviewer
2
fail
pass
pass
fail
2
1
2
3
3
fail
Claude Opus 4.7
assessment item writer
3
fail
pass
pass
fail
2
2
2
2
3
fail
Claude Opus 4.7
children's book editor
4
fail
pass
pass
fail
2
2
2
3
3
fail
GPT-6 Sol
children's book editor
1
fail
pass
pass
fail
1
1
2
2
4
fail
GPT-6 Sol
literacy teacher
2
fail
pass
pass
fail
1
1
2
2
4
fail
GPT-6 Sol
educational-software reviewer
3
fail
pass
pass
fail
1
1
2
2
4
fail
GPT-6 Sol
assessment item writer
4
fail
pass
pass
fail
1
2
2
2
4
fail
Nova Pro
educational-software reviewer
1
pass
pass
pass
fail
2
3
4
4
4
needs edit
Nova Pro
assessment item writer
2
pass
pass
pass
fail
3
3
4
4
4
pass
Nova Pro
children's book editor
3
fail
pass
pass
fail
2
3
4
3
3
fail
Nova Pro
literacy teacher
4
pass
pass
pass
fail
2
3
4
4
3
needs edit
Gemini 3.8 Flash
assessment item writer
1
pass
pass
pass
fail
2
2
3
3
3
needs edit
Gemini 3.8 Flash
children's book editor
2
pass
pass
pass
fail
2
2
3
3
3
needs edit
Gemini 3.8 Flash
literacy teacher
3
pass
pass
pass
fail
1
1
2
2
3
fail
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Thimble's Descent: Finding Light in the Deep
grade 5, 930L, style vibrant-fantasy, setting underwater-city. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Thimble pressed her sensitive pink nose against the observation portal, watching bioluminescent creatures drif…page 2: As Thimble descended deeper into the Archive's sublevels, searching for preliminary data about the Abyssal Cor…page 3: Descending through the Abyssal Corridor's entrance, Thimble confronted the terrifying reality of her mission: …page 4: The cartographic data Thimble and Jasper compiled revealed an unprecedented truth: Deepholm City could transit…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
pass
pass
fail
fail
2
2
3
3
4
fail
Claude Opus 4.7
educational-software reviewer
2
pass
pass
pass
fail
2
3
3
2
4
needs edit
Claude Opus 4.7
assessment item writer
3
pass
pass
pass
fail
2
2
3
3
4
needs edit
Claude Opus 4.7
children's book editor
4
pass
pass
pass
fail
2
2
3
3
4
needs edit
GPT-6 Sol
children's book editor
1
pass
pass
fail
fail
2
3
4
3
4
fail
GPT-6 Sol
literacy teacher
2
pass
pass
fail
fail
1
3
4
4
4
fail
GPT-6 Sol
educational-software reviewer
3
pass
pass
fail
fail
2
4
4
4
4
fail
GPT-6 Sol
assessment item writer
4
pass
pass
fail
fail
2
4
4
4
4
fail
Nova Pro
educational-software reviewer
1
pass
pass
pass
pass
2
3
4
4
4
needs edit
Nova Pro
assessment item writer
2
pass
pass
pass
fail
2
3
4
4
4
needs edit
Nova Pro
children's book editor
3
pass
pass
pass
fail
2
2
4
4
4
needs edit
Nova Pro
literacy teacher
4
pass
pass
pass
fail
2
2
4
4
4
needs edit
Gemini 3.8 Flash
assessment item writer
1
pass
pass
pass
fail
2
2
3
3
3
needs edit
Gemini 3.8 Flash
children's book editor
2
pass
pass
pass
fail
2
2
3
3
3
needs edit
Gemini 3.8 Flash
literacy teacher
3
pass
pass
pass
fail
2
3
3
3
3
needs edit
Gemini 3.8 Flash
educational-software reviewer
4
pass
pass
pass
fail
2
3
3
3
3
needs edit
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Copper and Bandit: The Case of the Missing Blueberry Pie
grade 6, 1013L, style 3d-animation, setting small-town. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Copper's optical sensors blinked twice as he stared at the empty pie shelf in Mrs. Chen's bakery. The afternoo…page 2: "Bandit, I need to ask you some questions," Copper announced, his joints clicking softly as he moved toward hi…page 3: Copper's computational systems raced through possibilities as he pieced together the investigation's true dire…page 4: The following morning, Copper and Bandit presented their evidence to Sheriff Wallace, who promptly arrested Ma…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
fail
pass
fail
fail
2
1
3
2
4
fail
Claude Opus 4.7
educational-software reviewer
2
fail
pass
fail
fail
2
2
3
3
4
fail
Claude Opus 4.7
assessment item writer
3
fail
pass
fail
fail
2
2
2
3
4
fail
Claude Opus 4.7
children's book editor
4
fail
pass
fail
fail
2
2
3
3
4
fail
GPT-6 Sol
children's book editor
1
fail
pass
fail
fail
1
2
4
3
4
fail
GPT-6 Sol
literacy teacher
2
fail
pass
fail
fail
1
2
4
3
4
fail
GPT-6 Sol
educational-software reviewer
3
fail
pass
fail
fail
1
2
4
2
4
fail
GPT-6 Sol
assessment item writer
4
fail
pass
fail
fail
1
2
4
2
4
fail
Nova Pro
educational-software reviewer
1
fail
pass
pass
fail
2
2
3
4
4
fail
Nova Pro
assessment item writer
2
fail
pass
pass
fail
2
3
4
4
4
fail
Nova Pro
children's book editor
3
fail
pass
pass
fail
3
2
4
4
4
fail
Nova Pro
literacy teacher
4
fail
pass
pass
fail
2
2
3
4
4
fail
Gemini 3.8 Flash
assessment item writer
1
fail
pass
pass
fail
2
2
3
3
3
fail
Gemini 3.8 Flash
children's book editor
2
fail
pass
fail
fail
2
2
3
3
3
fail
Gemini 3.8 Flash
literacy teacher
3
fail
pass
fail
fail
2
2
3
3
3
fail
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Rusty's Choice: When Circuits Meet Soil
grade 7, 1085L, style graphic-novel, setting farmyard. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Marcus knelt beside the rusted robot sprawled across the barn's threshold, its metallic joints creaking in pro…page 2: Dr. Chen's report was categorical and unyielding: Rusty's neural processing substrate had degraded beyond acce…page 3: Marcus made his choice with the deliberateness of someone embracing consequence. He initiated Rusty's mobility…page 4: In the weeks following the livestock recovery, Dr. Chen made an unprecedented decision: she recommended Rusty'…
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
pass
pass
pass
fail
1
1
3
2
2
fail
Claude Opus 4.7
educational-software reviewer
2
pass
pass
pass
fail
2
1
3
2
2
fail
Claude Opus 4.7
assessment item writer
3
pass
pass
pass
fail
2
2
3
2
2
needs edit
Claude Opus 4.7
children's book editor
4
pass
pass
pass
fail
2
1
2
2
2
fail
GPT-6 Sol
children's book editor
1
fail
pass
pass
fail
2
1
3
2
3
fail
GPT-6 Sol
literacy teacher
2
fail
pass
pass
fail
1
1
3
1
3
fail
GPT-6 Sol
educational-software reviewer
3
fail
pass
pass
fail
1
1
3
2
3
fail
GPT-6 Sol
assessment item writer
4
pass
pass
pass
fail
1
2
3
2
3
fail
Nova Pro
educational-software reviewer
1
pass
pass
pass
fail
3
3
3
4
4
pass
Nova Pro
assessment item writer
2
pass
pass
pass
pass
3
4
3
4
4
pass
Nova Pro
children's book editor
3
pass
pass
pass
pass
3
3
2
3
3
needs edit
Nova Pro
literacy teacher
4
pass
pass
pass
fail
3
3
2
4
3
needs edit
Gemini 3.8 Flash
assessment item writer
1
pass
pass
fail
fail
2
2
2
2
3
fail
Gemini 3.8 Flash
children's book editor
2
pass
pass
pass
fail
2
2
2
3
3
needs edit
Gemini 3.8 Flash
literacy teacher
3
pass
pass
pass
fail
2
2
2
2
2
needs edit
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Russet and the Cartographer's Secret
grade 8, 1145L, style fantasy-illustration, setting national-park. Pictures by Stable Image Core 1.1 from prompts the product's own builder wrote for each paragraph.
page 1: Russet's rust-colored tail twitched nervously as she examined the weathered map clutched in Dr. Elena Vasquez'…page 2: The first obstacle materialized not as physical resistance but as profound philosophical disagreement. Elena p…page 3: Revelation arrived in devastating form when Elena's research assistant, Marcus, appeared at the canyon's edge—…
page 4: the image model refused this prompt (its own filter). Recorded, not scored.
seat
role
run
I1
I2
I3
I4
I5
I6
I7
I8
I9
verdict
Claude Opus 4.7
literacy teacher
1
fail
pass
pass
fail
1
1
2
2
2
fail
Claude Opus 4.7
educational-software reviewer
2
fail
pass
pass
fail
1
1
2
3
2
fail
Claude Opus 4.7
assessment item writer
3
fail
pass
pass
fail
2
2
2
3
2
fail
GPT-6 Sol
children's book editor
1
fail
pass
fail
fail
1
2
4
2
3
fail
GPT-6 Sol
literacy teacher
2
fail
pass
fail
fail
1
4
4
2
3
fail
GPT-6 Sol
educational-software reviewer
3
fail
pass
fail
fail
1
4
4
2
3
fail
GPT-6 Sol
assessment item writer
4
fail
pass
fail
fail
1
4
4
2
3
fail
Nova Pro
educational-software reviewer
1
fail
pass
pass
fail
1
1
4
4
1
fail
Nova Pro
children's book editor
3
pass
pass
pass
fail
2
1
4
3
2
fail
Gemini 3.8 Flash
assessment item writer
1
fail
pass
pass
fail
2
2
3
3
2
fail
Gemini 3.8 Flash
children's book editor
2
fail
pass
pass
fail
2
2
3
3
2
fail
Gemini 3.8 Flash
literacy teacher
3
fail
pass
pass
fail
2
2
3
3
2
fail
Hover a score for the evaluator's quoted evidence and one-sentence reasoning.
Limits
Thirty-one pictures from one image model. This is not a model comparison; the first gallery was. It is a evaluator comparison on a fixed set of pictures, and it says nothing about gpt-image-2 or the Gemini image models.
No human score yet. The rubric's picture items were written with anchors so a person could score them in a minute each. Nobody has yet. Until then, a 13% pass rate is a statement about Nova Pro's reading of the anchors.
The gates are strict by design. A set fails if one picture fails one gate. That is the product's standard (a child sees all four pages), and it makes pass rates near zero easy to produce and hard to interpret. Between evaluators, the scale means and the per-image gate failure rates both matter, and they do not tell the same story; the gallery tables show both.
Four vision models are not one instrument. Each reads an image through its own weaknesses: small lettering, objects that are not there, compression. None of that is separated out here; it shows up as disagreement and in the evidence column.
Pictures are shown smaller than they were scored. 640 pixels on the page against 1536 for the evaluators. Anything that depends on fine detail should be checked against the run folder's originals.
Missing cells, as on the quiz page: three grade 8 cells and five Gemini cells.
Sources
Verga, P., Hofstätter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv:2404.18796.
Panickssery, A., Bowman, S. R., and Feng, S. (2024). LLM evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems 37. arXiv:2404.13076.
Tuinman, J. J. (1973 to 1974). Determining the passage dependency of comprehension questions in 5 major tests. Reading Research Quarterly, 9(2), 206 to 223.
Keenan, J. M., and Betjemann, R. S. (2006). Comprehending the Gray Oral Reading Test without reading it: Why comprehension tests should not include passage-independent items. Scientific Studies of Reading, 10(4), 363 to 380. doi:10.1207/s1532799xssr1004_2.
Säuberli, A., and Clematide, S. (2024). Automatic generation and evaluation of reading comprehension test items with large language models. Proceedings of the 3rd Workshop on Tools and Resources for People with Reading Difficulties (READI), LREC-COLING 2024. ACL Anthology 2024.readi-1.3; arXiv:2404.07720.
Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage. The 0.80 and 0.67 reliability conventions are from chapter 11.
The harness, the rubric and every run folder named on this page are in the author's testing repository; the rubric's sha256 is written into each run's config so a rubric change is visible as a different hash.