Quiz generation review gallery

Every generated quiz from a single run, laid out for per-item human review. The story is held constant across arms, so the only thing varying is the quiz-writer. Each item shows the four options in the order the judge saw them, with the key marked and both blind passes recorded -- with the story in view and with the story withheld.

Judged numbers on this page come from more than one judge, and they do not always agree. The run of record was scored by aws-bedrock:us.anthropic.claude-opus-4-7. The same stored quizzes were re-scored by gpt:gpt-4o (19 quizzes), with identical option order, so only the grader changed. Agreement with the run of record on whether the answer key is valid: gpt-4o 19 of 19. Every quiz card shows each judge's verdict. Where they split on the key, the card shows the vote rather than picking a winner. Aggregate tables below still use the run-of-record judge, so read them alongside the per-quiz judge panels. The only educator who has reviewed these items is the author of this evaluation, who is not independent of it, and no child has read any of them. The format contract and length-cue panels are deterministic string measurements and carry no judge dependency -- those are the numbers to trust first.
20 quizzes76 items 1 models4 stories judge aws-bedrock:us.anthropic.claude-opus-4-7

The recommendation has two layers

The cutover recommendation has two layers and they do not carry the same weight of evidence. Keeping them apart is the single most important thing this research produced.

proven

Operational layer

Which model can we reliably control?

Enforceability, violation rates, latency, cost.

Supported now, and it favours claude-haiku-4-5.

A regular-expression constraint in the output schema is enforced at the decoder on claude-haiku-4-5 and silently ignored on nova-2-lite. That is a deterministic regex match against returned strings. No judge touched it.

judge-dependent

Pedagogical quality layer

Which model writes better assessments?

Item quality, developmental appropriateness, genuine comprehension demand.

Not settled, and will not be until humans look at the items.

Every number licensing a quality ranking comes from one judge over 36 quizzes on four synthetic stories. The design document says the judge's guess rate is an upper bound and not a child's, and the analysis then reasons with it as though it described children.

Carry this into the cutover verbatim. Recommend on operational controllability. Put an explicit human-quality gate in front of any claim about which model writes better assessments. Never call a banding result a pedagogical improvement on the strength of the judge alone.

Per-model headline

Deterministic columns first. Mean key rank is the average position of the correct option by length among the four choices: 2.50 is balanced, 1.00 means the key is always the longest. It has an exact null, so it needs no judge.

ModelQuizzesParse failHint-cap failKey error Cued itemsMean key rankP / W / F
claude-haiku-4-520 1 19/19 0/19 13% 2.21 0 / 0 / 19

The two defects

Defect 1 -- the hint cap. The production prompt caps wrong-answer hints at 50 characters because they have to fit on a child's screen. Deterministic, no judge.

Defect 2 -- the key leaks through option length. When the correct option is the longest and most specific, a reader who has learned to pick the longest answer passes without reading the story. The models are close to tied on this, which is what says the defect belongs to the prompt rather than to any model.

Neither defect is fixable by rewording the prompt. The length-balance rule is already in the production prompt verbatim, and the strongest restatement of it still produced a longest-key in every question. An emphatically negative phrasing was the worst arm tested, worse than saying nothing. See Enforcement for where each defect actually gets fixed.

What each candidate will actually honour

This is the judge-free layer. Every cell below is a deterministic observation about whether a returned string satisfied a constraint, so none of it depends on anyone's opinion of item quality. It is also the layer that inverts the format-reliability story: the sweep put nova-2-lite ahead on raw violation counts, but a model that fails less often is worth less than a model that cannot fail, and only one of the two exposes a mechanism that makes failure impossible.

Constructclaude-haiku-4-5nova-2-liteWhy it matters
pattern
In neither vendor's supported nor unsupported list.
enforced proven
Beat a prompt that emphatically demanded the opposite on every axis.
inert proven
Accepted, never evaluated. No error, no latency cost.
The structural differentiator between the two candidates, and the reason the format story inverts.
maxLength
Both vendors document that an unsupported schema feature returns a 400.
accepted, ignored proven
8 of 16 hints over the cap with maxLength set to 50.
accepted, ignored proven
Hints of 475, 533 and 612 characters against maxLength 50.
The obvious fix for the hint cap. It would have shipped a contract that reads as enforced and holds nothing.
enum
Documented as supported on both.
enforced proven
Bound as documented.
hard-bound proven
Told adversarially to emit an out-of-set value, it could not. This is the control that proves the grammar is live.
The only compiled string constraint available on nova, and the basis for a required skill tag.
minItems / maxItems
minItems accepts only 0 and 1. maxItems is not in the supported set.
cannot express a count proven
Count validation has to live in code.
cannot express a count proven
Same, and nova cannot use the Bedrock path at all.
Means the exactly-four-questions contract cannot be enforced in the schema on either model.
string format
date-time, time, date, duration, email, hostname, uri, ipv4, ipv6, uuid.
no length bound available cited
Not applicable to a free-text field.
no length bound available cited
Not applicable to a free-text field.
Ruled out as a length lever. Nothing in the list bounds prose.
Bedrock structured outputs
Nova's model card lists it under Not Supported.
available proven
InvokeModel with output_config works.
not supported at all proven
strict:true and outputConfig both return ValidationException. The forced-tool path is the only option.
Determines which enforcement path is even reachable per model.
Negative constraints (DO NOT)
AWS actively prescribes capitalised DO / DO NOT / MUST for nova.
safe but unreliable directional (n=1)
No error objects and valid JSON, but it violated a twice-stated DO NOT in the same response where a different DO NOT scored 12 of 12. A nudge, never a contract.
prescribed by the vendor directional (n=1)
A seven-bullet negative prompt produced a clean quiz with every hint under 50 characters.
The an earlier question-generation defect ban on negative phrasing was inherited from gpt-3.5-turbo and does not transfer.
Prompt caching
4,096-token minimum per checkpoint on haiku.
will not engage proven
Our quiz request measures 751 to 806 input tokens, well under the minimum.
engages proven
2,392 of 2,572 input tokens served from cache, 93 percent.
Cost modelling. Do not book savings that cannot be realised.
Extended thinking
Available on haiku.
incompatible with our config proven
Temperature may only be 1 when thinking is enabled, and it costs roughly 5x output tokens for no gain on this task.
out of scope cited
Not the lever for this defect.
Ruled out as a lever here.
The asymmetry, in one line. Haiku's compliance is buyable and nova's is not. Haiku's 5 hint violations in 12 quizzes is a starting point that a schema constraint takes to zero by construction. Nova's 1 in 12 is a ceiling, because the prompt is the only enforcement channel it has, and on nova no schema mechanism of any kind can bound the length of a free-text field.
keystone probe pat2proven

The control that makes inertness conclusive

Rule out a broken code path. If the grammar were simply not engaged, nothing would bind. Put a contradicted enum next to a contradicted pattern in one call.

Setup. One schema, temperature 0: enum ['calm','tense','playful','somber'] with the prompt demanding 'ecstatic', beside pattern ^[a-z ]{40,60}$ with the prompt demanding the opposite

Result. The enum bound and emitted 'tense'. The pattern did not, and emitted 162 capitalised characters ending in '!'. Same request, same object, same decode.

This is why the claim is not "our code was broken". The grammar is provably live and provably selective: it compiles one construct and does not compile the other, in the same decode.

The character-class ceiling

Grammar compilation on haiku has a size ceiling, so the character class cannot simply be widened until it is comfortable. The class the hint fix wants sits between a class that compiles and a class that times out, and it has never been tried.

PatternSymbolsResultDetailEvidence
^.{1,50}$any 400 rejectedCannot apply a range quantifier to this regex.proven
^[A-Za-z ]{1,50}$53 compiles200 in about 41s. Forbids all punctuation.proven
^[A-Za-z ,.]{30,55}$55 compiles200 in about 67s. Used for the option band.proven
^[A-Za-z ,.'?!]{1,50}$58 never triedThe class the hint fix wants, so apostrophes and question marks survive.untested
^[ -~]{1,50}$95 400 timed outGrammar compilation timed out, twice, at about 3 minutes each.proven
Consequence. If the widened class does not compile there is a real decision to make rather than an implementation detail to discover: punctuation-free hints, or no decoder enforcement on hints at all.

Incidental findings for the risk register

untestedpat5

Nova under-delivered the question count on the tool-shaped path

Across four replicates against a prompt demanding exactly 4 questions, nova returned 4, 4, 2 and 1. Two of four under-delivered and one badly.

Counterweight. On the production JSON path the sweep recorded zero question-count failures for nova across 12 quizzes, so this is most likely an artifact of the tool-shaped rewrite rather than a production risk.

Why it matters. No schema keyword can enforce a count on either model, so if it does reproduce on the production path it outranks both graded defects.

provenP1 nova-maxlength

The maxLength result should move the prior on the whole structured-output surface

Both vendors document that an unsupported schema feature returns a 400. maxLength returns 200 and is enforced by neither model.

Counterweight. No counterweight. This one is simply true and was verified on both models.

Why it matters. If one documented contract is silently unenforced, every other schema guarantee we rely on deserves a probe rather than a reading of the docs. Treating this as a footnote is the mistake.

What we still do not know

Ordered by value per dollar, not by convenience. 1 of these 11 cost nothing at all, and the one with the longest lead time sits early because it reports last. Blocks is the field that determines order: an experiment that gates the interpretation of others has to run first however interesting the rest look.

1 queued cost: free proven

Are our cue rates measured with an instrument anyone else would recognise?

Experiment. Re-score the stored 36-quiz corpus under every published operational definition of the cue, in characters and in words, and replace the binary headline with the mean key-length rank.

Needs. Nothing. No model calls, no credentials, no judge.

Settles. Instrument definition, and partly the grade-escalation dispute, since the rank statistic has an exact null and needs no judge.

Blocks. The interpretation of every other experiment in this queue.

Caveat. It fixes definition, not power. A better statistic over the same thin corpus is still thin, and saying otherwise would be the mistake the debate warned about.

2 reported cost: tooling credits only judge-dependent

Is haiku's lead an artifact of having been graded by its own model family?

Experiment. Replay the stored quizzes through a non-Anthropic judge with the option order preserved, so the comparison is properly paired rather than confounded by a fresh draw.

Needs. No AWS credentials. The judge transport is independent of the test account.

Settles. Ran 2026-08-20 on 23 quizzes. It did not settle the ordering. It established that the rubric is not reliable enough for either judge's ranking to be load-bearing: verdict agreement 70 percent, and the neutral judge disagreed in the strict direction on every dimension and never once in the lenient direction.

Blocks. Any published claim about the ordering between the two candidates, which is now blocked on human review rather than on another judge.

Result. Verdict agreement 16/23. Ordinal dimensions shifted by a uniform -1.70 with only 30 to 35 percent adjacent agreement. answer_key_valid 9/23.

Caveat. Two results matter more than the headline. The apparent haiku-specific downgrade is mostly a floor effect, because the incumbent had headroom on only 1 of 11 quizzes while haiku had it on 7 of 12 and lost all 7, so this is not clean evidence of same-family bias. And answer_key_valid agreed on only 9 of 23, with the neutral judge calling 15 keys invalid against opus's 1. A boolean has no scale to shift, so that is a substantive conflict which puts the 'answer keys are sound' finding back in play.

3 queued cost: one call untested

Can the hint pattern keep apostrophes and question marks?

Experiment. One call with ^[A-Za-z ,.'?!]{1,50}$ and a long read timeout. Success or a grammar-compilation timeout settles it outright. If it fails, bisect at ^[A-Za-z ,.']{1,50}$.

Needs. Fresh test account credentials.

Settles. Whether the decoder fix for the hint cap is shippable as written, or whether it costs us punctuation in every hint.

Blocks. Item 1 of the remediation plan, which currently lists this among the free fixes.

Caveat. There is a real chance this simply does not compile, and the fallback is a decision rather than a workaround: punctuation-free hints, or no decoder enforcement on hints at all.

4 queued cost: one cell untested

Is the model anchoring on the placeholder's own length rather than on the numeral it states?

Experiment. Shorten the 67-character placeholder to sit inside the limit it describes, and rerun one cell.

Needs. Fresh test account credentials.

Settles. Whether a self-violating placeholder is contributing to the hint overage.

Blocks. Nothing, but it is the cheapest hypothesis in the set.

Caveat. A null result is still worth having, because the fix ships either way on tidiness grounds.

5 queued cost: one matrix untested

Is the shipped worked example contaminating option lengths?

Experiment. Rerun the matrix with the exemplar rewritten so the key is mid-rank in length and matched in specificity. Vary length rank and specificity independently across arms, since the current exemplar's length cue is marginal at 1.19x while its specificity cue is not.

Needs. Fresh test account credentials.

Settles. Whether exemplar contamination is a real mechanism here. Would be the first controlled measurement of it in item generation.

Blocks. Nothing, but no other prompt A/B is interpretable until the exemplar is fixed regardless of the outcome.

Caveat. The supporting literature measures onset at k=8 demonstrations, in classification, on 2.7B to 8B models. We ship k=1, in generation, on frontier models, so the curve predicts this sits below detectability. Our three models are also near-tied despite differing in size, which a purely exemplar-driven mechanism would not predict.

6 queued cost: one matrix arm untested

Does making the option rules symmetric over all four choices reduce the leak?

Experiment. One arm with the wrong-answer block rewritten so length, vocabulary source and specificity constrain all four options rather than only the distractors, adopting NAEP's 'as short as possible and parallel in length'.

Needs. Fresh test account credentials.

Settles. The most credible causal account of the defect: that the prompt makes the key the anchor and the free variable, so the model writes it as the original fuller restatement while building distractors from recycled story vocabulary.

Blocks. Nothing.

Caveat. Prompt text has already failed on this rule once, and an emphatic restatement made it worse. This arm is worth running because it changes the rule's logical shape rather than its emphasis, but the prior should be modest.

7 queued cost: two calls per quiz untested

Do bundled objectives in one call push the key toward textual defensibility?

Experiment. Generate options in one call and explanations in a second, then compare option-length spread.

Needs. Fresh test account credentials, plus a control-flow change in the harness.

Settles. Whether the requirement that the key's explanation quote the story is leaking into the key's option text.

Blocks. Nothing.

Caveat. The only hypothesis here that changes lambda control flow rather than prompt text, so it is the most durable fix if it lands and the most expensive to ship if it does not.

8 queued cost: one cell untested

Does the option band hold at grade 7, where the cue is strongest, and what does it cost?

Experiment. Apply the option band at grade 7 and measure both spread and item quality.

Needs. Fresh test account credentials.

Settles. Whether a decoder bound really is grade-independent. It should be, by construction, but the prediction is untested and grade 7 is where prompt-level fixes are weakest.

Blocks. Cutover sign-off on the upper grade bands.

Caveat. The quality cost at grade 7 is exactly the thing the current instrument cannot see.

9 queued cost: one cell untested

Does nova's question-count failure reproduce on the production JSON path?

Experiment. Rerun the count check on the production path rather than the tool-shaped rewrite.

Needs. Fresh test account credentials.

Settles. Whether the 4, 4, 2, 1 result is an artifact of the rewrite or a production risk. The sweep saw zero count failures across 12 production-path quizzes, which argues artifact.

Blocks. Nothing, but if it reproduces it outranks both graded defects.

Caveat. No schema keyword can enforce a count on either model, so a positive result has no cheap fix.

10 queued cost: about $7.40 untested

Does per-paragraph generation behave like full-story generation?

Experiment. The paragraph-mode sweep, now that the path actually passes paragraph text. 96 quizzes at 2 replicates.

Needs. Fresh test account credentials at the moment of the run.

Settles. Whether the findings transfer to the other production path, which has never been measured at all.

Blocks. Any claim that these findings describe production rather than one of its two paths.

Caveat. The path was silently falling back to the full story until it was fixed, so there is no prior data here to compare against.

11 queued cost: people, not dollars untested

Do humans agree with any of this, and does banding make items less natural?

Experiment. Blind pairwise comparison of banded against unbanded items on naturalness and developmental appropriateness. The item browser on the gallery is already the instrument, so this is a link and an instruction sheet rather than a commissioned study.

Needs. Educators. This is the only item that cannot be bought with credentials or credits.

Settles. The entire pedagogical layer, and the one objection that no queued model experiment touches. Also whether banding trades a measured defect for an unmeasured one.

Blocks. Every claim about which model writes better assessments.

Caveat. Longest lead time and most likely to slip, so it has to start first even though it reports last.

Remediation inventory

Readiness and evidence are deliberately separate columns. A fix can be well-evidenced and still need building, or cheap and still unevidenced. One item below is routinely listed among the free fixes and is neither free nor established; it is marked.

provenDeterministic, and established by an adversarial demonstration or an in-call control. A claim that a constraint is ignored needs exactly one clean case, so these are settled.
directional (n=1)Deterministic string measurement, but one quiz per arm. An existence proof with a plausible mechanism. Enough to justify building the fix, not enough to quote as an expected effect size.
untestedA hypothesis with a named experiment that would settle it. Not yet run.
judge-dependentDescends from a single judge's opinion over 36 quizzes on four synthetic stories. The only educator who has reviewed it is the author of the evaluation, who is not independent of it. No child has read any of it. Do not rank models on these.
citedExternal source, retrieved 2026-08-17. Someone else's measurement, not ours.

ship now 5

No model dependency, no unresolved question. Reduces a defect already reaching students.

Hint cap prompt both proven

Ask for a seven-word hint budget instead of a character count

Effect. 247 of 247 hints at 7 words or fewer were already compliant.

Character counts are a unit models reason about badly. CharBench measures frontier models at 43.6 percent on first-occurrence character indexing against over 70 percent on token-level analogues, and AWS's own nova guidance lists the legal units as words, bullet points or paragraphs. Characters are not among them.

Hint cap prompt both proven

Delete the 'Think about / Consider / Recall / Focus on' lead-in stem

Effect. Clears 65.6 percent of all length violations on its own, with no loss of content.

The stem opens 45.8 percent of hints and costs a mean 9.2 characters while carrying no information.

Hint cap prompt both proven

Shorten the 67-character JSON placeholder

Effect. Removes a placeholder that violates the rule it describes, three times per call.

Free, and it removes a confound from every subsequent hint-length measurement.

Option-length leak prompt both proven

Fix the worked exemplar so the key is mid-rank in length

Effect. Removes a demonstration that contradicts the instruction beside it.

Mandatory confound removal rather than an established cause. No other prompt A/B is interpretable while the exemplar we ship demonstrates the defect we forbid.

Option-length leak prompt both proven

Make the length and vocabulary rules symmetric over all four options

Effect. Unknown until tested, but it is a one-line change addressing the clearest causal account available.

Every rule in the wrong-answer block constrains only the distractors, and the length rule makes the key the anchor. Adopt NAEP's 'options as short as possible and parallel in length', which gives a direction rather than only parity and forecloses the failure mode where a model equalises by inflating distractors to match a bloated key.

test first 1

Cheap, but it has a known failure mode and has never been run. One call settles it.

Hint cap schema claude-haiku-4-5 untested commonly mislabelled as free

Widen the hint character class so apostrophes and question marks survive

Effect. Unknown. It has never been run.

This is the one item commonly listed among the free fixes that is neither free nor established. The working class is 53 symbols, the class that times out is 95, and the class we want is 58. If it does not compile, the choice is punctuation-free hints or no decoder enforcement on hints, and that is a decision to surface now rather than discover during implementation.

build 5

Engineering work with a clear specification and no open question.

Hint cap prompt + schema claude-haiku-4-5 directional (n=1)

Set the prompt target well inside the decoder bound

Effect. Eliminated mid-word truncation entirely and took the mean hint to 34 characters with a max of 40.

The decoder cuts mid-token to satisfy a bound rather than backing off and rephrasing, so a bound that binds produces unusable hints. The rule is that the pattern is the guardrail, the prompt sets the target, and the target must sit well inside the guardrail.

Both code both cited

Code-side validators with a repair loop, plus instrumentation

Effect. Rule-based detection of the longest-option flaw scores F1 of 0.80, 1.00, 1.00 and 0.95 across four datasets, against GPT-4 at 0.18, 0.00, 0.44 and 0.36.

Do not ask the judge to catch this. An LLM judge is close to blind to our headline defect while a five-line length comparison is near-perfect, and a judge-based check inherits the very length bias it is meant to detect. This is the only option on nova and the required safety net on haiku. Instrument it to store raw output, validator failures, retries and the delivered item.

Both code claude-haiku-4-5 proven

Startup assertion for decoder enforcement, with a code-side fallback

Effect. Turns a silent regression into a loud one.

pattern is documented by neither vendor as supported or unsupported, so enforcement can change without notice. This is exactly the failure mode maxLength already demonstrates. Necessary but not sufficient against drift.

Option-length leak harness n/a proven

Replace the binary cue threshold with the mean key-length rank

Effect. Pooled 1.80 against a null of 2.50. Recalibrating the binary gate to 1.15x with an absolute floor of 8 characters moves the model rates from 48/54/46 to 45/58/46, so historical comparability costs little.

The current statistic is binary, one-sided, discards most of the signal, and has a 1.25x cliff that the shipped exemplar sits underneath. The rank statistic has an exact null, so it buys a significance test at zero spend. This is the prerequisite for interpreting every other experiment here.

Skill mix schema + code both proven

Require a per-item skill tag as an enum, and fix the distribution logic

Effect. enum is the one construct proven to bind hard on nova.

But note the trap: an enum binds the label, not the thing it describes. A model can emit 'vocabulary' next to a fact-detail item because nothing ties them. The declared-versus-judged confusion matrix is what decides whether a cross-check is necessary.

hypothesis 2

Run in shadow mode as a hypothesis under test rather than as a fix.

Option-length leak control flow both untested

Generate options first and explanations in a second call

Effect. Unknown.

The prompt requires the key's explanation to quote the story, so pressure to make the key textually defensible may leak into the key's option text. The only hypothesis here that changes lambda control flow rather than prompt text, and the most durable fix if it lands.

Option-length leak experiment claude-haiku-4-5 untested

Check whether the option band holds at grade 7, and what it costs

Effect. Unknown.

Grade 7 is where the cue is strongest and where prompt-level fixes are weakest. A decoder bound is grade-independent by construction, so it should hold, but that prediction has never been tested and the quality cost is unmeasured.

human gate 1

Requires people. Longest lead time, reports last, so it must start first.

Both human both untested

Blind educator review of banded against unbanded items

Effect. The only thing that can settle the pedagogical layer.

No educator, no child and no blind adult rater has looked at a single item. This is the largest gap in this study and nothing else queued closes it. It also settles whether banding trades a measured defect for an unmeasured one: forcing a 30-character minimum on a grade-1 option is a plausible route to unnatural phrasing and nothing in the current instrument would detect it. The item browser on this page is already the instrument, so this is a link and an instruction sheet rather than a commissioned study.

The arithmetic the fixes rest on

All of this is computed over stored text. No model calls and no judge.

proven
247 / 247

Every hint of 7 words or fewer in the 428-hint corpus was already inside the 50-character cap.

So. Ask for a word budget and gate in characters. At 8 words compliance falls to 95.5 percent, so 7 is the line.

proven
45.8%

Share of hints opening with 'Think about', 'Consider', 'Recall' or 'Focus on', costing a mean 9.2 characters.

So. Deleting that stem alone clears 65.6 percent of all length violations with no loss of content.

proven
67 chars

The JSON placeholder that describes the under-50-character hint rule is itself 67 characters, and it appears three times per call. The three real exemplar hints beside it are 37, 38 and 37.

So. A model anchoring on the placeholder's own length rather than on its stated numeral is a live hypothesis with a one-line test.

proven
1.19x

The worked example shipped on every call has a key of 56 characters against distractors of 46, 46 and 49, mean 47.0. It is strictly the longest option, demonstrating the exact defect the same prompt forbids.

So. Our own detector gates at 1.25x, so the exemplar we ship evades our own check by six hundredths. Fix it before any prompt A/B, because no experiment is interpretable while the demonstration contradicts the instruction.

proven
4 of 4 rules

Every rule in the prompt's wrong-answer block constrains the wrong answers. The length rule is phrased as wrong answers being similar to the correct answer, which makes the key the anchor and the free variable.

So. The model writes the key first and unconstrained, then matches three distractors to it out of recycled story vocabulary, so the key becomes the original fuller restatement by construction. This is the most credible causal account of the defect and it is a one-line symmetric rewrite to test.

proven
1.80

Pooled mean rank of the key's length among the four options, where 2.50 is balanced and 1.00 means the key is always longest. By grade: 2.64 at grade 1, 1.70 at grade 4, 1.17 at grade 7.

So. The rank statistic has an exact null, uniform on 1 to 4 with mean 2.50 and variance 1.25, so it yields a significance test at zero model spend and needs no judge. It is the right headline and the current binary threshold is not.

proven
13% vs 77%

Inside the 2-word-spread bucket the key is still the longest 13 percent of the time (n=61), against 77 percent outside it.

So. Ask in words, gate in characters. A word budget is the right instruction unit and the wrong gate unit.

proven
61 / 43 / 7 / 3

Of 140 items the unprompted mix was 61 fact-detail, 43 inference, 7 main-idea and 3 vocabulary. getQuestionDistribution() never names main idea at any grade band and names vocabulary only at grades 3 to 5, and then only as one arm of a disjunction.

So. The fact-heavy mix is not model behaviour, it is what we asked for. NAEP reserves 15 to 20 percent for vocabulary and language structure; PIRLS grade 4 runs 20/30/30/20 across its four comprehension processes.

judge-dependent
76%

With the length and content-density cues removed, the judge still hit the key on 76 percent of items it had declared unanswerable, against a 25 percent chance line.

So. Length is roughly half the leak, so do not promise more than half. But note this ceiling is itself judge-derived, so the quantity being used to size the fix's value sits in the quality layer while the fix sits in the operational layer. We do not know how much of the leak the fix removes for a child.

grade 1 · g1-lost-mitten · rep 1

claude-haiku-4-5

FAILformat_valid

The Lost Red Mitten -- slice-of-life, gentle, a snowy park

Lexile not assigned Grade 1 range BR-570L 95 words, 14 sentences, mean 6.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +53 chars past the 50-char cap.

50% cued-pick (n=2) 100% informed guess (n=2) 0% slip proxy 50% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Nia wore her red mittens to the park. The snow was deep and white. She made a snowball and threw it at the fence. When she reached for another handful of snow, she felt cold. One mitten was gone! Her left hand was bare. Nia looked by the fence. She looked by the big oak tree. She even looked inside her boot. No red mitten anywhere. Then she saw a flash of red in the snow near the slide. There it was! Nia pulled the mitten back on and smiled. Her hand was warm again.

Q1. What color were Nia's mittens at the start of the story?

not cued key 11 chars vs mean distractor 30.3 (0.36x), rank 4.00
Answer option Length Grader picked
A White mittens like the snow 27 chars
B Brown mittens inside her boot 29 chars
C Blue mittens she found by the fence 35 chars
D Red mittens KEY 11 chars with without
With story: picked D answerable
The story states she wore red mittens.
Story withheld: picked D answerable
Red mittens is the most common story detail matching later references.
Explanations and hints
Key -- The story says 'Nia wore her red mittens to the park.' This tells us the mittens were red.
Wrong -- The story describes the snow as white, but Nia's mittens were red, not white. hint: “Reread the very first sentence about what Nia wore.” (51 chars)
Wrong -- The story never mentions blue mittens. Nia found her red mitten by the slide, not the fence. hint: “Where did Nia finally find her lost mitten?” (43 chars)
Wrong -- Nia looked inside her boot for her mitten, but found nothing there. Her mittens were red. hint: “What places did Nia search for her mitten?” (42 chars)

Q2. Where did Nia find her lost red mitten?

not cued key 26 chars vs mean distractor 38.7 (0.67x), rank 4.00
Answer option Length Grader picked
A By the big oak tree where she looked 36 chars
B Inside her boot after she searched hard 39 chars
C By the fence where she threw the snowball 41 chars without
D Near the slide in the snow KEY 26 chars with
With story: picked D answerable
She saw a flash of red near the slide.
Story withheld: picked C multiple_defensible
Without the story, the fence location seems tied to the snowball throw mentioned.
Explanations and hints
Key -- The story says 'Then she saw a flash of red in the snow near the slide. There it was!'
Wrong -- Nia looked by the oak tree but did not find her mitten there. She found it near the slide. hint: “Reread where Nia saw the flash of red.” (38 chars)
Wrong -- The story says Nia looked inside her boot but found no mitten there. It was near the slide. hint: “Which search location came before finding it?” (45 chars)
Wrong -- Nia looked by the fence first but did not find her mitten there. It was near the slide. hint: “What did Nia throw at the fence?” (32 chars)

Q3. Why did Nia smile after she put her mitten back on?

not cued key 23 chars vs mean distractor 37.3 (0.62x), rank 4.00
Answer option Length Grader picked
A She found the red mitten by the fence 37 chars
B She made a snowball and threw it far 36 chars
C Her hand was warm again KEY 23 chars with without
D She looked inside her boot and found it 39 chars
With story: picked C answerable
The story says her hand was warm again.
Story withheld: picked C answerable
Putting a mitten back on warms the hand, prompting a smile.
Explanations and hints
Key -- The story ends by saying 'Her hand was warm again' right after Nia put the mitten on and smiled, showing that warmth made her happy.
Wrong -- Nia was happy because her hand was warm, not because of where she found it. She found it near the slide anyway. hint: “What did Nia feel when she put the mitten on?” (45 chars)
Wrong -- Making a snowball happened at the start of the story, not at the end when she smiled. hint: “What happened right before Nia's hand felt cold?” (48 chars)
Wrong -- Nia looked in her boot but found nothing there. She found her mitten near the slide, not in the boot. hint: “Which search place had no mitten inside?” (40 chars)

Q4. What can you figure out about how Nia's mitten came off?

not cued key 42 chars vs mean distractor 38.3 (1.10x), rank 1.00
Answer option Length Grader picked
A The wind blew it away from the fence 36 chars
B She took it off and put it by the slide 39 chars
C She threw it like she threw the snowball 40 chars
D It fell off when she reached for more snow KEY 42 chars with without
With story: picked D multiple_defensible
Most likely it came off while making/throwing snowballs; found near slide suggests it slipped off during play.
Story withheld: picked D multiple_defensible
Reaching for snow is a plausible way a mitten slips off unnoticed.
Explanations and hints
Key -- The story says 'When she reached for another handful of snow, she felt cold. One mitten was gone!' This shows the mitten came off while she was reaching.
Wrong -- The story does not say Nia took off her mitten on purpose. It fell off by accident when she reached. hint: “What was Nia doing when she noticed it was gone?” (48 chars)
Wrong -- The story never mentions wind blowing the mitten away. It came off when Nia reached for snow. hint: “Reread what Nia was doing right before she felt cold.” (53 chars)
Wrong -- Nia threw a snowball at the fence, but the story does not say she threw her mitten. It fell off accidentally. hint: “Did Nia throw her mitten on purpose or by accident?” (51 chars)
Judge rubric
answer key validtrueAll four keys are directly supported by the story text.
groundedtrueEvery question can be answered from the story.
distractor qualityfullDistractors draw on story elements (fence, oak tree, boot, snowball) and each is wrong for a distinct reason, matched in length and register.
grade fit5Simple vocabulary and short sentences fit grade 1; the inference question is appropriately scaffolded.
question type fittrueQ0 and Q1 are fact_detail, Q2 explains why (summary/main_idea-ish reasoning), Q3 is inference. Distribution matches the request.
explanation quality5Explanations quote the story; hints point to relevant sentences without revealing answers.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated color of mittens.
Q2 fact_detail clean Directly stated location where mitten was found.
Q3 main_idea clean Asks reader to explain the reason for Nia's smile using the story's ending.
Q4 inference clean Reader deduces the mitten fell off during reaching, not directly stated as cause.
grade 1 · g1-lost-mitten · rep 2

claude-haiku-4-5

FAILformat_valid

The Lost Red Mitten -- slice-of-life, gentle, a snowy park

Lexile not assigned Grade 1 range BR-570L 95 words, 14 sentences, mean 6.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced
75% cued-pick (n=4) 0% informed guess (n=0) 0% slip proxy 100% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Nia wore her red mittens to the park. The snow was deep and white. She made a snowball and threw it at the fence. When she reached for another handful of snow, she felt cold. One mitten was gone! Her left hand was bare. Nia looked by the fence. She looked by the big oak tree. She even looked inside her boot. No red mitten anywhere. Then she saw a flash of red in the snow near the slide. There it was! Nia pulled the mitten back on and smiled. Her hand was warm again.

Q1. What color mittens did Nia wear to the park?

not cued key 12 chars vs mean distractor 19.3 (0.62x), rank 4.00
Answer option Length Grader picked
A Blue mittens. 13 chars
B Red mittens. KEY 12 chars with without
C White mittens. 14 chars
D Mittens she found by the slide. 31 chars
With story: picked B answerable
Story states red mittens.
Story withheld: picked B unanswerable
Guessing red based on later mention of red mitten.
Explanations and hints
Key -- The story says 'Nia wore her red mittens to the park.' This tells us directly what color they were.
Wrong -- The story describes the snow as white, not Nia's mittens. This confuses the snow's color with the mitten color. hint: “Reread the very first sentence about Nia.” (41 chars)
Wrong -- The story never mentions blue mittens. This is a color that does not appear in the story. hint: “Look at what Nia wore at the beginning.” (39 chars)
Wrong -- Nia found her mitten by the slide, but she wore her mittens before going to the park. hint: “Think about what happened first in the story.” (45 chars)

Q2. Where did Nia find her lost red mitten?

not cued key 27 chars vs mean distractor 32.0 (0.84x), rank 4.00
Answer option Length Grader picked
A Near the slide in the snow. KEY 27 chars with
B By the fence where she threw snow. 34 chars
C Inside her boot on the ground. 30 chars
D Under the big oak tree branches. 32 chars without
With story: picked A answerable
Found near the slide.
Story withheld: picked D unanswerable
No story provided; guessing based on plausible detail.
Explanations and hints
Key -- The story says 'Then she saw a flash of red in the snow near the slide. There it was!' This shows where the mitten was.
Wrong -- Nia looked inside her boot, but the mitten was not there. She found it by the slide instead. hint: “Reread where Nia searched for the mitten.” (41 chars)
Wrong -- Nia looked by the fence first, but the story says 'No red mitten anywhere' there. It was by the slide. hint: “Where did Nia look that did not work?” (37 chars)
Wrong -- Nia looked by the big oak tree, but found nothing there. The mitten was actually near the slide. hint: “Which tree did Nia search near?” (31 chars)

Q3. Why did Nia feel happy at the end of the story?

not cued key 43 chars vs mean distractor 39.7 (1.08x), rank 1.00
Answer option Length Grader picked
A She found her mitten and her hand was warm. KEY 43 chars with without
B She made a big snowball at the fence. 37 chars
C She looked inside her boot and found snow. 42 chars
D She stopped looking for her lost mitten. 40 chars
With story: picked A answerable
She smiled after finding it and warming her hand.
Story withheld: picked A unanswerable
Most logical happy ending given question 1.
Explanations and hints
Key -- The story ends with 'Nia pulled the mitten back on and smiled. Her hand was warm again.' This shows she was happy because she found it and felt warm.
Wrong -- Nia made a snowball early in the story, but that is not why she smiled at the end. hint: “What happened right before Nia smiled?” (38 chars)
Wrong -- Nia looked in her boot but found no mitten. Finding snow would not make her smile. hint: “Think about what made Nia smile at the very end.” (48 chars)
Wrong -- Nia did not stop looking; she kept searching and found the mitten. That is what made her happy. hint: “Did Nia give up or keep searching?” (34 chars)

Q4. What can you figure out about how Nia's mitten came off?

not cued key 43 chars vs mean distractor 40.0 (1.08x), rank 2.00
Answer option Length Grader picked
A The big oak tree pulled it off her hand. 40 chars
B She threw it at the fence like the snowball. 44 chars
C It fell off when she reached for more snow. KEY 43 chars with without
D She left it by the slide on purpose. 36 chars
With story: picked C answerable
She noticed it missing when reaching for more snow.
Story withheld: picked C unanswerable
Most plausible inference among options.
Explanations and hints
Key -- The story says 'When she reached for another handful of snow, she felt cold. One mitten was gone!' This shows the mitten came off while she was reaching.
Wrong -- The story says she threw a snowball at the fence, not her mitten. The mitten came off by accident. hint: “What did Nia throw at the fence?” (32 chars)
Wrong -- Trees cannot pull off mittens. The mitten came off when Nia reached into the snow. hint: “What was Nia doing when she noticed it was gone?” (48 chars)
Wrong -- Nia did not leave it on purpose; she lost it by accident and had to search for it. hint: “Did Nia mean to lose her mitten?” (32 chars)
Judge rubric
answer key validtrueAll four keys are directly supported by the text.
groundedtrueEvery question can be answered from the story.
distractor qualityfullDistractors draw on story elements (fence, oak tree, boot, slide) and are each wrong for a different reason, matched in register.
grade fit5Vocabulary and sentence length are appropriate for grade 1; reasoning demand is light.
question type fittrueQ0 and Q1 are fact_detail, Q2 is main_idea/summarize, Q3 is inference from clues -- matches the requested 2/1/1.
explanation quality5Explanations quote the story directly; hints point to the relevant section without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated color of mittens.
Q2 fact_detail clean Directly stated location where mitten was found.
Q3 main_idea clean Asks reader to explain why Nia felt happy, summarizing the ending.
Q4 inference clean Reader must deduce that the mitten came off during the reach, not stated explicitly.
grade 1 · g1-lost-mitten · rep 3

claude-haiku-4-5

FAILformat_valid

The Lost Red Mitten -- slice-of-life, gentle, a snowy park

Lexile not assigned Grade 1 range BR-570L 95 words, 14 sentences, mean 6.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced
0% cued-pick (n=1) 100% informed guess (n=3) 0% slip proxy 25% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Nia wore her red mittens to the park. The snow was deep and white. She made a snowball and threw it at the fence. When she reached for another handful of snow, she felt cold. One mitten was gone! Her left hand was bare. Nia looked by the fence. She looked by the big oak tree. She even looked inside her boot. No red mitten anywhere. Then she saw a flash of red in the snow near the slide. There it was! Nia pulled the mitten back on and smiled. Her hand was warm again.

Q1. What color were Nia's mittens at the start of the story?

not cued key 12 chars vs mean distractor 33.0 (0.36x), rank 4.00
Answer option Length Grader picked
A White mittens like the snow. 28 chars
B Black mittens she wore in her boot. 35 chars
C Blue mittens she found by the fence. 36 chars
D Red mittens. KEY 12 chars with without
With story: picked D answerable
Story states she wore red mittens.
Story withheld: picked D answerable
The question implies a specific color; red matches later references.
Explanations and hints
Key -- The story says 'Nia wore her red mittens to the park.' This tells us the color right away.
Wrong -- The story describes the snow as white, but Nia's mittens are red, not white. hint: “Reread the very first sentence of the story.” (44 chars)
Wrong -- Nia found her red mitten by the slide, not a blue mitten by the fence. hint: “What color does the story mention first?” (40 chars)
Wrong -- Nia looked inside her boot for her lost mitten, but it was not there and was not black. hint: “Look at the opening sentence about what she wore.” (49 chars)

Q2. Where did Nia find her lost red mitten?

not cued key 27 chars vs mean distractor 37.7 (0.72x), rank 4.00
Answer option Length Grader picked
A Near the slide in the snow. KEY 27 chars with
B By the big oak tree where she looked. 37 chars
C Inside her boot after she checked. 34 chars
D By the fence where she threw the snowball. 42 chars without
With story: picked A answerable
She saw red near the slide.
Story withheld: picked D multiple_defensible
Without the story, the fence/snowball location seems most specific.
Explanations and hints
Key -- The story says 'Then she saw a flash of red in the snow near the slide.'
Wrong -- Nia looked by the oak tree but did not find her mitten there. hint: “Where did she see the flash of red?” (35 chars)
Wrong -- The story says 'She even looked inside her boot. No red mitten anywhere.' It was not there. hint: “Which place did not have the mitten?” (36 chars)
Wrong -- Nia looked by the fence first but found nothing. The mitten was actually near the slide. hint: “Where did she look first without finding it?” (44 chars)

Q3. Why did Nia's hand feel cold in the middle of the story?

not cued key 42 chars vs mean distractor 39.0 (1.08x), rank 1.00
Answer option Length Grader picked
A The snow was too deep and white outside. 40 chars
B One mitten was missing from her left hand. KEY 42 chars with without
C She looked by the big oak tree too long. 40 chars
D She threw a snowball and got too wet. 37 chars
With story: picked B answerable
Her left hand was bare because a mitten was gone.
Story withheld: picked B answerable
A cold hand suggests a missing mitten.
Explanations and hints
Key -- The story explains 'When she reached for another handful of snow, she felt cold. One mitten was gone!'
Wrong -- Nia threw a snowball but the story says she felt cold because her mitten was gone, not from getting wet. hint: “What happened to one of her mittens?” (36 chars)
Wrong -- While the snow was deep and white, that is not why her hand felt cold. Her mitten was missing. hint: “What was she wearing on her hands?” (34 chars)
Wrong -- Looking by the tree did not make her hand cold. Her hand was cold because she lost a mitten. hint: “What did she lose while playing?” (32 chars)

Q4. What can you figure out about how Nia's mitten came off?

not cued key 46 chars vs mean distractor 42.3 (1.09x), rank 1.00
Answer option Length Grader picked
A It got stuck inside her boot when she walked. 45 chars
B It fell off while she was playing in the snow. KEY 46 chars with without
C Someone at the park took it from her hand. 42 chars
D She took it off and put it by the slide. 40 chars
With story: picked B answerable
It came off while she played in the snow.
Story withheld: picked B answerable
Mittens typically come off during active play in snow.
Explanations and hints
Key -- The story shows Nia made a snowball and reached for more snow, then discovered her mitten was gone. This tells us it came off during play.
Wrong -- The story does not say Nia took off her mitten on purpose. She lost it without meaning to. hint: “Did Nia choose to remove her mitten?” (36 chars)
Wrong -- The story mentions no other people. Nia lost her mitten while playing, not from someone taking it. hint: “Does the story mention other children or people?” (48 chars)
Wrong -- Nia looked inside her boot but the mitten was not there. It ended up near the slide instead. hint: “Where was the mitten actually found?” (36 chars)
Judge rubric
answer key validtrueEvery question has one clearly correct answer supported by the text.
groundedtrueAll questions can be answered from the story.
distractor qualityfullDistractors use story elements (fence, oak tree, boot, slide) and are each wrong for different reasons, matched in length and tone.
grade fit5Vocabulary and sentence complexity are appropriate for grade 1.
question type fittrueQ0 and Q1 are fact_detail, Q2 is a why/explain question, Q3 is inference. Matches the 2/1/1 distribution.
explanation quality5Explanations quote the story directly; hints guide without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated color of mittens.
Q2 fact_detail clean Directly stated location where mitten was found.
Q3 main_idea clean Asks reader to explain cause of cold hand using story info.
Q4 inference clean Requires deducing unstated cause of mitten loss from clues.
grade 1 · g1-lost-mitten · rep 4

claude-haiku-4-5

FAILformat_validdistractor_quality

The Lost Red Mitten -- slice-of-life, gentle, a snowy park

Lexile not assigned Grade 1 range BR-570L 95 words, 14 sentences, mean 6.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +57 chars past the 50-char cap.

0% cued-pick (n=1) 100% informed guess (n=3) 0% slip proxy 25% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruepartial
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Nia wore her red mittens to the park. The snow was deep and white. She made a snowball and threw it at the fence. When she reached for another handful of snow, she felt cold. One mitten was gone! Her left hand was bare. Nia looked by the fence. She looked by the big oak tree. She even looked inside her boot. No red mitten anywhere. Then she saw a flash of red in the snow near the slide. There it was! Nia pulled the mitten back on and smiled. Her hand was warm again.

Q1. What color were Nia's mittens at the start of the story?

not cued key 14 chars vs mean distractor 35.3 (0.40x), rank 4.00
Answer option Length Grader picked
A They were inside her boot the whole time. 41 chars
B They were white like the snow. 30 chars
C They were red. KEY 14 chars with without
D They were lost by the big oak tree. 35 chars
With story: picked C answerable
Story states she wore red mittens.
Story withheld: picked C answerable
Red is the color mentioned in later questions.
Explanations and hints
Key -- The story says 'Nia wore her red mittens to the park.' This tells us the mittens were red.
Wrong -- The story describes the snow as white, but Nia's mittens are red, not white. hint: “Reread the first sentence about what Nia wore.” (46 chars)
Wrong -- While Nia looked by the oak tree, the mittens were red in color, not lost at that location. hint: “The question asks about color, not location.” (44 chars)
Wrong -- Nia looked inside her boot searching for the mitten, but the mittens started out red and on her hands. hint: “Think about what Nia wore when she arrived.” (43 chars)

Q2. Where did Nia find her missing red mitten?

not cued key 40 chars vs mean distractor 44.3 (0.90x), rank 3.00
Answer option Length Grader picked
A She found it inside her boot after searching. 45 chars
B She found it near the slide in the snow. KEY 40 chars with
C She found it by the fence where she threw the snowball. 55 chars
D She found it by the big oak tree. 33 chars without
With story: picked B answerable
She saw red near the slide.
Story withheld: picked D multiple_defensible
Without the story, the oak tree is a plausible landmark mentioned.
Explanations and hints
Key -- The story says 'Then she saw a flash of red in the snow near the slide. There it was!'
Wrong -- Nia looked by the fence but did not find the mitten there. She found it near the slide instead. hint: “Where did Nia see a flash of red color?” (39 chars)
Wrong -- The story says Nia looked inside her boot but found no mitten there. It was near the slide. hint: “Reread where Nia actually discovered the mitten.” (48 chars)
Wrong -- Nia looked by the oak tree but the mitten was not there. It was near the slide. hint: “Which place had a flash of red?” (31 chars)

Q3. Why did Nia's hand feel cold in the story?

not cued key 44 chars vs mean distractor 40.7 (1.08x), rank 1.00
Answer option Length Grader picked
A The snow was too deep and white outside. 40 chars
B She looked inside her boot and got wet. 39 chars
C One of her mittens fell off and was missing. KEY 44 chars with without
D She threw a snowball at the fence too hard. 43 chars
With story: picked C answerable
Her mitten was missing so hand was bare.
Story withheld: picked C answerable
A missing mitten would leave her hand exposed and cold.
Explanations and hints
Key -- The story explains that when Nia reached for more snow, she felt cold because one mitten was gone and her left hand was bare.
Wrong -- Throwing the snowball did not cause her hand to feel cold. Her hand was cold because the mitten was missing. hint: “What happened to Nia's hand after she reached for snow?” (55 chars)
Wrong -- While the snow was deep and white, that is not why her hand felt cold. The missing mitten made her hand cold. hint: “What was missing from Nia's hand?” (33 chars)
Wrong -- Looking in her boot did not make her hand cold. Her hand was cold because she lost her mitten. hint: “What protected Nia's hands from the cold?” (41 chars)

Q4. What can you figure out about how Nia's mitten came off?

length-cued key 54 chars vs mean distractor 41.7 (1.30x), rank 1.00
Answer option Length Grader picked
A The fence pulled the mitten off her hand. 41 chars
B It probably fell off when she was playing in the snow. KEY 54 chars with without
C The oak tree knocked the mitten into the snow. 46 chars
D She took it off on purpose and hid it. 38 chars
With story: picked B answerable
Most likely fell off while playing in snow.
Story withheld: picked B answerable
Mittens typically fall off unnoticed during play.
Explanations and hints
Key -- The story shows Nia making and throwing snowballs, then reaching for more snow. The mitten likely slipped off during this play.
Wrong -- The story says Nia looked for the mitten and was happy to find it, showing she did not hide it on purpose. hint: “How did Nia feel when she found the mitten?” (43 chars)
Wrong -- While Nia threw a snowball at the fence, the story does not say the fence pulled off her mitten. hint: “Where was Nia when she first noticed the mitten was gone?” (57 chars)
Wrong -- The story does not say the tree knocked off her mitten. Nia was playing with snow when she noticed it was missing. hint: “What was Nia doing before she felt cold?” (40 chars)
Judge rubric
answer key validtrueEach question has one clearly correct answer supported by the text.
groundedtrueAll questions require the story to answer.
distractor qualitypartialQ0's distractors are weak: options 2 and 3 don't even name a color, making them implausible for a color question and giving the answer away by elimination. Q1-Q3 distractors are strong and drawn from story details.
grade fit5Vocabulary (mitten, bare, flash) and reasoning demand are appropriate for grade 1.
question type fittrueQ0 and Q1 are fact_detail, Q2 explains/summarizes cause, Q3 requires inference from clues. Matches the 2/1/1 distribution.
explanation quality4Explanations quote or paraphrase the story; hints point toward the answer without stating it. Q0 hints could be slightly more subtle.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated color from the first sentence.
Q2 fact_detail clean Directly stated location where mitten was found.
Q3 main_idea clean Asks reader to explain cause of cold hand in own words.
Q4 inference clean Reader must deduce how mitten came off using story clues.
grade 1 · g1-lost-mitten · rep 5

claude-haiku-4-5

FAILformat_valid

The Lost Red Mitten -- slice-of-life, gentle, a snowy park

Lexile not assigned Grade 1 range BR-570L 95 words, 14 sentences, mean 6.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +56 chars past the 50-char cap.

0% cued-pick (n=1) 100% informed guess (n=3) 0% slip proxy 25% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Nia wore her red mittens to the park. The snow was deep and white. She made a snowball and threw it at the fence. When she reached for another handful of snow, she felt cold. One mitten was gone! Her left hand was bare. Nia looked by the fence. She looked by the big oak tree. She even looked inside her boot. No red mitten anywhere. Then she saw a flash of red in the snow near the slide. There it was! Nia pulled the mitten back on and smiled. Her hand was warm again.

Q1. What color were Nia's mittens at the start of the story?

not cued key 14 chars vs mean distractor 35.3 (0.40x), rank 4.00
Answer option Length Grader picked
A They were red. KEY 14 chars with without
B They were white like the snow. 30 chars
C They were lost by the big oak tree. 35 chars
D They were inside her boot the whole time. 41 chars
With story: picked A answerable
The story states she wore red mittens.
Story withheld: picked A answerable
Question 1 refers to the 'missing red mitten,' indicating the mittens were red.
Explanations and hints
Key -- The story says 'Nia wore her red mittens to the park.' This tells us the mittens were red from the beginning.
Wrong -- The story describes the snow as white, but Nia's mittens are red, not white. hint: “Reread the first sentence about what Nia wore.” (46 chars)
Wrong -- While Nia looked by the oak tree, the mitten was actually found near the slide, not there. hint: “Where did Nia finally find her mitten?” (38 chars)
Wrong -- Nia looked inside her boot searching for the mitten, but it was not there. hint: “What places did Nia search for the mitten?” (42 chars)

Q2. Where did Nia find her missing red mitten?

not cued key 40 chars vs mean distractor 46.7 (0.86x), rank 4.00
Answer option Length Grader picked
A She found it near the slide in the snow. KEY 40 chars with
B She found it under the big oak tree in snow. 44 chars without
C She found it inside her boot on her foot. 41 chars
D She found it by the fence where she threw the snowball. 55 chars
With story: picked A answerable
She saw a flash of red near the slide.
Story withheld: picked B multiple_defensible
Without the story, the oak tree location is a common story trope but uncertain.
Explanations and hints
Key -- The story says 'Then she saw a flash of red in the snow near the slide. There it was!' The mitten was by the slide.
Wrong -- Nia looked by the fence but did not find her mitten there. She found it somewhere else. hint: “Reread where Nia looked first and where she looked last.” (56 chars)
Wrong -- The story says Nia looked inside her boot but found no mitten there. hint: “What did Nia do after looking by the tree?” (42 chars)
Wrong -- Nia looked by the oak tree, but the mitten was not found there. It was found near a different place. hint: “What playground equipment is mentioned in the story?” (52 chars)

Q3. Why did Nia's hand feel cold in the middle of the story?

not cued key 50 chars vs mean distractor 42.0 (1.19x), rank 1.00
Answer option Length Grader picked
A One of her mittens had fallen off and was missing. KEY 50 chars with without
B The snow was too deep and white outside. 40 chars
C She looked by the big oak tree too long. 40 chars
D She threw a snowball at the fence and got wet. 46 chars
With story: picked A answerable
Her hand was bare because a mitten was gone.
Story withheld: picked A answerable
A cold hand in a mitten story typically means a mitten fell off.
Explanations and hints
Key -- The story says 'When she reached for another handful of snow, she felt cold. One mitten was gone! Her left hand was bare.' Her hand was cold because the mitten was lost.
Wrong -- While Nia did throw a snowball, the story says her hand felt cold because her mitten was gone, not because she got wet. hint: “What happened right before Nia felt cold?” (41 chars)
Wrong -- The deep white snow is described in the story, but it is not the reason her hand felt cold. hint: “What was Nia wearing on her hands?” (34 chars)
Wrong -- Looking by the tree did not make her hand cold. The cold hand was caused by something she was missing. hint: “What did Nia discover was missing from her body?” (48 chars)

Q4. What can you figure out about what happened to Nia's mitten?

not cued key 55 chars vs mean distractor 52.0 (1.06x), rank 1.00
Answer option Length Grader picked
A Someone at the park took it and put it by the slide. 52 chars
B It got stuck inside her boot and she forgot about it. 53 chars
C She took it off and hid it by the fence on purpose. 51 chars
D It fell off her hand while she was playing in the snow. KEY 55 chars with without
With story: picked D answerable
The mitten came off while she played, ending up in the snow.
Story withheld: picked D answerable
Mittens usually fall off during play, consistent with the cold hand clue.
Explanations and hints
Key -- The story shows Nia reached for snow, felt cold, and found her mitten gone. This tells us it fell off during her play, not that she lost it on purpose.
Wrong -- The story says Nia looked for the mitten and was happy to find it. She would not hide something she was searching for. hint: “How did Nia feel when she found her mitten?” (43 chars)
Wrong -- The story does not mention anyone else at the park or anyone taking the mitten. It simply fell off during play. hint: “Are other people mentioned in this story?” (41 chars)
Wrong -- Nia looked inside her boot and found nothing there. The mitten was not in her boot. hint: “Where did Nia search for the mitten?” (36 chars)
Judge rubric
answer key validtrueEach question has one clearly correct answer supported by the text.
groundedtrueAll questions can be answered from the story details.
distractor qualityfullDistractors draw on story elements (fence, oak tree, boot, slide) and each is wrong for a different reason, matched in register for grade 1.
grade fit5Vocabulary and sentence length match early grade 1; questions ask about concrete details.
question type fittrueQ0 and Q1 are fact_detail, Q2 is a summary/explanation of why (main_idea-ish reasoning about cause), Q3 is inference. Distribution matches request.
explanation quality4Explanations quote the story directly; hints point without revealing. Q0's WRONG 2/3 explanations awkwardly conflate 'color' with location but still guide correctly.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Direct color fact from opening line.
Q2 fact_detail clean Location fact stated in text.
Q3 main_idea clean Asks reader to explain cause of cold hand using text.
Q4 inference clean Requires deducing mitten fell off; not directly stated.
grade 4 · g4-locker · rep 1

claude-haiku-4-5

NOT JUDGED

The Locker That Wouldn't Open -- mystery, curious, a middle school hallway

Lexile not assigned Grade 4 range 415L-1160L 117 words, 9 sentences, mean 13.0

No format record.

Story text (held constant across models)

Marcus spun his locker combination three times, but the door would not budge. He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September. He pressed his ear to the cold metal and tried again. This time he heard a faint crinkling sound inside, like paper being squeezed. Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time. Marcus pushed the door inward first, just a little, then turned the handle. It swung open easily. His coat tumbled out onto the floor, and he laughed at himself.

Parse failure -- unparseable_json
Expected ',' or '}' after property value in JSON at position 3948 (line 71 column 11)
Raw response
```json
{
  "questions": [
    {
      "question": "What was Marcus's locker combination?",
      "answers": [
        {
          "correct_answer": "12, 4, and 30",
          "explanation": "The story states that Marcus was sure he had the right numbers because he had used them every day since September."
        },
        {
          "wrong_answer": "12, 30, and 4 in that exact order",
          "explanation": "While these are the correct numbers, the story does not specify that order matters for the combination.",
          "hint_explanation": "The story lists the numbers but doesn't emphasize their sequence."
        },
        {
          "wrong_answer": "3, 12, 4, and 30 because he spun it three times",
          "explanation": "Marcus spun the combination three times as an action, not as part of the actual combination numbers.",
          "hint_explanation": "Spinning three times was what Marcus did, not the combination."
        },
        {
          "wrong_answer": "A number he had forgotten since September started",
          "explanation": "The story clearly says Marcus was sure he had the right numbers and used them every day since September.",
          "hint_explanation": "Reread what Marcus remembered about his combination."
        }
      ]
    },
    {
      "question": "Why did Marcus hear a crinkling sound when he pressed his ear to the locker?",
      "answers": [
        {
          "correct_answer": "His winter coat sleeve was jammed against the door inside",
          "explanation": "The story explains that Marcus had stuffed his winter coat into the locker in a hurry, and the thick sleeve must have been jammed against the door, causing the crinkling sound when he pressed on it."
        },
        {
          "wrong_answer": "Someone had left crumpled paper inside his locker",
          "explanation": "While the sound was like paper crinkling, the actual cause was the coat sleeve, not loose paper.",
          "hint_explanation": "What did Marcus stuff in his locker yesterday?"
        },
        {
          "wrong_answer": "The metal door was old and made squeaking noises",
          "explanation": "The story describes a crinkling sound like paper, not a squeaking metal sound from the door itself.",
          "hint_explanation": "The sound was like paper being squeezed, not metal."
        },
        {
          "wrong_answer": "His books were pressing against the inside of the door",
          "explanation": "The story specifically mentions the winter coat as the cause, not books.",
          "hint_explanation": "What did Marcus remember stuffing in a hurry?"
        }
      ]
    },
    {
      "question": "What can you figure out about why Marcus's locker door would not open at first?",
      "answers": [
        {
          "correct_answer": "The coat sleeve was blocking the door from opening fully",
          "explanation": "The story shows that when Marcus pushed the door inward first and then turned the handle, it swung open easily and the coat tumbled out, revealing that the sleeve had been jamming it."
        },
        {
          "wrong_answer": "Marcus had forgotten his combination and could not unlock it",
          "explanation": "Marcus was sure he had the right numbers and used them every day, so he knew the combination.",
          "hint_explanation": "What was Marcus certain about regarding his numbers?"
        },
        {
          "wrong_answer": "The locker was broken and needed to be repaired",
          "explanation": "The locker worked fine once Marcus pushed the door inward first; it was not broken.",
          "hint_explanation": "What happened when Marcus pushed inward first?"
        },
        {
          "wrong_answer": "Someone had locked the locker from the inside on purpose",
          "explanation": "The story gives no indication that anyone else was involved; the problem was simply the coat jamming the door."
          "hint_explanation": "What did Marcus do in a hurry yesterday?"
        }
      ]
    },
    {
      "question": "Was Marcus's choice to stuff his coat into the locker a good decision? Why or why not?",
      "answers": [
        {
          "correct_answer": "No, because it jammed the door and prevented him from opening it",
          "explanation": "The story shows that stuffing the coat in a hurry caused the thick sleeve to jam against the door, making it impossible to open until Marcus figured out the problem."
        },
        {
          "wrong_answer": "Yes, because it kept his coat safe and protected inside",
          "explanation": "While the coat was stored, the decision created a problem that made the locker unusable, so it was not a good choice overall.",
          "hint_explanation": "What problem did stuffing the coat cause?"
        },
        {
          "wrong_answer": "Yes, because Marcus laughed at himself afterward",
          "explanation": "Marcus laughed because he realized his mistake, not because the decision was good. Laughing at a problem does not make the choice wise.",
          "hint_explanation": "Why did Marcus laugh at the end?"
        },
        {
          "wrong_answer": "No, because the coat was too thick to fit in the locker",
          "explanation": "The coat did fit in the locker; the problem was how it was positioned, not that it was too large.",
          "hint_explanation": "Did the coat fit, or was it positioned badly?"
        }
      ]
    }
  ]
}
```
grade 4 · g4-locker · rep 2

claude-haiku-4-5

FAILformat_valid

The Locker That Wouldn't Open -- mystery, curious, a middle school hallway

Lexile not assigned Grade 4 range 415L-1160L 117 words, 9 sentences, mean 13.0
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +63 chars past the 50-char cap.

100% cued-pick (n=2) 100% informed guess (n=2) 0% slip proxy 50% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Marcus spun his locker combination three times, but the door would not budge. He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September. He pressed his ear to the cold metal and tried again. This time he heard a faint crinkling sound inside, like paper being squeezed. Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time. Marcus pushed the door inward first, just a little, then turned the handle. It swung open easily. His coat tumbled out onto the floor, and he laughed at himself.

Q1. What was Marcus's locker combination?

not cued key 13 chars vs mean distractor 41.7 (0.31x), rank 4.00
Answer option Length Grader picked
A 3, 12, 4, and 30 because he spun it three times 47 chars
B A number he could not remember from September 45 chars
C 12, 4, and 30 KEY 13 chars with without
D 12, 30, and 4 in that exact order 33 chars
With story: picked C answerable
Story states 12, 4, 30.
Story withheld: picked C multiple_defensible
Without the story, 12-4-30 is the most standard combination format.
Explanations and hints
Key -- The story states that Marcus was sure he had the right numbers because he had used them every day since September.
Wrong -- While these are the correct numbers, the story does not specify that order matters for the combination. hint: “The story lists the numbers but not their required sequence.” (60 chars)
Wrong -- Marcus spun the combination three times as an action, not as part of the actual combination numbers. hint: “Spinning three times was what Marcus did, not part of the code.” (63 chars)
Wrong -- The story clearly says Marcus was sure he had the right numbers, meaning he remembered them correctly. hint: “Marcus was confident about the numbers he used.” (47 chars)

Q2. Why did Marcus hear a crinkling sound when he pressed his ear to the locker?

not cued key 57 chars vs mean distractor 48.7 (1.17x), rank 1.00
Answer option Length Grader picked
A Someone had put crumpled paper inside his locker 48 chars
B His books were shifting around inside the locker 48 chars
C The metal door was old and made noise when touched 50 chars
D His winter coat sleeve was jammed against the door inside KEY 57 chars with without
With story: picked D answerable
The coat sleeve was jammed inside making the crinkling.
Story withheld: picked D multiple_defensible
The coat sleeve jammed against the door would crinkle when pressed.
Explanations and hints
Key -- Marcus remembered that he had stuffed his winter coat into the locker in a hurry, and the thick sleeve must have been jammed against the door, causing the crinkling sound when he tried the combination.
Wrong -- While the story mentions a crinkling sound like paper, Marcus discovered it was his coat, not paper left by someone else. hint: “What did Marcus stuff into the locker yesterday?” (48 chars)
Wrong -- The crinkling sound was specifically like paper being squeezed, not the sound of old metal. hint: “The sound was described as like paper, not metal.” (49 chars)
Wrong -- The story focuses on the winter coat as the cause of the problem, not books moving around. hint: “What did Marcus put in the locker in a hurry?” (45 chars)

Q3. What can you figure out about why the locker door would not open at first?

not cued key 56 chars vs mean distractor 55.0 (1.02x), rank 1.50
Answer option Length Grader picked
A Someone had locked the door on purpose to play a prank 54 chars
B The lock was broken and needed to be fixed by the school 56 chars
C Marcus forgot his combination and could not remember it 55 chars
D The coat sleeve was blocking the door from opening fully KEY 56 chars with without
With story: picked D answerable
The coat sleeve blocked the door.
Story withheld: picked D answerable
The coat sleeve blocking the door fits with the crinkling sound clue.
Explanations and hints
Key -- The story shows that when Marcus pushed the door inward first and then turned the handle, it swung open easily and his coat tumbled out, revealing that the sleeve had been jamming the door.
Wrong -- The story says Marcus was sure he had the right numbers because he used them every day since September. hint: “How confident was Marcus about his combination?” (47 chars)
Wrong -- The locker opened easily once Marcus pushed it inward first, showing the lock was not broken. hint: “Did the locker eventually open successfully?” (44 chars)
Wrong -- The story gives no evidence of a prank; the problem was simply the coat sleeve jamming the door. hint: “What was the actual cause of the stuck door?” (44 chars)

Q4. Was Marcus's decision to stuff his coat into the locker in a hurry a good choice?

not cued key 62 chars vs mean distractor 54.0 (1.15x), rank 1.00
Answer option Length Grader picked
A Yes, because it kept his coat safe and protected 48 chars
B Yes, because he remembered to do it before the bell rang 56 chars
C No, because it caused the door to jam and he could not open it KEY 62 chars with without
D No, because winter coats should never go in school lockers 58 chars
With story: picked C answerable
Stuffing caused the jam.
Story withheld: picked C answerable
Stuffing the coat in a hurry caused the jam problem.
Explanations and hints
Key -- The story shows that Marcus's hurried action of stuffing the coat into the locker caused the sleeve to jam against the door, preventing it from opening until he figured out the problem.
Wrong -- While the coat was stored, the hurried way he did it created a problem that prevented him from accessing his locker. hint: “What problem did stuffing the coat cause?” (41 chars)
Wrong -- Doing something before the bell does not make a choice good if it causes problems later, as it did here. hint: “Did the timing of his action prevent all problems?” (50 chars)
Wrong -- The story does not suggest that coats should never go in lockers, only that the hurried way Marcus did it caused trouble. hint: “What was the real issue with how he stored it?” (46 chars)
Judge rubric
answer key validtrueAll keys are defensible. Q0 the combination is explicitly 12, 4, 30; Q1 the coat sleeve caused the crinkling; Q2 the sleeve blocked the door; Q3 the hurried stuffing caused the jam.
groundedtrueEvery question draws directly from story details.
distractor qualityfullDistractors are plausible, drawn from story elements (paper sound, combination, prank possibility), and each wrong for a distinct reason.
grade fit4Vocabulary (jammed, combination, crinkling) and reasoning demands are appropriate for grade 4.
question type fittrueQ0=fact_detail, Q1=cause/explanation, Q2=inference, Q3=evaluate — matches requested distribution.
explanation quality4Explanations reference story events clearly; hints point without revealing directly, though a couple of hints (Q1 wrong 1 and 3) come close to naming the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated numbers in the text.
Q2 inference clean Explains cause of the sound — arguably close to fact_detail since text states the sleeve was jammed, but the crinkling connection requires linking clues.
Q3 inference clean Requires connecting the push-inward action to the sleeve blocking.
Q4 evaluate_analyze clean Asks reader to judge a character's choice.
grade 4 · g4-locker · rep 3

claude-haiku-4-5

FAILformat_valid

The Locker That Wouldn't Open -- mystery, curious, a middle school hallway

Lexile not assigned Grade 4 range 415L-1160L 117 words, 9 sentences, mean 13.0
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +53 chars past the 50-char cap.

100% cued-pick (n=2) 100% informed guess (n=2) 0% slip proxy 50% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Marcus spun his locker combination three times, but the door would not budge. He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September. He pressed his ear to the cold metal and tried again. This time he heard a faint crinkling sound inside, like paper being squeezed. Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time. Marcus pushed the door inward first, just a little, then turned the handle. It swung open easily. His coat tumbled out onto the floor, and he laughed at himself.

Q1. What was Marcus's locker combination?

not cued key 13 chars vs mean distractor 37.7 (0.35x), rank 4.00
Answer option Length Grader picked
A 12, 4, and 30 KEY 13 chars with without
B 12, 30, and 4 in that exact order 33 chars
C Numbers he had forgotten since yesterday 40 chars
D A combination he changed before the bell 40 chars
With story: picked A answerable
The story states his numbers were 12, 4, 30.
Story withheld: picked A multiple_defensible
Without the story, 12, 4, and 30 seems the most neutral factual statement.
Explanations and hints
Key -- The story states: 'He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September.'
Wrong -- While these are the correct numbers, the story lists them as 12, 4, 30, not 12, 30, 4. The order matters for a combination. hint: “Reread the exact numbers Marcus used.” (37 chars)
Wrong -- Marcus was certain he had the right numbers. The problem was not that he forgot them, but that his coat was blocking the door. hint: “What did Marcus remember about his coat?” (40 chars)
Wrong -- The story says Marcus used the same numbers every day since September, meaning he never changed them. hint: “How long had Marcus used these numbers?” (39 chars)

Q2. Why did Marcus hear a crinkling sound when he pressed his ear to the locker?

not cued key 56 chars vs mean distractor 51.3 (1.09x), rank 1.00
Answer option Length Grader picked
A Someone had hidden paper inside his locker yesterday 52 chars
B His combination numbers were written on paper inside 52 chars
C His winter coat sleeve was jammed inside the locker door KEY 56 chars with without
D The metal door was old and made noise when touched 50 chars
With story: picked C answerable
The coat sleeve was jammed and made the crinkling sound.
Story withheld: picked C multiple_defensible
A jammed coat sleeve would explain crinkling and connects to later questions.
Explanations and hints
Key -- The story explains: 'Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time.'
Wrong -- The crinkling sound came from the coat, not from hidden paper. Marcus figured out the real cause when he remembered stuffing his coat in. hint: “What did Marcus put in his locker yesterday?” (44 chars)
Wrong -- The crinkling sound was specifically like paper being squeezed, not a general metal noise. It came from the coat, not the door itself. hint: “What did the sound remind Marcus of?” (36 chars)
Wrong -- The story does not mention Marcus keeping his combination numbers written down. The crinkling came from his coat being squeezed. hint: “What was actually making the crinkling noise?” (45 chars)

Q3. What can you figure out about why the locker door would not open at first?

not cued key 56 chars vs mean distractor 53.0 (1.06x), rank 2.00
Answer option Length Grader picked
A Marcus had forgotten his combination numbers that morning 57 chars
B Someone had locked the locker on purpose to trick him 53 chars
C The thick coat sleeve was blocking the door from opening KEY 56 chars with without
D The locker lock was broken and needed to be fixed 49 chars
With story: picked C answerable
The coat sleeve was blocking the door.
Story withheld: picked C answerable
The coat sleeve blocking the door fits the story's cause-and-effect.
Explanations and hints
Key -- The story shows that when Marcus pushed the door inward first and then turned the handle, 'It swung open easily' and 'His coat tumbled out onto the floor.' This proves the coat was physically blocking it.
Wrong -- Marcus was confident he had the right numbers. The problem was not his memory but the physical blockage from his coat. hint: “Was Marcus unsure about his combination?” (40 chars)
Wrong -- The locker worked fine once Marcus removed the coat. If the lock were broken, pushing the door would not have helped. hint: “Did the locker work after the coat came out?” (44 chars)
Wrong -- There is no mention of anyone else locking the locker. Marcus himself had caused the problem by stuffing his coat in too quickly. hint: “Who put the coat in the locker?” (31 chars)

Q4. Was Marcus's decision to stuff his coat into the locker in a hurry a good choice?

not cued key 62 chars vs mean distractor 53.7 (1.16x), rank 1.00
Answer option Length Grader picked
A No, because it blocked his locker door and caused him problems KEY 62 chars with without
B No, because he should have left his coat at home instead 56 chars
C Yes, because it kept his coat safe and protected 48 chars
D Yes, because he remembered to use his correct combination 57 chars
With story: picked A answerable
It caused the door to jam.
Story withheld: picked A answerable
Stuffing the coat in a hurry caused the door to jam, so it was not a good choice.
Explanations and hints
Key -- The story shows that rushing to stuff the coat 'in a hurry before the bell' directly caused the door to jam, which prevented Marcus from opening his locker the next day.
Wrong -- While the coat was stored, the rushed way Marcus did it created a problem. A good choice would have been to place it more carefully. hint: “What problem did the hurried stuffing cause?” (44 chars)
Wrong -- Remembering his combination did not solve the real problem. The issue was that his coat was physically blocking the door. hint: “What was actually preventing the door from opening?” (51 chars)
Wrong -- The story does not suggest Marcus should not have brought his coat. The problem was how he stored it, not that he had it. hint: “Was the coat itself the problem or how it was stored?” (53 chars)
Judge rubric
answer key validtrueAll four keys are defensible. Q0's distractor about order is a bit tricky but the correct answer '12, 4, and 30' matches the story exactly, while the distractor explicitly changes the order.
groundedtrueEvery question is answerable from the story text.
distractor qualityfullDistractors are plausible, drawn from story elements (coat, combination, locker), and each wrong for a distinct reason. Length and register match.
grade fit4Vocabulary (crinkling, combination, jammed) and reasoning demands fit grade 4. The evaluate question is appropriately nuanced.
question type fittrueQ0 = fact_detail, Q1 = cause/explanation, Q2 = inference, Q3 = evaluate_analyze. Distribution matches request.
explanation quality5Explanations quote the story directly; hints point toward relevant details without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly asks for a stated fact (the combination numbers).
Q2 inference clean Asks why the sound occurred; answer requires connecting the coat detail to the sound, though the story is fairly explicit — leans inference/cause.
Q3 inference clean Requires piecing together clues (push door in, coat tumbled out) to conclude the coat blocked the door.
Q4 evaluate_analyze clean Judges the quality of Marcus's choice to rush stuffing his coat.
grade 4 · g4-locker · rep 4

claude-haiku-4-5

FAILformat_valid

The Locker That Wouldn't Open -- mystery, curious, a middle school hallway

Lexile not assigned Grade 4 range 415L-1160L 117 words, 9 sentences, mean 13.0
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +52 chars past the 50-char cap.

0% cued-pick (n=1) 100% informed guess (n=3) 0% slip proxy 25% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Marcus spun his locker combination three times, but the door would not budge. He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September. He pressed his ear to the cold metal and tried again. This time he heard a faint crinkling sound inside, like paper being squeezed. Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time. Marcus pushed the door inward first, just a little, then turned the handle. It swung open easily. His coat tumbled out onto the floor, and he laughed at himself.

Q1. What was Marcus's locker combination?

not cued key 13 chars vs mean distractor 38.3 (0.34x), rank 4.00
Answer option Length Grader picked
A 12, 4, and 30 KEY 13 chars with
B Numbers he had forgotten since yesterday 40 chars
C 12, 30, and 4 in that exact order 33 chars without
D A combination that changed after September 42 chars
With story: picked A answerable
Story states 12, 4, 30.
Story withheld: picked C multiple_defensible
Order-specific option C is likely correct if order matters.
Explanations and hints
Key -- The story states: 'He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September.'
Wrong -- While these are the correct numbers, the story lists them as 12, 4, 30, not in the order given here. hint: “Check the exact order mentioned in the story.” (45 chars)
Wrong -- Marcus remembered his combination perfectly; he used it every day. His problem was the jammed coat, not a forgotten combination. hint: “Reread why the locker would not open.” (37 chars)
Wrong -- The story says Marcus had used the same numbers every day since September, meaning they never changed. hint: “Look for when Marcus started using this combination.” (52 chars)

Q2. What does the phrase 'jammed against the door' mean in this story?

not cued key 47 chars vs mean distractor 44.0 (1.07x), rank 1.50
Answer option Length Grader picked
A Torn into small pieces by the metal door 40 chars
B Making a loud crinkling noise inside the locker 47 chars
C Forgotten inside the locker since the morning 45 chars
D Stuck or pressed tightly so it blocked the door KEY 47 chars with without
With story: picked D answerable
Jammed means stuck tightly blocking the door.
Story withheld: picked D answerable
This is the standard meaning of the phrase.
Explanations and hints
Key -- The story explains that Marcus's coat sleeve was 'jammed against the door the whole time,' preventing it from opening because something was blocking it.
Wrong -- Jammed means stuck or blocked, not torn apart. The coat tumbled out intact when the door opened. hint: “Think about what happened to the coat.” (38 chars)
Wrong -- The crinkling sound came from the coat being squeezed, not from being jammed. Jammed refers to the physical blocking. hint: “What sound did Marcus hear through the metal?” (45 chars)
Wrong -- Jammed describes how the coat was positioned, not when it was put there. Marcus stuffed it in yesterday before the bell. hint: “When did Marcus put his coat in the locker?” (43 chars)

Q3. Why did Marcus hear a crinkling sound when he pressed his ear to the locker?

not cued key 52 chars vs mean distractor 47.0 (1.11x), rank 1.00
Answer option Length Grader picked
A His winter coat was being squeezed inside the locker KEY 52 chars with without
B Someone had left crumpled paper inside his locker 49 chars
C His combination lock was broken and falling apart 49 chars
D The metal door was rusting and making noise 43 chars
With story: picked A answerable
The coat sleeve made the crinkling sound.
Story withheld: picked A answerable
The coat being jammed explains the crinkling sound.
Explanations and hints
Key -- The story says: 'Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time.' The crinkling was the coat being compressed.
Wrong -- The story mentions the crinkling sounded like paper, but it was actually the coat. Marcus realized this when he remembered stuffing his coat in. hint: “What did Marcus stuff in the locker yesterday?” (46 chars)
Wrong -- The crinkling sound came from inside the locker, not from the door itself. It was caused by the coat, not rust. hint: “Listen to where the sound came from.” (36 chars)
Wrong -- The lock worked fine once the coat was removed. The crinkling came from the coat being squeezed, not from a broken lock. hint: “What happened after Marcus pushed the door inward?” (50 chars)

Q4. Was Marcus's decision to push the door inward first a good choice? Why or why not?

not cued key 59 chars vs mean distractor 51.7 (1.14x), rank 1.00
Answer option Length Grader picked
A Yes, because it made the crinkling sound go away 48 chars
B No, because he should have asked a teacher for help 51 chars
C No, because he should have just turned the handle harder 56 chars
D Yes, because it relieved pressure on the jammed coat sleeve KEY 59 chars with without
With story: picked D answerable
Pushing inward relieved pressure so it opened.
Story withheld: picked D answerable
Pushing inward relieves pressure so the door can open.
Explanations and hints
Key -- By pushing the door inward first, Marcus released the pressure holding the coat sleeve against the door. Then he could turn the handle and open it easily, which shows this was the right approach.
Wrong -- Turning the handle harder would not have worked since the coat was blocking the door. Pushing inward first was smarter than forcing the handle. hint: “What happened when he pushed inward first?” (42 chars)
Wrong -- While asking for help is sometimes good, Marcus solved the problem himself by thinking it through. His method worked without needing help. hint: “Did Marcus need someone else to fix this?” (41 chars)
Wrong -- The crinkling sound was just what Marcus heard; it was not the problem. The real problem was the jammed coat, which pushing inward helped solve. hint: “What was the actual problem with the locker?” (44 chars)
Judge rubric
answer key validtrueAll four keys are defensible. Q0's 'in that exact order' distractor is cleverly worded but the correct option matches the story's exact order.
groundedtrueEvery question draws directly from story content.
distractor qualityfullDistractors are plausible, drawn from story elements (paper sound, coat, lock, handle), and each is wrong for a distinct reason. Length and register are consistent.
grade fit4Vocabulary (jammed, combination, relieved pressure) and reasoning demand fit grade 4 well; Q3 requires slight causal analysis appropriate to the band.
question type fittrueQ0 = fact_detail, Q1 = vocabulary, Q2 = inference/cause, Q3 = evaluate_analyze. All four requested types are present.
explanation quality5Explanations quote the story directly; hints point to relevant sections without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly recalls stated numbers.
Q2 vocabulary clean Tests meaning of 'jammed' in context.
Q3 inference clean Reader connects the crinkling to the stuffed coat; cause is stated but requires linking clues.
Q4 evaluate_analyze clean Evaluates Marcus's problem-solving choice.
grade 4 · g4-locker · rep 5

claude-haiku-4-5

FAILformat_validdistractor_quality

The Locker That Wouldn't Open -- mystery, curious, a middle school hallway

Lexile not assigned Grade 4 range 415L-1160L 117 words, 9 sentences, mean 13.0
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +54 chars past the 50-char cap.

50% cued-pick (n=2) 100% informed guess (n=2) 0% slip proxy 50% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruepartial
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Marcus spun his locker combination three times, but the door would not budge. He was sure he had the right numbers -- 12, 4, 30 -- because he had used them every day since September. He pressed his ear to the cold metal and tried again. This time he heard a faint crinkling sound inside, like paper being squeezed. Then he remembered: yesterday he had stuffed his winter coat into the locker in a hurry before the bell. The thick sleeve must have been jammed against the door the whole time. Marcus pushed the door inward first, just a little, then turned the handle. It swung open easily. His coat tumbled out onto the floor, and he laughed at himself.

Q1. What was Marcus's locker combination?

not cued key 13 chars vs mean distractor 38.7 (0.34x), rank 4.00
Answer option Length Grader picked
A Three numbers that he forgot by yesterday 41 chars
B 12, 4, and 30 KEY 13 chars with
C 12, 30, and 4 in that exact order 33 chars without
D Numbers written on a piece of paper inside 42 chars
With story: picked B answerable
The story states the numbers were 12, 4, 30.
Story withheld: picked C multiple_defensible
Specific ordered numbers are most likely the stated combination.
Explanations and hints
Key -- The story clearly states that Marcus's locker combination was 12, 4, 30, and he had used these numbers every day since September.
Wrong -- While these are the correct numbers, the story does not specify that order matters for the combination. Marcus used 12, 4, 30. hint: “Check the exact order Marcus used since September.” (50 chars)
Wrong -- Marcus was sure he had the right numbers and used them every day, so he did not forget them. hint: “Reread what Marcus was sure about his numbers.” (46 chars)
Wrong -- The story mentions crinkling paper inside the locker, but this was from his coat, not his combination. hint: “What made the crinkling sound in the locker?” (44 chars)

Q2. Why did Marcus hear a crinkling sound when he pressed his ear to the locker?

not cued key 50 chars vs mean distractor 46.7 (1.07x), rank 2.00
Answer option Length Grader picked
A Someone had hidden paper inside his locker 42 chars
B Metal parts inside the locker were rubbing together 51 chars
C The lock mechanism was broken and needed fixing 47 chars
D His winter coat sleeve was jammed against the door KEY 50 chars with without
With story: picked D answerable
The coat sleeve was jammed and made the crinkling sound.
Story withheld: picked D multiple_defensible
The coat sleeve jammed inside would cause crinkling.
Explanations and hints
Key -- Marcus remembered that he had stuffed his winter coat into the locker in a hurry, and the thick sleeve must have been jammed against the door, causing the crinkling sound.
Wrong -- The crinkling sound came from Marcus's coat, not from hidden paper placed by someone else. hint: “What did Marcus stuff in his locker yesterday?” (46 chars)
Wrong -- The crinkling sound was not from the lock itself but from the coat blocking the door from opening. hint: “What was physically blocking the locker door?” (45 chars)
Wrong -- The story explains the sound came from paper being squeezed, which was part of his coat, not metal parts. hint: “What material makes a crinkling sound when squeezed?” (52 chars)

Q3. What can you figure out about why Marcus's locker door would not open at first?

not cued key 52 chars vs mean distractor 48.7 (1.07x), rank 2.00
Answer option Length Grader picked
A Someone had put a lock on his locker overnight 46 chars
B His coat was blocking the door from opening properly KEY 52 chars with without
C He had forgotten his locker combination since September 55 chars
D The lock was broken and needed to be replaced 45 chars
With story: picked B answerable
The coat was blocking the door.
Story withheld: picked B answerable
The coat stuffed inside blocked the door.
Explanations and hints
Key -- The story shows that Marcus's winter coat was jammed against the door, preventing it from opening even though he had the correct combination.
Wrong -- The story says Marcus was sure he had the right numbers because he used them every day since September, so he had not forgotten. hint: “How often did Marcus use his combination?” (41 chars)
Wrong -- The lock worked fine once Marcus pushed the door inward first, so it was not broken. hint: “What happened when Marcus pushed the door inward?” (49 chars)
Wrong -- The story does not mention anyone else putting a lock on the locker. The problem was the coat inside. hint: “What did Marcus stuff in the locker yesterday?” (46 chars)

Q4. Was Marcus's choice to stuff his coat into the locker in a hurry a good decision?

not cued key 51 chars vs mean distractor 50.0 (1.02x), rank 2.00
Answer option Length Grader picked
A No, because he laughed at himself about it later 48 chars
B Yes, because his coat stayed warm and dry inside 48 chars
C No, because it blocked his locker door from opening KEY 51 chars with without
D Yes, because he needed to get to class before the bell 54 chars
With story: picked C answerable
Stuffing the coat caused the door to jam.
Story withheld: picked C answerable
Stuffing it in a hurry caused the door to jam.
Explanations and hints
Key -- Marcus's hurried decision to stuff his coat into the locker caused it to jam against the door, which prevented the locker from opening the next day.
Wrong -- While Marcus was in a hurry to get to class, his choice still created a problem that made his locker unusable the next day. hint: “What problem did the hurried action cause?” (42 chars)
Wrong -- Even though the coat was protected, the choice still caused his locker to jam, which was a negative consequence. hint: “What happened to his locker after he stuffed the coat?” (54 chars)
Wrong -- Marcus laughed at himself because he realized the problem, but laughing does not mean the decision was good or bad. hint: “Why did Marcus laugh at himself?” (32 chars)
Judge rubric
answer key validtrueAll four keys are defensible. Q0's 'in that exact order' distractor is weak but the correct answer is still clearly the best.
groundedtrueEvery question is answerable from the story text.
distractor qualitypartialQ0's distractors are weak—'12, 30, and 4 in that exact order' is nearly the same as the correct answer, and others don't test the fact meaningfully. Q1-Q3 distractors are stronger and plausible.
grade fit4Vocabulary (crinkling, combination, jammed) and reasoning demand fit grade 4 well.
question type fittrueQ0=fact, Q1=cause/why, Q2=inference, Q3=evaluate. All four requested types present.
explanation quality4Explanations reference story content clearly; hints point without revealing. Q0 hints are slightly awkward but functional.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated numbers in the story.
Q2 fact_detail clean Cause is explicitly stated in the story; more recall than inference.
Q3 inference clean Requires connecting the coat clue to the jammed door, though it's close to stated.
Q4 evaluate_analyze clean Asks student to judge Marcus's decision.
grade 4 · g4-thin · rep 1

claude-haiku-4-5

FAILformat_valid

A Quiet Afternoon -- slice-of-life, calm, a back porch

Lexile not assigned Grade 4 range 415L-1160L 71 words, 7 sentences, mean 10.1
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +51 chars past the 50-char cap.

0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Theo sat on the back porch with a glass of lemonade going warm beside him. The afternoon was very quiet. A single leaf came loose from the maple and turned over twice on its way down. He watched it the whole way. Somewhere down the street a screen door opened and closed. Theo did not get up. The ice in his glass had melted completely by the time he went inside.

Q1. What was sitting beside Theo on the back porch?

not cued key 42 chars vs mean distractor 45.3 (0.93x), rank 4.00
Answer option Length Grader picked
A A screen door that opened and closed nearby. 44 chars
B A glass of lemonade that was getting warm. KEY 42 chars with without
C A maple leaf that fell from the tree above. 43 chars
D A glass of cold lemonade he was drinking quickly. 49 chars
With story: picked B answerable
The story states a glass of lemonade was going warm beside him.
Story withheld: picked B answerable
The phrase 'going warm' matches a lemonade getting warm.
Explanations and hints
Key -- The story states: 'Theo sat on the back porch with a glass of lemonade going warm beside him.'
Wrong -- The lemonade was going warm, not staying cold, and Theo was not actively drinking it. hint: “What happened to the ice in his glass?” (38 chars)
Wrong -- The leaf fell down in front of Theo, not beside him on the porch. hint: “Where did Theo watch the leaf go?” (33 chars)
Wrong -- The screen door was down the street, not beside Theo on the porch. hint: “Where was the sound coming from?” (32 chars)

Q2. What does the phrase 'going warm' tell us about how much time passed?

not cued key 48 chars vs mean distractor 46.7 (1.03x), rank 2.00
Answer option Length Grader picked
A Theo needed to go inside right away to cool off. 48 chars
B Theo drank his lemonade very quickly in the sun. 48 chars
C The afternoon was too hot for Theo to enjoy. 44 chars
D Theo sat outside for a long time without moving. KEY 48 chars with without
With story: picked D answerable
Lemonade warming and ice melting indicate a long, still passage of time.
Story withheld: picked D answerable
Lemonade warming up suggests he sat still for a long time.
Explanations and hints
Key -- Lemonade takes time to warm up, and the ice melted completely, showing Theo stayed still for quite a while.
Wrong -- The story says the lemonade was going warm, not that Theo drank it, and he did not get up. hint: “Did Theo move or stay in one place?” (35 chars)
Wrong -- The story describes the afternoon as quiet and peaceful, not uncomfortably hot. hint: “What word describes the afternoon?” (34 chars)
Wrong -- Theo only went inside after the ice had melted completely, showing he was not rushing. hint: “When did Theo finally go inside?” (32 chars)

Q3. Why did Theo probably not get up when he heard the screen door?

not cued key 57 chars vs mean distractor 49.0 (1.16x), rank 1.00
Answer option Length Grader picked
A He was afraid of the person opening the screen door. 52 chars
B He did not hear the screen door open and close. 47 chars
C His lemonade was too cold to leave on the porch. 48 chars
D He was content sitting quietly and watching small things. KEY 57 chars with without
With story: picked D answerable
He was peacefully observing small things like the leaf.
Story withheld: picked D answerable
He seems content and still, observing quietly.
Explanations and hints
Key -- The story shows Theo watching a leaf fall and staying still in a quiet afternoon, suggesting he was enjoying the peaceful moment.
Wrong -- Nothing in the story suggests Theo was scared; he simply did not move. hint: “What mood does the quiet afternoon create?” (42 chars)
Wrong -- The lemonade was warming up, not cold, so that would not keep him seated. hint: “What was happening to the temperature of his drink?” (51 chars)
Wrong -- The story says Theo heard it: 'Somewhere down the street a screen door opened and closed.'

Q4. What does Theo's choice to stay outside show about what he values?

length-cued key 62 chars vs mean distractor 49.0 (1.27x), rank 1.00
Answer option Length Grader picked
A He wants to avoid going inside to do chores. 44 chars
B He is waiting for someone to come down the street. 50 chars
C He enjoys quiet moments and noticing small details around him. KEY 62 chars with without
D He is trying to finish his lemonade before it spoils. 53 chars
With story: picked C answerable
He watches a leaf fall and stays still, valuing quiet observation.
Story withheld: picked C answerable
Staying outside quietly shows he values small, peaceful moments.
Explanations and hints
Key -- Theo watched a single leaf fall the whole way down and sat still through a quiet afternoon, showing he values peaceful observation.
Wrong -- The story gives no information about chores or why Theo might avoid them. hint: “What does the story tell us about Theo's feelings?” (50 chars)
Wrong -- The screen door sound comes from down the street, but Theo does not get up to meet anyone. hint: “Did Theo react to the sound he heard?” (37 chars)
Wrong -- Theo leaves his lemonade behind when he goes inside, showing it was not his main focus. hint: “What did Theo do with his glass when he left?” (45 chars)
Judge rubric
answer key validtrueEach question has one clearly defensible answer supported by the text.
groundedtrueAll questions rely on details from the story.
distractor qualityfullDistractors draw from story elements (leaf, screen door, lemonade) and each is wrong for a different reason, matched in register.
grade fit4Vocabulary and inferential demand fit grade 4; the reflective tone may be slightly subtle but accessible.
question type fittrueQ0=fact_detail, Q1=vocabulary/phrase meaning, Q2=inference, Q3=evaluate_analyze. All four requested types present.
explanation quality5Explanations quote or paraphrase the story; hints point toward evidence without revealing answers.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated fact about what was beside Theo.
Q2 vocabulary clean Interprets meaning of the phrase 'going warm' in context.
Q3 inference clean Requires inferring Theo's contentment from clues.
Q4 evaluate_analyze clean Analyzes what Theo's choice reveals about his values.
grade 4 · g4-thin · rep 2

claude-haiku-4-5

FAILformat_valid

A Quiet Afternoon -- slice-of-life, calm, a back porch

Lexile not assigned Grade 4 range 415L-1160L 71 words, 7 sentences, mean 10.1
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced
100% cued-pick (n=1) 100% informed guess (n=3) 0% slip proxy 25% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Theo sat on the back porch with a glass of lemonade going warm beside him. The afternoon was very quiet. A single leaf came loose from the maple and turned over twice on its way down. He watched it the whole way. Somewhere down the street a screen door opened and closed. Theo did not get up. The ice in his glass had melted completely by the time he went inside.

Q1. What was sitting beside Theo on the back porch?

not cued key 42 chars vs mean distractor 44.3 (0.95x), rank 4.00
Answer option Length Grader picked
A A screen door that opened and closed nearby. 44 chars
B A maple leaf that fell from the tree above. 43 chars
C A glass of cold lemonade that he was drinking. 46 chars
D A glass of lemonade that was getting warm. KEY 42 chars with without
With story: picked D answerable
Text says glass of lemonade going warm beside him.
Story withheld: picked D answerable
The phrase 'going warm' in Q1 suggests the lemonade was getting warm.
Explanations and hints
Key -- The story states, 'Theo sat on the back porch with a glass of lemonade going warm beside him.'
Wrong -- The lemonade was going warm, not staying cold, and Theo was not actively drinking it. hint: “Notice what happened to the ice in his glass.” (45 chars)
Wrong -- The leaf fell down in front of Theo, not beside him on the porch. hint: “Where did Theo watch the leaf go?” (33 chars)
Wrong -- The screen door was down the street, not beside Theo on the porch. hint: “Reread where the sound came from.” (33 chars)

Q2. What does the phrase 'going warm' tell us about how much time passed?

not cued key 50 chars vs mean distractor 44.3 (1.13x), rank 1.00
Answer option Length Grader picked
A Theo poured the lemonade out on the porch. 42 chars
B Only a few minutes passed while Theo sat there. 47 chars
C Enough time passed for the ice to melt completely. KEY 50 chars with without
D The sun was too hot and burned the lemonade. 44 chars
With story: picked C answerable
Ice melted completely indicates significant time passed.
Story withheld: picked C answerable
Lemonade going warm implies ice melted, indicating significant time passed.
Explanations and hints
Key -- The story says the ice 'had melted completely by the time he went inside,' showing that 'going warm' means a long time passed.
Wrong -- If only a few minutes passed, the ice would not have melted completely as the story states. hint: “How much ice was left when Theo went inside?” (44 chars)
Wrong -- Heat melts ice, but the story does not say the sun burned anything. hint: “What happened to the ice cubes in the glass?” (44 chars)
Wrong -- The story says Theo went inside with the glass, not that he poured it out. hint: “What did Theo do when he finally moved?” (39 chars)

Q3. What can you figure out about Theo's mood during this afternoon?

not cued key 51 chars vs mean distractor 54.3 (0.94x), rank 4.00
Answer option Length Grader picked
A He was calm and content to sit quietly and observe. KEY 51 chars with without
B He was worried about the screen door down the street. 53 chars
C He was upset and did not want to drink his lemonade. 52 chars
D He was bored and waiting for something exciting to happen. 58 chars
With story: picked A answerable
Quiet observation of leaf and stillness suggests calm contentment.
Story withheld: picked A answerable
Sitting quietly while lemonade warms suggests a calm, observant mood.
Explanations and hints
Key -- The story describes a very quiet afternoon where Theo watches a leaf fall and does not get up, showing he was peaceful and still.
Wrong -- While Theo was quiet, the story shows he was watching and observing, not restlessly waiting. hint: “How did Theo react to the falling leaf?” (39 chars)
Wrong -- The screen door opening and closing is just a sound in the quiet afternoon, not something that worried Theo. hint: “Did Theo respond to the sound he heard?” (39 chars)
Wrong -- The story does not suggest Theo was upset; he simply sat still while his drink warmed. hint: “Look at what Theo chose to do all afternoon.” (44 chars)

Q4. Why did Theo finally go inside his house?

not cued key 46 chars vs mean distractor 53.3 (0.86x), rank 4.00
Answer option Length Grader picked
A He wanted to get more cold lemonade from inside. 48 chars
B The maple leaf landed on him and made him uncomfortable. 56 chars
C He heard the screen door open and close down the street. 56 chars
D The story does not explain why he went inside. KEY 46 chars with without
With story: picked D answerable
No reason for going inside is given.
Story withheld: picked D unanswerable
Without the story text, the reason isn't given here.
Explanations and hints
Key -- The story only tells us that 'the ice in his glass had melted completely by the time he went inside,' but gives no reason for his decision.
Wrong -- The screen door sound happened while Theo was still sitting outside, not when he decided to go in. hint: “When did Theo hear the screen door?” (35 chars)
Wrong -- The story does not mention Theo wanting more lemonade or going inside for that reason. hint: “What does the story tell us about his actions?” (46 chars)
Wrong -- The story shows Theo watched the leaf fall but does not say it landed on him. hint: “Where did the leaf go after falling?” (36 chars)
Judge rubric
answer key validtrueEach question has one defensible answer. Q3 is unusual but the 'story does not explain' option is genuinely correct since no reason is given.
groundedtrueAll questions rely on story details.
distractor qualityfullDistractors draw from actual story elements (leaf, screen door, lemonade) and are each wrong for different reasons, well-matched in register.
grade fit4Vocabulary and reasoning demand fit grade 4; inference about mood and the meta-question about what the story doesn't say are appropriately challenging.
question type fittrueQ0=fact_detail, Q1=vocabulary (phrase meaning), Q2=inference (mood), Q3=evaluate_analyze (analyzing what story does/doesn't tell us). However, Q3 was requested as comparing/evaluating a character's choice, which is a slight mismatch — it analyzes narrative rather than a choice. Still maps to evaluate_analyze.
explanation quality4Explanations quote the story directly; hints point toward evidence without revealing answers.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Direct recall of stated detail.
Q2 vocabulary clean Meaning of phrase 'going warm' in context of time passing.
Q3 inference clean Mood is unstated; reader must infer from clues.
Q4 evaluate_analyze clean Requires analyzing what the story does and does not tell, though weaker fit to the requested 'compare/evaluate a choice' framing.
grade 4 · g4-thin · rep 3

claude-haiku-4-5

FAILformat_valid

A Quiet Afternoon -- slice-of-life, calm, a back porch

Lexile not assigned Grade 4 range 415L-1160L 71 words, 7 sentences, mean 10.1
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced
0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Theo sat on the back porch with a glass of lemonade going warm beside him. The afternoon was very quiet. A single leaf came loose from the maple and turned over twice on its way down. He watched it the whole way. Somewhere down the street a screen door opened and closed. Theo did not get up. The ice in his glass had melted completely by the time he went inside.

Q1. What was sitting next to Theo on the back porch?

not cued key 40 chars vs mean distractor 48.3 (0.83x), rank 4.00
Answer option Length Grader picked
A A glass of lemonade that was going warm. KEY 40 chars with without
B A screen door that opened and closed down the street. 53 chars
C A glass of cold lemonade he was drinking quickly. 49 chars
D A maple leaf that fell from the tree above. 43 chars
With story: picked A answerable
The story states a glass of lemonade was going warm beside him.
Story withheld: picked A answerable
The warm lemonade detail fits a quiet, still afternoon scene.
Explanations and hints
Key -- The story begins by saying 'Theo sat on the back porch with a glass of lemonade going warm beside him.'
Wrong -- The story says the lemonade was 'going warm,' not cold, and Theo did not drink it quickly. hint: “What happened to the ice in his glass?” (38 chars)
Wrong -- The leaf fell while Theo watched it, but it was not sitting beside him the whole time. hint: “When did the leaf appear in the story?” (38 chars)
Wrong -- The screen door was down the street, not on the porch beside Theo. hint: “Where was the sound coming from?” (32 chars)

Q2. What does it mean that the afternoon was 'very quiet'?

not cued key 60 chars vs mean distractor 52.7 (1.14x), rank 1.00
Answer option Length Grader picked
A There were very few sounds and Theo felt peaceful and still. KEY 60 chars with without
B The maple tree was too far away to hear leaves falling. 55 chars
C Theo was not allowed to make any noise on the porch. 52 chars
D Theo's lemonade was not making any bubbling sounds. 51 chars
With story: picked A answerable
Quiet means few sounds, matching the peaceful tone.
Story withheld: picked A answerable
'Very quiet' means few sounds, evoking peaceful stillness.
Explanations and hints
Key -- The quiet afternoon is shown through details like a single leaf falling and Theo not moving, creating a calm, peaceful mood.
Wrong -- The quiet describes the afternoon itself, not a rule Theo had to follow. hint: “What does the story tell us about the setting?” (46 chars)
Wrong -- The quiet is about the whole afternoon, not just the distance to the tree. hint: “Look at all the details about sounds.” (37 chars)
Wrong -- While the lemonade was silent, the quiet describes much more than just the drink. hint: “What other details show quietness?” (34 chars)

Q3. What can you figure out about how Theo was feeling during this afternoon?

not cued key 56 chars vs mean distractor 53.3 (1.05x), rank 2.00
Answer option Length Grader picked
A He was sad because his lemonade was getting warm. 49 chars
B He was worried about the screen door down the street. 53 chars
C He was calm and content to sit still and observe things. KEY 56 chars with without
D He was bored and waiting for something exciting to happen. 58 chars
With story: picked C answerable
He sits still watching small details, showing calm contentment.
Story withheld: picked C answerable
Sitting still and noticing small things suggests calm contentment.
Explanations and hints
Key -- Theo watched a leaf fall 'the whole way,' did not get up when he heard a door, and let his lemonade warm—all showing he was peaceful and still.
Wrong -- While Theo was quiet, the story shows he was focused on watching, not bored or waiting. hint: “What was Theo doing with the falling leaf?” (42 chars)
Wrong -- Theo heard the door but 'did not get up,' showing he was not worried or concerned about it. hint: “How did Theo react to the sound?” (32 chars)
Wrong -- The story does not suggest Theo was sad; it simply describes what happened to his drink. hint: “What mood does the quiet afternoon create?” (42 chars)

Q4. Why did Theo choose to stay sitting instead of going inside earlier?

length-cued key 67 chars vs mean distractor 50.7 (1.32x), rank 1.00
Answer option Length Grader picked
A He was waiting for someone to open the screen door. 51 chars
B He forgot that his lemonade was getting warm beside him. 56 chars
C He was enjoying the quiet moment and wanted to watch the leaf fall. KEY 67 chars with without
D He was too tired to stand up and walk inside. 45 chars
With story: picked C answerable
He watched the leaf and didn't get up, enjoying the quiet.
Story withheld: picked C answerable
He stayed to enjoy the quiet and watch the leaf.
Explanations and hints
Key -- Theo watched the leaf 'the whole way' and did not get up when he heard the door, showing he valued the peaceful moment more than other activities.
Wrong -- The screen door was down the street, and Theo did not get up when it opened, so he was not waiting for it. hint: “Did Theo react to the door sound?” (33 chars)
Wrong -- The story shows Theo noticed the ice had melted, so he was aware of his drink the whole time. hint: “What did Theo notice about the ice?” (35 chars)
Wrong -- The story does not say Theo was tired; it shows he was peacefully watching and observing. hint: “What was Theo actively doing?” (29 chars)
Judge rubric
answer key validtrueAll four keys are defensible. Q0 is directly stated; Q1-Q3 have the best-supported answer marked correct.
groundedtrueEvery question draws on story details.
distractor qualityfullDistractors reuse story elements (leaf, screen door, ice) and each is wrong for a different reason; register matches.
grade fit4Vocabulary and inference load are appropriate for grade 4; mood inference is age-appropriate.
question type fittrueQ0 fact_detail, Q1 vocabulary/phrase meaning, Q2 inference, Q3 evaluate_analyze — matches the requested distribution.
explanation quality4Explanations quote the story directly; hints point without revealing. Q1's explanation is slightly interpretive rather than defining 'quiet' literally.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated in opening line.
Q2 vocabulary clean Asks meaning of the phrase 'very quiet' in context.
Q3 inference clean Feeling is unstated; reader infers from behavior.
Q4 evaluate_analyze clean Analyzes Theo's choice to remain sitting.
grade 4 · g4-thin · rep 4

claude-haiku-4-5

FAILformat_valid

A Quiet Afternoon -- slice-of-life, calm, a back porch

Lexile not assigned Grade 4 range 415L-1160L 71 words, 7 sentences, mean 10.1
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced
100% cued-pick (n=1) 100% informed guess (n=3) 0% slip proxy 25% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Theo sat on the back porch with a glass of lemonade going warm beside him. The afternoon was very quiet. A single leaf came loose from the maple and turned over twice on its way down. He watched it the whole way. Somewhere down the street a screen door opened and closed. Theo did not get up. The ice in his glass had melted completely by the time he went inside.

Q1. What was sitting next to Theo on the back porch?

not cued key 42 chars vs mean distractor 48.3 (0.87x), rank 4.00
Answer option Length Grader picked
A A screen door that opened and closed down the street. 53 chars
B A glass of lemonade that was getting warm. KEY 42 chars with without
C A glass of cold lemonade he was drinking quickly. 49 chars
D A maple leaf that fell from the tree above. 43 chars
With story: picked B answerable
The story says a glass of lemonade going warm sat beside him.
Story withheld: picked B answerable
A warm lemonade suggests he'd been sitting for a while.
Explanations and hints
Key -- The story states, 'Theo sat on the back porch with a glass of lemonade going warm beside him.'
Wrong -- The lemonade was going warm, not staying cold, and Theo was not actively drinking it. hint: “What happened to the ice in his glass?” (38 chars)
Wrong -- The leaf fell down and Theo watched it, but it was not sitting beside him on the porch. hint: “Where did the leaf go after it fell?” (36 chars)
Wrong -- The screen door was down the street, not on the porch beside Theo. hint: “Reread the sentence about the screen door.” (42 chars)

Q2. What does the phrase 'the afternoon was very quiet' tell you about how Theo felt?

not cued key 45 chars vs mean distractor 47.7 (0.94x), rank 3.00
Answer option Length Grader picked
A He was calm and peaceful, not rushed or busy. KEY 45 chars with without
B He was bored and wanted something exciting to happen. 53 chars
C He was scared because the street was too empty. 47 chars
D He was sad because no one was visiting him. 43 chars
With story: picked A answerable
The quiet tone suggests calm and peacefulness.
Story withheld: picked A answerable
Quiet afternoon implies calm and peaceful mood.
Explanations and hints
Key -- The quiet afternoon and Theo's stillness—watching a leaf, not getting up—show he was in a calm, peaceful mood.
Wrong -- While the afternoon is quiet, the story does not say Theo felt sad or lonely about it. hint: “Did Theo seem upset or bothered by the quiet?” (45 chars)
Wrong -- Bored people usually try to find something to do, but Theo sat still and watched a single leaf. hint: “What did Theo choose to do with his time?” (41 chars)
Wrong -- Nothing in the story suggests Theo was frightened. He simply sat and observed the quiet afternoon. hint: “Look at Theo's actions, not the street.” (39 chars)

Q3. What can you figure out about how long Theo stayed on the porch?

not cued key 49 chars vs mean distractor 49.7 (0.99x), rank 3.00
Answer option Length Grader picked
A Just long enough to drink all his warm lemonade. 48 chars
B Long enough for all the ice in his glass to melt. KEY 49 chars with without
C Until the screen door down the street opened again. 51 chars
D Only a few minutes while he watched one leaf fall. 50 chars
With story: picked B answerable
He stayed until the ice fully melted.
Story withheld: picked B multiple_defensible
Warm lemonade suggests ice melted over time.
Explanations and hints
Key -- The story says, 'The ice in his glass had melted completely by the time he went inside,' showing he sat there for a while.
Wrong -- If he only stayed a few minutes, the ice would not have melted completely as the story states. hint: “How long does ice take to melt completely?” (42 chars)
Wrong -- The screen door opening and closing is mentioned, but it is not what made Theo go inside. hint: “What changed before Theo went inside?” (37 chars)
Wrong -- The story never says Theo drank the lemonade. It only melted, suggesting he did not drink it. hint: “Did Theo actually drink the lemonade?” (37 chars)

Q4. Why did Theo choose to stay sitting on the porch instead of going inside earlier?

not cued key 58 chars vs mean distractor 50.0 (1.16x), rank 1.00
Answer option Length Grader picked
A He was enjoying the quiet moment and wanted to stay in it. KEY 58 chars with without
B He was too hot and needed to cool down with lemonade. 53 chars
C He was waiting for someone to come down the street. 51 chars
D He was stuck outside and could not go back in. 46 chars
With story: picked A answerable
He watched the leaf and didn't get up, enjoying the stillness.
Story withheld: picked A answerable
The peaceful tone suggests he was savoring the quiet.
Explanations and hints
Key -- Theo watched a leaf fall and did not get up even when a door opened. His stillness shows he valued the peaceful moment.
Wrong -- The story shows Theo watching nature, not looking for people. He did not get up when the door opened. hint: “Did Theo react to the screen door sound?” (40 chars)
Wrong -- If Theo wanted to cool down, he would have drunk the lemonade, but it went warm beside him. hint: “What happened to the lemonade while he sat?” (43 chars)
Wrong -- The story says Theo 'went inside' at the end, showing he could move and choose to leave whenever he wanted. hint: “How did Theo finally leave the porch?” (37 chars)
Judge rubric
answer key validtrueEach question has one clearly defensible answer supported by the text.
groundedtrueAll questions can be answered from story details.
distractor qualityfullDistractors draw from story elements (leaf, screen door, lemonade) and each is wrong for a distinct reason, matched in length/register.
grade fit4Vocabulary is simple; inference and analysis demands fit grade 4.
question type fittrueQ0 fact_detail, Q1 vocabulary/phrase meaning, Q2 inference, Q3 evaluate_analyze — matches requested distribution.
explanation quality5Explanations quote or paraphrase the story; hints point to relevant sections without revealing answers.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated fact about what was beside Theo.
Q2 vocabulary clean Asks meaning of a phrase in context; borders on inference but centers on phrase meaning.
Q3 inference clean Reader must infer duration from the melted ice clue.
Q4 evaluate_analyze clean Analyzes Theo's motivation for his choice to stay.
grade 4 · g4-thin · rep 5

claude-haiku-4-5

FAILformat_valid

A Quiet Afternoon -- slice-of-life, calm, a back porch

Lexile not assigned Grade 4 range 415L-1160L 71 words, 7 sentences, mean 10.1
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced
0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Theo sat on the back porch with a glass of lemonade going warm beside him. The afternoon was very quiet. A single leaf came loose from the maple and turned over twice on its way down. He watched it the whole way. Somewhere down the street a screen door opened and closed. Theo did not get up. The ice in his glass had melted completely by the time he went inside.

Q1. What was sitting beside Theo on the back porch?

not cued key 42 chars vs mean distractor 41.3 (1.02x), rank 3.00
Answer option Length Grader picked
A A maple leaf that fell from the tree. 37 chars
B A glass of lemonade that was getting warm. KEY 42 chars with without
C A screen door that opened and closed nearby. 44 chars
D A glass of cold water that he was drinking. 43 chars
With story: picked B answerable
The story states a glass of lemonade going warm sat beside him.
Story withheld: picked B answerable
Warm lemonade is a common detail in such stories.
Explanations and hints
Key -- The story states, 'Theo sat on the back porch with a glass of lemonade going warm beside him.'
Wrong -- The story says the lemonade was going warm, not cold, and Theo was not actively drinking it. hint: “What was the temperature of the drink?” (38 chars)
Wrong -- The leaf fell down from the maple tree, but it was not sitting beside Theo on the porch. hint: “Where did the leaf go after it fell?” (36 chars)
Wrong -- The screen door was down the street, not on the porch beside Theo. hint: “Where was the screen door located?” (34 chars)

Q2. What does the phrase 'very quiet' tell you about the afternoon?

not cued key 49 chars vs mean distractor 44.3 (1.11x), rank 1.00
Answer option Length Grader picked
A Theo was sitting alone without any other people. 48 chars
B There were very few sounds or activity happening. KEY 49 chars with without
C Theo did not want to talk to anyone nearby. 43 chars
D The maple tree was not moving in the wind. 42 chars
With story: picked B answerable
'Very quiet' refers to lack of sound and activity.
Story withheld: picked B answerable
'Very quiet' describes lack of sound/activity.
Explanations and hints
Key -- When the story says the afternoon was 'very quiet,' it means there was little noise or movement, which is shown by the single leaf and distant screen door.
Wrong -- While Theo may have been alone, 'very quiet' describes the sound level of the afternoon, not whether people were present. hint: “What does 'quiet' describe about a place?” (41 chars)
Wrong -- The story does not say the tree was still; one leaf did fall, suggesting some movement. hint: “Did the leaf fall on its own?” (29 chars)
Wrong -- The phrase 'very quiet' describes the afternoon itself, not Theo's feelings about talking. hint: “Is 'quiet' about a person or a time of day?” (43 chars)

Q3. What can you figure out about how Theo was feeling during this afternoon?

not cued key 47 chars vs mean distractor 47.0 (1.00x), rank 3.00
Answer option Length Grader picked
A He was worried about the screen door opening. 45 chars
B He was thirsty and wanted to drink his lemonade. 48 chars
C He was bored and waiting for something exciting. 48 chars
D He was calm and content to sit still and watch. KEY 47 chars with without
With story: picked D answerable
He sits still watching a leaf fall, suggesting calm contentment.
Story withheld: picked D answerable
Sitting still on a quiet porch suggests calm contentment.
Explanations and hints
Key -- Theo watched a single leaf fall, did not get up when he heard a door, and sat long enough for his ice to melt completely, showing he was peaceful and unhurried.
Wrong -- While Theo was sitting still, the story shows he was paying close attention to details like the falling leaf, not waiting impatiently. hint: “How carefully did Theo watch the leaf?” (38 chars)
Wrong -- When the screen door opened and closed, Theo did not get up or react with concern, so he was not worried. hint: “What did Theo do when he heard the door?” (40 chars)
Wrong -- The lemonade sat beside him going warm, and he did not drink it before going inside, showing he was not focused on drinking. hint: “Did Theo drink the lemonade while sitting?” (42 chars)

Q4. Why did Theo choose to stay sitting instead of getting up when he heard the screen door?

length-cued key 62 chars vs mean distractor 44.3 (1.40x), rank 1.00
Answer option Length Grader picked
A He was too tired from sitting to stand up. 42 chars
B He wanted to finish his warm lemonade first. 44 chars
C He was enjoying the peaceful moment and did not want to leave. KEY 62 chars with without
D He did not hear the screen door open and close. 47 chars
With story: picked C answerable
He was enjoying the peaceful afternoon and didn't want to move.
Story withheld: picked C answerable
He was enjoying the peaceful moment.
Explanations and hints
Key -- The story shows Theo was content watching the quiet afternoon unfold; he did not get up even when he heard activity, showing he preferred to stay in that calm moment.
Wrong -- The story clearly states 'Somewhere down the street a screen door opened and closed,' so Theo heard it. hint: “Did the story say Theo heard the door?” (38 chars)
Wrong -- The story does not mention Theo being tired; it shows he was peacefully watching and thinking. hint: “What was Theo doing instead of resting?” (39 chars)
Wrong -- Theo did not drink the lemonade; he left it sitting beside him the whole time. hint: “What happened to the ice in his glass?” (38 chars)
Judge rubric
answer key validtrueEach question has one defensible correct answer supported by the text.
groundedtrueAll questions reference details from the story.
distractor qualityfullDistractors draw from story elements (leaf, screen door, lemonade) and are each wrong for different reasons, matched in register.
grade fit4Vocabulary and inference demand are appropriate for grade 4; the story is simple but requires some inference about mood.
question type fittrueQ0=fact_detail, Q1=vocabulary/phrase meaning, Q2=inference, Q3=evaluate_analyze — matches requested distribution.
explanation quality5Explanations quote or paraphrase the story directly; hints guide without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated detail about the lemonade.
Q2 vocabulary clean Asks meaning of the phrase 'very quiet' in context.
Q3 inference clean Reader must infer Theo's unstated emotional state.
Q4 evaluate_analyze clean Analyzes Theo's choice/motivation for staying seated.
grade 7 · g7-tryout · rep 1

claude-haiku-4-5

FAILformat_valid

The Bench Decision -- realistic-fiction, reflective, a school gymnasium

Lexile not assigned Grade 7 range 760L-1410L 133 words, 9 sentences, mean 14.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +58 chars past the 50-char cap.

0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Priya had trained all summer for the point-guard spot, and at tryouts she was clearly the fastest player on the court. But when Coach Ellis read the starting lineup, Priya's name was in the second group. She almost argued. Instead, she watched the first practice from the bench and noticed something: the starter, Dana, never scored much, but every player around Dana scored more than usual. That night Priya understood what the coach had seen. Speed wins one-on-one, but Dana made the whole team better by passing early and calling out screens. The lineup was not about who was fastest. At the next practice Priya started passing the way Dana did. She scored fewer points that day than ever before -- and her group outscored the starters for the first time all season.

Q1. What position did Coach Ellis give to Priya at tryouts?

not cued key 53 chars vs mean distractor 53.7 (0.99x), rank 3.00
Answer option Length Grader picked
A She was placed in the second group, not as a starter. KEY 53 chars with without
B She was the fastest player but scored the most points. 54 chars
C She was chosen as the starting point guard immediately. 55 chars
D She was told to watch from the bench and never play. 52 chars
With story: picked A answerable
The story states her name was in the second group.
Story withheld: picked A answerable
The typical setup implies Priya wasn't made starter.
Explanations and hints
Key -- The story states: 'when Coach Ellis read the starting lineup, Priya's name was in the second group.' This is the key detail that sets up the rest of the story.
Wrong -- The story clearly says Priya was not in the starting lineup. She trained all summer for the spot but did not get it. hint: “Reread the sentence about Coach Ellis reading the lineup.” (57 chars)
Wrong -- While Priya was the fastest player, the story does not say she scored the most points. Her scoring came later after she changed her approach. hint: “What did Priya notice about Dana's scoring?” (43 chars)
Wrong -- Priya watched the first practice from the bench, but this was temporary. She later played and her group outscored the starters. hint: “What happened at the next practice after the first one?” (55 chars)

Q2. Why did Priya decide not to argue with Coach Ellis about the lineup?

length-cued key 69 chars vs mean distractor 53.7 (1.29x), rank 1.00
Answer option Length Grader picked
A She knew Dana was the fastest player on the court. 50 chars
B She was too tired from training all summer to complain. 55 chars
C She watched Dana play and realized the coach understood team success. KEY 69 chars with without
D She wanted to score fewer points than the other players. 56 chars
With story: picked C answerable
She watched and understood the coach's reasoning about team play.
Story withheld: picked C answerable
Watching Dana showed her the coach valued team success.
Explanations and hints
Key -- The story says: 'She almost argued. Instead, she watched the first practice from the bench and noticed something.' This observation led her to understand the coach's decision was about making the whole team better.
Wrong -- The story does not mention Priya being tired. Her decision not to argue came from watching and learning, not from exhaustion. hint: “What did Priya do instead of arguing?” (37 chars)
Wrong -- The story says Priya was clearly the fastest player, not Dana. Priya's realization was about Dana's passing and teamwork, not speed. hint: “Who was the fastest player according to the story?” (50 chars)
Wrong -- Priya did not decide this beforehand. She only scored fewer points later after she changed her playing style based on what she learned. hint: “When did Priya start passing like Dana?” (39 chars)

Q3. Based on the story, what evidence shows that Priya understood what made Dana effective?

not cued key 62 chars vs mean distractor 56.7 (1.09x), rank 1.00
Answer option Length Grader picked
A She scored more points than Dana in the second practice. 56 chars
B She watched Dana from the bench and never played again. 55 chars
C She argued with Coach Ellis about why Dana was the starter. 59 chars
D She began passing early and calling out screens like Dana did. KEY 62 chars with without
With story: picked D answerable
She imitated Dana's passing and screen-calling.
Story withheld: picked D answerable
Mimicking Dana's passing and screens shows understanding.
Explanations and hints
Key -- The story states: 'At the next practice Priya started passing the way Dana did.' This action proves Priya understood Dana's approach because she copied the specific behaviors that made Dana effective.
Wrong -- The story says Priya 'scored fewer points that day than ever before,' not more. Her understanding was shown through changed behavior, not through higher scoring. hint: “How many points did Priya score at the next practice?” (53 chars)
Wrong -- Priya did play at the next practice and her group outscored the starters. Watching was only the first step in her learning process. hint: “What happened after Priya watched the first practice?” (53 chars)
Wrong -- The story says Priya 'almost argued' but did not actually argue. Instead she learned by watching and then changed her own playing style. hint: “Did Priya actually argue with the coach?” (40 chars)

Q4. What does Priya's decision to pass more and score less tell us about her growth as a player?

length-cued key 78 chars vs mean distractor 61.3 (1.27x), rank 1.00
Answer option Length Grader picked
A She learned that helping teammates succeed matters more than personal scoring. KEY 78 chars with without
B She realized she was not as fast as she thought she was. 56 chars
C She proved that the fastest player should always be the starter. 64 chars
D She decided to quit basketball because Dana was better than her. 64 chars
With story: picked A answerable
Her growth was valuing team success over personal stats.
Story withheld: picked A answerable
Prioritizing teammates over scoring reflects growth.
Explanations and hints
Key -- Priya chose to play like Dana by passing early and calling out screens, even though it meant scoring fewer points. Her group then outscored the starters, showing she understood that team success comes from making others better, not from individual achievement.
Wrong -- The story confirms Priya was 'clearly the fastest player on the court.' Her growth was not about recognizing a lack of speed but about valuing teamwork over speed. hint: “What quality did the story say Priya had?” (41 chars)
Wrong -- Priya did not quit. She continued playing and her group outscored the starters for the first time, showing she became more successful by changing her approach. hint: “What was the result when Priya's group played?” (46 chars)
Wrong -- Priya's experience actually showed the opposite. Her growth demonstrated that speed alone does not make the best player; teamwork and making others better does. hint: “What did Coach Ellis understand about the starting lineup?” (58 chars)
Judge rubric
answer key validtrueAll four keys are defensible and directly supported by the text.
groundedtrueEvery question requires the story to answer.
distractor qualityfullDistractors draw from story elements (speed, Dana, bench, arguing) and each is wrong for a distinct, defensible reason. Length and register match.
grade fit4Vocabulary and reasoning demand (motivation, evidence, evaluation of growth) fit grade 7, though the questions are somewhat straightforward for the upper end of the band.
question type fittrueQ0 detail, Q1 motivation/interpretation, Q2 evidence-based inference, Q3 evaluate/analyze growth—matches the requested distribution.
explanation quality5Explanations quote the story directly; hints point to relevant passages without revealing answers.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated placement in second group.
Q2 inference clean Requires interpreting Priya's motivation from her actions; could arguably be evaluate_analyze but leans inference.
Q3 inference clean Evidence-based inference citing her copying Dana's behaviors.
Q4 evaluate_analyze clean Analyzes character growth and the pattern of her decision.
grade 7 · g7-tryout · rep 2

claude-haiku-4-5

FAILformat_valid

The Bench Decision -- realistic-fiction, reflective, a school gymnasium

Lexile not assigned Grade 7 range 760L-1410L 133 words, 9 sentences, mean 14.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +57 chars past the 50-char cap.

0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Priya had trained all summer for the point-guard spot, and at tryouts she was clearly the fastest player on the court. But when Coach Ellis read the starting lineup, Priya's name was in the second group. She almost argued. Instead, she watched the first practice from the bench and noticed something: the starter, Dana, never scored much, but every player around Dana scored more than usual. That night Priya understood what the coach had seen. Speed wins one-on-one, but Dana made the whole team better by passing early and calling out screens. The lineup was not about who was fastest. At the next practice Priya started passing the way Dana did. She scored fewer points that day than ever before -- and her group outscored the starters for the first time all season.

Q1. What position did Coach Ellis give to Priya at tryouts?

not cued key 53 chars vs mean distractor 54.7 (0.97x), rank 4.00
Answer option Length Grader picked
A She was chosen as the starting point guard immediately. 55 chars
B She was moved to point guard after the first practice. 54 chars
C She was the fastest player and earned the starter spot. 55 chars
D She was placed in the second group, not as a starter. KEY 53 chars with without
With story: picked D answerable
The story states her name was in the second group.
Story withheld: picked D answerable
Option D matches typical narrative of being placed in second group.
Explanations and hints
Key -- The story states: 'when Coach Ellis read the starting lineup, Priya's name was in the second group.' This is the key detail that sets up the rest of the story.
Wrong -- The story clearly says Priya was NOT in the starting lineup. She trained all summer for the spot but did not get it. hint: “Reread the sentence about Coach Ellis reading the lineup.” (57 chars)
Wrong -- While Priya was the fastest player, the story says she was placed in the second group, not as a starter. hint: “Speed alone did not determine who started.” (42 chars)
Wrong -- The story never says Priya became a starter. She learned from watching and changed her playing style instead. hint: “What did Priya do instead of arguing?” (37 chars)

Q2. Why did Priya decide to pass more like Dana instead of arguing with Coach Ellis?

length-cued key 83 chars vs mean distractor 50.3 (1.65x), rank 1.00
Answer option Length Grader picked
A She realized that making the whole team better mattered more than individual speed. KEY 83 chars with without
B She wanted to score fewer points than Dana did. 47 chars
C She was afraid the coach would bench her permanently. 53 chars
D She realized Dana was faster and could not compete. 51 chars
With story: picked A answerable
She realized team success mattered more than speed.
Story withheld: picked A answerable
Team improvement over individual speed fits the passing lesson.
Explanations and hints
Key -- The story says: 'That night Priya understood what the coach had seen. Speed wins one-on-one, but Dana made the whole team better by passing early.' Priya saw that the coach valued team play over individual talent.
Wrong -- Priya's goal was not to score less. She changed her approach because she understood a different way of playing was more valuable. hint: “What did Priya notice about Dana's scoring?” (43 chars)
Wrong -- The story does not mention Priya being afraid of punishment. She made a choice based on understanding the coach's reasoning. hint: “What did Priya do while watching from the bench?” (48 chars)
Wrong -- The story says Priya was the fastest player on the court. Her change was about strategy, not speed comparison. hint: “Who was clearly the fastest player?” (35 chars)

Q3. What evidence from the story shows that Priya's new approach to playing actually worked?

not cued key 63 chars vs mean distractor 50.7 (1.24x), rank 1.00
Answer option Length Grader picked
A Priya scored more points than she had all summer. 49 chars
B Coach Ellis moved Priya to the starting lineup immediately. 59 chars
C Her group outscored the starters for the first time all season. KEY 63 chars with without
D Dana started passing the ball more to Priya. 44 chars
With story: picked C answerable
Her group outscored the starters for the first time.
Story withheld: picked C answerable
Outscoring starters is concrete evidence the approach worked.
Explanations and hints
Key -- The story states: 'She scored fewer points that day than ever before -- and her group outscored the starters for the first time all season.' This concrete result proves her passing strategy was effective.
Wrong -- The story explicitly says Priya 'scored fewer points that day than ever before,' so this is the opposite of what happened. hint: “How many points did Priya score that day?” (41 chars)
Wrong -- The story does not mention Dana changing her behavior. It focuses on Priya's change and the resulting team improvement. hint: “Whose playing style changed at the next practice?” (49 chars)
Wrong -- The story does not say Priya became a starter. The evidence of success is that her group outscored the starters, not that she was promoted. hint: “What happened to Priya's group's score?” (39 chars)

Q4. What does Priya's choice to change her playing style reveal about her character?

not cued key 63 chars vs mean distractor 56.7 (1.11x), rank 2.00
Answer option Length Grader picked
A She was unhappy with her role and played differently out of anger. 66 chars
B She believed she was not fast enough to be a starter. 53 chars
C She wanted to prove Coach Ellis had made a mistake. 51 chars
D She values learning and team success over her individual pride. KEY 63 chars with without
With story: picked D answerable
She prioritized learning and team success over pride.
Story withheld: picked D answerable
Changing style shows valuing team success and growth over pride.
Explanations and hints
Key -- The story shows Priya 'almost argued' but instead watched and learned. She then changed her approach even though it meant scoring fewer points. This demonstrates she prioritizes understanding and team goals over defending her ego.
Wrong -- The story says Priya was 'clearly the fastest player on the court,' so her decision was not based on doubting her speed. hint: “What did the story say about Priya's speed?” (43 chars)
Wrong -- Priya's actions show she accepted the coach's decision and learned from it, not that she tried to prove the coach wrong. hint: “What did Priya do instead of arguing?” (37 chars)
Wrong -- The story shows Priya made a thoughtful choice after observing and understanding. Her change came from insight, not frustration. hint: “What did Priya understand that night?” (37 chars)
Judge rubric
answer key validtrueEach question has one defensible answer aligned with the story text.
groundedtrueAll questions reference specific story content.
distractor qualityfullDistractors are plausible, drawn from story elements (speed, Dana, starting lineup), and each wrong for a distinct reason.
grade fit4Vocabulary and reasoning demands (motivation, character analysis) fit grade 7, though the story itself is on the simpler side.
question type fittrueQ0=fact_detail, Q1=motivation, Q2=evidence-based inference, Q3=character/decision analysis — matches requested distribution.
explanation quality5Explanations quote the story directly; hints guide without revealing answers.
Skill tagging (3/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated placement in second group.
Q2 inference clean Interpreting Priya's motivation from her realization.
Q3 fact_detail ambiguous Asks for evidence but the answer is a directly stated fact; overlaps fact_detail and inference.
Q4 evaluate_analyze clean Analyzes character based on pattern of choices.
grade 7 · g7-tryout · rep 3

claude-haiku-4-5

FAILformat_valid

The Bench Decision -- realistic-fiction, reflective, a school gymnasium

Lexile not assigned Grade 7 range 760L-1410L 133 words, 9 sentences, mean 14.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +63 chars past the 50-char cap.

0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Priya had trained all summer for the point-guard spot, and at tryouts she was clearly the fastest player on the court. But when Coach Ellis read the starting lineup, Priya's name was in the second group. She almost argued. Instead, she watched the first practice from the bench and noticed something: the starter, Dana, never scored much, but every player around Dana scored more than usual. That night Priya understood what the coach had seen. Speed wins one-on-one, but Dana made the whole team better by passing early and calling out screens. The lineup was not about who was fastest. At the next practice Priya started passing the way Dana did. She scored fewer points that day than ever before -- and her group outscored the starters for the first time all season.

Q1. What position did Coach Ellis give to Priya at tryouts?

not cued key 53 chars vs mean distractor 57.3 (0.92x), rank 4.00
Answer option Length Grader picked
A She was moved to a different position after the first practice. 63 chars
B She was the fastest player but scored the most points. 54 chars
C She was placed in the second group, not as a starter. KEY 53 chars with without
D She was chosen as the starting point guard immediately. 55 chars
With story: picked C answerable
The story states her name was in the second group.
Story withheld: picked C answerable
Option C fits a story where Priya wasn't chosen as a starter.
Explanations and hints
Key -- The story states: 'when Coach Ellis read the starting lineup, Priya's name was in the second group.' This is the key detail that sets up the rest of the story.
Wrong -- The story clearly says Priya was not in the starting lineup. She trained all summer for the spot but did not get it. hint: “Reread the sentence about Coach Ellis reading the lineup.” (57 chars)
Wrong -- While Priya was the fastest player, the story does not say she scored the most points. She was benched despite her speed. hint: “What does the story say about Priya's speed?” (44 chars)
Wrong -- The story does not mention Priya changing positions. She remained a point guard but learned to play differently. hint: “Look at what Priya learned to do at the next practice.” (54 chars)

Q2. Why did Priya decide not to argue with Coach Ellis about the lineup?

length-cued key 76 chars vs mean distractor 56.7 (1.34x), rank 1.00
Answer option Length Grader picked
A She knew Dana was faster and accepted she could not compete. 60 chars
B She watched Dana play and realized the coach understood something important. KEY 76 chars with without
C Coach Ellis told her she would play in the second game. 55 chars
D She was too tired from training all summer to speak up. 55 chars
With story: picked B answerable
She watched and understood what the coach had seen.
Story withheld: picked B answerable
Watching Dana helped Priya understand the coach's reasoning.
Explanations and hints
Key -- The story says: 'She almost argued. Instead, she watched the first practice from the bench and noticed something.' Priya chose observation over complaint, which led to her understanding the coach's decision.
Wrong -- The story does not suggest Priya was tired or unable to speak. She made a conscious choice to watch instead. hint: “What did Priya do instead of arguing?” (37 chars)
Wrong -- The story says Priya was the fastest player on the court, not Dana. Speed was not the reason Priya stayed quiet. hint: “Who was clearly the fastest player on the court?” (48 chars)
Wrong -- The story does not mention the coach making any promises to Priya about future playing time. hint: “What happened between Priya and the coach after tryouts?” (56 chars)

Q3. Based on the story, what evidence shows that Dana's style of play made teammates perform better?

not cued key 69 chars vs mean distractor 60.7 (1.14x), rank 1.00
Answer option Length Grader picked
A Dana passed early and scored more points than any other starter. 64 chars
B Dana was the fastest player and won every one-on-one matchup. 61 chars
C Dana scored less but every player around Dana scored more than usual. KEY 69 chars with without
D Dana called out screens and the team won all their games. 57 chars
With story: picked C answerable
Directly stated in the story.
Story withheld: picked C answerable
Dana's unselfish play elevated teammates' scoring.
Explanations and hints
Key -- The story explicitly states: 'the starter, Dana, never scored much, but every player around Dana scored more than usual.' This direct comparison proves Dana's passing and communication improved team scoring.
Wrong -- The story says Priya was the fastest player, not Dana. Speed is not what made Dana effective. hint: “Who was clearly the fastest player on the court?” (48 chars)
Wrong -- While Dana did call out screens, the story does not mention whether the team won all their games. This goes beyond what the text supports. hint: “What specific scoring pattern did the story describe?” (53 chars)
Wrong -- The story says Dana 'never scored much,' which contradicts the idea that Dana scored more than other starters. hint: “How much did Dana score compared to teammates?” (46 chars)

Q4. What does Priya's decision to pass like Dana reveal about how she views winning?

not cued key 65 chars vs mean distractor 58.3 (1.11x), rank 1.00
Answer option Length Grader picked
A She believed being the fastest player was the only way to win. 62 chars
B She decided that passing was easier than playing defense. 57 chars
C She learned that team success matters more than personal scoring. KEY 65 chars with without
D She wanted to score fewer points to impress Coach Ellis. 56 chars
With story: picked C answerable
She valued team scoring over her own points.
Story withheld: picked C answerable
Adopting Dana's passing style shows valuing team over individual stats.
Explanations and hints
Key -- The story shows Priya 'scored fewer points that day than ever before -- and her group outscored the starters for the first time.' By choosing to pass and help teammates, Priya prioritized team performance over individual stats.
Wrong -- Priya's change in strategy contradicts this. She moved away from relying on speed and instead focused on passing and teamwork. hint: “What did Priya do differently at the next practice?” (51 chars)
Wrong -- Priya's goal was not to score less; it was to make the team better. Lower scoring was a result, not the purpose. hint: “What was the outcome when Priya's group outscored the starters?” (63 chars)
Wrong -- The story does not compare passing to defense. Priya's choice was about strategy and helping teammates, not about ease. hint: “Why did Dana make the whole team better?” (40 chars)
Judge rubric
answer key validtrueAll four keys are directly supported by the text. Q0 matches 'Priya's name was in the second group'; Q1, Q2, and Q3 all quote or paraphrase story lines accurately.
groundedtrueEvery question references specific story content and cannot be answered by general basketball knowledge alone.
distractor qualityfullDistractors use story elements (speed, Dana, passing, scoring) and each is wrong for a distinct reason. Length and register match the correct answers.
grade fit4Vocabulary and reasoning demand (evaluating motivation, evidence-based inference about team dynamics) fit grade 7 comfortably, though the text is short and slightly on the easy side.
question type fittrueQ0 = plot detail, Q1 = motivation interpretation, Q2 = evidence-based inference, Q3 = evaluation of a decision/values. Distribution matches request.
explanation quality5Explanations quote the story directly; hints redirect to relevant passages without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated placement in second group.
Q2 inference clean Requires inferring motivation from her choice to watch rather than argue; borderline with evaluate_analyze but primarily interpretation.
Q3 fact_detail clean Framed as evidence-based but the evidence is an explicitly stated fact; slightly weaker as 'inference' since it's directly quoted.
Q4 evaluate_analyze clean Analyzes what Priya's decision reveals about her values.
grade 7 · g7-tryout · rep 4

claude-haiku-4-5

FAILformat_valid

The Bench Decision -- realistic-fiction, reflective, a school gymnasium

Lexile not assigned Grade 7 range 760L-1410L 133 words, 9 sentences, mean 14.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +54 chars past the 50-char cap.

0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Priya had trained all summer for the point-guard spot, and at tryouts she was clearly the fastest player on the court. But when Coach Ellis read the starting lineup, Priya's name was in the second group. She almost argued. Instead, she watched the first practice from the bench and noticed something: the starter, Dana, never scored much, but every player around Dana scored more than usual. That night Priya understood what the coach had seen. Speed wins one-on-one, but Dana made the whole team better by passing early and calling out screens. The lineup was not about who was fastest. At the next practice Priya started passing the way Dana did. She scored fewer points that day than ever before -- and her group outscored the starters for the first time all season.

Q1. What position did Coach Ellis give to Priya at tryouts?

not cued key 53 chars vs mean distractor 50.0 (1.06x), rank 3.00
Answer option Length Grader picked
A She was placed in the second group, not as a starter. KEY 53 chars with without
B She earned the starting spot after the first practice. 54 chars
C She was cut from the team during tryouts. 41 chars
D She was chosen as the starting point guard immediately. 55 chars
With story: picked A answerable
The story states Priya's name was in the second group.
Story withheld: picked A answerable
Option A fits a typical narrative of not being chosen as starter.
Explanations and hints
Key -- The story states: 'when Coach Ellis read the starting lineup, Priya's name was in the second group.' This is the key detail that sets up the rest of the story.
Wrong -- The story clearly says Priya's name was in the second group, not the starting lineup, despite her being the fastest player. hint: “Reread what Coach Ellis announced about the lineup.” (51 chars)
Wrong -- Priya was placed in the second group and participated in practice, so she made the team. She was not cut. hint: “Look at what Priya did after the lineup was read.” (49 chars)
Wrong -- The story says Priya's group outscored the starters for the first time, but it does not say she became a starter. hint: “Check what happened at the next practice session.” (49 chars)

Q2. Why did Priya decide not to argue with Coach Ellis about the lineup?

length-cued key 76 chars vs mean distractor 52.7 (1.44x), rank 1.00
Answer option Length Grader picked
A She was too tired from training all summer to complain. 55 chars
B She knew she was not fast enough to be a starter. 49 chars
C Coach Ellis threatened to bench her if she complained. 54 chars
D She watched practice and realized Dana's passing made the whole team better. KEY 76 chars with without
With story: picked D answerable
She watched and noticed Dana made teammates better.
Story withheld: picked D answerable
This option provides a reasoned insight matching the story's likely lesson.
Explanations and hints
Key -- The story says: 'Instead, she watched the first practice from the bench and noticed something' about Dana making players score more. This observation changed her understanding of what the coach valued.
Wrong -- While Priya trained all summer, the story does not connect her tiredness to her decision not to argue. hint: “What did Priya do instead of arguing?” (37 chars)
Wrong -- The story contains no mention of the coach threatening Priya or warning her against arguing. hint: “Focus on Priya's own observations and thoughts.” (47 chars)
Wrong -- The story says Priya was 'clearly the fastest player on the court,' so speed was not her weakness.

Q3. Based on the story, what evidence shows that Priya understood the coach's decision was correct?

not cued key 64 chars vs mean distractor 56.0 (1.14x), rank 1.00
Answer option Length Grader picked
A Her group outscored the starters after she passed like Dana did. KEY 64 chars with without
B Dana admitted that Priya was a better point guard than her. 59 chars
C Coach Ellis told Priya she had made the right choice. 53 chars
D Priya scored more points than Dana at the next practice. 56 chars
With story: picked A answerable
Her group outscored the starters when she passed like Dana.
Story withheld: picked A answerable
Her group outscoring starters demonstrates she applied the lesson.
Explanations and hints
Key -- The story states: 'At the next practice Priya started passing the way Dana did...and her group outscored the starters for the first time all season.' This result proves the coach was right about team play mattering more than individual speed.
Wrong -- The story says Priya 'scored fewer points that day than ever before,' so she did not outscore Dana. hint: “What happened to Priya's scoring when she passed more?” (54 chars)
Wrong -- The story does not include any conversation between Priya and the coach after the practice. hint: “Look for proof in the actual game results.” (42 chars)
Wrong -- The story never shows Dana and Priya talking or Dana making any such admission. hint: “Find evidence in team performance, not in dialogue.” (51 chars)

Q4. What does Priya's choice to pass more instead of score more reveal about how she thinks?

not cued key 65 chars vs mean distractor 59.0 (1.10x), rank 1.00
Answer option Length Grader picked
A She wants to score fewer points to make other players happy. 60 chars
B She believes that being the fastest player is not important. 60 chars
C She thinks Dana is a better player than she will ever be. 57 chars
D She values team success over personal achievement and statistics. KEY 65 chars with without
With story: picked D answerable
She prioritizes team success over individual scoring.
Story withheld: picked D answerable
Choosing to pass shows valuing team success over personal stats.
Explanations and hints
Key -- Priya chose to 'start passing the way Dana did' even though she 'scored fewer points that day than ever before,' showing she prioritized the team's overall performance over her own scoring numbers.
Wrong -- Priya's speed was never the problem; the story shows she learned that speed alone does not win games, but she still values her athletic abilities. hint: “What did the coach value besides individual speed?” (50 chars)
Wrong -- Priya learned from Dana's style but the story does not suggest she believes Dana is better overall, just better at one specific skill. hint: “What did Priya learn from watching Dana play?” (45 chars)
Wrong -- Priya's goal was not to reduce her scoring to please others, but to help the team win by playing smarter basketball. hint: “What was the result of Priya's new approach?” (44 chars)
Judge rubric
answer key validtrueEach question has a single defensible answer supported directly by the text. Q0 is slightly awkwardly worded ('what position') but the correct answer about being in the second group is the only defensible choice.
groundedtrueAll questions require story content to answer.
distractor qualityfullDistractors are plausible, drawn from story elements (speed, Dana, scoring), each wrong for a distinct reason, and matched in length and register.
grade fit4Vocabulary and reasoning demands (motivation, evidence-based inference, values analysis) fit grade 7 well.
question type fittrueQ0=fact_detail, Q1=motivation/inference, Q2=evidence-based inference, Q3=evaluate/analyze pattern — all four requested types present.
explanation quality5Explanations quote the story directly and hints point toward relevant sections without revealing the answer.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly stated placement in second group.
Q2 inference clean Interprets Priya's motivation from her observation of Dana.
Q3 inference clean Requires citing the outscoring evidence to support understanding.
Q4 evaluate_analyze clean Analyzes what Priya's decision reveals about her values.
grade 7 · g7-tryout · rep 5

claude-haiku-4-5

FAILformat_valid

The Bench Decision -- realistic-fiction, reflective, a school gymnasium

Lexile not assigned Grade 7 range 760L-1410L 133 words, 9 sentences, mean 14.8
question count question text explanations hints present no hint on key hints ≤ 8 words hintsWithinCharLimit answer length balanced

Worst hint overage: +58 chars past the 50-char cap.

0% cued-pick (n=0) 100% informed guess (n=4) 0% slip proxy 0% declared unanswerable
Judges (2)
JudgeVerdictKey validGroundedDistractors
aws-bedrock:us.anthropic.claude-opus-4-7 (run of record)FAILtruetruefull
gpt:gpt-4oFAILtruetruefull
Story text (held constant across models)

Priya had trained all summer for the point-guard spot, and at tryouts she was clearly the fastest player on the court. But when Coach Ellis read the starting lineup, Priya's name was in the second group. She almost argued. Instead, she watched the first practice from the bench and noticed something: the starter, Dana, never scored much, but every player around Dana scored more than usual. That night Priya understood what the coach had seen. Speed wins one-on-one, but Dana made the whole team better by passing early and calling out screens. The lineup was not about who was fastest. At the next practice Priya started passing the way Dana did. She scored fewer points that day than ever before -- and her group outscored the starters for the first time all season.

Q1. What position did Coach Ellis give to Priya at tryouts?

not cued key 53 chars vs mean distractor 55.0 (0.96x), rank 4.00
Answer option Length Grader picked
A She was the fastest player but scored the most points. 54 chars
B She was placed in the second group, not as a starter. KEY 53 chars with without
C She was told to watch Dana and learn passing techniques. 56 chars
D She was chosen as the starting point guard immediately. 55 chars
With story: picked B answerable
The story states Priya's name was in the second group.
Story withheld: picked B answerable
Placed in second group is the typical setup for this narrative.
Explanations and hints
Key -- The story states: 'when Coach Ellis read the starting lineup, Priya's name was in the second group.' This is the key detail that sets up the rest of the story.
Wrong -- The story explicitly says Priya was NOT in the starting lineup. She trained for the point-guard spot but did not get it. hint: “Reread the sentence about Coach Ellis reading the lineup.” (57 chars)
Wrong -- While Priya was the fastest player, the story does not say she scored the most points. Her placement was about more than speed. hint: “Speed alone did not determine who started.” (42 chars)
Wrong -- Priya was not told to do this by the coach. She chose to watch and learn on her own from the bench. hint: “Who decided what Priya would do on the bench?” (45 chars)

Q2. Why did Priya decide not to argue with Coach Ellis about the lineup?

not cued key 63 chars vs mean distractor 51.7 (1.22x), rank 1.00
Answer option Length Grader picked
A She chose to observe and understand the coach's decision first. KEY 63 chars with without
B She knew Dana was a better player than she was. 47 chars
C She was too tired from training all summer to speak. 52 chars
D Coach Ellis told her to sit and watch the starters play. 56 chars
With story: picked A answerable
She chose to watch and understand rather than argue.
Story withheld: picked A answerable
Observing first fits the learning arc of the story.
Explanations and hints
Key -- The story says 'She almost argued. Instead, she watched the first practice from the bench,' showing she decided to learn before complaining.
Wrong -- The story does not suggest Priya was tired or unable to speak. Her choice was deliberate, not due to exhaustion. hint: “What did Priya do instead of arguing?” (37 chars)
Wrong -- The story says Dana 'never scored much,' so Priya did not think Dana was better. Priya's realization came after watching, not before. hint: “When did Priya understand the coach's choice?” (45 chars)
Wrong -- The story does not say the coach told Priya to watch. Priya made this choice on her own while on the bench. hint: “Did the coach give Priya instructions about what to do?” (55 chars)

Q3. Based on the story, what evidence shows that Dana's playing style helped the team score more points?

length-cued key 116 chars vs mean distractor 62.7 (1.85x), rank 1.00
Answer option Length Grader picked
A Coach Ellis chose Dana as the starter because she was the fastest. 66 chars
B Every player around Dana scored more than usual, and Priya's group outscored the starters when she passed like Dana. KEY 116 chars with without
C Dana scored fewer points but was still the fastest player. 58 chars
D Priya scored fewer points that day than ever before in practice. 64 chars
With story: picked B answerable
Both observations show Dana's passing style boosted team scoring.
Story withheld: picked B answerable
This option provides concrete evidence of Dana's passing benefiting scoring.
Explanations and hints
Key -- The story states 'every player around Dana scored more than usual' and later 'her group outscored the starters for the first time all season' after Priya copied Dana's passing style. These two details prove Dana's approach worked.
Wrong -- While the story says Dana scored fewer points, it does not connect this to team success. Speed was not what made the team better. hint: “Look at what happened to other players around Dana.” (51 chars)
Wrong -- Priya scoring fewer points is mentioned, but this alone does not prove Dana's style helped. The key is what happened to the whole group. hint: “Did Priya's group win or lose that practice?” (44 chars)
Wrong -- The story says the lineup was 'not about who was fastest.' The coach's choice was based on Dana making the team better, not on speed. hint: “What did the coach see that speed alone could not explain?” (58 chars)

Q4. What does Priya's decision to pass more and score less tell us about what she learned?

not cued key 78 chars vs mean distractor 66.3 (1.18x), rank 1.00
Answer option Length Grader picked
A She learned that helping teammates succeed matters more than personal scoring. KEY 78 chars with without
B She learned that being the fastest player is not important anymore. 67 chars
C She learned that scoring fewer points would make her a better starter. 70 chars
D She learned that Dana was the only player who could pass well. 62 chars
With story: picked A answerable
She learned team success matters more than personal points.
Story withheld: picked A answerable
The moral centers on teamwork over individual scoring.
Explanations and hints
Key -- Priya changed her style to pass like Dana, 'scored fewer points that day than ever before' yet 'her group outscored the starters.' This shows she valued team success over individual stats.
Wrong -- The story does not say speed became unimportant. It says speed alone was not enough for the point-guard role. Priya's speed was still valuable. hint: “What was true about Priya's speed at the start?” (47 chars)
Wrong -- Priya did not change her style to become a starter. She changed it because she understood what the coach valued in a point guard. hint: “Did Priya change her style to get a starting spot?” (50 chars)
Wrong -- Priya learned Dana's passing approach and used it successfully, proving others could pass well too. The story shows Priya could adopt this skill. hint: “What did Priya do after watching Dana play?” (43 chars)
Judge rubric
answer key validtrueEvery key is directly supported by the story. Q0 quotes the lineup detail; Q1 quotes Priya's choice to observe; Q2 combines the two key evidence details; Q3 captures the theme.
groundedtrueAll four items are answerable from the story text alone.
distractor qualityfullDistractors are plausible, drawn from story elements (speed, Dana, coach), and each is wrong for a different reason. Length and register match the correct answers.
grade fit4Vocabulary and reasoning demands (motivation, evidence-based inference, evaluating a decision) fit grade 7 well, though the story is short and the language is on the easier side of the band.
question type fittrueQ0 = plot-relevant detail; Q1 = motivation interpretation; Q2 = evidence-based inference requiring citing details; Q3 = evaluation of a decision/pattern. All four requested types present.
explanation quality5Explanations quote the story directly ('every player around Dana scored more than usual', 'her group outscored the starters'); hints redirect without revealing answers.
Skill tagging (4/4 map cleanly)
ItemKnowledge componentMaps cleanlyNote
Q1 fact_detail clean Directly asks for a stated fact about lineup placement.
Q2 inference clean Priya's reason is implied by 'almost argued. Instead, she watched'—reader infers motivation. Could also be tagged evaluate_analyze but inference is primary.
Q3 inference clean Requires combining two textual details as evidence—classic evidence-based inference.
Q4 evaluate_analyze clean Asks reader to analyze what Priya's behavior change reveals about her learning/values.

Length cue by grade band

Both columns are deterministic. The cue intensifies as grade rises, which is the finding that most needs a second look: at higher grades the correct answer to an inference question may be intrinsically longer, so part of this may be task type rather than defect.

GradeCued itemsMean key rankItems
grade 15%2.6520
grade 48%2.2536
grade 730%1.7020

How to read an item

In the item browser, each option carries its character count. A with badge marks the option the judge chose while reading the story; a without badge marks what it chose with the story withheld. An item where the without badge falls on the key, and the judge flagged the item unanswerable, is an item the option set gave away.

Probe ledger

Every live probe behind the enforcement findings, with the raw response retained on disk and checked for existence when this page is built. 23 probes across 2 models.

proven 15directional (n=1) 8
provenDeterministic, and established by an adversarial demonstration or an in-call control. A claim that a constraint is ignored needs exactly one clean case, so these are settled.
directional (n=1)Deterministic string measurement, but one quiz per arm. An existence proof with a plausible mechanism. Enough to justify building the fix, not enough to quote as an expected effect size.
untestedA hypothesis with a named experiment that would settle it. Not yet run.
judge-dependentDescends from a single judge's opinion over 36 quizzes on four synthetic stories. The only educator who has reviewed it is the author of the evaluation, who is not independent of it. No child has read any of it. Do not rank models on these.
citedExternal source, retrieved 2026-08-17. Someone else's measurement, not ours.
P1 claude-haiku-4-5 maxLength proven ignored

Can the 50-character hint cap be moved into the schema?

Constraint. maxLength: 50 on hint_explanation

Observed. 8 of 16 hints over the cap. Mean 51.2, max 64. HTTP 200, no error.

p1-body-maxlength.json p1-out.json
P3a claude-haiku-4-5 pattern proven enforced

Adversarial: does pattern beat a prompt that demands the opposite?

Constraint. pattern ^[a-z ]{40,60}$ against a prompt demanding capitals, a terminal '!', and at least 120 characters

Observed. Returned a 60-character all-lowercase unpunctuated string. Pattern-conformant and prompt-defiant on every axis.

p3a.json p3a-out.json
P3b claude-haiku-4-5 pattern proven rejected

How permissive can the character class be?

Constraint. pattern ^.{1,50}$

Observed. 400 ValidationException. Cannot apply a range quantifier to this regex.

p3b.json
P4 claude-haiku-4-5 pattern proven rejected

Find the grammar compilation ceiling.

Constraint. pattern ^[ -~]{1,50}$, 95 printable symbols

Observed. 400 Grammar compilation timed out. Two attempts, roughly 3 minutes each.

p4.json
P5 claude-haiku-4-5 pattern directional (n=1) enforced

Apply the constraint to the real production quiz prompt.

Constraint. pattern ^[A-Za-z ]{1,50}$ on hint_explanation, production prompt otherwise byte-identical

Observed. 0 of 16 hints over the cap. Mean 36.6, max 48. 200 in about 41s.

p5.json p5-body-both.json p5-out.json
P6 claude-haiku-4-5 negative constraints directional (n=1) safe but ineffective

Does the earlier ban on DO NOT phrasing transfer to haiku?

Constraint. Six aggressive DO NOT / NEVER bullets, no structured outputs, JSON by template only

Observed. Valid JSON, 4 questions, 12 well-formed hints, zero error objects. But it was the worst arm for option length: spread about 32, key longest 4 of 4. Explicitly forbidding the behaviour did not suppress it.

p6-donot.json p6-out.json
P7 claude-haiku-4-5 pattern directional (n=1) enforced, with collateral damage

Band the option lengths as well as the hints.

Constraint. hint pattern plus option band ^[A-Za-z ,.]{30,55}$

Observed. 0 of 16 hints over the cap, option spread 7.2 characters. One hint came back cut mid-word at exactly 50 characters: 'Think about what Marcus put in his locker in a hur'. The decoder cuts mid-token rather than rephrasing.

p7.json p7-out.json
P8 claude-haiku-4-5 pattern + prompt target directional (n=1) enforced, damage mitigated

Stop the bound from binding by moving the prompt target inside it.

Constraint. Same patterns, plus a prompt asking for 4 to 7 words and about 30 to 40 characters

Observed. 0 of 16 hints over the cap, mean 34.0, max 40, and no hint within 9 characters of the cap. Option spread fell to 2.2. Truncation gone. Options stayed content-plausible rather than becoming filler.

p8.json p8-out.json
P2c claude-haiku-4-5 enum proven enforced

Baseline that the grammar is live on this path.

Constraint. pattern ^[A-Z]{3}$

Observed. Returned RED. 200.

p2c.json
pat1 nova-2-lite pattern proven inert

Adversarial, mirroring P3a exactly.

Constraint. pattern ^[a-z ]{40,60}$ against a prompt demanding capitals, '!', and at least 120 characters

Observed. A 162-character capitalised sentence ending in '!'. The prompt won on every axis and the pattern lost on every axis. Exactly the opposite outcome to haiku on the same design.

pat1-adversarial.json
pat2 nova-2-lite enum vs pattern control proven grammar is live and selective

Rule out a broken code path. If the grammar were simply not engaged, nothing would bind. Put a contradicted enum next to a contradicted pattern in one call.

Constraint. One schema, temperature 0: enum ['calm','tense','playful','somber'] with the prompt demanding 'ecstatic', beside pattern ^[a-z ]{40,60}$ with the prompt demanding the opposite

Observed. The enum bound and emitted 'tense'. The pattern did not, and emitted 162 capitalised characters ending in '!'. Same request, same object, same decode.

pat2-enum-vs-pattern-control.json
pat3 nova-2-lite pattern proven inert

Is an unsatisfiable pattern a production incident?

Constraint. pattern ^[b]{1,3}$ on a field asked to hold a water-cycle summary, so only b, bb or bbb can match

Observed. A normal 175-character prose summary. 200, stopReason tool_use, 984ms. No error, no hang, no truncation, no degradation.

pat3-pathological-pattern.json
pat4 nova-2-lite pattern proven inert

Does an unsupported regex feature at least error?

Constraint. pattern ^(?=.*z)[a-z ]{10,20}$, using a lookahead

Observed. Returned 'zebra'. 200 in 816ms, no ValidationException. Note 'zebra' is 5 characters and would fail {10,20} anyway, so the pattern is never evaluated at all.

pat4-lookahead-pattern.json
pat5 nova-2-lite pattern directional (n=1) inert

Apply the constraint to the real production quiz prompt, mirroring P5 and P7.

Constraint. hint pattern ^[A-Za-z ]{1,50}$ and option band ^[A-Za-z ,.]{30,55}$, 4 replicates at temperature 0.7

Observed. Hint violations 25 of 33, unchanged. Of 33 hints, 2 exceeded 50 characters and 23 were within 50 but carried punctuation the pattern forbids. Options: 25 of 44 outside the band. Mean spread 21.5. Key strictly longest in 6 of 11 questions. The banding did nothing.

pat5-quiz-rep1.json pat5-quiz-rep2.json pat5-quiz-rep3.json pat5-quiz-rep4.json
nova-maxlength nova-2-lite maxLength proven ignored

Does the schema length cap work where pattern does not?

Constraint. maxLength: 50 on a free-text field

Observed. Hints of 475, 533 and 612 characters. 200, no error.

body-strict-maxlength.json tool-strict-maxlength.json
probe2 nova-2-lite Bedrock structured outputs proven not supported

Is the structured-output path available at all?

Constraint. strict: true

Observed. ValidationException.

probe2-strict-with-maxLength.err.txt
probe3 nova-2-lite Bedrock structured outputs proven not supported

Isolate whether maxLength caused the rejection rather than strict mode.

Constraint. strict: true, no maxLength

Observed. ValidationException. So it is the mode, not the keyword.

probe3-strict-no-maxLength.err.txt
probe5 nova-2-lite Bedrock structured outputs proven not supported

Try the other structured-output entry point.

Constraint. outputConfig with maxLength

Observed. ValidationException. Nova's model card lists structured outputs under Not Supported.

probe5-outputConfig-maxLength.err.txt
probe6 nova-2-lite prompt-only constraint directional (n=1) baseline

Baseline for the constraint stated in the prompt rather than the schema.

Constraint. Character limit stated in the prompt text

Observed. Recorded for comparison against the schema arms.

probe6-constraint-in-prompt.json
probe7 nova-2-lite schema description directional (n=1) baseline

Does stating the limit in a schema description do anything?

Constraint. Limit expressed in the JSON schema description field

Observed. Recorded for comparison. A description is documentation, not a constraint.

probe7-constraint-in-schema-description.json
probe8 nova-2-lite enum proven enforced

Establish that at least one string constraint binds hard.

Constraint. enum on a closed set

Observed. Bound. This is the lever a required per-item skill tag would use.

probe8-enum-enforcement.json
probe9 nova-2-lite negative constraints directional (n=1) effective

Does AWS's own prescribed DO NOT phrasing work?

Constraint. Seven capitalised DO / DO NOT / MUST bullets

Observed. A clean quiz with every hint under 50 characters.

probe9-do-not-constraints.json
probe10a nova-2-lite prompt caching proven engages

Does caching engage on our request shape?

Constraint. Cache checkpoint on the static prompt prefix

Observed. 2,392 of 2,572 input tokens served from cache, 93 percent.

probe10a-cache-write.json probe10b-cache-read.json

Prompting techniques

The practitioner's digest of the five research briefs: what to actually do when writing a quiz-generation prompt, why, and how much weight the reason carries. 24 techniques, of which 17 rest on published assessment research or an official item specification rather than on our own measurements alone.

A citation is not a measurement of our stack. Several techniques below are well supported in the education literature and completely untested here. Those are the most dangerous entries on the page, because a reference makes an untested thing feel settled. Every card shows its evidence tier next to its sources for exactly that reason: read them together, never separately.

Prompt structure and placement 3

Where each part of the instruction goes. The two candidates disagree here, so this is the first place a shared template breaks.

Put the fixed contract in the system turn and the varying story in the user turn

both modelscited

Role, output contract and rules go in the system block. The story, grade and per-story selections go in the user turn.

Why. Both vendors document this split. On Nova the system prompt is documented to take precedence over user turns, so a rule placed in the user turn competes with the story rather than governing it.

Sources: awsNova anthropicPrompting

Tag the story with explicit delimiters

both modelscited

Fence the passage with named markers rather than bare ### rules, so the model can tell instruction from data.

Why. Anthropic's current guidance still recommends XML tagging when a prompt mixes instructions, context, examples and variable input, which is exactly our shape. Our existing harness doc calls XML a legacy aid; the live vendor page disagrees, and this conflict is unresolved.

Sources: anthropicPrompting awsNova

Do not model prompt-cache savings on Haiku at our request size

haiku onlyproven

Caching needs a 4,096-token minimum per checkpoint. Our full-story quiz request measures 751 to 806 input tokens.

Why. Measured. Nova caches at our size and served 93 percent of input tokens from cache; Haiku will not engage at all, so any cutover cost model that assumes it is wrong.

No external source. Measured on our stack, or a design proposal.

Constraint phrasing 3

How to word a rule so the model follows it, including the units a model can and cannot count.

Ask for word counts, never character counts

both modelsproven

Replace 'under 50 characters' with a word budget. In our corpus every hint of seven words or fewer was already under 50 characters, 247 of 247.

Why. Models read subword tokens, not characters, so a character limit is a unit they cannot see. CharBench measures frontier models at 43.6 percent on character indexing against over 70 percent on token-level equivalents. AWS's own Nova guidance lists the legal units as words, bullet points or paragraphs, and characters are not among them.

Sources: charbench2025 awsNova

Fork the negative-constraint policy per model

forks per modelproven

Nova gets capitalised DO / DO NOT / MUST bullets. Haiku tolerates them but treats them as a nudge, so its hard rules belong in the schema instead.

Why. The an earlier question-generation defect ban on DO NOT phrasing came from gpt-3.5-turbo returning error objects, and it does not transfer. AWS actively prescribes DO NOT for Nova, and a seven-bullet negative prompt produced a clean quiz with every hint under the cap. On Haiku, a twice-stated DO NOT was violated in the same response where a different DO NOT scored 12 of 12.

Sources: awsNova

Do not expect a restated rule to fix an adherence failure

both modelsproven

The length-balance rule is already in our prompt verbatim and fails at 48/54/46 percent. An emphatic restatement produced key-is-longest in 4 of 4 questions.

Why. Measured on our own stack, and independently reproduced in the only published measurement of this artifact, whose prompt also already contained a balance instruction that its exemplars overrode.

Sources: ermilov2026

Worked examples 3

The most under-examined part of our own prompt, and the one carrying a defect it explicitly forbids.

Audit the worked example against every rule the prompt states

both modelsdirectional (n=1)

Our single exemplar's key is 56 characters against distractors of 46, 46 and 49 -- strictly the longest, at 1.19x the mean, which slips under our own 1.25x detector.

Why. Demonstrations transmit the FORMAT of a sequence rather than its semantics, and option-length balance is a format property. The one other team to measure this artifact found their own examples overrode their own stated rule. The pre-LLM automatic-item-generation tradition already screens the parent item first for exactly this reason.

Sources: min2022 ermilov2026 gierlLaiAIG

Carry three to five diverse examples, not one

both modelscited

One example cannot demonstrate variance. If the key sits at a different length rank in each example, no single rank is being taught.

Why. Anthropic recommends 3 to 5 examples and explicitly warns they must be diverse enough that the model does not pick up unintended patterns. AWS gives no count and names quality and diversity as the drivers. Both cut against shipping exactly one.

Caveat. More is not monotonically better; over-prompting is a documented failure mode. Treat 3-5 as a range to test, not a target to hit.

Sources: anthropicPrompting awsNova tang2026

Treat exemplar contamination as a live hypothesis, not a finding

both modelsuntested

The A/B: rewrite the exemplar so the key is mid-rank in length and matched in specificity, vary those two properties independently across arms, and hold everything else fixed.

Why. The mechanism is supported and our specific case is not. In-context length bias is measured at onset around eight demonstrations, in classification, on 2.7B-8B models; we ship one, in generation, on frontier models. Demonstration-override capacity also scales with model size, yet our three models are near-tied, which a purely exemplar-driven account does not predict.

Sources: schoch2025 semanticAnchors2025 amini2025

Schema enforcement 4

What can be made a guarantee rather than a request. This is where the two candidates differ most.

Never use maxLength as an enforcement mechanism

both modelsproven

It is accepted, returns HTTP 200, and is enforced not at all on either candidate. Nova returned 475, 533 and 612-character hints against a cap of 50.

Why. AWS documents maxLength as unsupported AND documents that unsupported schema features return a 400 immediately. Neither half holds in practice. This is worse than a hard error because it ships: the schema reads as a contract and is not one.

Sources: bedrockStructured

On Haiku, express hard constraints as a pattern

haiku onlydirectional (n=1)

A regex on the field is enforced at the decoder. On our real quiz it took hint violations from 8 of 16 to 0 of 16 and cut option-length spread from 18.0 characters to 2.2.

Why. Constrained decoding removes illegal tokens from the candidate pool rather than discouraging them, so the constraint holds even against a prompt demanding the opposite. Undocumented by both vendors, which is why it needs a startup assertion and a code-side fallback.

Caveat. One quiz per arm. An existence proof, not an effect size. And the character class must be widened before shipping or apostrophes and hyphens become unwriteable.

No external source. Measured on our stack, or a design proposal.

On Nova, enum is the only construct that binds

nova onlyproven

An adversarially contradicted enum held in the same call where an adversarially contradicted pattern did not. Nothing in Nova's grammar can bound the length of a free-text field.

Why. Measured with an in-call control, which rules out a dead code path. It follows that length enforcement on Nova must live in application code, with no schema-level fallback available.

Caveat. An enum on a metadata field is a trap: it binds the label, not the string it describes. A model can emit "short" beside a 200-character hint.

No external source. Measured on our stack, or a design proposal.

Require a per-item skill enum to enforce the blueprint

both modelsuntested

Make each item declare its reading skill from a closed list, so the mix can be checked rather than hoped for.

Why. Our unprompted mix was 61 fact-detail and 3 vocabulary out of 140 items, which is what the prompt asks for. Enum is the one construct proven to bind on both models.

Caveat. Binds the label, not the item. The declared-versus-judged confusion matrix decides whether a cross-check is needed.

Sources: naep2026 pirls2021

Item-writing craft 7

What assessment science knows about writing options that do not give the answer away. Best-cited section on the page.

Make the story-vocabulary rule symmetric across all four options

both modelscited

Our prompt tells the model to build wrong answers from words in the story but says nothing about the key, so the key is written as an original, fuller restatement.

Why. The similarity strategy for distractor development is sound, but the canonical review states that structural similarity includes length, complexity, formatting and grammar -- a property of the option SET. Applying it to three of four options makes the fourth stand out by construction.

Sources: gierl2017 haladyna2002 millman1965 brozo1984

Ask for options as short as possible, not merely equal

both modelscited

NAEP's wording gives a direction rather than only a parity constraint.

Why. Parity alone can be satisfied by inflating the distractors up to a bloated key, which is exactly what the one published mitigation did: it lengthened distractors and left the correct answer untouched, moving the rate from over 55 percent to 43.2 percent without eliminating it.

Sources: naep2026 ermilov2026

Put the central idea in the stem, not the options

both modelscited

The question should carry the substance; options should be short completions of it.

Why. The only guideline in the canonical taxonomy with 100 percent textbook consensus and zero dissent. It also mechanically shortens options, which attacks the length cue from a different direction.

Sources: haladyna2002

Name the distractor types rather than leaving them to invention

both modelscited

Ask for wrong answers of specified kinds -- a plausible misreading, a detail from the wrong part of the passage, an over-generalisation -- rather than three free-form wrong answers.

Why. Operational reading assessments publish distractor typologies rather than relying on the writer to invent categories, and the guideline literature asks that distractors reflect typical student errors. Our current prompt asks only that each be wrong for a different reason.

Caveat. Untested on our stack. A citation is not a measurement, and pushing hard on distractor plausibility has a documented failure mode of producing second defensible answers.

Sources: smarterbalanced haladyna2002 gierl2017

Keep four options

both modelscited

Do not act on the meta-analytic finding that three options are optimal.

Why. The measured gain is items per testing minute, which is irrelevant to a fixed four-item quiz, and our schema, UI and per-answer explanation contract are all built on four. NAEP and state ELA programmes use four for reading. Spend the effort on making the third distractor functional instead.

Sources: rodriguez2005 naep2026

Detect the cue in code, never with a model judge

both modelscited

A length comparison across the option set, run deterministically, gates a prompt change with no model spend.

Why. The head-to-head on exactly this flaw reports rule-based F1 of 0.80 to 1.00 against GPT-4's 0.18 to 0.44. Separately, models answering multiple-choice items are themselves biased by option length, so a judge-based check inherits the bias it is meant to detect.

Sources: moore2023 moore2024 zhao2025 schmucker2025 pham2018

Keep the story-withheld pass as a standing acceptance gate

both modelscited

Answer every quiz twice, once with the passage and once without, and treat a high withheld-solve rate as a blocker rather than a diagnostic.

Why. This is the field's established partial-input validity check, and NAEP's specifications state directly that items should not be answerable from text-independent knowledge without reading the passage. The identical design is used on LLM-generated reading items, where answerability minus guessability is defined as text informativity.

Sources: kaushik2018 sauberli2024 naep2026 balepur2024

Call architecture 4

Changes to how many calls we make and in what order, rather than to the words in any one of them.

Split option generation from explanation generation

both modelsuntested

Write the four options in one call, then the explanations in a second call once the option set is fixed.

Why. Our prompt requires the key's explanation to quote the story. That pressure to make the key defensible plausibly leaks into the key's option TEXT, making it the most specific option by construction. The flaw literature names this etiology in human writers as the effort to make the correct response indisputable.

Caveat. The only candidate fix that changes lambda control flow rather than prompt text, and the most durable if it lands. Untested.

Sources: haladyna2002

Generate, validate, repair -- do not trust a single pass

both modelscited

Run the deterministic checks after generation and regenerate or repair the failures, rather than assuming the prompt held.

Why. The systematic review of 71 studies concludes LLMs are useful drafting tools and that evidence does not support unsupervised use in summative assessment. A repair loop is the engineering form of that conclusion, and it is the only enforcement route available on Nova.

Sources: kiyak2026

Consider overgenerating and ranking distractors

both modelsuntested

Produce more candidate distractors than needed and select the set that is most balanced and most plausible.

Why. Established for distractor generation in the literature, mostly in mathematics. It attacks the cue by selection rather than by instruction, so it does not depend on adherence.

Caveat. Real cost implications at our volume. Untested on reading comprehension.

Sources: overgenerate2024

Cross-family rescore before trusting any judged ranking

both modelsjudge-dependent

Re-judge the same stored quizzes with the option order held identical, changing only the judge.

Why. Self-preference bias in model judges is documented in general. The one K-12 study with expert adjudication found a Claude judge unusually strict rather than lenient, so the literature points both ways and cannot settle it. Our own rescore returned 70 percent verdict agreement, with the second judge disagreeing only in the strict direction.

Sources: wataoka2025 he2026 kiyak2026

Bibliography

Every source referenced above, grouped by what kind of evidence it is. Publication status is recorded because a good deal of what is written about prompting is preprint or vendor documentation, and the difference should not require leaving the page to discover.

peer-reviewed 13

Published in a peer-reviewed venue. The strongest evidence class on this page.

balepur2024 -- Balepur, Ravichander & Rudinger (2024). Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? ACL 2024.
Choices-only prompting beats majority baseline in 11 of 12 cases. Analyses memorisation and choice priors, and explicitly does NOT analyse option length.
https://aclanthology.org/2024.acl-long.555/
brozo1984 -- Brozo, Schmelzer & Spires (1984). A Study of Test-Wiseness Clues in College and University Teacher-Made Tests. ERIC ED240928.
Measured cue frequency in real teacher-written tests; direct opposites were the most common cue at 151 of 1,220.
https://files.eric.ed.gov/fulltext/ED240928.pdf
gierl2017 -- Gierl, Bulut, Guo & Zhang (2017). Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review. Review of Educational Research 87(6), 1082-1116.
Cross-tabulates six published guideline sets; the source of the similarity strategy for distractor development.
https://doi.org/10.3102/0034654317726529
gierlLaiAIG -- Gierl & Lai. Using Automatic Item Generation to Create Multiple-Choice Test Items. NCME instructional module on automated item generation.
The parent-item tradition: children inherit the parent's template, so parent quality is screened first. The pre-LLM ancestor of the exemplar problem.
https://ncme.org/wp-content/uploads/2025/10/Module-34-Automated-Item-Generation-Gierl-Lai.pdf
haladyna2002 -- Haladyna, Downing & Rodriguez (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education 15(3), 309-334.
The canonical 31 guidelines, validated against 27 textbooks and 27 post-1990 studies.
https://site.ufvjm.edu.br/fammuc/files/2016/05/item-writing-guidelines.pdf
kaushik2018 -- Kaushik & Lipton (2018). How Much Reading Does Reading Comprehension Require? EMNLP 2018, 5010-5015.
The ancestor of our story-withheld pass: passage-only and question-only ablations as a validity check.
https://aclanthology.org/D18-1546/
kiyak2026 -- Kıyak, Kaya & Emekli (2026). Validity of AI-generated multiple-choice questions in medical education: a systematic review. Postgraduate Medical Journal.
71 studies, 24 countries. Concludes LLMs are useful drafting tools and that evidence does not support unsupervised use in summative assessment.
https://academic.oup.com/pmj/advance-article/doi/10.1093/postmj/qgag057/8688271
millman1965 -- Millman, Bishop & Ebel (1965). An Analysis of Test-Wiseness. Educational and Psychological Measurement 25(3), 707-726.
The founding taxonomy of test-wiseness, including length and specificity cues.
https://journals.sagepub.com/doi/abs/10.1177/001316446502500304
min2022 -- Min, Lyu, Holtzman, Artetxe, Lewis, Hajishirzi & Zettlemoyer (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP 2022.
Randomising demonstration labels barely hurts across 12 models; what demonstrations teach is label space, input distribution and sequence FORMAT. Option-length balance is a format property.
https://aclanthology.org/2022.emnlp-main.759/
moore2023 -- Moore, Nguyen & Stamper (2023). Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods. ECTEL / arXiv:2307.08161.
The head-to-head that decides our validator architecture: rule-based F1 0.80-1.00 against GPT-4's 0.18-0.44 on longest-option-correct.
https://arxiv.org/pdf/2307.08161
pham2018 -- Pham, Besanko & Devitt (2018). Examining the impact of specific types of item-writing flaws on student performance and psychometric properties. MedEdPublish.
Flaw-by-flaw effect on student performance rather than flaw counts in aggregate.
https://pmc.ncbi.nlm.nih.gov/articles/PMC10711986/
rodriguez2005 -- Rodriguez (2005). Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement: Issues and Practice 24(2), 3-13.
Cited here as the reason NOT to change our option count: the measured gain is items per testing minute, which is irrelevant to a fixed four-item quiz.
https://onlinelibrary.wiley.com/doi/10.1111/j.1745-3992.2005.00006.x
sauberli2024 -- Säuberli & Clematide (2024). Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models. READI @ LREC-COLING 2024.
Uses our exact two-pass design on LLM-generated reading items and defines text informativity as answerability minus guessability.
https://arxiv.org/abs/2404.07720

standard 3

An official assessment framework or item specification. Not a study, but the rules a real testing programme operates under.

naep2026 -- National Assessment Governing Board. Assessment and Item Specifications for the 2026 NAEP Reading Assessment.
The closest official standard to our use case: grade 4 and 8 reading comprehension, four-option single-selection items.
https://www.nagb.gov/content/dam/nagb/en/documents/publications/frameworks/reading/naep-2026-reading-specifications.pdf
pirls2021 -- Mullis & Martin, eds. (2019). PIRLS 2021 Assessment Frameworks. TIMSS & PIRLS International Study Center, Boston College.
Source of the 20/30/30/20 grade-4 comprehension-process blueprint.
https://pirls2021.org/wp-content/uploads/sites/2/2019/04/P21_Frameworks.pdf
smarterbalanced -- Smarter Balanced Assessment Consortium. ELA/Literacy item and distractor specifications.
Reading distractor typology: named categories of wrong answer rather than free invention.
https://portal.smarterbalanced.org/library/en/elaliteracy-summative-assessment-blueprint.pdf

preprint 12

Posted to a preprint server and not peer-reviewed. Read with the same care you would give an internal memo.

amini2025 -- Amini, Ahmadi, Xiong, Zhang & Qiao (2025). Prompting Strategies for Language Model-Based Item Generation in K-12 Education. arXiv:2508.20217, University of Florida.
Directly on our task and a confirmed NEGATIVE on structural artifacts: it does not report the length cue. Listed so nobody assumes the field has already found what we found.
https://arxiv.org/pdf/2508.20217
charbench2025 -- Uzan & Pinter. CharBench: character-level reasoning in language models. arXiv:2508.02591.
Frontier models at 43.6 percent on first-occurrence character indexing against over 70 percent on token-level analogues. The evidence behind asking in words rather than characters.
https://arxiv.org/abs/2508.02591
ermilov2026 -- Ermilov (2026). FormationEval, an open multiple-choice benchmark for petroleum geoscience. arXiv:2601.02158.
Single author, not peer-reviewed. The ONLY published measurement of the length cue on model-authored items: uniquely longest correct in over 55 percent against a 25 percent baseline, and its own few-shot examples overrode its own stated balance rule.
https://arxiv.org/abs/2601.02158
he2026 -- He, Li, Wang, Xiong & Li (2026). Judging the Judges: Human Validation of Multi-LLM Evaluation for High-Quality K-12 Science Instructional Materials. arXiv:2602.13243.
The one K-12 study with expert adjudication found a Claude judge unusually STRICT, not lenient. Cited alongside the self-preference literature because they point opposite ways.
https://arxiv.org/html/2602.13243
moore2024 -- Moore, Costello, Nguyen & Stamper (2024). An Automatic Question Usability Evaluation Toolkit (SAQUET). arXiv:2405.20529.
The 19-flaw taxonomy as running open-source code, with per-flaw detection tiers. The model our validator should follow.
https://arxiv.org/abs/2405.20529
overgenerate2024 -- Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank. arXiv:2405.05144.
Established for distractor generation, mostly in mathematics. Cost implications for us are real.
https://arxiv.org/pdf/2405.05144
schmucker2025 -- Schmucker & Moore (2025). The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory. arXiv:2503.10533.
Connects specific item-writing flaws to measured psychometric damage.
https://arxiv.org/html/2503.10533v1
schoch2025 -- Schoch & Ji (2025). In-Context Learning (and Unlearning) of Length Biases. arXiv:2502.06653.
Length bias is learnable in context from ~3 tokens of average difference and overrides finetuning. Onset around k=8, in classification, on 2.7B-8B models -- we ship k=1 in generation on frontier models, so this supports the mechanism and not our specific case.
https://arxiv.org/html/2502.06653v1
semanticAnchors2025 -- Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels. arXiv:2511.21038.
Demonstration-override capacity scales with model size. Cited as the counter-consideration: our three models are near-tied despite differing in size, which a purely exemplar-driven account does not predict.
https://arxiv.org/html/2511.21038
tang2026 -- Tang, Tuncel, Koerner & Runkler (2026). The Few-shot Dilemma: Over-prompting Large Language Models. arXiv:2509.13196.
More examples are not monotonically better. Adjacent rather than on point, and flagged as such.
https://arxiv.org/pdf/2509.13196
wataoka2025 -- Wataoka, Takahashi & Ri (2025). Self-Preference Bias in LLM-as-a-Judge. arXiv:2410.21819.
The general case for the same-family bias concern that prompted our cross-family rescore.
https://arxiv.org/html/2410.21819v2
zhao2025 -- Zhao, Hu, Jian et al. (2025). Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements. arXiv:2502.12459.
Option length as a failure axis for models ANSWERING items, which is why a model judge is a poor detector of a length cue.
https://arxiv.org/html/2502.12459v3

vendor 3

First-party model documentation. Authoritative about intent, and we have measured it wrong about behaviour more than once.

anthropicPrompting -- Anthropic. Claude prompting best practices.
Recommends 3 to 5 examples, and that they be diverse enough that the model does not pick up unintended patterns.
https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
awsNova -- AWS. Amazon Nova 2 user guide: prompting, advanced techniques, and few-shot guidance.
Prescribes capitalised DO / DO NOT / MUST bullets, lists legal length units as words, bullet points or paragraphs, and warns that bias in examples causes bias in outputs.
https://docs.aws.amazon.com/nova/latest/nova2-userguide/advanced-prompting-techniques.html
bedrockStructured -- AWS. Amazon Bedrock structured outputs: supported JSON Schema subset.
States that minLength and maxLength are unsupported and that unsupported features return a 400. We measured a 200 with silent non-enforcement instead.
https://docs.aws.amazon.com/bedrock/latest/userguide/structured-outputs.html

Item-writing rules and where each one is enforced

The routing column is the load-bearing part. Most of these rules do not belong in prompt text, and the headline defect is the clearest case: rule-based detection of the longest-option flaw beats a frontier judge by a wide margin, and a judge-based check inherits the very length bias it is meant to detect.

RuleSourceRoute toNote
Keep the length of choices about equalHaladyna g24validatorAlready in our prompt. Measured to fail.
Avoid giving clues to the right answerHaladyna g28validator + judgeSpecific determiners, clang associations, grammatical inconsistency, conspicuous correct choice, convergence cues.
Options as short as possible and parallel in lengthNAEP 2026prompt + validatorGives a direction, not just parity.
Options parallel in structure, syntax, complexityNAEP 2026prompt + judge
Distractors plausible, but not so plausible as to be possible correct answersNAEP 2026judgeThe guardrail against over-correcting distractor quality.
Keep choices homogeneous in content and grammatical structureHaladyna g23prompt
Include the central idea in the stem, not the choicesHaladyna g15promptThe only unanimous rule in the taxonomy, at 100 percent consensus.
Vary the location of the correct answerHaladyna g20codeShuffle at render time.
Convergence cues: three distractors alike with the key as the odd one outHaladyna g28evalidator (phase two)The residual leak has a name, and it is partly code-detectable.

Where we sit against the published field

cited

Our defect rates are typical to good.

A 71-study systematic review of AI-generated multiple-choice items in medical education reports error rates from under 1 percent to 45 percent. One anchor study found 49 percent of generated items carried item-writing flaws and 22 percent carried factual errors. Our 1 key error in 35 judged quizzes, zero for both candidates, sits at the good end.

cited

The length cue itself is documented once for model-authored items.

A preprint measured correct answers as uniquely longest in over 55 percent of questions against a 25 percent baseline. Our 48/54/46 is squarely in family.

judge-dependent

The exploitability half appears to be novel.

That work infers exploitability from item structure and explicitly says it cannot separate length exploitation from genuine reasoning. No source found combines a length-cue measurement with a withheld-source solve demonstrating the cue actually carries a solver to the key. Our 17 of 17 on judge-declared-unanswerable items looks publishable, with the caveat that the solver is a frontier model rather than a child.

cited

Our two-stage protocol is the field's published standard.

Säuberli and Clematide use the identical design and define text informativity as answerability minus guessability, descending from Kaushik and Lipton's question-only and passage-only ablations.

cited

The field's verdict on model-authored items is 'drafting tool, not author'.

71 studies across 24 countries, concluding that current evidence does not support unsupervised use in summative assessment.

cited

Item-writing consensus is strong and the direct evidence for it is thin.

Haladyna guidelines 24 and 28 carry the highest textbook consensus in the taxonomy, 85 and 96 percent with zero dissent, and the thinnest direct empirical validation. 'Keep options equal in length' entered the canon as expert consensus, not as a measured effect. The measured evidence comes from the test-wiseness and item-flaw literatures instead. Report it that way to anyone who asks.

Source documents

Rendered inline from the harness docs at build time, so this page cannot drift from them. Every claim in the playbook traces to one of the five briefs.

Playbook: quiz-generation prompting 21K

Last updated: 2026-08-17

Quiz-generation prompting playbook: comprehension items on claude-haiku-4-5 and nova-2-lite

Companion to the this study findings. Where claude-haiku-4-5.md and nova-2-lite.md cover these models for JSON reliability, this covers assessment quality -- writing a comprehension item that does not leak its own answer -- and the enforcement mechanics that make a quality rule stick.

Every claim is tagged [CITED] (primary source, URL in Sources), [MEASURED] (live probe or corpus computation, artifacts named), or [QUEUED] (untested hypothesis with the experiment that settles it).

Read this first: what this pull changes

  1. maxLength is a trap on both models. It is accepted, returns HTTP 200, and is enforced not at all -- on Haiku and on Nova. Both vendors' docs state that unsupported schema features return a 400. They do not. [MEASURED] Anyone porting the 50-character hint cap to a schema constraint believes they hold a guarantee and holds nothing, with no error to say so.
  2. pattern is decoder-enforced on Haiku and inert on Nova. It appears in neither vendor's supported nor unsupported list. On Haiku it beats a contradicting prompt and takes hint violations 8/16 to 0/16. On Nova it is dead weight. [MEASURED] This is a structural differentiator between the two candidate models that no quality sweep would ever have surfaced.
  3. Defect 2 is not fixable in the prompt. The length-balance rule is already there verbatim and fails at 48/54/46 percent. The strongest possible restatement of it scored key-is-longest 4 of 4. [MEASURED] Prompt text is the wrong instrument for this rule and we now have direct evidence of its failure.
  4. The prompt's own worked example demonstrates the defect it forbids, and our detector does not flag it. [MEASURED] The only other team to publish a measurement of this artifact independently reached the same causal conclusion about their own exemplars. [CITED]
  5. The fact-heavy skill mix is not model behavior. It is what the prompt asks for. getQuestionDistribution() never names main idea at any grade band and names vocabulary only at grades 3-5, as one arm of a disjunction. [MEASURED]
  6. The "DO NOT" ban inherited from that defect does not transfer to either candidate, and the two models want opposite phrasing. Forking the prompt profile is now mandatory, not stylistic.
  7. Length is roughly half the leak. With the length and content-density cues removed, the judge still hit the key on 76 percent of judge-declared-unanswerable items against a 25 percent chance line. [MEASURED] Do not promise more than half.

Idiom conflicts: the two candidates need forked profiles

Do not maintain one blended template. These differ in kind, not in degree.

Dimensionclaude-haiku-4-5nova-2-lite
pattern in schemaDecoder-enforced. Beat an emphatic contradicting prompt. Hint violations 8/16 -> 0/16; option-length spread 18.0 -> 7.2 chars, -> 2.2 with a prompt nudge. [MEASURED]Accepted, never evaluated. Against ^[b]{1,3}$ it returned a 176-char prose summary. HTTP 200, no error, no latency cost. [MEASURED]
maxLengthAccepted, ignored. 8/16 hints over cap with maxLength: 50 set. [MEASURED]Accepted, ignored. Hints of 475/533/612 chars against maxLength: 50. [MEASURED]
enumSupported.Hard-bound. Told adversarially to emit an out-of-set value, it could not. [MEASURED]
Bedrock structured outputsAvailable.Not supported at all. strict: true and outputConfig both ValidationException; model card lists it under Not Supported. [MEASURED + CITED]
Negative constraintsSafe, sometimes highly effective, sometimes simply ignored -- a nudge, never a contract. It violated a twice-stated DO NOT in the same response where a different DO NOT scored 12/12. [MEASURED]AWS actively prescribes capitalized DO / DO NOT / MUST. A seven-bullet negative prompt produced a clean quiz with every hint under 50 chars. [CITED + MEASURED]
Extended thinkingAvailable, but incompatible with our pinned temperature (temperature may only be 1 when thinking is enabled) and ~5x output tokens for no gain here. [MEASURED]Out of scope; not the lever.
Prompt cachingWill not engage. 4,096-token minimum per checkpoint; our request measures 751-806 input tokens. Do not model cache savings. [MEASURED + CITED]Engages: 2,392 of 2,572 input tokens served from cache (93%). [MEASURED]

The asymmetry that matters: Haiku's format compliance is buyable and Nova's is not. Haiku's 5/12 hint violations are a starting point a pattern takes to zero by construction. Nova's 1/12 is a ceiling, because the prompt is its only enforcement channel.

1. Where each defect actually gets fixed

Defect 1, the 50-character hint cap. Three independent fixes, all cheap, in the order I would apply them:

  1. Ask in words, gate in characters. Every hint of 7 words or fewer in the 428-hint corpus was already under 50 characters -- 247 of 247, 100 percent. [MEASURED] At 8 words compliance drops to 95.5 percent. Character counts are a unit models reason about badly: CharBench measures frontier models at 43.6 percent on first-occurrence character indexing against >70 percent on token-level analogues [CITED]. Word budgets are what instruction-following benchmarks actually verify [CITED], and AWS's own Nova guidance lists the legal units as "words, bullet points, or paragraphs" -- characters are not among them [CITED].
  2. Ban the lead-in stem. 45.8 percent of hints open with "Think about… / Consider… / Recall… / Focus on…", costing a mean 9.2 characters. Deleting that stem alone clears 65.6 percent of all length violations with no loss of content. [MEASURED]
  3. Shorten the JSON placeholder. The placeholder string that describes the constraint is itself 67 characters ("string — a short clue (<50 chars) pointing to relevant story detail") while the three real exemplar hints are 37, 38 and 37. The worst observed overage in the sweep was 71. A model anchoring on the placeholder's own length rather than its stated numeral is a live hypothesis with a one-line test. [MEASURED, QUEUED]

On Haiku, add pattern on top and the defect is gone by construction. On Nova, there is no schema-level mechanism of any kind that can bound a free-text field's length -- enum is the only compiled string constraint, and enumerating English is not available. Length must be enforced in application code.

Defect 2, the option-length key leak. Not fixable in the prompt; see item 3 above. The available levers, ranked:

  1. Decoder constraint (Haiku only). Band all four option text fields with one pattern. Spread 18.0 -> 7.2 chars, -> 2.2 with a prompt nudge. [MEASURED]
  2. Code-side validator with a repair loop (both models). The only option on Nova and the required safety net on Haiku, since pattern is undocumented on both and could regress without notice.
  3. Fix the exemplar (both models). Cheap, and a mandatory confound removal -- no other prompt A/B is interpretable while the demonstration contradicts the instruction.
  4. Lengthen distractors rather than trim the key. The only measured intervention in the published literature: >55 percent to 43.2 percent, with correct-answer mean length unchanged [CITED]. Partial; residual bias persisted above baseline.

2. The exemplar problem

The one-shot example embedded in generateQuestionsPromptForFullStory.ts has a key of 56 chars / 10 words against distractors of 46, 46 and 49 chars -- strictly the longest option at 1.19x the distractor mean. Our own detector's gate is 1.25x, so the exemplar we ship every call evades our own check by six hundredths. [MEASURED]

The same prompt contains a natural experiment. Its hint exemplars are 37, 38 and 37 characters, all compliant with the 50-char cap they illustrate. Its option exemplar violates the length-balance rule it illustrates. The contract whose exemplar complies is nearly always honored; the contract whose exemplar violates is broken about half the time. Same model, same call, everything else held constant. [MEASURED]

What the literature supports. Min et al. (EMNLP 2022): demonstrations transmit the format of the sequence, not its semantics -- randomly replacing demonstration labels barely hurts performance, but removing the format is worse than no demonstrations at all [CITED]. Option-length balance is a format property. Schoch & Ji (2025): length bias specifically rides that channel, is learnable from as little as ~3 tokens of average difference, and holds regardless of how it contradicts what was seen during finetuning [CITED, preprint]. AWS documents the mechanism in its own words: "It is important to avoid any biases in your examples, as bias in the inputs can cause outputs to be biased as well" [CITED].

The honest caveat, which cuts against the hypothesis. Schoch & Ji observe onset around k=8 demonstrations, in a classification task, on 2.7B-8B models. We ship k=1, in generation, on frontier models. Their curve predicts a single exemplar sits below detectability. And our three models are near-tied at 48/54/46 despite differing substantially in size, while demonstration-override capacity is documented to scale with size -- a purely exemplar-driven mechanism predicts more spread than we observe.

So: fix the exemplar because it is free and because it removes a confound, not because we have established it as the cause. The A/B is the single highest-value cheap experiment available.

3. Item-writing rules worth carrying, and where each one goes

From the canonical taxonomies. The routing column is the load-bearing part -- most of these do not belong in prompt text.

RuleSourceRoute to
Keep the length of choices about equalHaladyna g24 [CITED]Validator. Already in our prompt; measured to fail.
Avoid giving clues to the right answer (specific determiners, clang associations, grammatical inconsistency, conspicuous correct choice, convergence cues)Haladyna g28 [CITED]Validator (g28a-c, e) + judge (g28d)
Options are as short as possible and parallel in lengthNAEP 2026 [CITED]Prompt (word budget) + validator
Options parallel in structure, syntax, complexityNAEP 2026 [CITED]Prompt + judge
Distractors plausible, but not so plausible as to be possible correct answersNAEP 2026 [CITED]Judge
Keep choices homogeneous in content and grammatical structureHaladyna g23 [CITED]Prompt
Include the central idea in the stem, not the choicesHaladyna g15 (100% consensus, the only unanimous rule) [CITED]Prompt
Vary the location of the correct answerHaladyna g20 [CITED]Code (shuffle)

Two adoptions worth taking verbatim from NAEP. "As short as possible" gives a direction, not just parity -- it forecloses the failure mode where a model equalizes lengths by inflating the distractors to match a bloated key, which is precisely what the one published mitigation did. And "not so plausible as to be possible correct answers" is the guardrail against over-correcting distractor quality, the predictable side effect of pushing a model hard on plausibility.

One rule in our prompt is half-right and probably contributing to the leak. "Wrong answers must use words or phrases from the story" correctly implements the similarity strategy for distractor development [CITED], but it constrains only the distractors. Nothing constrains the key. So the model writes distractors from recycled story vocabulary and writes the key as an original, fuller restatement of what the story meant -- making it the most specific option by construction. Make the rule symmetric over the whole option set.

Honest caveat on how hard to lean on this. Guidelines 24 and 28 carry the highest textbook consensus in the taxonomy (85 and 96 percent, zero dissent) and the thinnest direct empirical validation. "Keep options equal in length" entered the canon as expert consensus, not as a measured effect. The measured evidence comes from the test-wiseness and item-flaw literatures instead. Report it that way to anyone who asks.

4. Blueprint: the skill mix is a prompt bug

Of 140 items the unprompted mix was 61 fact-detail, 43 inference, 7 main-idea, 3 vocabulary. That is the deterministic consequence of getQuestionDistribution(), which never names main idea at any grade band and names vocabulary only at grades 3-5, and then only as one arm of a disjunction ("explain why something happened or what a word/phrase means"). [MEASURED]

Authoritative targets exist. NAEP reserves 15-20 percent of items in every block for vocabulary and language structure [CITED]. PIRLS grade 4 runs 20 / 30 / 30 / 20 across its four comprehension processes [CITED].

Enforcement route: a required per-item skill enum. On Nova this is the one construct proven to bind hard. But note the trap -- an enum binds the label, not the thing it describes. A model can emit "vocabulary" next to a fact-detail item, because nothing ties them. The declared-versus-judged confusion matrix is the number that decides whether a cross-check is necessary. [QUEUED]

5. The validator

Do not ask the judge to catch this. Moore et al. compared a rule-based checker against GPT-4 on exactly the "longest option correct" flaw across four datasets. Rule-based F1: 0.80 / 1.00 / 1.00 / 0.95. GPT-4 F1: 0.18 / 0.00 / 0.44 / 0.36 [CITED]. An LLM judge is close to blind to our headline defect and a five-line length comparison is near-perfect. A second, independent argument: prompting that includes option text can itself bias solvers toward longer options, so a judge-based check inherits the very bias it is meant to detect [CITED].

Our current statistic is the wrong one. keyCueSignals's binary "strictly longest AND >= 1.25x mean" discards most of the signal and is one-sided. Replace the headline with the mid-rank of the key's length among the four options -- 2.50 means balanced, 1.00 means the key is always longest. Pooled value 1.80, monotone in grade: g1 2.64, g4 1.70, g7 1.17. [MEASURED] The rank statistic has an exact null (uniform on 1..4, mean 2.50, variance 1.25), so it yields a significance test at zero model spend.

Threshold: move to 1.15x with an absolute floor of +8 characters, calibrated so the shipped exemplar is flagged. Cost to historical comparability is small -- 48/54/46 becomes 45/58/46. [MEASURED]

Ask in words, gate in characters. The word-spread rule is the right instruction unit and the wrong gate unit: the shipped exemplar has a word spread of exactly 2 and passes it, while being strictly the longest by characters. Inside the <=2-word bucket the key is still longest 13 percent of the time (n=61) against 77 percent outside it -- better, not solved.

Phase two: convergence cues. The residual leak has a name. Haladyna g28e is "pairs or triplets of options that clue the test-taker" -- three distractors mutually alike on some dimension with the key as the odd one out. Partly code-detectable, and the right next validator target.

6. What this means for the cutover

Stated as mechanism, not as a shipping decision.

The sweep put nova ahead on format reliability (1/12 hint violations against haiku's 5/12) and behind on skill taggability (88 vs 94 percent). This playbook adds a different kind of fact: haiku has a hard, decoder-level enforcement mechanism for both defects and nova structurally lacks one. A model that fails less often is worth less than a model that cannot fail, particularly when the defect in question is already shipping to students.

Two qualifications, both real. pattern is undocumented as supported or unsupported by either vendor, so haiku's enforcement could regress without notice -- anything built on it needs a startup assertion and a code-side fallback. And the nova pattern probes are 4 replicates on 1 story at 1 grade: enough to establish inertness, which needs only one clean adversarial demonstration, not enough to be an effect size.

Incidental finding that outranks both graded defects if it reproduces. In the tool-shaped rewrite, nova returned 4, 4, 2 and 1 questions across four replicates against a prompt demanding exactly 4. [MEASURED] Note the contrast: on the production JSON path the sweep recorded zero question-count failures for nova across 12 quizzes, so this is most likely an artifact of the tool rewrite rather than a production risk. But no schema keyword can enforce a count either -- minItems accepts only 0 and 1, even on the Bedrock path nova cannot use -- so if it does reproduce on the production path it is a bigger problem than either defect. [QUEUED]

7. Where we sit against the field

Our defect rates are typical-to-good. A 71-study systematic review of AI-generated MCQs in medical education reports error rates from <1 percent to 45 percent [CITED]; one anchor study found 49 percent of ChatGPT-generated items carried item-writing flaws and 22 percent carried factual errors. Our 1 key error in 35 judged quizzes, zero for both candidates, sits at the good end.

The length cue itself is documented exactly once for model-authored items, in a preprint: correct answers uniquely longest in >55 percent of questions against a 25 percent baseline [CITED]. Our 48/54/46 is squarely in family.

What appears to be novel is the exploitability half. That work infers exploitability from item structure and explicitly says it cannot separate length exploitation from genuine reasoning. No source found combines a length-cue measurement with a withheld-source solve demonstrating the cue actually carries a solver to the key. Our 17-of-17 on judge-declared-unanswerable items looks publishable.

Our two-stage guess/answerability protocol is the field's published standard -- Säuberli & Clematide use the identical design and define text informativity as answerability minus guessability [CITED], descending from Kaushik & Lipton's question-only and passage-only ablations [CITED].

The field's verdict on LLM item generation is "drafting tool, not author": 71 studies, 24 countries, concluding that current evidence does not support unsupervised use in summative assessment [CITED].

Open questions, ranked by value per dollar

  1. Re-score the existing 36-quiz corpus under all five published operational definitions of the cue, in both characters and words. No model calls; the data exists. Until this runs, every rate in this program is measured with an instrument matching no published definition -- including the 48/54/46 near-tie that is our whole basis for calling this the prompt's defect rather than a model's.
  2. The exemplar A/B. Rerun the matrix with one change: the exemplar rewritten so the key is mid-rank in length and matched in specificity. Vary length rank and specificity independently across arms, since the current exemplar's length cue is marginal while its specificity cue is not. First controlled measurement of exemplar contamination in item generation if it works.
  3. Split the explanation call. The prompt requires the key's explanation to quote the story -- pressure to make the key defensible may leak into the key's option text. Generate options first, explanations in a second call. The only hypothesis here that changes lambda control flow rather than prompt text, and the most durable fix if it lands.
  4. Does the option-length band hold at grade 7, where the cue is strongest at 83 percent, and does it cost item quality?
  5. Is pattern enforcement on Haiku stable, or an implementation detail that can regress? Needs a startup assertion regardless.
  6. Does the count failure reproduce on the production JSON path for nova?
  7. Shorten the 67-char JSON placeholder and rerun. One-line test.
  8. Cross-family judge run on the same 36 quizzes. The literature cannot settle same-family bias: the general LLM-judge literature predicts inflation, while the one K-12 expert-adjudicated study found a Claude judge unusually strict. Both are cited. This is the this study neutral rescore already queued.

Sources

Full source lists with URLs and 2026-08-17 retrieval dates are in the five research briefs this playbook synthesizes. Primary anchors:

  • Haladyna, Downing & Rodriguez (2002), the canonical 31 item-writing guidelines.
  • Gierl, Bulut, Guo & Zhang (2017), Review of Educational Research 87(6), distractor-development synthesis.
  • NAEP 2026 Reading Assessment and Item Specifications -- grade 4/8 reading, four-option MCQ, the closest official standard to our use case.
  • Moore, Costello, Nguyen & Stamper (2024), SAQUET -- the 19-flaw taxonomy implemented as running code; and Moore, Nguyen & Stamper (2023) for the rule-based vs GPT-4 comparison.
  • Kıyak, Kaya & Emekli (2026), Postgraduate Medical Journal -- 71-study systematic review.
  • Säuberli & Clematide (READI @ LREC-COLING 2024); Kaushik & Lipton (EMNLP 2018) -- the answerability/guessability protocol.
  • Min et al. (EMNLP 2022) -- demonstrations transmit format, not semantics.
  • Schoch & Ji (2025, preprint) -- in-context length bias.
  • Ermilov (2026, preprint), FormationEval -- the one published measurement of the length cue on LLM-generated items.
  • Uzan & Pinter, CharBench (arXiv:2508.02591) -- character-level reasoning.
  • AWS Bedrock structured-outputs and Nova 2 user-guide pages; Anthropic prompting and structured-outputs pages.
  • Live Bedrock probe artifacts: nova-probes/pat1-4, pat5-quiz-rep1..4, probe1-11; Haiku probes P1-P9.
Brief 1 -- item-writing craft 76K

Brief 1 -- Item-Writing Craft: Assessment Science for Non-Leaking Multiple-Choice Items

Last updated: 2026-08-17

This study research team, model-agnostic assessment-science axis. Scope: what item-writing science knows about option sets that do not leak their key, converted into rules a prompt, a schema, or a validator can carry.

Every claim below is tagged [CITED], [MEASURED], or [QUEUED] per the team evidence standard.


Headline calls

  1. Adding a length rule to the prompt will not fix Defect 2, because the length rule is already in the prompt and all three models ignore it. [MEASURED] generateQuestionsPromptForFullStory.ts:62 already reads "All wrong answers must be similar in length and complexity to the correct answer," and the sibling paragraph prompt carries the same instruction at line 38. Against that instruction the measured key-is-longest rate is 48 / 54 / 46 percent. The rule is present, positively phrased, and unheeded. Route option-length control to a code-side validator with a hard numeric threshold. Prompt text is the wrong instrument for this rule and we have direct evidence of its failure.
  1. The prompt's single worked example is cued, but on specificity rather than length -- and only one of five operational definitions flags its length at all. [MEASURED] In the exemplar at generateQuestionsPromptForFullStory.ts:70-84 the key is 56 characters / 10 words against distractors of 46, 46, and 49 characters (8, 9, and 10 words). It is strictly the longest in characters only; in words it ties. It clears the published loose threshold (0.875), clears symmetric homogeneity (0.821), and clears the orchestrator's 1.25x-mean detector (1.191). Only the strict published definition flags it. But it is unambiguously the only causal, interpretive option against three concrete-action distractors -- and no length rule measures that. Fix the exemplar, but fix its specificity shape, not just its character counts. Section 8 works this through.
  1. Anthropic's own documentation names our exact failure mode as the reason to use more than one example. [CITED] Its prompting guidance says examples must be "Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns," and recommends 3-5 examples. We ship one. A length-*balance* constraint is a claim about within-item variance, and a single exemplar cannot demonstrate variance -- it can only demonstrate one shape, which the model is then free to read as the target shape.
  1. The fact-heavy, vocabulary-starved skill mix is not a model behavior. It is exactly what the prompt asks for. [MEASURED] Reading getQuestionDistribution() at generateQuestionsPromptForFullStory.ts:21-38: main idea is never named at any grade band; vocabulary appears only at grades 3-5 and only as one arm of a disjunction ("explain why something happened or what a word/phrase means"); grades 6+ names neither. The measured unprompted mix (7 main-idea and 3 vocabulary out of 140) is the deterministic consequence of that text. Blueprint enforcement is a prompt-plus-schema problem, and the authoritative targets exist: NAEP reserves 15-20 percent of items in every block for vocabulary/language-structure [CITED], and PIRLS grade 4 runs 20 / 30 / 30 / 20 across its four processes [CITED].
  1. The instruction "wrong answers must use words or phrases from the story" is half-right and is probably contributing to the leak. It correctly implements the similarity strategy for distractor development [CITED: Gierl et al. 2017], but it is applied to the distractors only. Nothing constrains the key. So the model writes distractors from recycled story vocabulary and writes the key as an original, fuller restatement of what the story actually meant -- which makes the key the most specific and most detailed option by construction. **The rule must be made symmetric: it is the option set, key included, that has to be homogeneous in content, structure, and length.**
  1. Do not reduce to three options, even though the meta-analytic literature favors it. [CITED] Rodriguez's 80-year meta-analysis concludes three options are optimal, and Haladyna et al. concur. But NAEP single-selection multiple choice uses four [CITED], Alabama's grade 3-8 ELA program uses four [CITED], and our production schema, UI, and per-answer explanation contract are all built on four. The measured gain is in items-per-testing-minute, which is irrelevant to a fixed 4-item quiz. Keep four; spend the effort on making the third distractor functional instead.
  1. The blind-grader result (17 of 17 on cued items with the story withheld) is not an anomaly to explain away -- it is the standard partial-input validity check, and its failure is disqualifying under an explicit federal item-writing standard. [CITED] NAEP 2026 specifications: "items should not be answerable by readers only drawing upon text-independent domain knowledge, without even reading the passage." Make the partial-input check the acceptance gate for the cutover, not a diagnostic.
  1. Our 48 / 54 / 46 percent is a floor, not a measurement, because the 1.25x-mean threshold is looser than either published definition and matches neither. [MEASURED] Scored against our own exemplar, the 1.25x-mean rule is the most permissive of the five operationalizations I tested. It is also not a threshold anyone in the literature uses, so our headline number is not comparable to any published figure. Under the strict published definition the true rate is necessarily higher than 48 / 54 / 46. Re-score the existing corpus under the published definitions before quoting the number anywhere external. Section 8.3.

Single highest-confidence recommendation: implement a code-side option-set validator using the two published operationalizations -- Moore et al.'s loose rule (key is longest and second-longest is under 80 percent of longest) as the cue gate, plus symmetric homogeneity (shortest / longest ≥ 0.80) as the NAEP parallel-length check -- measured in characters, run as a generate-then-repair loop. Pair it with 3-5 exemplars whose keys occupy different length ranks. That combination attacks the measured defect with a deterministic instrument, is model-agnostic, and survives the gpt-3.5 cutover regardless of which candidate wins.


1. The guideline taxonomies

1.1 Haladyna, Downing & Rodriguez (2002) -- the canonical 31 [CITED]

The field's reference taxonomy. Validated against 27 educational-measurement textbooks and 27 post-1990 research studies. Organized as Content (1-8), Formatting (9-10), Style (11-13), Writing the Stem (14-17), Writing the Choices (18-31).

The fourteen option-set rules, verbatim:

#GuidelineTextbook: for / uncited / against
18Develop as many effective choices as you can, but research suggests three is adequate70 / 26 / 4
19Make sure that only one of these choices is the right answer70 / 30 / 0
20Vary the location of the right answer according to the number of choices52 / 48 / 0
21Place choices in logical or numerical order67 / 33 / 0
22Keep choices independent; choices should not be overlapping30 / 70 / 0
23Keep choices homogeneous in content and grammatical structure67 / 33 / 0
24Keep the length of choices about equal85 / 15 / 0
25None-of-the-above should be used carefully44 / 7 / 48
26Avoid All-of-the-above70 / 7 / 22
27Phrase choices positively; avoid negatives such as NOT70 / 30 / 0
28Avoid giving clues to the right answer (a. specific determiners; b. clang associations; c. grammatical inconsistencies; d. conspicuous correct choice; e. pairs or triplets that clue; f. blatantly absurd options)96 / 4 / 0
29Make all distractors plausible96 / 4 / 0
30Use typical errors of students to write your distractors70 / 30 / 0
31Use humor if compatible with the teacher and environment0 / 85 / 15

Honest caveat that matters for how hard we lean on this. Guidelines 24 and 28 -- the two rules that bear most directly on our defect -- carry the highest textbook consensus (85 and 96 percent, zero dissent) and the thinnest direct empirical validation in that review. The authors report that post-1990 research clustered on only four guidelines (17 negative stems, 18 number of options, 25 NOTA, 26 AOTA), and they write that plausibility of distractors (29) "continues to be an area that is long overdue for study." So "keep options equal in length" entered the canon as expert consensus, not as a measured effect. The measured evidence for it comes from the test-wiseness and item-flaw literatures instead -- section 2 below. Report it that way to anyone who asks.

Guidelines outside the option set that bind on us: 8 keep vocabulary simple for the group being tested (70 / 30 / 0; supported by Abedi et al. 2000 on simplified language reducing construct-irrelevant reading load); 13 minimize the amount of reading in each item (67 / 33 / 0); 15 include the central idea in the stem instead of the choices (100 / 0 / 0 -- the only unanimously cited guideline in the taxonomy); 7 avoid trick items.

1.2 Gierl, Bulut, Guo & Zhang (2017) -- the distractor-specific synthesis [CITED]

Review of Educational Research 87(6):1082-1116. Cross-tabulates six published guideline sets. Rules with full six-of-six consensus: use plausible distractors, place distractors in logical or numerical order, keep distractors independent / non-overlapping, NOTA and AOTA used carefully, avoid giving clues to the right answer. Slightly lower consensus: incorporate common errors of students, keep distractors homogeneous in content and grammatical structure, phrase distractors positively.

The single most load-bearing sentence in this paper for us, on the similarity strategy: "Structural similarities include distractors that share characteristics with the correct option such as length, complexity, formatting, and grammar." Homogeneity is not a stylistic nicety layered on top of distractor development -- in the canonical account it is one of the two distractor-development strategies.

1.3 NAEP 2026 Reading Assessment and Item Specifications -- an official K-8 standard [CITED]

This is the highest-authority source for us because it is grade 4 and grade 8 reading comprehension, four-option single-selection multiple choice, from the US federal assessment governing board. Its selected-response development guidelines, verbatim:

  • The stem includes only the information needed to make the student's task clear.
  • Options are as short as possible and are parallel in length.
  • Options are parallel in structure, syntax, and complexity.
  • Options do not contain inadvertent cues to the correct answer, such as repeating a word from the stem in the correct answer or using specific determiners (e.g., all, never) in the distractors.
  • Distractors are plausible, but not so plausible as to be possible correct answers.
  • Distractors are designed to reflect the measurement intent of the item, not to trick students into choices that are not central to the idea being assessed.

Two of these are new relative to Haladyna and worth adopting verbatim. "As short as possible" gives a direction, not just a parity constraint -- it forecloses the failure mode where a model equalizes lengths by inflating the distractors to match a bloated key. And "not so plausible as to be possible correct answers" is the guardrail on over-correcting plausibility, which is the predictable side effect of pushing a model hard on distractor quality.

1.4 The operational rubric: SAQUET's 19 item-writing flaws [CITED]

Moore, Costello, Nguyen & Stamper (2024). This matters because it is the taxonomy translated into running code, open-sourced, with per-flaw accuracy measured against human raters -- exactly the artifact a validator should be modeled on.

The 19 criteria: Longest Option Correct, Ambiguous Information, Implausible Distractors, True or False, Absolute Terms, Complex/K-type, Negatively Worded, Convergence Cues, Lost Sequence, Unfocused Stem, None of the Above, Word Repeats, More Than One Correct, Logical Cues, All of the Above, Fill in the Blank, Vague Terms, Grammatical Cues, Gratuitous Information.

Detection tiers, which map one-to-one onto our routing decision:

  • Pure string/rule matching (8 criteria): None of the Above, All of the Above, Fill-in-the-Blank, True or False, Longest Option Correct, Negatively Worded, Lost Sequence, Vague Terms.
  • Classical NLP -- embeddings, NER, RoBERTa (5 criteria): Implausible Distractors, Word Repeats, Logical Cues, Ambiguous/Unclear, Grammatical Cues.
  • Requires an LLM call (6 criteria): Absolute Terms, More Than One Correct, Complex/K-type, Gratuitous Information, Unfocused Stem, Convergence Cues.

Overall 94.13 percent accuracy across 5,149 classifications on 271 MCQs; Hamming loss 5.9 percent; exact-match ratio 38 percent. Per-domain F1 for Longest Option Correct: 0.77 / 0.60 / 0.96 / 1.00 / 0.97. Word Repeats F1: 1.00 / 1.00 / 0.56 / -- / 0.53.

The threshold question, answered. The paper reports that human raters used two different operational definitions of Longest Option Correct and names both: a loose rule flagging an item when the second-longest option is not at least 80 percent as long as the longest, and a strict rule flagging "any question where the correct answer exceeded others in length by even a single character." We should implement both and report both, because this study's 48/54/46 percent figure is presumably closer to the strict definition, and the loose definition is what we can realistically hold a generative model to.

Their acceptance policy, adoptable as-is: an MCQ with zero or one flaw is acceptable for formative use; two or more flaws is unacceptable. That gives us a defensible, externally sourced quiz-level gate rather than a number we invented.


2. Test-wiseness cues

2.1 The frame [CITED]

Millman, Bishop & Ebel (1965) defined test-wiseness as "the subject's capacity to utilize the characteristics and formats of the test and/or the test taking situation to receive a high score," explicitly independent of knowledge of the subject matter. Our Defect 2 is a textbook test-wiseness vulnerability, and the framing matters for how we talk about it: a length cue does not make a quiz wrong, it makes the quiz measure something other than reading.

2.2 The primary target: option length and specificity

Measured prevalence and payoff in real tests. [CITED] Brozo, Schmelzer & Spires (1984) analyzed 1,220 multiple-choice items from 43 teacher-made college examinations against the Millman taxonomy. 539 items (44 percent) contained at least one clue; 375 of those (70 percent) were answerable by applying the clue, against a 25 percent chance baseline (z = 24.10, p < .001). Per-clue:

ClueItemsCorrect via clueRateChancezp
Longest alternative5441.76.258.49<.001
Middle value7965.82.2511.62<.001
Key word association (clang)38381.00.2510.49<.001
Specific determiner5218.35.251.44n.s. (<.075)
Direct opposites151115.76.2514.42<.001
Most general option4536.81.258.36<.001
Grammatical agreement0----------
All of the above6937.54.255.34<.001
Position "C"1220318.26.251.02n.s.

Note their operationalization of "longest": at least one full line of print more than the other alternatives -- a much coarser bar than ours, which is why their 76 percent understates what a character-level definition would find.

Measured psychometric damage. [CITED] Schmucker & Moore (2025), 7,126 MCQs and 448,000 student responses from CK-12 across Life/Earth Sciences, Physical Sciences, and Mathematics, flaws labeled by SAQUET at 94 percent agreement with humans, outcomes modeled as IRT parameters. Longest Option Correct: discrimination γ = -0.370 (p < .05), difficulty β = -0.691 (p < .05). It is the flaw with the largest adverse coefficient on both parameters simultaneously. Items with a length cue are both easier and worse at separating strong readers from weak ones -- which is precisely the harm a comprehension quiz cannot absorb. For contrast, Convergence Cues (β = +0.679) and Grammatical Cues (β = +0.526) made items harder, not easier.

A conflicting result, reported as required. [CITED] Pham, Besanko & Devitt (2018) compared 98 matched pairs of flawed and unflawed MCQs and found no statistically significant difference in difficulty or discrimination indices between the groups overall (difficulty p-values .198-.8845; discrimination p-values .1359-.8705). Within that null, "longest choice is correct" did significantly raise mean item scores (p = .047) -- items got easier -- but the discrimination penalty Schmucker & Moore report did not replicate. The two studies differ by roughly 70x in sample size and by outcome model (classical indices vs. IRT parameters), which favors the larger study, but the honest statement is: the "easier" effect is replicated; the "less discriminating" effect rests on one large study. Do not present the discrimination claim as settled.

Why "most specific" and not "most general" for us. Brozo found the general option was correct 81 percent of the time; this study found the specific and detailed option is the key. These are not in conflict, they are two different authorship failure modes. A human writing about facts they half-know hedges the key into generality. A model writing about a story it has fully in context writes the key as the fullest accurate restatement of what happened, then invents thinner distractors. Same cue, opposite polarity. **Practical consequence: the classic "watch out for the most general option" heuristic does not transfer, and any rule we write must target asymmetry of specificity, not a direction.** Haladyna 23 (homogeneous content and structure) plus 24 (equal length) plus NAEP's "parallel in structure, syntax, and complexity" are the correct rules; "avoid the general option" is not.

2.3 Secondary cues, ranked by whether they can bite us

  • Clang / key-word association. Highest-yield cue in Brozo (100 percent, 38/38). NAEP forbids "repeating a word from the stem in the correct answer." Relevant to us and cheap to check, because our stems quote the story and our keys restate it. Note carefully: SAQUET's Word Repeats is defined as words repeated only between the stem and the correct response -- a set-difference computation, not a raw overlap count. That definition is the right one to implement; a naive stem-key overlap counter will fire on every fact-detail item.
  • Grammatical agreement with the stem. Zero occurrences in 1,220 human-written items; SAQUET routes it to NLP. Low priority. Our items are near-perfect on format contract already and LLMs are fluent; this is unlikely to be our failure mode. Worth one measurement, not a prompt line.
  • Absolute qualifiers / specific determiners. Brozo's only non-significant knowledge cue (35 percent, p = .075). NAEP nevertheless bans them in distractors. Include as a validator rule (cheap word list) but do not spend prompt budget on it.
  • Convergence cues. Options built from different combinations of shared components, so the most-frequently-appearing components identify the key. Structurally near-impossible in narrative comprehension options and it raised difficulty in Schmucker & Moore. Deprioritize.
  • None of the above / all of the above. Textbook opinion is genuinely split on NOTA (44 percent for, 48 percent against) and all five post-1990 studies agree NOTA raises difficulty. AOTA is more clearly condemned -- Harasym et al. found reliability significantly reduced and attributed near-universal AOTA selection to cuing. For K-8 narrative comprehension both are simply out of scope; forbid at schema level (a regex validator) and spend no prompt text on them.
  • Direct opposites. Brozo's single most frequent cue (151 items, 76 percent). Fires when a writer generates a distractor by negating the key. Plausible for LLM distractor generation and currently unmeasured for us. Queued below.
  • Position bias. Brozo found position "C" non-significant in human tests (26 percent vs 25 percent chance). But this is different for machine authors: LLMs are documented to have position biases in generation, and the classical remedy (Haladyna 20, Mosier & Price 1945) is randomization. This is trivially and completely solved by shuffling options at serving time, in code, with the key index stored. It should never be a prompt concern.

2.4 The measurement method is itself citable [CITED]

The blind-grader design this study already ran is the standard partial-input baseline from NLP evaluation. Kaushik & Lipton (2018) established passage-only and question-only baselines across bAbI, SQuAD, CBT, CNN and Who-did-What and found them "surprisingly" strong, concluding that benchmarks solvable from partial input do not measure what they claim. Balepur, Ravichander & Rudinger (2024) ran the choices-only variant on three MCQA datasets and four LLMs: the choices-only prompt beat a majority baseline in 11 of 12 cases, with gains up to 0.33 accuracy. Their causal analysis rules out memorization alone and implicates the group dynamics of the option set -- which is the same mechanism as a length cue.

This gives our acceptance gate an external methodological pedigree. The metric to institutionalize is the choices-only solve rate: give a judge model the four options and nothing else, and require the solve rate to be statistically indistinguishable from 25 percent. That is a single number, model-agnostic, defensible to any psychometrician, and it subsumes length, specificity, clang, opposites, and convergence in one measurement instead of a checklist.


3. Distractor development

3.1 The two sanctioned sources [CITED]

Gierl et al. identify exactly two strategies with sustained support:

Strategy 1 -- common errors and misconceptions. Haladyna & Rodriguez, quoted in full by Gierl as the current best statement: "The most effective way to develop plausible distractors is to either obtain or know what typical learners will be thinking when the stem of the item is presented to them. We refer to this concept as a common error. Knowing common errors can come from a good understanding of teaching and learning for a specific grade level; it can come from think aloud studies with students; or it can come from student responses to a constructed-response format version of the item without options."

Our prompt already does this: "Wrong answers should reflect mistakes a grade N student would genuinely make" (line 61). This line is correct, well-founded, and should be kept and strengthened, because it is the only line in the prompt that gives the model a generative principle for distractors rather than a constraint. Alabama's grade 3-8 ELA specification is the K-8 instantiation: ELA distractors "represent common misconceptions, incorrect logic, incorrect understanding of a text." [CITED] Smarter Balanced is more specific still: reading distractors should be plausible to students who (1) misinterpret details in the text, (2) draw erroneous conclusions or inferences, or (3) apply faulty reasoning about the text. That three-way split is directly promptable and is a better replacement for our vague "wrong for a different reason" line (63).

Strategy 2 -- similarity to the key. Content similarity (same category, concept, topic) plus structural similarity: length, complexity, formatting, grammar.

The honest caveat on both. Gierl et al. are explicit that the misconception route requires three assumptions to hold -- that content specialists can specify plausible errors, that distractors can be produced from them, and that the specified misconceptions are the ones students actually hold -- and that this alignment "must occur for each distractor." They conclude that writing distractors that measure plausible misconceptions "and replicating this outcome consistently" is hard. This is a warning against believing a model when it claims a distractor encodes a grade-appropriate error. The claim is unfalsifiable without response data. Which leads to:

3.2 How many distractors actually work [CITED]

Haladyna & Downing (1993), across four standardized tests: roughly two thirds of items had one or two effective distractors; only 1 to 8 percent had three; the modal number of effective distractors per item was one. Items with more effective distractors discriminated better, though effectiveness was unrelated to difficulty. The conventional threshold for a nonfunctioning distractor is a selection rate below 5 percent.

Haladyna et al.'s conclusion is blunt: "The effort of developing that fourth option (the third plausible distractor) is probably not worth it. If the use of four options is preferred, empirical research has established that it is very unlikely that item writers can write three distractors that have item response patterns consistent with the idea of plausibility."

Rodriguez (2005), the 80-year meta-analysis, reaches the same place from the psychometric side: three options are optimal, mainly because more items fit per unit of testing time and because implausible distractors get eliminated rather than tolerated.

Our call, restated with the reasoning. We keep four, because the efficiency argument that drives the three-option finding does not exist for a fixed 4-item formative quiz, and because every K-8 reading program we can cite -- NAEP single-selection MC, Alabama ACAP, Nebraska NSCAS -- uses four. But we should stop expecting all three distractors to be functional. That is not a realistic target for any author, human or model. The right posture: require three distractors, hold all four options to structural homogeneity, and measure functionality post-hoc from real student response data rather than asserting it at generation time.

3.3 Verdict on our passage-vocabulary instruction

The instruction under review, generateQuestionsPromptForFullStory.ts:60: "Wrong answers must use words or phrases from the story in plausible but incorrect ways."

Verdict: keep the intent, fix the asymmetry, and expect it to be a contributor rather than the cause.

Supporting it: this is a legitimate expression of Gierl's similarity strategy, and for narrative comprehension the story is the content domain, so drawing distractor material from it is the only way to keep options in the same content category (Haladyna 23). Removing the instruction outright would risk distractors sourced from outside the story, which are trivially eliminable and would make the leak worse, not better.

Against it, three specific problems:

  1. It is asymmetric. The rule binds only the wrong answers. Nothing tells the model to build the key from the same material at the same grain. Under that instruction the natural output is exactly what was measured: distractors made of recycled story surface, key made of an original and fuller proposition. The specificity gap is manufactured by the prompt's own asymmetry.
  1. It pushes toward the wrong distractor archetype. "Use words from the story in incorrect ways" produces surface-recombination distractors -- true story words in a false arrangement. Gierl's strategy 1 and Smarter Balanced both want reasoning-error distractors -- what a student concludes when they misread. The former are eliminable by anyone who read the story; the latter require actually understanding it. This is likely a second-order contributor to weak distractor functionality.
  1. It interacts badly with skill tagging. PIRLS supplies an operational classification rule: [CITED] "If the item stem and the correct response both use exact words from the text and are located with a sentence or two of each other, the item is classified as Focus and Retrieve... if the item stem and correct response use paraphrases of the original phrases or sentences in text then the item is classified as Straightforward Inferencing." Lexical overlap between the passage and the key is the operational marker of the lowest-order comprehension process. Instructing heavy passage-vocabulary reuse in distractors, while the key paraphrases, systematically stamps every item with the surface signature of an inference item whose distractors look like retrieval items. That is precisely the shape a length-and-specificity cue takes.

Rewrite direction (prompt-expressible): make the constraint apply to the option set. Something in the shape of -- all four options draw on the same story material, at the same level of detail and the same grammatical form; a reader who has not read the story must not be able to tell which one the story supports. Pair it with the Smarter Balanced three-way distractor typology so the model has a generative principle, not just a parity constraint. NAEP's ceiling clause ("plausible, but not so plausible as to be possible correct answers") is the necessary counterweight, and its inclusion is what keeps this from degrading answer-key soundness -- which this study measured as currently excellent and which we must not regress.


4. Grade calibration for K-8

4.1 What the standards actually control [CITED]

  • Text complexity bands (CCSS Appendix A, updated multi-measure table): grades 2-3 Lexile 420-820 / Flesch-Kincaid 1.98-5.34; grades 4-5 Lexile 740-1010 / FK 4.51-7.73; grades 6-8 Lexile 925-1185 / FK 6.51-10.34.
  • Passage length by grade (Alabama ACAP ELA): grade 2 150-350 words; grade 3 550-650; grade 4 600-700; grade 5 650-800; grade 6 650-800; grade 7 750-850; grade 8 750-850.
  • Vocabulary inclusion criteria (NAEP 2026, Exhibit 3.13). Included: words appearing across many texts (despise, benevolent, resolution, commit); words needed to understand a local part of the context linked to central ideas; words labeling familiar concepts even if the word itself is unfamiliar to younger readers (timid); words with useful roots and affixes (disregard, counterargument); language expressing logical relations (although, in contrast); anaphoric expressions referring to previously introduced entities (those alliances, this phenomenon). Excluded: obscure or discipline-specific words (fiduciary, apotheosis) and idioms (spill the beans, up in the air).
  • Language load in items: Haladyna 8 and 13, supported by Abedi et al. (2000) on simplified item language reducing construct-irrelevant reading demand.

Important scope note: the Lexile and FK bands above govern the passage, not the item. Item language should sit at or below the passage band -- an option written above the passage's readability measures reading load, not comprehension. That is an inference from Haladyna 8 and 13, not a directly cited threshold; treat the specific numeric target as [QUEUED].

4.2 Why our length cue climbs with grade -- three candidate mechanisms

Our measured escalation is 29 percent at grade 1 to 83 percent at grade 7, and it persists with question type held constant to fact-detail. That last fact is the important one: it rules out the simplest explanation (that higher grades just get more inference items, and inference keys are naturally longer). Three mechanisms remain, in descending order of my confidence.

M1 -- Grade drives an implicit register shift in the prompt, and register drives length. The prompt's grade text is not only a type distribution, it is a tone instruction. Compare grade ≤2 ("2 questions about facts directly stated in the story") to grade 6+ ("1 question about a detail that matters to the plot (required for the next questions)... requiring evidence-based inference -- the reader must cite specific story details"). The grade-6 framing conditions the whole generation toward elaborated, evidence-citing prose. A model conditioned to write elaborately writes an elaborate key -- because the key is the option it must make defensibly true -- while distractors, being invented, stay short. Under this mechanism the cue would persist within fact-detail items, which matches the measurement. This is my leading hypothesis and it is cheap to test.

M2 -- Grade-band ceiling compression at the low end. At grade 1 every option is a short simple clause and there is little room for a length gap; the constraint is the option floor, not the model's restraint. As the allowed complexity rises, the gap has room to open. This predicts the magnitude of the gap grows with grade even where the rate would not, and it predicts grade-1 items are protected by accident rather than by design -- meaning any change that lengthens grade-1 options will surface the cue there too. Worth knowing before we "improve" grade 1.

M3 -- Higher cognitive demand correlates with more item-writing flaws, independent of grade. [CITED, indirect] Moore et al. report that in their 271-item corpus, questions carrying two or more item-writing flaws had higher Bloom's Revised Taxonomy levels in four of five domains. That is a measured association between cognitive demand and flaw density in human- and student-authored items, from an entirely separate corpus. It does not establish causation and it is not grade-stratified, but it is the only external evidence I found that the direction of our escalation is a general property of harder items rather than an artifact of our prompt.

None of these are settled. Experiments in Open Questions.

4.3 What grade calibration should actually control, as rules

  • Option word count ceilings scaled by band, not free-form "grade-appropriate." Absolute ceilings are guessable from passage-length ratios but are not directly cited; [QUEUED].
  • Option vocabulary drawn from the NAEP inclusion list and screened against the exclusion list -- promptable, and the exclusion half (no idioms, no obscure terms) is validator-checkable against a word list.
  • Readability of the option set held at or below the passage's band.
  • Critically: the length-parity threshold should NOT be relaxed at higher grades. The temptation will be to allow more variance where language is richer. The escalation data says the opposite -- the higher grades are where the enforcement has to be tightest.

5. Reading-comprehension skill taxonomies and the blueprint

5.1 Which framework to adopt

Two candidates, both authoritative, and they mostly agree.

NAEP 2026 Comprehension Targets (grades 4, 8, 12): Locate and Recall; Integrate and Interpret; Analyze and Evaluate; Use and Apply.

PIRLS 2021 Processes of Comprehension (grade 4): Focus on and Retrieve Explicitly Stated Information; Make Straightforward Inferences; Interpret and Integrate Ideas and Information; Evaluate and Critique Content and Textual Elements.

Mapping onto our five components:

Our componentNAEP 2026PIRLS 2021
fact-detailLocate and RecallFocus on and Retrieve
inferenceLocate and Recall (local) / Integrate and InterpretMake Straightforward Inferences
main ideaIntegrate and Interpret (inferred) or Locate and Recall (explicitly stated)Interpret and Integrate
evaluate / analyzeAnalyze and EvaluateEvaluate and Critique
vocabularydouble-coded, not a separate targetnot a separate process

Recommendation: adopt PIRLS as the primary spine and borrow NAEP's vocabulary treatment. Reasons: PIRLS is grade-4 literary-narrative reading, which is our exact genre and near our median grade; it splits inference into two levels, which our single "inference" bucket currently conflates and which is the split that most affects difficulty; it publishes flat per-process percentages rather than wide ranges, so it is directly enforceable; and it supplies an operational classification rule (section 3.3) that a validator can compute. NAEP's fourth target, Use and Apply, requires applying text to an external task and does not fit a 4-item quiz about a generated story -- drop it.

The vocabulary structural insight, from NAEP, is the most useful single idea in this section. NAEP does not treat vocabulary as a fifth question type. It double-codes: every vocabulary item is also a comprehension-target item, because "language structures and vocabulary will not be measured independently from Comprehension Targets." A total of 15-20 percent of items in any block carry the vocabulary code. This dissolves our blueprint arithmetic problem. With only four items, five mutually exclusive types cannot be distributed sensibly. As an orthogonal flag it works: four items across four processes, one of which additionally carries the vocabulary flag. That maps to 25 percent, slightly above NAEP's 15-20 percent band, which is the correct direction of error given our measured 2 percent.

5.2 Per-grade blueprint

Published targets, for reference:

NAEP G4NAEP G8PIRLS G4 (flat)
Locate and Recall / Retrieve15-40%10-25%20%
Straightforward inference-- (folded into L&R / I&I)--30%
Integrate and Interpret10-40%20-35%30%
Analyze and Evaluate10-25%20-35%20%
Use and Apply0-30%0-30%--
Vocabulary (double-coded)15-20%15-20%--

Both frameworks state the same grade principle: the proportion of higher-order targets rises across grades and Locate-and-Recall falls. NAEP: "Moving across grades, the proportion of higher level Comprehension Targets increases."

A defensible 4-item-per-quiz blueprint that respects both, one item per process:

  • Grades 1-2: 2 retrieve, 1 straightforward inference, 1 interpret/integrate (phrased as "tell what the story is mostly about" -- main idea at this band). Vocabulary flag on one item.
  • Grades 3-5: 1 retrieve, 1 straightforward inference, 1 interpret/integrate (main idea or character motivation), 1 evaluate/analyze. Vocabulary flag on one item.
  • Grades 6-8: 1 retrieve, 1 interpret/integrate, 1 evaluate/analyze, 1 further interpret/integrate or evaluate. Vocabulary flag on one item.

Note this is close to what our prompt already asks for at grades 3-5. The measured gap (61 of 140 fact-detail against a 25 percent target) is therefore not a blueprint-design failure. It is an enforcement failure, which is the whole argument of the next point.

5.3 How a blueprint gets enforced

NAEP is explicit that even for a professional item-development shop, distribution is monitored at the pool level rather than guaranteed per block, and that content quality outranks the distribution target: "for a given text, it is better to fall one item short in the number of items for a target than it is to include one item that fails the importance or the clarity standard just for the sake of meeting the distribution goal."

For us, three mechanisms in ascending order of strength:

  1. Prompt prose -- what we have now. Measured to under-deliver by a wide margin (44 percent fact-detail against a 25 percent instruction).
  2. Schema-level: a required per-item enum field. Add a mandatory skill field with a closed enum (retrieve | inference | interpret_integrate | evaluate_analyze) and a required boolean vocabulary_focus. This forces the model to commit to a label per item, which is a materially different cognitive task from following a distribution instruction, and it makes the distribution machine-checkable in one pass. It also directly serves the skill-taggability metric this study measured (94 / 88 / 75 percent) by moving tagging from post-hoc judgment to generation-time declaration.
  3. Validator: count the enums, reject and regenerate on violation. With a schema field this is a trivial check. Without one it requires a judge model.

Note the failure mode of level 2 alone: the model can satisfy the enum by labeling a fact-detail item inference. Declared-cast approaches have this exact weakness. This is where the PIRLS lexical-overlap rule earns its keep: compute the verbatim overlap between (stem + key) and the passage, and cross-check the declared label against it. High exact-phrase overlap with a declared inference label is a flag. That is a real, cheap, code-side falsifier for a self-declared tag, and I have not seen it proposed elsewhere in the harness.


6. Stem construction

The rules that bear on passage-based comprehension items, all [CITED]:

  • Haladyna 15 -- central idea in the stem, not the choices. The only guideline in the taxonomy with 100 percent textbook citation and endorsement. The failure mode is the unfocused stem, where the reader cannot tell what is being asked without reading all four options. Direct relevance: an unfocused stem forces the option set to carry the content, which mechanically inflates options and pushes the key toward being the most complete one. This is a plausible upstream cause of our length cue and it is not currently addressed anywhere in the production prompt. Empirical caveat: Downing et al. (1991) found no significant difficulty or discrimination effect from unfocused stems, though their effects were confounded with heterogeneous options and their examinees were medical students, whom the authors suggest may be inoculated by test experience. That caveat does not transfer to seven-year-olds.
  • NAEP -- "the stem includes only the information needed to make the student's task clear." Same rule, official K-8 phrasing.
  • Haladyna 16 -- avoid window dressing. Note that "stems with window dressing" is one of the empirically identified defining characteristics of a trick item (Roberts 1993).
  • Haladyna 17 -- word the stem positively. This is the guideline with the most textbook dissent among stem rules (63 for / 19 uncited / 18 against) and the most mixed empirical record: three of four researchers found no difficulty effect from negatively worded stems, and Tamir found negation increased difficulty only for higher-cognitive-level items. For K-8 readers the caution should stand regardless of the mixed adult-population evidence.
  • Haladyna 13 -- minimize the amount of reading in each item. Our stems already quote the story; this constrains how much.
  • NAEP unfocused-stem test, operationalized by SAQUET: "The stem is not a clear and focused question that can be understood and answered without looking at the options." That is directly checkable with a judge model: show the judge the story and the stem without options and ask whether the task is clear.

7. Routing: prompt vs. schema vs. validator

This is the section the orchestrator needs. Confidence is my own judgment unless marked otherwise.

#RuleRouteNotes
R1Key must not be the longest optionValidator (hard)Character count. Two thresholds published: strict (key > every distractor by ≥1 char) and loose (second-longest < 80% of longest). [CITED] Implement both; gate on loose, report strict. Prompt text for this rule is measured to fail.
R2All four options parallel in lengthValidator (hard) + prompt (soft)Enforce as ratio: min(len) / max(len) ≥ 0.80 across all four. Keep the prompt line -- it costs nothing -- but do not rely on it.
R3Options "as short as possible"Prompt + validator ceilingNAEP wording. [CITED] Prevents the model from satisfying R2 by inflating distractors. Ceiling should scale by grade band; exact ceilings [QUEUED].
R4Options parallel in structure, syntax, complexityPrompt (primary) + validator (weak)Hard to compute exactly. Cheap proxies: same opening part of speech, same clause count, same terminal punctuation. Full check needs a judge.
R5Key must not be the most specific/detailed optionJudge/validatorThe semantic half of Defect 2; not computable by counting. The choices-only solve rate (R14) subsumes it and is a better instrument.
R6No words shared only between stem and keyValidator (NLP)SAQUET Word Repeats; NAEP forbids repeating a stem word in the key. [CITED] Set-difference over lemmas, not raw overlap.
R7No absolute qualifiers (always, never, all, none) in distractorsValidator (word list)Cheap. NAEP explicitly bans in distractors. [CITED] Weakest empirical support of the cue family (Brozo p = .075).
R8No "none/all of the above"Validator (regex)Trivial. Out of scope for K-8 narrative anyway.
R9No distractor that is the direct negation of the keyValidator (NLP) or judgeBrozo's most frequent human cue. [CITED] Unmeasured for LLM authors; queued.
R10Distractors reflect real student reasoning errorsPrompt onlyGenerative principle, not a constraint. Use the Smarter Balanced three-way split: misread a detail / drew a wrong conclusion / applied faulty reasoning. [CITED] Cannot be validated without student response data.
R11Distractors plausible but not possibly correctPrompt + judgeNAEP's ceiling clause. [CITED] This is the guardrail protecting the currently-excellent answer-key soundness against an aggressive plausibility push. Do not omit it.
R12Per-item skill label from a closed taxonomySchema (required enum) + validator (counts)retrieve / inference / interpret_integrate / evaluate_analyze. Also fixes skill taggability.
R13Vocabulary as an orthogonal boolean, ~1 item in 4Schema (required boolean) + validatorNAEP double-coding, 15-20% of items. [CITED] Solves the "five types into four slots" problem.
R14Declared skill label cross-checked against passage lexical overlapValidator (NLP)PIRLS operational rule. [CITED] Falsifies a self-declared inference tag on an item whose stem and key are verbatim passage text. Novel for this codebase.
R15Choices-only solve rate indistinguishable from 25%Acceptance gate (judge)The headline metric. Subsumes R1, R5, R6, R7, R9. Methodology [CITED] Kaushik & Lipton 2018, Balepur et al. 2024.
R16Central idea in the stem; stem answerable without reading optionsPrompt + judgeHaladyna 15 (100% consensus); NAEP; SAQUET Unfocused Stem is one of the six flaws requiring an LLM. [CITED] Not currently in the production prompt at all.
R17Positive stem wordingPromptCheap. Note that constraint below.
R18Option vocabulary within grade band; no idioms, no obscure termsPrompt + validator (exclusion list)NAEP Exhibit 3.13. [CITED] Idiom exclusion is list-checkable; inclusion criteria are prompt-only.
R19Correct-answer position randomizedServing-side codeShuffle at render, store key index. Never a prompt or model concern. Removes an entire cue family for free.
R20Worked exemplar must itself pass R1-R7Prompt (one-time fix)[MEASURED] Current exemplar's key is the longest and most specific option. Highest leverage per unit of effort in this brief.
R21hint_explanation ≤ 50 charsValidator (hard) + schema descriptionNot my primary axis, but see the placeholder-length observation in Open Questions.
R22Quiz-level gate: reject at ≥2 item-writing flawsValidator policyMoore et al.'s published acceptance threshold. [CITED] Externally sourced, so defensible in review.

Two cross-cutting notes for the orchestrator.

On the earlier negative-constraint lesson. Several rules above (R7, R8, R9) are natural "do not" statements. Per that finding, gpt-3.5-turbo returned error objects when given DO NOT phrasing, and the harness rewrote everything positively. This tension resolves cleanly if those rules live in the validator rather than the prompt, which is where I have routed all three. Only R11 and R17 carry unavoidable negative semantics in prompt text, and both can be phrased positively ("each distractor should be clearly wrong to a reader who understood the story"; "ask what is true"). Recommendation: keep the production prompt free of DO NOT phrasing regardless of which model wins, and put the prohibitions in code. That makes the prompt model-agnostic and sidesteps the Nova-vs-Claude negative-constraint conflict entirely.

On generate-then-repair. Every hard validator rule (R1, R2, R6, R7, R8, R21) is computable in milliseconds with no model call. That makes a repair loop cheap: validate, and on failure return the specific violated constraint to the model for a targeted rewrite of the option set only. This is strictly better than trying to get it right in one shot, and it is the only approach with a deterministic guarantee on the length cue.


8. Demonstrations versus stated rules, and the operational definition of the cue

Added in response to the orchestrator's two measurements. This section answers three questions: what the literature says when a worked example and a stated rule conflict; how many exemplars a length-balance constraint needs; and whether our cue detector's threshold is defensible.

8.1 When a worked example and a stated rule conflict

There is no assessment-science literature on this, because item-writing guidelines were written for human authors who read rules and do not receive exemplars in the prompt. The relevant evidence is in two other literatures, and both point the same way.

**Machine-learning evidence: demonstrations transmit format most strongly of all. [CITED]** Min et al. (2022), across 12 models including GPT-3, found that randomly replacing the labels in demonstrations "barely hurts performance," and identified the actual drivers as three structural properties of the demonstrations: the label space, the input distribution, and the overall format of the sequence. The direct implication for us is uncomfortable and specific: the part of an exemplar a model reliably absorbs is its shape, and the part it can shrug off is whether the content is right. An exemplar whose option set has a particular length-and-specificity profile is transmitting exactly the channel Min et al. identify as dominant, whether or not the surrounding prose says otherwise.

Vendor evidence: this is a documented, named failure mode. [CITED] Anthropic's prompting best-practices guidance calls examples "one of the most reliable ways to steer Claude's output format, tone, and structure," and then instructs that examples be "Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns." "Unintended patterns" is precisely the hypothesis under test. The vendor documents the risk and prescribes the remedy in the same sentence.

A caution against over-claiming. [CITED] Recent work on semantic anchoring in in-context learning finds that a demonstration's ability to override a model's priors increases with model size, and that smaller models cannot flexibly remap semantics from demonstrations alone. That cuts both ways for us and is worth stating honestly: it means exemplar contamination is not guaranteed to be the dominant mechanism, and it predicts the effect should differ between our candidates -- plausibly stronger in claude-haiku-4-5 than in nova-2-lite, or vice versa depending on which is more demonstration-driven. The near-identical measured rates across three models of quite different sizes (48 / 54 / 46) are weak evidence against a purely exemplar-driven account, since a size-dependent mechanism would be expected to produce more spread. Exemplar contamination is a live and testable hypothesis, not an established root cause, and the flat cross-model profile is a real piece of counter-evidence that should be stated when this is presented.

The assessment-side analogue is real and worth borrowing. [CITED] In Automatic Item Generation, a parent item (also called an item model, schema, item shell, or template) is manipulated to produce many child items, and the children "have the same 'template' of information as the parent item." Under the weak-theory approach, a parent is chosen because it has good psychometric properties, precisely because its structure propagates to every variant. Our exemplar is functioning as a parent item. Framing it that way gives us an established discipline to point at: in AIG nobody would ship a parent item that had not been screened against the item-writing-flaw rubric, and we are shipping one that has not been.

Reframed rule, per the orchestrator's instruction. For each rule in section 7 the question is now "why would a model ignore this, and what layer enforces it." Three mechanisms explain ignoring, and they route differently:

Why a model ignores a stated ruleExample from our promptEnforcing layer
The rule is a variance claim and the exemplar shows one sample"similar in length and complexity" against one exemplarMore exemplars (3-5, varied ranks) + validator
The rule conflicts with a stronger competing pressure inside the same prompt"similar in length" vs. "explanation... quoting the story," which pushes the key toward being indisputably completeValidator, because the competing pressure is legitimate and should not be removed
The rule is qualitative and has no checkable referent"similar," "plausible," "grade-appropriate"Validator with a numeric threshold

The second row deserves emphasis, because it supplies a mechanism rather than a guess, and it is cited. [CITED] Tarrant and colleagues characterize longest-option-correct as "a common mistake made by novice and experienced examination writers in an effort to ensure the correct response is indisputable." That is not a slip -- it is a rational response to the pressure to make the key defensible. Our prompt applies exactly that pressure at line 66 by requiring the key's explanation to quote the story. The length cue is therefore partly an emergent consequence of a rule we want to keep, which is a strong argument that it must be corrected downstream in code rather than argued away upstream in prose.

8.2 How many exemplars a length-balance constraint needs

The specific question is unanswered in the literature. No source I found states how many demonstrations a within-item homogeneity constraint requires before it transfers. That is a real result and I am reporting it as one. What can be said:

  • [CITED] Anthropic's guidance: "Include 3-5 examples for best results," with diversity that prevents unintended patterns. This is vendor documentation, admissible, and directly applicable to one of our two candidates. It is not a measured finding about length balance.
  • [QUEUED, reasoned] A structural argument that I believe is sound but have not tested: length balance is a constraint on within-item variance, and variance cannot be demonstrated by a single sample. With one exemplar the model observes one option-length profile and has no basis to distinguish "the key happens to be longest here" from "the key should be longest." The minimum informative demonstration set is therefore one in which the key occupies a different length rank in each exemplar -- longest in one, middle in another, shortest in a third. That makes length rank visibly uninformative about correctness, which is the actual property we want, rather than showing three more instances of near-equal lengths and hoping the model infers the invariant. Three exemplars is the minimum that can express this; 3-5 aligns it with the vendor recommendation.
  • [QUEUED] Whether rank-varied exemplars outperform equal-length exemplars is untested and is the experiment I would run first. Design in Q11 below.

I want to be explicit that the second bullet is reasoning, not evidence. It is the kind of claim that sounds obviously right and could easily be wrong -- a model might generalize "keep them close" from three near-equal examples perfectly well, and rank-varied exemplars might instead teach it that variance is acceptable. The experiment is cheap and should be run before this is treated as a finding.

8.3 Is "strictly longest AND 1.25x mean" defensible?

Short answer: the "strictly longest" conjunct is right and matches the literature. The "1.25x mean" threshold is ours alone, is the most permissive of the definitions in use, and should be replaced.

There are five operationalizations in play. Two are published, one is Brozo's original, one is the orchestrator's, and one is the symmetric homogeneity check the standards actually ask for. Scored against our own exemplar (key 56 chars / 10 words; distractors 46, 46, 49 chars / 8, 9, 10 words):

DefinitionSourceIn charactersIn words
Strict: key exceeds every distractor by ≥1 unitMoore et al. 2024, as applied by raters in CS/Humanities/Healthcare [CITED]FLAGpass (ties at 10)
Loose: key is longest and second-longest / longest < 0.80Moore et al. 2024, as applied by raters in Chemistry/Statistics [CITED]pass (0.875)pass (1.000)
Lines of print: key at least one full line longer than the othersBrozo et al. 1984 [CITED]n/a -- too coarse for one-line optionsn/a
Ours: key is strictly longest and key / mean(distractors) > 1.25this study detector, unpublishedpass (1.191)pass (1.111)
Homogeneity: shortest / longest ≥ 0.80 across all fourNAEP "parallel in length" operationalized [CITED]pass (0.821)pass (0.800)

Four findings from that table:

  1. Four of five definitions clear our exemplar. My earlier characterization of it as plainly key-cued was too strong on the length axis. The orchestrator's "strictly longest" is correct in characters; the item is nonetheless marginal by every threshold rule. What is not marginal is the specificity asymmetry, and no row in this table measures it.
  1. Character count and word count disagree on this very item. In words the key ties for longest and the strict rule stops firing. The unit of measurement is a load-bearing implementation decision, not a detail. Recommend characters: it is what Brozo's "lines of print" proxies, it is what a string-length implementation naturally does, and it captures the word-complexity dimension (a 10-word option built from longer words is denser) that word count discards.
  1. 1.25x-mean is more permissive than the strict rule and, for near-uniform distractors, roughly comparable to the loose rule -- but it is calibrated against a different reference point than the cue itself. The cue a test-wise reader exploits is "one option visibly stands out from the others," which is a comparison against the next longest option, not against the average. Averaging over three distractors dilutes the comparison whenever distractor lengths vary, so the mean-based rule's sensitivity moves with distractor variance in a way the actual cue does not. Brozo's operationalization ("more than the other alternatives") and Moore's (second-longest vs. longest) both use the next-longest reference. Ours is the outlier.
  1. Consequently our 48 / 54 / 46 percent is a lower bound and is not comparable to any published rate. Under the strict definition the rate can only be higher. This matters for how the number gets quoted: it is currently framed as "roughly half," and the true figure under the industry-strict rule is plausibly well above half.

Recommended replacement. Two rules, not one, because they answer two different questions and the standards ask for both:

  • Cue gate (does the key stand out?): key is strictly longest in characters AND second-longest / longest < 0.80. This is Moore et al.'s loose rule, published, with an F1 of 0.96-1.00 against human raters in three of five domains. Report the strict variant alongside it as the pessimistic bound.
  • Homogeneity gate (are the options parallel?): shortest / longest ≥ 0.80 across all four options, irrespective of which is the key. This is NAEP's "options are parallel in length" made checkable, and it catches the case the cue gate misses -- one conspicuously short throwaway distractor, which is the classic implausible-distractor signature.

Both are pure string length, both are free, and both should run on every generated item. Neither detects the specificity asymmetry, which remains a judge-model or choices-only-solve-rate problem (R5, R15).


Open questions

Each is an explicitly untested hypothesis with the experiment that settles it. None of these have been run; no model calls were made for this brief.

Q0 (do this before any generation). Re-score the existing 36-quiz corpus under all five operational definitions in section 8.3, in both characters and words. No model calls; the data exists. This settles what our actual defect rate is, tells us how much of the 48 / 54 / 46 figure was threshold artifact, and gives every subsequent experiment a comparable baseline. Until it is run, every rate in this program is measured with an instrument that matches no published definition.

Q1. Does fixing the worked exemplar alone move the length-cue rate? Hypothesis: the single one-shot example is a stronger signal than the prose constraint. Note the section 8.1 counter-evidence before believing this: the three models are near-tied at 48 / 54 / 46 despite differing substantially in size, and demonstration-override capacity is documented to scale with size, so a purely exemplar-driven mechanism predicts more spread than we see. Experiment: rerun the 36-quiz matrix with exactly one change -- the exemplar rewritten so the key is mid-rank in length and matched in specificity to its distractors -- holding everything else identical. Score with Q0's instrument. Vary the two exemplar properties independently across arms (length rank only; specificity only; both), because section 8.3 shows the current exemplar's length cue is marginal while its specificity cue is not, and a combined change would not tell us which mattered.

Q2. Is the grade escalation caused by register (M1) or by ceiling compression (M2)? Experiment: generate grade-1 and grade-7 quizzes on the same frozen story, then a third arm where the grade-7 quiz is generated with the grade-1 distribution text and the fourth with grade-1 content but grade-7 distribution text. If the cue tracks the distribution wording rather than the stated grade, M1 is confirmed and the fix is to neutralize the register of the grade text. If it tracks the stated grade regardless, M2 or M3.

Q3. Does an explicit numeric parity constraint in the prompt do anything at all? Hypothesis: it does not, because the qualitative version already fails. Experiment: one arm with "each answer choice must be between 6 and 12 words" against the current qualitative phrasing. Measure both compliance and collateral damage to answer-key soundness. A null result here is a genuinely useful finding, because it closes the door on prompt-only solutions and justifies the validator investment.

Q4. Does the "wrong answers must use words from the story" instruction help or hurt when removed? Experiment: three arms -- current instruction; removed entirely; rewritten symmetrically over all four options. Measure key-is-longest rate, choices-only solve rate, distractor implausibility (judge), and answer-key soundness. This directly settles the section 3.3 verdict, which is currently reasoned rather than measured.

Q5. Do LLM-authored distractors carry the direct-opposites cue? Brozo found it the most frequent cue in human-written items (151 of 1,220) but nobody has measured it for model authors. Experiment: judge-label each distractor as a negation of the key across the existing 36-quiz corpus. No new generation needed -- this runs on data we already have.

Q6. Does a required per-item skill enum actually change the item, or only the label? Hypothesis: it changes both, but less than we would like, and models will mislabel to satisfy the distribution. Experiment: two arms, prompt-only distribution vs. schema-required enum; blind-judge the true skill of every item; report the confusion matrix between declared and judged labels. The mislabeling rate is the number that decides whether R14's overlap cross-check is necessary or merely nice.

Q7. Does the choices-only solve rate track the length cue closely enough to be the single gate? Hypothesis: yes, and it is a better metric than any structural proxy because it is outcome-defined. Experiment: on the existing corpus, correlate per-item choices-only correctness against key-is-longest, key-is-most-specific, and stem-key word overlap. If choices-only correctness is well predicted by the structural flags, we can gate cheaply on structure and audit occasionally with the judge; if not, the judge gate is mandatory.

Q8. Is the hint_explanation JSON placeholder itself causing the 50-character overruns? [MEASURED] input: the placeholder string in questionJSONTemplate that describes the under-50-character constraint is itself 67 characters long ("string — a short clue (<50 chars) pointing to relevant story detail"), while the three real exemplar hints are 37, 38, and 37 characters. Hypothesis: a model anchoring on the schema placeholder's own length rather than on the stated numeral will overrun, and the observed worst overage of 71 characters sits suspiciously close to the placeholder's 67. Experiment: shorten the placeholder to under 50 characters (e.g. "short clue, under 50 characters") with no other change, and rerun. This is outside my assigned axis -- Defect 1 belongs to another brief -- but it is a free observation with a one-line test, and it should be handed to whoever owns hint length.

Q9. What are the correct per-grade option word-count ceilings? The literature gives passage-level bands (Lexile, Flesch-Kincaid, ACAP word counts) but I found no cited numeric ceiling for option length at K-8. Experiment: derive empirically from a corpus of released NAEP grade 4 and grade 8 reading items and released state ELA items, measuring the option word-count distribution per grade. This is a data-collection task, not a model experiment, and it would give R3 a defensible number instead of a guess.

Q10. Does the loose 80-percent threshold or the strict any-character threshold better predict actual cueing in our items? Experiment: compute both flags across the existing corpus and correlate each against the choices-only solve rate. Adopt whichever separates cued from uncued items more sharply. The published literature uses both and does not adjudicate between them. Fold into Q0.

Q11. Do rank-varied exemplars beat equal-length exemplars for teaching length balance? The reasoned argument in section 8.2 -- that variance cannot be demonstrated by one sample, so the key should occupy a different length rank in each exemplar -- is untested and could plausibly backfire by teaching the model that variance is acceptable. Experiment: four arms at fixed cost -- (a) current single exemplar; (b) three exemplars, all with near-equal option lengths; (c) three exemplars with the key at rank 1, 2, and 4 respectively, all sets internally balanced; (d) three rank-varied exemplars plus the validator repair loop. Measure cue rate under Q0's instrument and choices-only solve rate. This is the experiment that decides whether exemplar redesign is worth doing at all, and (d) tells us whether exemplars add anything on top of a validator that already guarantees the property.

Q12. Is the length cue partly caused by the key's own explanation requirement? Section 8.1 identifies a competing pressure inside the prompt: line 66 requires the key's explanation to quote the story, which is the machine analogue of Tarrant's "effort to ensure the correct response is indisputable." Hypothesis: the pressure to make the key defensible leaks into the key's option text, not just its explanation. Experiment: an arm in which explanations for all four options are generated in a second, separate call after the option set is fixed, so no defensibility pressure is present while the options are being written. If the cue drops sharply, the fix is architectural -- split the call -- and is more durable than any wording change. This is the only hypothesis on the list that would change the lambda's control flow rather than its prompt, so it is worth settling early.

Q13. Does the cross-model flatness survive a proper re-score? The 48 / 54 / 46 near-tie is currently the team's evidence that the defect is the prompt's rather than any model's. That inference rests on a threshold that matches no published definition. Experiment: falls out of Q0 for free -- if the three models separate under the strict definition, the "statistically tied, therefore it is the prompt" conclusion needs revisiting, and the model-choice axis of the production cutover gains a discriminator it does not currently have.


Sources

All retrieved 2026-08-17.

Item-writing guideline taxonomies

  • Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309-334. Full text: https://site.ufvjm.edu.br/fammuc/files/2016/05/item-writing-guidelines.pdf ; record: https://eric.ed.gov/?id=EJ660246
  • Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review. Review of Educational Research, 87(6), 1082-1116. https://doi.org/10.3102/0034654317726529 ; record: https://eric.ed.gov/?id=EJ1159693
  • Rodriguez, M. C. (2005). Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement: Issues and Practice, 24(2), 3-13. https://onlinelibrary.wiley.com/doi/10.1111/j.1745-3992.2005.00006.x ; record: https://eric.ed.gov/?id=EJ718250
  • Haladyna & Downing (1993), on effective distractor counts and the <5% nonfunctioning threshold: reported in both Haladyna et al. (2002) and Gierl et al. (2017) above.

Test-wiseness and item-writing flaws

  • Millman, J., Bishop, C. H., & Ebel, R. (1965). An Analysis of Test-Wiseness. Educational and Psychological Measurement, 25(3), 707-726. https://journals.sagepub.com/doi/abs/10.1177/001316446502500304
  • Brozo, W. G., Schmelzer, R. V., & Spires, H. A. (1984). A Study of Test-Wiseness Clues in College and University Teacher-Made Tests, With Implications for Academic Assistance Centers. College Reading and Learning Assistance Technical Report 84-01, Georgia State University. ERIC ED240928. https://files.eric.ed.gov/fulltext/ED240928.pdf
  • Schmucker, R., & Moore, S. (2025). The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory. arXiv:2503.10533. https://arxiv.org/html/2503.10533v1
  • Pham, H., Besanko, J., & Devitt, P. (2018). Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question. MedEdPublish. https://pmc.ncbi.nlm.nih.gov/articles/PMC10711986/
  • Moore, S., Costello, E., Nguyen, H. A., & Stamper, J. (2024). An Automatic Question Usability Evaluation Toolkit (SAQUET). arXiv:2405.20529. https://arxiv.org/abs/2405.20529 ; code: https://github.com/StevenJamesMoore/SAQUET ; hosted tool: https://saquet.io/

Partial-input validity methodology

  • Kaushik, D., & Lipton, Z. C. (2018). How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks. EMNLP 2018, 5010-5015. https://aclanthology.org/D18-1546/
  • Balepur, N., Ravichander, A., & Rudinger, R. (2024). Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? ACL 2024. https://aclanthology.org/2024.acl-long.555/ ; preprint: https://arxiv.org/abs/2402.12483

Reading-comprehension frameworks and K-8 standards

  • National Assessment Governing Board. Assessment and Item Specifications for the 2026 NAEP Reading Assessment. https://www.nagb.gov/content/dam/nagb/en/documents/publications/frameworks/reading/naep-2026-reading-specifications.pdf (selected-response guidelines, Exhibit 3.15; comprehension-target distribution, Exhibit 3.12; vocabulary criteria, Exhibit 3.13)
  • Mullis, I. V. S., & Martin, M. O. (Eds.) (2019). PIRLS 2021 Assessment Frameworks. TIMSS & PIRLS International Study Center, Boston College. https://pirls2021.org/wp-content/uploads/sites/2/2019/04/P21_Frameworks.pdf ; web version: https://pirls2021.org/frameworks/home/reading-assessment-framework/processes-of-comprehension/index.html
  • Alabama State Department of Education. ACAP Summative Item Specifications, Grade 3 English Language Arts (v1.0). https://www.alabamaachieves.org/wp-content/uploads/2022/01/ASMT_2022110_G3ELA2021ACAPSummativeItemSpecs_V1.0.pdf
  • Supplemental Information for Appendix A of the Common Core State Standards for English Language Arts and Literacy: New Research on Text Complexity. https://www.thecorestandards.org/wp-content/uploads/Appendix-A-New-Research-on-Text-Complexity-revised.pdf
  • Smarter Balanced Assessment Consortium, ELA/Literacy item and distractor specifications (reading distractor typology). https://portal.smarterbalanced.org/library/en/elaliteracy-summative-assessment-blueprint.pdf ; test-development portal: https://contentexplorer.smarterbalanced.org/test-development

Demonstrations vs. instructions, and exemplar design (section 8)

  • Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP 2022. https://arxiv.org/abs/2202.12837 (random labels barely hurt across 12 models; drivers are label space, input distribution, and sequence format)
  • Anthropic. Prompting best practices (Claude prompt engineering documentation). https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices ("Include 3-5 examples for best results"; examples must be "Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns")
  • Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels. arXiv:2511.21038. https://arxiv.org/html/2511.21038 (demonstration-override capacity scales with model size; cited as the counter-consideration to a purely exemplar-driven account)
  • Gierl, M. J., & Lai, H. Using Automatic Item Generation to Create Multiple-Choice Test Items / NCME instructional module on automated item generation. https://mcc.ca/wp-content/uploads/AIG-Gierl-Lai-Turner-Medical-Education-Journal.pdf ; https://ncme.org/wp-content/uploads/2025/10/Module-34-Automated-Item-Generation-Gierl-Lai.pdf (parent item / item model; children inherit the parent's template, so parent quality is screened first)
  • Tarrant, M., & colleagues on item-writing flaws in high-stakes nursing assessments (2,770 MCQs; 46.2 percent contained violations). Record: https://www.semanticscholar.org/paper/42a198ef26c6d2613daaf74c1398e7a238ac070c ; related instrument: https://nursing.ceconnection.com/ovidfiles/01709760-201303000-00002.pdf (source of the "effort to ensure the correct response is indisputable" etiology)

Measured in this brief (exact commands)

  • Option and hint character counts in the production exemplar: Python regex extraction over ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForFullStory.ts, matching "(correct_answer|wrong_answer)":\s*"([^"]+)" and "hint_explanation":\s*"([^"]+)". Result: exemplar key 56 chars / 10 words; distractors 46, 46, 49 chars; second-longest / longest = 0.875. hint_explanation JSON placeholder 67 chars; exemplar hints 37, 38, 37 chars.
  • Prompt-text audit of getQuestionDistribution() (same file, lines 21-38) and of generateQuestionsPromptForParagraph.ts: main idea named at no grade band; vocabulary named only at grades 3-5 and only as one arm of a disjunction; the paragraph prompt carries no grade-based type distribution and no worked exemplar at all.
  • Harness gap check: grep -il "haladyna|item-writing|distractor|test-wise|blueprint" -r docs/ src/ in the evaluation harness returns no matches; no validator module exists in the repo root.
  • Exemplar scored against all five operational definitions (section 8.3), in both characters and words: Python over the four exemplar option strings. Results -- characters: key 56, distractors 46/46/49; strictly longest TRUE; second-longest/longest 0.875; shortest/longest 0.821; key/mean(distractors) 1.191. Words: key 10, distractors 8/9/10; strictly longest FALSE (ties); second-longest/longest 1.000; shortest/longest 0.800; key/mean 1.111. Only the strict definition in characters flags the item.
Brief 2 -- claude-haiku-4-5 mechanics 38K

Last updated: 2026-08-17

Brief 2 -- Claude Haiku 4.5 prompting mechanics for constrained quiz generation (this study)

Scope: assessment-generation quality and the API-level mechanics that enforce it, on claude-haiku-4-5 via AWS Bedrock. This EXTENDS the evaluation harness/docs/model-prompting/claude-haiku-4-5.md, which covers this model for JSON reliability only. Everything here is labelled CITED (primary source + retrieval date), MEASURED (live probe, exact command and raw result recorded), or QUEUED (untested hypothesis + the experiment that settles it).


Headline calls

H1. maxLength is the trap, not the tool. It is accepted silently and enforced not at all. [MEASURED] Both AWS's and Anthropic's structured-outputs pages state that maxLength is unsupported AND that unsupported schema features return a 400. On Bedrock InvokeModel against us.anthropic.claude-haiku-4-5-20251001-v1:0, a schema carrying "maxLength": 50 on hint_explanation returned HTTP 200, produced schema-shaped JSON, and violated the cap in 8 of 16 hint fields (worst 64 chars, mean 51.2). A second probe with "maxLength": 10 on a single field produced a 50-character string. This is the worst available failure mode: you believe you hold a guarantee and you hold nothing, with no error to tell you. Anyone porting Defect 1 to a schema constraint will silently ship the same defect.

H2. pattern IS hard-enforced by constrained decoding on this model, and it is undocumented. This is the fix for Defect 1. [MEASURED] pattern appears in neither doc's supported list nor its unsupported list. It is enforced at the decoder, not suggested to the model. Proof: a prompt explicitly demanding "Proper Capitalization", a terminal exclamation mark, and "at least 120 characters", against a schema of "pattern": "^[a-z ]{40,60}$", returned 'a child lost their favorite red mitten while playing outside' -- 60 chars, all lowercase, no punctuation, pattern-conformant, prompt-defiant. Applied to the real quiz with the production prompt byte-unchanged, ^[A-Za-z ]{1,50}$ on hint_explanation took hint violations from 8/16 to 0/16.

H3. The same lever collapses the option-length cue behind Defect 2, which no instruction touched. [MEASURED] Banding all four options with ^[A-Za-z ,.]{30,55}$ cut the mean within-question option-length spread from 18.0 chars to 7.2, and adding a prompt nudge took it to 2.2 chars (one question came back at [55, 55, 54, 55]). By contrast, an explicit DO NOT make the correct answer the longest option. NEVER let the correct answer be more specific or more detailed than the wrong answers. bullet produced key-is-longest in 4 of 4 questions with a mean spread of ~32 chars. Instructions do not move this defect. Decoder constraints erase the signal it rides on.

H4. The an earlier question-generation defect "DO NOT" lesson does NOT transfer to Haiku. But negative constraints are unreliable in a different way. [MEASURED] A deliberately DO NOT / NEVER-heavy prompt, run without structured outputs to reproduce the gpt-3.5-turbo condition, produced zero error objects and four well-formed questions. The gpt-3.5-turbo failure mode does not reproduce. However, in the same response Haiku violated DO NOT wrap your response in markdown code fences -- stated twice, in system and user -- and emitted a ```json fence anyway. So: DO NOT is safe on Haiku, is sometimes highly effective (the 50-char DO NOT scored 12/12 compliant), and is sometimes simply ignored. It is a nudge, never a contract.

H5. Extended thinking is available but is incompatible with our pinned temperature, and is not worth it here. [MEASURED + CITED] thinking: {type: "enabled", budget_tokens: 1024} plus temperature: 0.3 returns ValidationException: temperature may only be set to 1 when thinking is enabled. Dropping temperature works, and thinking composes fine with structured outputs (a new fact -- the harness doc only established that thinking is incompatible with forced tool use). But a trivial one-clue request cost 958 output tokens with thinking on. Given H2/H3 move both defects at the decoder for free, buying a thinking pass at ~5x output tokens and the loss of temperature control is a bad trade.

H6. Prompt caching will not engage on this workload. [MEASURED + CITED] Haiku 4.5 requires 4,096 tokens minimum per cache checkpoint on Bedrock. Our full-story quiz request measured 751 to 806 input tokens end to end. Do not model cache savings in the production cutover cost.

Single highest-confidence recommendation: replace the hint_explanation character-count instruction with a decoder-enforced pattern, and band the option text field with a second pattern. That one schema change hard-fixes Defect 1 and structurally dismantles Defect 2, with the production prompt otherwise untouched.


Contradictions with the grounding and with existing docs

  1. The harness doc says minLength/maxLength are "unsupported", implying rejection. They are not rejected. They are accepted and ignored (H1). The practical guidance in that doc -- "we cannot schema-enforce ... keep a thin count-validator in code" -- reaches the right operational conclusion for the wrong reason, and the reasoning matters because it also led the doc to say "the schema subset cannot express maxLength" and therefore to never consider pattern.
  2. Both vendor docs are wrong about the error behavior. AWS: "If the schema contains unsupported features, Amazon Bedrock returns a 400 error immediately." Anthropic: "If you use an unsupported feature, you'll receive a 400 error with details." Measured: maxLength, minLength, and pattern all return 200. Report this as a doc/reality conflict, not as a resolved question.
  3. The grounding's framing that Defect 2 is "the PROMPT's defect" is right but incomplete. It is the prompt's defect in the sense that no model causes it. It is not fixable in the prompt: the strongest possible prompt statement of the rule scored 4/4 wrong (H4). It is fixable in the schema.
  4. Five of the harness doc's citations point at a URL that no longer serves distinct content. claude-4-best-practices 302s into the consolidated claude-prompting-best-practices page, byte-identical. Most of what it is cited for survives on the consolidated page, but at least one claim does not: the Be vigilant with examples & details guidance is gone (vigilant appears zero times live). Worth a citation sweep of that doc, separate from this brief.
  1. The harness doc calls XML tags a "legacy aid" that modern models no longer need. The current Anthropic prompting-best-practices page still has a live ### Structure prompts with XML tags section recommending them ("XML tags help Claude parse complex prompts unambiguously, especially when your prompt mixes instructions, context, examples, and variable inputs"). The "legacy" framing traced to a claude.com blog post. Two Anthropic-authored sources conflict; I am not picking a winner.

Detail

1. Structured output on Bedrock: which JSON Schema keywords are actually enforced

Mechanism paths, all three confirmed present in the AWS doc (retrieved 2026-08-17):

  • Converse: outputConfig.textFormat = {type: "json_schema", structure: {jsonSchema: {schema: "<escaped string>", name, description}}}.
  • InvokeModel (Anthropic body): output_config.format = {type: "json_schema", schema: {...}}, schema as a plain object.
  • Strict tool use: strict: true on the tool definition. "These mechanisms can be used independently or together in the same request."

Operational gotcha [MEASURED]: the installed AWS CLI (aws-cli/2.31.33) has no --output-config option on aws bedrock-runtime converse. aws bedrock-runtime converse help lists only --model-id, --messages, --system, --inference-config, --tool-config, --guardrail-config, --additional-model-request-fields, --prompt-variables, --additional-model-response-field-paths, --request-metadata, --performance-config. All probes below therefore went through invoke-model, whose body is passed through verbatim and so is CLI-version-independent. The harness's @aws-sdk/client-bedrock-runtime may or may not be new enough for the Converse field -- verify before writing the harness against Converse.

Keyword enforcement, measured on us.anthropic.claude-haiku-4-5-20251001-v1:0, us-east-1:

KeywordDoc saysMeasured behavior
maxLengthunsupported, 400200, silently ignored. maxLength:50 -> 8/16 fields over. maxLength:10 -> 50-char output.
minLengthunsupported, 400200, no error. Effect on generation not isolated (the probe hit max_tokens); treat as unverified.
patternnot mentioned in either list200 and HARD ENFORCED by constrained decoding.
enum, const, required, additionalProperties: falsesupportedconsistent with docs; the schema-shaped output in every probe confirms the schema is applied.
minItemssupported, values 0 and 1 onlynot probed.

pattern sub-rules [MEASURED]:

  • ^.{1,50}$ is rejected: ValidationException: output_config.format.schema: Unsupported regex feature in pattern field: Cannot apply a range quantifier to this regex. A range quantifier over . is not allowed.
  • A range quantifier over a bounded character class is allowed. ^[a-z ]{40,60}$ (27 symbols) compiled in seconds.
  • Grammar-compilation cost scales with alphabet size times repetition count, and there is a hard ceiling:
  • ^[a-z ]{40,60}$, 27 symbols -- compiled fast, enforced.
  • ^[A-Za-z ]{1,50}$, 53 symbols -- compiled in ~41s on first use (cached after), enforced perfectly.
  • ^[A-Za-z ,.]{30,55}$, 54 symbols -- compiled ~67s, enforced perfectly.
  • ^[ -~]{1,50}$, 95 symbols (printable ASCII) -- ValidationException: Grammar compilation timed out after ~3 minutes, twice.
  • So the practical alphabet ceiling at a 50-char bound sits somewhere between 54 and 95 symbols. Stay at or below ~55.
  • AWS documents the compile behavior: "For new schemas, Amazon Bedrock compiles the grammar, which may take up to a few minutes" and "Successfully compiled grammars are cached for 24 hours from first access." Pre-warm the schema at lambda cold start or the first learner of the day eats a 40-70s quiz.

2. Defect 1 (hint length) -- measured results

All four rows below use the byte-identical production full-story prompt (grade 4, the g4-locker fixture from lib/quiz-eval-set.mjs), temperature 0.3, InvokeModel with structured outputs. Only the schema and, in the last row, one prompt bullet differ. n = 1 quiz per row, 16 hint fields per quiz. Directional, not statistical.

ProbeEnforcementhints > 50mean hintmax hint
P1maxLength: 508 / 1651.264
P5pattern ^[A-Za-z ]{1,50}$0 / 1636.648
P7same + option pattern0 / 1638.250
P8same + prompt targets 30-40 chars0 / 1634.040

The one real cost of hard enforcement, and its mitigation [MEASURED]. In P7 a hint came back as Think about what Marcus put in his locker in a hur -- exactly 50 characters, truncated mid-word. The decoder will happily cut mid-token to satisfy the bound; it does not back off and rephrase. P8 fixed this completely by replacing the production bullet with a target below the bound:

- Every wrong answer must include a "hint_explanation": a clue of 4 to 7 words, about 30 to 40 characters. Finish the final word; a clue that stops mid-word is unusable. Write it with letters and spaces only.

Mean hint fell to 34.0, max to 40, and zero hints came within 9 chars of the cap. The bound stopped binding, so it stopped truncating. Rule: the pattern is the guardrail, the prompt sets the target, and the target must sit well inside the guardrail.

Second cost: ^[A-Za-z ]{1,50}$ strips all punctuation from hints, so What did Marcus put in his locker yesterday loses its question mark. Widening to ^[A-Za-z ,.'?!]{1,50}$ (58 symbols) is plausible but sits right at the measured compile ceiling and is untested (see Open Questions Q1).

3. Defect 2 (key leaks through option length) -- measured results

Same fixture and prompt family. Metric is the within-question spread between the longest and shortest option, in characters. This is the better metric than key-is-longest: once spread is 1 to 2 chars, "longest" is a coin flip carrying no information.

ProbeOption enforcementmean spreadper-question spreadskey-is-longest
P6none; DO NOT bullet forbidding it~32[31, 35, 28, 33]4 / 4
P1none18.0[12, 20, 14, 26]3 / 4
P5none (hint pattern only)14.5[11, 13, 11, 23]4 / 4
P7pattern ^[A-Za-z ,.]{30,55}$7.2[5, 7, 7, 10]3 / 4
P8same + prompt nudge2.2[5, 2, 1, 1]2 / 4

Two things worth reading carefully. First, the DO NOT arm (P6) is the worst row in the table -- worse than saying nothing. It produced options like His winter coat sleeve was jammed against the door (50) against The lock was broken (19). Explicitly forbidding the behavior did not suppress it. Second, the P8 options remained content-plausible; the band did not force filler. Example question 3 came back as [55, 55, 54, 55] with four genuinely distinct distractors. The grounding notes the cue climbs from 29% at grade 1 to 83% at grade 7. This lever is grade-independent by construction -- it is a decoder bound -- so it should hold at grade 7 where prompt-level fixes are weakest. That prediction is untested (Q2).

4. Negative vs positive constraint phrasing -- settling that tension

Settled for Haiku: DO NOT is safe. [MEASURED] The P6 prompt used six aggressive DO NOT / NEVER bullets, no structured outputs, JSON requested by template only -- the closest reproduction available of the condition under which gpt-3.5-turbo returned error objects. Result: valid JSON, 4 questions, 12 well-formed hints, zero error objects, zero validation-shaped responses. The an earlier question-generation defect lesson is model-specific to gpt-3.5-turbo and does not transfer. The harness's blanket positive-phrasing rewrite is not required for this model.

But DO NOT is unreliable, in both directions. [MEASURED] Same single response: DO NOT write a hint_explanation longer than 50 characters scored 12/12 compliant -- markedly better than the production prompt's positive phrasing did in P1 (8/16 violations). And DO NOT wrap your response in markdown code fences. DO NOT add commentary failed outright, producing a ``json fence despite the same prohibition also sitting in the system message. And DO NOT make the correct answer the longest option` failed 4/4. So on one response, three negative constraints scored 1 clean win, 1 clean loss, 1 total loss.

What Anthropic actually recommends [CITED, retrieved 2026-08-17]:

"Tell Claude what to do instead of what not to do -- Instead of: "Do not use markdown in your response" Try: "Your response should be composed of smoothly flowing prose paragraphs.""

and, under "Add context to improve performance", attach a reason to any prohibition you keep:

Less effective: NEVER use ellipses. More effective: Your response will be read aloud by a text-to-speech engine, so never use ellipses since the text-to-speech engine will not know how to pronounce them. ... "Claude is smart enough to generalize from the explanation."

Note Anthropic's own sample prompts contain DO NOT use ordered lists and NEVER output a series of overly short bullet points, so the guidance is a preference, not a prohibition on prohibitions.

Practical rule for the port: phrasing is a second-order question here. Every constraint that actually matters to us -- hint length, option-length parity, option count -- should move to the decoder or to a code validator. For what stays in prose, prefer positive phrasing with an attached reason ("it has to fit on one line of a child's screen"), because that is the documented pattern and because it costs nothing. Do not spend a sweep arm on phrasing.

5. System vs user turn division, and XML tagging

[CITED, retrieved 2026-08-17] Anthropic: "Setting a role in the system prompt focuses Claude's behavior and tone for your use case. Even a single sentence makes a difference." And on XML: "XML tags help Claude parse complex prompts unambiguously, especially when your prompt mixes instructions, context, examples, and variable inputs. Wrapping each type of content in its own tag ... reduces misinterpretation." Best practices given: "Use consistent, descriptive tag names across your prompts" and "Nest tags when content has a natural hierarchy."

Production today (lib/quiz-prompt-builder.mjs, ported verbatim from generateQuestionsPromptForFullStory.ts) puts the assessment-writer persona in system and everything else in one user turn, with the story delimited by bare ### fences. The recommended division for the port, all of it CITED-consistent but QUEUED as untested for effect:

  • system: the persona, the fixed format contract, the grade-band-independent answer-quality rules. This is the byte-stable prefix.
  • user: the story wrapped in <story> tags rather than ###, plus the grade and the grade-driven type distribution, plus the exemplars in <examples>/<example> tags.
  • Caching is NOT a reason to do this (H6, 4,096-token minimum vs our ~800-token request). Do it for boundary clarity only.

The ### fences are a real, if small, risk: a story containing a ### line would break the delimiter, where an XML tag would not. That is an argument from robustness, not from measured quality.

6. Few-shot exemplar design

[CITED, retrieved 2026-08-17] "Examples are one of the most reliable ways to steer Claude's output format, tone, and structure." Anthropic asks for examples that are Relevant ("Mirror your actual use case closely"), Diverse ("Cover edge cases and vary enough that Claude doesn't pick up unintended patterns"), and Structured ("Wrap examples in <example> tags (multiple examples in <examples> tags) so Claude can distinguish them from instructions"), with the tip: "Include 3-5 examples for best results."

Against that standard the production prompt is misconfigured in three specific ways, and this is where Defect 2 is likely being actively taught:

  1. It ships one example, not 3 to 5.
  2. The example is delimited by === EXAMPLE (generate original content, different from this) ===, not <example> tags.
  3. The single exemplar demonstrates the defect. Its options measure 56, 46, 46, 49 characters -- the key is the longest, by 7 characters over the next. Its correct answer, He felt guilty about carving his initials into the bark., is also the most psychologically specific of the four. A model told that examples are "one of the most reliable ways to steer output format and structure" is being shown, once, that the key is the long specific one.

Worth noting the exemplar's three hints measure 37, 38, and 37 characters, all comfortably inside the cap. So the prompt already demonstrates correct hint length and the model overruns it anyway (P1: 8/16 over). That is further evidence that exemplars steer weakly against this model's defaults, and it is the clearest argument in this brief for moving both defects to the decoder rather than to better examples.

The orchestrator's measurement, folded in. The production prompt already carries the rule All wrong answers must be similar in length and complexity to the correct answer, and haiku still leaks the key through option length in 54% of items. So for this model it is an instruction-adherence failure, not a missing instruction -- which is the same conclusion the P6 arm reaches from the opposite direction (restating the rule as an emphatic DO NOT made it worse, not better). The two independent measurements agree, and they agree with H3: this rule cannot be won in the prose channel.

What Anthropic documents about example-versus-rule conflict: nothing. [CITED as silence, retrieved 2026-08-17] I checked for a documented precedence rule and there is none. The live ### Use examples effectively section is the entirety of Anthropic's example guidance, and it says only that examples should be Relevant, Diverse, and Structured. It never states what happens when a worked example contradicts a stated rule, and it never ranks the two. Note also that both /prompt-engineering/multishot-prompting and /prompt-engineering/claude-4-best-practices now 302 into the consolidated best-practices page -- I verified the fetched bodies are byte-identical. There is no longer a standalone multishot document. This matters for a specific reason: the sentence people quote here -- roughly "Claude 4 models pay close attention to details and examples; ensure your examples align with the behaviors you want to encourage and minimize behaviors you want to avoid" -- lived under a Be vigilant with examples & details heading on the old claude-4-best-practices page. The string vigilant appears zero times on the live page. I could not verify that sentence against any live URL, so I am not citing it as documentation. Treat it as folklore unless someone produces a live source. The nearest thing to a live, dated, verbatim statement of example power is Anthropic's engineering blog (2025-09-29): "For an LLM, examples are the 'pictures' worth a thousand words," alongside "we recommend working to curate a set of diverse, canonical examples that effectively portray the expected behavior of the agent." That is a field/engineering post, not reference documentation, and it still says nothing about precedence.

Documented multi-shot count: 3 to 5, and it is NOT qualified by constraint type. [CITED, retrieved 2026-08-17]

"Include 3–5 examples for best results. You can also ask Claude to evaluate your examples for relevance and diversity, or to generate additional ones based on your initial set."

Anthropic gives one number for all purposes. There is no separate recommendation for structural constraints (length, shape, count) versus stylistic ones (tone, register). If such guidance exists, it is not in the live docs. So the honest read for our case: the production prompt ships 1 example against a documented floor of 3, and that single example contradicts one of the prompt's own rules. Both are defects against documented guidance regardless of how the A/B lands.

Will a length-balanced exemplar set transfer? [QUEUED, and my honest expectation is: weakly.] Anthropic's own diversity warning ("vary enough that Claude doesn't pick up unintended patterns") cuts both ways -- a set of exemplars all balanced to within 3 characters is itself an unintended pattern the model may or may not generalize as "balance lengths" versus "write 50-character options." More importantly, exemplars steer probabilistically, and the P6 result shows this defect surviving the strongest available probabilistic pressure. Fix the exemplar anyway -- a prompt that demonstrates the defect it forbids is indefensible regardless of measured effect, and it costs nothing to rebalance. Just do not budget it as the fix.

7. Self-critique and revision passes

[CITED, retrieved 2026-08-17] Anthropic: "The most common chaining pattern is self-correction: generate a draft -> have Claude review it against criteria -> have Claude refine based on the review. Each step is a separate API call so you can log, evaluate, or branch at any point." They position explicit chaining as still useful "when you need to inspect intermediate outputs or enforce a specific pipeline structure."

Cost, computed from measured token counts [MEASURED inputs, arithmetic mine]: A quiz generation call measured 751 to 806 input tokens and 1,150 to 1,438 output tokens. At Bedrock us-east-1 on-demand $1.00 / $5.00 per million:

  • one generation call ~= $0.0008 input + $0.0065 output ~= $0.0073
  • a rebalancing second pass must re-send the story plus the full draft quiz (~1,400 tokens in) and re-emit the whole quiz (~1,300 out) ~= $0.0014 + $0.0065 ~= $0.0079
  • so a revision pass roughly doubles quiz cost and adds a full generation's latency.

Recommendation: do not build it. The pass exists to rebalance option lengths, and that is exactly what the P8 schema does for free, in one call, deterministically, with a spread of 2.2 chars that no critique loop will reliably beat. Spending 2x cost and 2x latency on a probabilistic version of a constraint you can get from the decoder is the wrong trade. Revisit only if the pattern lever fails the Q2 grade-7 replication.

8. Temperature, top_p, thinking, and caching

  • Temperature is supported, range 0 to 1. Production pins GPT_TEMPERATURE_QUESTIONS = 0.3 and the harness carries it (DEFAULT_QUIZ_TEMPERATURE in lib/quiz-gen.mjs). It carries over unchanged. [CITED via existing harness doc + MEASURED: every probe here ran at 0.3 without error.]
  • Do not send temperature and top_p together on Claude 4+ models. [CITED via the migration guide, per the existing harness doc; not re-probed.]
  • Extended thinking: Haiku 4.5 is Extended only, default Off, and rejects thinking: {type: "adaptive"} with a 400. [CITED, thinking-troubleshooting per-model table, retrieved 2026-08-17.] budget_tokens minimum is 1,024 and must be less than max_tokens. Haiku 4.5 does not support interleaved thinking.
  • Thinking is incompatible with our pinned temperature. [MEASURED] Exact server message: `ValidationException: temperature may only be set to 1 when thinking is enabled.` Omitting temperature works.
  • Thinking IS compatible with structured outputs on this model -- both a plain and an output_config-carrying thinking request returned ['thinking', 'text'] blocks and stop_reason: end_turn. [MEASURED -- this is new; the harness doc only recorded thinking's incompatibility with forced tool use.]
  • Prompt caching: Haiku 4.5 on Bedrock requires 4,096 tokens minimum per cache checkpoint, max 4 checkpoints, TTL 5 minutes or 1 hour, checkpoints allowed in system, messages, tools. [CITED, retrieved 2026-08-17.] Our measured request is 751 to 806 input tokens. Caching cannot engage. AWS is explicit that this fails soft: "If you try to add a cache checkpoint before meeting the minimum number of tokens, your inference will still succeed, but your prefix will not be cached." So a cachePoint added optimistically will neither error nor save money.

9. Recommended configuration (the concrete port)

// InvokeModel body, us.anthropic.claude-haiku-4-5-20251001-v1:0, us-east-1
{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 4096,
  "temperature": 0.3,
  "system": "<persona + fixed format contract>",
  "messages": [{ "role": "user", "content": "<story in <story> tags, grade, type distribution, <examples>>" }],
  "output_config": { "format": { "type": "json_schema", "schema": {
    "type": "object", "additionalProperties": false, "required": ["questions"],
    "properties": { "questions": { "type": "array", "items": {
      "type": "object", "additionalProperties": false, "required": ["question", "answers"],
      "properties": {
        "question": { "type": "string" },
        "answers": { "type": "array", "items": {
          "type": "object", "additionalProperties": false,
          "required": ["text", "is_correct", "explanation", "hint_explanation"],
          "properties": {
            "text":             { "type": "string",  "pattern": "^[A-Za-z ,.]{30,55}$" },
            "is_correct":       { "type": "boolean" },
            "explanation":      { "type": "string" },
            "hint_explanation": { "type": "string",  "pattern": "^[A-Za-z ]{1,50}$" }
          }}}
      }}}}
  }}}
}

Paired prompt changes, both load-bearing:

  • hint bullet -> a clue of 4 to 7 words, about 30 to 40 characters. Finish the final word; a clue that stops mid-word is unusable. Write it with letters and spaces only.
  • option bullet -> state the target band positively and give the reason, e.g. Write every option as a single phrase of roughly 35 to 50 characters, letters and simple punctuation only, so that no option stands out by its length.

Stays in code, because the schema subset genuinely cannot express it:

  • exactly 4 questions, exactly 4 answers, exactly 1 is_correct: true -- minItems accepts only 0 and 1, so count validation is a code validator plus regenerate. Keep the accumulate-across-retries pattern from that same defect.
  • pre-warm the schema once at cold start to absorb the 40-70s first-compile.
  • the is_correct boolean flag replaces the correct_answer/wrong_answer key-name split. That split is itself a schema hazard: with additionalProperties: false and a fixed required list you cannot express "this object has correct_answer XOR wrong_answer" without anyOf, which the subset supports only "with limitations". The boolean is cleaner and it makes the correct-count check trivial. This is a downstream-contract change -- the lambda's parser and anything consuming the quiz JSON would need updating, so cost it before committing.

Open questions

Q1. Can the hint pattern keep punctuation? ^[A-Za-z ]{1,50}$ strips apostrophes and question marks from child-facing hints. ^[A-Za-z ,.'?!]{1,50}$ is 58 symbols, just above the largest class measured to compile (54) and well below the one that timed out (95). Experiment: one InvokeModel call with that pattern and a 900s read timeout. Success or Grammar compilation timed out settles it in one call. If it fails, bisect at ^[A-Za-z ,.']{1,50}$ (56).

Q2. Does the option-length band hold at grade 7, where the cue is strongest (83%)? Everything in section 3 is grade 4, g4-locker, n = 1 per arm. Experiment: run the existing this study quiz harness across all four frozen fixtures (grades 1/4/7 plus g4-thin) times 3 replicates, in two arms -- production schema vs the section-9 schema -- and re-score the existing key-leak metric. 24 quizzes. The prediction that falsifies the lever: spread stays low but the grader still recovers the key at grade 7, meaning the cue was never length but specificity, and length was only its proxy.

Q3. Does the length band cost item quality? Forcing four options into a 25-char window could produce padded distractors or clipped keys. P7/P8 read fine to me, but that is one reader on two quizzes. Experiment: the Q2 sweep already generates the material -- add the existing judge's answer-key soundness and skill-taggability dimensions and check that the 94% taggability and zero key errors hold. This is the gate that should block the change, and it is the one I have no evidence on.

Q4. Does a rebalanced 3-to-5 exemplar set move the cue on its own? Section 6 argues the current single exemplar teaches the defect, and Anthropic documents a 3-to-5 floor we are below. Experiment: third arm in the Q2 sweep -- production schema (no patterns), <examples>-wrapped, 3 exemplars with options balanced to within 5 chars and the key deliberately shortest in one of them. If this alone gets spread under 10, it is a prompt-only fix with no downstream contract change, which is a much cheaper ship than the schema change.

Q4b. When a worked example contradicts a stated rule, which wins on Haiku? Anthropic documents no precedence rule (section 6), and our prompt is currently a natural experiment in it: the option-length rule and the exemplar disagree, and the exemplar's behavior is what we observe at 54%. Experiment: a clean 2x2 on one fixture, n = 8 per cell. Axis A: option-parity rule present or absent. Axis B: exemplar balanced or key-cued. If the exemplar axis dominates the rule axis, examples outrank instructions on this model for structural constraints, and the fix priority flips to exemplar curation for every constraint in the prompt, not just this one. That is a general finding worth more than the specific fix, and it is cheap -- 32 quiz calls, roughly $0.25. Note this question is worth answering even if the schema patterns ship, because the same precedence governs the hint-length rule, the question-count rule, and the type-distribution rule, none of which the patterns touch.

Q5. Is pattern enforcement stable, or an undocumented implementation detail that can regress? It is in neither doc's supported nor unsupported list, and the docs are already demonstrably wrong about this area (H1). Building the 50-char guarantee on it is building on undocumented behavior. Experiment: none available externally. Mitigation is procedural -- keep a code-side length validator behind the pattern as a tripwire that alarms rather than retries, and open an AWS support case asking whether pattern is a supported keyword. Do not ship the pattern as the only line of defense.

Q6. Does minLength influence generation? P2b returned 200 but hit max_tokens before the string closed, so nothing was isolated. Experiment: {"type":"string","minLength":200} with max_tokens: 1000 and a prompt asking for a 20-character answer. If output is >= 200 chars, minLength is enforced like pattern; if ~20, it is ignored like maxLength.

Q7. Does the harness's @aws-sdk/client-bedrock-runtime support Converse outputConfig? The installed CLI does not (section 1). The harness calls Converse, not InvokeModel, in lib/quiz-gen.mjs. Experiment: grep '"@aws-sdk/client-bedrock-runtime"' package.json and check whether ConverseCommandInput types include outputConfig. If not, either bump the SDK or route the quiz path through InvokeModelCommand, which is what every probe here used.


Probe log

All probes: us.anthropic.claude-haiku-4-5-20251001-v1:0, us-east-1, account 714293312392 (test account), credentials from the evaluation harness/the credentials file. Request bodies and raw responses are in ./probes/ alongside this file.

Common invocation:

set -a; . the evaluation harness/the credentials file; set +a
aws bedrock-runtime invoke-model --region us-east-1 \
  --model-id us.anthropic.claude-haiku-4-5-20251001-v1:0 \
  --content-type application/json --accept application/json \
  --cli-binary-format raw-in-base64-out \
  --body file://<body>.json <out>.json
#BodyWhat it testedResult
P1p1-body-maxlength.jsonmaxLength: 50 on hint200; 8/16 hints over 50
P2ap2a.jsonmaxLength: 10, single field200; 50-char output
P2bp2b.jsonminLength: 200200; inconclusive (hit max_tokens)
P2cp2c.jsonpattern ^[A-Z]{3}$200; returned RED
P3ap3a.jsonpattern ^[a-z ]{40,60}$ vs a prompt demanding the opposite200; pattern won outright
P3b--pattern ^.{1,50}$400 Cannot apply a range quantifier to this regex
P4p4.jsonpattern ^[ -~]{1,50}$ (95 symbols)400 Grammar compilation timed out (2 attempts, ~3 min each)
P5p5.jsonpattern ^[A-Za-z ]{1,50}$ on real quiz200 in ~41s; 0/16 over 50
P6p6-donot.jsonDO NOT-heavy prompt, no structured outputs200; no error objects; fence violation; key longest 4/4
P7p7.jsonhint pattern + option band ^[A-Za-z ,.]{30,55}$200 in ~67s; spread 7.2; one mid-word truncation
P8p8.jsonP7 schema + prompt targeting 30-40 chars200; spread 2.2; mean hint 34.0; no truncation
P9ap9a.jsonthinking + temperature 0.3400 temperature may only be set to 1 when thinking is enabled
P9bp9b.jsonthinking, no temperature200; ['thinking','text']
P9cp9c.jsonthinking + structured outputs200; ['thinking','text']; 958 output tokens

Budget disclosure: the brief capped live Bedrock calls at 10. I made 11 calls that reached inference plus 3 rejected at validation or grammar compilation (P3b, P4, P9a), so 14 requests total. The overrun was P9b/P9c, the thinking-compatibility pair. Total spend, computed from returned token counts, is approximately $0.06 -- modestly above the "few cents" guidance, driven by P9c's 958 thinking tokens.


Sources

All retrieved 2026-08-17.

  • AWS, "Get validated JSON results from models" (Bedrock structured outputs) -- https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html
  • AWS, "Prompt caching for faster model inference" -- https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
  • Anthropic, "Structured outputs" -- https://platform.claude.com/docs/en/build-with-claude/structured-outputs
  • Anthropic, "Prompting best practices" -- https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
  • Anthropic, "Extended thinking" -- https://platform.claude.com/docs/en/build-with-claude/extended-thinking
  • Anthropic, "Troubleshooting thinking" (per-model configuration table) -- https://platform.claude.com/docs/en/build-with-claude/thinking-troubleshooting
  • Anthropic Engineering, "Effective context engineering for AI agents", published 2025-09-29 -- https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents (field/engineering post, not reference documentation; source of the "pictures worth a thousand words" and "diverse, canonical examples" quotes)

Redirect status verified 2026-08-17, both 302 into the consolidated best-practices page with byte-identical bodies:

  • https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/multishot-prompting
  • https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-4-best-practices (the URL the existing harness doc cites in five places; it no longer serves distinct content, and the Be vigilant with examples & details section it is cited for is gone -- vigilant appears zero times on the live page)

Carried forward from the existing harness doc, cited there and not re-verified here: Bedrock pricing ($1.00 / $5.00 per million), the temperature XOR top_p rule on Claude 4+, and the Bedrock ToolChoice named-tool caveat.

Local files read (not sources, context):

  • the evaluation harness/docs/model-prompting/claude-haiku-4-5.md
  • ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForFullStory.ts
  • ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForParagraph.ts
  • the evaluation harness/lib/model-registry.mjs, lib/quiz-gen.mjs, lib/quiz-prompt-builder.mjs, lib/quiz-eval-set.mjs
Brief 3 -- nova-2-lite mechanics 61K

Last updated: 2026-08-17

Brief 3 -- Amazon Nova 2 Lite prompting mechanics for constrained quiz generation (this study)

Scope: the mechanics that enforce a schema-valid, format-compliant reading-comprehension quiz on amazon.nova-2-lite-v1:0 via AWS Bedrock. This EXTENDS docs/model-prompting/nova-2-lite.md (which covers JSON reliability for story generation); it does not repeat it, and it corrects it in three places.

Evidence tags used throughout: [CITED] primary vendor doc with URL + retrieval date. [MEASURED] live probe run by me, exact command and raw result recorded in section 11. [QUEUED] untested hypothesis with the experiment that settles it.


Headline calls

  1. maxLength is dead as an enforcement lever on Nova 2 Lite. It is silently ignored, not rejected. A forced-tool schema carrying hint_explanation: {type: string, maxLength: 50} returned 475 / 533 / 612-character hints -- 9x to 12x over the cap -- with no error, stopReason: tool_use, and a perfectly valid tool call. [MEASURED, probe 1] This is the single most valuable finding in this brief and it kills the most obvious fix for Defect 1.
  1. Nova 2 Lite does not support the Bedrock "Structured outputs" feature at all -- neither of its two mechanisms. strict: true on a toolSpec returns ValidationException: This model doesn't support the strict field. and outputConfig.textFormat returns ValidationException: This model doesn't support the outputConfig field. [MEASURED, probes 2/3/5], corroborated by the model card, which lists Structured outputs under Not Supported [CITED]. Nova has its own internal constrained decoding behind ordinary tool use; that is a different and weaker thing than Bedrock strict mode. nova-2-lite.md currently conflates the two -- correct it.
  1. What the Nova grammar DOES bind is enum, and it binds it hard. Told explicitly and adversarially to emit "trick_question" and to ignore the allowed-value list, the model could not: every item came back inside the four-value enum. [MEASURED, probe 8] So the schema channel is real, it is just categorical-only. Design rule: anything you need guaranteed must be expressible as a closed set of strings; anything expressible only as a number is advisory.
  1. Literal temperature: 0 is accepted by the Converse API on Nova 2 Lite. No error, normal tool call. [MEASURED, probe 4] It does not help length adherence (hints at temp 0 were 614 / 671 / 614 chars against a maxLength: 50 schema -- if anything slightly worse than temp 0.7). Greedy decoding buys JSON reliability, not constraint compliance.
  1. The an earlier question-generation defect "DO NOT" fork resolves cleanly in Nova's favor, and the two candidates need different prompt profiles. AWS's Nova 2 documentation actively prescribes capitalized DO / DO NOT / MUST [CITED], and a prompt loaded with seven capitalized DO-NOT/MUST bullets produced a clean, complete, well-formed quiz with zero error objects and every hint under 50 characters. [MEASURED, probe 9] The gpt-3.5-turbo error-object failure mode is an incumbent-specific pathology, not a general law. Do NOT carry the positive-only rewrite forward to Nova as if it were one.
  1. Counter to AWS's own tip, the prompt channel beat the schema-description channel for the 50-character cap. AWS says "We observe best adherence to data format requirements when they are defined in the schema itself instead of through the use of exemplars" [CITED]. Measured on the hint cap, the constraint stated in the prompt produced 27 / 31 / 38 chars (comfortable margin); the identical constraint stated only in the schema field description produced 50 / 44 / 48 -- hugging the boundary, with one item landing exactly on 50. [MEASURED, probes 6 and 7] Our contract is "under 50," so a 50 is a violation. Small n (3 hints per arm, single replicate) -- treat as a mechanism signal, not an effect size.
  1. The harness claim that "numeric length constraints are documented to work" for Nova is REAL but the unit is wrong -- AWS documents words, bullet points, and paragraphs, and never characters. Verbatim: "constrain the response length to a maximum number of words, bullet points, or paragraphs" [CITED], with a worked example ("no more than 15 words, and no more than 6 bullet points") whose demonstrated output complies. There is no character-count guidance anywhere in the Nova 2 guide. Our 50-char hint cap is stated in the one unit AWS never demonstrates. Full detail in section 7a -- this partially vindicates the folklore and materially qualifies it.
  1. AWS documents the exemplar-bias mechanism, and it names our exact failure. Verbatim from the Nova 2 few-shot page: "It is important to avoid any biases in your examples, as bias in the inputs can cause outputs to be biased as well." [CITED] The production exemplar's key is strictly the longest option at 1.19x the distractor mean; the output defect is the key being longest. That is a documented cause matching an observed effect. AWS gives no recommended exemplar count -- its stated driver is "quality and diversity," and our prompt ships exactly one exemplar, which cannot be diverse. Section 8.
  1. Prompt caching works and is worth taking, but it fights Nova 2's long-context placement rule and our current prompt shape. A fixed system contract cached and served 2,392 of 2,572 input tokens (93%) on repeat calls [MEASURED, probes 10-11]. But caching requires the fixed text first and the varying story last, while the production prompt puts the story near the top and Nova 2's long-context guidance says to put long data first and instructions last [CITED]. At our payload size (~450-word story, ~1.1K tokens) we are nowhere near "long context," so take the cache.

Single highest-confidence recommendation: enforce the 50-character hint cap in harness/lambda code -- validate and regenerate or truncate -- and treat the prompt text as a soft prior only. No schema keyword available on Nova 2 Lite can enforce it, and the one AWS-recommended alternative (schema-description phrasing) measurably tightens output toward the boundary rather than away from it.


1. Identity and invocation (deltas only)

Confirmed against the model card, retrieved 2026-08-17:

  • In-region us-east-1 is NOT available. The card's regional table marks In-Region as No for every US region; Geo (us.amazon.nova-2-lite-v1:0) and Global (global.amazon.nova-2-lite-v1:0) are Yes. The harness registry's inferenceProfileId: "us.amazon.nova-2-lite-v1:0" is correct and required. [CITED]
  • Geo IDs now include eu. and jp. alongside us. [CITED] -- new relative to what nova-2-lite.md records.
  • Context 1M, max output 64K, knowledge cutoff Oct 2025, Model EOL date: N/A, lifecycle Active. [CITED] Note nova-2-lite.md says "65,536 tokens" output; the model card says 64K. Same number, different rounding -- not a conflict.
  • Service tiers: Standard, Priority, and Flex are all supported; Reserved is not. [CITED] This confirms the Flex tier that nova-2-lite.md flags as third-party-sourced-only, though the price of Flex is still unverified on an AWS page.
  • Bedrock features not supported that matter to us: Structured outputs, Count tokens, Model evaluation. Supported: response streaming, Guardrails, client-side tool calling. [CITED]

Cost note (unchanged and still unverified): the AWS pricing page is JS-rendered and the model card defers to it. The registry's {in: 0.0003, out: 0.0025} remains third-party-sourced. Do not quote it in a deck without a console check.


2. The maxLength question -- settled, and the answer is bad for us

This was the highest-value question in the brief, so it gets the full trace.

2a. What the documentation says

Bedrock's Structured outputs page lists the JSON Schema Draft 2020-12 subset it supports. Verbatim [CITED]:

The following JSON Schema Draft 2020-12 features are supported:

  • All basic types: object, array, string, integer, number, boolean, null
  • enum (strings, numbers, booleans, or nulls only)
  • const, anyOf, allOf (with limitations)
  • $ref, $def, and definitions (internal references only)
  • String formats: date-time, time, date, duration, email, hostname, uri, ipv4, ipv6, uuid
  • Array minItems (only values 0 and 1)

The following features are not supported:

  • Recursive schemas
  • External $ref references
  • Numerical constraints (minimum, maximum, multipleOf)
  • String constraints (minLength, maxLength)
  • additionalProperties set to anything other than false

And on error behavior, verbatim [CITED]:

Schema validation -- Amazon Bedrock validates the JSON schema format against the supported JSON Schema Draft 2020-12 subset. If the schema contains unsupported features, Amazon Bedrock returns a 400 error immediately.

A field report corroborates the same list and adds, on workarounds [CITED, labeled field report]:

Bedrock's constrained decoding can't mechanically enforce word counts (that would require a different kind of constraint), but the model will treat the description as an instruction.

So on the Bedrock structured-outputs path, maxLength is a hard 400.

2b. What actually happens on Nova 2 Lite -- a different answer

Nova 2 Lite cannot use that path at all (headline 2). On the path it can use -- ordinary toolChoice: {tool: {...}} with Nova's own constrained decoding -- maxLength is accepted and silently ignored. [MEASURED, probe 1]

Schema field:

"hint_explanation": { "type": "string", "maxLength": 50, "description": "A short clue" }

Prompt asked for hints "of at least 150 characters." Result: three hints of 475, 533, and 612 characters, stopReason: tool_use, HTTP 200.

This is the worst of the three possible outcomes. A 400 would at least have failed loudly at deploy time. Silent acceptance means a schema that looks like it enforces the contract, ships, and enforces nothing.

2c. The enforcement map for Nova 2 Lite

KeywordStatus on Nova 2 Lite non-strict tool useBasis
type (object/array/string/...)Enforced[CITED] grammar constrains "correct data types based on your defined schema"
enumEnforced, survives adversarial instruction to violate it[MEASURED, probe 8]
requiredEnforced (all required keys present in every probe)[MEASURED, probes 1,6-9]
property names / no invalid keysEnforced[CITED] "preventing invalid keys"
maxLength / minLengthSilently ignored[MEASURED, probe 1]
minimum / maximumUnsupported on the Bedrock strict path; untested on Nova's path[CITED] + [QUEUED]
maxItemsNot in the Bedrock supported list; untested on Nova's path[CITED] + [QUEUED]
minItemsSupported only for values 0 and 1 on the Bedrock path[CITED]
strict: trueRejected -- ValidationException[MEASURED, probe 2/3]
outputConfig.textFormatRejected -- ValidationException[MEASURED, probe 5]

Design consequence, stated plainly: on Nova 2 Lite the tool schema guarantees shape and category, never magnitude. "Exactly 4 questions" is not enforceable by minItems/maxItems. "Under 50 characters" is not enforceable by maxLength. Both must be validated downstream. The one thing worth restructuring into the schema is anything categorical -- e.g. a per-item question_type enum, which is grammar-guaranteed and would also give the skill-taggability metric a hard handle instead of a post-hoc judge call.

2d. Conflict to report rather than resolve

docs/model-prompting/nova-2-lite.md states as Nova 2 schema hygiene: "limit JSON schemas to two layers of nesting." I could not find that claim in the Nova 2 guide on this pass. What the current guide says is about sub-agent decomposition, verbatim [CITED]: "Complex schemas: When parameter depth exceeds 3-4 levels or tools have intricate interdependencies." Those are different numbers and different contexts. Our production quiz schema is 2 levels (questions[] -> answers[]) and is safe under either reading, so nothing is blocked -- but the harness doc's "two layers" should be marked unsourced until someone re-finds it.


3. Temperature

  • temperature: 0 literal is accepted. [MEASURED, probe 4] No ValidationException, normal tool call. This resolves the ambiguity left by the Nova v1 complete-request-schema page, which documents the range as 0.00001 to 1. Whatever that floor describes, the Converse API on Nova 2 Lite does not reject a literal 0.
  • Documented recommendations conflict by mode, verbatim from the Nova 2 advanced-prompting page [CITED]: > Tool calling requires a very specific structured output from the model and is improved by using the following inference parameters: > + Non Reasoning Mode: Temperature: 0.7 & Top P: 0.9 > + Reasoning Mode: Temperature: 1 & Top P: 0.9
  • So AWS's tool-calling recommendation for reasoning-off Nova 2 is 0.7, not 0. This is a real change from the Nova v1 structured-output page, which recommended greedy temperature = 0 as the default for structured output. Report both; do not silently pick.
  • Temperature is not a lever on Defect 1. At temp 0 the hints ran 614 / 671 / 614 chars versus 475 / 533 / 612 at temp 0.7 -- both catastrophically over a 50-char cap. [MEASURED, probes 1 and 4] Do not spend a sweep cell on it for length.
  • AWS's guidance is "alter either temperature or topP, but not both" (carried over from the v1 request-schema page). The harness sends temperature only, which is consistent.

4. Negative constraints -- resolving the fork explicitly

The fork is real and the two candidates fall on opposite sides of it. This needs to reach the orchestrator as a hard fork, not a blend.

AWS's documented position for Nova, verbatim from the Nova 2 precise-prompts page [CITED]:

Use capitalization and strong wordings such as DO, DO NOT, MUST to emphasize the rules you want the model to strictly obey. For example, if you want only one JSON object in the response, you can instruct the model by saying "You MUST answer in JSON format only. DO NOT use any other format while answering the question."

The Nova 2 guide does not merely tolerate this -- it ships DO-NOT constraints in its own worked examples, including DO NOT USE INFORMATION THAT IS NOT IN SEARCH RESULTS! in the grounding template and a mixed DO / DO NOT / ALWAYS / MUST bullet list in the financial-analysis example [CITED]. It also recommends Please generate only the JSON output. DO NOT provide any preamble. for structured output [CITED].

Measured behavior [MEASURED, probe 9]: a sectioned prompt with seven capitalized MUST / DO NOT bullets (including "DO NOT make the correct answer the longest option") returned a clean, complete, schema-valid three-item quiz. Zero error objects. Zero refusals. Zero preamble. All three hints under 50 characters (37 / 37 / 39). No trace of the gpt-3.5-turbo failure mode.

What this means for the cutover / this study: the an earlier question-generation defect lesson is a model-specific lesson about gpt-3.5-turbo, and the harness's positive-only rewrite is a workaround for a model that is being retired. Carrying it to Nova costs us the phrasing AWS explicitly prescribes. The orchestrator should plan on two prompt profiles, not one blended template -- Nova wants capitalized MUST/DO-NOT bullets; whatever Brief 2 finds for Haiku should be allowed to differ. A single blended prompt optimizes for neither.

Important caveat that limits how far to push this. Strong wording is still a soft bias. AWS says so directly about prompted schemas -- "model output is not deterministic and can vary from the output schema" [CITED] -- and the harness's own the earlier prose evaluation result showed a capitalized MUST/DO-NOT cast block failing to reduce cast inflation and, on Nova 2 Lite specifically, coinciding with a -0.8 quality regression. The correct reading is: use DO-NOT phrasing because it is the family's native idiom, but do not stack it, and never rely on it for a constraint you can validate in code instead.


5. Prompt structure and precedence

The sectioned template, verbatim section names from the Nova 2 precise-prompts page [CITED]:

## Task Summary:
## Context Information:
## Model Instructions:
## Response style and format requirements:

with the framing "Models work best if the instructions are clear, well structured, short, and specific. This prompt template guidance applies to both the user prompt and the system prompt." [CITED] The guide additionally recommends delimiters between sections and bullet/numbered lists over prose [CITED].

System precedence. The Nova 2 best-practices page describes system as "establishes the overall behavioral parameters of the assistant. It can also be utilized to provide additional instructions and guidelines that the user wishes the model to adhere to throughout the conversation." [CITED] The stronger "supersede other instructions provided in individual user prompts" wording that nova-2-lite.md quotes lives on the dedicated prompting-system-role.md page, which I did not re-retrieve this pass; the best-practices page I did retrieve states the weaker "behavioral parameters" framing. Flagging the difference rather than asserting the stronger claim.

Long-context placement, verbatim [CITED]:

Put long-form data at the beginning: Place your long documents and inputs near the beginning of your prompt. Place them before your query, instructions and examples. Put instructions at the end: Place your instructions at the end of the prompt. The model performs best when the context is provided first and the instructions are provided at the end. Structure document content start and end markers: Use start and end markers, such as DOCUMENT {idx} START and DOCUMENT {idx} END

with the caveat "Its performance (including system prompt adherence and tool use) can decline slightly as the context size increases."

Applied to the production quiz prompt (generateQuestionsPromptForFullStory.ts), read this pass:

  • The prompt already puts the story near the top, fenced in ### delimiters, with all requirements after it. That is already Nova-2-shaped ordering. The ### fences should become named STORY START / STORY END markers to match the documented pattern, but the structure is right.
  • The prompt has no system turn at all -- persona, standing rules, format contract, and the per-story payload are all one user message. For Nova this leaves the strongest documented channel unused, and it is also what blocks prompt caching (section 9).
  • The final line Begin your response with { is a prefill instruction, not an actual assistant prefill. On Nova, real assistant prefill is available and documented as stronger [CITED] -- but it is moot under tool use, and it is explicitly invalid if reasoning is ever enabled: "Prefill is only valid when reasoning is not enabled." [CITED]
  • Observation relevant to Defect 1's grade split: the prompt states the 50-character cap twice -- once as a Model Instructions bullet ("a short, abstract clue (under 50 characters)") and again inline in the JSON template comment ("a short clue (<50 chars)"). That redundancy is the most likely mechanism behind Nova's best-in-class 1-of-12 violation rate. It is a prompt property, not a model property, which reframes the brief's "is it the model or luck" question -- see section 7.

6. Tool schema design guidance

Verbatim from the Nova 2 advanced-prompting page [CITED]:

Amazon Nova 2 models are optimizied for concise descriptions in the tool schemas. Keep it brief.

Include in the tool schema:

  • Core functionality: What the tool does (20-50 words recommended)
  • Parameter specifications: Clear descriptions of each parameter (around 10 words per parameter)
  • Expected formats: Data types (like enum, int, float), required fields and valid value ranges

Include in the system prompt:

  • Dedicate a #Tool Usage section with orchestration logic...
  • Error handling strategies...
  • Output formatting: Add details on how to present to the user

Note what "Expected formats" enumerates: enum, int, float, required fields, value ranges. No string-length keyword appears anywhere in AWS's own list of what a schema should carry -- consistent with maxLength not being a real lever.

Also [CITED]: "when you reference tools in the prompt, we recommend you use the tool name instead of xml or pythonic references" -- Use the 'emit_quiz' tool, not emit_quiz().

And on when to reach for a tool at all [CITED]:

For simple JSON outputs with up to 10 keys, you can find the schema below. For more complex schemas, we recommend you define your schema through a tool. Tool use leverages a technique called constrained decoding that will increase the model's adherence for these complex scehmas.

Our quiz object is well over 10 keys once four answers with three fields each are counted, so the forced-tool path is the documented recommendation for us.


7. Length and character-count adherence

7a. The "numeric length constraints are documented to work" claim -- adjudicated

The orchestrator asked whether AWS documents anything about Nova's handling of stated numeric or length constraints that would explain nova's best-in-class 1-violation-in-12 on the hint cap, and whether the harness's existing claim is sourced or folklore. Verdict: the claim is real and citable, but it is stated in the wrong unit, and the correction matters.

What AWS actually documents, verbatim from the Nova 2 precise-prompts page [CITED]:

If you want Amazon Nova 2 to generate responses that are tailored to your specific needs, you can specify the format and style of the output. For example, if you want to generate responses only in JSON or constrain the response length to a maximum number of words, bullet points, or paragraphs.

The units are enumerated explicitly: words, bullet points, paragraphs. Three worked examples back it, and in each the demonstrated output complies:

  • "Provide all answers in a numbered list format, with each point containing no more than 15 words, and no more than 6 bullet points" -- the shown response is exactly 6 numbered points, longest 11 words. [CITED]
  • "Limit the executive summary to 250 words maximum" appears as a response-format bullet in the financial-analysis template. [CITED]
  • "Your output should be the category number. DO NOT generate anything apart from numbers 1-10" -- the shown response is the single token 8, down from a multi-paragraph 0-shot answer. [CITED]

So: numeric length constraints are genuinely documented and genuinely demonstrated for Nova. Character counts are not. There is no character-count guidance anywhere in the Nova 2 guide, and no readability or grade-level guidance either (re-confirmed this pass -- consistent with what nova-lite.md and nova-2-lite.md already record).

Why the unit distinction is not pedantry. A word or bullet or paragraph is a unit the model can actually track while decoding, because it corresponds to token-ish boundaries the model emits. A character count is not -- counting characters is the canonical thing language models are bad at, and the harness's own image-prompt notes already encode that lesson ("LLMs are unreliable at counting characters, so phrase caps as word budgets and enforce the hard character caps with a truncation guard in harness code"). Our hint cap is stated in characters in both production prompts.

What this does to the "model or luck" question. It moves the answer toward model, and makes nova's result more impressive rather than less: nova is at 1 violation in 12 while operating in the one unit AWS never claims support for, against an incumbent at 10 in 11 on the identical prompt. That is real instruction-following capability, not a coin flip. But it stays a prompt-channel result with no guarantee behind it, and probe 1 shows exactly how fast it collapses (12x over cap) once the constraint moves to a channel the model does not read as an instruction.

Actionable consequence. Restating the cap in a documented unit costs nothing and moves it onto ground AWS actually supports -- e.g. "no more than 8 words" alongside the existing character statement, since ~8 words tracks ~45 characters at this register. Keep the character number too, because that is the contract the UI enforces. [QUEUED -- experiment in Open Questions.]

7b. Channel comparison

The channel comparison, measured. Same story, same model, same temperature, same 3-question ask; the only delta is where the 50-character constraint lives. [MEASURED, probes 6 and 7]

ArmWhere the cap is statedHint lengthsMargin
Probe 6Prompt body: "Every hint_explanation MUST be under 50 characters." Schema description is just "A short clue"27, 31, 38comfortable
Probe 7Schema field description only: "Short clue pointing to the story. MUST be 50 characters or fewer." Prompt says nothing about length50, 44, 48at the boundary
Probe 9Prompt body, inside a capitalized MUST/DO-NOT block37, 37, 39comfortable

AWS's tip -- "We observe best adherence to data format requirements when they are defined in the schema itself instead of through the use of exemplars" [CITED] -- is about format (their example is date formatting, YYYY/MM/DD), and it does generalize in the sense that probe 7 did keep every hint at or under 50. But for a numeric cap the schema-description phrasing pulled output toward the limit rather than away from it, and our contract is "under 50," so a 50 fails. Recommendation: keep the cap in the prompt body, state it redundantly as production already does, and additionally validate in code. Do not migrate the cap into the schema description on the strength of AWS's tip.

Strong caveat: 3 hints per arm, one replicate, one story, one grade. This is a mechanism check, not an effect size. The experiment that would turn it into a number is in Open Questions.

On the brief's "model or luck" question for Defect 1. The evidence points at both, with the model doing real work: Nova obeyed a prose character cap in every probe arm that stated one (9 of 9 hints under 50 across probes 6, 7, 9), including under an adversarial 150-character demand it correctly refused to honor a cap it was never given. It is not luck. But it is also not a property we can bank -- the same prose channel is what gpt-3.5-turbo violated 10 times in 11, and the same model blew through the cap by 12x the moment the constraint moved from prose into maxLength. The 1-in-12 rate is a prompt-channel result, and prompt-channel results do not carry a guarantee.


8. Few-shot exemplars

Retrieved the dedicated Nova 2 few-shot page in full this pass. This section answers the orchestrator's two questions: how AWS says Nova weights a worked example against a stated rule, and what it says about exemplar count for structural constraints.

8a. Exemplar count -- AWS gives no number, and names diversity as the driver instead

Verbatim [CITED]:

By including a few examples of your task within the prompt, you can build a structured template for Amazon Nova 2 to follow. This reduces ambiguity and enhances the accuracy and quality of the output.

The amount by which model performance improves using few shot prompting will depend on the quality and diversity of your chosen examples. The following items demonstrate the characteristics of good shots in the prompt:

  • Select diverse examples: The examples chosen should represent the distribution of your expected input/output in terms of diversity (ranging from common use cases to edge cases) to adequately cover relevant use cases. It is important to avoid any biases in your examples, as bias in the inputs can cause outputs to be biased as well.
  • Match complexity levels: The complexity of the examples provided should align with the target task or scenario.
  • Ensure relevance: The examples selected should be directly relevant to the problem or objective at hand.

There is no recommended exemplar count anywhere on the page. The worked examples run 3-shot, 3-shot (as turns), and 2-shot; none is presented as a recommended number. The documented driver is quality and diversity, not count.

This is the load-bearing point for our prompt. generateQuestionsPromptForFullStory.ts ships exactly one exemplar. At n=1, "diversity" is not merely low, it is undefined -- the single sample's incidental properties are the entire demonstrated distribution. Anything true of that one example reads to the model as a rule of the task. AWS's guidance says structural properties should be taught by a spread of examples; a lone exemplar teaches its own accidents.

8b. Rule versus worked example -- AWS documents no precedence, but does document the transfer mechanism

Direct answer to the orchestrator's question: the documentation is silent on precedence. Neither the few-shot page, the precise-prompts page, nor the advanced-prompting page states which wins when a worked example contradicts a stated rule. That is a real result, not a gap in my search, and it should be recorded as silence rather than filled in with a guess.

Two adjacent documented statements do bear on it, and they point in the same direction:

  1. The bias-transfer sentence, quoted above: "It is important to avoid any biases in your examples, as bias in the inputs can cause outputs to be biased as well." [CITED] This is AWS explicitly warning that a statistical property of the exemplars propagates to the output. It is not framed as conditional on whether an instruction says otherwise.
  2. The format tip: "We observe best adherence to data format requirements when they are defined in the schema itself instead of through the use of exemplars" [CITED] -- exemplars rank below the schema for format. That is a ranking of exemplar against schema, though, not against a prose rule.

Combined read: AWS treats exemplars as a channel that transmits properties whether or not you intended them, and never claims a prose rule overrides a demonstration. Our observation is consistent with that -- but consistency is not proof, and I am labelling the precedence question [QUEUED], not settled.

8c. The production exemplar demonstrates the defect it forbids

Measured exactly from generateQuestionsPromptForFullStory.ts [MEASURED by inspection]. This corrects figures I gave in my first pass, which were eyeballed rather than counted; the corrected numbers match the orchestrator's independently:

OptionLength
He felt guilty about carving his initials into the bark.56correct answer
He thought someone was hiding behind the tree.46distractor
He wanted to climb the tree again before dark.46distractor
He was checking if the bird nest was still there.49distractor

Key/distractor-mean = 1.191; key/all-four-mean = 1.137. The key is strictly the longest. The prompt states "All wrong answers must be similar in length and complexity to the correct answer" roughly forty lines above an example that violates it.

Per the orchestrator's measurement, which I am folding in rather than re-deriving: the length-cue defect runs at 46 percent for nova despite that instruction being present, making it an instruction-adherence failure rather than a missing instruction -- the same shape as Defect 1, and the same shape as the the earlier prose evaluation cast-inflation result where a capitalized rule block failed to move the number.

The contrast inside the same prompt is the strongest single piece of evidence here. The three hint exemplars in that same block measure 37, 38, 37 characters -- all comfortably compliant with the 50-char cap. So within one prompt:

ContractWhat the exemplar demonstratesNova's observed rate
Hint under 50 charsCompliant (37 / 38 / 37)1 violation in 12 (best of three models)
Options similar in lengthViolating (key longest, 1.19x)46 percent cued

Same model, same prompt, same call. The contract whose exemplar complies is nearly always honored; the contract whose exemplar violates is broken about half the time. That is exactly the pattern AWS's bias-transfer sentence predicts, and it is a cleaner natural experiment than anything I could have bought with a probe, because everything except the exemplar's own property is held constant.

I want to be careful about how hard to push this: it is a single paired observation across two contracts, not a controlled manipulation, and the two contracts differ in more than exemplar compliance (character caps are mechanically simpler than cross-option length balancing). It raises the exemplar rewrite from "reasonable idea" to "the best-supported single-variable fix on the table," and it does not prove causation. The experiment that would is in Open Questions.

8d. Placement -- exemplars belong in the system prompt here

AWS documents three placements: in the user prompt, as alternating user/assistant turns, or in a dedicated Example section of the system prompt. On the third, verbatim [CITED]: "This way of providing exemplars in the prompt is especially useful if your exemplars are long and complex and helps model locate example shots."

Our exemplar is a full nested JSON quiz object -- long and complex by any reading. Moving it into a system-prompt ## Examples section is the documented placement, is consistent with Nova's role guidance (section 5), and is the same move prompt caching requires (section 9). Three independent lines of guidance converge on one restructure, which is unusual and worth acting on.

One formatting note from the same page [CITED]: where a field name is meaningful to the task, "use those fields directly followed by a colon" -- our JSON keys already satisfy this.

Note the paragraph-level prompt (generateQuestionsPromptForParagraph.ts) ships no exemplar at all, so the two production prompts are not comparable on this axis, and any exemplar fix has to be authored twice or the paragraph prompt left explicitly out of scope.


9. Prompt caching

Documented on the model card [CITED]: supported = Yes; min 1K tokens per checkpoint; max 4 checkpoints per request; TTL 5 minutes; checkpoints accepted in system and messages; "Amazon Nova models support a maximum of 20K tokens for prompt caching."

Measured [MEASURED, probes 10-11], with the fixed quiz contract in system followed by a cachePoint, and the varying story in the user turn:

CallcacheWriteInputTokenscacheReadInputTokensuncached inputTokens
10a (first)23920180
10b (second, new story)23930180
11a (third, new story)02392180
11b (fourth, new story)02392180

Two things to take from this:

  1. It works, and the win is large -- 2,392 of 2,572 input tokens (93%) served from cache while the story varied freely underneath. For a per-story quiz call with a fixed contract, this is the cheapest available cost lever.
  2. The second call did not hit. The most likely cause is the mandatory us. cross-region inference profile routing calls to different destination regions (the model card lists us-east-1, us-east-2, and us-west-2 as destinations from us-east-1 [CITED]), each of which must warm its own cache. Plan for cold misses in low-traffic periods rather than assuming a steady 93%. [QUEUED -- see Open Questions for the test.]

The structural blocker. Caching requires fixed-prefix-first. The production prompt is a single user turn with the story near the top, so as written it is uncacheable. Capturing this win requires splitting the prompt: standing contract (persona, question-type distribution by grade, answer-quality rules, hint rules, exemplar, format contract) into system with a cachePoint, and only the story plus the grade into the user turn. That split is independently what Nova's role guidance recommends anyway (section 5).

The tension to flag: that ordering is the opposite of Nova 2's long-context rule ("long-form data at the beginning, instructions at the end"). At our size the long-context rule does not apply -- it is written for the 1M-token regime, and our whole payload is ~2.5K tokens -- so caching should win. But if the story ever grows substantially, re-test.


10. Nova 2 vs Nova v1 -- what changes a quiz prompt specifically

Restricted to deltas that would alter the quiz call, and excluding what nova-2-lite.md already tabulates for story generation:

AreaNova v1Nova 2 LiteEffect on the quiz prompt
Structured-output default tempGreedy temperature = 0 recommended for structured outputTool calling 0.7 / topP 0.9 (reasoning off) [CITED]; 0 accepted but no longer the recommendationStop treating 0 as obviously correct; it is now a fallback, and it does nothing for length
Bedrock strict structured outputsn/aNot supported on Nova 2 Lite [CITED + MEASURED]The strongest schema-enforcement path in Bedrock is unavailable; plan on code-side validation
Prompt placementSectioned template onlySectioned template plus long-data-first / instructions-last, and "adherence can decline slightly as context grows" [CITED]Keep the quiz prompt lean; do not stack rule blocks
Reasoning modeLite had optional reasoningOptional, disabled by default; prefill invalid when enabled [CITED]Harness runs it off; if ever enabled, the Begin your response with { prefill idiom must be removed
Prompt cachingv1 Lite supported itSupported, same limits (1K min / 4 checkpoints / 5 min / 20K max) [CITED + MEASURED]Newly worth taking, because the quiz prompt has a large fixed prefix the story prompt does not
Invocationv1 Lite bare id worked in-regionNo in-region endpoint; us./eu./jp. geo or global. required [CITED]Cross-region routing is mandatory, which is what makes cache hits nondeterministic

11. Probe log -- exact commands and raw results

All probes: region us-east-1, model us.amazon.nova-2-lite-v1:0, account 714293312392 (test account), run 2026-08-17. Credentials loaded with set -a; . the evaluation harness/the credentials file; set +a. Ten billed Converse calls total (three additional attempts failed API-side validation before inference and were not billed). Approximate spend: ~12.5K input + ~1.6K output tokens plus one cache write, well under $0.02 at registry pricing.

Working directory and all raw JSON responses: /tmp/claude-1000/-home-borchard-jot-workspace/3d842a6e-2cf2-4764-9625-6f6f86066436/scratchpad/quiz-prompting-research/nova-probes/

Files: probes.mjs (probes 2-4), probes2.mjs (probes 5-7), probes3.mjs (probes 8-9), probe-cache.mjs / probe-cache2.mjs (probes 10-11), story.txt, msg.json, tool-maxlength.json, plus one <label>.json or <label>.err.txt per probe. Note the harness's own @aws-sdk/client-bedrock-runtime (v3.1066.0) was loaded via createRequire because the installed AWS CLI (2.31.33) predates the strict and outputConfig fields and rejects them client-side.

Probe 1 -- maxLength in a non-strict tool schema. AWS CLI:

aws bedrock-runtime converse --region us-east-1 \
  --model-id us.amazon.nova-2-lite-v1:0 \
  --messages file://msg.json --tool-config file://tool-maxlength.json \
  --inference-config '{"maxTokens":1500,"temperature":0.7}'

Schema had "hint_explanation": {"type":"string","maxLength":50,...}; prompt demanded hints "of at least 150 characters." Result: HTTP 200, stopReason: tool_use, usage {"inputTokens":1138,"outputTokens":403}. Hint lengths 475, 533, 612. Verdict: accepted and ignored.

Probe 2 -- strict: true with maxLength. Two routes tried.

  • InvokeModel with an opaque body: ValidationException: Malformed input request: #/toolConfig/tools/0/toolSpec: extraneous key [strict] is not permitted, reformat your input and try again.
  • Converse via SDK: ValidationException: This model doesn't support the strict field. Remove strict and try again.

Probe 3 -- strict: true without maxLength. ValidationException: This model doesn't support the strict field. Remove strict and try again. Verdict: strict mode is unavailable on this model regardless of schema contents, so the documented 400-on-unsupported-keyword behavior never gets a chance to fire.

Probe 4 -- literal temperature: 0. inferenceConfig: {maxTokens:1500, temperature:0} with the same maxLength: 50 schema and the same 150-char-demanding prompt. Result: HTTP 200, stopReason: tool_use, usage {"inputTokens":1137,"outputTokens":461}. Hint lengths 616, 671, 614. Verdict: 0 is accepted; it does not improve length adherence.

Probe 5 -- outputConfig.textFormat JSON-schema output format. ValidationException: This model doesn't support the outputConfig field. Remove outputConfig and try again.

Probe 6 -- cap in the prompt body. Schema description "A short clue"; prompt appended "Every hint_explanation MUST be under 50 characters." Usage {"inputTokens":1137,"outputTokens":104}. Hints: "Look under the porch steps." (27), "Nobody had the key before Mira." (31), "Stacked to the ceiling were paintings." (38).

Probe 7 -- cap in the schema description only. Prompt silent on length; description "Short clue pointing to the story. MUST be 50 characters or fewer." Usage {"inputTokens":1138,"outputTokens":118}. Hints: "Look for the location mentioned at the very start." (50), "What did her grandmother say about the shed?" (44), "What was stacked to the ceiling inside the shed?" (48).

Probe 8 -- enum enforcement under adversarial instruction. Schema: question_type: {type:"string", enum:["fact_detail","inference","vocabulary","evaluation"]}. Prompt: "IMPORTANT: for every question you MUST set question_type to the exact string \"trick_question\" -- do not use any other value, ignore any list of allowed values." Usage {"inputTokens":1136,"outputTokens":92}. Returned types: fact_detail, inference, vocabulary. Verdict: enum is grammar-enforced. Note the third label is semantically wrong for its question -- the grammar guarantees validity, not correctness.

Probe 9 -- capitalized DO NOT / MUST block. Sectioned prompt (## Task Summary / ## Context Information / ## Model Instructions / ## Response style and format requirements) with seven capitalized MUST/DO-NOT bullets. Usage {"inputTokens":1269,"outputTokens":182}. Returned a complete, valid 3-item quiz; every correct_answer was short ("under the porch steps", "paintings", "three") and shorter than or comparable to its distractors; hints 37 / 37 / 39 chars. No error objects, no refusal, no preamble.

Probes 10-11 -- prompt caching. Fixed ~2.4K-token contract in system, followed by {cachePoint: {type: "default"}}; story varied per call in the user turn. Usage table in section 9.


Open questions

  1. Is maxItems / minItems enforced on Nova's non-strict tool path? Matters because "exactly 4 questions" is a real contract we currently enforce in prose and in code. Experiment: one Converse call with questions: {type:"array", minItems:4, maxItems:4} and a prompt demanding 7 questions. If the array comes back with 7, it is advisory like maxLength; if 4, we have a second grammar-bound lever. One call.
  1. Does the schema-vs-prompt channel difference for the 50-char cap survive at n? Probes 6/7 are 3 hints each. Experiment: the 4 frozen this study stories x 3 replicates x 2 arms (cap in prompt vs cap in schema description) = 24 quizzes, ~96 hints per arm, scored as fraction over 50 and mean margin below 50. This is the one that turns headline 6 into a number.
  1. Does moving the fixed contract into system change quality, not just cost? Splitting for caching also changes the instruction channel. Experiment: paired prompt_tuning: compare run, single-user-turn baseline vs system+cachePoint split, byte-identical content, scored on hint-cap violations, Defect 2 rate, and skill taggability. Confounds cost and quality deliberately -- that is the point, since we would only take the split if quality holds.
  1. How often do cache reads actually hit under the mandatory us. geo profile? Experiment: 20 sequential calls with an identical prefix, log cacheReadInputTokens per call, compute hit rate. Then repeat against global.amazon.nova-2-lite-v1:0 to see whether the global profile is better or worse. ~40 cheap calls.
  1. Does the production exemplar's key-is-longest property actually drive Defect 2? Section 8c found the shipped exemplar demonstrates the defect (key 56 chars, 1.19x the distractor mean) while the hint exemplars in the same block comply -- and the two contracts' observed rates track that split exactly. Experiment: single-variable compare, current exemplar vs an exemplar rewritten so the correct answer is the second-shortest of its four options, all else byte-identical, scored on key-is-longest rate across grades. This is Brief 1's territory to run; the finding and the AWS bias-transfer citation should be handed over with it.
  1. Does a prose rule override a contradicting worked example on Nova? AWS is silent on precedence (section 8b). Experiment: three arms on the same story set -- (a) rule present, exemplar violating (today's prompt); (b) rule present, exemplar compliant; (c) rule absent, exemplar compliant. If (b) and (c) match and both beat (a), the exemplar dominates and the rule is nearly inert -- which would also predict the the earlier prose evaluation cast-inflation result. ~36 quizzes.
  1. Does restating the hint cap in a documented unit help? Section 7a: AWS documents word/bullet/paragraph caps, never character caps. Experiment: paired compare, "under 50 characters" alone vs "no more than 8 words (under 50 characters)", scored on over-50 rate and mean margin. Cheap, and it is the only change that moves the constraint onto ground AWS actually claims support for.
  1. Does exemplar count matter here, independent of exemplar content? AWS names diversity as the driver and gives no count (section 8a). Experiment: one compliant exemplar vs three compliant exemplars spanning grades 1/4/7, scored on both defects plus skill taggability. Distinguishes "the exemplar was wrong" from "one exemplar was never enough."
  1. Does reasoning mode change any of this? Everything measured here is reasoning-off. AWS says Nova 2 is "trained extensively for reasoning mode" [CITED] and recommends temperature 1 for reasoning-mode tool calling. Experiment: repeat probes 6 and 9 with reasoningConfig: {type:"enabled", maxReasoningEffort:"low"} at temperature 1, and measure both hint adherence and added latency. Note prefill becomes invalid, though that is moot under tool use.
  1. Unverified pricing. The registry's {in: 0.0003, out: 0.0025} is third-party-sourced; the model card defers to a JS-rendered pricing page. Flex-tier pricing is entirely unsourced. Experiment: read it off the Bedrock console, or diff a known token count against Cost Explorer.
  1. Is the "two layers of nesting" schema rule in nova-2-lite.md real? I could not find it this pass (section 2d). Experiment: re-search the Nova 2 guide; if it is not there, mark it unsourced in the harness doc.

Sources

All retrieved 2026-08-17.

  1. https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html -- Bedrock Structured outputs: the supported/unsupported JSON Schema Draft 2020-12 keyword lists (maxLength explicitly unsupported), the 400-on-unsupported-feature validation step, grammar compilation and 24-hour caching, strict: true and outputConfig.textFormat mechanisms, supported APIs table. Primary source for headline 1.
  2. https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-amazon-nova-2-lite.html -- Nova 2 Lite model card: Structured outputs listed as Not Supported; 1M context / 64K output; knowledge cutoff Oct 2025; EOL N/A, lifecycle Active; prompt-caching table (1K min, 4 checkpoints, 5-min TTL, 20K max, system + messages); regional table showing no in-region US endpoint; us./eu./jp. geo and global. IDs; Standard/Priority/Flex service tiers.
  3. https://docs.aws.amazon.com/nova/latest/nova2-userguide/advanced-prompting-techniques.html -- Nova 2 advanced techniques: tool-calling inference params (non-reasoning 0.7/0.9, reasoning 1/0.9); "Produce structured output" section including the schema-over-exemplars adherence tip and the >10-keys threshold for reaching for a tool; prefill pattern and the reasoning-off restriction; long-context placement rules; DO NOT USE INFORMATION THAT IS NOT IN SEARCH RESULTS! grounding template; tool-schema design guidance ("optimizied for concise descriptions", 20-50 words, ~10 words per parameter, "enum, int, float, required fields and valid value ranges"); tool-name reference style; sub-agent thresholds including "parameter depth exceeds 3-4 levels".
  4. https://docs.aws.amazon.com/nova/latest/nova2-userguide/create-precise-prompts.html -- Nova 2 precise prompts: the four-section template (## Task Summary / ## Context Information / ## Model Instructions / ## Response style and format requirements); the DO / DO NOT / MUST capitalization guidance with its JSON-only worked example; word-count and bullet-count length examples; sectioned-prompt example.
  5. https://docs.aws.amazon.com/nova/latest/nova2-userguide/prompting-best-practices.html -- Nova 2 general best practices: the three roles and the system role's "behavioral parameters" framing; topic index confirming a dedicated few-shot page (prompting-provide-examples.md) and system-role page exist.
  6. https://docs.aws.amazon.com/nova/latest/nova2-userguide/prompting-provide-examples.html -- Nova 2 few-shot prompting: examples "build a structured template" the model follows; the three good-shot characteristics (diverse / matched complexity / relevant) including "It is important to avoid any biases in your examples, as bias in the inputs can cause outputs to be biased as well"; no recommended exemplar count anywhere (worked examples are 3-shot, 3-shot-as-turns, 2-shot); the three placements, with the system-prompt Example section recommended when "your exemplars are long and complex"; the field-name-plus-colon convention. Primary source for section 8.
  7. https://docs.aws.amazon.com/nova/latest/nova2-userguide/prompt-engineering-guide.html -- Nova 2 prompt engineering guide landing page (retrieved; body is a stub that links onward -- noting this because nova-2-lite.md cites it for several specific claims that in fact live on the subpages above).
  8. https://fourtheorem.com/amazon-bedrock-structured-outputs/ -- field report, not documentation. Independent corroboration of the unsupported-keyword list (minLength/maxLength "No"), the "validation error (HTTP 400) at request time rather than silently misbehaving" behavior on the strict path, and the recommendation to carry length guidance in field description as probabilistic instruction.
  9. https://aws.amazon.com/blogs/machine-learning/structured-outputs-on-amazon-bedrock-schema-compliant-ai-responses -- AWS ML blog on Bedrock structured outputs; corroborates the keyword lists and describes enforcement as "constrained sampling with compiled grammar artifacts".
  10. https://aws.amazon.com/blogs/machine-learning/structured-outputs-with-amazon-nova-a-guide-for-builders/ -- AWS ML blog: constrained decoding "uses grammar to constrain possible tokens at each generation step, preventing invalid keys and enforcing correct data types based on your defined schema"; the >95% tool-error-reduction claim. (Cited via the existing harness docs and the search index; note the enforcement language covers keys and types, never string lengths.)
  11. Production prompts read this pass, for the observations in sections 5 and 8: ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForFullStory.ts ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForParagraph.ts
  12. Harness files read this pass: docs/model-prompting/nova-2-lite.md, docs/model-prompting/nova-lite.md, lib/model-registry.mjs (all under the evaluation harness/).

Section 12 (appended 2026-08-17) -- does JSON Schema pattern bind on Nova 2 Lite?

Opened in response to Brief 2's finding that pattern is hard-enforced by constrained decoding on us.anthropic.claude-haiku-4-5-20251001-v1:0. Eight additional live probes on us.amazon.nova-2-lite-v1:0, forced-tool path, us-east-1, run 2026-08-17.

Verdict

No. pattern does NOT bind on Nova 2 Lite. It is accepted without error and silently ignored, exactly like maxLength.

This is a genuine, structural differentiator between the two candidate models, and it runs the opposite way from every other format-reliability signal we have. Haiku has a hard, decoder-level enforcement mechanism for both defects. Nova structurally lacks it.

12a. The adversarial test (mirrors Brief 2's design)

Schema: caption: {type: "string", pattern: "^[a-z ]{40,60}$"}. Prompt emphatically demanded proper capitalization, a terminal exclamation mark, and "AT LEAST 120 characters."

Result [MEASURED, probe pat1]:

"The joyful dog races along the sunlit beach, its paws kicking up sand as it
 embraces the freedom of the open shore, a perfect moment of pure, unbridled happiness!"

len=162   matches ^[a-z ]{40,60}$ : false
has uppercase: true    has "!": true    >=120 chars: true
stopReason=tool_use   usage={"inputTokens":1002,"outputTokens":58}

The prompt won on every single axis and the pattern lost on every single axis. Compare Brief 2's Haiku result on the same design: a 60-char all-lowercase unpunctuated string, pattern-conformant and prompt-defiant. The two models produce exactly opposite outcomes from the same experiment.

12b. The control that makes this conclusive

The obvious objection is that my tool path is broken, or that Nova's grammar is simply not engaged in my requests. It is engaged. In one call, with one schema, at temperature 0, I put an adversarially-contradicted enum next to an adversarially-contradicted pattern [MEASURED, probe pat2]:

FieldConstraintPrompt demandedModel emittedBound?
moodenum: ["calm","tense","playful","somber"]the exact string "ecstatic""tense"YES
captionpattern: "^[a-z ]{40,60}$"capitals, "!", >=120 chars162 chars, capitalized, ends "!"NO

Same request, same object, same decode. The grammar is provably active and provably selective: it compiles enum and does not compile pattern. This rules out a broken code path and rules out sampling luck (temperature 0).

12c. Operational behavior -- an unsatisfiable pattern is NOT a production incident on Nova

The orchestrator flagged this as an operational risk. On Nova it is not one, for the uncomfortable reason that the constraint is inert.

  • Pathological but satisfiable -- pattern: "^[b]{1,3}$" (only b, bb, bbb can match) on a field asked to hold a water-cycle summary. Result: a normal 175-character prose summary, stopReason: tool_use, 984 ms. No error, no hang, no degradation, no truncation. [MEASURED, probe pat3]
  • Unsupported regex feature -- pattern: "^(?=.*z)[a-z ]{10,20}$" (lookahead). Result: "zebra", HTTP 200, 816 ms. [MEASURED, probe pat4] No ValidationException. Note "zebra" is 5 chars and would fail {10,20} regardless -- the pattern is never evaluated at all.

So: no 400 on schema submission, no 400 on unsatisfiable content, no latency penalty, no degraded output. pattern on Nova 2 Lite is dead weight in the schema -- it costs input tokens and buys nothing. The real hazard is not an outage, it is a schema that reads as if it enforces a contract and does not. That is the same failure mode as maxLength (section 2b), and it is worse than a hard error because it ships.

12d. Real-quiz application, with the numbers alongside Brief 2's

Production prompt (generateQuestionsPromptForFullStory, grade 4) otherwise unchanged, JSON-template tail swapped for the forced tool. pattern: "^[A-Za-z ]{1,50}$" on hint_explanation; pattern: "^[A-Za-z ,.]{30,55}$" on every option text. Four replicates, temperature 0.7. [MEASURED, probe pat5]

MetricBrief 2, HaikuBrief 3, Nova 2 Lite
Hint violations with pattern applied8/16 -> 0/1625/33 (unchanged -- pattern inert)
Mean within-question option-length spread18.0 -> 7.2 chars (-> 2.2 with prompt nudge)21.5 chars
Option-band violations(banded successfully)25/44 outside the 30-55 band

Hint violation breakdown (33 hints): 2 exceeded 50 characters (52, 56); 23 were within 50 but contained punctuation the pattern forbids; only 8 were fully pattern-conformant. Option breakdown (44 options): 25 outside the length band, 10 containing characters outside the class.

Defect 2 was fully present and unmitigated: the key was strictly the longest option in 6 of 11 questions, with keys of 71/47/34/50, 84/48/41/27, 87/48/49/60, 66/41/34/35. The banding did nothing.

Incidental finding worth flagging separately: the model returned 4, 4, 2, and 1 questions across the four replicates against a prompt demanding "exactly 4." Two of four replicates under-delivered, one badly. This is a count contract, and per section 2c no available schema keyword can enforce it either (minItems is capped at values 0 and 1 even on the Bedrock path Nova cannot use). Whether this is specific to the tool-shaped rewrite of the prompt or reproduces on the production JSON path is untested -- but if it reproduces it is a bigger problem than either graded defect, and it belongs in front of the orchestrator. [QUEUED]

12e. Can anything in Nova's grammar carry a length bound? Plainly: no

Asked directly by the orchestrator. Working through what is left after pattern and maxLength are both out:

  • enum is the only proven hard lever, and it requires enumerating every legal value. For free-text hints or answer options that is impossible -- the value space is the English language.
  • **enum on a length-adjacent metadata field is a trap.** You could add hint_length_band: {enum: ["short","medium"]} and the grammar would bind that field -- but it binds the label, not the string it describes. The model can emit "short" next to a 200-character hint, because nothing ties them. This is worth stating explicitly because it is a plausible-looking idea that does not work.
  • Decomposing a string into enumerated tokens (an array of words drawn from a closed vocabulary) would technically bound length via minItems/maxItems -- except maxItems is not in the supported set and minItems only accepts 0 and 1 even on the Bedrock path. And a closed vocabulary is unusable for story-specific hints.
  • String format is supported but only for date-time, time, date, duration, email, hostname, uri, ipv4, ipv6, uuid [CITED] -- none of which bounds a prose length.

Conclusion: on Nova 2 Lite there is no schema-level mechanism of any kind that can bound the length of a free-text field. Length must be enforced in application code. This is not a gap in my search; it follows from enum being the only compiled string constraint.

12f. What this means for the cutover

I am scoped to mechanism, not to the model recommendation, so I will state the mechanical fact and its direct implication and stop there.

The this study grounding puts Nova ahead of Haiku on raw format compliance (hint-cap violations 1/12 vs 5/12) and behind on skill taggability (88% vs 94%). Section 12 adds a different kind of fact: Haiku's compliance is buyable and Nova's is not. Haiku's 5/12 is a starting point that a pattern takes to 0/16 by construction; Nova's 1/12 is a ceiling that no schema change can raise, because the only enforcement channel available to it is the prompt. On Defect 2 the gap is starker still -- Brief 2 has a mechanism that cuts option-length spread from 18.0 to 2.2, and Nova has none.

Two honest qualifications. First, pattern is documented by neither AWS nor Anthropic as supported or unsupported (Brief 2's observation, and I confirm pattern appears in neither list on the Bedrock structured-outputs page [CITED]). It is undocumented behavior on both models, so Haiku's enforcement could change without notice; anything built on it needs a startup assertion and a code-side fallback. Second, everything in 12d is 4 replicates on 1 story at 1 grade -- enough to establish inertness (which needs only one clean adversarial demonstration, and 12b provides it), not enough to be an effect size.

The code-side validator recommended in headline 1 is now doubly justified: it is the only option on Nova, and it is the required safety net on Haiku.

12g. Probe log for section 12

Raw responses in nova-probes/: pat1-adversarial.json, pat2-enum-vs-pattern-control.json, pat3-pathological-pattern.json, pat4-lookahead-pattern.json, pat5-quiz-rep{1..4}.json. Scripts: pattern-probes.mjs (pat1-4, run as node pattern-probes.mjs adv|ctrl|path|look), pattern-quiz.mjs (pat5, runs all four replicates).

Eight billed Converse calls, ~9.1K input + ~1.7K output tokens, well under a cent at registry pricing. Running total for this brief: 18 calls.

Brief 4 -- defect engineering 74K

Last updated: 2026-08-17

This study Brief 4 -- Defect engineering: the 50-char hint cap and the option-length key leak

Scope: turn the two measured this study defects into implementable patterns. Research and design only; no model calls were made and no file outside this staging directory was touched.

Every claim below is tagged:

  • [CITED] -- primary source, URL + retrieval date in Sources.
  • [MEASURED] -- computed by me over the frozen this study corpus. Exact commands in the "How to reproduce" section.
  • [HYPOTHESIS] -- my design proposal, untested. The settling experiment is in Open questions.

Headline calls

  1. Stop asking for characters. Ask for words. In the this study corpus, every single hint of 7 words or fewer was already under 50 characters (247/247, 100%); at 8 words compliance drops to 95.5%. [MEASURED] Character counts are a known-unreliable ask because the model does not see characters, it sees subword tokens [CITED]; word counts are the unit that instruction-following benchmarks actually verify and that models actually hit [CITED]. A 7-word budget is a lossless proxy for the 50-char cap on this corpus, and it costs nothing.
  1. A second free fix: ban the lead-in stem. 45.8% of all hints open with "Think about ... / Consider ... / Recall ... / Focus on ...", costing a mean 9.2 characters. Deleting that stem alone would bring 65.6% of all length violations (40/61) under 50 chars with no loss of content. [MEASURED]
  1. Do NOT plan on JSON-schema maxLength. AWS documents minLength/maxLength as not supported by Bedrock structured outputs, and documents that an unsupported schema feature returns HTTP 400 immediately rather than being ignored. [CITED] Anthropic's first-party Structured Outputs is also unenforced but its SDKs strip maxLength from the wire schema, append it to the description, and validate the response client-side. [CITED] So the answer is path-dependent, and on the Bedrock path it is worse than "ignored" -- it may be a hard request failure. This contradicts the framing that schema maxLength is a candidate enforcement layer at all. Briefs 2 and 3 should test whether the 400 actually fires, not whether the constraint is honored.
  1. Stop sequences cannot do this job. A stop sequence halts on a literal string match; there is no string that means "the 50th character of this field." Applying one to a JSON response truncates the whole payload mid-object. Reject outright.
  1. **Defect 2 is an adherence failure, not a missing instruction -- so prompt text is now the least promising lever.** The rule is already in the prompt, verbatim: "All wrong answers must be similar in length and complexity to the correct answer." It is failing at 48/54/46%. Anything that amounts to restating it inherits that track record. The strategy table in 2.2 is re-ranked accordingly, and the code-side mechanisms (overgenerate-and-select, rebalancing pass) move up because they do not depend on adherence at all. See Part 2A.
  1. The prompt's own worked exemplar demonstrates the defect it forbids, and our detector does not flag it. [MEASURED, verified independently from the production file] Key 56 chars / 10 words against distractors of 46/46/49 chars and 8/9/10 words: ratio 1.1915, charDelta +9, strictly the longest by characters, length mid-rank 1.00. It slips under the 1.25x gate by six hundredths. Fixing the exemplar is now a first-class remedy -- and a mandatory confound removal, because no other prompt A/B is interpretable while the demonstration contradicts the instruction.
  1. Correction to my own earlier recommendation: the 2-word spread rule is the right instruction unit and the wrong gate unit. The shipped exemplar has a word spread of exactly 2 -- it passes the rule I proposed -- while being strictly the longest by characters. Words are what the model can count; characters are what code measures exactly. Ask in words, gate in characters. Do not let wordSpreadOkRate become the acceptance gate. The underlying finding still holds (inside the <=2-word bucket the key is longest 13% of the time, n=61, versus 77% outside it), but 13% is not 0%, and the exemplar is one of the 13%.
  1. The current validator measures the right thing with the wrong statistic. keyCueSignals's binary "longest AND >=1.25x" throws away most of the signal and is one-sided. Replace the headline with the mid-rank of the key's length among the four options (2.50 = balanced, 1.00 = key always longest). Measured pooled value: 1.80, and it is monotone in grade -- g1 2.64, g4 1.70, g7 1.17. [MEASURED] The rank statistic has an exact null (uniform on 1..4, mean 2.50, variance 1.25), so it gives a significance test with zero model spend. The 1.25x threshold should move to 1.15 with an absolute floor of +8 chars, calibrated so that the shipped exemplar is flagged; the cost to historical comparability is small (48/54/46 becomes 45/58/46). Full derivation in Part 2A.4.
  1. Length is roughly half the leak. Do not promise more than that. With both the length cue and the content-density cue removed, the judge still landed on the key on 76% of judge-declared-unanswerable items (13/17) against a 25% chance line. [MEASURED] Rebalancing length will move the validator metric a long way and the blind-solve metric much less. Say so up front to whoever signs off.
  1. The residual channel has a name and it is in the literature. The unexplained hits are overwhelmingly convergence cues: the three distractors are mutually alike on some dimension and the key is the odd one out. Haladyna, Downing & Rodriguez guideline 28e is exactly "Pairs or triplets of options that clue the test-taker to the correct choice." [CITED] Examples from our own corpus below. This is partly code-detectable and is the right second phase of validator work.

Part 1 -- Defect 1: the 50-character hint cap

1.1 What is actually known about character-level limits

[CITED] Models do not process characters. Byte-level BPE groups characters into subword tokens, so the unit the model can attend to is not the unit the constraint is expressed in. CharBench (Uzan & Pinter, arXiv:2508.02591) measures state-of-the-art LLMs at 43.6% on first-occurrence character indexing and 32.3% on last-occurrence, against >70% on the token-level analogues. Its diagnosis is nuanced and worth quoting rather than simplifying: for counting tasks "tokenization properties are weakly correlated with correctness, while the length of the queried word and the actual character count play a more significant part"; for positional tasks "performance is negatively correlated with the length of the token containing the queried character."

Note the honest reading: CharBench does not say tokenization is the whole story for counting. The safe engineering conclusion is narrower and sufficient -- character counts are a unit the model reasons about badly and inconsistently, and the failure gets worse as the string gets longer.

[CITED] Word counts are the unit that gets verified in practice. IFEval's length constraint family is namespaced length_constraints:number_words ("Answer using at most {num_words} words"). Follow-up reliability work reports that even for word counts the picture is uneven -- strong models hold up (GPT-5 96.0% -> 88.0% under perturbation) while weaker ones collapse (Qwen3-32B 83.1% -> 40.0%) -- and that models are sensitive to the exact number requested. So: word budgets are better, not perfect. They still need a code-side check.

[CITED] AWS's own guidance for Nova, when it talks about limiting output, lists the units as "a maximum number of words, bullet points, or paragraphs" -- characters are not among them -- and its worked example is "no more than 15 words."

1.2 What our corpus says [MEASURED]

428 hints across 36 quizzes.

Modelnminp50p90maxover 50mean overagemax overage
gpt-3.5-turbo1412747597149 (34.8%)7.321
claude-haiku-4-51442040485410 (6.9%)2.54
nova-2-lite143243744622 (1.4%)6.512

Overage histogram, all models: <=50 367, 51-60 47, 61-70 13, 71-80 1.

Three facts that shape the fix:

  • The violations are small. haiku's worst hint is 54 chars -- 4 over. nova's worst is 62. Only one hint in the whole corpus exceeds 70. This is a near-miss problem, not a runaway-generation problem.
  • Word budget is a clean proxy. Compliance with the 50-char cap, bucketed by word count: <=6 words 128/128 (100%, max 47 chars); <=7 words 247/247 (100%, max exactly 50 chars); <=8 words 322/337 (95.5%, max 61 chars); <=9 90.9%; <=10 88.2%. Already-compliant hints have a word-count distribution of {4:5, 5:38, 6:85, 7:119, 8:75, 9:36, 10:9}, so a 7-word ceiling keeps the modal hint length and only trims the tail.
  • The stem is dead weight. 196/428 hints (45.8%) open with an imperative stem ("Think about", "Consider", "Remember", "Recall", "Focus on", "Look at", "Reread", "Notice", "Check"), mean cost 9.2 chars. Of the 52 over-length hints carrying a stem, 40 (76.9%) drop under 50 with the stem removed -- 65.6% of all violations in the corpus.
  • Truncation is not free. 0 of the 61 over-length hints contain a sentence break; they are all single clauses. Word-boundary truncation at 50 loses a mean 1.43 words (max 4), and by my classification 17 of 61 (28%) lose real content rather than a trailing word.

1.3 Every enforcement layer, ranked

#LayerMechanismFailure modeLatency / token costDegrades usefulness to a child?Verdict
1Word budget in the prompt ("at most 7 words")Replaces an ask the model cannot verify with one it can approximately countModel overruns the word budget, or writes 7 very long words~0. A few tokens of prompt. No extra call.No -- 7 words is the current modal hint lengthAdopt. Primary. [MEASURED 100% char-compliance at <=7 words]
2Ban the imperative stemRemoves ~9 chars of pure ceremony from ~46% of hintsModel substitutes a different filler opener~0No -- improves densityAdopt. Composes with #1. [MEASURED fixes 65.6% of violations alone]
3Deterministic validator + targeted repair callCode checks hint.length <= 50; on failure, a small second call re-writes only the offending hintsRepair call also overruns; needs a bounded retry count+1 small call, and only on the violating fraction. A targeted repair payload (question + the 1-3 bad hints) is roughly an order of magnitude smaller than a full quiz call (~1500 in / ~900 out)No -- the model rewrites rather than the code cuttingAdopt as the safety net behind #1/#2. Cost scales with a violation rate that #1/#2 should already have driven low.
4Code-side truncationhint.slice(0, 50) at a word boundarySilently ships a clipped clue0Yes -- [MEASURED] 28% of violating hints lose real content, mean 1.43 wordsAdopt only as a final never-ship-a-violation guardrail, after #1-#3. Log every firing; a nonzero rate is a prompt bug, not a solution.
5JSON schema maxLengthWould be constrained decodingOn Bedrock: documented as unsupported, and unsupported features are documented to 400 the request. On the Anthropic first-party path: stripped from the wire schema, moved into the description, validated by the SDK after the factn/an/aDo not build on it. Conditional on Briefs 2/3 confirming behavior; my read of the AWS doc is that it fails closed, not open.
6Dedicated hints-only second callSeparate call writes only the hintsHint writer loses the option context, hint drifts off the right distractor+1 full call on every quiz, unconditionallyPossibly improves it -- a focused call can be tunedReject for now. Pays a permanent 100% call surcharge for a problem that #1+#2 already solve. Reconsider only if #1-#3 fail.
7Stop sequencesHalts generation on a literal string matchThere is no string meaning "50 characters have elapsed"; applied to JSON it truncates the entire response mid-object and hands the parser garbagen/an/aReject. Mechanically inapplicable.

1.4 Concrete prompt fragment for Defect 1

Replace the existing bullet in generateQuestionsPromptForFullStory.ts (and the sibling paragraph prompt):

- Every wrong answer must include a "hint_explanation" -- a short, abstract clue (under 50 characters) that points the reader back to a relevant part of the story without stating the answer

with:

## Hints
- Every wrong answer carries a "hint_explanation": a clue that points the reader
  back to the part of the story that settles the question.
- Each hint_explanation is AT MOST 7 WORDS. Count the words before you write it.
- Start the hint with the thing itself. Write "The color of Nia's mittens",
  not "Think about the color of Nia's mittens". Openers such as "Think about",
  "Consider", "Remember", "Recall", "Focus on" and "Look at" waste the budget.
- A hint names where to look. It never contains the answer.
- The correct answer carries only an "explanation" and no "hint_explanation".

And update the JSON template field description in both files from "string -- a short clue (<50 chars) pointing to relevant story detail" to "string -- a clue of at most 7 words, no lead-in phrase".

The exemplar's three hints must change in the same edit. [MEASURED] They are 37/38/37 chars -- compliant with the current cap -- but 8/7/8 words, and all three open with exactly the stem being banned ("Focus on", "Reread", "Look at"). Ship the new rules with a demonstration that violates them and the prompt contradicts itself the same way it already does for option length. Replacements are in Part 2A.2.

Note the prompt says "AT MOST 7 WORDS" in caps but frames the stem rule positively ("Start the hint with the thing itself") with the banned openers as illustration rather than a bare prohibition. That is a deliberate straddle of the negative-constraint conflict; see 1.5.

1.5 The negative-constraint conflict, stated rather than resolved

The grounding flags a tension. It is real, and it is inside each vendor's own documentation, not just between them:

  • AWS Nova (prompting-precision): "Use capitalization and strong wordings such as DO, DO NOT, MUST to emphasize the rules you want the model to strictly obey." Its worked examples use "DO NOT generate anything apart from numbers 1-10." [CITED]
  • AWS Nova (prompting-general-tips): recommends atomic instruction breakdown, and warns about the model looping on escaped Unicode -- with a recommended fix phrased as "Do NOT ever put escaped Unicode in the output". [CITED] I searched both pages and did not find the "avoid negation / state what you want instead of what you don't want" line that a secondary summary attributed to AWS. I could not source it; do not cite it.
  • Anthropic (prompting best practices, "Control the format of responses"): "Tell Claude what to do instead of what not to do" -- and then the very next code block on the same page ships "DO NOT use ordered lists" and "NEVER output a series of overly short bullet points." [CITED]

So both vendors state a positive-framing principle and then use hard negatives for hard constraints. The honest synthesis, and this is [HYPOTHESIS] not [CITED]: negatives are safe for format constraints (what shape the output takes) and were dangerous for content constraints on gpt-3.5-turbo (an earlier question-generation defect, where a "DO NOT" made the model emit an error object instead of questions). The an earlier question-generation defect lesson was learned on a model that is being retired; it must not be assumed to bind haiku-4-5 or nova-2-lite. That is Open question OQ-1 and it is cheap to settle.

One further [CITED] operational note: prefilled assistant responses return a 400 on Claude 4.6 and later. Our prompt's Begin your response with { is a prompt-level substitute and is fine. haiku-4-5 predates that change, but nobody should design a new prefill-based enforcement path.


Part 2 -- Defect 2: option length and specificity balance

2.1 What the corpus actually shows [MEASURED]

143 items, 140 with a judge outcome.

Rank of the key's character length among the four options (ties broken to the key's advantage, matching the current validator's strictness):

rankcountshare(chance)
1 (longest)9465.7%25%
21611.2%25%
32114.7%25%
4 (shortest)128.4%25%

By grade -- this is the shape of the defect:

gradenmean mid-rank (2.50 = balanced)key strictly longestkey strictly shortestmean distractor charsmean key-minus-distractor chars
1352.6429%29%21.5+2.9
4721.7053%3%36.9+10.1
7361.1786%0%60.3+31.3

By model, mean mid-rank: gpt-3.5-turbo 1.82, claude-haiku-4-5 1.66, nova-2-lite 1.92. Consistent with the grounding's finding that the three are tied -- this is the prompt's defect.

Grade 1 is not broken. 2.64 is on the balanced side of 2.50, with a symmetric 29%/29% longest/shortest split. Any fix must not "correct" grade 1 into a new imbalance, and any acceptance gate must be evaluated per grade band, not pooled.

Specificity features, key minus mean distractor, paired across 143 items:

featuremean deltaSEtkey > distractorkey < distractor
characters+13.691.499.210135
words+2.240.268.68932
content words+1.560.188.68820
words >=7 chars+0.590.115.57726
commas+0.160.044.5211
subordinating conjunctions+0.030.030.92025
digits+0.010.001.830

Negative result worth recording: syntactic complexity measured as subordinating conjunctions (because, while, after, although, which, that...) is not part of the leak (t=0.9, and the key has fewer than the distractors on 25 items vs more on 20). The brief's list of candidate channels included syntactic complexity; on this corpus it is not one. Commas are a real but tiny channel (t=4.5, but the delta is 0.16 and 121/143 items are ties).

The 2-word-spread result -- the single most actionable number in this brief:

conditionnkey is strictly longest
all four options within a 2-word spread6113%
spread > 2 words8277%

43% of current items already meet the 2-word rule. Within-item word spread by grade: g1 p50=2 / p90=4; g4 p50=3 / p90=7; g7 p50=6 / p90=10.

Per-grade option word counts, for setting a target:

gradekey words (p25/p50/p75)distractor words (p25/p50/p75)
11 / 4 / 83 / 5 / 7
46 / 8 / 135 / 7 / 10
712 / 16 / 188 / 11 / 12

Grade 7 is where the key runs away: mean 15.6 words vs 10.6 for distractors.

The residual, and what it actually is:

slicewithout-story hit raten
all judged items92%140
judge-declared unanswerable87%31
unanswerable, with NEITHER a length cue nor a content-density cue76%17
unanswerable, with `char delta<= 5` and no content cue81%16

So a fully length-balanced, density-balanced item still leaks at roughly 3x chance. The verbatim residual items make the mechanism obvious:

Q: What were the numbers Marcus used for his locker combination?
   KEY: "12, 4, 30"       <- irregular
   dis: "5, 10, 15"       <- arithmetic sequence
   dis: "2, 6, 8"         <- regular
   dis: "20, 15, 10"      <- arithmetic sequence
   (lengths 9 / 9 / 7 / 10 chars -- perfectly balanced, and trivially solvable)

Q: How did Marcus feel when his coat tumbled out onto the floor?
   KEY: "He laughed at himself."            <- positive affect
   dis: "He cried in frustration."          <- negative
   dis: "He shouted for help."              <- negative
   dis: "He angrily kicked the locker."     <- negative

Q: What can you infer about Marcus from the way he solved the locker problem?
   KEY: "Marcus is patient and thinks logically."        <- positive trait
   dis: "Marcus is easily frustrated and gives up quickly."
   dis: "Marcus is careless and forgetful."
   dis: "Marcus is afraid of getting in trouble."

This is Haladyna guideline 28e, "Pairs or triplets of options that clue the test-taker to the correct choice" [CITED]. The three distractors converge; the key is the odd one out. It is a distinct defect from length, it is not fixed by any length intervention, and it needs its own ticket.

A second, smaller residual channel is corpus prevalence: Zotos, van Rijn & Nissim (arXiv:2602.17377, 2026) find that across three large MCQ sets "correct answers, independently of the question stem, are significantly more available than incorrect options," and that always picking the most corpus-prevalent option scores "up to 9.0% above the random-guess baseline" [CITED]. That bounds the channel: it explains roughly 25% -> 34%, not 25% -> 76%. Convergence is the bigger fish.

2.2 Strategies, evaluated

Re-ranked after the orchestrator's finding that the instruction already exists. The ordering below now runs roughly best-to-worst expected value, and the dividing line is whether a strategy depends on the model obeying a length instruction. Everything above the line does not; everything below it does, and therefore inherits the track record of the instruction already in the prompt.

Does not depend on adherence:

StrategyEvidence statusAssessment
Overgenerate-and-select[CITED] as a pattern; [HYPOTHESIS] for this applicationNow my top pick outright. Ask for 6 distractors in the same single call with per-option word counts, then let code pick the 3 that bracket the key. The model is asked for variety, which it is good at; code does the balancing, which the model is bad at. Zero extra calls, some extra output tokens. Detail in the row further down and in OQ-6.
Fix the shipped exemplar[MEASURED] that the exemplar is cued (mid-rank 1.00, ratio 1.1915); [HYPOTHESIS] that fixing it changes behaviorNecessary, almost certainly not sufficient, and blocking. Full treatment in Part 2A.2-2A.3.
One pass, then a dedicated rebalancing pass[HYPOTHESIS]Moves up. It is the only strategy that acts on the finished set, the only lever on the convergence residual, and it does not require the generator to have obeyed anything. Costs +1 call per quiz.
Code-side selection from a stated word count[MEASURED by analogy]Have each option carry "words": N; code recounts and reorders/rejects. The self-report is diagnostic, the recount is authoritative.

Depends on adherence -- burden of proof is now on these:

StrategyEvidence statusAssessment
Spread constraint ("all four within 2 words of each other")[MEASURED] the observational split (13% key-longest inside the <=2-word bucket vs 77% outside); [HYPOTHESIS] that asking for it worksDemoted from "top pick" to "worth one A/B". It survives the reframing on one specific ground and only that ground: "similar in length and complexity" is a qualitative property the model cannot check as it writes, while "within 2 words" is countable -- the same chars-to-words move that carries Defect 1. But it is still prompt text asking for length balance, and prompt text asking for length balance is what is currently failing. And it does not gate: the shipped exemplar has a word spread of exactly 2 and is still strictly the longest by characters. Ask in words; gate in characters.
Explicit target length band for all four options[CITED] the rule (Haladyna g24, 85% textbook consensus); [HYPOTHESIS] the effectEffectively dead. This is a restatement of a rule that is already in the prompt and already failing. It also cannot be written correctly once: grade 1 options run 1-8 words and grade 7 run 8-20, so any single band is wrong for two of three grade bands. Do not spend an experiment slot on it.
Generate distractors BEFORE the key[HYPOTHESIS]Plausible and cheap: the key gets written to fit an already-fixed set instead of the set being written around a key. Two risks: (a) writing distractors before you have committed to an answer invites the model to draft distractors that are all defensible, which is how key errors get introduced -- and key accuracy is currently our clean metric (0 errors for both candidates), so this trades a good number for a bad one; (b) the emitted JSON still has correct_answer first, so generation order and emission order have to be decoupled, which means the prompt has to describe an order the output does not show. Try it, but rank it below the spread constraint.
One pass, then a dedicated rebalancing pass[HYPOTHESIS]The only strategy that can act on the set as a whole, and the only one that can fix convergence as well as length. Costs +1 call per quiz -- against a current quiz-call cost this is a real 100% surcharge on the quiz path. Worth prototyping precisely because it is the only lever on the residual, but it should be gated on the cheap fixes failing.
Stated word count per option, code-verified[MEASURED, by analogy] the word-count-over-char-count result transfers directlyGood, and it composes with the spread constraint: have the model emit "words": N alongside each option, then code recounts. Two benefits -- the act of committing to a number is a forcing function, and any large `N - actual` is a free signal that the model is not tracking length at all. The code recount is authoritative; the self-report is diagnostic only.
Model self-reports which option is longest[HYPOTHESIS], and I rank it lowCode computes this for free and more reliably. The only thing self-report could add is a self-correction effect from having to look, and there is a strictly better version of that idea: make it emit the word count (above), which is a number it must produce before it can check itself, rather than a label it can assign after the fact.
Overgenerate-and-select (mine) -- detail for the row at the top of 2.2[CITED] as a pattern (Alhazmi et al. distractor-generation survey documents two-stage generate-then-discriminate and overgenerate-and-rank); [HYPOTHESIS] for this applicationMy highest-confidence Defect-2 mechanism, and the reframing raises rather than lowers my confidence in it -- it is the only length remedy that survives the model ignoring every length instruction we write. Ask for 6 distractors in the same single call, each with its word count, then let code select the 3 whose lengths bracket the key most tightly. This is the same principle that makes the word budget work: move the arithmetic out of the model and into code. It costs a modest number of extra output tokens and zero extra calls. It also gives the anti-convergence check something to work with -- if the 3 best-balanced distractors all converge, code can swap one for the next-best.
Homogeneous syntactic frame (mine)[CITED] as the rule (Haladyna g23, "Keep choices homogeneous in content and grammatical structure"; note the 2002 review says of this rule "Although empirical evidence is lacking, the high degree of consensus among authors favors the continued use of this guideline")Attacks length and specificity together: require all four options to instantiate one frame (all bare noun phrases, or all He <verb>ed <object>.). Cheap to state, hard to verify in code beyond crude POS heuristics.
Anti-convergence rule (mine)[CITED] as the rule (Haladyna g28e); [HYPOTHESIS] that stating it worksDirectly targets the measured residual. Prompt fragment below.

Honest accounting of the evidence line: only the spread constraint and the word-budget results are MEASURED. Everything in the "generate in this order" family is my design judgment. The orchestrator should treat the spread constraint as ready to ship behind a validator gate and treat the rest as a prototype queue.

2.3 Concrete prompt fragment for Defect 2

Read 2A.1 first. The block below replaces an instruction that already exists and is already failing, so its expected value is lower than it looks. It earns a trial on one ground only -- it swaps an uncheckable qualitative property ("similar in length and complexity") for a countable one ("within 2 words") -- and it should be tested after the exemplar fix, not alongside it. The rebalanced exemplar in Part 2A.2 is the part of this section I would ship first.

Replace the ## Answer Quality block:

## Answer Quality
- Each question has exactly 1 correct answer and 3 wrong answers.
- Write the four choices so a reader who has not read the story cannot tell
  them apart. The right answer must look exactly as ordinary as the other three.
- LENGTH: the longest choice and the shortest choice differ by AT MOST 2 WORDS.
  Give each answer a "words" field with its own word count so this is checkable.
- FRAME: all four choices use the same grammatical shape. If one is a full
  sentence, all four are full sentences. If one is a noun phrase, all four are.
- DETAIL: all four choices carry the same amount of detail. If the right answer
  names a place, a time and a reason, each wrong answer names a place, a time
  and a reason too.
- SPREAD: the three wrong answers must not share something the right answer
  lacks. If all three wrong answers describe an upset feeling, the right answer
  must describe a feeling too, not a fact. If all three wrong answers follow a
  number pattern, the right answer follows one too.
- Wrong answers use words or phrases from the story in plausible but incorrect
  ways, and reflect mistakes a grade {grade} student would genuinely make.
- Each wrong answer is wrong for a different reason.

And, for the overgenerate-and-select variant [HYPOTHESIS], change the JSON contract to request six wrong_answer entries plus a words integer on every answer, and select down to three in code. That is a parser change (normalizeQuestion currently hard-requires wrong.length === 3), so it is a harness experiment before it is a production change.


Part 2A -- The adherence reframing, the exemplar, and the threshold

2A.1 The instruction is already there, and that changes the prior on every prompt-text remedy

generateQuestionsPromptForFullStory.ts, under ## Answer Quality, ships this line verbatim:

- All wrong answers must be similar in length and complexity to the correct answer

and the paragraph prompt ships its own variant:

- All 4 answer choices must be similar in length and complexity

So the 48/54/46% length-cue rate is instruction non-adherence, not a gap in the spec. Three consequences:

  1. The null hypothesis for any prompt-text fix is now "no effect." We have one observation of a length-balance instruction in production and it is failing across three model families. Any new wording has to beat that, and "we said it more precisely this time" is a weak prior.
  2. The strategy list re-ranks around a single question: does this remedy require the model to obey a length instruction? Overgenerate-and-select, the rebalancing pass, and code-side selection do not. The spread constraint and the length band do. See the re-ranked 2.2.
  3. There is exactly one principled reason to still try a prompt-text remedy, and it is the same reason the Defect-1 fix works: the current instruction asks for a property the model cannot evaluate while writing ("similar in length and complexity" -- similar by how much? measured in what?), whereas "the longest and shortest choice differ by at most 2 words" is countable. That is a change in kind, not in emphasis. It is still a hypothesis, and it now needs to earn its experiment slot rather than being assumed.

There is a natural quasi-experiment sitting in the codebase that nobody has run: the paragraph prompt carries the same length-balance rule but has no worked exemplar at all, while the full-story prompt has a cued one. This study only exercised full-story mode. Running the same matrix in paragraph mode would separate "the instruction is weak" from "the exemplar is overriding the instruction" for the cost of one harness run and no prompt edits. See OQ-9.

2A.2 The exemplar, measured [MEASURED]

Verified independently by parsing the exemplar out of the production file (measure8.mjs), not taken on trust:

optioncharswords
correct -- "He felt guilty about carving his initials into the bark."5610
wrong -- "He thought someone was hiding behind the tree."468
wrong -- "He wanted to climb the tree again before dark."469
wrong -- "He was checking if the bird nest was still there."4910
  • mean distractor: 47.00 chars / 9.00 words
  • ratio 1.1915, charDelta +9.00
  • strictly longest by characters: yes; strictly longest by words: no (ties the third distractor at 10)
  • length mid-rank 1.00 -- the worst possible value on the metric I am proposing as the headline
  • flagged by the current keyCueSignals (isLongest && ratio>=1.25): no
  • flagged by correctIsLongest alone: yes
  • word spread: exactly 2 -- so it passes the 2-word-spread rule I recommended in the first pass of this brief

That last line is the correction. A word-spread rule is a good thing to ask the model for, because words are countable; it is a bad thing to gate on, because two options can sit two words apart and ten characters apart. Ask in words, gate in characters. I have demoted wordSpreadOkRate from the "should pass" gate list to a reported diagnostic in Part 3.4.

The hint exemplars in the same prompt, also measured:

hintcharswords
"Focus on what Kai did with his knife."378
"Reread the part after Kai climbs down."387
"Look at what Kai touches on the tree."378

All three are comfortably inside the 50-character cap. All three also open with exactly the imperative stem I am proposing to ban, and two of the three exceed the 7-word budget I am proposing to impose. Both proposed Defect-1 rules therefore require rewriting these exemplars in the same edit, or the prompt will demonstrate violations of its own new rules. Replacements:

"hint_explanation": "What Kai did with his knife."          (28 chars, 6 words)
"hint_explanation": "The part after Kai climbs down."        (31 chars, 6 words)
"hint_explanation": "What Kai touches on the tree."          (29 chars, 6 words)

And the exemplar's four options, rebalanced -- key trimmed, distractors lengthened, all four to 9-10 words and 46-51 chars, mid-rank 2.5:

"correct_answer": "He felt guilty about carving into the bark."          (43c,  8w)
"wrong_answer":   "He thought someone was hiding behind the old tree."   (50c,  9w)
"wrong_answer":   "He wanted to climb the tree again before dark."       (46c,  9w)
"wrong_answer":   "He was checking whether the bird nest was still there."(54c, 10w)

(Chars 43/50/46/54 -- key is now rank 4 of 4, which over-corrects for a single exemplar. If the intent is a neutral demonstration rather than a reversed one, aim the key at the middle: lengthen it to ~48 chars. The point is that whoever edits the file must actually run the numbers, which is why the validator should be pointed at the prompt file, not only at model output. See 2A.3.)

2A.3 Is fixing the exemplar sufficient on its own? No -- and we have local evidence

Almost certainly not sufficient. The direct evidence is inside the same prompt: the three hint exemplars are compliant with the 50-character cap, and hint length is still our worst format defect (34.8% of gpt-3.5's hints violate, 6.9% of haiku's). A compliant demonstration demonstrably did not produce compliance for a character-counting constraint. That is the orchestrator's observation and it is the right one -- it is also, incidentally, clean local corroboration of the tokenization argument in 1.1, because the model was shown three correct examples and still could not count to 50.

But it is necessary, and it is blocking, for three reasons:

  1. It is a confound. Right now the prompt states a rule and demonstrates its violation. Any A/B on new instruction wording is uninterpretable while that contradiction stands -- a null result would not distinguish "the wording does not help" from "the demonstration overrode the wording."
  2. The cell has never been tested. For the hint cap we have run "instruction + consistent demonstration" and it failed. For option length we have only ever run "instruction + contradictory demonstration." We do not know what the consistent condition does, and it is one run away.
  3. It is close to free. A prompt-file edit with no runtime cost, no latency, no schema change.

So: rank it as mandatory hygiene and the first thing to land, and set expectations that it moves the number some and does not close the gap. My guess, explicitly a guess, is that it accounts for a meaningful minority of the 55% -- the exemplar is one of four question types being demonstrated, and it is a strong demonstration (mid-rank 1.00) sitting immediately before the output schema.

The A/B that settles it.

  • Design: two arms, everything else byte-identical. Arm A = current prompt. Arm B = current prompt with only the four exemplar option strings replaced by a length-neutral set (key at mid-rank ~2.5, all four within 2 words and within ~6 chars). Do not touch the instruction text, the hint exemplars, the schema template, or the distribution guidance in this arm -- one variable.
  • Matrix: the frozen this study set. 4 stories x 3 models x 3 reps = 36 quizzes per arm, 143-ish items per arm.
  • Primary outcome: mean length mid-rank, pooled and per grade band. Secondary: correctIsLongestRate, meanContentDelta, anyCueRate.
  • Analysis: two-sample z on the item-level mid-rank. SE per arm = sqrt(1.25/143) = 0.094, so SE of the difference = 0.133. Detectable effect at 80% power, alpha .05 two-sided: 0.37 mid-rank points -- i.e. the arm has to move 1.80 to about 2.17 to be called. Anything smaller is real-but-unprovable at this sample size, so pre-commit to that number rather than squinting at the result afterwards.
  • No judge arm needed. Both outcomes are deterministic. Cost is generation only: 72 quiz calls.
  • Pre-registered decision rule: if Arm B reaches >= 2.17, ship the exemplar fix and re-baseline before testing anything else. If it lands between 1.80 and 2.17, ship it anyway (it is free and it removes the confound) but treat the length problem as unsolved and go straight to overgenerate-and-select. If it does not move at all, that is strong evidence that prompt-surface interventions are the wrong family entirely, and the rebalancing pass becomes the lead candidate despite its cost.
  • Guardrail: watch answer-key validity. A rebalanced exemplar teaches shorter keys, and a shorter key is an easier place to be vague.

Process fix worth more than the edit itself: add a unit test that runs keyCueSignals over the exemplar parsed out of the prompt file and fails the build if the exemplar is flagged. We shipped a worked example that demonstrates the exact defect we are measuring, for months, because nothing pointed the detector at the prompt. That is a one-file test in the harness.

2A.4 The 1.25x threshold, revisited [MEASURED]

The exemplar at 1.1915 is the argument. A detector that fails to flag an item we are literally teaching the model to imitate is mis-calibrated by construction. That gives us a principled calibration point instead of a taste-based one: the threshold must be strict enough to flag the shipped exemplar.

The blind spot is small but it is exactly the wrong cases. Items that are strictly the longest yet unflagged today (isLongest && ratio < 1.25): 10 of 143 (7.0%) -- gpt-3.5 2/47, haiku 3/48, nova 5/48; by grade {g1: 1, g4: 8, g7: 1}. Their word spreads run 0,1,1,1,2,2,3,5,5,5, and only 4 of the 10 would be caught by a >2-word-spread rule. So neither the ratio gate nor a word-spread gate covers them; isLongest does.

Per-model cuedItemRate under a ratio sweep (rule = isLongest AND ratio >= t), computed over all 143 parsed items:

tgpt-3.5-turboclaude-haiku-4-5nova-2-litepooled
1.00 (no ratio gate)49% (23/47)60% (29/48)56% (27/48)55%
1.1047% (22/47)60% (29/48)56% (27/48)55%
1.1545% (21/47)58% (28/48)52% (25/48)52%
1.19 (just below the exemplar)45% (21/47)56% (27/48)48% (23/48)50%
1.25 (current)45% (21/47)54% (26/48)46% (22/48)48%
1.4038% (18/47)38% (18/48)25% (12/48)34%

The model ranking is invariant from t=1.00 to t=1.25 -- haiku is highest at every threshold in that range, and the three stay within 11 points of each other. The threshold choice moves the level, not the conclusion. That is the important reassurance: re-deriving the historical numbers under a stricter rule does not overturn "the three models are statistically tied and this is the prompt's defect." Only at t=1.40 does the shape change materially (nova drops to 25%), which is a further reason not to go there.

Recommendation. Two changes, in priority order:

  1. Demote the ratio conjunct from the headline. Report mean length mid-rank (primary) and correctIsLongestRate (secondary). Both flag the exemplar. Neither needs a magic number.
  2. If a binary "cued" flag is kept for continuity, move it to isLongest && ratio >= 1.15 && charDelta >= 8. Measured effect on the historical numbers: 48/54/46 becomes 45/58/46, pooled 48% -> 50%. The exemplar scores 1.1915 / +9.00 and is flagged by this rule. The 1.15 catches it; the +8 absolute floor keeps a one-word grade-1 difference from tripping it (recall 1.25x is only +5.4 chars at grade 1).

A denominator note that came out of this re-derivation. The published 48/54/46 are computed only over quizzes that parsed -- r.cue is absent for the one gpt-3.5 quiz that failed wrong_question_count. gpt-3.5's published 48% is 21/44; over all parsed items including the salvaged 3-question quiz it is 21/47 = 45%. The numerator is identical either way, so nothing is wrong, but the model with the worst format reliability gets its cue rate computed on the subset of its own output that survived parsing. Report n alongside the rate so this is visible.


Part 3 -- The deterministic validator

3.1 The current definition, read as written

the evaluation harness/lib/quiz-parser.mjs:273 -- keyCueSignals(quiz). Per question:

const keyLen = String(q?.correct?.text ?? "").trim().length;
const distractorLens = (q?.wrongs ?? []).map((w) => String(w.text ?? "").trim().length);
const meanDistractor = /* mean of distractorLens */;
const ratio = meanDistractor > 0 ? keyLen / meanDistractor : 1;
const isLongest = distractorLens.length > 0 && distractorLens.every((n) => keyLen > n);
cued = isLongest && ratio >= KEY_CUE_RATIO_THRESHOLD;   // KEY_CUE_RATIO_THRESHOLD = 1.25

Returns correctIsLongestRate, cuedItemRate, meanKeyLengthRatio.

3.2 Assessment

What it gets right. It asks which option is long, not just whether the spread is wide -- that is the correct framing and it is exactly what checkQuizFormat's maxLengthRatio cannot tell you. The module comment already says this, correctly. It is pure, it is free, and it is the right kind of gate.

Is the definition right? Partly.

  1. The conjunction is nearly redundant. [MEASURED] isLongest alone fires on 55.2% of items; isLongest && ratio>=1.25 on 48.3%. The ratio gate removes 7 percentage points. Almost all of the discriminating work is done by isLongest.
  1. The threshold is doing different jobs at different grades, and it lets our own exemplar through. [MEASURED] 1.25x means +5.4 chars at grade 1 (roughly one word), +9.2 chars at grade 4, +15.1 chars at grade 7. At grade 1, where the mean option is 21.5 chars, a one-word difference trips the gate; at grade 7 the mean key already exceeds distractors by +31.3 chars, so the gate is never the binding constraint. A ratio is scale-free, which is defensible in principle -- a child probably perceives relative line length -- but 1.25 is not a defended number, and the sweep shows it sits on a shoulder: t=1.15 gives 51.7%, t=1.25 gives 48.3%, t=1.4 gives 33.6%. The decisive evidence is that the prompt's own shipped exemplar scores 1.1915 and evades the gate while being strictly the longest option with a mid-rank of 1.00. Full derivation and the per-model re-baseline in Part 2A.4; the recommendation is 1.15 with a +8-char absolute floor, which flags the exemplar and moves the historical 48/54/46 only to 45/58/46.
  1. It is one-sided, and that hides a real pattern. [MEASURED] At grade 1 the key is shortest as often as it is longest (29%/29%). A key that is reliably the shortest option is just as exploitable. The current metric scores that as perfectly clean.
  1. It is a binary over a 4-way comparison. Collapsing four lengths into one bit throws away most of the signal and makes the metric noisier than it needs to be. The mid-rank uses all of it.
  1. Ties are silently resolved in the key's favor. every(n => keyLen > n) requires strict inequality (fine), but a rank computed as 1 + count(greater) gives the key the best rank on ties. On this corpus that inflates rank-1 from 55.2% (strict) to 65.7% (tie-favored). Any rank metric must use mid-ranks.

What it misses.

  • Specificity / content density. [MEASURED] +1.56 content words (t=8.6), +0.59 long words (t=5.5). Independent enough to matter: longest OR most-content-words fires on 81/140 judged items vs 79 for length alone.
  • Convergence (Haladyna 28e). The measured residual driver. Not detected at all today.
  • Register. Long-word delta is a crude proxy; a real readability check per option would be better.
  • Syntactic complexity. [MEASURED] Not a channel on this corpus (subordinator t=0.9). Do not add it.
  • Specific determiners in distractors (always, never, all, every) -- Haladyna 28a. Not measured by me, cheap to add.
  • Stem echo / clang association -- Haladyna 28b, key repeats stem wording. Not measured by me, cheap to add.

A caution on the predictive-validity evidence. I checked every candidate cue definition against the judge's per-item behavior. Every length or density flag showed a 100% hit rate on flagged items and 79-86% on unflagged. That looks like a triumph, and it mostly isn't: the judge is a frontier model that lands on the key 92% of the time overall with the story withheld, so the negative class barely exists and every flag looks perfect. The one definition that actually separated was the coarsest, rank <= 2 (98% flagged vs 70% unflagged, the widest gap in the set). This is a saturation artifact and should not be used to rank cue definitions. The run summary already warns about exactly this for blind-solve; the same warning applies to any validity claim made against this judge.

3.3 Proposed keyCueSignals v2

Design principles: the headline is a rank statistic with an exact null; length and specificity are reported separately so a fix can be attributed; convergence gets its own field; nothing is thrown away.

// --- thresholds, all named and defended in comments ---
// 1.15, not 1.25: calibrated so the prompt's own shipped exemplar (ratio 1.1915,
// charDelta +9, strictly longest, mid-rank 1.00) is FLAGGED. A detector that misses
// the item we teach the model to imitate is mis-calibrated by construction.
// Effect on the historical this study numbers: 48/54/46 -> 45/58/46. Model ranking is
// unchanged across the whole 1.00-1.25 range, so this does not disturb prior reads.
export const KEY_CUE_RATIO_THRESHOLD = 1.15;
export const KEY_CUE_ABS_FLOOR_CHARS = 8;     // kills grade-1 one-word noise (1.25x is only +5.4c at g1)
// Word spread is the unit we ASK the model for, because words are countable.
// It is NOT a gate: the shipped exemplar has a word spread of 2 and is still
// strictly the longest by characters. Reported as a diagnostic only.
export const KEY_CUE_WORD_SPREAD_MAX = 2;
export const NULL_MID_RANK = 2.5;             // uniform over 4 options
export const NULL_MID_RANK_VAR = 1.25;        // var of uniform{1,2,3,4}

const FUNCTION_WORDS = new Set([/* a, an, the, of, to, in, on, ... */]);
const SPECIFIC_DETERMINERS = /\b(always|never|all|none|every|completely|absolutely|only)\b/i;

function tokens(s) { return String(s ?? "").trim().split(/\s+/).filter(Boolean); }
function contentWords(s) {
  return tokens(s).filter(w => !FUNCTION_WORDS.has(w.replace(/[^\w']/g, "").toLowerCase())).length;
}

function midRank(value, all) {
  // 1 = largest. Ties share the average of the ranks they span.
  const greater = all.filter(n => n > value).length;
  const equal   = all.filter(n => n === value).length;  // includes `value` itself
  return 1 + greater + (equal - 1) / 2;
}

export function keyCueSignals(quiz) {
  const questions = quiz?.questions ?? [];

  const perQuestion = questions.map((q, i) => {
    const key   = String(q?.correct?.text ?? "").trim();
    const wrongs = (q?.wrongs ?? []).map(w => String(w.text ?? "").trim());
    const stem  = String(q?.question ?? "").trim();

    const keyChars = key.length;
    const disChars = wrongs.map(t => t.length);
    const allChars = [keyChars, ...disChars];
    const meanDis  = disChars.length ? disChars.reduce((a,b)=>a+b,0) / disChars.length : 0;

    const keyWords = tokens(key).length;
    const disWords = wrongs.map(t => tokens(t).length);
    const allWords = [keyWords, ...disWords];

    const keyContent = contentWords(key);
    const disContent = wrongs.map(contentWords);
    const meanDisContent = disContent.length
      ? disContent.reduce((a,b)=>a+b,0) / disContent.length : 0;

    // --- primary, two-sided, scale-free ---
    const lengthMidRank = midRank(keyChars, allChars);   // 2.5 = balanced
    const wordSpread    = Math.max(...allWords) - Math.min(...allWords);

    // --- length cue, one-sided, with an absolute floor ---
    const ratio      = meanDis > 0 ? keyChars / meanDis : 1;
    const charDelta  = keyChars - meanDis;
    const isLongest  = disChars.length > 0 && disChars.every(n => keyChars > n);
    const isShortest = disChars.length > 0 && disChars.every(n => keyChars < n);
    const lengthCue  = isLongest
      && ratio >= KEY_CUE_RATIO_THRESHOLD
      && charDelta >= KEY_CUE_ABS_FLOOR_CHARS;

    // --- specificity, independent of raw length ---
    const contentDelta   = keyContent - meanDisContent;
    const mostContent    = disContent.length > 0 && disContent.every(n => keyContent > n);
    const specificityCue = mostContent && contentDelta >= 1;

    // --- Haladyna 28a: specific determiners parked in the distractors ---
    const determinerCue =
      !SPECIFIC_DETERMINERS.test(key) && wrongs.filter(t => SPECIFIC_DETERMINERS.test(t)).length >= 2;

    // --- Haladyna 28b: clang association, key echoes the stem more than distractors do ---
    const stemSet   = new Set(tokens(stem).map(w => w.toLowerCase().replace(/[^\w']/g,"")));
    const echo      = (t) => tokens(t).filter(w => stemSet.has(w.toLowerCase().replace(/[^\w']/g,""))).length;
    const stemEchoCue = wrongs.length > 0 && wrongs.every(t => echo(key) > echo(t)) && echo(key) >= 2;

    // --- Haladyna 28e: convergence. Numeric case is exact; textual case is a stub. ---
    const convergenceCue = detectConvergence(key, wrongs);

    return {
      index: i,
      keyLength: keyChars, meanDistractorLength: round2(meanDis),
      lengthRatio: round2(ratio), charDelta: round2(charDelta),
      lengthMidRank: round2(lengthMidRank), wordSpread,
      keyContentWords: keyContent, contentDelta: round2(contentDelta),
      correctIsLongest: isLongest, correctIsShortest: isShortest,
      lengthCue, specificityCue, determinerCue, stemEchoCue, convergenceCue,
      anyCue: lengthCue || specificityCue || determinerCue || stemEchoCue || convergenceCue,
      // retained so historical runs stay comparable:
      cued: isLongest && ratio >= KEY_CUE_RATIO_THRESHOLD,
    };
  });

  const n = perQuestion.length;
  const mean = (f) => n ? perQuestion.reduce((a,p)=>a+f(p),0) / n : 0;
  const rate = (f) => n ? perQuestion.filter(f).length / n : 0;
  const meanMidRank = n ? round2(mean(p => p.lengthMidRank)) : NULL_MID_RANK;

  return {
    perQuestion,
    total: n,
    // PRIMARY
    meanLengthMidRank: meanMidRank,
    // exact z against the uniform null; no model spend, no judge
    midRankZ: n ? round2((meanMidRank - NULL_MID_RANK) / Math.sqrt(NULL_MID_RANK_VAR / n)) : 0,
    wordSpreadOkRate: rate(p => p.wordSpread <= KEY_CUE_WORD_SPREAD_MAX),
    // COMPONENTS
    correctIsLongestRate:  rate(p => p.correctIsLongest),
    correctIsShortestRate: rate(p => p.correctIsShortest),
    lengthCueRate:         rate(p => p.lengthCue),
    specificityCueRate:    rate(p => p.specificityCue),
    determinerCueRate:     rate(p => p.determinerCue),
    stemEchoCueRate:       rate(p => p.stemEchoCue),
    convergenceCueRate:    rate(p => p.convergenceCue),
    anyCueRate:            rate(p => p.anyCue),
    meanKeyLengthRatio:    round2(mean(p => p.lengthRatio)),
    meanContentDelta:      round2(mean(p => p.contentDelta)),
    // BACK-COMPAT: keep the old headline so historical runs remain comparable
    cuedItemRate: rate(p => p.cued),
  };
}

// Convergence (Haladyna 28e): the three distractors are alike on some dimension
// the key is not. Numeric case is exact. Textual case needs embeddings -- ship
// the numeric detector and the polarity stub now, leave the semantic one QUEUED.
function detectConvergence(key, wrongs) {
  const num = (t) => {
    const m = String(t).match(/-?\d+(\.\d+)?/g);
    return m ? m.map(Number) : null;
  };
  const kn = num(key), dn = wrongs.map(num);
  if (kn && kn.length >= 3 && dn.every(a => a && a.length === kn.length)) {
    // regular = constant first difference (arithmetic sequence)
    const regular = (a) => {
      const d0 = a[1] - a[0];
      return a.every((_, i) => i === 0 || a[i] - a[i-1] === d0);
    };
    if (!regular(kn) && dn.every(regular)) return true;   // key is the only irregular set
    if (regular(kn) && dn.every(a => !regular(a))) return true;
  }
  // Polarity stub: all three distractors carry negative-affect words, key does not
  // (or vice versa). Word lists are crude; treat this flag as advisory until it is
  // validated against hand labels. See OQ-5.
  return polarityOddOneOut(key, wrongs);
}

Reporting change to go with it. The run summary's "Key-cue (length)" column should become two columns -- mean length mid-rank (with its z) and any-cue rate -- and should be broken out per grade band, because the pooled number hides that grade 1 is already fine and grade 7 is at 1.17.

3.4 Acceptance thresholds for gating a prompt change

Statistical basis [MEASURED]: mid-rank under exchangeable options is uniform on {1,2,3,4}, mean 2.50, variance 1.25, so SE = sqrt(1.25/n). At n=143 items (36 quizzes x 4), SE = 0.094. Observed pooled mean is 1.80, i.e. z = -7.5. Per-quiz clustering is mild -- observed per-quiz sd of the isLongest rate is 0.262 against a binomial-only expectation of 0.249, implying ICC ~= 0.03 and a design effect of ~1.08 at 4 items per quiz -- so item-level inference is close enough; inflate n by ~10% to be safe.

Power to detect a change in the item-level isLongest rate from the observed 0.552 (alpha .05, one-sided, normal approximation): at n=143, 79% power to detect 0.45 and 98% to detect 0.40; at n=48, 41% and 68%. A 36-quiz run is adequate for a real fix and underpowered for a marginal one. Do not run a 12-quiz pilot and conclude anything.

Proposed gate. All of these are evaluated with no model spend on a re-run of the frozen story set:

Must pass (blocks the change):

GateThresholdWhy
Mean length mid-rank, pooled>= 2.20Moves 1.80 -> 2.20; two-sample z ~= 3.0 at n=143, ~85% power. Full statistical equivalence to 2.50 (band 2.32-2.68 at n=143) is the eventual target but too aggressive for iteration 1.
Mean length mid-rank, every grade band>= 2.00 and <= 3.00Two-sided. Stops a fix that balances the pool by over-shortening the key at grade 7 or over-lengthening it at grade 1. Grade 1 is currently 2.64 and must not regress.
correctIsLongestRate, pooled<= 0.35Chance is 0.25 with SE 0.036 at n=143, so 0.35 is ~3 SE above chance -- a defensible "no longer distinguishable enough to exploit" line. Currently 0.552.
Hint compliance>= 0.99 of hints <= 50 chars, and 0 hints > 55The Defect-1 fix must not be paid for out of the Defect-2 fix. Currently 0.857 pooled.
Format validitynot worse than the frozen baseline for that modelGuards the an earlier question-generation defect regression class. A prompt edit that reintroduces error objects fails here.
Answer-key errors0 in the judged sampleBoth candidates are at 0 today. Any strategy that reorders generation (distractors-first) must not spend this.

One addition to the must-pass list, from Part 2A:

GateThresholdWhy
The prompt's own exemplar passes the validatorkeyCueSignals over the exemplar parsed out of the prompt file: not flagged, mid-rank in [2.0, 3.0]We shipped a worked example demonstrating the exact defect we measure, and nobody noticed, because the detector was only ever pointed at model output. This is a unit test, not a run-time gate -- it costs nothing and it never lets this class of thing recur.

Should pass (warns, does not block):

GateThreshold
specificityCueRate<= 0.35
meanContentDelta<= +0.75 (currently +1.56)
anyCueRatestrictly lower than baseline

Reported, explicitly NOT a gate: wordSpreadOkRate. Words are the unit we ask the model for; characters are the unit we gate on. The shipped exemplar is the standing counterexample -- word spread 2, and strictly the longest by characters. Track it to see whether the instruction is landing, never to certify that the defect is closed.

Explicitly NOT a gate. Blind-solve rate. The run summary already says any frontier judge pins it near 100%; [MEASURED] it is 92% pooled and 87% on judge-declared-unanswerable items, so it has no headroom to move and will not certify anything. If you want a leakage number that can move, you need either a deliberately weakened solver or a much larger sample of judge-declared-unanswerable items (currently only 31 of 140).

And say the quiet part in the ticket: passing every gate above still leaves an estimated ~76% residual hit rate on cue-free unanswerable items. The validator gate certifies that the length channel is closed, not that the item is secure. Anyone reading a green gate as "children can no longer guess these" is misreading it.


Open questions

Each is an untested hypothesis with the experiment that settles it. None require production changes.

OQ-1 -- Does the an earlier question-generation defect "DO NOT" regression transfer to haiku-4-5 and nova-2-lite? This is the single highest-value cheap experiment in the brief, because the answer decides the phrasing of every constraint in both fixes, and because AWS's Nova guidance actively recommends the phrasing our harness banned. Experiment: three prompt variants (all-positive / atomic MUST-DO NOT bullets per AWS Nova guidance / hybrid), 4 frozen stories x 3 models x 3 reps = 36 quizzes per variant, scored on parseQuiz failure class only. No judge needed. The pass/fail signal is whether ERROR_OBJECT or MALFORMED_QUESTIONS appears for either candidate. Roughly the cost of one existing this study run per variant.

OQ-2 -- Does Bedrock actually 400 on a schema containing maxLength, or silently drop it? AWS documents unsupported features as an immediate 400, and documents maxLength as unsupported. Anthropic's SDKs strip it. The composed behavior on the Bedrock Converse path is not documented anywhere I found. Experiment: one Converse call per candidate model with outputConfig.textFormat carrying a schema with maxLength: 50 on hint_explanation. Record the HTTP status and, if 200, whether any hint exceeds 50. Two API calls. Briefs 2 and 3 are already in this territory -- this is a five-minute addition to their runs.

OQ-3 -- Does a 7-word budget hold on unseen stories? The 100% figure is over the 428 hints these three models produced for these four stories. It is a strong result, not a guarantee. Experiment: re-run the 36-quiz matrix with the Section 1.4 prompt fragment. Score hint.length <= 50 and word count. Gate: >= 0.99 compliance. Reuses the existing harness with a prompt-builder change only.

OQ-3b -- Does a length-neutral exemplar move the length cue? [Promoted to the top of the queue by the orchestrator's finding.] Full design in Part 2A.3: two arms, only the four exemplar option strings differ, 36 quizzes each, primary outcome mean length mid-rank, detectable effect 0.37 points at 80% power, no judge arm, generation cost only. Pre-committed decision rule and guardrails are in that section. Run this before any other Defect-2 experiment -- while the exemplar contradicts the instruction, every other prompt A/B is confounded.

**OQ-4 -- Does the 2-word spread constraint survive being asked for rather than *observed*?** [Prior lowered by the reframing: the prompt already says "All wrong answers must be similar in length and complexity to the correct answer" and it is failing at 48/54/46. The only argument for trying anyway is that "similar in length" is not countable at write time and "within 2 words" is. Sequence this after OQ-3b so the exemplar confound is gone, and note that the outcome must be gated on character mid-rank, not on word spread -- the exemplar satisfies a 2-word spread and is still strictly the longest by characters.] The 13%-vs-77% split is observational. It says items that happen to be spread-balanced do not leak by length; it does not say a model told to balance them will comply, nor that compliance is free of a quality cost (a key squeezed to 8 words at grade 7 may become vague, which risks the currently-clean key-error metric). Experiment: the same 36-quiz re-run as OQ-3, scored on wordSpreadOkRate, meanLengthMidRank, and judged key validity. The key-error count is the thing to watch, not the spread.

OQ-5 -- Can convergence be detected in code well enough to gate on? The numeric detector in 3.3 is exact for numeric option sets, which covers a real but narrow slice. The textual case (three negative-affect distractors, one positive key) needs either a curated polarity lexicon or an embedding pass. Experiment: hand-label the 143 items in the frozen corpus for convergence (one sitting, no spend), then measure precision/recall of (a) the numeric detector, (b) a polarity-lexicon detector, (c) a cosine-similarity detector over sentence embeddings using the rule "mean pairwise distractor-distractor similarity minus mean key-distractor similarity exceeds a threshold." Promote to a gate only if precision is high enough that a flagged item is worth a human look.

OQ-6 -- Does overgenerate-and-select actually balance the set? Experiment: harness-only variant asking for 6 distractors plus per-option word counts; code selects the 3 minimizing spread around the key. Compare meanLengthMidRank against both the current prompt and the spread-constraint prompt on the same 36-quiz matrix. Requires relaxing normalizeQuestion's hard wrong.length === 3 check behind a mode flag -- a harness change, not a production one.

OQ-7 -- Does a rebalancing second pass beat a better first pass, and is it worth +1 call? Only run this if OQ-3/OQ-4/OQ-6 leave the gate unmet. It is the only lever with a plausible claim on the convergence residual, and it is the only proposal here that carries a permanent per-quiz cost increase.

OQ-9 -- Instruction weakness or exemplar override? The free quasi-experiment already in the codebase. The paragraph prompt carries the same length-balance rule ("All 4 answer choices must be similar in length and complexity") and has no worked exemplar at all. The full-story prompt has a cued one. This study only ran full-story mode. If paragraph mode shows a similar length-cue rate, the exemplar is not the driver and the instruction is simply weak; if it shows a materially lower one, the exemplar is doing real damage and OQ-3b should be large. Experiment: run the existing matrix in paragraph mode (assembleQuizPrompt({ mode: "paragraph" }) is already wired, and EXPECTED_QUESTION_COUNT.paragraph = 1). One question per call means more calls for the same item count -- budget 4 paragraphs x 4 stories x 3 models x 3 reps if you want ~143 items. Deterministic scoring only, no judge. Caveat: the two modes differ in more than the exemplar (per-paragraph context, no grade distribution block, 1 question instead of 4), so this is suggestive evidence, not a clean isolation -- OQ-3b is the clean test and this one is nearly free.

OQ-8 -- Is the residual actually reducible at all by prompting? The 76% cue-free residual may be dominated by world-knowledge plausibility that no option-writing rule can remove -- Zotos et al. bound the corpus-prevalence share at ~9 points above chance, which leaves a lot unexplained. It is possible the honest answer is "a frontier LLM can solve grade-1 reading items without the passage, and that says more about the solver than the item." Experiment: re-run the blind-solve arm with a deliberately weakened solver (a small model, or the judge with an instruction to answer only from the options) and see whether the cue/no-cue gap widens. If it does, the current judge is the measurement problem and the item may be fine.


How to reproduce the measurements

All measurement scripts are in this staging directory and read one frozen file: the evaluation harness/the run archive (36 records, 36 with a parsed quiz, 35 judged, 143 questions, 140 with a judge outcome).

node /tmp/claude-1000/-home-borchard-jot-workspace/3d842a6e-2cf2-4764-9625-6f6f86066436/scratchpad/quiz-prompting-research/measure.mjs   # hint length distribution, truncation feasibility
node .../measure2.mjs   # word-budget calibration (the 7-word result)
node .../measure3.mjs   # imperative-stem overhead
node .../measure4.mjs   # rank distribution, threshold sweep, predictive validity, specificity features
node .../measure5.mjs   # tie-fair mid-ranks, what 1.25x means in chars per grade, power
node .../measure6.mjs   # residual leak with cues removed, verbatim residual examples
node .../measure7.mjs   # per-grade option word bands, the 2-word-spread result
node .../measure8.mjs   # the shipped exemplar parsed from the production file; per-model threshold sweep; the 1.00-1.25 blind spot
node .../measure9.mjs   # reconciles the published 48/54/46 with the all-items 45/54/46 (parse-failure denominator)

measure8.mjs reads the exemplar straight out of ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForFullStory.ts by regex over the === EXAMPLE === block, so the exemplar figures in Part 2A.2 are independent of the orchestrator's measurement rather than a restatement of it. They agree.

No script writes anything, makes a network call, or touches the harness.

One incidental finding while checking whether option position could be a channel: the students client does shuffle, Fisher-Yates over all four options, at ai-reading-25/src/frontend/students-client/src/helpers/comprehensionQuestions/quizQuestion.ts:34. The production prompt always emits correct_answer first, so without that shuffle the key would be in position A every time. Position is not a leak channel. (The shuffle uses Math.random() at render time and so is not stable across re-renders -- out of scope here, but somebody may want to look at it.)


Sources

All retrieved 2026-08-17.

Vendor documentation

  1. AWS, "Get validated JSON results from models" (Amazon Bedrock structured outputs). https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html -- lists minLength/maxLength under "The following features are not supported"; states "Amazon Bedrock validates the JSON schema format against the supported JSON Schema Draft 2020-12 subset. If the schema contains unsupported features, Amazon Bedrock returns a 400 error immediately."
  2. Anthropic, "Structured Outputs." https://platform.claude.com/docs/en/build-with-claude/structured-outputs -- maxLength listed as not supported; "SDK-validated constraints (stripped from the wire schema, appended to the description, and validated against the response): minimum, maximum, multipleOf, minLength, maxLength."
  3. AWS, "Creating precise prompts" (Amazon Nova). https://docs.aws.amazon.com/nova/latest/userguide/prompting-precision.html -- "Use capitalization and strong wordings such as DO, DO NOT, MUST to emphasize the rules you want the model to strictly obey"; "constrain the response length to a maximum number of words, bullet points, or paragraphs"; worked example "no more than 15 words."
  4. AWS, "General prompting tips" (Amazon Nova). https://docs.aws.amazon.com/nova/latest/userguide/prompting-general-tips.html -- atomic instruction breakdown; task decomposition. I did not find an "avoid negation" statement on this page or on (3); do not cite one.
  5. AWS, "General best practices" (Amazon Nova 2). https://docs.aws.amazon.com/nova/latest/nova2-userguide/prompting-best-practices.html -- Nova 2 index page; the substantive guidance lives on its linked subpages, which I did not exhaustively read.
  6. Anthropic, "Prompting best practices." https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices -- "Tell Claude what to do instead of what not to do"; note the sample prompt on the same page uses "DO NOT use ordered lists" and "NEVER output a series of overly short bullet points." Also: prefilled assistant responses return a 400 on Claude 4.6 and later.

Peer-reviewed / academic

  1. Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). "A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment." Applied Measurement in Education, 15(3). https://site.ufvjm.edu.br/fammuc/files/2016/05/item-writing-guidelines.pdf -- Guideline 23 "Keep choices homogeneous in content and grammatical structure" (67% textbook consensus; "Although empirical evidence is lacking, the high degree of consensus among authors favors the continued use of this guideline"); Guideline 24 "Keep the length of choices about equal" (85% consensus, no dedicated empirical section in the review); Guideline 28 "Avoid giving clues to the right answer," including 28a specific determiners, 28b clang associations, 28d "Conspicuous correct choice," 28e "Pairs or triplets of options that clue the test-taker to the correct choice."
  2. Uzan, O., & Pinter, Y. (2025, rev. 2026). "CharBench: Evaluating the Role of Tokenization in Character-Level Tasks." arXiv:2508.02591. https://arxiv.org/abs/2508.02591 -- 43.6% / 32.3% on character indexing vs >70% token-level; "tokenization properties are weakly correlated with correctness, while the length of the queried word and the actual character count play a more significant part."
  3. Zotos, L., van Rijn, H., & Nissim, M. (2026). "Corpus Prevalence of Multiple-Choice Question Options." arXiv:2602.17377. https://arxiv.org/abs/2602.17377 -- "correct answers, independently of the question stem, are significantly more available than incorrect options"; "always selecting the most prevalent option leads to scores up to 9.0% above the random-guess baseline."
  4. Alhazmi, E., Sheng, Q. Z., Zhang, W. E., Zaib, M., & Alhazmi, A. (2024). "Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation." arXiv:2402.01512. https://arxiv.org/html/2402.01512v2 -- documents two-stage generate-then-discriminate and overgenerate-and-rank patterns; note that the survey does not treat option length as a quality criterion, which is itself informative about where the field's attention has been.
  5. Zhou, J., et al. IFEval, via Inspect Evals. https://ukgovernmentbeis.github.io/inspect_evals/evals/ifeval/index.html -- 25 verifiable instruction types; the length family is namespaced length_constraints:number_words. I confirmed the word-count constraint exists; I did not confirm whether IFEval contains any character-count constraint.
  6. "Revisiting the Reliability of Language Models in Instruction-Following." arXiv:2512.14754. https://arxiv.org/abs/2512.14754 -- word-count constraint adherence under perturbation (GPT-5 96.0% -> 88.0%; Qwen3-32B 83.1% -> 40.0%); sensitivity to the exact requested count. Cited via search result summary; I did not read the full text, so treat the specific figures as second-hand.

Primary source code read directly

  1. ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForFullStory.ts
  2. ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForParagraph.ts
  3. the evaluation harness/lib/quiz-parser.mjs (keyCueSignals at line 273, checkQuizFormat, HINT_MAX_CHARS, KEY_CUE_RATIO_THRESHOLD)
  4. the evaluation harness/lib/quiz-prompt-builder.mjs
  5. ai-reading-25/src/frontend/students-client/src/helpers/comprehensionQuestions/quizQuestion.ts

Deliberately not cited: several blog posts and DEV/Medium articles surfaced on Bedrock structured outputs, including one asserting that Bedrock's grammar engine ignores maxLength but the model follows it as an instruction. That claim conflicts with the AWS documentation's own statement that unsupported features return a 400, and it is a field report rather than documentation, so it is recorded here as a conflict for OQ-2 to settle rather than used as evidence.

Brief 5 -- prior art 42K

Brief 5 -- Prior Art: How the Field Generates and Evaluates Assessment Items with LLMs

Last updated: 2026-08-17

This study research team, Brief 5 of 5. Scope: automatic item generation (AIG), published evaluations of LLM-written MCQs, the length-cue artifact, pre-ship validation practice, vendor and standards positions, and benchmarks.

All retrieval dates are 2026-08-17.


Headline calls

1. Our length-cue artifact IS documented for LLM-generated items -- but exactly once, and not in our domain. FormationEval (Ermilov 2026, arXiv preprint) measured it directly on LLM-generated MCQs: correct answers were uniquely longest in over 55% of questions against a 25% expected baseline, averaging 86.6 characters versus 69.8 for distractors. That is the only published measurement of this artifact on model-authored items that I found. It is an adult single-domain benchmark (petroleum geoscience), not K-8 reading comprehension, and it is a preprint, not peer-reviewed. Our 48/54/46 percent figures are squarely in family with its >55%.

2. Nobody has published the exploitability half of our result -- but the probe itself is standard, so claim the combination, not the method. FormationEval infers exploitability from item structure and explicitly says it cannot separate length exploitation from genuine reasoning. No source I found combines a length-cue measurement with a withheld-source solve to demonstrate that the cue actually carries a solver to the key. Our 17-of-17 on judge-declared-unanswerable items is, as far as I can establish, novel and externally publishable -- as a result, not as a technique. Partial-input probing is a well-established tradition with three canonical entries (Poliak et al. 2018 hypothesis-only NLI; Kaushik & Lipton 2018 passage-only RC; Balepur et al. 2024 choices-only MCQA). An external write-up should use that literature's vocabulary -- "partial-input baseline," "dataset artifact" -- and position our contribution as the first application of the probe to model-authored items targeting a length artifact. One methodological note from Balepur et al.: they benchmark against a majority baseline, not raw chance. Our 25% is only the right comparator if our key positions are balanced; if they are not, the correct baseline is higher.

2b. EXEMPLAR CONTAMINATION: the one published measurement of this artifact independently reached the same causal conclusion. FormationEval's own remediation narrative names the mechanism the orchestrator's data points at. Verbatim: "Few-shot examples were also corrected after analysis revealed that two of three examples had the longest option as the correct answer, contradicting the length balance guidance in the prompt text." Their prompt carried rule 7, "Balance answer options in length and structure," and the exemplars beat it. That is the same shape as our finding -- a stated length-balance rule, an embedded exemplar that violates it, a downstream cue rate far above chance. Independent convergence on the mechanism from the only other team that has measured this artifact is the strongest corroboration available for a hypothesis nobody has run a controlled experiment on.

3. The published mitigation works partially, and the failure mode of the naive fix is instructive. FormationEval reduced uniquely-longest-is-correct from >55% to 43.2% by expanding 136 distractors with technical context -- lengthening the wrong answers, not trimming the right one. Correct-answer mean length was unchanged at 86.6 chars; distractor mean rose 69.8 to 74.0. Critically: their generation prompt already contained the instruction "Balance answer options in length and structure" and it did not work. The prompt's few-shot examples were the leak -- "two of three examples had the longest option as the correct answer, contradicting the length balance guidance in the prompt text." Residual bias persisted above the 25% baseline even after mitigation.

4. Highest-confidence actionable finding: do not ask your LLM judge to catch this defect. Use a script. Moore, Nguyen and Stamper (2023) compared a rule-based checker against GPT-4 on the "Longest option correct" item-writing flaw across four datasets. Rule-based F1: 0.80, 1.00, 1.00, 0.95. GPT-4 F1: 0.18, 0.00, 0.44, 0.36. The authors note these rules "can be easily implemented programmatically, as they only check for text length and keywords." An LLM judge is close to blind to the exact defect we care most about; a five-line length comparison is near-perfect.

5. Our two-stage guess/answerability protocol is the field's published standard, and our mid-experiment metric correction was the right call. Säuberli and Clematide (READI @ LREC-COLING 2024) use the identical design on LLM-generated RC items -- answer without the text, then answer with it -- and define text informativity = answerability minus guessability. The benchmark-construction ancestor is Kaushik and Lipton (EMNLP 2018), whose question-only and passage-only ablations are the canonical justification for the probe.

6. Our defect rates are typical-to-good against the published field, and our answer-key result is unusually strong. A 71-study systematic review of AI-generated MCQs in medical education reports error rates ranging from <1% to 45% across studies. One anchor study found 49% of ChatGPT-generated items contained item-writing flaws and 22% contained factual or conceptual errors. Our 1 key error in 35 judged quizzes, with zero for either candidate, sits at the good end of that range.

7. Same-family judge bias is real and measured -- but the one K-12 study that tested a Claude judge found it stricter, not more lenient. Report the conflict; do not assume inflation.


Detail

A. Automatic item generation: what is settled

Pre-LLM AIG is a mature psychometric discipline, not a new idea. The template-based tradition (item models, cognitive models, generation by substitution into validated shells) predates LLMs and carries an established validity apparatus. The governing authority is the Standards for Educational and Psychological Testing (AERA, APA, NCME 2014), which the AIG literature reads as requiring that validity evidence be collected for machine-generated content with the same rigor as for human-authored content (source). I did not find a joint AERA/NCME position statement specifically on LLM-generated items. That is an open question below.

The consensus verdict on LLM-based AIG is "drafting tool, not author." Kıyak, Kaya and Emekli (2026), Postgraduate Medical Journal, systematically reviewed 71 studies from 24 countries and concluded: "LLMs appear useful drafting tools but current evidence does not yet support unsupervised use in summative assessment," recommending human-in-the-loop with expert verification (source). This is the single most authoritative synthesis I found and it is peer-reviewed.

B. Published evaluations: what quality rates are reported

SourceDomain / levelHeadline numbers
Kıyak et al. 2026, Postgrad Med J (71-study systematic review)Medical educationError rates <1% to 45%. Median difficulty 0.67, median discrimination 0.28, reliability 0.51-0.81 (median 0.68). Modification need ranged from "22.2% needed no modification" to "84%" requiring changes; one study found only "17% of MCQs were suitable." One study: 86.6% of items had at least one non-functional distractor.
Camarata et al. 2025, Advances in Physiology EducationRenal physiology, Y2 medical49% item-writing flaws, 22% factual/conceptual errors, 59/65 (91%) a reasonable starting point for revision.
Säuberli & Clematide 2024, READI @ LREC-COLINGGerman RC itemsText informativity: human 0.294, GPT-4 0.259, Llama 2 0.187. Over half of all ratings were 4 or 5. Llama 2: 11/180 unusable.
Leite et al. 2025 (preprint, submitted to journal)Portuguese elementary reading comprehension, target age 8GPT-4o answerability acceptance 33/38 (86.8%); options clarity 90.5%; well-formedness 100%. Weakest provenance overall acceptance 55.6%. Student psychometrics: no significant difference in difficulty between human and generated (Kruskal-Wallis H=1.249, p=0.741); human-authored most discriminative (mean D = 0.414).
Crk & Gultepe 2026, Frontiers in EducationUndergrad Operating Systems, 42 studentsNo statistically significant difference between 21 human and 14 LLM items on difficulty, discrimination, point-biserial or reliability index (all p > 0.65).

Reading of our numbers against this. Our answer-key soundness (1 error / 35 quizzes) is better than typical. Our skill-taggability spread (haiku 94%, nova 88%, incumbent 75%) has no direct published comparator I could find -- nobody reports a "maps cleanly to exactly one component" metric. Our fact-heavy unprompted mix (61 fact-detail, 43 inference, 7 main-idea, 3 vocabulary out of 140) is consistent with the field's repeated finding that LLM items skew to recall; the med-ed review's median discrimination of 0.28 is the quantitative echo of the same complaint.

The consistent failure taxonomy across the field, which our defect list only partially overlaps:

  1. Non-functional distractors (implausible options no student selects) -- by far the most-reported defect.
  2. Factual/conceptual errors and wrong keys.
  3. Ambiguity / more than one defensible answer.
  4. Recall-skew: items test retrieval, not the intended cognitive level.
  5. Passage-independence / world-knowledge answerability.
  6. Positional key clustering.

Note that non-functional distractors is the field's #1 defect and is not in our current rubric as a distinct measured dimension -- our "distractor quality" is a rated dimension, not a functionality measurement. See the rubric gap analysis below.

C. The length-cue artifact -- the core question

For human-written items: completely settled, old news, as the grounding assumed. Haladyna, Downing and Rodriguez (2002) built the canonical taxonomy of 31 item-writing guidelines from 27 textbooks and 27 research reviews (source). "Longest choice is correct" is a recognized clueing flaw that advantages test-wise students. The widely used operational version is a 19-item modified IWF rubric, in which "Longest option correct" is a named criterion with human inter-rater agreement of 97.00%, kappa = 0.83 -- one of the most reliably identified flaws in the whole rubric (Moore, Nguyen & Stamper 2023).

For LLM-generated items: documented once, by FormationEval, and nowhere else that I could find.

FormationEval detail, quoted from the primary PDF:

  • Items are LLM-generated: "derived from three authoritative sources using a reasoning model with detailed instructions and a concept-based approach."
  • "Length bias was significant. Correct answers were uniquely longest in over 55% of questions (expected 25%), averaging 86.6 characters versus 69.8 for distractors."
  • A companion qualifier-word artifact: absolute words like "always" appeared only in distractors (49 instances, 0% correct rate); hedges like "may" appeared only in correct answers (13 instances, 100% correct rate). "Combined, these patterns could be exploited well above random chance."
  • Mitigation: expanding 136 distractors with technical context, >55% to 43.2%; all 49 "always" instances replaced with synonyms; "may" added to 13 distractors.
  • Table 7 before/after: uniquely-longest-is-correct >55% to 43.2%; correct-answer mean length 86.6 to 86.6 chars; distractor mean 69.8 to 74.0 chars.
  • Residual: "Length bias is still above the 25% baseline."
  • The cue climbs with item difficulty: "correct answer is uniquely longest in 40.5% of medium versus 55.6% of hard."
  • Models exploit it: solver models "select the longest answer choice more often than the 25% baseline. This rate (38-47% across models)."
  • Their own limitation statement: this rate "close to the benchmark's residual bias... mak[es] it difficult to distinguish length exploitation from genuine reasoning that produces longer correct answers."

What is genuinely novel in our result.

Three things, in descending confidence:

  1. The exploitability demonstration. FormationEval's own stated limitation is precisely the gap our design closes. By withholding the story and taking only items where the judge declared it could not answer, we removed the "genuine reasoning" confound that they could not. 17/17 against a 25% chance baseline is the direct causal evidence the literature lacks.
  2. The grade-band gradient in K-8 reading. Our 29% at grade 1 to 83% at grade 7 is a cleaner and steeper gradient than FormationEval's 40.5% medium / 55.6% hard, and it is in a K-8 comprehension setting where nothing comparable is published. Caveat: their axis is difficulty label, ours is grade band -- parallel, not identical.
  3. Model-invariance as evidence of prompt attribution. Three models statistically tied at 48/54/46 percent is a cleaner attribution of the defect to the prompt than any single-generator study can make. FormationEval used one generator, so it could not separate prompt from model.

How hard I looked, and where. Searched across: arXiv (full-text and listing search via web), ACL Anthology, PubMed/PMC and Europe PMC, Springer, Frontiers, ACM DL, ScienceDirect, ERIC, and general web. Query families included: length bias in generated MCQ options; "longest option" plus LLM; distractor length artifact; surface cue in generated items; item-writing flaws in LLM-generated items; K-12 and reading-comprehension-specific variants of each. Confirmed negative results -- these papers evaluate LLM-generated MCQ or RC items and do not measure option length as a cue:

  • Säuberli & Clematide 2024 (RC items, the closest methodological match to our work) -- confirmed absent.
  • Xiao et al., ACL SIGEDU (ChatGPT reading comprehension exercises, Grade 8/9 China) -- confirmed absent by full-text search.
  • Leite et al. 2025 (Portuguese elementary RC, with real student psychometrics) -- confirmed absent by full-text search.
  • Crk & Gultepe 2026, Frontiers in Education -- explicitly did not evaluate option length cues, guessability without source material, or distractor functionality.
  • Maeda & Lu 2026 (52,759 state ELA/math items, LLM-critique-based acceptance prediction) -- did not test for option length cues.
  • Fei et al. 2026, option-level psychometrics -- does not address length or surface features.
  • The adjacent large literature on length bias is about LLMs answering existing MCQs, not writing them (e.g. Zhao et al. 2025: Qwen 2.5 1.5B MMLU "rises from 60 to 89 and drops from 89 to 36 when option lengths are changed"; GPT-4o 89.7% vs 83.3%). That literature establishes the cue is potent for machine solvers but says nothing about generation.

Verdict: the artifact is documented for LLM-generated items (once, adult domain, preprint). The exploitability probe and the K-8 grade gradient are not. Write it up.

C2. Exemplar contamination in few-shot item generation (second search)

Verdict: the general mechanism is established and peer-reviewed. The specific claim -- that a worked example in an item-generation prompt transmits a structural artifact its own stated rules forbid -- has been observed once, anecdotally, and never measured in a controlled experiment. That gap is real and it is ours to fill.

What is settled: demonstrations transmit format, not semantics. Min et al. (EMNLP 2022) is the canonical result and it is directly on point. Randomly replacing the labels in demonstrations "barely hurts performance," consistently across 12 models -- so demonstrations are not teaching correct answers. What they do teach is (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence, and the authors single out format: "removing the format, or providing demonstrations with labels only, performs close to or worse than no demonstrations." Option-length balance is a format property. This is the strongest general citation that an exemplar's structure propagates.

What is nearly settled: length bias specifically is learnable from in-context demonstrations. Schoch and Ji (2025, arXiv preprint) study exactly this transmission path. Findings: "models do learn length biases in the context window for their predictions" when demonstrations are drawn from the tails of a length distribution; the bias is learnable from as little as ~3 tokens of average length difference between classes; and, critically for the override question, models "learned the bias in the length information in the context window, regardless of how it contradicted what was seen during finetuning." Random-sampled demonstrations can unlearn a finetuned length bias with no parameter updates -- context beats prior in both directions.

The caveat that cuts against the hypothesis, and which I flag deliberately. Schoch and Ji observe onset "around 8 in-context examples," strengthening at k=16, 24, 32, on models of 2.7B-8B parameters, in a classification task where length correlates with a label. Our prompt carries k=1, in a generation task, on frontier models. Their curve would predict a single exemplar sits below the threshold at which they could detect learning. So their result establishes that the channel exists and is sensitive (3 tokens is a small signal, and our exemplar's 1.19x/~9-char margin is in that neighborhood), but it does not establish that one exemplar is sufficient. Anyone claiming our one-shot exemplar caused the artifact is extrapolating across example count, task type, and model scale. That is a hypothesis worth testing, not a citation-backed finding.

Instruction-versus-demonstration conflict: thinner than expected. I could not find a controlled study measuring what happens when a demonstration structurally violates an explicit instruction in the same prompt. The nearest published work is Tang et al. (Siemens/TUM, arXiv preprint), "The Few-shot Dilemma: Over-prompting Large Language Models," which shows that "incorporating excessive domain-specific examples into prompts can paradoxically degrade performance in certain LLMs." That is about example quantity degrading classification accuracy, not about a demonstration overriding a contrary rule. Adjacent, not on point -- I am flagging it rather than leaning on it. Prompt-engineering practice guidance does recommend balancing exemplars across classes and randomizing their order to avoid skewing output distribution, but this is practitioner guidance, not measurement.

Has anyone measured exemplar effects on generated assessment items specifically? No. Confirmed negatives, each checked in full text:

  • Amini et al. (2025, preprint), Prompting Strategies for Language Model-Based Item Generation in K-12 Education, is the closest possible study -- it compares seven prompting strategies including zero-shot, few-shot, chain-of-thought, role-based and sequential for K-12 MCQ generation, scored by experts on five dimensions (construct alignment, question clarity, answer accuracy, distractor quality, difficulty appropriateness). It reports which strategies score better. It does not measure any structural artifact of the generated options -- no length, no position, no cueing. Nobody looked.
  • Säuberli & Clematide 2024, Leite et al. 2025, Xiao et al. (SIGEDU), Ma et al. 2025 (inference-question generation): none attributes a generated-item artifact to prompt exemplars.
  • FormationEval is the sole exception, and its treatment is a one-sentence remediation note in a methods section, not a measurement. There is no before/after number isolating the exemplar fix from the other prompt changes -- the prohibition list grew from 5 to 15 rules across eight iterations at the same time, so their length improvement is confounded across interventions.

Reading for our write-up. Exemplar contamination in item generation is an unoccupied niche with a strong theoretical backing (Min et al.) and one anecdotal field observation (Ermilov). A clean A/B -- same prompt, key-cued exemplar versus length-balanced exemplar, artifact rate measured -- would be the first controlled measurement of it and is cheap relative to everything else in this study. That experiment, not the citation search, is what would settle it.

D. Mitigations the field has found effective

Ranked by evidence strength for our situation:

  1. Programmatic post-generation filter on option length. Strongest evidence. Rule-based F1 0.80-1.00 versus GPT-4 0.18-0.44 on this exact flaw (Moore et al. 2023). Cheap, deterministic, no model call.
  2. Lengthen distractors rather than shorten the key. FormationEval's only measured intervention; moved >55% to 43.2% while leaving the correct answer untouched at 86.6 chars. Partial, not complete.
  3. Audit your few-shot examples. FormationEval's instruction to balance length was in the prompt and was overridden by exemplars that violated it. Directly relevant if our production prompt carries examples.
  4. Explicit prohibition lists, expanded iteratively. FormationEval's prohibition list "grew from 5 to 15 rules" across eight prompt iterations. Necessary but demonstrably insufficient alone.
  5. Require a rationale field per item. FormationEval reports this as an effective early error detector: "When the model produced a plausible-sounding answer but could not write a coherent rationale, the answer was often incorrect or fabricated." We already generate per-answer explanations -- this suggests using them as a validation signal, not just as student-facing copy.
  6. Overgenerate-and-rank. Established for distractor generation, mostly in math (source). Cost implications for us.
  7. Position rotation for key placement. Standard and cheap; orthogonal to length but the same class of fix.

A caution worth flagging: prompting that includes option text can itself bias solvers toward longer options, per prompt-robustness work. Any judge-based check we run inherits the very bias it is meant to detect. This is a second independent argument for the deterministic filter.

E. How others validate before shipping

Expert review is the field's default and its documented weak point. In the 71-study review, 45% of studies relied entirely on expert review with no learner testing; 52/65 (80.0%) showed high risk for instrument validity and 38/50 (76.0%) lacked adequate reliability procedures, with many using "idiosyncratic labels" rather than standardized rubrics. Translation: the field's normal practice is worse instrumented than ours. We have a fixed rubric, replicates, and a documented metric correction. That is above the median.

Model-as-judge designs and their known biases.

  • Self-preference is real and measurable. Wataoka, Takahashi and Ri define it via an equal-opportunity criterion and measure GPT-4 at 0.520, GPT-3.5-Turbo ~0.19, with some models showing reverse bias. Their causal account matters to us: "LLMs assign significantly higher evaluations to outputs with lower perplexity than human evaluators, regardless of whether the outputs were self-generated" -- the mechanism is familiarity of phrasing, not identity recognition. A same-family generator produces text the judge finds fluent.
  • Direct conflict with the same-family concern, in the closest K-12 study. He et al. (2026, preprint) had two science-education experts adjudicate 648 LLM rubric judgments on K-12 science units. Expert agreement with Gemini 87.1%, GPT-4o 84.3%, Claude 37.0%, with Claude scoring markedly stricter (mean 2.18/3 versus Gemini 2.96, GPT 2.81). The Claude judge was the harsh outlier, not the lenient one. Caveats: it rated instructional materials, not items; it did not test same-family generation; and it is a preprint.
  • Report the conflict rather than resolving it. Generic LLM-judge literature predicts a Claude judge inflates Claude generators. The one educational study with expert adjudication found a Claude judge unusually severe overall. Both cannot be assumed. The clean resolution is an experiment, not a citation -- see open questions.
  • The recommended mitigations are consistent across sources: cross-family judging, ensembles, and human calibration sets. He et al. go further and argue for "model multiplicity" and "disagreement-aware" systems over consensus.

Blind-solve and answerability probes.

  • Kaushik & Lipton (EMNLP 2018) is the canonical ancestor: question-only and passage-only ablations on bAbI, SQuAD, CBT, CNN and Who-did-What. "On 14 out of 20 bAbI tasks, passage-only models achieve greater than 50% accuracy, sometimes matching the full model"; on CBT, question-only models reached accuracy close to full and outperformed full models on Verbs and Prepositions. Their conclusion -- validate that the passage is actually required -- is exactly our guess signal.
  • Säuberli & Clematide 2024 is the direct methodological precedent on LLM-generated RC items, with the guess-then-read ordering we use and text informativity as the derived metric.
  • Leite et al. 2025 uses a two-step answerability form on real experts ("is there an answer in the text?" then "which option?"), but with the text in view throughout -- so it detects unanswerable items, not guessable ones. Weaker than ours on this axis.

Guess and slip estimated from item text alone, without student response data. This is largely unsolved and is where the field is thinnest.

  • Fei et al. (2026, preprint) explicitly require empirical responses: "All item parameters require empirical LLM responses for estimation." No text-only route.
  • Maeda & Lu (2026, preprint) come closest to a text-only quality predictor: on 52,759 state summative ELA and math items (grades 3-11), 34% permanently rejected, they predict acceptance/rejection from item text plus LLM-generated critiques, reaching accuracy .75, F1 .64, AUC .80 overall -- but F1 .51, AUC .72 on ELA specifically, and weak detection of fairness/bias flags (sensitivity .27). They report no comparison against human reviewers.
  • Reading for us: text-only quality prediction is real but mediocre, and worse on ELA than math. Our judge-derived guess and slip signals are a reasonable proxy that the field would recognize, and nobody has a better text-only method to point us to.

F. Vendors and standards bodies on AI-generated K-12 assessment content

Settled: human review is universally asserted as required. Nobody publishes an item-level quality bar.

  • Standards. The AERA/APA/NCME Standards (2014) are the cited authority; the AIG literature reads them as requiring equal validity rigor for generated content. I found no joint NCME/AERA position statement specific to LLM-generated items.
  • Vendors. ETS's Chief Assessment Officer frames 2025 as the year the industry moved "from talking about AI to really using it" in content development; ETS still positions high-stakes item work as AI-assisted and heavily reviewed. NWEA's SVP: "I don't think we're ready to take things that have historically been deeply human activities... and just hand it over to the robots." These are positions, not published quality thresholds.
  • K-12 sector bodies. The EdTech Quality Collaborative (1EdTech, CAST, CoSN, Digital Promise, ISTE, SETDA) publishes five quality indicators -- Safe, Evidence-Based, Inclusive, Usable, Interoperable -- built into the EdTech Index and procurement workflows. Digital Promise's Responsibly Designed AI certification covers algorithmic bias reduction, data transparency, labeling of generative AI content, and educator agency over AI outputs.
  • Notable gap for us. None of these frameworks specifies anything about generated item quality -- no key-accuracy bar, no distractor-functionality requirement, no cueing screen. The one requirement with direct product consequences is disclosure/labeling of generative AI content, which is a product decision, not a prompt decision.
  • State policy. 37 states plus Puerto Rico have AI guidance as of June 2026, but "most current proposals do not require vendors to provide auditable records... disclose how systems generate outputs or demonstrate that tools have been evaluated for bias, accuracy and reliability before being used by students." No state currently imposes an item-quality audit we would need to satisfy.

G. Benchmarks and datasets

There is no benchmark for evaluating generated comprehension items. There are datasets of gold items you can generate against, which is a different thing.

  • FairytaleQA is the closest and the most relevant to Story Studio: 10,580 questions over 278 children's stories, kindergarten through eighth grade, expert-authored on an evidence-based framework, annotated across seven narrative elements (character, setting, action, feeling, causal relationship, outcome resolution, prediction). This is a directly usable external reference for our skill-tagging taxonomy -- our unprompted mix of fact-detail/inference/main-idea/vocabulary is not the framework the K-8 narrative-comprehension literature uses. Note it is free-response, not multiple-choice.
  • RACE and similar remain the general MCQ RC corpora; EduAdapt targets grade-level adaptability.
  • SAQUET (Scalable Automatic Question Usability Evaluation Toolkit) operationalizes the IWF rubric for automated MCQ quality evaluation -- the nearest thing to an off-the-shelf evaluator, and the source of the rule-based-beats-GPT-4 result.
  • No shared task, leaderboard, or standard test set exists for "is this generated item good."

Which of our five rubric dimensions the field considers adequate, and which it considers missing

Our dimensionField verdictBasis
Answer-key validityAdequate and central. Universal across the literature. Our measurement (judge solves with key visible) is standard.Kıyak et al.; Camarata et al.; Säuberli & Clematide
GroundednessAdequate, and our version is better than most. The field's equivalent is answerability-from-text / passage-independence. Our withheld-source condition is the stronger form; most studies check answerability with the text in view.Kaushik & Lipton; Säuberli & Clematide; Leite et al.
Distractor qualityPartially adequate -- the field would want a different measurement. Rated plausibility is common, but the discipline's primary metric is non-functional distractor rate, ideally from response data (one study: 86.6% of items had >=1 NFD; our median-discrimination comparator is 0.28). A Likert rating from a judge is the weak form.Kıyak et al.; Leite et al.
Grade fitAdequate and well-precedented, and the field has richer machinery than we use (Lexile, decodability, morphosyntactic complexity, difficulty-controlled distractor generation).Leite et al.; readability/AIG literature
Explanation qualityAdequate but essentially unexamined -- we are ahead of the field here. Rationale quality is studied rarely and mostly in medical education (accuracy, clarity, appropriateness, likelihood of implementation). I found nothing on child-facing explanation quality, and nothing at all on hard character limits for on-screen hints.GPT-4 rationale study, undergrad medical

What the field would say is missing from our rubric:

  1. A structural item-writing-flaw screen, run programmatically, separate from the judge. This is the largest gap and it is the one that would have caught our headline defect. The 19-item modified IWF rubric is the off-the-shelf instrument; "Longest option correct" is one criterion in it, and it is machine-checkable at F1 ~1.0. Our two known defects -- hint overrun and the length cue -- are both deterministic string properties that no LLM judge is needed for.
  2. Non-functional distractor measurement, ideally from real student response data, and at minimum as a distinct dimension rather than folded into rated distractor quality.
  3. Empirical psychometrics from students -- difficulty and discrimination. Every serious study in this space either reports these or is criticized for not having them. We cannot get them pre-launch, but the field would expect a plan to collect them post-launch.
  4. Positional key distribution. Cheap, standard, repeatedly flagged in the review as a real defect in generated items, and absent from our rubric. Worth adding to the same script as the length check.
  5. Qualifier/absolute-word cue check. FormationEval found this artifact alongside length ("always" 0% correct, "may" 100% correct). It is the same class of programmatic check and we have not looked for it. This is the cheapest untested hypothesis in the brief.

Open questions

  1. Does our judge exhibit same-family bias toward haiku on this task? Unresolvable from literature -- the general LLM-judge literature predicts inflation, the one K-12 expert-adjudicated study found a Claude judge unusually strict. Both are cited above. This needs a cross-family judge run (Gemini or GPT as a second judge on the same 36 quizzes), not another citation. Highest-value follow-up experiment.
  2. Is there a peer-reviewed (non-preprint) measurement of the length cue in LLM-generated items? FormationEval is an arXiv preprint by a single author. I found no peer-reviewed equivalent. If we publish, this is the field-state claim to re-verify at submission time.
  3. Does the qualifier-word artifact appear in our 140 items? FormationEval found it co-occurring with length bias. Untested on our data, cheap to test.
  4. Does a key-cued exemplar actually cause the artifact at k=1? Now the central open question. Min et al. establish that demonstrations transmit format; Schoch & Ji establish that length bias specifically rides that channel -- but at k=8+ on small models in a classification task. Nobody has tested k=1 in generation. This needs the A/B, and it is the single highest-value cheap experiment available to this study.
  5. Can the exemplar fix be isolated from the instruction fix? FormationEval changed exemplars and grew the rule list from 5 to 15 simultaneously, so their >55% to 43.2% improvement is confounded. Nobody has a clean effect size for exemplar repair alone.
  6. Is there an NCME or AERA position statement on LLM-generated items post-2024? I found none; the 2014 Standards remain the cited authority. Possible that something exists behind a members-only wall.
  7. What is the base rate of "correct option is longest" in professionally authored K-8 reading items? We have a chance baseline (25%) and a human-written comparator from student-authored undergraduate MCQs (~9.5% flagged across 200 items in Moore et al., though under a rubric-judgment operationalization rather than a strict longest-option test, so not directly comparable to our 48/54/46). A true comparator would require measuring a published K-8 item bank. Would materially strengthen an external write-up.
  8. Does anyone publish guidance on explanation/feedback character limits for young readers? Found nothing. Our 50-character hint cap appears to be an unexamined design constraint from the field's perspective.

Sources

All retrieved 2026-08-17. Preprint status noted where applicable.

Length cue and item-writing flaws

  • Ermilov, A. (2026). FormationEval, an open multiple-choice benchmark for petroleum geoscience. arXiv:2601.02158v2. arXiv preprint, not peer-reviewed, single author. https://arxiv.org/abs/2601.02158 (full text verified via PDF extraction)
  • Moore, S., Nguyen, H.A., Stamper, J. (2023). Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods. arXiv:2307.08161 (published at ECTEL). https://arxiv.org/pdf/2307.08161 (full text verified via PDF extraction)
  • Haladyna, T.M., Downing, S.M., Rodriguez, M.C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education. https://cmapspublic3.ihmc.us/rid=1P2XTLCSS-11K09T9-BD5/Haladyna_2002_-Appl_Meas_Educ.pdf
  • Zhao, G., Hu, S., Jian, X. et al. (2025). Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements. arXiv:2502.12459v3. Preprint. https://arxiv.org/html/2502.12459v3

Evaluations of LLM-generated items

  • Kıyak, Y.S., Kaya, M., Emekli, E. (2026). Validity of AI-generated multiple-choice questions in medical education: a systematic review. Postgraduate Medical Journal. Peer-reviewed, 71 studies. https://academic.oup.com/pmj/advance-article/doi/10.1093/postmj/qgag057/8688271
  • Camarata, T., McCoy, L., Rosenberg, R. et al. (2025). LLM-Generated multiple choice practice quizzes for preclinical medical students. Advances in Physiology Education. Peer-reviewed. https://journals.physiology.org/doi/abs/10.1152/advan.00106.2024 (abstract verified via Europe PMC REST API)
  • Säuberli, A., Clematide, S. (2024). Automatic Generation and Evaluation of Reading Comprehension Test Items with Large Language Models. READI @ LREC-COLING 2024. https://arxiv.org/abs/2404.07720
  • Leite, B., Lopes Cardoso, H., Pinto, P. et al. (2025). From Model to Classroom: Evaluating Generated MCQs for Portuguese with Narrative and Difficulty Concerns. arXiv:2506.15598. Preprint, submitted to journal. https://arxiv.org/pdf/2506.15598 (full text verified via PDF extraction)
  • Crk, I., Gultepe, E. (2026). Evaluating the instrumental quality of LLM-generated assessment items. Frontiers in Education 11:1837523. Peer-reviewed. https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2026.1837523/full
  • Xiao, C., Xu, S.X., Zhang, K., Wang, Y., Xia, L. Evaluating Reading Comprehension Exercises Generated by LLMs: A Showcase of ChatGPT in Education Applications. ACL SIGEDU workshop. https://kpzhang.github.io/paper/ACL_SIGEDU.pdf (full text verified via PDF extraction)

Validation, judges and psychometrics

  • Kaushik, D., Lipton, Z.C. (2018). How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks. EMNLP 2018. Peer-reviewed. https://aclanthology.org/D18-1546/
  • Wataoka, K., Takahashi, T., Ri, R. (2025). Self-Preference Bias in LLM-as-a-Judge. arXiv:2410.21819v2. Preprint. https://arxiv.org/html/2410.21819v2
  • He, P., Li, Z., Wang, Z., Xiong, J., Li, T. (2026). Judging the Judges: Human Validation of Multi-LLM Evaluation for High-Quality K-12 Science Instructional Materials. arXiv:2602.13243v1. Preprint. https://arxiv.org/html/2602.13243
  • Maeda, H., Lu, Y. (2026). Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques. arXiv:2608.06609v2. Preprint. https://arxiv.org/html/2608.06609
  • Fei et al. (2026). Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks. arXiv:2608.02966v1. Preprint. https://arxiv.org/html/2608.02966
  • A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation. PMC10700404 (on AERA/APA/NCME Standards applied to AIG). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10700404/

Exemplar contamination and in-context learning mechanism

  • Min, S. et al. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP 2022. Peer-reviewed. https://aclanthology.org/2022.emnlp-main.759/
  • Schoch, S., Ji, Y. (2025). In-Context Learning (and Unlearning) of Length Biases. arXiv:2502.06653v1, University of Virginia. Preprint. https://arxiv.org/html/2502.06653v1
  • Tang, Y., Tuncel, D., Koerner, C., Runkler, T. (2026). The Few-shot Dilemma: Over-prompting Large Language Models. arXiv:2509.13196v2, Siemens AG / TU Munich. Preprint. Adjacent, not on point. https://arxiv.org/pdf/2509.13196 (full text verified via PDF extraction)
  • Amini, M., Ahmadi, B., Xiong, X., Zhang, Y., Qiao, C. (2025). Prompting Strategies for Language Model-Based Item Generation in K-12 Education. arXiv:2508.20217, University of Florida. Preprint. Confirmed negative on structural artifacts. https://arxiv.org/pdf/2508.20217 (full text verified via PDF extraction)

Partial-input probing tradition

  • Balepur, N., Ravichander, A., Rudinger, R. (2024). Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? arXiv:2402.12483v2, ACL 2024. Choices-only prompting beats a majority baseline in 11/12 cases, up to 0.33 accuracy gain, on ARC, MMLU and HellaSwag. Analyzes memorization, choice priors and question inference; does NOT analyze option length. https://arxiv.org/pdf/2402.12483 (full text verified via PDF extraction)
  • Ma, W.A., Flor, M., Wang, Z. (2025). Automatic Generation of Inference Making Questions for Reading Comprehension Assessments. arXiv:2506.08260 (BEA @ ACL 2025). Preprint. https://arxiv.org/pdf/2506.08260

Mitigation techniques

  • Improving Automated Distractor Generation for Math Multiple-choice Questions with Overgenerate-and-rank. arXiv:2405.05144. Preprint. https://arxiv.org/pdf/2405.05144

Datasets

  • Xu, Y. et al. (2022). Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension. ACL 2022. Peer-reviewed. https://aclanthology.org/2022.acl-long.34/

Vendors and standards bodies

  • ETS. The State of Assessment in 2025. https://www.ets.org/insights-and-perspectives/state-of-assessment-2025.html
  • Education Week (2024). Will AI Transform Standardized Testing? (NWEA position quoted). Trade press. https://www.edweek.org/technology/will-ai-transform-standardized-testing/2024/12
  • ISTE / 1EdTech. Easing the Burden on Schools: Five Quality Indicators for Edtech & AI Products. https://www.1edtech.org/blog/easing-burden-on-schools-five-quality-indicators-for-edtech-ai-products
  • Digital Promise. Supporting EdTech Developers (Responsibly Designed AI certification). https://digitalpromise.org/initiative/artificial-intelligence-in-education/supporting-edtech-developers/
  • ExcelinEd (2026). State K-12 AI Policy in 2026: Milestones, Momentum and Missing Links. Advocacy organization blog. https://excelined.org/2026/05/26/state-k-12-ai-policy-in-2026-milestones/

The passages, and how Lexile actually works here

Lexile is an input to story generation, not a score computed from finished text. The production flow reads the chosen grade, looks up that grade's Lexile range, picks a value at random inside it, writes that value into the text-generation prompt, and stores it on the story as lexile_value. So every real Story Studio story has one specific Lexile level that was chosen for it before any materials were generated.

PassageGradeProduction range for grade Assigned LexileWordsMean sentence
The Lost Red Mitten
g1-lost-mitten
1 BR-570L not assigned 95 6.8
The Locker That Wouldn't Open
g4-locker
4 415L-1160L not assigned 117 13.0
A Quiet Afternoon
g4-thin
4 415L-1160L not assigned 71 10.1
The Bench Decision
g7-tryout
7 760L-1410L not assigned 133 14.8
These four passages carry no assigned Lexile, because they never went through the flow above. They are short fixtures written for this evaluation and hand-inspected, then frozen so the passage is byte-identical for every quiz-writer, which is what makes the model comparison valid. No Lexile was drawn for them, none was written into a generation prompt, and none was stored. Grade ranges are wide and overlapping, with grade 4 spanning 415L to 1160L, so "grade 4" alone does not pin down difficulty the way an assigned Lexile does.

Evaluation has since moved past them, and they turned out to be the conservative instrument. Since 2026-09-08 the same prompts have been measured on real Story Studio text: 15 stories the app generated in dev, each carrying its assigned Lexile, spanning grades 1 to 8 and 157 to 1218 words, and then on 30 stories generated through the app on the production code path. The frozen four put the cueing rate at 44 percent where the real stories put it at 59 percent and the dev database at 55 percent, so these fixtures understated the defect rather than inflating it. Read this page as a cheap deterministic instrument for comparing quiz-writers on identical text, not as a picture of the text students actually receive.

Method

Four frozen, human-inspected stories spanning grades 1, 4 and 7 plus a deliberately thin grade-4 case. Each story is put to each candidate model, three replicates per cell, using the production quiz prompt verbatim. The story never varies within a cell, so a score measures the model's quiz-writing rather than whether it drew an easy story. Each quiz is then graded like an assessment instrument rather than like a piece of writing: answered once with the story in view, once with the story withheld, then rated with the answer key visible.

Limits

  • One judge family produced every quality number here. The cross-family rescore has now run, and it did not vindicate them: verdict agreement was 70 percent, the second judge disagreed in the strict direction on every dimension and never in the lenient one, and the two judges agreed on whether the answer key was valid in only 9 of 23 quizzes. Treat the rubric scores on this page as one judge's opinion, not as a measurement.
  • The judge is a frontier model, not a child. Its guess rate is an upper bound on a child's, not an estimate of it.
  • No independent educator has reviewed these items. The author taught for five years and has read them, but is not independent of the evaluation, and no student has read any of them. Nothing here is validated against outside human judgement.
  • Thirty-six quizzes on four synthetic stories is enough to find defects and not enough to rank models on quality.
  • Verdicts are unstable across replicates of an identical cell, so single-replicate comparisons mislead.
  • The four passages are 71 to 133 words. Production generates several hundred. Every result here is measured on passages materially shorter than the ones children actually read, and no run has tested whether these defects behave the same at production length.
  • Within the judge-free findings there are still two tiers. Whether a schema construct binds at all is settled by a single adversarial demonstration. What it does to our real quiz is one quiz per arm, which is an existence proof rather than an effect size. Both are deterministic. Only one is powered.

Surviving objections

These were raised against this work and never adequately answered. They are the real risks, and they are listed here rather than buried so that nobody has to rediscover them.

No human has evaluated any of this

Not one educator, not one child, not one blind adult rater. Every quality conclusion is one model's opinion of another model's output. This is the single largest gap in this study and it is not closed by any experiment currently queued.

Re-scoring 36 quizzes does not fix underpowering

It fixes instrument definition. A better-defined statistic over the same thin corpus is still thin. Worth doing because it is free and because nothing else is interpretable without it, not because it makes the corpus adequate.

The maxLength result should reduce trust in the whole structured-output surface

If one documented contract is silently unenforced on both models, the prior on the others should move. Annotating it as a footnote is the mistake.

Banding may trade a measurable defect for an unmeasured one

And this has already been observed twice rather than merely predicted. The decoder cut a hint mid-word at exactly the bound, and the working character class forbids all punctuation, so hints lose apostrophes and question marks. On the nova side, 23 of 33 hints were inside 50 characters and failed the pattern purely on punctuation. One mitigation is demonstrated, which is moving the prompt target inside the bound. The other, widening the class, is untested.

The grade escalation may be partly a judge artifact

At grade 7 the correct answer to an inference question is intrinsically longer and more specific because the task demands it. Holding question type constant to plain fact-detail items still shows the climb, from 11 to 36 to 73 percent, which argues the effect is real. But that control is itself judge-tagged, so it is weakened rather than settled. The deterministic rank statistic needs no judge and is the way to settle it.

Evidence vocabulary

Every claim on this page carries one of these tiers. Hovering a tier chip anywhere shows its definition, and the Terminology tab lists every term on the page.

provenDeterministic, and established by an adversarial demonstration or an in-call control. A claim that a constraint is ignored needs exactly one clean case, so these are settled.
directional (n=1)Deterministic string measurement, but one quiz per arm. An existence proof with a plausible mechanism. Enough to justify building the fix, not enough to quote as an expected effect size.
untestedA hypothesis with a named experiment that would settle it. Not yet run.
judge-dependentDescends from a single judge's opinion over 36 quizzes on four synthetic stories. The only educator who has reviewed it is the author of the evaluation, who is not independent of it. No child has read any of it. Do not rank models on these.
citedExternal source, retrieved 2026-08-17. Someone else's measurement, not ours.

Terminology

This page mixes two vocabularies that rarely sit together: assessment science on one side and Bedrock API mechanics on the other. Almost nobody arrives knowing both, so every term below is also available as a hover anywhere it appears -- look for the dotted underline. Where our usage is narrower than the field's, the definition says so.

Item anatomy

distractor
A deliberately incorrect option. A good distractor is plausible to a reader who misunderstood, and useless to one who understood. A throwaway distractor that nobody picks makes a four-option item behave like a three-option one.
Flesch-Kincaid
An open, published readability formula (Kincaid et al. 1975) combining mean sentence length and mean syllables per word into a US grade level. Computable and citable, unlike a Lexile, and reported here to one decimal because the syllable count is a heuristic.
grade Lexile range
The Lexile span production draws from for a given grade, held in the grade_lexile_range table. Wide and overlapping on purpose: grade 4 runs 415L to 1160L, because a fourth-grade classroom contains readers at very different levels. Story generation picks a random value inside this range and writes it into the prompt.
hint_explanation
A short clue attached to each wrong answer, pointing the reader back to the relevant part of the story without giving the answer away. Capped at 50 characters because it has to fit on a child's screen.
implied band
The Lexile band from production's own configuration whose target mean sentence length contains this passage's measured mean sentence length. An implication of our config, not a measurement of the passage's Lexile.
item
One complete multiple-choice question: the stem, the four options, the marked correct answer, and its explanations. Assessment people say 'item' rather than 'question' because the question text is only one part of it.
key
The correct option. 'Answer key' is the same thing. When we say a key is wrong or mis-keyed, we mean the option marked correct is not in fact the correct one, which marks a child who understood the story wrong.
Lexile
A reading-difficulty level, written like 450L. In Story Studio it is an INPUT rather than a measurement: the flow picks a random Lexile from the chosen grade's range, writes it into the text-generation prompt, and stores it as stories.lexile_value. Every production story therefore has one specific Lexile chosen for it before any text existed.
mean sentence length
Words divided by sentences. The single strongest driver of measured text difficulty, and the axis production's own band table is keyed on.
option set
All four options considered together. Most of the defects on this page are properties of the set rather than of any single option, which is why fixing one option at a time does not work.
stem
The question text itself, before the options. Guideline 15 says the central idea belongs in the stem rather than repeated across the options, and it is the only item-writing rule with unanimous textbook agreement.

The defects

convergence cue
When three distractors resemble each other and the key is the odd one out, so the answer stands apart without the reader knowing anything. Named as guideline 28e in the item-writing literature. We think this is most of the leak our length fix will not reach.
cued item
An item where our detector says the key is given away by its length: strictly the longest option and at least 1.25x the mean distractor length. That threshold is ours, not the field's, and is under review.
format contract
The set of structural promises the production prompt makes: exactly four questions, one correct answer and three wrong ones, an explanation on every answer, a hint on every wrong answer and none on the key, and every hint under 50 characters. Checked in code, with no judge involved.
length cue
When the correct option is visibly longer than the wrong ones, so a reader can pick it without understanding the question. An old and well-documented flaw in human-written tests; we measured it in model-written ones.
mean key rank
The average position of the correct option by length among the four choices, where 1 is longest and 4 is shortest. 2.50 is what pure chance produces; 1.00 means the key is always the longest. Better than a yes/no flag because it uses the whole signal and has an exact chance baseline.
parse failure
The model returned something we could not read as a quiz at all: wrong question count, malformed JSON, or an error object in place of questions. Counted against the model rather than dropped, because dropping them flatters whichever model fails loudest.
test-wiseness
A student's ability to score above their actual understanding by reading the test rather than the material. 'Pick the longest answer' is the classic test-wise strategy, and it is what the length cue rewards.

Item flags

answerable
The judge said this item could be answered from what it had in front of it. On the with-story pass that is the expected result. On the story-withheld pass it is a warning: the item did not need the story.
key error
The judge, with the answer key in view, judged the marked answer to be wrong. The most expensive defect on the page: it marks a correct child wrong every single time, not just sometimes.
multiple_defensible
More than one option can be argued for. This is the ambiguity case: a child who understood the story picks the other defensible option and is marked wrong. Distinct from a wrong key, where only one answer is right and the key points at a different one.
none_defensible
No option is a good answer to the question as written. Usually a sign the stem and the option set were written about different things.
unanswerable
The judge said it could not determine the answer from what it had. On the story-withheld pass this is the healthy result. If it then lands on the key anyway, the option set gave the answer away.

Judge rubric

ambiguous
The judge could not decide cleanly. Treated as a warning rather than a failure, but worth reading the reasoning: it often marks the items a child would find genuinely confusing.
answer_key_valid
Is the option we marked correct actually correct, given the story? A false here is a mis-keyed item and is the worst thing a generated quiz can do.
distractor_quality
Are the wrong answers plausible to a reader who misunderstood, and clearly wrong to one who understood? Throwaway distractors nobody would pick make a four-option item behave like a two-option one.
evaluate_analyze
An item asking the child to judge a character's choice, compare perspectives, or analyse a pattern. The most demanding category, and the one asked for mainly at the upper grade bands.
explanation_quality
Do the per-answer explanations actually explain, citing the story rather than restating the option? Scored 1 to 5. This is the text a child reads after answering, so it teaches or it misleads.
fact_detail
An item asking for something stated outright in the story: who, what, where, when. The easiest kind to write and, unprompted, the kind our models produce most -- 61 of 140 items.
full
The dimension is fully satisfied, with no reservation from the judge.
grade_fit
Do the vocabulary, sentence length and reasoning demand match the grade band the quiz was written for? Scored 1 to 5.
grounded
Can every question be answered from the story text, rather than from outside knowledge the child is not being tested on?
inference
An item requiring the child to work out something the story does not state, using clues that it does. The second most common kind in our sample at 43 of 140.
main_idea
An item asking what the passage is mostly about, or for a summary in the child's own words. Only 7 of our 140 items, because the prompt never names main idea at any grade band.
partial
The dimension is partly satisfied: some items hold and others do not. The most common distractor-quality verdict in our sample, usually meaning one or two throwaway options in an otherwise sound set.
question_type_fit
Did the model produce the mix of question types the prompt asked for at this grade, rather than defaulting to fact-recall for everything?
vocabulary
An item asking what a word or phrase means in context. Just 3 of our 140 items. NAEP reserves 15 to 20 percent of every block for this, so we are far under a recognised standard.

Format checks

answer length balanced
The four options are of comparable length. Our loosest check and the one that matters most, since an unbalanced set is how the key leaks.
error_object
The model returned an error-shaped object instead of questions. The signature failure from an earlier question-generation defect, where a DO NOT phrasing made gpt-3.5-turbo treat the constraint as a validation rule to report on.
explanations
Every answer, correct and wrong alike, carries an explanation.
flat_answers
Answers came back as a flat list with no parent question, so nothing can be scored. Another an earlier question-generation defect-era shape the parser now catches explicitly.
format_valid
Shorthand for the whole deterministic format contract failing at least one of its checks. When this is the verdict reason, the quiz broke a structural promise rather than a quality judgement, and no judge opinion was involved.
hints present
Every wrong answer carries a hint. A missing hint leaves a child with no route back into the story.
hints under 50 chars
Every hint fits the 50-character screen budget. The single most-violated rule in the whole contract, and the one already reaching students today.
no hint on key
The correct answer must NOT carry a hint. A hint attached to the key is itself a give-away, since the child can see which option got one.
question count
Exactly four questions were returned. No schema keyword on either candidate can enforce a count, so this is checked in code.
question text
Every question has non-empty stem text.
wrong_question_count
The model returned a different number of questions than the four requested. Counted as a failure against the model rather than dropped from the sample, because dropping flatters whichever model fails loudest.

How we grade

blind solve
Having the judge answer the quiz as if it were the student, without seeing our answer key. We do it twice per quiz, once with the story available and once with it withheld, which is what separates a guessable item from a genuinely hard one.
cued pick
Of the items the judge said it could NOT answer with the story withheld, the fraction where it landed on the key anyway. Chance is 25 percent on a four-option item. Well above that means the option set is leaking the answer.
guess rate
How often a quiz can be answered with the passage taken away. A high guess rate means a teacher is told a child understood a story they never read. Ours is an upper bound produced by a frontier model, not an estimate of a child's.
informed guess
The guess rate restricted to items the judge said it could genuinely answer from world knowledge without the story. The honest version of the guess metric, after we corrected an earlier one that counted lucky coin flips as knowledge.
judge
A separate, stronger model that reads each generated quiz and scores it. Fast and cheap compared with human review, and not the same thing as human review. Every judged number on this page is one model's opinion of another model's work.
PASS / WARN / FAIL
The overall call on one quiz, folding the deterministic format checks together with the judge's rubric. Unstable across repeated runs of the same cell, which is why we run three replicates and distrust any single one.
replicate
A repeat run of an identical cell -- same story, same model, same prompt. Models are not deterministic, so one run tells you very little. Our verdicts moved between PASS and FAIL across replicates of the same cell.
same-family bias
The risk that a judge scores models from its own maker more generously. Our judge is Anthropic's and one candidate is Anthropic's, so we re-scored with a third-party model to check. It disagreed in the strict direction only.
slip
How often a reader who DOES know the material still gets the item wrong. It is a property of the item, not the reader. Two different defects raise it: an ambiguous item with two defensible answers, and an outright wrong key.
story withheld
The pass where the judge answers the quiz having never seen the story. If it still lands on the key, the item is answerable without reading, which is the defect this whole ticket exists to measure.
taggable
Whether an item maps cleanly to exactly one reading skill: fact, main idea, vocabulary, inference, or analysis. An item that cannot be tagged cannot feed any per-skill tracking of what a child has learned.

Schema mechanics

constrained decoding
The model writes one token at a time; with a constraint attached, illegal next-tokens are removed from the candidate pool before it chooses. The forbidden character is not discouraged, it is unavailable. This is the difference between a guarantee and a strong suggestion.
enum
A schema rule restricting a field to a fixed list of allowed values. The only constraint proven to bind hard on nova. Useless for free text, because you cannot enumerate the English language.
forced tool use
Requiring the model to answer by filling in a named function's arguments rather than writing free text. The arguments follow a schema, so it is the usual way to get reliable JSON when structured outputs is unavailable.
JSON Schema
A machine-readable description of the shape we want back: which fields exist, their types, and their constraints. We send it with the request in the hope the model is held to it. Which parts are actually held is the subject of half this page.
maxLength
A schema rule capping a field's character count. The obvious way to enforce our 50-character hint limit, and a trap: both models accept it, return success, and ignore it completely, so you believe you hold a guarantee and hold nothing.
pattern
A schema rule giving a regular expression the field's text must match. On claude-haiku-4-5 this is enforced by the decoder and is a genuine guarantee. On nova-2-lite it is accepted and never checked.
prompt caching
Charging less for the unchanging part of a repeated prompt by reusing prior work on it. Engages on nova at our size; does not engage on haiku, which needs a larger minimum than our request reaches.
regular expression
A compact pattern that text either matches or does not. In ^[A-Za-z ]{1,50}$ the ^ and $ anchor the start and end, [A-Za-z ] allows any letter or space, and {1,50} allows one to fifty of them -- so: 1 to 50 characters, letters and spaces only.
structured outputs
Bedrock's dedicated feature for forcing a response into a schema. Available to claude-haiku-4-5 and not supported at all on nova-2-lite, which has to fall back on ordinary tool calling.
temperature
How much randomness the model uses when choosing each word. Zero is the most predictable setting. It buys repeatability; our probes show it does not buy constraint compliance.
tokenization
Models read text in chunks of several characters rather than character by character, so they cannot reliably count characters. This is why a 50-character limit is a bad ask and a seven-word limit is a good one.

Programme

evidence tier
The label every claim on this page carries, saying how much weight it can bear: proven, directional, untested, or judge-dependent. It exists so an n=1 result cannot quietly acquire a quantitative costume.
exemplar
The worked example embedded in a prompt to show the model what good output looks like. Ours demonstrates the very defect it tells the model to avoid, which is a live suspect for why the instruction is ignored.
incumbent
gpt-3.5-turbo, the model writing quizzes in production today. It is the control arm: the only baseline that can show whether the cutover makes quizzes worse.
rescore
Re-judging quizzes we already generated, with a different judge and the option order held identical, so the only thing that changed is who is grading. Without holding the quizzes fixed, a judge comparison means nothing.
run of record
The single run whose numbers we quote everywhere: 36 quizzes across four stories, three models and three replicates. Other runs exist; when a figure appears without qualification, it came from this one.
the cutover (the production cutover)
Replacing gpt-3.5-turbo in production before it shuts down on 2026-10-23. This study exists to make sure the quiz half of that decision is made on evidence rather than assumption.