Brief 1 -- Item-Writing Craft: Assessment Science for Non-Leaking Multiple-Choice Items
Last updated: 2026-08-17
This study research team, model-agnostic assessment-science axis. Scope: what item-writing science knows about option sets that do not leak their key, converted into rules a prompt, a schema, or a validator can carry.
Every claim below is tagged [CITED], [MEASURED], or [QUEUED] per the team evidence standard.
Headline calls
- Adding a length rule to the prompt will not fix Defect 2, because the length rule is already in the prompt and all three models ignore it. [MEASURED]
generateQuestionsPromptForFullStory.ts:62 already reads "All wrong answers must be similar in length and complexity to the correct answer," and the sibling paragraph prompt carries the same instruction at line 38. Against that instruction the measured key-is-longest rate is 48 / 54 / 46 percent. The rule is present, positively phrased, and unheeded. Route option-length control to a code-side validator with a hard numeric threshold. Prompt text is the wrong instrument for this rule and we have direct evidence of its failure.
- The prompt's single worked example is cued, but on specificity rather than length -- and only one of five operational definitions flags its length at all. [MEASURED] In the exemplar at
generateQuestionsPromptForFullStory.ts:70-84 the key is 56 characters / 10 words against distractors of 46, 46, and 49 characters (8, 9, and 10 words). It is strictly the longest in characters only; in words it ties. It clears the published loose threshold (0.875), clears symmetric homogeneity (0.821), and clears the orchestrator's 1.25x-mean detector (1.191). Only the strict published definition flags it. But it is unambiguously the only causal, interpretive option against three concrete-action distractors -- and no length rule measures that. Fix the exemplar, but fix its specificity shape, not just its character counts. Section 8 works this through.
- Anthropic's own documentation names our exact failure mode as the reason to use more than one example. [CITED] Its prompting guidance says examples must be "Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns," and recommends 3-5 examples. We ship one. A length-*balance* constraint is a claim about within-item variance, and a single exemplar cannot demonstrate variance -- it can only demonstrate one shape, which the model is then free to read as the target shape.
- The fact-heavy, vocabulary-starved skill mix is not a model behavior. It is exactly what the prompt asks for. [MEASURED] Reading
getQuestionDistribution() at generateQuestionsPromptForFullStory.ts:21-38: main idea is never named at any grade band; vocabulary appears only at grades 3-5 and only as one arm of a disjunction ("explain why something happened or what a word/phrase means"); grades 6+ names neither. The measured unprompted mix (7 main-idea and 3 vocabulary out of 140) is the deterministic consequence of that text. Blueprint enforcement is a prompt-plus-schema problem, and the authoritative targets exist: NAEP reserves 15-20 percent of items in every block for vocabulary/language-structure [CITED], and PIRLS grade 4 runs 20 / 30 / 30 / 20 across its four processes [CITED].
- The instruction "wrong answers must use words or phrases from the story" is half-right and is probably contributing to the leak. It correctly implements the similarity strategy for distractor development [CITED: Gierl et al. 2017], but it is applied to the distractors only. Nothing constrains the key. So the model writes distractors from recycled story vocabulary and writes the key as an original, fuller restatement of what the story actually meant -- which makes the key the most specific and most detailed option by construction. **The rule must be made symmetric: it is the option set, key included, that has to be homogeneous in content, structure, and length.**
- Do not reduce to three options, even though the meta-analytic literature favors it. [CITED] Rodriguez's 80-year meta-analysis concludes three options are optimal, and Haladyna et al. concur. But NAEP single-selection multiple choice uses four [CITED], Alabama's grade 3-8 ELA program uses four [CITED], and our production schema, UI, and per-answer explanation contract are all built on four. The measured gain is in items-per-testing-minute, which is irrelevant to a fixed 4-item quiz. Keep four; spend the effort on making the third distractor functional instead.
- The blind-grader result (17 of 17 on cued items with the story withheld) is not an anomaly to explain away -- it is the standard partial-input validity check, and its failure is disqualifying under an explicit federal item-writing standard. [CITED] NAEP 2026 specifications: "items should not be answerable by readers only drawing upon text-independent domain knowledge, without even reading the passage." Make the partial-input check the acceptance gate for the cutover, not a diagnostic.
- Our 48 / 54 / 46 percent is a floor, not a measurement, because the 1.25x-mean threshold is looser than either published definition and matches neither. [MEASURED] Scored against our own exemplar, the 1.25x-mean rule is the most permissive of the five operationalizations I tested. It is also not a threshold anyone in the literature uses, so our headline number is not comparable to any published figure. Under the strict published definition the true rate is necessarily higher than 48 / 54 / 46. Re-score the existing corpus under the published definitions before quoting the number anywhere external. Section 8.3.
Single highest-confidence recommendation: implement a code-side option-set validator using the two published operationalizations -- Moore et al.'s loose rule (key is longest and second-longest is under 80 percent of longest) as the cue gate, plus symmetric homogeneity (shortest / longest ≥ 0.80) as the NAEP parallel-length check -- measured in characters, run as a generate-then-repair loop. Pair it with 3-5 exemplars whose keys occupy different length ranks. That combination attacks the measured defect with a deterministic instrument, is model-agnostic, and survives the gpt-3.5 cutover regardless of which candidate wins.
1. The guideline taxonomies
1.1 Haladyna, Downing & Rodriguez (2002) -- the canonical 31 [CITED]
The field's reference taxonomy. Validated against 27 educational-measurement textbooks and 27 post-1990 research studies. Organized as Content (1-8), Formatting (9-10), Style (11-13), Writing the Stem (14-17), Writing the Choices (18-31).
The fourteen option-set rules, verbatim:
Honest caveat that matters for how hard we lean on this. Guidelines 24 and 28 -- the two rules that bear most directly on our defect -- carry the highest textbook consensus (85 and 96 percent, zero dissent) and the thinnest direct empirical validation in that review. The authors report that post-1990 research clustered on only four guidelines (17 negative stems, 18 number of options, 25 NOTA, 26 AOTA), and they write that plausibility of distractors (29) "continues to be an area that is long overdue for study." So "keep options equal in length" entered the canon as expert consensus, not as a measured effect. The measured evidence for it comes from the test-wiseness and item-flaw literatures instead -- section 2 below. Report it that way to anyone who asks.
Guidelines outside the option set that bind on us: 8 keep vocabulary simple for the group being tested (70 / 30 / 0; supported by Abedi et al. 2000 on simplified language reducing construct-irrelevant reading load); 13 minimize the amount of reading in each item (67 / 33 / 0); 15 include the central idea in the stem instead of the choices (100 / 0 / 0 -- the only unanimously cited guideline in the taxonomy); 7 avoid trick items.
1.2 Gierl, Bulut, Guo & Zhang (2017) -- the distractor-specific synthesis [CITED]
Review of Educational Research 87(6):1082-1116. Cross-tabulates six published guideline sets. Rules with full six-of-six consensus: use plausible distractors, place distractors in logical or numerical order, keep distractors independent / non-overlapping, NOTA and AOTA used carefully, avoid giving clues to the right answer. Slightly lower consensus: incorporate common errors of students, keep distractors homogeneous in content and grammatical structure, phrase distractors positively.
The single most load-bearing sentence in this paper for us, on the similarity strategy: "Structural similarities include distractors that share characteristics with the correct option such as length, complexity, formatting, and grammar." Homogeneity is not a stylistic nicety layered on top of distractor development -- in the canonical account it is one of the two distractor-development strategies.
1.3 NAEP 2026 Reading Assessment and Item Specifications -- an official K-8 standard [CITED]
This is the highest-authority source for us because it is grade 4 and grade 8 reading comprehension, four-option single-selection multiple choice, from the US federal assessment governing board. Its selected-response development guidelines, verbatim:
- The stem includes only the information needed to make the student's task clear.
- Options are as short as possible and are parallel in length.
- Options are parallel in structure, syntax, and complexity.
- Options do not contain inadvertent cues to the correct answer, such as repeating a word from the stem in the correct answer or using specific determiners (e.g., all, never) in the distractors.
- Distractors are plausible, but not so plausible as to be possible correct answers.
- Distractors are designed to reflect the measurement intent of the item, not to trick students into choices that are not central to the idea being assessed.
Two of these are new relative to Haladyna and worth adopting verbatim. "As short as possible" gives a direction, not just a parity constraint -- it forecloses the failure mode where a model equalizes lengths by inflating the distractors to match a bloated key. And "not so plausible as to be possible correct answers" is the guardrail on over-correcting plausibility, which is the predictable side effect of pushing a model hard on distractor quality.
1.4 The operational rubric: SAQUET's 19 item-writing flaws [CITED]
Moore, Costello, Nguyen & Stamper (2024). This matters because it is the taxonomy translated into running code, open-sourced, with per-flaw accuracy measured against human raters -- exactly the artifact a validator should be modeled on.
The 19 criteria: Longest Option Correct, Ambiguous Information, Implausible Distractors, True or False, Absolute Terms, Complex/K-type, Negatively Worded, Convergence Cues, Lost Sequence, Unfocused Stem, None of the Above, Word Repeats, More Than One Correct, Logical Cues, All of the Above, Fill in the Blank, Vague Terms, Grammatical Cues, Gratuitous Information.
Detection tiers, which map one-to-one onto our routing decision:
- Pure string/rule matching (8 criteria): None of the Above, All of the Above, Fill-in-the-Blank, True or False, Longest Option Correct, Negatively Worded, Lost Sequence, Vague Terms.
- Classical NLP -- embeddings, NER, RoBERTa (5 criteria): Implausible Distractors, Word Repeats, Logical Cues, Ambiguous/Unclear, Grammatical Cues.
- Requires an LLM call (6 criteria): Absolute Terms, More Than One Correct, Complex/K-type, Gratuitous Information, Unfocused Stem, Convergence Cues.
Overall 94.13 percent accuracy across 5,149 classifications on 271 MCQs; Hamming loss 5.9 percent; exact-match ratio 38 percent. Per-domain F1 for Longest Option Correct: 0.77 / 0.60 / 0.96 / 1.00 / 0.97. Word Repeats F1: 1.00 / 1.00 / 0.56 / -- / 0.53.
The threshold question, answered. The paper reports that human raters used two different operational definitions of Longest Option Correct and names both: a loose rule flagging an item when the second-longest option is not at least 80 percent as long as the longest, and a strict rule flagging "any question where the correct answer exceeded others in length by even a single character." We should implement both and report both, because this study's 48/54/46 percent figure is presumably closer to the strict definition, and the loose definition is what we can realistically hold a generative model to.
Their acceptance policy, adoptable as-is: an MCQ with zero or one flaw is acceptable for formative use; two or more flaws is unacceptable. That gives us a defensible, externally sourced quiz-level gate rather than a number we invented.
2. Test-wiseness cues
2.1 The frame [CITED]
Millman, Bishop & Ebel (1965) defined test-wiseness as "the subject's capacity to utilize the characteristics and formats of the test and/or the test taking situation to receive a high score," explicitly independent of knowledge of the subject matter. Our Defect 2 is a textbook test-wiseness vulnerability, and the framing matters for how we talk about it: a length cue does not make a quiz wrong, it makes the quiz measure something other than reading.
2.2 The primary target: option length and specificity
Measured prevalence and payoff in real tests. [CITED] Brozo, Schmelzer & Spires (1984) analyzed 1,220 multiple-choice items from 43 teacher-made college examinations against the Millman taxonomy. 539 items (44 percent) contained at least one clue; 375 of those (70 percent) were answerable by applying the clue, against a 25 percent chance baseline (z = 24.10, p < .001). Per-clue:
Note their operationalization of "longest": at least one full line of print more than the other alternatives -- a much coarser bar than ours, which is why their 76 percent understates what a character-level definition would find.
Measured psychometric damage. [CITED] Schmucker & Moore (2025), 7,126 MCQs and 448,000 student responses from CK-12 across Life/Earth Sciences, Physical Sciences, and Mathematics, flaws labeled by SAQUET at 94 percent agreement with humans, outcomes modeled as IRT parameters. Longest Option Correct: discrimination γ = -0.370 (p < .05), difficulty β = -0.691 (p < .05). It is the flaw with the largest adverse coefficient on both parameters simultaneously. Items with a length cue are both easier and worse at separating strong readers from weak ones -- which is precisely the harm a comprehension quiz cannot absorb. For contrast, Convergence Cues (β = +0.679) and Grammatical Cues (β = +0.526) made items harder, not easier.
A conflicting result, reported as required. [CITED] Pham, Besanko & Devitt (2018) compared 98 matched pairs of flawed and unflawed MCQs and found no statistically significant difference in difficulty or discrimination indices between the groups overall (difficulty p-values .198-.8845; discrimination p-values .1359-.8705). Within that null, "longest choice is correct" did significantly raise mean item scores (p = .047) -- items got easier -- but the discrimination penalty Schmucker & Moore report did not replicate. The two studies differ by roughly 70x in sample size and by outcome model (classical indices vs. IRT parameters), which favors the larger study, but the honest statement is: the "easier" effect is replicated; the "less discriminating" effect rests on one large study. Do not present the discrimination claim as settled.
Why "most specific" and not "most general" for us. Brozo found the general option was correct 81 percent of the time; this study found the specific and detailed option is the key. These are not in conflict, they are two different authorship failure modes. A human writing about facts they half-know hedges the key into generality. A model writing about a story it has fully in context writes the key as the fullest accurate restatement of what happened, then invents thinner distractors. Same cue, opposite polarity. **Practical consequence: the classic "watch out for the most general option" heuristic does not transfer, and any rule we write must target asymmetry of specificity, not a direction.** Haladyna 23 (homogeneous content and structure) plus 24 (equal length) plus NAEP's "parallel in structure, syntax, and complexity" are the correct rules; "avoid the general option" is not.
2.3 Secondary cues, ranked by whether they can bite us
- Clang / key-word association. Highest-yield cue in Brozo (100 percent, 38/38). NAEP forbids "repeating a word from the stem in the correct answer." Relevant to us and cheap to check, because our stems quote the story and our keys restate it. Note carefully: SAQUET's Word Repeats is defined as words repeated only between the stem and the correct response -- a set-difference computation, not a raw overlap count. That definition is the right one to implement; a naive stem-key overlap counter will fire on every fact-detail item.
- Grammatical agreement with the stem. Zero occurrences in 1,220 human-written items; SAQUET routes it to NLP. Low priority. Our items are near-perfect on format contract already and LLMs are fluent; this is unlikely to be our failure mode. Worth one measurement, not a prompt line.
- Absolute qualifiers / specific determiners. Brozo's only non-significant knowledge cue (35 percent, p = .075). NAEP nevertheless bans them in distractors. Include as a validator rule (cheap word list) but do not spend prompt budget on it.
- Convergence cues. Options built from different combinations of shared components, so the most-frequently-appearing components identify the key. Structurally near-impossible in narrative comprehension options and it raised difficulty in Schmucker & Moore. Deprioritize.
- None of the above / all of the above. Textbook opinion is genuinely split on NOTA (44 percent for, 48 percent against) and all five post-1990 studies agree NOTA raises difficulty. AOTA is more clearly condemned -- Harasym et al. found reliability significantly reduced and attributed near-universal AOTA selection to cuing. For K-8 narrative comprehension both are simply out of scope; forbid at schema level (a regex validator) and spend no prompt text on them.
- Direct opposites. Brozo's single most frequent cue (151 items, 76 percent). Fires when a writer generates a distractor by negating the key. Plausible for LLM distractor generation and currently unmeasured for us. Queued below.
- Position bias. Brozo found position "C" non-significant in human tests (26 percent vs 25 percent chance). But this is different for machine authors: LLMs are documented to have position biases in generation, and the classical remedy (Haladyna 20, Mosier & Price 1945) is randomization. This is trivially and completely solved by shuffling options at serving time, in code, with the key index stored. It should never be a prompt concern.
2.4 The measurement method is itself citable [CITED]
The blind-grader design this study already ran is the standard partial-input baseline from NLP evaluation. Kaushik & Lipton (2018) established passage-only and question-only baselines across bAbI, SQuAD, CBT, CNN and Who-did-What and found them "surprisingly" strong, concluding that benchmarks solvable from partial input do not measure what they claim. Balepur, Ravichander & Rudinger (2024) ran the choices-only variant on three MCQA datasets and four LLMs: the choices-only prompt beat a majority baseline in 11 of 12 cases, with gains up to 0.33 accuracy. Their causal analysis rules out memorization alone and implicates the group dynamics of the option set -- which is the same mechanism as a length cue.
This gives our acceptance gate an external methodological pedigree. The metric to institutionalize is the choices-only solve rate: give a judge model the four options and nothing else, and require the solve rate to be statistically indistinguishable from 25 percent. That is a single number, model-agnostic, defensible to any psychometrician, and it subsumes length, specificity, clang, opposites, and convergence in one measurement instead of a checklist.
3. Distractor development
3.1 The two sanctioned sources [CITED]
Gierl et al. identify exactly two strategies with sustained support:
Strategy 1 -- common errors and misconceptions. Haladyna & Rodriguez, quoted in full by Gierl as the current best statement: "The most effective way to develop plausible distractors is to either obtain or know what typical learners will be thinking when the stem of the item is presented to them. We refer to this concept as a common error. Knowing common errors can come from a good understanding of teaching and learning for a specific grade level; it can come from think aloud studies with students; or it can come from student responses to a constructed-response format version of the item without options."
Our prompt already does this: "Wrong answers should reflect mistakes a grade N student would genuinely make" (line 61). This line is correct, well-founded, and should be kept and strengthened, because it is the only line in the prompt that gives the model a generative principle for distractors rather than a constraint. Alabama's grade 3-8 ELA specification is the K-8 instantiation: ELA distractors "represent common misconceptions, incorrect logic, incorrect understanding of a text." [CITED] Smarter Balanced is more specific still: reading distractors should be plausible to students who (1) misinterpret details in the text, (2) draw erroneous conclusions or inferences, or (3) apply faulty reasoning about the text. That three-way split is directly promptable and is a better replacement for our vague "wrong for a different reason" line (63).
Strategy 2 -- similarity to the key. Content similarity (same category, concept, topic) plus structural similarity: length, complexity, formatting, grammar.
The honest caveat on both. Gierl et al. are explicit that the misconception route requires three assumptions to hold -- that content specialists can specify plausible errors, that distractors can be produced from them, and that the specified misconceptions are the ones students actually hold -- and that this alignment "must occur for each distractor." They conclude that writing distractors that measure plausible misconceptions "and replicating this outcome consistently" is hard. This is a warning against believing a model when it claims a distractor encodes a grade-appropriate error. The claim is unfalsifiable without response data. Which leads to:
3.2 How many distractors actually work [CITED]
Haladyna & Downing (1993), across four standardized tests: roughly two thirds of items had one or two effective distractors; only 1 to 8 percent had three; the modal number of effective distractors per item was one. Items with more effective distractors discriminated better, though effectiveness was unrelated to difficulty. The conventional threshold for a nonfunctioning distractor is a selection rate below 5 percent.
Haladyna et al.'s conclusion is blunt: "The effort of developing that fourth option (the third plausible distractor) is probably not worth it. If the use of four options is preferred, empirical research has established that it is very unlikely that item writers can write three distractors that have item response patterns consistent with the idea of plausibility."
Rodriguez (2005), the 80-year meta-analysis, reaches the same place from the psychometric side: three options are optimal, mainly because more items fit per unit of testing time and because implausible distractors get eliminated rather than tolerated.
Our call, restated with the reasoning. We keep four, because the efficiency argument that drives the three-option finding does not exist for a fixed 4-item formative quiz, and because every K-8 reading program we can cite -- NAEP single-selection MC, Alabama ACAP, Nebraska NSCAS -- uses four. But we should stop expecting all three distractors to be functional. That is not a realistic target for any author, human or model. The right posture: require three distractors, hold all four options to structural homogeneity, and measure functionality post-hoc from real student response data rather than asserting it at generation time.
3.3 Verdict on our passage-vocabulary instruction
The instruction under review, generateQuestionsPromptForFullStory.ts:60: "Wrong answers must use words or phrases from the story in plausible but incorrect ways."
Verdict: keep the intent, fix the asymmetry, and expect it to be a contributor rather than the cause.
Supporting it: this is a legitimate expression of Gierl's similarity strategy, and for narrative comprehension the story is the content domain, so drawing distractor material from it is the only way to keep options in the same content category (Haladyna 23). Removing the instruction outright would risk distractors sourced from outside the story, which are trivially eliminable and would make the leak worse, not better.
Against it, three specific problems:
- It is asymmetric. The rule binds only the wrong answers. Nothing tells the model to build the key from the same material at the same grain. Under that instruction the natural output is exactly what was measured: distractors made of recycled story surface, key made of an original and fuller proposition. The specificity gap is manufactured by the prompt's own asymmetry.
- It pushes toward the wrong distractor archetype. "Use words from the story in incorrect ways" produces surface-recombination distractors -- true story words in a false arrangement. Gierl's strategy 1 and Smarter Balanced both want reasoning-error distractors -- what a student concludes when they misread. The former are eliminable by anyone who read the story; the latter require actually understanding it. This is likely a second-order contributor to weak distractor functionality.
- It interacts badly with skill tagging. PIRLS supplies an operational classification rule: [CITED] "If the item stem and the correct response both use exact words from the text and are located with a sentence or two of each other, the item is classified as Focus and Retrieve... if the item stem and correct response use paraphrases of the original phrases or sentences in text then the item is classified as Straightforward Inferencing." Lexical overlap between the passage and the key is the operational marker of the lowest-order comprehension process. Instructing heavy passage-vocabulary reuse in distractors, while the key paraphrases, systematically stamps every item with the surface signature of an inference item whose distractors look like retrieval items. That is precisely the shape a length-and-specificity cue takes.
Rewrite direction (prompt-expressible): make the constraint apply to the option set. Something in the shape of -- all four options draw on the same story material, at the same level of detail and the same grammatical form; a reader who has not read the story must not be able to tell which one the story supports. Pair it with the Smarter Balanced three-way distractor typology so the model has a generative principle, not just a parity constraint. NAEP's ceiling clause ("plausible, but not so plausible as to be possible correct answers") is the necessary counterweight, and its inclusion is what keeps this from degrading answer-key soundness -- which this study measured as currently excellent and which we must not regress.
4. Grade calibration for K-8
4.1 What the standards actually control [CITED]
- Text complexity bands (CCSS Appendix A, updated multi-measure table): grades 2-3 Lexile 420-820 / Flesch-Kincaid 1.98-5.34; grades 4-5 Lexile 740-1010 / FK 4.51-7.73; grades 6-8 Lexile 925-1185 / FK 6.51-10.34.
- Passage length by grade (Alabama ACAP ELA): grade 2 150-350 words; grade 3 550-650; grade 4 600-700; grade 5 650-800; grade 6 650-800; grade 7 750-850; grade 8 750-850.
- Vocabulary inclusion criteria (NAEP 2026, Exhibit 3.13). Included: words appearing across many texts (despise, benevolent, resolution, commit); words needed to understand a local part of the context linked to central ideas; words labeling familiar concepts even if the word itself is unfamiliar to younger readers (timid); words with useful roots and affixes (disregard, counterargument); language expressing logical relations (although, in contrast); anaphoric expressions referring to previously introduced entities (those alliances, this phenomenon). Excluded: obscure or discipline-specific words (fiduciary, apotheosis) and idioms (spill the beans, up in the air).
- Language load in items: Haladyna 8 and 13, supported by Abedi et al. (2000) on simplified item language reducing construct-irrelevant reading demand.
Important scope note: the Lexile and FK bands above govern the passage, not the item. Item language should sit at or below the passage band -- an option written above the passage's readability measures reading load, not comprehension. That is an inference from Haladyna 8 and 13, not a directly cited threshold; treat the specific numeric target as [QUEUED].
4.2 Why our length cue climbs with grade -- three candidate mechanisms
Our measured escalation is 29 percent at grade 1 to 83 percent at grade 7, and it persists with question type held constant to fact-detail. That last fact is the important one: it rules out the simplest explanation (that higher grades just get more inference items, and inference keys are naturally longer). Three mechanisms remain, in descending order of my confidence.
M1 -- Grade drives an implicit register shift in the prompt, and register drives length. The prompt's grade text is not only a type distribution, it is a tone instruction. Compare grade ≤2 ("2 questions about facts directly stated in the story") to grade 6+ ("1 question about a detail that matters to the plot (required for the next questions)... requiring evidence-based inference -- the reader must cite specific story details"). The grade-6 framing conditions the whole generation toward elaborated, evidence-citing prose. A model conditioned to write elaborately writes an elaborate key -- because the key is the option it must make defensibly true -- while distractors, being invented, stay short. Under this mechanism the cue would persist within fact-detail items, which matches the measurement. This is my leading hypothesis and it is cheap to test.
M2 -- Grade-band ceiling compression at the low end. At grade 1 every option is a short simple clause and there is little room for a length gap; the constraint is the option floor, not the model's restraint. As the allowed complexity rises, the gap has room to open. This predicts the magnitude of the gap grows with grade even where the rate would not, and it predicts grade-1 items are protected by accident rather than by design -- meaning any change that lengthens grade-1 options will surface the cue there too. Worth knowing before we "improve" grade 1.
M3 -- Higher cognitive demand correlates with more item-writing flaws, independent of grade. [CITED, indirect] Moore et al. report that in their 271-item corpus, questions carrying two or more item-writing flaws had higher Bloom's Revised Taxonomy levels in four of five domains. That is a measured association between cognitive demand and flaw density in human- and student-authored items, from an entirely separate corpus. It does not establish causation and it is not grade-stratified, but it is the only external evidence I found that the direction of our escalation is a general property of harder items rather than an artifact of our prompt.
None of these are settled. Experiments in Open Questions.
4.3 What grade calibration should actually control, as rules
- Option word count ceilings scaled by band, not free-form "grade-appropriate." Absolute ceilings are guessable from passage-length ratios but are not directly cited; [QUEUED].
- Option vocabulary drawn from the NAEP inclusion list and screened against the exclusion list -- promptable, and the exclusion half (no idioms, no obscure terms) is validator-checkable against a word list.
- Readability of the option set held at or below the passage's band.
- Critically: the length-parity threshold should NOT be relaxed at higher grades. The temptation will be to allow more variance where language is richer. The escalation data says the opposite -- the higher grades are where the enforcement has to be tightest.
5. Reading-comprehension skill taxonomies and the blueprint
5.1 Which framework to adopt
Two candidates, both authoritative, and they mostly agree.
NAEP 2026 Comprehension Targets (grades 4, 8, 12): Locate and Recall; Integrate and Interpret; Analyze and Evaluate; Use and Apply.
PIRLS 2021 Processes of Comprehension (grade 4): Focus on and Retrieve Explicitly Stated Information; Make Straightforward Inferences; Interpret and Integrate Ideas and Information; Evaluate and Critique Content and Textual Elements.
Mapping onto our five components:
Recommendation: adopt PIRLS as the primary spine and borrow NAEP's vocabulary treatment. Reasons: PIRLS is grade-4 literary-narrative reading, which is our exact genre and near our median grade; it splits inference into two levels, which our single "inference" bucket currently conflates and which is the split that most affects difficulty; it publishes flat per-process percentages rather than wide ranges, so it is directly enforceable; and it supplies an operational classification rule (section 3.3) that a validator can compute. NAEP's fourth target, Use and Apply, requires applying text to an external task and does not fit a 4-item quiz about a generated story -- drop it.
The vocabulary structural insight, from NAEP, is the most useful single idea in this section. NAEP does not treat vocabulary as a fifth question type. It double-codes: every vocabulary item is also a comprehension-target item, because "language structures and vocabulary will not be measured independently from Comprehension Targets." A total of 15-20 percent of items in any block carry the vocabulary code. This dissolves our blueprint arithmetic problem. With only four items, five mutually exclusive types cannot be distributed sensibly. As an orthogonal flag it works: four items across four processes, one of which additionally carries the vocabulary flag. That maps to 25 percent, slightly above NAEP's 15-20 percent band, which is the correct direction of error given our measured 2 percent.
5.2 Per-grade blueprint
Published targets, for reference:
Both frameworks state the same grade principle: the proportion of higher-order targets rises across grades and Locate-and-Recall falls. NAEP: "Moving across grades, the proportion of higher level Comprehension Targets increases."
A defensible 4-item-per-quiz blueprint that respects both, one item per process:
- Grades 1-2: 2 retrieve, 1 straightforward inference, 1 interpret/integrate (phrased as "tell what the story is mostly about" -- main idea at this band). Vocabulary flag on one item.
- Grades 3-5: 1 retrieve, 1 straightforward inference, 1 interpret/integrate (main idea or character motivation), 1 evaluate/analyze. Vocabulary flag on one item.
- Grades 6-8: 1 retrieve, 1 interpret/integrate, 1 evaluate/analyze, 1 further interpret/integrate or evaluate. Vocabulary flag on one item.
Note this is close to what our prompt already asks for at grades 3-5. The measured gap (61 of 140 fact-detail against a 25 percent target) is therefore not a blueprint-design failure. It is an enforcement failure, which is the whole argument of the next point.
5.3 How a blueprint gets enforced
NAEP is explicit that even for a professional item-development shop, distribution is monitored at the pool level rather than guaranteed per block, and that content quality outranks the distribution target: "for a given text, it is better to fall one item short in the number of items for a target than it is to include one item that fails the importance or the clarity standard just for the sake of meeting the distribution goal."
For us, three mechanisms in ascending order of strength:
- Prompt prose -- what we have now. Measured to under-deliver by a wide margin (44 percent fact-detail against a 25 percent instruction).
- Schema-level: a required per-item enum field. Add a mandatory
skill field with a closed enum (retrieve | inference | interpret_integrate | evaluate_analyze) and a required boolean vocabulary_focus. This forces the model to commit to a label per item, which is a materially different cognitive task from following a distribution instruction, and it makes the distribution machine-checkable in one pass. It also directly serves the skill-taggability metric this study measured (94 / 88 / 75 percent) by moving tagging from post-hoc judgment to generation-time declaration.
- Validator: count the enums, reject and regenerate on violation. With a schema field this is a trivial check. Without one it requires a judge model.
Note the failure mode of level 2 alone: the model can satisfy the enum by labeling a fact-detail item inference. Declared-cast approaches have this exact weakness. This is where the PIRLS lexical-overlap rule earns its keep: compute the verbatim overlap between (stem + key) and the passage, and cross-check the declared label against it. High exact-phrase overlap with a declared inference label is a flag. That is a real, cheap, code-side falsifier for a self-declared tag, and I have not seen it proposed elsewhere in the harness.
6. Stem construction
The rules that bear on passage-based comprehension items, all [CITED]:
- Haladyna 15 -- central idea in the stem, not the choices. The only guideline in the taxonomy with 100 percent textbook citation and endorsement. The failure mode is the unfocused stem, where the reader cannot tell what is being asked without reading all four options. Direct relevance: an unfocused stem forces the option set to carry the content, which mechanically inflates options and pushes the key toward being the most complete one. This is a plausible upstream cause of our length cue and it is not currently addressed anywhere in the production prompt. Empirical caveat: Downing et al. (1991) found no significant difficulty or discrimination effect from unfocused stems, though their effects were confounded with heterogeneous options and their examinees were medical students, whom the authors suggest may be inoculated by test experience. That caveat does not transfer to seven-year-olds.
- NAEP -- "the stem includes only the information needed to make the student's task clear." Same rule, official K-8 phrasing.
- Haladyna 16 -- avoid window dressing. Note that "stems with window dressing" is one of the empirically identified defining characteristics of a trick item (Roberts 1993).
- Haladyna 17 -- word the stem positively. This is the guideline with the most textbook dissent among stem rules (63 for / 19 uncited / 18 against) and the most mixed empirical record: three of four researchers found no difficulty effect from negatively worded stems, and Tamir found negation increased difficulty only for higher-cognitive-level items. For K-8 readers the caution should stand regardless of the mixed adult-population evidence.
- Haladyna 13 -- minimize the amount of reading in each item. Our stems already quote the story; this constrains how much.
- NAEP unfocused-stem test, operationalized by SAQUET: "The stem is not a clear and focused question that can be understood and answered without looking at the options." That is directly checkable with a judge model: show the judge the story and the stem without options and ask whether the task is clear.
7. Routing: prompt vs. schema vs. validator
This is the section the orchestrator needs. Confidence is my own judgment unless marked otherwise.
Two cross-cutting notes for the orchestrator.
On the earlier negative-constraint lesson. Several rules above (R7, R8, R9) are natural "do not" statements. Per that finding, gpt-3.5-turbo returned error objects when given DO NOT phrasing, and the harness rewrote everything positively. This tension resolves cleanly if those rules live in the validator rather than the prompt, which is where I have routed all three. Only R11 and R17 carry unavoidable negative semantics in prompt text, and both can be phrased positively ("each distractor should be clearly wrong to a reader who understood the story"; "ask what is true"). Recommendation: keep the production prompt free of DO NOT phrasing regardless of which model wins, and put the prohibitions in code. That makes the prompt model-agnostic and sidesteps the Nova-vs-Claude negative-constraint conflict entirely.
On generate-then-repair. Every hard validator rule (R1, R2, R6, R7, R8, R21) is computable in milliseconds with no model call. That makes a repair loop cheap: validate, and on failure return the specific violated constraint to the model for a targeted rewrite of the option set only. This is strictly better than trying to get it right in one shot, and it is the only approach with a deterministic guarantee on the length cue.
8. Demonstrations versus stated rules, and the operational definition of the cue
Added in response to the orchestrator's two measurements. This section answers three questions: what the literature says when a worked example and a stated rule conflict; how many exemplars a length-balance constraint needs; and whether our cue detector's threshold is defensible.
8.1 When a worked example and a stated rule conflict
There is no assessment-science literature on this, because item-writing guidelines were written for human authors who read rules and do not receive exemplars in the prompt. The relevant evidence is in two other literatures, and both point the same way.
**Machine-learning evidence: demonstrations transmit format most strongly of all. [CITED]** Min et al. (2022), across 12 models including GPT-3, found that randomly replacing the labels in demonstrations "barely hurts performance," and identified the actual drivers as three structural properties of the demonstrations: the label space, the input distribution, and the overall format of the sequence. The direct implication for us is uncomfortable and specific: the part of an exemplar a model reliably absorbs is its shape, and the part it can shrug off is whether the content is right. An exemplar whose option set has a particular length-and-specificity profile is transmitting exactly the channel Min et al. identify as dominant, whether or not the surrounding prose says otherwise.
Vendor evidence: this is a documented, named failure mode. [CITED] Anthropic's prompting best-practices guidance calls examples "one of the most reliable ways to steer Claude's output format, tone, and structure," and then instructs that examples be "Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns." "Unintended patterns" is precisely the hypothesis under test. The vendor documents the risk and prescribes the remedy in the same sentence.
A caution against over-claiming. [CITED] Recent work on semantic anchoring in in-context learning finds that a demonstration's ability to override a model's priors increases with model size, and that smaller models cannot flexibly remap semantics from demonstrations alone. That cuts both ways for us and is worth stating honestly: it means exemplar contamination is not guaranteed to be the dominant mechanism, and it predicts the effect should differ between our candidates -- plausibly stronger in claude-haiku-4-5 than in nova-2-lite, or vice versa depending on which is more demonstration-driven. The near-identical measured rates across three models of quite different sizes (48 / 54 / 46) are weak evidence against a purely exemplar-driven account, since a size-dependent mechanism would be expected to produce more spread. Exemplar contamination is a live and testable hypothesis, not an established root cause, and the flat cross-model profile is a real piece of counter-evidence that should be stated when this is presented.
The assessment-side analogue is real and worth borrowing. [CITED] In Automatic Item Generation, a parent item (also called an item model, schema, item shell, or template) is manipulated to produce many child items, and the children "have the same 'template' of information as the parent item." Under the weak-theory approach, a parent is chosen because it has good psychometric properties, precisely because its structure propagates to every variant. Our exemplar is functioning as a parent item. Framing it that way gives us an established discipline to point at: in AIG nobody would ship a parent item that had not been screened against the item-writing-flaw rubric, and we are shipping one that has not been.
Reframed rule, per the orchestrator's instruction. For each rule in section 7 the question is now "why would a model ignore this, and what layer enforces it." Three mechanisms explain ignoring, and they route differently:
The second row deserves emphasis, because it supplies a mechanism rather than a guess, and it is cited. [CITED] Tarrant and colleagues characterize longest-option-correct as "a common mistake made by novice and experienced examination writers in an effort to ensure the correct response is indisputable." That is not a slip -- it is a rational response to the pressure to make the key defensible. Our prompt applies exactly that pressure at line 66 by requiring the key's explanation to quote the story. The length cue is therefore partly an emergent consequence of a rule we want to keep, which is a strong argument that it must be corrected downstream in code rather than argued away upstream in prose.
8.2 How many exemplars a length-balance constraint needs
The specific question is unanswered in the literature. No source I found states how many demonstrations a within-item homogeneity constraint requires before it transfers. That is a real result and I am reporting it as one. What can be said:
- [CITED] Anthropic's guidance: "Include 3-5 examples for best results," with diversity that prevents unintended patterns. This is vendor documentation, admissible, and directly applicable to one of our two candidates. It is not a measured finding about length balance.
- [QUEUED, reasoned] A structural argument that I believe is sound but have not tested: length balance is a constraint on within-item variance, and variance cannot be demonstrated by a single sample. With one exemplar the model observes one option-length profile and has no basis to distinguish "the key happens to be longest here" from "the key should be longest." The minimum informative demonstration set is therefore one in which the key occupies a different length rank in each exemplar -- longest in one, middle in another, shortest in a third. That makes length rank visibly uninformative about correctness, which is the actual property we want, rather than showing three more instances of near-equal lengths and hoping the model infers the invariant. Three exemplars is the minimum that can express this; 3-5 aligns it with the vendor recommendation.
- [QUEUED] Whether rank-varied exemplars outperform equal-length exemplars is untested and is the experiment I would run first. Design in Q11 below.
I want to be explicit that the second bullet is reasoning, not evidence. It is the kind of claim that sounds obviously right and could easily be wrong -- a model might generalize "keep them close" from three near-equal examples perfectly well, and rank-varied exemplars might instead teach it that variance is acceptable. The experiment is cheap and should be run before this is treated as a finding.
8.3 Is "strictly longest AND 1.25x mean" defensible?
Short answer: the "strictly longest" conjunct is right and matches the literature. The "1.25x mean" threshold is ours alone, is the most permissive of the definitions in use, and should be replaced.
There are five operationalizations in play. Two are published, one is Brozo's original, one is the orchestrator's, and one is the symmetric homogeneity check the standards actually ask for. Scored against our own exemplar (key 56 chars / 10 words; distractors 46, 46, 49 chars / 8, 9, 10 words):
Four findings from that table:
- Four of five definitions clear our exemplar. My earlier characterization of it as plainly key-cued was too strong on the length axis. The orchestrator's "strictly longest" is correct in characters; the item is nonetheless marginal by every threshold rule. What is not marginal is the specificity asymmetry, and no row in this table measures it.
- Character count and word count disagree on this very item. In words the key ties for longest and the strict rule stops firing. The unit of measurement is a load-bearing implementation decision, not a detail. Recommend characters: it is what Brozo's "lines of print" proxies, it is what a string-length implementation naturally does, and it captures the word-complexity dimension (a 10-word option built from longer words is denser) that word count discards.
- 1.25x-mean is more permissive than the strict rule and, for near-uniform distractors, roughly comparable to the loose rule -- but it is calibrated against a different reference point than the cue itself. The cue a test-wise reader exploits is "one option visibly stands out from the others," which is a comparison against the next longest option, not against the average. Averaging over three distractors dilutes the comparison whenever distractor lengths vary, so the mean-based rule's sensitivity moves with distractor variance in a way the actual cue does not. Brozo's operationalization ("more than the other alternatives") and Moore's (second-longest vs. longest) both use the next-longest reference. Ours is the outlier.
- Consequently our 48 / 54 / 46 percent is a lower bound and is not comparable to any published rate. Under the strict definition the rate can only be higher. This matters for how the number gets quoted: it is currently framed as "roughly half," and the true figure under the industry-strict rule is plausibly well above half.
Recommended replacement. Two rules, not one, because they answer two different questions and the standards ask for both:
- Cue gate (does the key stand out?): key is strictly longest in characters AND second-longest / longest < 0.80. This is Moore et al.'s loose rule, published, with an F1 of 0.96-1.00 against human raters in three of five domains. Report the strict variant alongside it as the pessimistic bound.
- Homogeneity gate (are the options parallel?): shortest / longest ≥ 0.80 across all four options, irrespective of which is the key. This is NAEP's "options are parallel in length" made checkable, and it catches the case the cue gate misses -- one conspicuously short throwaway distractor, which is the classic implausible-distractor signature.
Both are pure string length, both are free, and both should run on every generated item. Neither detects the specificity asymmetry, which remains a judge-model or choices-only-solve-rate problem (R5, R15).
Open questions
Each is an explicitly untested hypothesis with the experiment that settles it. None of these have been run; no model calls were made for this brief.
Q0 (do this before any generation). Re-score the existing 36-quiz corpus under all five operational definitions in section 8.3, in both characters and words. No model calls; the data exists. This settles what our actual defect rate is, tells us how much of the 48 / 54 / 46 figure was threshold artifact, and gives every subsequent experiment a comparable baseline. Until it is run, every rate in this program is measured with an instrument that matches no published definition.
Q1. Does fixing the worked exemplar alone move the length-cue rate? Hypothesis: the single one-shot example is a stronger signal than the prose constraint. Note the section 8.1 counter-evidence before believing this: the three models are near-tied at 48 / 54 / 46 despite differing substantially in size, and demonstration-override capacity is documented to scale with size, so a purely exemplar-driven mechanism predicts more spread than we see. Experiment: rerun the 36-quiz matrix with exactly one change -- the exemplar rewritten so the key is mid-rank in length and matched in specificity to its distractors -- holding everything else identical. Score with Q0's instrument. Vary the two exemplar properties independently across arms (length rank only; specificity only; both), because section 8.3 shows the current exemplar's length cue is marginal while its specificity cue is not, and a combined change would not tell us which mattered.
Q2. Is the grade escalation caused by register (M1) or by ceiling compression (M2)? Experiment: generate grade-1 and grade-7 quizzes on the same frozen story, then a third arm where the grade-7 quiz is generated with the grade-1 distribution text and the fourth with grade-1 content but grade-7 distribution text. If the cue tracks the distribution wording rather than the stated grade, M1 is confirmed and the fix is to neutralize the register of the grade text. If it tracks the stated grade regardless, M2 or M3.
Q3. Does an explicit numeric parity constraint in the prompt do anything at all? Hypothesis: it does not, because the qualitative version already fails. Experiment: one arm with "each answer choice must be between 6 and 12 words" against the current qualitative phrasing. Measure both compliance and collateral damage to answer-key soundness. A null result here is a genuinely useful finding, because it closes the door on prompt-only solutions and justifies the validator investment.
Q4. Does the "wrong answers must use words from the story" instruction help or hurt when removed? Experiment: three arms -- current instruction; removed entirely; rewritten symmetrically over all four options. Measure key-is-longest rate, choices-only solve rate, distractor implausibility (judge), and answer-key soundness. This directly settles the section 3.3 verdict, which is currently reasoned rather than measured.
Q5. Do LLM-authored distractors carry the direct-opposites cue? Brozo found it the most frequent cue in human-written items (151 of 1,220) but nobody has measured it for model authors. Experiment: judge-label each distractor as a negation of the key across the existing 36-quiz corpus. No new generation needed -- this runs on data we already have.
Q6. Does a required per-item skill enum actually change the item, or only the label? Hypothesis: it changes both, but less than we would like, and models will mislabel to satisfy the distribution. Experiment: two arms, prompt-only distribution vs. schema-required enum; blind-judge the true skill of every item; report the confusion matrix between declared and judged labels. The mislabeling rate is the number that decides whether R14's overlap cross-check is necessary or merely nice.
Q7. Does the choices-only solve rate track the length cue closely enough to be the single gate? Hypothesis: yes, and it is a better metric than any structural proxy because it is outcome-defined. Experiment: on the existing corpus, correlate per-item choices-only correctness against key-is-longest, key-is-most-specific, and stem-key word overlap. If choices-only correctness is well predicted by the structural flags, we can gate cheaply on structure and audit occasionally with the judge; if not, the judge gate is mandatory.
Q8. Is the hint_explanation JSON placeholder itself causing the 50-character overruns? [MEASURED] input: the placeholder string in questionJSONTemplate that describes the under-50-character constraint is itself 67 characters long ("string — a short clue (<50 chars) pointing to relevant story detail"), while the three real exemplar hints are 37, 38, and 37 characters. Hypothesis: a model anchoring on the schema placeholder's own length rather than on the stated numeral will overrun, and the observed worst overage of 71 characters sits suspiciously close to the placeholder's 67. Experiment: shorten the placeholder to under 50 characters (e.g. "short clue, under 50 characters") with no other change, and rerun. This is outside my assigned axis -- Defect 1 belongs to another brief -- but it is a free observation with a one-line test, and it should be handed to whoever owns hint length.
Q9. What are the correct per-grade option word-count ceilings? The literature gives passage-level bands (Lexile, Flesch-Kincaid, ACAP word counts) but I found no cited numeric ceiling for option length at K-8. Experiment: derive empirically from a corpus of released NAEP grade 4 and grade 8 reading items and released state ELA items, measuring the option word-count distribution per grade. This is a data-collection task, not a model experiment, and it would give R3 a defensible number instead of a guess.
Q10. Does the loose 80-percent threshold or the strict any-character threshold better predict actual cueing in our items? Experiment: compute both flags across the existing corpus and correlate each against the choices-only solve rate. Adopt whichever separates cued from uncued items more sharply. The published literature uses both and does not adjudicate between them. Fold into Q0.
Q11. Do rank-varied exemplars beat equal-length exemplars for teaching length balance? The reasoned argument in section 8.2 -- that variance cannot be demonstrated by one sample, so the key should occupy a different length rank in each exemplar -- is untested and could plausibly backfire by teaching the model that variance is acceptable. Experiment: four arms at fixed cost -- (a) current single exemplar; (b) three exemplars, all with near-equal option lengths; (c) three exemplars with the key at rank 1, 2, and 4 respectively, all sets internally balanced; (d) three rank-varied exemplars plus the validator repair loop. Measure cue rate under Q0's instrument and choices-only solve rate. This is the experiment that decides whether exemplar redesign is worth doing at all, and (d) tells us whether exemplars add anything on top of a validator that already guarantees the property.
Q12. Is the length cue partly caused by the key's own explanation requirement? Section 8.1 identifies a competing pressure inside the prompt: line 66 requires the key's explanation to quote the story, which is the machine analogue of Tarrant's "effort to ensure the correct response is indisputable." Hypothesis: the pressure to make the key defensible leaks into the key's option text, not just its explanation. Experiment: an arm in which explanations for all four options are generated in a second, separate call after the option set is fixed, so no defensibility pressure is present while the options are being written. If the cue drops sharply, the fix is architectural -- split the call -- and is more durable than any wording change. This is the only hypothesis on the list that would change the lambda's control flow rather than its prompt, so it is worth settling early.
Q13. Does the cross-model flatness survive a proper re-score? The 48 / 54 / 46 near-tie is currently the team's evidence that the defect is the prompt's rather than any model's. That inference rests on a threshold that matches no published definition. Experiment: falls out of Q0 for free -- if the three models separate under the strict definition, the "statistically tied, therefore it is the prompt" conclusion needs revisiting, and the model-choice axis of the production cutover gains a discriminator it does not currently have.
Sources
All retrieved 2026-08-17.
Item-writing guideline taxonomies
- Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education, 15(3), 309-334. Full text: https://site.ufvjm.edu.br/fammuc/files/2016/05/item-writing-guidelines.pdf ; record: https://eric.ed.gov/?id=EJ660246
- Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review. Review of Educational Research, 87(6), 1082-1116. https://doi.org/10.3102/0034654317726529 ; record: https://eric.ed.gov/?id=EJ1159693
- Rodriguez, M. C. (2005). Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research. Educational Measurement: Issues and Practice, 24(2), 3-13. https://onlinelibrary.wiley.com/doi/10.1111/j.1745-3992.2005.00006.x ; record: https://eric.ed.gov/?id=EJ718250
- Haladyna & Downing (1993), on effective distractor counts and the <5% nonfunctioning threshold: reported in both Haladyna et al. (2002) and Gierl et al. (2017) above.
Test-wiseness and item-writing flaws
- Millman, J., Bishop, C. H., & Ebel, R. (1965). An Analysis of Test-Wiseness. Educational and Psychological Measurement, 25(3), 707-726. https://journals.sagepub.com/doi/abs/10.1177/001316446502500304
- Brozo, W. G., Schmelzer, R. V., & Spires, H. A. (1984). A Study of Test-Wiseness Clues in College and University Teacher-Made Tests, With Implications for Academic Assistance Centers. College Reading and Learning Assistance Technical Report 84-01, Georgia State University. ERIC ED240928. https://files.eric.ed.gov/fulltext/ED240928.pdf
- Schmucker, R., & Moore, S. (2025). The Impact of Item-Writing Flaws on Difficulty and Discrimination in Item Response Theory. arXiv:2503.10533. https://arxiv.org/html/2503.10533v1
- Pham, H., Besanko, J., & Devitt, P. (2018). Examining the impact of specific types of item-writing flaws on student performance and psychometric properties of the multiple choice question. MedEdPublish. https://pmc.ncbi.nlm.nih.gov/articles/PMC10711986/
- Moore, S., Costello, E., Nguyen, H. A., & Stamper, J. (2024). An Automatic Question Usability Evaluation Toolkit (SAQUET). arXiv:2405.20529. https://arxiv.org/abs/2405.20529 ; code: https://github.com/StevenJamesMoore/SAQUET ; hosted tool: https://saquet.io/
Partial-input validity methodology
- Kaushik, D., & Lipton, Z. C. (2018). How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks. EMNLP 2018, 5010-5015. https://aclanthology.org/D18-1546/
- Balepur, N., Ravichander, A., & Rudinger, R. (2024). Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question? ACL 2024. https://aclanthology.org/2024.acl-long.555/ ; preprint: https://arxiv.org/abs/2402.12483
Reading-comprehension frameworks and K-8 standards
- National Assessment Governing Board. Assessment and Item Specifications for the 2026 NAEP Reading Assessment. https://www.nagb.gov/content/dam/nagb/en/documents/publications/frameworks/reading/naep-2026-reading-specifications.pdf (selected-response guidelines, Exhibit 3.15; comprehension-target distribution, Exhibit 3.12; vocabulary criteria, Exhibit 3.13)
- Mullis, I. V. S., & Martin, M. O. (Eds.) (2019). PIRLS 2021 Assessment Frameworks. TIMSS & PIRLS International Study Center, Boston College. https://pirls2021.org/wp-content/uploads/sites/2/2019/04/P21_Frameworks.pdf ; web version: https://pirls2021.org/frameworks/home/reading-assessment-framework/processes-of-comprehension/index.html
- Alabama State Department of Education. ACAP Summative Item Specifications, Grade 3 English Language Arts (v1.0). https://www.alabamaachieves.org/wp-content/uploads/2022/01/ASMT_2022110_G3ELA2021ACAPSummativeItemSpecs_V1.0.pdf
- Supplemental Information for Appendix A of the Common Core State Standards for English Language Arts and Literacy: New Research on Text Complexity. https://www.thecorestandards.org/wp-content/uploads/Appendix-A-New-Research-on-Text-Complexity-revised.pdf
- Smarter Balanced Assessment Consortium, ELA/Literacy item and distractor specifications (reading distractor typology). https://portal.smarterbalanced.org/library/en/elaliteracy-summative-assessment-blueprint.pdf ; test-development portal: https://contentexplorer.smarterbalanced.org/test-development
Demonstrations vs. instructions, and exemplar design (section 8)
- Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP 2022. https://arxiv.org/abs/2202.12837 (random labels barely hurt across 12 models; drivers are label space, input distribution, and sequence format)
- Anthropic. Prompting best practices (Claude prompt engineering documentation). https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices ("Include 3-5 examples for best results"; examples must be "Diverse: Cover edge cases and vary enough that Claude doesn't pick up unintended patterns")
- Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labels. arXiv:2511.21038. https://arxiv.org/html/2511.21038 (demonstration-override capacity scales with model size; cited as the counter-consideration to a purely exemplar-driven account)
- Gierl, M. J., & Lai, H. Using Automatic Item Generation to Create Multiple-Choice Test Items / NCME instructional module on automated item generation. https://mcc.ca/wp-content/uploads/AIG-Gierl-Lai-Turner-Medical-Education-Journal.pdf ; https://ncme.org/wp-content/uploads/2025/10/Module-34-Automated-Item-Generation-Gierl-Lai.pdf (parent item / item model; children inherit the parent's template, so parent quality is screened first)
- Tarrant, M., & colleagues on item-writing flaws in high-stakes nursing assessments (2,770 MCQs; 46.2 percent contained violations). Record: https://www.semanticscholar.org/paper/42a198ef26c6d2613daaf74c1398e7a238ac070c ; related instrument: https://nursing.ceconnection.com/ovidfiles/01709760-201303000-00002.pdf (source of the "effort to ensure the correct response is indisputable" etiology)
Measured in this brief (exact commands)
- Option and hint character counts in the production exemplar: Python regex extraction over
ai-reading-25/src/lambdas/text-generation/lambda_function/src/helpers/prompts/generateQuestionsPromptForFullStory.ts, matching "(correct_answer|wrong_answer)":\s*"([^"]+)" and "hint_explanation":\s*"([^"]+)". Result: exemplar key 56 chars / 10 words; distractors 46, 46, 49 chars; second-longest / longest = 0.875. hint_explanation JSON placeholder 67 chars; exemplar hints 37, 38, 37 chars.
- Prompt-text audit of
getQuestionDistribution() (same file, lines 21-38) and of generateQuestionsPromptForParagraph.ts: main idea named at no grade band; vocabulary named only at grades 3-5 and only as one arm of a disjunction; the paragraph prompt carries no grade-based type distribution and no worked exemplar at all.
- Harness gap check:
grep -il "haladyna|item-writing|distractor|test-wise|blueprint" -r docs/ src/ in the evaluation harness returns no matches; no validator module exists in the repo root.
- Exemplar scored against all five operational definitions (section 8.3), in both characters and words: Python over the four exemplar option strings. Results -- characters: key 56, distractors 46/46/49; strictly longest TRUE; second-longest/longest 0.875; shortest/longest 0.821; key/mean(distractors) 1.191. Words: key 10, distractors 8/9/10; strictly longest FALSE (ties); second-longest/longest 1.000; shortest/longest 0.800; key/mean 1.111. Only the strict definition in characters flags the item.