Here is a question from Story Studio, which writes children's stories and a short reading quiz to go with each one. Don't read the story. There isn't one here.

What does the narrator learn about giving after skating with the polar bear? A. Giving is only important during holidays B. Giving means always sharing your favorite toys C. Giving means expecting something back in return D. Giving is about sharing time and kindness, not just presents
A real item from the development database, shown without its story. The answer key is not marked.

It's D. You knew that. Every story we generate ends on a warm lesson, so the cynical answer, the rigid answer and the selfish answer can all be crossed off by someone who has never heard of the polar bear.

I spotted that reading a single quiz. What I wanted to know was whether it was one bad item or a property of the whole collection, and reading more quizzes was never going to answer that. Counting them would.

What a cue is

A cue is anything about the answer choices, other than being correct, that lets a reader pick the right one. The right answer is the longest. The right answer is the only one that sounds like a lesson. The wrong answers repeat the question's wording and the right one doesn't.

None of this is new to people who write tests. Thorndike counted test-wiseness among the lasting traits that inflate test scores in 1951, and in 1965 Millman, Bishop and Ebel gave it the taxonomy people still cite. Its cue-using strategies include a keyed option longer than the rest, specific determiners like always and never, and options that resemble the stem. The standard item-writing guidelines (Haladyna, Downing and Rodriguez, 2002) turn the catalogue into rules. Keep the choices about equal in length. Avoid absolute words and clang associations. Make every distractor plausible. And in 1990 Katz and colleagues had college students answer SAT reading questions with the passages deleted. They did well above chance.

Machine learning ran into a close relative in 2017 and 2018 and called it annotation artifacts: patterns left in a dataset by the way its examples were written, rather than by what they mean. Gururangan and colleagues got about 67% of the SNLI inference benchmark right with a classifier that saw only the hypothesis and never the premise, partly because the people writing "contradiction" examples reached for words like no and never. Cai, Tu and Gimpel scored 72.5% on a two-choice story-ending test reading only the endings, and sentiment was one of the clues. More recently, Proebsting and Poliak rebuilt part of SNLI with language models writing the examples. The same hypothesis-only classifier that got 72% on the human-written data got 86 to 96% on the model-written versions.

That last result is the situation I'm in. When one model writes the right answer and the three wrong ones for thousands of items, any habit it has becomes a pattern across the whole set. No single item looks wrong. You only see it when you count.

One shape for every cue

We had already measured one cue, answer length, and built that check as a one-off. The second one was tone, and I could see a third and fourth coming, so I rebuilt the check so that any attribute plugs into the same test.

For every item and every attribute, score all four options. Then ask one question: is the correct answer strictly the highest?

export function keyPosition(scores, keyIndex) {
  const n = scores.length;
  const max = Math.max(...scores);
  const nMax = scores.filter((s) => s === max).length;
  return {
    top: nMax === 1 && scores[keyIndex] === max,
    topChance: nMax === 1 ? 1 / n : 0,
  };
}

The detail that matters is topChance. The lazy baseline is 25%, one in four. But if two options tie for the top score, no option is strictly highest, so a random pick has zero chance of being "the top one." Counting those items against 25% understates a cue. Ignoring them overstates it. Computing chance per item and summing gives the expected count the data actually allows. Divide observed by expected for a lift, and use the per-item probabilities for a z-score, so ties neither flatter the result nor hide it.

Everything after that is attributes. Some cost nothing to compute: length in words, absolute words like "only" and "always," how many words an option shares with the question, what share of its words appear in the story. Some need a reader, which comes up below.

The cleaning found problems before the analysis did

I pulled every published quiz from our development database: 73,742 items. Before computing a single cue, the loader that normalises them found three things.

82% of the database was a load test. 60,274 items were created over three weeks in December by 1,084 accounts. The stored answer on every response was the literal string default_answer, and the median time to answer a question was 276 milliseconds. No child reads that fast. Any average taken over that database without excluding December was mostly describing a load test.

1,225 more items were placeholders, with choices like "Right Answer" and "Distractor 1."

73,742 items every published quiz in the dev database
−60,274 December load test: default_answer, 276 ms per answer
−1,225 −160 placeholders ("Right Answer"), then duplicates
12,083 items October 2023 to September 2026
What the loader removed before any cue was measured. Five out of six items in the database were not real quizzes.

Three of those placeholders were in the 95-question sample we had used to show the quizzes are guessable. Four AI graders had answered those 95 with the story withheld and got 89.5% to 97.5% right. Three of the questions had a right answer that was literally the text "Right Answer," and every grader got them right on every run. Taking them out moves the figures by at most 0.4 points, so the conclusion stands. The sample didn't.

A check built to find giveaways found one in the evidence for the giveaway claim. Run your cue battery on your evaluation set before you run your evaluation.

What counting found

12,083 real items remained, from October 2023 to September 2026, written by two different models. With that many items the counted rates are precise to about a point either way. Each cue shows up under both generators, but not at the same strength. Story echo, for one, runs at roughly four times chance through 2025 and under two times chance in September 2026, after the generator changed. A cue is a habit of whatever wrote the items, so it moves when the writer does.

The right answer is the option...SeenChanceLift
sharing most words with the story66%19%3.5x
that is longest41%15%2.7x
sharing fewest words with the question11%4%2.7x
most positive, by word list24%14%1.6x
with the most absolute words3%3%0.95x
with the fewest absolute words0.4%0.2%1.9x (43 items)

Chance here is the expected rate given each item's ties, which is why it is not 25% anywhere.

Three of those deserve a closer look.

The right answer copies the story. In 66% of items, the correct option reuses more of the story's own wording than any wrong one, against 19% expected. That doesn't make the question guessable without the story. For a detail question, finding the right sentence may be exactly what the item is meant to test. For a question about why something happened, or what it meant, it means a child can match instead of infer.

The wrong answers echo the question. The right answer is the option that repeats the question's words least, 2.7 times as often as chance. A test-wise reader learns to skip the option that parrots the question.

"Only," "always" and "never" are not a cue here. They show up on right and wrong answers at the same rate across all 12,083 items. The polar-bear question happens to have two of them, which is exactly why one striking example is a bad guide to a collection.

Tone needs a reader

A sentiment word list costs nothing and never changes its mind, so I tried it first. It scored "Giving is only important during holidays" as more positive than the right answer, because "important" is a positive word. Cynicism and selfishness mostly use ordinary words. A word list scores vocabulary, and what gives the polar-bear item away is what the sentence means about people.

So tone is rated, by a model, under rules meant to keep the answer key out of it. It sees the question and the four options in a fixed shuffled order. It never sees the story. It is never told which option is correct. For each option it gives a valence from -2 to +2, a score for how much it sounds like a children's-story lesson, and a flag for cynical or selfish.

Blind to the key is not the same as blind to the answer. A rater that can tell which option sounds like the lesson is, in a sense, solving the question. For this cue I'll accept that, because the claim being tested is exactly that the kind option can be found without the story. What the setup rules out is the rater being told.

Two raters from two companies, Anthropic's Claude Opus (A) and OpenAI's gpt-5.4-mini (B), ran on the same items with the same instructions. Neither model wrote them. Two models agreeing is not the same as people agreeing; both were tuned on similar ideas of what a children's story should sound like, and they could share a blind spot.

How often the right answer was the single most positive option:

SetRater ARater BChance
120 recent items49%50%~15%
586-600 items, sampled across three years45%44%~15%
All 665 lesson questions84%81%~22%

They agree item by item as well as in total. Their per-option tone scores correlate at 0.85 to 0.93 (Pearson, across every option in a set), and when both name a single most positive option they name the same one 98 to 99.8% of the time.

Overall, the correct answer is the most positive option a little under half the time, roughly three times chance. That average mixes very different kinds of question, though.

Share of items where the right answer is strictly the most positive option, rater A, by question type. Lesson is every lesson question in the corpus; the rest come from the 586-item sample. Random placement would give about 15 to 22% depending on ties.

On questions that ask what a character learned, the right answer was the most positive option 84% of the time across all 665 such questions (81% for the second rater). When exactly one option was the most positive, it was the right answer 96% of the time, for both raters. On the 530 questions where both raters independently named one kindest option, they named the same one 529 times, and it was right 519 times (98%; 95% interval 96 to 99%). On plain detail questions, "what color was the map," it was 35%.

A few checks on that. Some stories have more than one lesson question, so the items aren't fully independent; keeping just one per story gives 84% on 604 stories, the same as before. The rate held across the years, 77% at its lowest year and 92% at its highest. The 19 lesson questions written since the generator changed this summer come in at 13, too few to say whether it's moving. And the "lesson" label comes from simple rules on the question's wording, so I read 30 of them: 27 asked what a character learned, two asked what a character discovered, and one asked what a character promised to do.

For lesson questions, "pick the nicest answer" is very nearly a perfect strategy. In the item-writing guidelines' terms, the distractors aren't plausible: in a collection where every story ends kindly, a cynical lesson is the "blatantly absurd" option the 2002 guidelines warn about, just absurd in tone instead of in content. Read one at a time, each question looks fine. The cue only exists across the collection, because everything we generate shares one kind voice.

I found no published study of this cue in tests given to students. Passage-free answering is well studied, but as a total solve rate rather than as tone. The nearest case is a machine benchmark, the story-ending test above. There, the right ending is measurably more positive than the wrong one, and simply picking the more positive ending scores about 59% against 50% chance. When Sharma and colleagues rebuilt that test in 2018, part of the fix was telling writers to keep the right and wrong endings similar in tone. That's the same fix this quiz generator needs.

What it doesn't explain

I expected the cues to explain the guessability. They explain a little of it.

Right answer stands out on...ItemsSolved without the story
at least one measured cue8096%
nothing measured1288%
(random guessing)25%

These are the 92 real questions from the guessability sample, each answered without the story by four AI evaluators. Items where the right answer stands out on at least one cue were solved 96% of the time. The twelve where it stands out on nothing we measure were solved 88% of the time. Tone and length each account for about seven points. The rest of the gap to 25% has two leading candidates: cues nobody has measured yet (grammatical fit, how specific an option is, how natural it sounds), or graders that bring world knowledge and plausibility reasoning a nine-year-old doesn't have. The NLP literature argues about the same split: Balepur and colleagues found that models answering from the choices alone sometimes reconstruct the question rather than exploit a surface pattern. I haven't separated the two here. A weaker solver, or people answering without the story, would.

That result is the reason to build the battery rather than to stop at one cue. If I'd measured tone alone and found it large, it would have been easy to believe it explained everything.

Limits

These are model ratings and model solvers. No child has answered any of these items in this analysis; the development data has no real student responses in it. The tone figures rest on two model families agreeing, not on people. No teacher or child has rated these options. The question-type labels come from simple rules on the question's wording. Absolute-word and question-echo counts use a stopword list and a crude stemmer, which is fine for rates across thousands of items and wrong for any single one.

Using it on your own data

Any multiple-choice set where the same system writes the right answer and the wrong ones can carry this, whether it's a quiz generator, a benchmark built with a model's help, or a support bot's suggested replies.

Sources