Print the whole workshop

Part of a series written for one product team. The companion series it refers to, From Concern to Evidence, is not online yet; where this page points at that packet, the team has it in hand.

Vocabulary, the concern worksheet, the three-criteria table, the audit sheet and the five closing questions are in the From Concern to Evidence packet. Everything below is from Story Studio QA test stories and the release's own records. No real child's data appears anywhere.

Part 1: Say what good looks like, then compare

Before anything else

Write your own definition of a good quiz item in one sentence. Hand it in. It is read without your name.

Do it live

Open the live activity on a phone or laptop. It uses the quizzes from the 18 QA test stories made on 2026-11-09, without the stories. For each question you pick the answer you think is right and say how sure you are, from 25 to 100. Then you say how sure you are that a student in the target grade would get it right without the story, on the same scale. After the answer shows, pick a giveaway if the question was too easy to guess. No names are collected. At the end, copy your results and send them to Mason, who adds up the group.

The session set is 12 questions and takes about ten minutes. The grade sets and the full 72 are there for anyone who wants more. If you have no device, the activity page prints a paper version.

Two versions of one question

Written by Mason as the example the quiz writer is held to. The story is about Brad and Bruno in the desert.

What lesson do Brad and Bruno learn by the end of the story?

First version:

Second version:

Which version is better, and what exactly is different? Write before you talk. Then, without the story: can you pick the key in the first version? In the second? For each pick, choose the one giveaway from the list below.

The giveaway list

These are the words we use for a guessable question in reviews, tickets and the workshops. When you pick an answer without the story, pick the one reason your guess was right. If your guess was wrong, pick "Nothing: a kid would need the story".

Giveaway What it means
Only kind, brave or moral-sounding choice The key is the only choice where someone is kind, brave, honest or does the right thing, so a child picks it on values alone.
Repeats the theme the child picked The key echoes the theme, tone or setting the student chose before the story was written (friendship, courage, space...).
Sounds like a lesson (not just..., rather than...) The key is phrased like a moral, often with a contrast such as "not just", "rather than" or "instead of".
Longest choice The key is visibly longer than the other choices, by words or just by how much space it takes on screen.
Wrong answers don't answer the question Some wrong answers are about something else entirely, so only the key fits the question.
Other choices make no sense on their own The wrong answers are odd or impossible even without the story, so the key is the only sensible one left.
The question's wording points at it A word or phrase in the question also appears in the key, or the question's framing tells you which choice it wants.
Common sense: what a story like this would say Anyone who knows how children's stories usually go could predict the key (the problem gets solved, the friends make up).
Nothing: a kid would need the story Your guess was wrong, or right only by luck. This is the answer we want for every question.

The five rules for answer choices

These bind every quiz generator from 2026-11-05 on.

  1. No one-word or short-phrase choices at grades 4 to 8. Every choice is a full sentence. K-3 should also prefer full sentences.
  2. Wrong answers carry as much specific, plausible detail as the key. The key must not be the longest, the most specific, or the only one with a reason. Pad the wrong answers with specific detail (a named place, a reason, a consequence), not the key with less.
  3. Lesson questions: every choice is a lesson-shaped statement. Only the story decides which one it taught.
  4. No choice that is one of the student's own selections (setting, theme, genre, tone, characters).
  5. No two questions about the same moment in one quiz.

Map each rule to the thing it fixes in the first Brad and Bruno version. Two of the rules need the student's selections or the rest of the quiz, which you do not have here; for those, write "not assessable from this item" and say what you would need.

The rules are about the answer choices. An item can keep all five and still fail: the key has to answer the question that was asked, and it has to be the only choice the story supports.

Apply the rules to current QA items

Item 1 (Sprocket and Imani's Impossible Race, Lexile 736). What did Imani do when she handed Sprocket the specialized wrench?

Item 2 (Pip's Wings in the Withered Wood, Lexile 722). Why did Pip feel so lost at the start, even though she had always lived there?

First, without the story, pick the key and choose a giveaway. Then: which rule does it keep, which does it break, and which words break it? Is the key an answer to the question that was asked?

Rewrite one wrong answer

Pick a wrong answer from Item 1 and rewrite it so it carries as much specific detail as the key, stays wrong, and stays true to the kind of thing the story could have said. You do not have the story here, so check your version against it before calling it wrong. Mason can open it.

Part 2: Language, reading load and who the question is for

Read this aloud at a child's pace

Nadia's Crumbling Palace Adventure, Lexile 315. What problem did Nadia want to solve when she crouched behind the pillar?

Time yourself. Would a kindergartener finish the four choices? Yes, no, or depends, and on what.

Two readings of one rubric line

The literacy reviewer checks every choice against a 15-word ceiling. The judge compares the choices with the story's own sentences. Both are scoring the rubric's reading-load line. On Zara's Quiet Garden, the literacy reviewer gave 3 and the judge gave 2 on all four questions.

Young-grade why questions with two sensible reasons

In the exemplar bank's hold-out run, 4 of the 36 K-2 items had a second answer a teacher would accept. Two were grade-0 motivation items about why Rivet slowed down for Ziggy, one was a grade-1 motivation item about why Gizmo felt embarrassed, and one was a grade-1 "just after" item. For the two Rivet items: is the fault in the question, the key, or the story? What would you need to see to decide?

Reading-load criterion

Grade band Observable anchor for "readable at this grade" Not readable when Cannot tell when
K-2
3-5
6-8

Test it on the Nadia item, the Pip item, and one item from a 6-8 story in the review set.

Part 3: When the rater and the judge disagree, who decides?

Zara's Quiet Garden

The literacy reviewer and the model judge scored the same story against the same rubric. They agreed on 51 of 72 cells. Most of the differences are one point apart on image and text scores. These five are about what each was measuring:

Code Question Rater Judge What each measured, and where they part
Q1 4 Pass Fail Key says "pebble collection"; the story has one pebble and never says collection. The literacy reviewer passed it as what the story meant; the judge failed it on the words.
Q2 4 Fail Pass Both picked the key without reading. The literacy reviewer failed the item as guessable; the judge passed it because it labelled its own pick a guess.
Q5 2 Fail Pass Key ties two wrong answers at 9 words. The literacy reviewer fails ties; the code counts a tie as no cue.
Q11 1 to 4 3 2 The literacy reviewer checks the 15-word ceiling; the judge compares with the story's sentences.
Q12 1 to 3 2 2 The cell agrees. The literacy reviewer reads the hints on the wrong answers; the judge reads the hint on the key. Same score for two different things.

For each: who is right, and why? Then sort: definition gap (fix the rubric wording), judgment call (the literacy reviewer decides), open (needs more cases). A case can fall in more than one pile; say so when it does. If you think the fault is in the judge and not the quiz, say that instead.

Decision record

Field Your record
The disagreement, in one sentence
The ruling
Who made it and on what expertise
The rubric line it changes, and the new wording
What the judge is told differently as a result
Cases that would reopen it
Review date

Two questions back

When you read another group's record, return the two questions that would stop you acting on it.