Whose “Good Enough”?: the whole workshop
Design, participant packet, facilitator notes with keys, and appendix on one page. This is the facilitator's copy: print the participant packet on its own for the group.
Part of a series written for one product team. The companion series it refers to, From Concern to Evidence, is not online yet; where this page points at that packet, the team has it in hand.
Design
Three 75-minute sessions for the moment when the literacy reviewer, the engineers and the manager mean different things by "a good quiz question". The parent design's sequence was: surface assumptions, build literacy-quality criteria, examine language and representation, negotiate disagreements, record an accountable decision. This version compresses the first two and the last two, because the team already has written rules and written bars, and the live work is agreeing what they mean on a real item. It overlaps Sessions 1 and 4 of From Concern to Evidence (From Concern to Evidence) and reuses their worksheets. The literacy reviewer leads more of this series than any other.
What the team should be able to do afterward
- State their own standard for a quiz item before hearing anyone else's, and see where it differs from a colleague's.
- Apply the five answer-choice rules to a real item and say which rule it breaks, with the words that break it.
- Judge reading load and wording at a grade, with the literacy reviewer's ceiling as the anchor.
- Record a disagreement between the human rater and the model judge as a decision with an owner, so the number does not settle it by default.
The three sessions
Same shape as From Concern to Evidence: write first, short explanation, supported practice, individual exit task, the five anonymous questions (the From Concern to Evidence packet, closing section).
Session 1: Say what good looks like, then compare
Cases (packet Part 1): Mason's good and bad versions of a lesson question about Brad and Bruno; the five answer-choice rules; a current QA item whose key is a detail rather than an answer.
| Time | Activity |
|---|---|
| 0 to 8 | Purpose, norms. Each person writes their own one-sentence definition of a good quiz item and hands it in. |
| 8 to 18 | Alone: the bad and good Brad and Bruno items, side by side. Which is better, and what exactly is different? |
| 18 to 30 | The literacy reviewer reads the definitions without names and groups them. Where do the engineers' and the reviewer's definitions part? |
| 30 to 42 | Mason presents the five rules, two minutes, then the group maps each rule to the thing it fixes in the bad item. |
| 42 to 47 | Break. |
| 47 to 62 | Pairs apply the rules to the Sprocket wrench item and one other current QA item. Name the rule broken and the words that break it. |
| 62 to 75 | Individual: rewrite one wrong answer so it carries as much detail as the key. Five questions. |
- Facilitator move: when a definition says "fair" or "appropriate", ask what the literacy reviewer would point to.
- Success check: every pair names a rule and the words, and at least one rewritten wrong answer is as specific as its key.
- Live activity, ten to twelve minutes: run the session set from the packet's link before Mason presents the rules, so the picks are not shaped by them. Everyone can do it alone before Session 1, or Session 1 can run to 87 minutes with the activity right after the definitions are handed in. Put the combined tally on the board before anyone talks.
- Between sessions: apply the five rules to one quiz on your own QA account and write which rule, if any, it breaks.
Session 2: Language, reading load and who the question is for
Cases (packet Part 2): a Lexile 315 story with answer choices of 11 to 13 words; the K-2 hold-out items where young-grade why questions had two sensible reasons; the literacy reviewer's 15-word ceiling and the judge's different reading of the same rubric line.
| Time | Activity |
|---|---|
| 0 to 8 | Report back. Retrieve the five rules. |
| 8 to 18 | Alone: read the Lexile 315 item aloud at a child's pace and time it. Would a kindergartener finish the choices? Write yes, no or depends, and on what. |
| 18 to 30 | The literacy reviewer leads: what reading load means at K-2, 3-5 and 6-8, and why full sentences at grade 4 and up and shorter choices at K-3 are both rules. |
| 30 to 45 | Pairs: the two young-grade why items where a teacher accepted a second answer. Is the fault in the question, the key, or the story? |
| 45 to 50 | Break. |
| 50 to 65 | Groups write a reading-load criterion with an observable anchor per grade band, and test it on three items. |
| 65 to 75 | Individual: one place where "right at this grade" and "right" come apart. Five questions. |
- Facilitator move: do not let the group settle on a word count alone. Ask what else makes a sentence hard at six.
- Success check: the criterion has a different anchor per band and a "cannot tell" case.
- Between sessions: none.
Session 3: When the rater and the judge disagree, who decides?
Cases (packet Part 3): the Zara's Quiet Garden comparison, 51 of 72 cells agreed; the five disagreements with their reasons; the open question of which one is the reference.
| Time | Activity |
|---|---|
| 0 to 8 | Report back. Retrieve the Zara disagreements. |
| 8 to 18 | Alone: for each of the five disagreements, write who you think is right and why. |
| 18 to 30 | Reveal the split. Sort the five into: definition gap (fix the rubric wording), judgment call (the literacy reviewer decides), and open (needs more cases). |
| 30 to 45 | Groups draft the decision record for one disagreement: the ruling, the owner, the rubric line it changes, the date it is reviewed. |
| 45 to 50 | Break. |
| 50 to 65 | Groups trade records and ask the two questions that would stop them acting on it. |
| 65 to 75 | Individual: the one line in the rubric you would change first. Five questions. |
- Facilitator move: the judge is not the tiebreaker because it is faster. Ask who has the expertise for each disagreement, and whether that expertise has been written down anywhere the judge could read it.
- Success check: each record names a ruling, an owner and a rubric line; none says "the model will decide".
- Afterward: Mason brings the agreed rubric changes to the next harness run and reports which disagreements closed.
What the series keeps from the research
Three points, where they change what happens in a session:
- The rater's standard is the one the series is building toward, so it is heard first and written down. Peters and Armstrong's Type Three teaching has the teacher as "a member of the group of learners" whose knowledge "does not necessarily supersede that of the other learners" (1998, p. 79). Here that cuts both ways: Mason's harness does not outrank the literacy reviewer's reading, and the literacy reviewer's reading is written down so the engineers can build to it. Say at the start what is open. On a literacy ruling the literacy reviewer has the final word by role. What the group negotiates is the wording, the evidence and what the judge gets told.
- The disagreement stays in the record instead of being averaged away. Mezirow's conditions for discourse include being "free from coercion" (1997, p. 10), and Brookfield's reply is that a leader's avowal of openness does not make it so; the structure has to do it (Brookfield, 2006, p. 22, on who speaks when passing is allowed). The written-first rule and the anonymous cards are that structure.
- A standard needs a worked example at each grade, and less scaffolding for the person who already reads at that grade for a living. Kirschner, Sweller and Clark on worked examples (2006, p. 80) and on guidance receding with prior knowledge (p. 75). The literacy reviewer does not need the Lexile 315 item read aloud to her; the engineers do.
Reuse from From Concern to Evidence
- Vocabulary, concern worksheet, three-criteria table and audit sheet: the From Concern to Evidence packet, Parts 1 and 4.
- The Zara disagreements and the four Part 4 items: the From Concern to Evidence packet, Part 4.
- Five closing questions and opening eight minutes: From Concern to Evidence.
Appendix
The appendix is a short optional note on reading a bar as a rate with a sample. It adds three fields to the decision record: the accepted rate and its sample, the per-child reading of that rate, and the slice the bar was not checked on.
Citation audit
All three sources were verified for From Concern to Evidence on 2026-11-07 (Crossref match or publisher record, read in the copy on disk). Peters and Armstrong (1998), doi:10.1002/ace.7908, p. 79: verified. Mezirow (1997), doi:10.1002/ace.7401, p. 10: verified. Brookfield (2006) workshop packet, author's site, p. 22: verified, practitioner account. Kirschner, Sweller and Clark (2006), doi:10.1207/s15326985ep4102_1, pp. 75 and 80: verified. No other sources are cited in this folder.
Participant packet
Vocabulary, the concern worksheet, the three-criteria table, the audit sheet and the five closing questions are in the From Concern to Evidence packet. Everything below is from Story Studio QA test stories and the release's own records. No real child's data appears anywhere.
Part 1: Say what good looks like, then compare
Before anything else
Write your own definition of a good quiz item in one sentence. Hand it in. It is read without your name.
Do it live
Open the live activity on a phone or laptop. It uses the quizzes from the 18 QA test stories made on 2026-11-09, without the stories. For each question you pick the answer you think is right and say how sure you are, from 25 to 100. Then you say how sure you are that a student in the target grade would get it right without the story, on the same scale. After the answer shows, pick a giveaway if the question was too easy to guess. No names are collected. At the end, copy your results and send them to Mason, who adds up the group.
The session set is 12 questions and takes about ten minutes. The grade sets and the full 72 are there for anyone who wants more. If you have no device, the activity page prints a paper version.
Two versions of one question
Written by Mason as the example the quiz writer is held to. The story is about Brad and Bruno in the desert.
What lesson do Brad and Bruno learn by the end of the story?
First version:
- A. That sandwiches should always be hidden very well
- B. That the desert is a dangerous place to visit alone
- C. That monkeys are the meanest animals in the desert
- D. That courage and kindness can help overcome challenges and make friends
Second version:
- A. That conflicts can usually be overcome by making fair deals with others
- B. That the desert is a dangerous place to visit alone and it's important to come prepared
- C. That monkeys are more aggressive compared to other animals in the desert
- D. That daunting challenges can be overcome by strategizing with others
Which version is better, and what exactly is different? Write before you talk. Then, without the story: can you pick the key in the first version? In the second? For each pick, choose the one giveaway from the list below.
The giveaway list
These are the words we use for a guessable question in reviews, tickets and the workshops. When you pick an answer without the story, pick the one reason your guess was right. If your guess was wrong, pick "Nothing: a kid would need the story".
| Giveaway | What it means |
|---|---|
| Only kind, brave or moral-sounding choice | The key is the only choice where someone is kind, brave, honest or does the right thing, so a child picks it on values alone. |
| Repeats the theme the child picked | The key echoes the theme, tone or setting the student chose before the story was written (friendship, courage, space...). |
| Sounds like a lesson (not just..., rather than...) | The key is phrased like a moral, often with a contrast such as "not just", "rather than" or "instead of". |
| Longest choice | The key is visibly longer than the other choices, by words or just by how much space it takes on screen. |
| Wrong answers don't answer the question | Some wrong answers are about something else entirely, so only the key fits the question. |
| Other choices make no sense on their own | The wrong answers are odd or impossible even without the story, so the key is the only sensible one left. |
| The question's wording points at it | A word or phrase in the question also appears in the key, or the question's framing tells you which choice it wants. |
| Common sense: what a story like this would say | Anyone who knows how children's stories usually go could predict the key (the problem gets solved, the friends make up). |
| Nothing: a kid would need the story | Your guess was wrong, or right only by luck. This is the answer we want for every question. |
The five rules for answer choices
These bind every quiz generator from 2026-11-05 on.
- No one-word or short-phrase choices at grades 4 to 8. Every choice is a full sentence. K-3 should also prefer full sentences.
- Wrong answers carry as much specific, plausible detail as the key. The key must not be the longest, the most specific, or the only one with a reason. Pad the wrong answers with specific detail (a named place, a reason, a consequence), not the key with less.
- Lesson questions: every choice is a lesson-shaped statement. Only the story decides which one it taught.
- No choice that is one of the student's own selections (setting, theme, genre, tone, characters).
- No two questions about the same moment in one quiz.
Map each rule to the thing it fixes in the first Brad and Bruno version. Two of the rules need the student's selections or the rest of the quiz, which you do not have here; for those, write "not assessable from this item" and say what you would need.
The rules are about the answer choices. An item can keep all five and still fail: the key has to answer the question that was asked, and it has to be the only choice the story supports.
Apply the rules to current QA items
Item 1 (Sprocket and Imani's Impossible Race, Lexile 736). What did Imani do when she handed Sprocket the specialized wrench?
- A. Imani gave Sprocket encouraging words to keep trying hard.
- B. Their fingers brushed as the exchange happened between them.
- C. Imani loosened the bolts while Sprocket held the wrench.
- D. Imani worked alone to replace the thermal regulator in three minutes.
Item 2 (Pip's Wings in the Withered Wood, Lexile 722). Why did Pip feel so lost at the start, even though she had always lived there?
- A. Her parents had vanished, and Biscuit said every old path fell into a chasm.
- B. Her parents had vanished, and fog had hidden the bridge she used to find home.
- C. Her parents had vanished, and the once-golden forest had become a maze.
- D. Her parents had vanished, and her tired wings could not rise above the haze.
First, without the story, pick the key and choose a giveaway. Then: which rule does it keep, which does it break, and which words break it? Is the key an answer to the question that was asked?
Rewrite one wrong answer
Pick a wrong answer from Item 1 and rewrite it so it carries as much specific detail as the key, stays wrong, and stays true to the kind of thing the story could have said. You do not have the story here, so check your version against it before calling it wrong. Mason can open it.
Part 2: Language, reading load and who the question is for
Read this aloud at a child's pace
Nadia's Crumbling Palace Adventure, Lexile 315. What problem did Nadia want to solve when she crouched behind the pillar?
- A. Nadia wanted to find the ancient rope hidden in the palace.
- B. Nadia wanted to explore the forbidden throne room that nobody had entered.
- C. Nadia wanted to prove to Maya that she was not afraid.
- D. Nadia wanted to reach the golden box at the bottom of the pit.
Time yourself. Would a kindergartener finish the four choices? Yes, no, or depends, and on what.
Two readings of one rubric line
The literacy reviewer checks every choice against a 15-word ceiling. The judge compares the choices with the story's own sentences. Both are scoring the rubric's reading-load line. On Zara's Quiet Garden, the literacy reviewer gave 3 and the judge gave 2 on all four questions.
Young-grade why questions with two sensible reasons
In the exemplar bank's hold-out run, 4 of the 36 K-2 items had a second answer a teacher would accept. Two were grade-0 motivation items about why Rivet slowed down for Ziggy, one was a grade-1 motivation item about why Gizmo felt embarrassed, and one was a grade-1 "just after" item. For the two Rivet items: is the fault in the question, the key, or the story? What would you need to see to decide?
Reading-load criterion
| Grade band | Observable anchor for "readable at this grade" | Not readable when | Cannot tell when |
|---|---|---|---|
| K-2 | |||
| 3-5 | |||
| 6-8 |
Test it on the Nadia item, the Pip item, and one item from a 6-8 story in the review set.
Part 3: When the rater and the judge disagree, who decides?
Zara's Quiet Garden
The literacy reviewer and the model judge scored the same story against the same rubric. They agreed on 51 of 72 cells. Most of the differences are one point apart on image and text scores. These five are about what each was measuring:
| Code | Question | Rater | Judge | What each measured, and where they part |
|---|---|---|---|---|
| Q1 | 4 | Pass | Fail | Key says "pebble collection"; the story has one pebble and never says collection. The literacy reviewer passed it as what the story meant; the judge failed it on the words. |
| Q2 | 4 | Fail | Pass | Both picked the key without reading. The literacy reviewer failed the item as guessable; the judge passed it because it labelled its own pick a guess. |
| Q5 | 2 | Fail | Pass | Key ties two wrong answers at 9 words. The literacy reviewer fails ties; the code counts a tie as no cue. |
| Q11 | 1 to 4 | 3 | 2 | The literacy reviewer checks the 15-word ceiling; the judge compares with the story's sentences. |
| Q12 | 1 to 3 | 2 | 2 | The cell agrees. The literacy reviewer reads the hints on the wrong answers; the judge reads the hint on the key. Same score for two different things. |
For each: who is right, and why? Then sort: definition gap (fix the rubric wording), judgment call (the literacy reviewer decides), open (needs more cases). A case can fall in more than one pile; say so when it does. If you think the fault is in the judge and not the quiz, say that instead.
Decision record
| Field | Your record |
|---|---|
| The disagreement, in one sentence | |
| The ruling | |
| Who made it and on what expertise | |
| The rubric line it changes, and the new wording | |
| What the judge is told differently as a result | |
| Cases that would reopen it | |
| Review date |
Two questions back
When you read another group's record, return the two questions that would stop you acting on it.
Facilitator notes
Reuse from From Concern to Evidence (From Concern to Evidence)
- Vocabulary, concern worksheet, three-criteria table, audit sheet: the From Concern to Evidence packet, Parts 1 and 4.
- The Nadia item is From Concern to Evidence's Part 4 Item D; the Zara table is From Concern to Evidence's Part 4 list, laid out as a table here.
- Five closing questions, opening eight minutes, scoring: From Concern to Evidence.
If the team has done From Concern to Evidence Session 4, Session 3 here can be cut to the decision record, about 40 minutes.
Number sources
- Brad and Bruno, both versions, word for word: bad_vs_good_examples_multiple_choice.md in the release reference folder. Only Question 1 of that file is filled in.
- The five rules: 2026-11-05_mason-rules-distractor-specificity.md.
- Sprocket wrench item (id 90102) and Pip item (90246): QA read on 2026-11-07. Letters are the packet's shuffled order.
- Nadia item (90055): QA read on 2026-11-07.
- The literacy reviewer 3, judge 2 on Q11; the five disagreements; 51 of 72: the 2026-10-27 rater-versus-judge comparison.
- K-2 hold-out: 4 of 36 items with a second answer, the Rivet and Gizmo items: 2026-11-06_quiz-exemplar-bank-tightened-results.md, "By grade band".
Keys
Keys: open after the judging is done
| Item | Key |
|---|---|
| Brad and Bruno, first version | D |
| Brad and Bruno, second version | the file does not mark one; Mason's point is that only the story decides |
| Sprocket wrench | B (their fingers brushed) |
| Pip | C (the once-golden forest had become a maze) |
| Nadia | B (explore the forbidden throne room) |
Intended reasoning
Session 1
Brad and Bruno: in the first version the key is the only warm, lesson-shaped choice, so a reader picks it without the story. In the second, all four are lesson-shaped and about the same length, so only the story can decide. Rules 2 and 3 are the ones it illustrates. Someone will say the second version is harder. It is harder to guess. Whether it is harder for a child who read the story is unmeasured, since similar choices can also raise the reading demand, and there is no child data on this item.
Sprocket wrench item: the key, "Their fingers brushed as the exchange happened between them", is a detail from the sentence, not an answer to "what did Imani do". Wrong answer C is a reversal of a real event, which is good practice; wrong answer A is a filler. Rule 2 is kept on length; the fault is that the question and key do not match, which is a validity fault the five rules do not cover. Say it outright: the rules are about the choices, and a question can keep all five and still be a bad question.
Pip item: all four choices begin "Her parents had vanished, and", which is rule 2 done well; each wrong answer names a real thing from the story at the wrong time. The key is right. This is the item to hold up as what the rules produce when they work.
Session 2
Nadia: the story's Lexile is 315, low in the app's range, and the four choices are 11 to 13 words each. An adult reading aloud at a guessed pace is a rough check, not a measure of what children can do, so say that when you time it. Rule 1 says K-3 should prefer full sentences; the literacy reviewer's ceiling is 15 words. Both are met. The question is whether "under 15 words" is the right anchor at K, or whether the anchor at K is something else (sentence shape, vocabulary, one clause). Do not settle it; write the candidates down for the literacy reviewer's criterion.
Q11: this is a definition gap, and it is in the rubric wording. The fix is to state which reading the line means.
Rivet items: the tightened-results file lists them as young-grade motivation items "with two sensible reasons", and does not say whether the story states one reason. The answer is "cannot tell without the story", and the next action is to open it.
Session 3
The sort to aim for, and the packet's three piles do not hold all of it: Q5 and Q11 are definition gaps (rubric wording, Mason owns the change, the literacy reviewer approves the wording). Q1 is a judgment call that goes to the literacy reviewer, and the record should say that a teacher's reading of "collection" stands over the judge's word match. Q2 is a judge fault, not a disagreement about the quiz: a judge that passes an item because it labelled its own answer a guess is grading itself; the record is "the code counts a blind pick as a pick, labelled or not". Q12 is a definition gap about which hint is read.
End on how these were found. A person scored the same story the judge scored without seeing the judge's answers. The review sheet's collapsed columns exist for that, which is why the rule says not to open them until you are done.
Giveaway tally (Session 1)
Tally the giveaways for the first Brad and Bruno version before showing the rules. Expect "Only kind, brave or moral-sounding choice" and "Sounds like a lesson", with some "Longest choice". That tally makes the argument for rules 2 and 3 before Mason states them. On the second version, expect "Nothing" and a scatter. Tally the Sprocket and Pip items before the rules are applied too. "Nothing" after a wrong guess is a recording convention from the sheet. A wrong guess does not prove the item has no cue, and a right one does not prove it has one. Pip should draw mostly "Nothing", and it is held up for that reason. The list comes from the READ ME tab of the 2026-11-09 review sheet.
Running the live activity
Send the link to the session set: https://masonborchard.com/workshops/whose-good-enough/activity.html?set=session. Everyone works alone and nobody talks until the tally is up. Each person copies their results at the end and sends them to Mason, who pastes them into the facilitator box at the bottom of the results screen. Nothing leaves anyone's browser until they send it, and the page collects no names. The box gives a per-question table to put on the screen.
- Source. The questions, keys and explanations are from the 2026-11-09 review sheet of 18 QA test stories. Logins, learner accounts, story links and the rater's name are left out of the page.
- Choice order. The sheet stores every key as choice A. The page reorders the choices with a fixed seed, so everyone sees the same letters, and no quiz has three keys in the same position.
- Scale. 25 is a blind guess on four choices, and the same scale is used for the question about a student.
- BQTH scores. BQTH has not judged this set, so the harness columns are empty and the reveal says so. When they are filled, rerun the build script and each reveal shows the two blind readers' picks and the rubric judge's Q1 and Q2 results, with letters matched to the page's order. Nothing in the page has to change.
cd ~/code/masonborchard.com && python3 Scripts/build_quiz_activity.py
Appendix numbers
The item counts (156 on the 39 frozen stories, 96 on the 24 new ones) assume four items per quiz, which is the gate in the One Change One Claim pipeline table; say so if the group asks, and confirm from the harness output whether the second-answer figures (3.8 and 5.3) are counted over items before using the per-item point values; 5.3 percent is not a whole number of items out of 96, so the measure may be weighted or defined differently. The K-2 figure (4 of 36) is the one already in this packet. The 3-over-n rule is arithmetic, and the fuller working is in From Concern to Evidence's appendix.
Appendix
Optional reading. The arithmetic and its assumptions are worked out in the fuller appendix in the From Concern to Evidence series.
Why the bar cannot be zero
The quiz writer is probabilistic. A bar of "no second acceptable answer, ever" cannot be checked on a sample. Mason's bars are rates for that reason: strong-reader guessing at most 55 percent, child reader at most 38, second acceptable answer at most 5 percent. A ruling that sets or moves a bar should also say what sample the rate stands on and who bears the misses.
Three things to add to the decision record
The decision record in the packet has seven fields. When a ruling sets or moves a bar, add these:
| Field | Why |
|---|---|
| The accepted rate and the sample it was measured on | The pipeline table in One Change One Claim shows a second-answer rate of 3.8 percent on the 39 frozen stories and 5.3 percent on the 24 new ones. If the measure counts items and each quiz has four, those sets hold about 156 and 96 items, so one item moves the first figure by about 0.6 points and the second by about 1 point. Confirm the denominator from the harness output before reading the gap between them as a change, and remember that the four items in a quiz share a story, so they are not independent observations. |
| The per-child reading of that rate | A 5 percent second-answer rate means about one item in 20 has a second answer a teacher would accept, which the key marks wrong if a child picks it. That is an item rate, not a child's chance of being marked wrong, and it is still the sentence to write next to the number. |
| The slice the bar was not checked on | In the K-2 hold-out in Session 2, 4 of 36 items had a second answer a teacher would accept. A bar met on the whole set can be missed on a slice. |
One reader is also a sample
The literacy reviewer's reading is the standard this series builds toward, and it is also a sample. A rater who reads 39 stories closely sees what those 39 contain. A rare failure is more likely to be found where more stories are read with a check that recognizes it. Without such a check, the first report may come from a classroom. The Session 3 question, when the rater and the judge disagree, who decides, has a companion: for the cases neither has seen yet, which check would find them?
Repeatable under a condition, or a rate
Before ruling on a disagreement, ask whether the fault reproduces every time under a condition that can be stated (a rubric line that fires on every tie for longest answer) or has no such condition yet (an occasional second acceptable answer). The first is settled by reproducing it and changing the rule or the wording, then rerunning the same cases. The second is a rate to measure against the bar while the cause is looked for. Neither answer by itself says what the fix is.