Whose “Good Enough”?: the design
Three 75-minute sessions for a product team whose literacy reviewer, engineers and manager mean different things by “a good quiz question”.
Part of a series written for one product team. The companion series it refers to, From Concern to Evidence, is not online yet; where this page points at that packet, the team has it in hand.
Three 75-minute sessions for the moment when the literacy reviewer, the engineers and the manager mean different things by "a good quiz question". The parent design's sequence was: surface assumptions, build literacy-quality criteria, examine language and representation, negotiate disagreements, record an accountable decision. This version compresses the first two and the last two, because the team already has written rules and written bars, and the live work is agreeing what they mean on a real item. It overlaps Sessions 1 and 4 of From Concern to Evidence (From Concern to Evidence) and reuses their worksheets. The literacy reviewer leads more of this series than any other.
What the team should be able to do afterward
- State their own standard for a quiz item before hearing anyone else's, and see where it differs from a colleague's.
- Apply the five answer-choice rules to a real item and say which rule it breaks, with the words that break it.
- Judge reading load and wording at a grade, with the literacy reviewer's ceiling as the anchor.
- Record a disagreement between the human rater and the model judge as a decision with an owner, so the number does not settle it by default.
The three sessions
Same shape as From Concern to Evidence: write first, short explanation, supported practice, individual exit task, the five anonymous questions (the From Concern to Evidence packet, closing section).
Session 1: Say what good looks like, then compare
Cases (packet Part 1): Mason's good and bad versions of a lesson question about Brad and Bruno; the five answer-choice rules; a current QA item whose key is a detail rather than an answer.
| Time | Activity |
|---|---|
| 0 to 8 | Purpose, norms. Each person writes their own one-sentence definition of a good quiz item and hands it in. |
| 8 to 18 | Alone: the bad and good Brad and Bruno items, side by side. Which is better, and what exactly is different? |
| 18 to 30 | The literacy reviewer reads the definitions without names and groups them. Where do the engineers' and the reviewer's definitions part? |
| 30 to 42 | Mason presents the five rules, two minutes, then the group maps each rule to the thing it fixes in the bad item. |
| 42 to 47 | Break. |
| 47 to 62 | Pairs apply the rules to the Sprocket wrench item and one other current QA item. Name the rule broken and the words that break it. |
| 62 to 75 | Individual: rewrite one wrong answer so it carries as much detail as the key. Five questions. |
- Facilitator move: when a definition says "fair" or "appropriate", ask what the literacy reviewer would point to.
- Success check: every pair names a rule and the words, and at least one rewritten wrong answer is as specific as its key.
- Live activity, ten to twelve minutes: run the session set from the packet's link before Mason presents the rules, so the picks are not shaped by them. Everyone can do it alone before Session 1, or Session 1 can run to 87 minutes with the activity right after the definitions are handed in. Put the combined tally on the board before anyone talks.
- Between sessions: apply the five rules to one quiz on your own QA account and write which rule, if any, it breaks.
Session 2: Language, reading load and who the question is for
Cases (packet Part 2): a Lexile 315 story with answer choices of 11 to 13 words; the K-2 hold-out items where young-grade why questions had two sensible reasons; the literacy reviewer's 15-word ceiling and the judge's different reading of the same rubric line.
| Time | Activity |
|---|---|
| 0 to 8 | Report back. Retrieve the five rules. |
| 8 to 18 | Alone: read the Lexile 315 item aloud at a child's pace and time it. Would a kindergartener finish the choices? Write yes, no or depends, and on what. |
| 18 to 30 | The literacy reviewer leads: what reading load means at K-2, 3-5 and 6-8, and why full sentences at grade 4 and up and shorter choices at K-3 are both rules. |
| 30 to 45 | Pairs: the two young-grade why items where a teacher accepted a second answer. Is the fault in the question, the key, or the story? |
| 45 to 50 | Break. |
| 50 to 65 | Groups write a reading-load criterion with an observable anchor per grade band, and test it on three items. |
| 65 to 75 | Individual: one place where "right at this grade" and "right" come apart. Five questions. |
- Facilitator move: do not let the group settle on a word count alone. Ask what else makes a sentence hard at six.
- Success check: the criterion has a different anchor per band and a "cannot tell" case.
- Between sessions: none.
Session 3: When the rater and the judge disagree, who decides?
Cases (packet Part 3): the Zara's Quiet Garden comparison, 51 of 72 cells agreed; the five disagreements with their reasons; the open question of which one is the reference.
| Time | Activity |
|---|---|
| 0 to 8 | Report back. Retrieve the Zara disagreements. |
| 8 to 18 | Alone: for each of the five disagreements, write who you think is right and why. |
| 18 to 30 | Reveal the split. Sort the five into: definition gap (fix the rubric wording), judgment call (the literacy reviewer decides), and open (needs more cases). |
| 30 to 45 | Groups draft the decision record for one disagreement: the ruling, the owner, the rubric line it changes, the date it is reviewed. |
| 45 to 50 | Break. |
| 50 to 65 | Groups trade records and ask the two questions that would stop them acting on it. |
| 65 to 75 | Individual: the one line in the rubric you would change first. Five questions. |
- Facilitator move: the judge is not the tiebreaker because it is faster. Ask who has the expertise for each disagreement, and whether that expertise has been written down anywhere the judge could read it.
- Success check: each record names a ruling, an owner and a rubric line; none says "the model will decide".
- Afterward: Mason brings the agreed rubric changes to the next harness run and reports which disagreements closed.
What the series keeps from the research
Three points, where they change what happens in a session:
- The rater's standard is the one the series is building toward, so it is heard first and written down. Peters and Armstrong's Type Three teaching has the teacher as "a member of the group of learners" whose knowledge "does not necessarily supersede that of the other learners" (1998, p. 79). Here that cuts both ways: Mason's harness does not outrank the literacy reviewer's reading, and the literacy reviewer's reading is written down so the engineers can build to it. Say at the start what is open. On a literacy ruling the literacy reviewer has the final word by role. What the group negotiates is the wording, the evidence and what the judge gets told.
- The disagreement stays in the record instead of being averaged away. Mezirow's conditions for discourse include being "free from coercion" (1997, p. 10), and Brookfield's reply is that a leader's avowal of openness does not make it so; the structure has to do it (Brookfield, 2006, p. 22, on who speaks when passing is allowed). The written-first rule and the anonymous cards are that structure.
- A standard needs a worked example at each grade, and less scaffolding for the person who already reads at that grade for a living. Kirschner, Sweller and Clark on worked examples (2006, p. 80) and on guidance receding with prior knowledge (p. 75). The literacy reviewer does not need the Lexile 315 item read aloud to her; the engineers do.
Reuse from From Concern to Evidence
- Vocabulary, concern worksheet, three-criteria table and audit sheet: the From Concern to Evidence packet, Parts 1 and 4.
- The Zara disagreements and the four Part 4 items: the From Concern to Evidence packet, Part 4.
- Five closing questions and opening eight minutes: From Concern to Evidence.
Appendix
The appendix is a short optional note on reading a bar as a rate with a sample. It adds three fields to the decision record: the accepted rate and its sample, the per-child reading of that rate, and the slice the bar was not checked on.
Citation audit
All three sources were verified for From Concern to Evidence on 2026-11-07 (Crossref match or publisher record, read in the copy on disk). Peters and Armstrong (1998), doi:10.1002/ace.7908, p. 79: verified. Mezirow (1997), doi:10.1002/ace.7401, p. 10: verified. Brookfield (2006) workshop packet, author's site, p. 22: verified, practitioner account. Kirschner, Sweller and Clark (2006), doi:10.1207/s15326985ep4102_1, pp. 75 and 80: verified. No other sources are cited in this folder.