Print the whole workshop

Part of a series written for one product team. The companion series it refers to, From Concern to Evidence, is not online yet; where this page points at that packet, the team has it in hand.

Optional reading. The arithmetic and its assumptions are worked out in the fuller appendix in the From Concern to Evidence series.

Why the bar cannot be zero

The quiz writer is probabilistic. A bar of "no second acceptable answer, ever" cannot be checked on a sample. Mason's bars are rates for that reason: strong-reader guessing at most 55 percent, child reader at most 38, second acceptable answer at most 5 percent. A ruling that sets or moves a bar should also say what sample the rate stands on and who bears the misses.

Three things to add to the decision record

The decision record in the packet has seven fields. When a ruling sets or moves a bar, add these:

Field Why
The accepted rate and the sample it was measured on The pipeline table in One Change One Claim shows a second-answer rate of 3.8 percent on the 39 frozen stories and 5.3 percent on the 24 new ones. If the measure counts items and each quiz has four, those sets hold about 156 and 96 items, so one item moves the first figure by about 0.6 points and the second by about 1 point. Confirm the denominator from the harness output before reading the gap between them as a change, and remember that the four items in a quiz share a story, so they are not independent observations.
The per-child reading of that rate A 5 percent second-answer rate means about one item in 20 has a second answer a teacher would accept, which the key marks wrong if a child picks it. That is an item rate, not a child's chance of being marked wrong, and it is still the sentence to write next to the number.
The slice the bar was not checked on In the K-2 hold-out in Session 2, 4 of 36 items had a second answer a teacher would accept. A bar met on the whole set can be missed on a slice.

One reader is also a sample

The literacy reviewer's reading is the standard this series builds toward, and it is also a sample. A rater who reads 39 stories closely sees what those 39 contain. A rare failure is more likely to be found where more stories are read with a check that recognizes it. Without such a check, the first report may come from a classroom. The Session 3 question, when the rater and the judge disagree, who decides, has a companion: for the cases neither has seen yet, which check would find them?

Repeatable under a condition, or a rate

Before ruling on a disagreement, ask whether the fault reproduces every time under a condition that can be stated (a rubric line that fires on every tie for longest answer) or has no such condition yet (an occasional second acceptable answer). The first is settled by reproducing it and changing the rule or the wording, then rerunning the same cases. The second is a rate to measure against the bar while the cause is looked for. Neither answer by itself says what the fix is.