SCORE IT YOURSELF
How reliable is the AI that grades your AI? Five ways a quality score lies to you, and three questions to ask about any number someone hands you.
Thursday, 2:00 PM, Theater 3
Score It Yourself: How Reliable Is the AI That Grades Your AI?
Thursday 8 October 2026, 2:00 to 2:30 PM
Theater 3, The Cinemas, 11 Fulton Street, at the Seaport. Theater 3 is confirmed; check the
agenda on the day in case it moves.
A 30-minute session: about 26 minutes of talk, one short exercise you do from your seat, then questions.
Nothing to install, nothing to bring.
For the people who approve AI work and the engineers who build it. No technical background needed.
Abstract
Many leaders approve AI features they have never inspected themselves. The demo goes well, a number gets reported, and the decision rests on someone else's judgment about what "good enough" means. Once a product is working at volume, nobody reads all of the output by hand. They sample a few, count what a computer can count, and more and more often let another AI score the rest. That score is what reaches the person who signs off.
This session takes apart the numbers behind one AI product, a reading app used in grades 1 through 8, and shows where each one went wrong. You start as the test taker: comprehension questions an AI wrote, shown without the stories they belong to. You answer them anyway. Then you see how four AI evaluators answered the same questions with the stories withheld, three runs each. After that, 57 children on fifty questions written to a stricter rule, with the giveaways in the choices removed so they are harder to guess without the story. From there, five ways a quality score lies to you, each from a real result: it rewards sameness; it cannot move; the evaluator is the variable; the failures are not in the average; nothing was compared. Each comes with its fix, in one sentence a non-engineer can say to an engineer.
The talk closes with three questions to ask about any AI quality number, and a page that carries every quiz with each evaluator's verdict beside it.
About the speaker
Mason Ulrika Borchard is a senior software engineer at Stride, Inc., where they lead engineering for Story Studio, the company's AI reading app for K-8 students. They came to engineering from teaching. The job has kept both: they write the code, and they answer for whether what it generates holds up as teaching material, from reading-level calibration to grade-level content standards.
They also built and run the evaluation program behind it. Over the past five months that has meant thousands of evaluator runs over generated stories, quizzes and illustrations, always on fixed test material, with the scoring rules written down before anything is scored. Its results have reversed a model choice before cutover. Twice it has sent Mason back to withdraw a finding they had already reported. The current version seats four AI evaluators from four LLM families. Each takes a turn in all four prompt roles, and the published scores carry the evidence the evaluator quoted for them. The children's result in the talk, fifty rewritten comprehension questions answered on paper by 57 students in kindergarten through grade 8 with the stories withheld, was a product check on whether the questions could be guessed, run inside the app's own quality process.
Their doctoral work at Valdosta State University, in adult learning and development, is separate from Story Studio. They are building software that uses AI to simulate real research sites for undergraduate students in ecology and evolutionary biology. Many of those students never get a traditional undergraduate research experience. Some work full-time alongside school and cannot take on unpaid research; others live far from the places they want to study, or attend universities that can fund only a handful of positions.
Outside Stride they are technology lead and a program planner for The Synapse, a conference for women working across consciousness, AI, cognitive science and robotics. With Timothy Beach they build Aegix Linux, a turn-key Linux distribution on Artix with suckless software, LUKS encryption and the BTRFS filesystem. They co-founded Ionosphere, a consciousness-research platform built around a quantum random number generator, and lead its statistical analysis. Its first two experiments were presented at the 2026 ACORN Symposium; a sealed follow-up is on the program at The Science of Consciousness 2026.
Before engineering they taught science in a Title I high school and in a juvenile detention facility. They hold an MAT from USC Rossier, where they were a Robert Noyce Scholar. Story Studio began as their winning entry in Stride's 2023 hackathon, and they are recognized in the XPRIZE Hall of Fame as a Pandemic Response Challenge finalist.
The full deck
The complete 60-minute version of the talk, with every slide from the 30-minute session and the sections there was no time for: the second exercise, what held up, what changed in BQTH, and the checks that caught the checkers.
- VIEW IN THE BROWSER Score It Yourself: the full deck, arrow keys or click to move through it
- DOWNLOAD The full deck as a PDF, 44 pages
The exercise, with the answers
Six comprehension questions an AI wrote for a children's reading app, each belonging to a story you were not shown. In the room you answered questions 1 and 6 by a show of hands. Four AI evaluators answered all six, three runs each, with the stories withheld. The first five were written under the old item rules; question 6 under the new ones.
Based on specific details in the story, explain why Bubbles and Cat could not have saved the alien ship if only one of them had tried. 695L, about grade 4.
A. The alien ship's signal was too weak for one frequency to boost.
B. The transmitter was too heavy for one animal to hold steady.
C. One animal would have gotten too tired before finishing.
D. The device required two different frequencies adjusted simultaneously, and each animal had different paws suited to different tasks.Answer: D. Old rules. Every evaluator, every run, without the story. The longest choice by a mile, and the only one with any detail. You answered this one in the room.
Why did Bubbles and Giggles decide to help the dragon instead of taking the treasure? 331L, about grade 2.
A. Giggles whispered that they should leave the cave.
B. They were too scared to take the gold coins.
C. The dragon showed them a map to the real treasure.
D. The dragon was hurt and needed help more than they needed gold.Answer: D. Old rules. Every evaluator, every run, without the story. The kind answer.
Why did Jose initially hesitate to tell anyone about the missing grain? 1090L, about grade 6.
A. He wanted to solve the problem by himself first.
B. He feared severe punishment for admitting the discrepancy.
C. He thought the grain had simply been miscounted by mistake.
D. He did not trust Maria to keep the secret safe.Answer: B. Old rules. Every evaluator, every run, without the story. The shortest choice, so it is not only length.
Based on specific details in the story, what evidence shows that the shared guardianship agreement actually worked? 908L, about grade 5.
A. Miguel became famous and received statues in his honor.
B. Two weeks later, both groups were working together on construction and irrigation, sharing food, and planning to teach their children the spring's history together.
C. No more disputes ever occurred in the enchanted lands.
D. Kess and the Thornheart leader became best friends immediately.Answer: B. Old rules. Every evaluator, every run, without the story. 24 words against 9, and the wrong ones say "immediately" and "ever".
What did Bubbles and Giggles find behind the pile of rocks? 331L, about grade 2.
A. Soft moss and smooth textures.
B. A young dragon with a hurt wing.
C. Gold coins and a golden egg.
D. A map showing the treasure path.Answer: B. Old rules. Three of the four evaluators, every run, without the story. Question 2 gives it away: keep questions independent.
What did Brad see in the sky over the ice? 1227L, about grade 8.
A. A spiral of green aurora light
B. A ring of falling winter stars
C. A wall of black storm cloud
D. A fissure like a cracked horizonAnswer: D. New rules. Without the story, the evaluators got it in 2 of 10 runs; with the story, 10 of 10. One student in six got it on paper without the story. You answered this one in the room too.
Fifty questions, 57 children, four evaluators
Fifty-seven students in kindergarten through grade 8 answered fifty of the rewritten questions on paper with the stories withheld, as a product check on whether the questions could be guessed. They answered 102 of 461 correctly, about one in five, at the 25 percent that guessing on four choices would give. Part of that is by construction: each wrong answer was written to be the natural guess from the question alone. The four AI evaluators answered the same questions without the stories at 40 to 59 percent and with the stories at 98 to 100 percent. No child has yet answered these questions with the story in front of them; that is the next check. Nothing identifying was recorded with the answers, and the result is reported as one pooled count.
| Who | Answers planned | Answers scored | Correct, story withheld | Correct, story shown |
|---|---|---|---|---|
| 57 children, on paper | -- | 461 | 102 (22%) | not yet tested |
| AI Evaluator #1 | 150 | 123 | 59% | 100% |
| AI Evaluator #2 | 150 | 150 | 51% | 100% |
| AI Evaluator #3 | 150 | 131 | 40% | 99% |
| AI Evaluator #4 | 150 | 142 | 48% | 98% |
Five ways a quality score lies to you
- The score rewards sameness.Ask what the cheapest way to score well would be. An image model told to keep one room the same across four pages earned top marks by returning the same picture four times, including the page that was supposed to be at night. If the cheapest way to pass is also a bad outcome, something will find it.
- The score cannot move.A run came back 32 of 32 at exactly 100%. Nothing short of a collapse would have registered, so the ordinary decline you actually need to catch was the decline nobody could see. Harder test material brought the same work back to 91%, and the test could see again.
- The evaluator is the variable.The same twelve images were scored 4.92 by one AI evaluator and 3.92 by another. On a set of reading questions, four evaluators spread from 89.5% to 97.5%, and the oldest of the four disagreed with itself, run to run, on ten of 24 stories.
- The failures are not in the average.A model-selection summary recommended the wrong model, because 11 of its 80 attempts failed before they could be scored. The model that failed more often looked better, because its worst results never reached the average. Ask how many started.
- Nothing was compared.A 20-point gain justified weeks of new engineering until a $2.50 experiment compared it against what the product already did, and found the whole gain sitting there. A number with nothing beside it is not a result. It is a reading.
Three questions for any AI quality number
1. Who did the judging, and who checked the judge?
2. What would move this number? A second AI evaluator, a second run, the failures put back in the average, harder test material.
3. What did the same inputs produce before the change? That is how you tell a fix from a coincidence.
The test you can run yourself
Take the source away. If an AI's output is supposed to be based on a document, withhold the document and ask again. It works on contract summaries, triaged support queues, answers from your own knowledge base, anything that claims to rest on a source.
It tells you whether the output leaned on the source. Whether that is a problem depends on what the output was for, and that judgment is still yours.
What to send the engineer before you sign
Thanks for the summary. Before I sign off I'd like three things beside it:
1. The rows. One line per item, per evaluator, per run, with the evaluator's written reason in the column next to its score. I'll read ten of them, and I'll pick which ten.
2. The funnel. How many items were requested, how many came back, how many were scored. If those three numbers differ, a line on what happened to the difference.
3. The before. The same items run on the version we have in production today, scored the same way, in a column beside the new one.
And one sentence: what is the cheapest way for this feature to score well on this test, and is that case in the test set?
Producing this is engineering. Reading it is not. If nobody can answer the third question inside a day, that is your Monday: freeze twenty real examples and run every future test on the same twenty.
Two ways to get to the sign-off
BQTH, the Borchard Qualitative Testing Harness, is the multi-LLM testing harness behind every number in this talk.
What we used to do. Release candidate, BQTH results, human QA that had already seen the score, sign-off, ship. Then things that used to work stopped working, and an unplanned fix release.
What I'm proposing. Release candidate. BQTH results, as a sheet with one row per item, per evaluator, per run. QA reads the rows, not the summary, and scores before seeing the evaluators' scores. The method goes with the results to everyone who signs. A UX designer, a subject-matter expert and a teacher read the same items, each privately first, then together. Children try the items on paper. Then the same sign-off. Some of this already happens: BQTH runs, the children's check ran, and blind scoring by QA is starting. BQTH runs on my own machine and is not yet a company repository. The rest is a proposal, and its results will be posted here when they exist.
For engineers: how the run is built
The panel. Four evaluators from four LLM families, each handed four prompt roles (teacher, reviewer, item writer, editor) in a Latin square, so every model plays every role once across four runs and no role is welded to one family. On a pilot batch of eight stories, switching the model moved the text score 0.7 points on a 1-to-4 scale and switching the role moved it 0.2. Chance-corrected, agreement within one evaluator across its four roles is 0.63 to 0.75; between evaluators it is 0.13 to 0.20. Eight stories is a pilot: a direction, not a ratio.
The sheet. One tab per item. One row per item, per evaluator, per role, per run, with the score in one column and the evaluator's quoted evidence in the next. Every fail must quote evidence.
The gallery. Every generated item beside its score, the rubric it was scored against, and the evidence the evaluator quoted, on one page a reviewer can scroll.
The blind block. For every item a person scores, the person's scoring block sits above the evaluators' block, and the evaluators' block stays collapsed until the person's score is in. It is a layout and a rule, not a lock.
The honest limit. These are several pipelines, not one. The pipeline that generates and scores runs one evaluator per story. The four-evaluator panel is a second script, run where the question demanded it. A third withholds the passage. No evaluator is pinned to a temperature; three runs each is how the run-to-run spread is measured.
Who scores, and how
Who. A QA engineer who did not build the system, and the engineering lead, each scoring a random fifth of every run without seeing the other's score or the evaluators' scores. A UX designer, the curriculum lead and, when available, an English language arts specialist read ten stories from that fifth, privately, before any discussion. Children answer the quiz items on paper with the story withheld.
The rule. The rubric is revised on a pilot batch and frozen before the scored run; every wording change is logged with its reason. Every item where evaluators split, or where a person and the panel split, is read by a person and gets a one-line reason.
What gets reported. Percent agreement of each person with each evaluator on the random fifth. Between two people, weighted kappa on scales and AC1 on pass/fail gates. The split list is a defect list, not a statistic. The human comparison is posted here when the scores exist, and not before.
How adults learn to read evaluation results
Most people who sign off on an AI feature were never taught how to read the evidence behind it. Four findings from adult learning and development research, and what each means in practice.
- Teach on their own problems.Adults learn best from problems in their own work, and want to know why before how (Knowles, 1980). Train reviewers on your own product's items, not a generic tutorial.
- Worked examples, then fading.A newcomer learns a judgment fastest by studying an expert's worked example, then taking over more of the steps while explaining each one (Atkinson, Derry, Renkl and Wortham, 2000; Atkinson, Renkl and Merrill, 2003). Walk a reviewer through three scored items with the evaluator's quoted evidence, then have them score the fourth and say why.
- A briefing is not training.Untrained readers judged AI-written stories at chance; instructions and annotated examples raised accuracy to 55% at best (Clark and colleagues, 2021). Practice on real items with feedback.
- One page, in layers.Guidance that helps a newcomer gets in an expert's way (Kalyuga, Ayres, Chandler and Sweller, 2003). Put the item and the verdict on top, the evidence and the rubric a click below.
For engineers: presenting qualitative results
- Counts, not percentages."3 of 50 failed" rather than "94%". People with no statistics training reason correctly far more often when information is given as natural frequencies (Gigerenzer and Hoffrage, 1995).
- The item beside its score.A bare number moves across a team without anyone having to understand it. The item, its score and the quoted evidence give other roles something to translate and push back on (Carlile, 2004).
- Define the terms first.Evaluator, rubric, passage dependence: one line each, before the first number, so the reader is not holding new words and a table at once (Mayer and Moreno, 2003).
- End with an if-then."If one evaluator flags it and the other does not, a person reads it before release." Plans in if-then form are carried out far more often than intentions (Gollwitzer and Sheeran, 2006).
- Overview first, details on demand.Open on the roll-up, let the reader filter, and show one item when asked (Shneiderman, 1996). Show outcomes people can count, a row per run rather than an error bar (Hullman, Resnick and Adar, 2015; Kay, Kola, Hullman and Munson, 2016). Let the reader change the view (Dragicevic and colleagues, 2019).
- Their judgment first.Scoring before seeing the answer is how people learn (Roediger and Karpicke, 2006). In the BQTH review sheet, each story tab has the reviewer's block above the evaluators' block, and the evaluators' block is collapsed until the reviewer has scored.
Read next
Every article below is one of the runs or practices the talk drew on. The note under each says which part of the talk it backs.
The results, item by item
- SEE IT Quiz Judging, Version 2: Four Evaluators, Four Roles, One Square. Every quiz from the evaluation panel slide, with each evaluator's verdict and quoted evidence, and the children-versus-evaluators table.
- SEE IT Image Judging, Version 2: Four Evaluators on the Same Pictures. The same pictures scored by four evaluators, every score beside the picture it describes.
The experiments behind the slides
- THE EXERCISE Can an AI Write a Quiz a Child Can't Game?. The first quiz evaluation: answer-length leaks, hint caps and blind scoring. Where the longest-answer giveaway in question 1 came from.
- THE EXERCISE The Nicest Answer Was Right 96% of the Time. A general check for answer giveaways across 12,083 generated quiz items. The "kind answer" in question 2, at scale.
- SIGN 1 Text-to-Image Prompting for Character Continuity, Setting Continuity, and Scene Fidelity. The continuity evaluation where one pinned seed scored five out of five by returning the same picture four times.
- IMAGES Can an Image Model Draw a Whole Comic Page?. A three-model evaluation of panel layout, story progression and cost, with every judged page.
The engineering practices
- RULE 2 Test the Stage You Changed, Not the Whole Pipeline. Change one thing and pin everything else: a harness that runs only the stage under test.
- WHAT SHIPPED Wire a Fallback Model Before Your Credits Run Out. The fallback that tells you every time it fires, from the list of what shipped because of the measuring.
- SIGN 4 The Validator Was the Bug. Stricter validation that quietly dropped stories. What to compare before you trust a count that changed.
- FOR ANYONE WHO USES CLAUDE CODE The five skills behind this talk. Including the one that caught the invented citations: files, install steps, what each does.
- FOR ENGINEERS Build Your Own Agent Skills: Model-Debate, Content-Matter-Expert and a Peer-Review Panel. A second LLM family that argues back, and a source-backed expert that checks claims.
- FOR ENGINEERS Put Your AI's Sparring Partner on Bedrock. Running the second-family evaluator on your own cloud account instead of a personal key.
Measurement discipline
- RULE 3 Lock One Analysis Battery Across All Datasets. One fixed analysis run over every dataset instead of a new one for each, for the same reason the scoring rules are written before anything is scored.
- RULE 3 When Your Stats Are Exploratory, Say So Up Front. How to tell readers which results were planned and which were found along the way.
Sources
Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181–214. doi.org/10.3102/00346543070002181
Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems. Journal of Educational Psychology, 95(4), 774–783. doi.org/10.1037/0022-0663.95.4.774
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. doi.org/10.1016/0149-7189(79)90048-X
Carlile, P. R. (2004). Transferring, translating, and transforming: An integrative framework for managing knowledge across boundaries. Organization Science, 15(5), 555–568. doi.org/10.1287/orsc.1040.0094
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021). All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. ACL-IJCNLP 2021. doi.org/10.18653/v1/2021.acl-long.565
Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019). Increasing the transparency of research papers with explorable multiverse analyses. CHI 2019. doi.org/10.1145/3290605.3300295
Gautam, S., Liu, H., Choi, Y., & Lease, M. (2026). How researchers navigate accountability, transparency, and trust when using AI tools in early-stage research: A think-aloud study. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency. doi.org/10.1145/3805689.3812262
Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. doi.org/10.1037/0033-295X.102.4.684
Gollwitzer, P. M., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta-analysis. Advances in Experimental Social Psychology, 38, 69–119. doi.org/10.1016/S0065-2601(06)38002-1
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333. doi.org/10.1207/S15324818AME1503_5
Haldar, R., & Hockenmaier, J. (2025). Rating roulette: Self-inconsistency in LLM-as-a-judge frameworks. Findings of the Association for Computational Linguistics: EMNLP 2025. arXiv:2510.27106
Hullman, J., Resnick, P., & Adar, E. (2015). Hypothetical outcome plots outperform error bars and violin plots. PLoS ONE, 10(11), e0142444. doi.org/10.1371/journal.pone.0142444
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31. doi.org/10.1207/S15326985EP3801_4
Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016). When (ish) is my bus? CHI 2016, 5092–5103. doi.org/10.1145/2858036.2858558
Keenan, J. M., & Betjemann, R. S. (2006). Comprehending the Gray Oral Reading Test without reading it: Why comprehension tests should not include passage-independent items. Scientific Studies of Reading, 10(4), 363–380. doi.org/10.1207/s1532799xssr1004_2
Knowles, M. S. (1980). The modern practice of adult education: From pedagogy to andragogy (Rev. ed.). Cambridge.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52. doi.org/10.1207/S15326985EP3801_6
Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37. arXiv:2404.13076
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. doi.org/10.1177/0018720810376055
Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning. Psychological Science, 17(3), 249–255. doi.org/10.1111/j.1467-9280.2006.01693.x
Säuberli, A., & Clematide, S. (2024). Automatic generation and evaluation of reading comprehension test items with large language models. Proceedings of the 3rd Workshop on Tools and Resources for People with Reading Difficulties (READI). ACL Anthology 2024.readi-1.3
Schooler, J. W. (2011). Unpublished results hide the decline effect. Nature, 470(7335), 437. doi.org/10.1038/470437a
Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. IEEE Symposium on Visual Languages, 336–343. doi.org/10.1109/VL.1996.545307
Strathern, M. (1997). “Improving ratings”: Audit in the British university system. European Review, 5(3), 305–321.
Tuinman, J. J. (1973–1974). Determining the passage dependency of comprehension questions in 5 major tests. Reading Research Quarterly, 9(2), 206–223. doi.org/10.2307/747135
Verga, P., et al. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv:2404.18796
Zheng, L., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. arXiv:2306.05685