Women Who Code Leadership Summit · New York · 8 October 2026

Score It Yourself

How reliable is the AI that grades your AI?

Mason Ulrika Borchard

Senior Software Engineer, Stride, Inc. · Doctoral Student, Valdosta State University

1

4.6 / 5

Story quality

94%

Pass rate

What would you want to know before you approved this?
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
2
Why the testing harness exists
the numbers Can we upgrade Story Studio's content generator from GPT-3.5? Even GPT-4.1 mini wins. Here's the research. Great evidence! Very thorough research! I agree! So... no. Let's circle back. SYNERGY DAVE, SEGUE-ING
Circling back all day without getting anything done.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
3

Story Studio

An AI reading app for grades K to 8: a story, its pictures and a quiz, generated for one child.

Began as the winning entry in the company's 2023 hackathon. The CEO asked for it to be built.

The team

One frontend engineer. One backend engineer. One QA engineer on two teams. One UX designer. A curriculum lead who reports to the CEO. Me, and since the budget cuts, the reading specialist too.

How it is tested

Automated checks that count. BQTH, the Borchard Qualitative Testing Harness: AI evaluators from four LLM families. A QA engineer, testing every release. A former teacher, reading.

What it cost

The first AI evaluator runs came out of my own pocket: a few hundred dollars. The experiment that stopped a multi-week build cost $2.50.

1,900+

stories

1,800+

images

1,500+

quizzes, in about five months

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
4
How we used to get to the sign-off
WHAT WE USED TO DO Release candidate BQTH results Human QA sees the score first Sign-off Ship Things that used to work stop working An unplanned fix release
Most of what follows was found after this process. Today: you take a test → how the number is made → five ways it lies → how people learn to read it → three questions
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
5
Exercise 1 · one question, no story · hands up

Based on specific details in the story, explain why Bubbles and Cat could not have saved the alien ship if only one of them had tried.

A question from a children’s reading app, 695L, about grade 4. The story is removed. Pick an answer anyway.

A  The alien ship's signal was too weak for one frequency to boost.
B  The transmitter was too heavy for one animal to hold steady.
C  One animal would have gotten too tired before finishing.
D  The device required two different frequencies adjusted simultaneously, and each animal had different paws suited to different tasks.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
6
Exercise 1 · one more · hands up

What did Brad see in the sky over the ice?

The same app, 1227L, about grade 8. The story is removed. Pick an answer anyway.

A  A spiral of green aurora light
B  A ring of falling winter stars
C  A wall of black storm cloud
D  A fissure like a cracked horizon
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
7
Exercise 1 · five written under the old rules, one under the new · four AI evaluators, no story, three runs each
#LevelKeyRight, every runWhy
1695LD4 of 4Longest choice by a mile, the only one with detail
2331LD4 of 4The kind answer
31090LB4 of 4The shortest choice. Still the one a person would feel
4908LB4 of 424 words against 9. "Immediately." "Ever."
5331LB3 of 4Question 2 gives it away
61227LD0 of 4Written under the new rules. With the story: 4 of 4. Students: 1 of 6
Fifty items under the new rules: evaluators 40 to 59% without the story, 98 to 100% with it. Fifty-seven children on paper, no story: at the chance line.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
8
57 students, kindergarten to grade 8 · fifty rewritten questions · a product check
Story withheld (dashed line: chance, 25%)Story shown
57 students, on paper
102 of 461 answers
22%not yet
Claude Opus 4.7
123 scored
59%
100%
GPT-6 Sol
150 scored
51%
100%
Gemini 3.8 Flash
142 scored
48%
98%
Nova Pro
131 scored
40%
99%

Guessing at random on four choices: 25%. Every wrong answer was written to be the natural guess from the question alone. Unreadable evaluator answers count as missing, not wrong.

At the chance line without the story. A guessability check, not a reading score.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
9
Passage dependence · 95 questions · 24 stories · grades 1 to 8 · 3 runs each
With the storyWithout
AI Evaluator #198.9%97.5%
AI Evaluator #299.3%96.8%
AI Evaluator #399.6%96.5%
AI Evaluator #499.6%89.5%

25%

Random guessing

Calibrating a judge against human labels would pass every one of these. Calibration checks the judge. It doesn't check the question.
Tuinman (1973); Keenan & Betjemann (2006); Säuberli & Clematide (2024). Full references on the last slide.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
10
The evaluation panel, on the latest batch
GENERATE Stories, quizzes and picture prompts the product’s own prompts; Haiku 4.5 wrote the batch Pictures Stable Image Core, gpt-image-2, Gemini image EVALUATE Anthropic Claude Opus 4.7, Bedrock text, pictures, quiz OpenAI GPT-6 Sol text, pictures, quiz Amazon Nova Pro, Bedrock text, pictures, quiz Google Gemini 3.8 Flash text, pictures, quiz Four LLM families, four prompt roles: teacher, reviewer, item writer, editor. Every model played every role once. 8 new stories, a pilot: switching the model moved the text score 0.7 points; switching the role, 0.2. T R W E R W E T W E T R E T R W four runs, logged COUNT Not AI: words, failures, cost free, and the same answer every time SHEET One row per item, per judge, per role, per run quoted evidence beside each score; 120 clean judgments of 128 on 8 stories, and every story’s verdict split across the evaluators ARGUE Research first it reads before it argues Every citation checked fetched and read, not trusted Another family argues against the conclusions A person reads the splits wherever evaluators disagree The second layer exists only to attack the first: it audits what BQTH concludes, not the stories. Panel of families: Verga et al. (2024). Self-preference in judges: Panickssery, Bowman & Feng (2024).
11
The harness, the version being built
GENERATE Stories, quizzes and picture prompts Haiku 4.5 writes the batch; gpt-3.5-turbo stays as the unchanged comparison Pictures Stable Image Core, gpt-image-2, Gemini image EVALUATE Anthropic Claude Opus 4.7 literacy teacher OpenAI GPT-6 Sol ed-software reviewer Amazon Nova Pro assessment item writer Google Gemini 3.8 Flash children’s book editor Four LLM families. A role is a prompt condition, not an expert. Every model plays every role once. Rubric: story text, pictures, quiz. Gates, checks, 1-to-4 scales with written anchors. T R W E R W E T W E T R E T R W four runs, logged COUNT Not AI: words, failures, cost free, and the same answer every time SHEET One row per item, per judge, per run, per role 20 percent of items scored twice by people; every split between a evaluator and a person read by a person ARGUE Research first it reads before it argues Every citation checked fetched and read, not trusted Another family argues against the conclusions A person reads the splits wherever evaluators disagree [human comparison: pending, arrives after this talk] Panel of families: Verga et al. (2024). Self-preference in judges: Panickssery, Bowman & Feng (2024).
12

The discipline

Four rules, none of them technical

What makes anything the machinery produces mean something.

01

The same test items, every run

The 95-question set has not changed since it was written, so a difference cannot come from the items.

02

An unchanged version beside every change

The four-version experiment under sign 5 ran each one against what production already did.

03

Scoring rules written before scoring

The rubric was fixed before the 300-story test. People cannot see the AI scores until theirs are in.

04

Everything runs more than once

Three runs each. One evaluator changed its own answer on 10 of 24 stories.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
13 · 1 of 4

The discipline

Four rules, none of them technical

What makes anything the machinery produces mean something.

01

The same test items, every run

The 95-question set has not changed since it was written, so a difference cannot come from the items.

02

An unchanged version beside every change

The four-version experiment under sign 5 ran each one against what production already did.

03

Scoring rules written before scoring

The rubric was fixed before the 300-story test. People cannot see the AI scores until theirs are in.

04

Everything runs more than once

Three runs each. One evaluator changed its own answer on 10 of 24 stories.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
13 · 2 of 4

The discipline

Four rules, none of them technical

What makes anything the machinery produces mean something.

01

The same test items, every run

The 95-question set has not changed since it was written, so a difference cannot come from the items.

02

An unchanged version beside every change

The four-version experiment under sign 5 ran each one against what production already did.

03

Scoring rules written before scoring

The rubric was fixed before the 300-story test. People cannot see the AI scores until theirs are in.

04

Everything runs more than once

Three runs each. One evaluator changed its own answer on 10 of 24 stories.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
13 · 3 of 4

The discipline

Four rules, none of them technical

What makes anything the machinery produces mean something.

01

The same test items, every run

The 95-question set has not changed since it was written, so a difference cannot come from the items.

02

An unchanged version beside every change

The four-version experiment under sign 5 ran each one against what production already did.

03

Scoring rules written before scoring

The rubric was fixed before the 300-story test. People cannot see the AI scores until theirs are in.

04

Everything runs more than once

Three runs each. One evaluator changed its own answer on 10 of 24 stories.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
13 · 4 of 4

Counting

How many words is the right answer?

How often did it come back broken?

What did it cost?

Almost free. Same answer every time.

Opinion

Is this a good question?

Is the marked answer actually right?

Costs real money. Moves.


Five versions of the instructions, 80 questions each, counting only: about 75 cents. 53% gave the answer away by length; the best alternative, 22%.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
14
Five ways a quality score lies to you
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
15
Warning sign 1 of 5
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
16
Warning sign 1 of 5 · the score rewards sameness
Same angle
page 1
Same light
page 2
Same furniture, same spot
page 3
The story says it's night
page 4
Prompt 1 reading in the beanbag by the round window, daytime
Prompt 2 kneeling at the bookshelf, searching for a brass key, low angle
Prompt 3 standing with Mr Penguin by the chalkboard, wide shot
Prompt 4 night: the room dark, lit by one small lamp

Four prompts, one pinned seed. Consistency score: 5 out of 5, every page. Night palette: 1 out of 5. Three of the four files are byte-for-byte identical.

If the cheapest way to pass is also a bad outcome, something will find it.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
17
Warning sign 1 of 5 · the score rewards sameness

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Campbell (1979), Evaluation and Program Planning, 2(1), 67 to 90. Goodhart’s law as usually quoted: Strathern (1997).

What is the cheapest way to score well on this?

A support team judged on tickets closed closes tickets without solving them. A summary scored on words shared with the document copies the document.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
18

Warning sign 1 of 5

The score rewards sameness

Same failure, a different product, no images anywhere near it.

Grade 6 answer choices over the length limit

55%

Automated check

Clean.

Not one flagged.

The check asked

Is the right answer the single longest choice? If so, the question gives itself away.

The hole

If every choice runs long, none stands out as the longest. The cheapest way to pass: make every answer long.

Nobody intended it.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
19

Warning sign 2 of 5

The score cannot move

32 of 32 test cells

100%

After harder test material

91%

Why it's a problem

A score at the top of its range can't go down. It would catch a collapse, not ordinary decline.

The fix

Harder material until it moves again. A score that sits at the ceiling for months retires.

At the ceiling, a score has gone blind to the decline you need to catch.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
20

Warning sign 3 of 5

The evaluator is the variable

The question this session is named after.

The same 12 images, scored out of 5 by two AI evaluators

4.92

AI Evaluator #1

3.92

AI Evaluator #2

Same pictures, a full point apart. If only one number reached you, you would never know the other existed.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
21
Warning sign 3 of 5 · the evaluator is the variable

Same evaluator, same 95 questions, three runs

Stories where the answer changed, of 24Single questions that flipped, of 95
AI Evaluator #4 (GPT-4o)1010
AI Evaluator #2 (Claude Opus)34
AI Evaluator #3 (gpt-5.4)33
AI Evaluator #1 (Gemini)11

AI Evaluator #4 is the oldest of the four. Its answers moved between runs; the choices were reshuffled every run, for every evaluator.

None of the four is a child. Use them, and check them.
Run-to-run inconsistency: Haldar & Hockenmaier (2025). Position and length bias: Zheng et al. (2023). Self-preference: Panickssery et al. (2024).
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
22
Warning sign 3 of 5 · the evaluator is the variable
1Two evaluators agree on the answer key in 9 of 23. I withdraw "the answer keys are sound."
2A third AI evaluator agrees with the first in 31 of 35.
 Of the 23 all three evaluated: a majority says valid in 21. The second AI evaluator is the lone holdout in 13.
The vote tells you where to look, not what's true.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
23
Warning sign 4 of 5 · the failures are not in the average

The run summary recommended Model A.

Stories requestedFailed before scoringStories scored
Model A801169
Model B80080
How many started, and what happened to the ones that aren't here?
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
24

Warning sign 5 of 5

Nothing was compared

The result

A 20-point gain

In how well pictures matched their story.

The plan

Weeks of building

A new stage in the product, on the strength of it.

The test

Four versions

8 images each, every one against what production already did.

$2.50

The whole gain was already in the step production ran. We didn't build it.

Compare every change against what already runs.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
25
Five signs, five fixes
The signWhat we did about it
The score rewards samenessAsk what the cheapest way to score well is, and put that case in the test set
The score can't moveMake the test harder until it moves. A score at the ceiling retires
The evaluator is the variableA second AI evaluator from another LLM family, a second run, and a person reads where they split
The failures aren't in the averageEvery summary reports requested, returned, scored
Nothing was comparedAn unchanged version beside every change, on the same items
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
26
Two ways to get to the sign-off
WHAT WE USED TO DO WHAT I'M PROPOSING Release candidate BQTH results Human QA sees the score first Sign-off Ship Things that used to work stop working An unplanned fix release Release candidate BQTH results the sheet: one row per item, per evaluator, per run QA reads the rows, not the summary scores first, sees the score second Method and results go to everyone who signs Read by UX, a subject-matter expert, a teacher each privately, then together; starting now Tried by the people it is for 57 children, on paper, no story Sign-off Ship Expected: fewer surprise fixes and the before is already there
What shipped because of the measuring: a length fix that holds on a second AI evaluator · a gate that says no to its own rewrites · a model pick reversed before the cutover · a build that was never built · a fallback that announces itself
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
27

Adult learning and development

How adults learn to read evaluation results

Most people who sign off on an AI feature were never taught how to read the evidence behind it.

01 · their own problems

Teach on your own items

Adults learn from problems in their own work, and want to know why before how.

02 · worked examples, then fading

Show three, then let them score one

Study an expert's worked example, then take over more of the steps while explaining each one.

03 · a briefing is not training

Practice with feedback

Untrained readers judged AI-written stories at chance. Instructions and examples raised that to 55% at best.

04 · the expertise reversal effect

One page, in layers

Help a newcomer needs gets in an expert's way. Verdict on top, evidence a click below.

Knowles (1980); Atkinson, Derry, Renkl & Wortham (2000); Atkinson, Renkl & Merrill (2003); Clark et al. (2021); Kalyuga, Ayres, Chandler & Sweller (2003).
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
28

For the engineers

How to present qualitative results so others can use them

Four habits, each grounded in research on how people read numbers and evidence.

01 · natural frequencies

Counts, not percentages

“3 of 50 failed” beats “94%”. Untrained readers reason correctly far more often with counts.

02 · a boundary object

The item beside its score

A bare number transfers. The item, its score and the quoted evidence let other roles translate it and push back.

03 · cognitive load

Define the terms first

Evaluator, rubric, passage dependence: one line each, before the first number.

04 · implementation intentions

End with an if-then

“If one evaluator flags it and the other does not, a person reads it before release.”

Gigerenzer & Hoffrage (1995); Carlile (2004); Mayer & Moreno (2003); Gollwitzer & Sheeran (2006); Roediger & Karpicke (2006).
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
29
What BQTH writes when it runs · one review workbook per grade
Read me firstWhat the sheet is, how to work a tab, what every column means
Roll-upOne row per story: the evaluators' verdicts filled in, the reviewer's left blank
One tab per storyFour blocks, always in this order →

1 · The story

Selections, a link to read it in the app, every paragraph with its picture

2 · The quiz

Questions, four choices with the key marked, explanations, hints

3 · Your assessment

One row per rubric item: your score, the evidence (quote it), notes

“Fill this in before expanding the block below.”

4 · The evaluators' assessment

The same rows, a score and the quoted reasoning per evaluator

Collapsed when the tab is created. Open it after your scores are in.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
30
Results people can explore · the published item browser
The item browser: one quiz question with its four choices and word counts, and a table of every evaluator, role and run with a pass or fail per rubric item, filterable by evaluator, role and story.

Overview, filter, details on demand

Roll-up first, one item when asked

Filter by evaluator and role before opening any item.

Outcomes people can count

A row per run, not an error bar

Non-experts read discrete outcomes more accurately than intervals.

Explorable reports

Let the reader change the view

Switching the analysis shows how solid a finding is.

Commit first

Their judgment before ours

The blind block in the sheet: score first, then open the evaluators.

Shneiderman (1996); Hullman, Resnick & Adar (2015); Kay, Kola, Hullman & Munson (2016); Dragicevic, Jansen, Sarma, Kay & Chevalier (2019).
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
31
The same result, written two ways · an illustration, not a real run

What usually reaches the sign-off

Quiz quality: 94% pass.

Approved.

What a reader can work with

47 of 50 quizzes passed, scored by two AI evaluators from different LLM families.

3 failed. Here they are, with the evidence each evaluator quoted.

The evaluators disagreed on 5. A QA engineer read those; 2 need rewriting.

The same 50 on the current model: 41 passed.

Rubric fixed before scoring. Every row is in the sheet.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
32
Exercise 2 · 3 minutes

A. Which model should we ship?

RequestedScoredSummary says
Model A8069Ship this one
Model B8080

B. Are the answer keys sound?

EvaluatorScoredAgrees with 1
135–
2239 of 23
33531 of 35

On the 23 all three evaluated, a majority says valid on 21. AI Evaluator #2 is the only dissenter on 13.

C. Did the sentence fix work?

StoriesSentencesOver level
Before314728%
After1280012%

Same model both rows. Only the instructions changed.

With the person next to you, for each one:
act on it, don't, or ask one specific question first.
If you'd ask, write the question down.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
33

Same prompt. Same grade. Two different stories.

15%

67%

The story mattered more than the thing I was testing.

Before and after on three items measures the three items. Ask: how many, and how different were they?
The decline effect: Schooler (2011), Nature, 470, 437.
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
34
What held
With the storyWithoutChildren, without
The 95 items from the start98.9 to 99.6%89.5 to 97.5%–
50 items under the new rules98 to 100%27 to 55%22%

Chance is 25%. Fifty-seven children, K to 8, on paper.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
35
What changed in BQTH
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
36
My current qualitative testing system
300 storieseach with a twinon the old model The testing teampeople, scoring first, blind Two AI judgestwo LLM families, same rubric One tab per storyyour scores on topthe judges' below, collapsed Where they split, the rubric changes

Every score carries a quoted piece of evidence. Word counts, key position and the length cue are counted, never judged.

Results: still being scored. The people's scores, and how they compare with the AI evaluators, will be posted at masonborchard.com/wwc2026.

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
37

Where people come in

The checks that caught the checkers

A rule, applied every time.

The reviewer AI

Cited two papers that don't exist

Misdescribed two that do. Conceded all four when challenged.

How it was caught

Every citation fetched and checked

A review skill reads the research first, then checks each source before anything leans on it.

My own tests

56 passed, 0 failed

They never ran. They sat below a line that ends the program.

Before the citation rule

Ten bad citations in one review

Four in the turn meant to correct the first six. None reached the page.

Ask any AI review process: what has it retracted?
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
38
Three questions for any AI quality number
1Who did the judging, and who checked the judge?
2What would move this number?
3What did the same inputs produce before the change?

Ask for the methods. Ask for the full dataset. Not the summary.

Before you sign, send for three things: the rows, the funnel, the before.

Training and instructions do not prevent over-reliance on automation, in experts as well as novices (Parasuraman & Manzey, 2010). Fifteen researchers watched using AI tools: the confident tone hid which outputs needed scrutiny, trust was fragile, and the ones who coped checked by hand and built trust by auditing step by step (Gautam, Liu, Choi, & Lease, 2026).
Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
39
Mason Ulrika Borchard in a purple race suit, holding a helmet, beside an orange car numbered 182

Thank you.

The full deck and the resources from the talk:

masonborchard.com/wwc2026

linkedin.com/in/masonborchard

masonulrikaborchard@gmail.com

QR code for linkedin.com/in/masonborchard

Connect on LinkedIn

40
References

Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181–214. https://doi.org/10.3102/00346543070002181

Atkinson, R. K., Renkl, A., & Merrill, M. M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), 774–783. https://doi.org/10.1037/0022-0663.95.4.774

Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X

Carlile, P. R. (2004). Transferring, translating, and transforming: An integrative framework for managing knowledge across boundaries. Organization Science, 15(5), 555–568. https://doi.org/10.1287/orsc.1040.0094

Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021). All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. Proceedings of ACL-IJCNLP 2021, 7282–7296. https://doi.org/10.18653/v1/2021.acl-long.565

Dragicevic, P., Jansen, Y., Sarma, A., Kay, M., & Chevalier, F. (2019). Increasing the transparency of research papers with explorable multiverse analyses. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3290605.3300295

Gautam, S., Liu, H., Choi, Y., & Lease, M. (2026). How researchers navigate accountability, transparency, and trust when using AI tools in early-stage research: A think-aloud study. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’26). ACM. https://doi.org/10.1145/3805689.3812262

Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4), 684–704. https://doi.org/10.1037/0033-295X.102.4.684

Gollwitzer, P. M., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta-analysis of effects and processes. Advances in Experimental Social Psychology, 38, 69–119. https://doi.org/10.1016/S0065-2601(06)38002-1

Haldar, R., & Hockenmaier, J. (2025). Rating roulette: Self-inconsistency in LLM-as-a-judge frameworks. Findings of the Association for Computational Linguistics: EMNLP 2025, 24986–25004. arXiv:2510.27106

Hullman, J., Resnick, P., & Adar, E. (2015). Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PLoS ONE, 10(11), e0142444. https://doi.org/10.1371/journal.pone.0142444

Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31. https://doi.org/10.1207/S15326985EP3801_4

Kay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016). When (ish) is my bus? User-centered visualizations of uncertainty in everyday, mobile predictive systems. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 5092–5103. https://doi.org/10.1145/2858036.2858558

Keenan, J. M., & Betjemann, R. S. (2006). Comprehending the Gray Oral Reading Test without reading it: Why comprehension tests should not include passage-independent items. Scientific Studies of Reading, 10(4), 363–380. https://doi.org/10.1207/s1532799xssr1004_2

Knowles, M. S. (1980). The modern practice of adult education: From pedagogy to andragogy (Rev. ed.). Cambridge.

Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52. https://doi.org/10.1207/S15326985EP3801_6

Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37. arXiv:2404.13076

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x

Schooler, J. W. (2011). Unpublished results hide the decline effect. Nature, 470(7335), 437. https://doi.org/10.1038/470437a

Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. Proceedings of the 1996 IEEE Symposium on Visual Languages, 336–343. https://doi.org/10.1109/VL.1996.545307

Strathern, M. (1997). “Improving ratings”: Audit in the British university system. European Review, 5(3), 305–321.

Säuberli, A., & Clematide, S. (2024). Automatic generation and evaluation of reading comprehension test items with large language models. Proceedings of the 3rd Workshop on Tools and Resources for People with Reading Difficulties (READI), LREC-COLING 2024. ACL Anthology 2024.readi-1.3

Tuinman, J. J. (1973–1974). Determining the passage dependency of comprehension questions in 5 major tests. Reading Research Quarterly, 9(2), 206–223. https://doi.org/10.2307/747135

Verga, P., Hofstätter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., & Lewis, P. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv:2404.18796

Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. arXiv:2306.05685

Score It Yourself · Mason Ulrika Borchard · masonborchard.com/wwc2026
41

00:00

1 / 31
← → move  ·  T light