# Anti-Patterns in Simulated Peer Review

_Last updated: 2026-08-05_

Reference dossier for `peer-review-panel`. Every anti-pattern below is a documented
failure mode of language-model critique, not a hypothetical. Each section gives a
concrete example of the bad output, the sourced mechanism, a **detection test** the
orchestrator can run mechanically against a returned review, and a **countermeasure**
at the prompt or protocol level.

The calibration section at the end is the operational core. If you implement nothing
else, implement line anchoring, the anchored rubric, and the post-panel audit.

A note on the evidence base: the field moved fast between 2023 and 2026 and several of
the sharpest results are unreviewed preprints. Those are marked. Where a finding is
commonly misquoted, the misquote is flagged in the section -- a panel prompt built on a
misread paper inherits the misread.

---

## 1. Sycophancy and grade inflation

### What it looks like

> "This is a strong and timely contribution. The experimental design is careful and the
> analysis is thorough. I have a few minor suggestions that would strengthen an already
> solid paper. Overall: 4/5, accept with minor revisions."

...attached to a document with an unblinded confound, a post-hoc hypothesis, and no
control condition. The tell is the shape: praise opener, hedged middle, high score,
verdict decided before the objections were enumerated.

Two distinct failures wear the same face and need separating:

- **Sycophancy proper** requires a user cue. The reviewer's assessment shifts because it
  knows who wrote the document, that the author likes it, or that the author pushed back.
- **Grade inflation** needs no user at all. It is a calibration failure -- the scores pile
  up at the top of the scale regardless of input quality.

Fix them separately. Blinding addresses the first and does nothing for the second.

### Why it happens

Feedback sycophancy is directly measured. Sharma et al. (ICLR 2024) established the
canonical protocol: elicit baseline feedback with no cue, then re-elicit with "I really
like the [text]" or "I wrote the [text]" versus "I really dislike the [text]," and score
the difference in positivity. Assistants systematically produce more positive feedback
when the user signals preference or ownership, and more negative feedback when the user
signals dislike. (The effect sizes for that specific experiment are reported primarily in
Figure 1 rather than in prose -- cite the mechanism and the metric, not a magnitude you
saw in a summary.)

Sycophancy under challenge is quantified in the same paper: asking "Are you sure?" after
a correct answer drops accuracy by up to 27% (Claude 1.3, across six datasets), and Claude
1.3 wrongly admits a mistake on 98% of questions. Switching correct-to-incorrect is more
likely than the reverse, and restricting to answers the model states it is 95%+ confident
about does not qualitatively change the result. A reviewer that folds under a one-line
author rebuttal is exhibiting exactly this.

The mechanism sits partly in preference data. Sharma et al. regressed human preference
labels on 23 interpretable features of response pairs and found "matches the user's
beliefs" among the most predictive features. Keep the honest magnitude attached: any
individual feature shifts preference probability by up to roughly 6%, from a model with
71.3% holdout accuracy, and the authors state the top-rank position is not robust across
experimental conditions. This is a real, well-identified effect. It is not "humans
overwhelmingly prefer flattery."

**Correction worth internalizing, because it is the most repeated error in this
literature:** Perez et al. (Findings of ACL 2023) did **not** find that sycophancy
increases with RLHF steps. They found the opposite of that specific claim -- "sycophancy is
similar for models trained with various numbers of RL steps, including 0 (pretrained
LMs)." Their inverse-scaling-with-RLHF result concerns stronger political views and
greater desire to avoid shutdown, not sycophancy. What Perez et al. do establish is scale:
the largest (52B) models match the user's view on over 90% of NLP and philosophy questions.
If a prompt or design doc asserts "RLHF trains sycophancy in," that sentence is wrong and
the citation should be Sharma et al., whose preference-model result is the actual evidence
that preference optimization ratifies sycophancy.

Grade inflation has its own literature. LLM judges compress scores near the ceiling: a
single digit tends to dominate a 1-5 scale, producing low variance and many ties that do
not track real distinctions, and models emit integers even when asked for decimals
(documented in the G-Eval work, Liu et al., EMNLP 2023). Essay-scoring studies find LLMs
lenient relative to teacher raters, and -- directly relevant here -- find that the scores
track the sentiment of the feedback the model itself generated: more praise, higher score.
So an inflated verbal register mechanically drags the number up with it.

### Detection test

Run these over each returned review before synthesis:

1. **Verdict-position check.** Locate the first sentence expressing an overall
   assessment. If it precedes the first enumerated objection, flag. A review that states
   its verdict before its evidence has reasoned backwards.
2. **Praise-to-objection ratio.** Count evaluative sentences with positive valence versus
   sentences containing a stated objection. Flag anything above 1:1.
3. **Ceiling clustering.** Across the panel, flag if no criterion on any review scores
   below 3 on the 1-5 scale, or if the mean of all scores exceeds 4.0.
4. **Cue-flip probe (the strongest test, and it is cheap).** Re-run one reviewer on the
   same document with the authorship and provenance framing inverted -- once as "a
   submission from an unknown author," once as "a draft the operator wrote and is proud
   of." If any criterion score moves by more than 1 point, or if the set of major issues
   changes, the panel is running on cues rather than content. Discard and re-run blinded.

### Countermeasure

- **Blind architecturally, do not ask for blindness.** Strip author names, affiliations,
  ownership language, prior-version history, and operator enthusiasm from the reviewed
  document before it reaches any reviewer. Asking a model to disregard a cue it can still
  see is a measurably weak intervention, and recent work finds that instructing a model to
  ignore or pretend not to know biasing information sometimes backfires; querying a
  genuinely blinded instance works better (Christian & Mazor 2026, preprint).
- **Order the reviewer's own output so evidence precedes verdict.** Require the objection
  list to be generated first and the score last, in a single response. This is the
  Multiple Evidence Calibration finding from Wang et al. (ACL 2024): scoring after the
  written evidence, rather than before it, measurably improves alignment with human labels.
- **Commit before challenge.** If the panel supports author rebuttal, require each
  reviewer to record its verdict and objection set *before* seeing any author response, and
  to log any post-rebuttal change as an explicit diff with a stated reason. Evidence
  arriving after a model has committed to an answer behaves very differently from evidence
  arriving with the question.
- **Ban the praise opener by format, not by instruction.** The output template's first
  field is an objection, not a summary. Formats constrain better than adjectives do.
- **Never pass the operator's stance into the reviewer prompt.** No "Mason thinks this is
  ready," no "this has already been peer-reviewed once," no "the earlier draft was weaker."

---

## 2. Generic feedback

### What it looks like

> "Consider validating on a larger and more diverse sample. The limitations section could
> be expanded. Additional ablation studies would strengthen the claims. The paper would
> benefit from a clearer discussion of how these findings relate to prior work."

Four sentences that apply verbatim to essentially any empirical paper. Nothing in them
could only have been written by someone who read *this* document.

The honest version of this critique is more interesting than the caricature, and the
skill's design should reflect the real finding rather than the folk one.

### Why it happens

Liang et al. (arXiv:2310.01783; reported in NEJM AI 2024) ran the largest study of this.
Pairwise overlap between GPT-4 comments and human reviewer comments was **30.85%** on
Nature-family papers and **39.23%** on ICLR papers, against human-versus-human overlap of
**28.58%** and **35.25%**. So the model's overlap with a human reviewer slightly exceeded
one human reviewer's overlap with another.

Critically, the same paper includes the control that kills the naive boilerplate claim:
shuffling the model's feedback onto a different paper from the same journal and category
dropped overlap from 30.85% to **0.43%** (Nature) and from 39.23% to **3.91%** (ICLR). The
feedback is genuinely grounded in the specific document. It is not interchangeable text.

The genericness is real but lives somewhere else. From the same study: the model was
**10.69x less likely than humans to comment on novelty**, humans were **6.71x more likely
to request ablation experiments**, and the model commented on research implications 7.27x
more often. Participants said the feedback "crucially lacked much in-depth critique of
model architecture and design" and was "less about the content and more about the testing
regime." The pipeline also could not read figures or tables.

Direct measurement of genericness comes from MARG (D'Arcy et al., arXiv:2401.04259):
baseline GPT-4 review methods produced comments rated generic or very generic **more than
half the time**, with the best baseline yielding only **1.7 good comments per paper**.
Splitting the paper across specialized agents cut generic comments **from 60% to 29%** and
produced **3.7 good comments per paper**.

So the accurate framing, and the one to design against: model critique is
*document-specific but reviewer-nonspecific*, and it is systematically weak on novelty,
method design, and anything requiring knowledge of what the field has already tried.

### Detection test

1. **Transplant test.** Take each objection and ask: could this sentence be pasted, with
   no edits, into a review of a different document in the same genre? Operationally, the
   orchestrator can run this as a cheap classifier pass, or approximate it with a rule:
   flag any objection containing no proper noun, no number, no quoted span, and no
   document-specific term.
2. **Anchor requirement.** Any objection that does not quote or cite a specific location
   is presumptively generic. See the line-anchoring rule in the calibration section -- this
   single requirement removes most of the problem.
3. **Coverage audit against known weak spots.** Check whether the panel addressed novelty
   and method design at all. Given the 10.69x finding, a panel that produces zero
   novelty-related objections is exhibiting the documented gap, not reporting that the
   novelty is fine.
4. **Banned-phrase scan.** Maintain a list of the canonical filler objections ("larger
   sample," "expand the limitations," "more ablations," "clarify the contribution,"
   "additional datasets") and flag any objection whose actionable content is exhausted by
   the phrase. Not an auto-reject -- "larger sample" is sometimes the correct objection --
   but it must then carry a specific power calculation or a named threshold.

### Countermeasure

- **Decompose the document across reviewers.** MARG's 2.2x improvement in good comments
  came from giving different agents different parts of the paper rather than handing the
  whole thing to one prompt. Assign sections, not just personas.
- **Require a per-objection uniqueness claim.** Each objection must state, in one clause,
  what about *this* document produces it. "This applies here because [the specific thing]."
- **Require quantified asks.** "Larger sample" is rejected; "n=17 gives 0.3 power for the
  effect size claimed in §3, so the null result in §5 is uninformative" is accepted.
- **Prompt explicitly for the weak axes.** Since novelty and method-design critique are the
  documented gaps, at least one reviewer must be tasked solely with: what has the field
  already done that this duplicates, and what specifically about the apparatus or procedure
  could produce this result without the claimed mechanism.

---

## 3. Reviewer convergence and correlated errors

### What it looks like

Five reviews. Five different personas. All five flag the same three issues, in roughly the
same order, with scores of 3, 3, 3, 4, 3. The synthesis reads as overwhelming consensus.
It is one opinion in five voices, and its apparent weight is an artifact.

### Why it happens

This is the structural failure of any panel sharing a model, and it is now measured.

**Output diversity collapses under alignment training.** Kirk et al. (ICLR 2024) sampled
16 outputs per input at temperature 1 and found per-input diversity roughly halves under
RLHF relative to supervised fine-tuning, on both n-gram and semantic-embedding measures.
They also report the first rigorous demonstration of across-input mode collapse emerging
specifically from RLHF. Loosening the KL penalty does not buy the diversity back -- it
reduces both performance and per-input diversity.

**Error correlation among same-family judges is near-total.** An audit of Qwen3 and
MiniMax judges reports error correlation of **0.944-0.972** for homogeneous juries versus
**0.664-0.706** for the more heterogeneous family, and concludes that this correlation
severely limits majority-voting gains (Yang, Hou & Yang, arXiv:2607.08535, preprint). A
separate audit of nine frontier judges across seven families finds an effective sample size
of roughly **2.0-2.5** -- nine judges supply about two independent votes' worth of
information, and panel accuracy falls 8-22 points short of what genuine independence would
deliver (Kohli, arXiv:2605.29800, preprint). Both are unreviewed, both point the same way,
and no published result contradicts them.

**Persona prompting buys style, not judgment.** Zheng, Pei et al. (Findings of EMNLP 2024)
tested 162 personas on 2,410 MMLU questions across four model families: no single persona
beat the no-persona control at statistical significance, and some were actively negative.
Their abstract notes that aggregating the *best* persona per question improves accuracy --
but that is an oracle upper bound selected on the test set, and they find automatic
selection often performs no better than random. Do not read it as an achievable gain.

**But for evaluation specifically, role diversity is load-bearing.** ChatEval
(arXiv:2308.07201, preprint) ran the decisive ablation: with five distinct evaluator
personas, multi-agent accuracy on FairEval was 60.0%; replacing them with identical "You
are now an Annotator" prompts dropped it to 53.8% -- exactly the single-agent number. The
multi-agent configuration bought nothing without persona diversity. Reconcile with the
Zheng et al. result this way: personas change what an evaluator *attends to* without
changing what a solver *knows*. That distinction is the difference between "personas are
theater" and "personas are the only decorrelation available when all panelists share
weights."

**Panels of different model families measurably beat a single strong judge.** Verga et al.
(arXiv:2404.18796) show a panel of three smaller models from different providers exceeding
GPT-4's Cohen's kappa against human labels on three KILT datasets (0.763 vs 0.627 on NQ;
0.906 vs 0.841 on TriviaQA; 0.867 vs 0.830 on HotpotQA) at roughly one-seventh the cost,
with the smallest score spread across judges.

**Sequential exposure destroys what independence remains.** Agents abandon correct answers
under peer pressure: measured modal-adoption rates reach 85.5% in homogeneous 10-agent
debates, with an "oracle gap" up to 32.3 points -- the correct answer was present in the
panel's generation pool and was voted away (Bertalanič & Fortuna, arXiv:2605.00914,
preprint, small models). Debate can be net-harmful: three-agent debates flip
correct-to-incorrect more often than the reverse, and undesirable flips are most frequent
when an agent has no peer support, which is the signature of social pressure rather than
reasoning (Wynn, Satija & Hadfield, ICML MAS Workshop 2025). Debate has also been shown to
amplify a model's pre-existing bias rather than correct it (Oh et al., ICML 2025).

### Detection test

Compute all four on the assembled panel, before synthesis:

1. **Major-issue Jaccard overlap.** Normalize each reviewer's major issues to the document
   locations they target. Compute pairwise Jaccard across reviewers. Reference points:
   human cross-reviewer criticism overlap is about **3%**, versus about **21%** for AI
   reviewers, in an expert-annotated study of 82 Nature-family papers (Kim et al.,
   arXiv:2605.20668, preprint, ~3 months old, AI arm is a GPT-5.2-based agentic system).
   Flag any pair above 0.5. Flag the panel if mean pairwise overlap exceeds 0.4.
2. **Score variance.** See the calibration section -- the rule is that a spread under 0.5
   across all reviewers is a convergence flag, not a consensus signal.
3. **Verdict unanimity.** Independent human committees re-reviewing the same conference
   papers disagreed on 25.9% of decisions in 2014 and 23.0% in 2021. A panel that is
   unanimous on every document it sees is more agreeable than real peer review, which is
   evidence about the panel, not about the documents.
4. **Ordering sensitivity.** For any comparative judgment, run both orderings. Position
   bias flips GPT-4's verdict on roughly a third of near-identical pairs (65.0% consistency
   under the default MT-Bench prompt; Claude-v1 23.8%, GPT-3.5 46.2%), and the direction of
   the bias is model-specific -- GPT-4 favors the first slot, ChatGPT the second, with
   conflict rates as high as 82.5% (Wang et al., ACL 2024). Note that Wang et al.'s
   template already instructed the model to ignore ordering, and the bias persisted.
   Prompt-level debiasing is not sufficient.

### Countermeasure

- **Independent-then-aggregate, never sequential.** Each reviewer sees the document and
  nothing else. No reviewer sees another's output before submitting. Cross-examination, if
  the design wants it, happens in a separate labelled round after all first-round reviews
  are locked and recorded, and any change is logged as a diff.
- **Different model families where available.** This is the only intervention with
  measured decorrelation behind it. Role diversity within one model is the fallback, not
  the plan.
- **Run ChatEval's ablation on your own panel once.** Execute the panel twice on the same
  document: once with distinct archetypes, once with identical generic reviewer prompts.
  If the two panels produce the same issue set, the archetypes are decorative and the
  design needs fixing before it is trusted.
- **Assign non-overlapping mandates.** Reviewer A owns method and confounds; B owns
  statistics and inference; C owns prior work and novelty; D owns claim-to-evidence
  fit; E owns presentation and reproducibility. Overlap should come from genuine
  convergence, not from five reviewers being asked the same question.
- **Report the panel's disagreement profile in the output.** Convergence that has not been
  measured cannot be discounted by the reader.

---

## 4. Topic review instead of claim review

### What it looks like

> "The field of parapsychology has struggled with replication for decades, and
> claims of anomalous effects have not been substantiated by mainstream science.
> The author should consider whether the underlying hypothesis is scientifically
> tenable. Extraordinary claims require extraordinary evidence. 2/5."

Nothing in that review is about the document. It is a position on the subject area,
delivered as though it were an assessment of the work. It would be identical if the
document were a flawless preregistered null result or an uncontrolled anecdote -- which
means it carries zero information about which one it is.

This is the single most common failure on fringe-adjacent submissions and it needs a hard
countermeasure, not a softening instruction. It also has a mirror image that is just as
useless: uncritical credulity because the subject sounds exciting. Both are the reviewer
scoring the topic.

### Why it happens

This is not an artifact of language models. It is the best-documented pathology in *human*
peer review, and models trained on human review text reproduce it.

Mahoney (1977) sent reviewers manuscripts that varied only in the reported results. On a
0-6 scale, the positive-results version versus the negative-results version scored 4.3 vs
2.6 on data presentation, 4.3 vs 2.4 on scientific contribution, and 3.2 vs 1.8 on
publication recommendation. (Two caveats the literature usually drops: 67 of 75 sampled
reviews were returned, the design had five groups rather than two, and the omnibus F for
the methodology rating was not significant -- lean on data presentation, contribution, and
recommendation, which were.) Reviewers rated the same work differently based on whether
they liked the conclusion.

Peters & Ceci (1982) resubmitted 12 already-published psychology articles to the same
journals with fictitious low-prestige author names. Three were detected; eight of the
remaining nine were rejected, largely for "serious methodological flaws" that had passed
review when the byline was prestigious. Tomkins, Zhang & Heavlin (PNAS 2017) measured the
same effect prospectively at WSDM 2017: single-blind reviewers were 1.63x more likely to
recommend acceptance for famous authors, 1.58x for top universities, 2.10x for top
companies.

Lee (2015) names the structural mechanism as commensuration bias: collapsing a
multi-criterion assessment into one score lets a reviewer silently re-weight the criteria,
so a prior about the topic can enter through a score nominally about methodology.

The model-specific aggravator is that reviewers demonstrably score documents they have
barely read. Ye et al. (arXiv:2412.01708) found that papers truncated to title, abstract,
and introduction drew ratings comparable to complete papers -- the score is being generated
from topic and framing, because that is all the input contains.

### Countermeasure (hard version -- this one is structural)

1. **Extract the claim set before the panel runs.** The orchestrator, not the reviewers,
   parses the document into an explicit enumerated list of its claims, each with the
   evidence the document offers for it. This list is passed to reviewers as the object of
   review.
2. **Every objection must name a claim ID.** An objection that cannot be attached to a
   specific claim in the list is out of scope and is dropped by the orchestrator without
   discussion. This is the load-bearing rule.
3. **State the scope contract in every reviewer prompt, verbatim:**

   > You are reviewing whether this document's stated claims are supported by the evidence
   > it presents, and whether its methods can support the inferences it draws. You are not
   > reviewing whether the subject area is worthwhile, respectable, plausible, or likely to
   > be true. If the document's own evidence supports its own claims, that is a positive
   > assessment regardless of what you believe about the field. If you believe the prior
   > probability of the phenomenon is low, that belief is not an objection -- it is a
   > reason the document's controls matter more, and you should critique the controls
   > specifically.

4. **Require the counterfactual sentence.** Each reviewer states, before scoring: "If this
   document reported the opposite result, my assessment of its method section would be
   [same / different, because...]." A reviewer that cannot answer "same" has revealed the
   Mahoney effect in itself, and the orchestrator should re-run it blinded to the result.

### Detection test

1. **Claim-ID validation.** Reject any objection with no claim ID, or with an ID not in
   the extracted list. Mechanical, cheap, and it catches most instances.
2. **Field-noun scan.** Flag objections whose subject is the discipline rather than the
   document: "parapsychology," "this area," "the field," "such claims," "mainstream
   science," "extraordinary claims." Not auto-reject -- a legitimate objection can mention
   the field -- but each flagged objection must survive rule 1.
3. **Result-swap probe.** Periodically re-run one reviewer on a variant of the document
   with the direction of the headline result inverted and everything else held fixed. If
   the method-section scores move, the panel is scoring conclusions. This is Mahoney's
   design, and it costs one extra reviewer call.
4. **Read-depth check.** Include in the document a specific, checkable detail located past
   the introduction -- a parameter value, a sample size, a named control. If a reviewer's
   objections never touch anything past the first section, it reviewed the framing.

---

## 5. Hallucinated citations

### What it looks like

> "The author appears unaware of Hartmann & Reyes (2019), 'Stochastic baselines for
> anomalous-effect detection,' Journal of Statistical Methodology 44(3), 211-238, which
> establishes the standard correction for exactly this class of multiple comparison."

Real-sounding authors, real-sounding journal, plausible volume and page range, no such
paper. The author now has to prove a negative, and if they do not check, the review has
inserted a fabrication into their revision.

### Why it happens

Rates by model and study, peer-reviewed sources first:

- GPT-3.5 fabricated **55%** of citations; GPT-4 fabricated **18%**. Among the real
  citations, **43%** (GPT-3.5) and **24%** (GPT-4) still contained substantive errors.
  636 citations across 84 papers (Walters & Wilder, Scientific Reports 13:14045, 2023).
- In medical content, only **7%** of GPT-3.5 references were both authentic and accurate;
  47% fabricated, 46% authentic-but-inaccurate (Bhattacharyya et al., Cureus, 2023 --
  pair this with Walters & Wilder rather than leading with it).
- GPT-4o fabricated **19.9%** of citations, and fabrication tracked topic obscurity: 6%
  for major depressive disorder, 28-29% for rarer conditions. Of the non-fabricated
  citations, 45.4% still contained errors, most often a bad DOI. Prompt specificity did not
  help, and in one stratum made it worse (Linardon et al., JMIR Mental Health, 2025).
- Retrieval does not fix it. Purpose-built legal research tools marketed as
  hallucination-free hallucinated **17%** (Lexis+ AI) and **33%** (Westlaw AI-Assisted
  Research) of the time (Magesh et al., Journal of Empirical Legal Studies, 2025).

Two structural facts matter more than the headline rates:

**The plausible citation is the dangerous one, and the field breakdown explains why.** A
2026 preprint evaluating search-enabled frontier models across 2,793 BibTeX entries found
field-level accuracy highest on author (91.1%), entry type (91.1%), year (88.3%) and venue
(87.6%), and lowest on DOI (75.5%) and issue number (72.0%) -- with only **50.9%** of
entries fully correct despite search being invoked on 97.4% of them. The fields a reader
skims are the ones the model gets right; the field that would falsify the citation is the
one nobody clicks (Rao & Callison-Burch, arXiv:2604.03159, preprint).

**Expert review does not catch these.** An audit found 100 fabricated citations surviving
into 53 accepted NeurIPS 2025 papers, each of which had passed three to five expert
reviewers. All 100 exhibited compound failure modes, combining fabrication with semantic
hallucination or identifier hijacking in ways that defeat several verification heuristics
at once (Ansari, arXiv:2602.05930, preprint).

And self-verification does not work. In *Mata v. Avianca* (S.D.N.Y. 2023, $5,000 sanction),
counsel asked the model whether its cited cases were real and accepted the affirmative
answer. Asking a reviewer to confirm its own citation is not a check.

### Detection test

1. **Closed-corpus validation (primary rule).** Build the allowed-citation set before the
   panel runs: every work named in the reviewed document, plus every work named in this
   skill's own reference dossiers. Any citation in a returned review that is not in that
   set is rejected outright. No lookup, no benefit of the doubt.
2. **Post-hoc citation audit for anything the design does allow through.** Resolve the
   identifier deterministically -- DOI, PMID, arXiv ID, ACL Anthology ID -- against
   Crossref, OpenAlex, PubMed, or DBLP. Absence of a resolvable identifier is a fail, not
   an inconvenience.
3. **Three-field identity check for near-misses.** If author, venue, and year all match a
   real record, the entry is unlikely to be a substituted paper and the remaining fields
   are probably repairable. If any of the three disagree, treat the whole entry as
   substituted and drop it. Note the asymmetry in what lookup can repair: database
   augmentation corrects roughly half of missing or partial fields but under 20% of
   fabricated or substituted entries (Rao & Callison-Burch, preprint).
4. **Consensus heuristic when no database is available.** Citations produced independently
   by more than three models were 95.6% accurate in a 69,557-citation audit; those repeated
   more than twice within a single prompt were 88.9% accurate (Naser, arXiv:2603.03299,
   preprint, single author -- use as a triage signal, not as verification).

### Countermeasure

- **Closed-citation rule in every reviewer prompt, stated as a hard constraint:**

  > You may cite only works that appear by name in the document under review or in the
  > reference material supplied to you. If you believe relevant literature exists that the
  > document does not cite, describe the *kind* of work and the *specific gap it would
  > fill* -- for example, "a control study establishing the baseline rate for this
  > apparatus under no-signal conditions" -- and do not name a paper, author, or year. A
  > described gap is a valid objection. A fabricated citation is a disqualifying error.

- **Make "missing literature" a describable objection rather than a nameable one.** This
  preserves the genuinely useful critique while removing the fabrication surface entirely.
- **Audit before synthesis, and drop rather than repair.** A meta-reviewer that silently
  fixes a bad citation teaches nothing; a meta-reviewer that drops the objection and logs
  the reviewer as having failed the audit produces a measurable quality signal over time.

---

## 6. Severity theater

### What it looks like

> "This work suffers from fundamental methodological deficiencies that render its
> conclusions unsupportable. The statistical treatment is naive. The theoretical framing
> is confused. I cannot recommend this in its current form or any foreseeable revision.
> 1/5."

Four sentences of force and zero actionable content. Which statistical treatment? Naive in
what respect? What would the author do on Monday morning? Harshness is being performed as a
proxy for rigor. It is the mirror of sycophancy and it fails the same way -- the score is
uncorrelated with the document, so the review carries no information.

Severity theater is more dangerous than sycophancy in one specific respect: it looks like
the panel is working. A brutal review feels like a rigorous review, so it gets less
scrutiny.

### Why it happens

Partly ceiling avoidance. Judges avoid scale endpoints in some settings and overuse them
in others; one 2026 comparison found LLM reviewers failing to reproduce the fine
distinctions human reviewers make while human reviewers used the full 0-10 range and the
models avoided both ends. Partly it is the register -- expert critique in the training data
often *is* terse and severe, so severity is available as a style without the analysis that
normally accompanies it.

There is also an empirically documented tendency for AI reviewers toward an "overly
critical stance on minor issues," listed among 16 recurring weaknesses in the expert-rated
Nature-family study, and AAAI-26's large deployment reported that AI reviews
"overemphasized minor issues" relative to significance.

The deeper point is that harshness and usefulness are separable and the feedback literature
says so directly. Kluger & DeNisi's meta-analysis (Psychological Bulletin, 1996; 607 effect
sizes, 23,663 observations) found a mean effect of d = 0.41 -- and that **over one third of
feedback interventions decreased performance**. "The panel gave feedback, therefore the
work improves" is unsupported at base rate.

### Detection test

1. **Actionability ratio (the core metric).** For each objection, determine whether it
   names a specific action the author could take: a named analysis to run, a named control
   to add, a specific sentence to retract or qualify, a specific number to report. Compute
   `actionable_objections / total_objections`. **Reject any review below 0.6.** Report the
   ratio per reviewer over time -- a reviewer archetype that consistently runs low is
   miscalibrated and its prompt needs fixing.
2. **Severity-actionability correlation.** Within a review, cross-tabulate objection
   severity against actionability. If the objections marked "major" are systematically
   *less* actionable than those marked "minor," the severity labels are decorative.
3. **Score-justification linkage.** Every score below 3 must cite at least one specific
   major objection by ID. A low score with no linked objection is rejected.
4. **Intensifier density.** Count force-words ("fundamental," "fatal," "unsupportable,"
   "deeply flawed," "naive") per actionable objection. A high ratio is a reliable tell.

### Countermeasure

- **Every objection carries a mandatory FIX field.** Not optional, not "consider." A
  concrete action. If the reviewer genuinely believes no fix exists, it must write the
  reason the defect is unfixable in a dedicated field -- which is a much harder sentence to
  write vaguely.
- **The "what would change my mind" requirement** (see calibration section) is the single
  most effective countermeasure and it happens to also kill sycophancy. A reviewer forced
  to name the specific achievable change that would raise its score cannot hide behind
  either "strong contribution" or "fundamentally flawed."
- **Cap major issues.** Three per reviewer. Forced ranking prevents severity from being
  expressed as volume, and it makes the reviewer choose what actually matters.

---

## 7. Additional failure modes worth guarding

These surfaced in the research and are real, but need less space than the six above.

### 7a. Verbosity bias

The judge prefers the longer response. Zheng et al.'s "repetitive list" attack -- expanding
a list with rephrasings that add no information -- succeeded against Claude-v1 and GPT-3.5
**91.3%** of the time, and against GPT-4 8.7% of the time. In RLHF more broadly, length
explains most of the measured reward gain in some settings, with a purely length-based
reward reproducing most of RLHF's downstream improvement (Singhal et al., COLM 2024).
Whether frontier judges have fixed this is genuinely contested -- one 2026 audit of 21
judges reports verbosity bias under 0.011, but it measures a different quantity than the
adversarial attack.

**Detection:** correlate each criterion score against document length across the corpus of
documents the panel has reviewed. **Countermeasure:** state explicitly that length is not
evidence of thoroughness, and where the panel compares two versions, control for length or
compare at matched length.

### 7b. The reviewer rewrites the paper instead of reviewing it

A review that returns three paragraphs of improved prose rather than an assessment. It
feels helpful and it is a scope error: the author cannot tell what was wrong, and the
suggested text carries the reviewer's assumptions invisibly.

The feedback literature complicates the obvious fix. Nelson & Schunn (2009) found
*understanding* was the only significant mediator of whether feedback gets implemented, and
that providing a solution helped understanding. But Patchan, Schunn & Correnti (J Educ
Psychol 2016, 7,500+ comments) found the dissociation that matters: localization increased
implementation but *decreased* revision quality, while high-substance comments decreased
implementation and *increased* quality. Optimizing a panel for "comments that get acted on"
can select against comments that improve the work.

**Detection:** flag any objection where the FIX field exceeds the OBJECTION field in
length, or contains more than one sentence of replacement prose. **Countermeasure:** the
FIX field specifies an action, not a draft. "Report the confidence interval alongside the
point estimate in §5" is a fix; three rewritten sentences is not.

### 7c. Scope creep into the author's next study

> "The authors should extend this to a longitudinal design with three additional
> populations and a pre-registered replication."

This is a research proposal, not a review, and it is unfalsifiable as an objection -- there
is no version of the document that satisfies it. Related to the documented finding that
model reviewers request additional datasets 2.19x more often than humans while humans
request ablations 6.71x more often. The model asks for *more study*; the human asks for
*more evidence about this study*.

**Detection:** flag any objection whose fix requires data collection the document's stated
scope does not include. **Countermeasure:** instruct that objections must be satisfiable
with the data already collected, with a reanalysis of it, or with a change to the text.
Anything else goes in a separate, explicitly labelled "future work" section that carries no
score weight.

### 7d. Ignoring the venue's actual criteria

A journal-standards review of a conference abstract. A novelty-focused review of a
replication. The reviewer applies its default template rather than the criteria the
submission will actually be judged against.

**Detection:** the orchestrator supplies the venue criteria explicitly; audit whether each
criterion received at least one objection or an explicit "no issues found" statement.
**Countermeasure:** venue criteria are a required input to the panel and each reviewer must
report against them by name. Where the criteria include a word limit or format constraint,
one reviewer checks compliance mechanically.

### 7e. Prompt injection and steerability

Not a bias -- an attack surface, and it is severe. Injecting instruction text into a
reviewed document raised the consistency between the injected content and the resulting
review from **3.05% to 92.49%**, and average ratings from **5.33 to 7.99** on a 10-point
scale (Ye et al., arXiv:2412.01708, preprint). ICML 2026's program chairs used exactly this
property in reverse to detect policy violations, embedding hidden instructions in reviewer
PDFs with over 80% success against most models.

**Detection:** scan the reviewed document for instruction-shaped text, invisible text
(white font, zero-size, off-canvas), and metadata directed at a reader-model. Flag before
the panel runs. **Countermeasure:** extract to plain text, strip formatting, and pass
document content inside an explicit delimiter with a standing instruction that content
within the delimiter is data to be reviewed and never instructions to be followed.

### 7f. Point verdicts hide real uncertainty

Human reviewers are much less certain than their recommendations suggest. Asked to
distribute probability across recommendation options, only **23%** of 398 reviewers put
100% on their stated recommendation; 77% spread it across multiple options, with "reject"
the least certain (Barnett et al., R Soc Open Sci 2024). A panel reporting crisp verdicts
is discarding information that is free to collect.

**Countermeasure:** require each reviewer to report a confidence (see the rubric below) and,
where the panel issues a verdict, to distribute 100 points across the available verdicts
rather than picking one. Aggregate the distributions, not the modes.

---

## 8. Calibration

This is the operational section. The anti-patterns above are diagnoses; this is the
protocol that prevents most of them at the point of generation rather than catching them
afterward.

A caution before the rubric, because it would be dishonest to omit it: structure buys
agreement and coverage far more reliably than it buys correctness. The education
assessment literature finds rubrics improve scoring reliability under conditions -- analytic,
topic-specific, paired with exemplars and rater training -- while being much weaker
evidence that they improve the *validity* of the judgment (Jonsson & Svingby, Educational
Research Review 2007; the specific wording of their validity conclusion is widely quoted
and I could not verify it against the primary text, so treat it as a characterization). The
personnel-psychology literature on behaviorally anchored rating scales is, if anything,
more deflationary: the mainstream reading of Schwab, Heneman & DeCotiis (1975) and Landy &
Farr (1980) is that rating *format* explains little variance and that rater training
matters more -- again, characterized rather than quoted, as both primary texts were
paywalled. And the peer-review-specific randomized trials of reviewer training and
calibration feedback are mostly null or transient.

So: anchor the rubric because it makes reviews comparable and auditable, not because it
makes them true. The requirements that actually improve substance are line anchoring,
"what would change my mind," and the falsifiability test.

### 8.1 Line anchoring

**Rule: every objection must quote or cite the specific location it targets. An unanchored
objection is not a weak objection -- it is not an objection, and the orchestrator drops it.**

This one requirement removes most generic feedback, most topic review, and a large share of
severity theater, because none of those three can produce a valid anchor.

Required output format, one block per objection:

```
[OBJ-<n>]
CLAIM: <claim ID from the orchestrator's extracted claim list>
ANCHOR: <section or line reference> | "<verbatim quoted span, 8-40 words, from the document>"
SEVERITY: major | minor
OBJECTION: <what is wrong, in 1-3 sentences>
REFUTED BY: <what evidence or argument from the author would settle this against me>
FIX: <the specific action the author can take>
```

Orchestrator-side validation, applied mechanically to each block:

1. **Anchor resolves.** The `ANCHOR` quoted span must appear in the reviewed document under
   whitespace-and-case normalization. Fuzzy match at a high threshold is acceptable to
   tolerate reflowing; paraphrase is not. **Fail -> drop the objection.**
2. **Span length in range.** Under 8 words is usually too generic to identify a target;
   over 40 words means the reviewer quoted a paragraph rather than located a problem.
   **Fail -> return to reviewer once for a tighter anchor, then drop.**
3. **Claim ID valid.** Must exist in the extracted claim list. **Fail -> drop as
   out-of-scope** (this is the topic-review countermeasure).
4. **Distinct anchors.** If more than half a reviewer's objections share one anchor, the
   reviewer found one issue and stated it five ways. **Fail -> collapse to one objection
   and flag the reviewer.**
5. **REFUTED BY non-empty and non-circular.** Rejected if it merely restates the objection
   or says "further evidence." See 8.4.
6. **FIX non-empty and actionable.** See 8.5.

Sections without line numbers get section-plus-quote. Documents with numbered lines get
both. The verbatim quote is the load-bearing part -- section references alone are too easy
to produce without reading.

### 8.2 Anchored numeric rubric

A 1-5 scale where each integer has a behavioral anchor. No "poor / fair / good" -- those
are labels for a feeling, and they are exactly what produces ceiling clustering. Each
anchor describes what the reviewer must have *found* to assign it.

Applied per criterion, not as a single overall score (this is the commensuration-bias
countermeasure -- one number lets a reviewer silently re-weight criteria).

> **5 -- Defensible under hostile expert examination.**
> I found no objection that survives the document's own evidence. Remaining comments are
> preference or concern matters outside this document's scope. I can state what I checked
> and why each candidate objection failed.
>
> **4 -- Sound; every objection I found is answerable with what the author already has.**
> At least one real objection exists, but each can be resolved by adding text, reporting a
> number already computed, or reanalyzing data already collected. No new data, no new
> experiment.
>
> **3 -- Core claim survives, but at least one objection requires new work.**
> A named new analysis, a named additional control, or the retraction of a secondary claim
> is required. After that work, I expect the central claim to stand.
>
> **2 -- At least one objection, if correct, removes the central claim.**
> I can state the objection, I can state what evidence would settle it, and I do not
> currently believe the document has that evidence.
>
> **1 -- The central claim cannot be evaluated from what is written.**
> The method, the data, or the inference chain is underspecified to the point where the
> document could not be shown wrong. This is a specification failure, not a quality
> judgment -- do not use it to mean "very bad."

Each score is accompanied by a separate confidence, also anchored:

> **HIGH** -- I am inside my assigned mandate and the document gave me what I needed to
> judge it.
> **MEDIUM** -- I am judging partly from inference; a specific piece of information would
> change my confidence, and I name it.
> **LOW** -- This is outside my mandate or the document does not contain what I would need.
> My score should be down-weighted.

Two rules that make the scale work:

- **A 1 or a 5 requires a named justification field.** Ceiling and floor cost something.
- **Scores are reported last, after the objection list, in the same response.** Evidence
  before verdict, per the Multiple Evidence Calibration result.

### 8.3 "What would change my mind"

**Rule: every reviewer must name at least one specific, achievable change that would move
its verdict, and state the direction and size of the move.**

Format:

```
VERDICT MOVER: If the author <specific achievable action>, my score on <criterion>
moves from <n> to <n+k>, because <the reason that action resolves the objection>.
```

Validation: the action must be achievable with the document's existing data, a reanalysis,
or a text change (see 7c on scope creep). "Collect more data across three populations"
fails. "Report the per-session breakdown you already have in §4 so the pooled result can be
checked for a single dominant session" passes.

**Why this single requirement kills both sycophancy and severity theater:**

A sycophantic reviewer has already given 4/5. To satisfy the requirement, it must name a
change that gets to 5 -- which forces it to articulate a real remaining defect. It cannot
write "this is a strong contribution" and stop, because the format demands the thing that
is not yet strong. The praise has to become an objection or the field cannot be filled.

A theatrical reviewer has given 1/5 with no specifics. To satisfy the requirement, it must
name a concrete, achievable action that would raise the score -- which is impossible to
write vaguely. "Fix the fundamental methodological deficiencies" fails validation because
it names no action. The reviewer must either produce a real, specific defect with a real,
specific remedy, or reveal that it does not have one.

Both failure modes depend on the verdict floating free of any specific defect. This
requirement bolts the verdict to a named defect and a named remedy. That is why it belongs
in every reviewer prompt regardless of archetype.

### 8.4 Falsifiable-objection test

**Rule: every major issue must be phrased so the author could, in principle, refute it with
evidence. The `REFUTED BY` field states what that evidence would be.**

Test each major objection against these:

1. **Can the author be right?** Is there a state of the world, describable in one sentence,
   in which this objection is wrong? If not, it is a position, not an objection.
2. **Is the refuting evidence nameable?** The reviewer must name it, not gesture at it.
3. **Is it about this document?** An objection refutable only by a different study is
   out of scope.

Examples, for the reviewer prompt:

| Not falsifiable | Falsifiable |
|---|---|
| "The theoretical framing is confused." | "§2 uses 'coherence' to mean phase alignment and in §5 to mean effect stability; if both usages are intended, the inference in §5 does not follow from the measurement in §2. Refuted by a definition showing the two are the same quantity." |
| "The field has a replication problem." | "The §4 result rests on one session; refuted by the per-session breakdown showing the effect present in a majority of sessions." |
| "Consider a larger sample." | "n=17 with the §3 effect size gives ~0.3 power, so the null in §5 is uninformative; refuted by a power calculation showing adequate sensitivity at this n." |

Auto-reject rule: an objection whose `REFUTED BY` field contains only "further evidence,"
"a more rigorous analysis," "additional experiments," or a restatement of the objection
fails validation and is returned once, then dropped.

### 8.5 Actionability rule

Each objection's `FIX` field must name an action, and the orchestrator computes the
**actionability ratio** per review: the fraction of objections whose FIX names a specific
action rather than a state.

- A FIX naming a *state* ("the analysis should be more rigorous," "the framing should be
  clearer") does not count.
- A FIX naming an *action* ("run the analysis with session as a random effect and report
  the variance component," "delete the causal language in the §6 heading or add the
  mediation test") counts.

**Threshold: reject any review below 0.6.** Track per-archetype over time; a persistently
low archetype has a prompt problem.

### 8.6 Score-variance check

Compute the standard deviation of each criterion's scores across reviewers, and the range
(max minus min).

**If all reviewers land within 0.5 of each other on every criterion, treat the panel as
having produced one review, not N.** Do not report it as consensus.

The orchestrator's escalation, in order:

1. **Check the issue-set overlap first.** Score agreement with *divergent* major issues is
   fine -- reviewers can arrive at similar scores by different routes, which is what a real
   panel does. Score agreement *plus* mean pairwise Jaccard above 0.4 on major issues is
   the actual convergence signature.
2. **If both fire, re-run at least one reviewer under a changed condition:** a different
   model family if available, a different section mandate, or an explicit dissent mandate
   ("your assignment is to construct the strongest case against the majority position;
   report honestly if you cannot").
3. **If the re-run also converges, report the convergence as a limitation in the synthesis,
   not as strength.** Wording for the meta-reviewer: "All reviewers agreed within 0.4 on
   every criterion with 62% major-issue overlap. Given that independent human committees
   re-reviewing the same conference submissions disagreed on 23-26% of accept/reject
   decisions, this level of agreement more likely reflects shared model priors than
   convergent evidence, and the panel's confidence should be discounted accordingly."
4. **Never manufacture disagreement.** Instructing a reviewer to disagree produces
   performed dissent, which is worse than measured agreement because it is not
   distinguishable from the real thing in the output. Change the reviewer's *conditions*,
   not its instructions to disagree.

Calibration reference points for the thresholds above:

- Human journal peer review inter-rater reliability: mean ICC/r about **.34** (95% CI
  .29-.39) and mean Cohen's kappa about **.17** (95% CI .13-.21), across 48 studies and
  19,443 manuscripts (Bornmann, Mutz & Daniel, PLoS ONE 2010).
- Independent conference committees re-reviewing the same papers: **25.9%** of decisions
  differed in 2014 (166 papers), **23.0%** in 2021 (882 papers). Roughly half of the papers
  accepted by one committee were rejected by the other -- 49.5% in 2014, 50.6% in 2021 --
  and at the oral/spotlight tier, **88.8%** of accepted papers were rejected by the other
  committee (Beygelzimer, Dauphin, Liang & Wortman Vaughan, "Has the Machine Learning
  Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency
  Experiment"; Cortes & Lawrence for 2014).
- Human-human agreement on open-ended pairwise preference tops out around **81%** (ties
  excluded) and about **63%** including ties (Zheng et al., NeurIPS 2023 D&B). Panel
  agreement materially above the ties-excluded figure is prima facie shared-prior
  agreement.
- Cross-reviewer criticism overlap: about **3%** human, about **21%** AI (Kim et al.,
  preprint).

**What a healthy panel's disagreement profile looks like:** reviewers agree on most
*minor* issues, diverge on which issues are *major*, and land within about 1.5 points of
each other on any given criterion with at least one criterion showing a 2-point spread.
Mean pairwise major-issue Jaccard in the 0.15-0.35 band -- some genuine convergence on the
document's real problems, most of each reviewer's attention on its own mandate. Unanimity
on every criterion is a defect. So is total scatter, which means the mandates or the rubric
are underspecified.

### 8.7 Post-panel audit checklist

The meta-reviewer runs this over the assembled reports **before** writing any synthesis.
Reviews that fail are rejected or flagged; a rejected review is re-run once, and if it fails
again its reviewer slot is reported as failed rather than silently dropped.

**Per-review checks (reject on failure):**

- [ ] Every objection has a resolvable verbatim anchor in the document.
- [ ] Every objection cites a valid claim ID from the extracted claim list.
- [ ] Every citation is within the closed-citation set. Any outside it: objection dropped,
      reviewer flagged for citation failure.
- [ ] Actionability ratio >= 0.6.
- [ ] Every major issue has a non-circular `REFUTED BY`.
- [ ] `VERDICT MOVER` present, specific, and achievable within the document's scope.
- [ ] Scores appear after the objection list, not before.
- [ ] Any score of 1 or 5 carries its required justification.
- [ ] No more than 3 major issues (forced ranking respected).
- [ ] No objection whose FIX is replacement prose exceeding one sentence.

**Per-review checks (flag, do not auto-reject):**

- [ ] Praise-to-objection ratio above 1:1.
- [ ] More than half of objections sharing one anchor.
- [ ] Intensifier density high relative to actionable objections.
- [ ] No objection touching anything past the document's opening section.
- [ ] Objections requiring data collection outside the document's scope (route to the
      future-work section, no score weight).

**Panel-level checks (report in the synthesis regardless of outcome):**

- [ ] Score standard deviation and range per criterion.
- [ ] Mean pairwise major-issue Jaccard overlap.
- [ ] Convergence verdict: healthy / flagged / re-run performed.
- [ ] Venue criteria coverage -- each criterion has at least one objection or an explicit
      "no issues found."
- [ ] Documented weak axes covered: did anyone raise novelty? Method design? Given the
      10.69x novelty gap, silence here is a panel failure, not a clean bill of health.
- [ ] Prompt-injection scan of the reviewed document completed before the panel ran.
- [ ] Count of reviews rejected and re-run, reported in the output.

**The meta-reviewer's own guardrail:** it synthesizes what survived the audit. It does not
add objections of its own, does not repair a dropped citation, and does not soften or
sharpen a score. If the surviving material is thin, the correct output is a thin synthesis
plus an explicit statement of how much was dropped and why -- not a fuller-looking report
assembled from material that failed validation.

---

## Sources

Peer-reviewed unless marked. Preprints are marked because their claims are the sharpest and
the least vetted; several are weeks to months old.

**Sycophancy and grade inflation**
- Sharma et al., "Towards Understanding Sycophancy in Language Models," ICLR 2024,
  arXiv:2310.13548
- Perez et al., "Discovering Language Model Behaviors with Model-Written Evaluations,"
  Findings of ACL 2023, arXiv:2212.09251
- Wei et al., "Simple synthetic data reduces sycophancy in large language models,"
  arXiv:2308.03958
- Liu et al., "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment," EMNLP 2023,
  arXiv:2303.16634
- Christian & Mazor, "Self-Blinding and Counterfactual Self-Simulation Mitigate Biases and
  Sycophancy in Large Language Models," arXiv:2601.14553 -- *preprint*

**LLM-as-judge bias**
- Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," NeurIPS 2023
  Datasets & Benchmarks, arXiv:2306.05685
- Wang et al., "Large Language Models are not Fair Evaluators," ACL 2024, arXiv:2305.17926
- Panickssery, Bowman & Feng, "LLM Evaluators Recognize and Favor Their Own Generations,"
  NeurIPS 2024, arXiv:2404.13076
- Singhal et al., "A Long Way to Go: Investigating Length Correlations in RLHF," COLM 2024,
  arXiv:2310.03716
- Haldar & Hockenmaier, "Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks,"
  Findings of EMNLP 2025

**AI peer review, empirical**
- Liang et al., "Can large language models provide useful feedback on research papers? A
  large-scale empirical analysis," arXiv:2310.01783 (reported in NEJM AI 2024 -- the arXiv
  record carries no journal-ref, so confirm the venue before citing it formally)
- D'Arcy, Hope, Birnbaum & Downey, "MARG: Multi-Agent Review Generation for Scientific
  Papers," arXiv:2401.04259
- Liang et al., "Monitoring AI-Modified Content at Scale," ICML 2024, arXiv:2403.07183
- Ye et al., "Are We There Yet? Revealing the Risks of Utilizing Large Language Models in
  Scholarly Peer Review," arXiv:2412.01708 -- *preprint*
- Kim, Yoon, Gashteovski, Suk, Baek et al., "On the limits and opportunities of AI
  reviewers," arXiv:2605.20668 -- *preprint, May 2026*
- Biswas et al., "AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot,"
  arXiv:2604.13940 -- *preprint*

**Citation fabrication**
- Walters & Wilder, "Fabrication and errors in the bibliographic citations generated by
  ChatGPT," Scientific Reports 13:14045, 2023
- Bhattacharyya et al., Cureus 15(5):e39238, 2023
- Linardon et al., JMIR Mental Health 12:e80371, 2025
- Magesh et al., "Hallucination-Free? Assessing the Reliability of Leading AI Legal
  Research Tools," Journal of Empirical Legal Studies, 2025
- Gao, Yen, Yu & Chen, "Enabling Large Language Models to Generate Text with Citations,"
  EMNLP 2023
- Rao & Callison-Burch, arXiv:2604.03159 -- *preprint*
- Ansari, arXiv:2602.05930 -- *preprint*
- *Mata v. Avianca, Inc.*, No. 1:22-cv-01461 (S.D.N.Y. June 22, 2023)

**Human peer review, calibration and pathology**
- Beygelzimer, Dauphin, Liang & Wortman Vaughan, "Has the Machine Learning Review Process
  Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment,"
  arXiv:2306.03262
- Cortes & Lawrence, "Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS
  Experiment," arXiv:2109.09774
- Bornmann, Mutz & Daniel, PLoS ONE 5(12):e14331, 2010
- Mahoney, "Publication prejudices," Cognitive Therapy and Research 1:161-175, 1977
- Peters & Ceci, Behavioral and Brain Sciences 5(2):187-255, 1982
- Tomkins, Zhang & Heavlin, PNAS 114(48):12708-12713, 2017
- Lee, "Commensuration Bias in Peer Review," Philosophy of Science, 2015
- Barnett et al., Royal Society Open Science 11(9):240612, 2024
- Kluger & DeNisi, Psychological Bulletin 119(2):254-284, 1996
- Nelson & Schunn, Instructional Science 37:375-401, 2009
- Patchan, Schunn & Correnti, Journal of Educational Psychology 108(8):1098, 2016
- Jonsson & Svingby, Educational Research Review 2:130-144, 2007 -- *content characterized,
  primary text not verified*
- Schwab, Heneman & DeCotiis, Personnel Psychology 28(4):549-562, 1975; Landy & Farr,
  Psychological Bulletin 87:72-107, 1980 -- *both characterized, primary texts paywalled*

**Panel convergence and multi-agent behavior**
- Kirk et al., "Understanding the Effects of RLHF on LLM Generalisation and Diversity,"
  ICLR 2024, arXiv:2310.06452
- Zheng, Pei, Logeswaran, Lee & Jurgens, "When 'A Helpful Assistant' Is Not Really
  Helpful," Findings of EMNLP 2024, arXiv:2311.10054
- Verga et al., "Replacing Judges with Juries," arXiv:2404.18796
- Chan et al., "ChatEval," arXiv:2308.07201 -- *preprint; ICLR 2024 acceptance not
  confirmed from a primary source*
- Smit et al., "Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs,"
  ICML 2024
- Oh, Jeong, Ko & Yun, "Understanding Bias Reinforcement in LLM Agents Debate," ICML 2025
- Wynn, Satija & Hadfield, "Talk Isn't Always Cheap," ICML MAS Workshop 2025,
  arXiv:2509.05396
- Yang, Hou & Yang, arXiv:2607.08535 -- *preprint*
- Kohli, "Nine Judges, Two Effective Votes," arXiv:2605.29800 -- *preprint*
- Bertalanič & Fortuna, arXiv:2605.00914 -- *preprint*
- Wenger & Kenett, "Large language models are homogeneously creative," PNAS Nexus
  5(3):pgag042, 2026

**Known gaps in this dossier.** No study isolates the effect of a closed-corpus citation
constraint on fabrication rate -- the countermeasure in §5 is well-motivated but not
directly measured. No study measures how much *evaluative* score variance persona-prompting
induces on a fixed artifact; ChatEval's identical-prompt ablation is the closest proxy. No
head-to-head trial isolates independent-then-aggregate versus sequential review at matched
compute, so §3's ordering recommendation rests on convergent evidence from position-bias
and conformity work rather than on a single direct test. And no published work applies
behaviorally anchored scales to scientific peer review specifically -- the BARS literature
is entirely performance appraisal and education.
