# Panel Mechanics

_Last updated: 2026-08-05_

Operational reference for running a multi-model simulated peer-review panel on this
machine. Load this at runtime. Every command block below was executed on 2026-08-05
and the observed results are recorded verbatim.

The short version: **independence before verdicts is the lever that works.** Model-family
diversity is a much weaker lever than the folklore claims, and a panel does not average
away correlated error. Design for the first, buy what you can of the second, and never
report panel agreement as if it were confirmation.

---

## 1. Bridges available on this machine

Three non-Claude bridges exist. All three take the **same stdin JSON shape** by design --
`bedrock_brad.py` and `gpt_brad.py` are deliberate mirrors, so request-building code is
shared. The cursor bridge differs (plain text on stdin, chat-id state).

Status as of 2026-08-05: **gpt_brad up, cursor up, bedrock down (expired SSO).**

### 1.1 gpt_brad -- OpenAI Responses API (primary non-Claude peer)

Script: `/home/borchard/.claude/skills/debate-gpt-brad/gpt_brad.py`

**Invocation:**

```bash
source ~/.zshenv && GPT_BRAD_MODEL=gpt-5.4-2026-03-05 \
  python3 ~/.claude/skills/debate-gpt-brad/gpt_brad.py ask < "$SCRATCHPAD/peer_req.json"
```

**Input JSON schema** (read off `ask()` at `gpt_brad.py:110`):

```json
{
  "system": "optional string -- becomes the Responses API `instructions` field",
  "model":  "optional string -- overrides $GPT_BRAD_MODEL and auto-detection",
  "messages": [
    {"role": "user",      "content": "..."},
    {"role": "assistant", "content": "..."}
  ]
}
```

- `messages` is **required and must be non-empty** -- empty exits with `No messages provided.`
- `system` is optional; omitted means no `instructions` field is sent.
- Precedence for model: request `model` field, then `$GPT_BRAD_MODEL`, then newest
  flagship `gpt-*` from `/v1/models`, then the hardcoded `FALLBACK_MODEL = "gpt-5"`.
- The bridge sends the whole `messages` array as the Responses API `input`. It is
  **stateless** -- there is no server-side thread. Every call must carry the full
  transcript you want the peer to see. For a review panel this is a feature: it is
  exactly how you enforce that a peer sees only what you decide it sees.

**Output shape:** plain text on stdout, `\n`-terminated. Not JSON. The bridge extracts
and concatenates every `output_text` block from the response and strips whitespace
(`_extract_text`, `gpt_brad.py:99`). If no text comes back it exits nonzero with the raw
payload dumped. If you want structured output from this peer, ask for JSON **in the
prompt** and parse stdout yourself.

**Key resolution:** `$OPENAI_API_KEY` if set, else regex-scraped from
`~/.config/secrets/api-keys.env`. The `source ~/.zshenv` prefix is belt-and-braces -- the
on-disk fallback means the bridge usually works without it.

**Timeout:** `TIMEOUT = 180` seconds, hardcoded at `gpt_brad.py:27`. Not configurable by
env var. A peer prompt that would take longer than 3 minutes must be split.

**Cheap reachability probe** (no credentials consumed on the models path, ~0.6s):

```bash
source ~/.zshenv && python3 ~/.claude/skills/debate-gpt-brad/gpt_brad.py model
# observed 2026-08-05 -> gpt-5.4-2026-03-05   (0.588s wall)
```

**Smoke test executed 2026-08-05.** Payload:

```json
{"system":"You are a terse reviewer. Answer in one sentence.","messages":[{"role":"user","content":"Reply with exactly one sentence naming the single biggest risk of using one model as sole judge of a paper."}]}
```

Verbatim stdout:

```
The biggest risk is systematic bias—one model can confidently and consistently misjudge the paper without any independent check.
```

**Latency: 2.35s wall clock** for a one-sentence answer. A full referee report on a
2,000-word draft should be budgeted at 30-90s.

Note the em dash in that output. Peer output is raw model text; if any of it reaches a
Mason-facing deliverable it must be normalized to `--` first.

**Degradation path:** exits nonzero with `OPENAI_API_KEY not in env and not found in
~/.config/secrets/api-keys.env.` on a missing key, or `OpenAI API error <code>: <body>`
on an API failure. Both are fatal `sys.exit` calls, so the failure is loud and
catchable on exit status. If gpt_brad is down, fall through to cursor (1.3) pinned to a
non-Anthropic model. **Never synthesize a peer turn yourself to cover a dead bridge** --
a fabricated referee report is worse than a three-person panel.

### 1.2 bedrock_brad -- AWS Bedrock Converse API (currently DOWN)

Script: `/home/borchard/.claude/skills/debate-bedrock-brad/bedrock_brad.py`

**Invocation:**

```bash
python3 ~/.claude/skills/debate-bedrock-brad/bedrock_brad.py ask < "$REQ"
```

**Input JSON schema:** identical to gpt_brad -- `{"system", "messages", "model"}`. Same
required/optional rules. The bridge rewrites each message into Converse's
`{"role":..., "content":[{"text":...}]}` form.

**Model selection:** request `model` field, then `$BEDROCK_BRAD_MODEL`, then default
`gpt-oss-120b`. Short keys map through `BEDROCK_IDS` (`bedrock_brad.py:48`):

| short key | Bedrock id |
|---|---|
| `gpt-oss-120b` (default) | `openai.gpt-oss-120b-1:0` |
| `gpt-oss-20b` | `openai.gpt-oss-20b-1:0` |
| `nova-lite` | `amazon.nova-lite-v1:0` |
| `nova-micro` | `amazon.nova-micro-v1:0` |
| `nova-2-lite` | `us.amazon.nova-2-lite-v1:0` |
| `llama3-3-70b` | `us.meta.llama3-3-70b-instruct-v1:0` |
| `claude-haiku-4-5` | `us.anthropic.claude-haiku-4-5-20251001-v1:0` |

A raw id containing `.` or `:` passes through untouched. An unknown bare key exits
loudly rather than silently routing elsewhere.

**Other env:** `$BEDROCK_BRAD_REGION` (default `us-east-1`),
`$BEDROCK_BRAD_MAX_TOKENS` (default 8000 for gpt-oss, 4000 otherwise -- gpt-oss bills
reasoning against output tokens, so the headroom prevents truncation).

**Output shape:** plain text on stdout. `reasoningContent` blocks are deliberately
dropped, so the caller never sees chain-of-thought.

**Timeout:** none set in the script -- inherits boto3/botocore defaults.

**Reachability probe -- and the trap:**

```bash
python3 ~/.claude/skills/debate-bedrock-brad/bedrock_brad.py model
# observed 2026-08-05 -> openai.gpt-oss-120b-1:0
```

This **does not touch credentials**. A clean print here is not evidence the bridge
works. Only `ask` exercises the credential chain.

**Smoke test executed 2026-08-05 -- FAILED.** Same payload as gpt_brad. Verbatim stderr,
exit status 1, 0.43s:

```
Bedrock call failed -- sandbox SSO creds look expired or unset; re-auth and retry.
```

**Degradation path:** this message is emitted for `NoCredentialsError`, for any
`ClientError` whose code is in `{ExpiredTokenException, ExpiredToken,
UnrecognizedClientException, InvalidSignatureException, AccessDeniedException,
UnauthorizedException}`, or for any error whose text contains `token`/`credential`/
`sso`/`expired`/`security token`. Anything else surfaces as `Bedrock call failed: <err>`.
A missing boto3 exits with `boto3 is not importable. Install it with: pip install --user boto3`.

When Bedrock is down: **seat that chair via cursor instead** (1.3), pin a non-Anthropic
model, and record the substitution in the panel transcript. Do not silently shrink the
panel -- a 3-peer panel reported as if it were 4 misstates the evidence base.

### 1.3 cursor -- second model inside Cursor (working fallback and third family)

Script: `/home/borchard/.claude/skills/debate-cursor-brad/cursor_debate.py`

This bridge is **stateful and differently shaped** -- it takes plain text on stdin, not
JSON, and it maintains a chat id across turns.

**Three subcommands:**

```bash
# 1. list models on the account
python3 ~/.claude/skills/debate-cursor-brad/cursor_debate.py models

# 2. open a throwaway chat -- one per peer, per panel
CURSOR_DEBATE_WORKSPACE=/home/borchard/ionosphere-workspace \
  python3 ~/.claude/skills/debate-cursor-brad/cursor_debate.py start
# -> {"chat_id": "9126ae95-0ec2-473f-a64f-dec61a0757d7", "workspace": "/home/borchard/ionosphere-workspace"}

# 3. send one turn (PLAIN TEXT on stdin, not JSON)
CURSOR_DEBATE_WORKSPACE=/home/borchard/ionosphere-workspace \
  python3 ~/.claude/skills/debate-cursor-brad/cursor_debate.py ask <chat-id> <model> < msg.txt
```

**Input:** raw text read from stdin and passed as the prompt. Empty stdin exits with
`No message on stdin.` There is no `system` field -- **put the persona and the rubric at
the top of every message**, because `--mode ask` gives the session no persistent persona.

**Model precedence:** `ask` argument, then `$CURSOR_DEBATE_MODEL`, then
`DEFAULT_MODEL = "gpt-5.2"`.

**Output shape:** the bridge runs `cursor-agent --output-format json`, parses the last
stdout line, checks `is_error`, and prints the `result` field. So you get **plain text on
stdout**, same as the other two.

**Timeout:** `TIMEOUT = 600` seconds (`cursor_debate.py:41`). The most generous of the three.

**Safety:** runs `--mode ask`, read-only. The peer argues; it cannot edit files or run
commands. Keep it that way -- a reviewer should not be able to modify what it reviews.

**Smoke tests executed 2026-08-05 -- BOTH PASSED.**

`which cursor-agent` -> `/home/borchard/.local/bin/cursor-agent`

`start` -> `{"chat_id": "9126ae95-0ec2-473f-a64f-dec61a0757d7", ...}` in **5.48s**.

`ask ... gpt-5.2`, same one-sentence prompt, verbatim stdout, **12.27s**:

```
The single biggest risk is **systematic bias**—a lone model can consistently misjudge novelty, rigor, or relevance due to its blind spots, leading to confident but wrong acceptance/rejection decisions.
```

Cursor is roughly **5x slower than gpt_brad** per turn plus a one-time ~5.5s chat setup.
Budget accordingly.

**Models available on the account** (from `models`, 2026-08-05, abridged to the useful
panel seats):

- Non-Anthropic: `gpt-5.2`, `gpt-5.5-high`, `gpt-5.6-sol-high`, `gpt-5.6-sol-xhigh`,
  `gpt-5.3-codex` (and low/high/xhigh/fast variants), `cursor-grok-4.5-high`,
  `cursor-grok-4.5-medium`, `cursor-grok-4.5-low`, `kimi-k3-high`, `composer-2.5`
- Anthropic (available, but see 4.2 before seating one): `claude-opus-5-*`,
  `claude-sonnet-5-thinking-*`, `claude-opus-4-8-*`,
  `claude-fable-5-thinking-high` / `-xhigh` -- **both Fable entries are marked `(NO ZDR)`.
  Do not send anything confidential through them.**

**Degradation path:** `cursor-agent not found on PATH.` if the binary is missing;
`cursor-agent did not answer within 600s.` on timeout; nonzero exit with captured
stderr if the CLI fails; `Unparseable cursor-agent output:` if the JSON line is
malformed. All fatal, all catchable on exit status.

### 1.4 A live demonstration of the whole problem

The two working bridges -- different vendors, different families, same one-sentence
prompt -- returned:

- gpt_brad: "The biggest risk is **systematic bias** -- one model can confidently and
  consistently misjudge the paper without any independent check."
- cursor/gpt-5.2: "The single biggest risk is **systematic bias** -- a lone model can
  consistently misjudge novelty, rigor, or relevance due to its blind spots..."

Near-identical, down to the phrase. That is correlated error, observed on this machine,
in the trivial case. It is also two GPT-family models, which is the honest read. Treat it
as a standing warning: **cross-vendor does not mean decorrelated**, and agreement between
two peers is weak evidence of anything. Section 3 is the response to this.

---

## 2. Claude model diversity via the Agent tool

### 2.1 Parameter surface (verified against the live tool schema)

```
Agent({
  description:       string   REQUIRED  -- 3-5 words, shown in the UI
  prompt:            string   REQUIRED  -- the full task for the agent
  subagent_type:     string             -- from the harness's agent list; defaults to general-purpose
  model:             "opus" | "sonnet" | "haiku" | "fable"
  name:              string             -- ^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$
  run_in_background: boolean            -- DEFAULT TRUE
  isolation:         "worktree" | "remote"
  mode:              DEPRECATED, ignored
  team_name:         DEPRECATED, ignored
})
```

Notes that matter operationally:

- **`model` accepts exactly four values on this harness: `opus`, `sonnet`, `haiku`,
  `fable`.** Not full model ids. It takes precedence over the agent definition's own
  `model` frontmatter. Omitted, the agent inherits the definition's model or the parent's.
- **`model` is ignored for `subagent_type: "fork"`** -- forks always inherit the parent
  model. Do not try to seat a diverse panel with forks.
- **`run_in_background` defaults to true.** For a fan-out of independent referees this is
  what you want -- issue all N in one message and collect the notifications. Set
  `run_in_background: false` only when you need one result before you can construct the
  next call.
- **`name` makes an agent addressable via `SendMessage({to: name})`** while it runs. This
  is how you run the rebuttal round without re-sending the whole draft: the peer keeps its
  context.
- **A NAMED AGENT CANNOT BE SPAWNED BY A TEAMMATE.** Verified the hard way on 2026-08-05.
  Passing `name` from inside a subagent returns:

  ```
  Teammates cannot spawn other teammates -- the team roster is flat.
  To spawn a subagent instead, omit the `name` parameter.
  ```

  So: if the panel skill runs in the **top-level** session, use `name` and get
  SendMessage-based rebuttals. If it runs **inside a subagent**, omit `name` -- and
  because unnamed subagents cannot be resumed by name, the rebuttal round must instead
  re-issue fresh agents carrying the phase-2 context in their prompt. Both paths are
  specified in section 5. Detect which case you are in by attempting the named spawn and
  falling back on that exact error string.

- Each named agent returns an `agentId` (format `a<hex>`). A completed background agent
  can be resumed with `SendMessage({to: "<agentId>"})` even after it finishes.

### 2.2 Fan-out syntax for a panel

All peers in **one message**, multiple tool calls. This is the mechanical guarantee of
independence -- see 3.2.

```
Agent({name: "peer-methods",  model: "opus",   subagent_type: "general-purpose",
       description: "Referee report: methods",  prompt: "<full referee prompt A>"})
Agent({name: "peer-novelty",  model: "sonnet", subagent_type: "general-purpose",
       description: "Referee report: novelty",  prompt: "<full referee prompt B>"})
Agent({name: "peer-clarity",  model: "haiku",  subagent_type: "general-purpose",
       description: "Referee report: clarity",  prompt: "<full referee prompt C>"})
```

Plus the non-Claude seat via Bash + gpt_brad, issued in the same message.

### 2.3 What each tier is good for in a reviewer role

These are role-fit judgments for this harness, not benchmark results.

| Tier | Reviewer role it fits | Why |
|---|---|---|
| `opus` | Methods referee, statistics referee, meta-reviewer | Deepest at finding the flaw that requires holding the whole argument at once. Use it where a missed error is expensive. The meta-reviewer especially -- synthesis across conflicting reports is the hardest job on the panel. |
| `sonnet` | Novelty / prior-work referee, general referee | Strong enough for substantive critique at materially lower cost and latency. The right default for the middle seats of a 4+ peer panel. |
| `haiku` | Clarity referee, rubric/format compliance, structured-field extraction | Fast and cheap. Good at "does this section do what it claims," bad at "is this claim true." Never seat haiku alone on correctness. |
| `fable` | Adversarial / devil's-advocate seat, framing critique | Different training lineage from the Opus/Sonnet/Haiku line, which is the closest thing to genuine within-Claude diversity available. **Constraint: the Cursor listing marks Fable `(NO ZDR)`.** That flag is about the Cursor route specifically, but treat Fable as the seat you do not send confidential material to unless Mason has cleared it. |

**Cost and latency:** relative ordering is `opus > sonnet > haiku` on both. I did not
verify current per-token prices in this pass and will not invent them -- **load the
`claude-api` skill for authoritative pricing before quoting any dollar figure.** The
cost table in section 6 is expressed in peer-turns, which is the unit you actually
control.

---

## 3. Independence protocol

### 3.1 Why this is the section that matters

The evidence is lopsided. Deliberation protocols, model diversity, aggregation rules --
all contested. **Independence before verdicts is the one lever with clean, converging,
peer-reviewed support**, and every result points the same way:

- Kaesberg et al., "Voting or Consensus? Decision-Making in Multi-Agent Debate,"
  Findings of ACL 2025 (arXiv:2502.19130). Seven decision protocols, everything else held
  fixed: voting protocols improved reasoning tasks by 13.2%; **more agents improved
  performance, but additional discussion rounds before voting reduced it.** This is the
  most actionable single finding in the literature.
- Jin et al., "AgentReview," EMNLP 2024 (aclanthology 2024.emnlp-main.70). A simulated
  five-phase review pipeline: **rating standard deviation dropped 27.2% after the
  discussion phase.** Deliberation manufactured consensus. That is herding measured in
  silico, in exactly our use case.
- Baumann, Pei, Koyejo & Hovy, "Position: Stop Automating Peer Review Without Rigorous
  Evaluation," ICML 2026 Oral, Position track (arXiv:2605.03202). LLM reviewers show a
  "hivemind effect of excessive agreement" that reduces perspective diversity. Note this
  is a **position paper**, argumentative by design -- cite it as such, not as an
  empirical result.
- Teplitskiy et al., field experiment with faculty reviewing real grant proposals:
  reviewers scored independently, were then shown a randomly-sampled peer score, and
  **47% updated -- in all but one case of ~185, toward the external score**, regardless of
  merit. (Working paper / AEA session paper; cite it as field-experimental working-paper
  evidence, not as a journal result.)
- Bailar, BBS 1991, the canonical statement of why this matters: "Too much agreement is
  in fact a sign that the review process is not working well, that reviewers are not
  properly selected for diversity, and that some are redundant."

Human peer review is the calibration point, and it is a low bar. Bornmann, Mutz & Daniel
(PLoS ONE 2010), meta-analysis of 48 studies / 19,443 manuscripts: mean ICC 0.34, mean
Cohen's kappa **0.17**. The NeurIPS 2021 consistency experiment (Beygelzimer et al.,
arXiv:2306.03262): 882 duplicated papers, 23.0% accept/reject disagreement against a
**35.1% random baseline**, and ~50.6% "arbitrariness" -- about half the accept list would
change on a rerun. Always quote that 35.1% baseline alongside the 50.6%; the bare number
is the most alarming framing of a paper that is more measured than that.

The transferable structural fact: **reviewers agree substantially about what is bad and
barely at all about what is good.** NeurIPS 2021 reject-flip was 14.9% against ~50%
accept-flip. Weight your panel's convergent negative findings heavily and its convergent
positive findings hardly at all.

### 3.2 The five rules

1. **No shared context before verdicts.** Every phase-1 peer receives the draft and its
   own rubric. It receives nothing about the other peers -- not their existence, not their
   count, not their assignments, not a partial finding. A peer that knows it is one of
   four behaves differently from one that thinks it is the only reviewer.

2. **Fan out in a single message.** This is not a performance optimization, it is the
   mechanical guarantee. Issue all peer calls as parallel tool uses in one assistant
   message. If you issue them sequentially, peer 2's prompt is constructed by a you that
   has already read peer 1's report, and anchoring leaks through your own prompt
   construction even when you intend it not to. Sequential fan-out is the single easiest
   way to silently destroy the panel's value.

3. **Separate prompts, not one template with a persona swapped in.** The degenerate
   configuration -- one model, one prompt template, N personas -- is the thing to design
   against, and the evidence against it is direct. Zheng, Pei, Jurgens et al., "When 'A
   Helpful Assistant' Is Not Really Helpful," Findings of EMNLP 2024 (arXiv:2311.10054):
   162 roles across 6 relationship types and 8 expertise domains, 4 model families, 2,410
   questions -- personas do not reliably help, and **automatic persona selection performs
   no better than random.** Give each peer a genuinely different *rubric and evidence
   focus*, not a costume. Different questions produce different answers; different
   costumes do not.

   The counter-evidence is real but narrow: ChatEval (Chan et al., ICLR 2024) found role
   diversity load-bearing for evaluator agents specifically -- single-agent 53.8%,
   same-role multi-agent 53.8%, **diverse-role 60.0%**. The reconciliation is that
   personas help when the task is subjective evaluation, where different criteria
   genuinely apply, and do not help on tasks with one verifiable answer. Peer review is
   the former, so personas earn their place here -- but as criterion carriers, not as
   flavor.

4. **No visibility of drafts, ever -- only extracted issues.** In phase 2 peers see a
   deduplicated list of major issues, stripped of authorship and of prose. They never see
   another peer's full report. Full reports transmit tone, confidence, and framing, all of
   which are anchoring vectors carrying no evidential content.

5. **Cap deliberation at one round.** Kaesberg et al. is explicit that rounds before
   voting hurt. The panel gets exactly one rebuttal pass. If it has not sharpened by then,
   more rounds will produce convergence, not accuracy.

### 3.3 The failure mode this is all aimed at

**All peers on one model with one prompt template is not a panel. It is one reviewer
sampled N times, and it will report high confidence.** Baumann et al. measured exactly
this. The symptom is a panel that agrees, which reads like corroboration and is not: the
effective sample size is near one, so the agreement is a property of the sampler, not of
the draft.

Instrument for it rather than assuming you avoided it. The cheap diagnostic is the
**oracle gap** -- the best finding present anywhere in the phase-1 reports minus the
finding that survives into the meta-review. A large gap means synthesis is discarding
correct minority findings, which is the specific way panels fail. Also track whether any
phase-1 issue was raised by exactly one peer: **a panel that produces zero singleton
findings has converged and is not worth its cost.**

---

## 4. Persona-to-model assignment

### 4.1 Principles

- **Assign by rubric first, model second.** The rubric is what actually decorrelates the
  reports. Pick the criterion each seat owns -- soundness, novelty/prior work,
  clarity/presentation, reproducibility, ethics -- then choose a model tier that fits the
  cognitive load of that criterion (2.3).
- **Copy ARR's soundness/excitement split.** ACL Rolling Review declares Excitement
  "orthogonal to the assessment of the soundness of the paper," and requires low Soundness
  scores to be justified in the review text. That separation -- "is it correct," arguable
  from evidence, versus "do I care," taste -- puts the ~50% subjective variance in a box
  where it can be read separately. Cheapest calibration constraint available.
- **Decompose into specific verification questions; do not ask a persona to opine.** Liu &
  Shah, "ReviewerGPT?" (arXiv:2306.00622): 13 short papers each with one deliberately
  inserted error -- GPT-4 found it in **7 of 13 when asked specifically to check
  correctness, and did markedly worse when simply asked to "write a review."** Targeted
  prompting beats holistic review prompting. This is the highest-leverage prompt-design
  finding for this skill.
- **Give the novelty seat retrieval, or do not let it judge novelty.** Bare LLMs
  systematically praise novelty in strengths and rarely critique it in weaknesses. Afzal
  et al., EACL 2026, show the failure is fixable specifically by forcing comparison
  against retrieved prior work: 86.5% alignment with human reasoning, 75.3% agreement on
  novelty conclusions. The mechanism is citing specific prior work rather than asserting
  novelty. A novelty referee without WebSearch is worse than no novelty referee.
- **Blind the panel to author identity.** Not optional. Tomkins, Zhang & Heavlin (PNAS
  2017, WSDM 2017 experiment): single-blind reviewers were **1.63x** more likely to
  recommend acceptance for famous authors, 1.58x for top universities, 2.10x for top
  companies. Macharla Vasu et al. (ACL 2026, arXiv:2509.13400) find LLM reviewers carry
  the same affiliation and seniority biases, and that these become **more** pronounced in
  token-level soft ratings -- score-level alignment can mask bias without removing it.
- **Absolute scores from an LLM panel are close to meaningless.** LLM reviewers show
  positivity/score inflation, and Ye et al. (arXiv:2412.01708) found an LLM can rate an
  *incomplete* paper higher than its complete version. Use rubric-decomposed, relative,
  and issue-level judgments. Do not report a panel mean as if it were a score.

### 4.2 Why a non-Claude peer -- the honest version

The team lead's framing was that a non-Claude peer "materially changes the panel's error
profile." That is **half-supported**, and the skill should not overclaim it. The
adversarial verification pass in this research explicitly flagged it as the one clause the
surviving evidence contradicts as written.

**What the evidence does NOT support: vendor diversity as a decorrelation lever.**

- Kohli, "Nine Judges, Two Effective Votes" (arXiv:2605.29800, May 2026): a 9-judge panel
  across 7 model families yields a Kish effective sample size of **1.99-2.48** independent
  votes out of 9. Same-family error correlation was phi=0.437/0.435 versus a cross-family
  mean of phi=0.389 -- **a gap of 0.047** -- and the three most-correlated pairs were all
  *cross*-family (Claude x Gemini phi=0.603). Majority vote, Dawid-Skene EM,
  accuracy-weighted and phi-optimal weighting closed at most 11% of the gap to the
  independence-implied accuracy. Dawid-Skene **underperformed plain majority vote.** The
  paper also concludes the best single judge matched or beat the full panel in every
  condition.
- Kim, Garg, Peng & Garg, "Correlated Errors in Large Language Models," ICML 2025
  (arXiv:2506.07962): across 350+ models, **models agree ~60% of the time conditional on
  both being wrong**, and "larger and more accurate models have highly correlated errors,
  even with distinct architectures and providers." Correlation gets *worse* as the panel
  gets stronger.

Caveat both: Kohli is a single-author 2026 preprint. Kim et al. is peer-reviewed ICML and
is the stronger citation. Note also that Kohli's numbers are on NLI-style labeling, not
open-ended review -- the transfer to referee reports is plausible but not measured.

**What the evidence DOES support, and what the non-Claude seat is actually for:**

1. **Self-preference and preference leakage.** Panickssery, Bowman & Feng (NeurIPS 2024)
   found a **linear correlation** between self-recognition ability and self-preference
   (the paper argues causally but hedges; do not upgrade this to "linear causal
   relationship"). Li et al., "Preference Leakage," ICML 2025 (arXiv:2502.01534): when
   generator and judge are the same model, in an inheritance relationship, or in the same
   family, the judge favors the related output. **The draft under review here was very
   often written with Claude's help.** That is a concrete, mechanism-level reason for a
   non-Claude referee that has nothing to do with generic error decorrelation.

   Two honest complications. Chen et al., EMNLP 2025 (arXiv:2506.02592) show self-preference
   bias **shrinks sharply with model size** -- Llama-3.1-8B 21.6% DBG versus 70B 0.4%;
   Qwen2.5-0.5B 41.7% versus 14B 2.1%. And Zhao et al.'s Language Model Council (NAACL 2025)
   found 12 of 20 models showed positive self-enhancement bias but that **ensembling
   effectively neutralized it.** So: on a frontier-model panel, self-preference is a
   smaller worry than folklore suggests, and panels genuinely do wash out *idiosyncratic*
   self-preference. What they do not wash out is *correlated* error, and that gets worse
   with scale. The two effects cut in opposite directions and the skill should say so.

2. **Panels do beat single judges sometimes.** Verga et al., PoLL (arXiv:2404.18796): a
   three-model panel from disjoint families (Command R, Claude Haiku, GPT-3.5) reached
   Cohen's kappa **0.763** on Natural Questions versus **0.627** for a single GPT-4 judge,
   at seven to eight times lower cost. Two cautions: the "7x cheaper" is an **April 2024
   price ratio** and has decayed badly -- do not quote it as current; and PoLL and Kohli
   are in direct conflict, PoLL finding the panel beats a strong single judge and Kohli
   finding the best single judge beats the panel. Different tasks, different eras,
   unresolved.

3. **Route-level independence.** The Claude peers all run through one harness with one
   set of system-prompt and safety layers. gpt_brad exits that entirely. Even if model
   errors correlate, *pipeline* errors do not -- a harness-level constraint that shapes
   all four Claude reports will not shape the OpenAI one. This is the least glamorous
   argument and probably the most robust.

**The defensible claim, and what the skill should say:** seat one non-Claude peer for
self-preference, preference leakage, and route independence. Do **not** claim it buys
error decorrelation -- state plainly that cross-vendor diversity is a weak lever
(0.047 phi difference, Kohli) and that the real independence work is done by separate
rubrics, blind phase-1, and the single-round cap. The 1.4 smoke-test coincidence is the
in-house illustration.

### 4.3 Recommended default seating (4 peers + meta)

| Seat | Rubric it owns | Where it runs |
|---|---|---|
| Methods / soundness | Claims vs evidence; stats; does the design support the conclusion | Agent, `model: "opus"` |
| Novelty / prior work | Retrieved-and-cited comparison to specific prior work. **Needs WebSearch.** | Agent, `model: "sonnet"` |
| Clarity / presentation | Structure, figures, reproducibility, whether each section does what it claims | Agent, `model: "haiku"` |
| Adversarial outsider | Hostile read: what would a skeptic in a different subfield say? | gpt_brad (`gpt-5.4-2026-03-05`); cursor `gpt-5.2` if down |
| Meta-reviewer | Phase 3 synthesis only. Sees reports, never the peers. | Agent, `model: "opus"` |

Swap the adversarial seat to Cursor `cursor-grok-4.5-high` or `kimi-k3-high` when you want
a third lineage, accepting the ~12s/turn latency.

---

## 5. Three-phase protocol

### Phase 1 -- Independent referee reports

**Context each agent gets:** the draft (author identity stripped), its own rubric, the
venue norms, and the output schema. **Nothing else.** No peer count, no other rubrics, no
prior findings. Fan out in ONE message (3.2 rule 2).

Prompt skeleton for each peer:

```
You are refereeing a submission. You are the only reviewer; write as if your report
is the sole basis for the decision.

YOUR ASSIGNED CRITERION: <soundness | novelty | clarity | adversarial outsider>
Judge primarily on this criterion. Note issues outside it only if severe.

<venue norms>

SPECIFIC QUESTIONS YOU MUST ANSWER: <5-8 targeted verification questions, e.g.
"Does every numeric claim in section 3 trace to a stated source?" -- NOT "write a
review". Targeted beats holistic (ReviewerGPT, 7/13 vs markedly worse).>

Report only what you can point to in the text. Quote the passage for every issue.
If you cannot support an issue with a quotation, do not raise it.
Return ONLY the JSON below.

SUBMISSION:
<draft>
```

**Must return** (identical schema for every peer, including the bridge peers -- ask for
it in the prompt and parse stdout):

```json
{
  "peer_id": "methods",
  "criterion": "soundness",
  "summary": "2-3 sentences: what the submission claims and does",
  "major_issues": [
    {"id": "M1", "quote": "verbatim passage", "location": "section 3, para 2",
     "issue": "what is wrong", "why_it_matters": "consequence if unfixed",
     "fix": "concrete, actionable", "confidence": "high|medium|low"}
  ],
  "minor_issues":  [ {"quote": "...", "issue": "...", "fix": "..."} ],
  "strengths":     ["specific, not generic"],
  "soundness":  {"score": 1-5, "justification": "required if score <= 2"},
  "excitement": {"score": 1-5, "justification": "..."},
  "reviewer_confidence": 1-5,
  "unable_to_assess": ["what this peer could not judge and why"]
}
```

Keep `soundness` and `excitement` orthogonal (ARR). Require justification text for low
soundness. `unable_to_assess` is load-bearing -- it is how the meta-reviewer knows a
finding's absence is not evidence of absence.

### Phase 2 -- Moderated rebuttal (exactly one round)

The moderator (you) extracts every `major_issues` entry from every phase-1 report,
deduplicates near-identical issues, and **strips peer attribution and all prose framing**.
What remains is a bare numbered issue list.

**Context each agent gets:** its own phase-1 report (it already has this if named; re-send
it if not), plus the anonymized combined issue list. **It never sees another peer's full
report.**

Named-agent path (top-level session):

```
SendMessage({to: "peer-methods", summary: "Rebuttal round: consolidated issues",
             message: "<rebuttal prompt + anonymized issue list>"})
```

Unnamed path (running inside a subagent -- see 2.1): re-issue a fresh Agent per peer with
the draft, that peer's own phase-1 report, and the anonymized issue list in the prompt.
More tokens, same protocol.

Rebuttal prompt:

```
Below are the major issues raised across the panel, anonymized and deduplicated.
Some are yours. You do not know which.

For each issue: SHARPEN (you can strengthen it with evidence from the text),
CONCEDE (you now think it is wrong or overstated -- say why), or ABSTAIN
(outside your criterion or competence -- this is a legitimate answer and you
should use it freely).

Do NOT change a position because other reviewers appear to hold it. Agreement is
not evidence. If you cannot point to text in the submission, ABSTAIN.

ISSUES:
<numbered anonymized list>
```

**Must return:**

```json
{
  "peer_id": "methods",
  "positions": [
    {"issue_id": "M1", "stance": "sharpen|concede|abstain",
     "reasoning": "must cite the submission text",
     "added_evidence": "quote, or null",
     "revised_severity": "blocking|major|minor|not-an-issue"}
  ],
  "new_issues": [ "<same object shape as major_issues; only if phase 2 genuinely surfaced it>" ],
  "position_changed": true
}
```

Track `position_changed` across peers. **A high rate of changed positions in the
sharpen/concede direction toward apparent majority is the herding signature** --
AgentReview's 27.2% variance collapse. If most peers moved toward the majority, say so in
the final output and discount the convergence explicitly.

**Stop here. No round two.**

### Phase 3 -- Meta-review synthesis

**Context:** all phase-1 reports, all phase-2 positions, and the herding diagnostics.
**Not** the draft in full -- the meta-reviewer synthesizes reports, it does not re-referee.
It must never have talked to the peers.

Run on `model: "opus"`, `run_in_background: false` (you need the result to finish).

Follow ICLR 2026's meta-review requirements: a description of the submission's claims with
strengths and weaknesses, a summary of the discussion including concessions, and a
recommendation with justification.

**Must return:**

```json
{
  "summary": "what the submission claims and does",
  "consensus_issues":  [{"issue": "...", "peers_raising": 3, "peers_conceding": 0,
                         "severity": "blocking|major|minor"}],
  "contested_issues":  [{"issue": "...", "case_for": "...", "case_against": "...",
                         "moderator_read": "..."}],
  "singleton_issues":  [{"issue": "...", "raised_by_criterion": "novelty",
                         "kept": true, "why": "..."}],
  "conceded_issues":   [{"issue": "...", "why_dropped": "..."}],
  "coverage_gaps":     ["what no peer could assess"],
  "priority_fixes": {"must_fix": [], "should_fix": [], "nice_to_have": []},
  "recommendation": "with justification",
  "panel_health": {
     "singleton_count": 4,
     "position_changes_toward_majority": 1,
     "oracle_gap_note": "any strong phase-1 finding dropped in synthesis, and why",
     "independence_caveat": "state plainly if peers converged"
  }
}
```

`singleton_issues` and `panel_health` are not decoration. A singleton finding from the one
peer qualified to see it is the highest-value output a panel produces, and majority-style
synthesis is exactly what destroys it. **Default to keeping singletons** and require a
stated reason to drop one.

---

## 6. Cost and latency tradeoffs

Measured components, 2026-08-05:

| Component | Wall clock |
|---|---|
| gpt_brad `model` probe | 0.59s |
| gpt_brad `ask`, one sentence | 2.35s |
| bedrock `ask` (failing fast on creds) | 0.43s |
| cursor `start` | 5.48s |
| cursor `ask`, one sentence | 12.27s |

Real referee reports are far longer than one sentence. Budget 30-90s per bridge peer and
1-4 min per Claude Agent peer on a substantial draft. **Because phase 1 fans out in
parallel, panel wall clock is set by the slowest peer, not the sum.**

Cost is in **peer-turns** (one model call producing a report or a rebuttal). Convert to
dollars with the `claude-api` skill; do not guess.

| Config | Peers | Phases | Peer-turns | Wall clock | Use it when |
|---|---|---|---|---|---|
| Quick read | 2 (1 opus, 1 gpt_brad) | 1 only | 2 | ~1-2 min | Early draft, "is this broken" |
| Standard | 3 (opus, sonnet, gpt_brad) | 1 + 3 | 4 | ~3-5 min | Routine review |
| **Default (recommended)** | **4 (opus, sonnet, haiku, gpt_brad) + meta** | **1 + 2 + 3** | **9** | **~6-10 min** | **Anything Mason submits or publishes** |
| Deep | 6 (2 opus, 2 sonnet, gpt_brad, cursor) + meta | 1 + 2 + 3 | 13 | ~12-20 min | Grant, paper, high-stakes submission |

**Recommended default: 4 peers, full three phases, one meta-reviewer -- 9 peer-turns.**

The reasoning against going bigger is empirical, not budgetary. Kaesberg et al. found more
agents help; Zhao et al.'s Language Model Council found the marginal benefit of an
additional judge falls off around **~9 judges**; and Kohli found the first 5 judges deliver
~90% of achievable independence, with n_eff plateauing near 2.6 regardless. Past roughly 5
peers you are paying for redundancy. Spend the marginal budget on **better rubrics and
retrieval for the novelty seat**, not on more seats.

---

## 7. Failure modes and recovery

| Failure | Detection | Recovery |
|---|---|---|
| **Agent returns nothing / empty** | No report, or report with empty `major_issues` and empty `unable_to_assess` | Re-issue **once** with the schema restated and an explicit "return only JSON" instruction. On second failure, drop the seat and record it in `coverage_gaps`. Never fill the gap yourself. |
| **Bridge times out** | gpt_brad: 180s hard cap, no env override. cursor: 600s. Both exit nonzero. | Split the prompt (shorter draft excerpt, fewer questions) and retry once. Then fall through: gpt_brad -> cursor `gpt-5.2` -> Claude seat, **labeling the substitution in `panel_health`**. |
| **Bedrock creds expired** | Exact string: `Bedrock call failed -- sandbox SSO creds look expired or unset; re-auth and retry.` Note `bedrock_brad.py model` prints fine anyway -- it never touches creds. | Tell Mason to re-auth the sandbox SSO. Do not block on it: run the panel with cursor in that seat and note it. |
| **Peer breaks format** | JSON parse fails, or required keys missing | Re-issue once with the malformed output quoted back and the schema restated. If it fails again, parse what you can by hand into the schema (the CME pattern) and mark the peer `format_degraded` in `panel_health`. Never discard substantive findings over a formatting failure. |
| **Partial panel** (n < planned) | Fewer reports than seats | Proceed, and **state the actual panel composition in the output.** A 2-peer panel reported as 4 misstates the evidence base. If only one peer survives, say plainly that this is a single review, not a panel. |
| **Panel converges (the real failure)** | `singleton_count == 0`, or most `position_changed` moved toward the majority | Report it as a finding. State in `independence_caveat` that the panel converged and its agreement should not be read as corroboration. Optionally re-run one seat on a different model with a deliberately hostile rubric. Do not silently present convergence as confidence. |
| **Peer reviews the wrong thing** | Report cites text not in the draft, or ignores its criterion | Almost always truncated context. Verify the draft was fully included, re-issue once. Repeated hallucinated quotes = drop the seat; the quote-required rule in the phase-1 prompt exists to make this detectable. |
| **Named spawn rejected** | `Teammates cannot spawn other teammates -- the team roster is flat.` | You are inside a subagent. Re-issue without `name`, and use the unnamed phase-2 path (re-issue fresh agents carrying context) instead of SendMessage. |
| **Em dashes in peer output** | Bridge peers emit them (both smoke tests did) | Normalize to `--` before any of it reaches a Mason-facing deliverable. |

**Standing rule:** if a bridge is down or an agent fails, degrade openly. Every recovery
above is safe except one -- writing a peer's report yourself. A fabricated referee report
makes the panel actively worse than not running it, because it launders your own opinion
as independent corroboration.
