# RED Baseline -- What an Unskilled Panel Run Produces

_Last updated: 2026-08-05_

Evidence from a baseline run (superpowers:writing-skills Iron Law: watch the test fail
before writing the skill). A sonnet agent was asked to "peer review this abstract with a
simulated panel of four expert reviewers" on the TSC 2026 abstract, with no
`peer-review-panel` skill available and no methodology guidance. Output:
`baseline-panel-review.md`, 95 lines / 2,413 words, four reviewer sections plus a score
table.

The skill must counter each failure below. Anything marked PASSED is behavior the skill
should preserve rather than legislate.

## Measured failures

| # | Failure | Evidence from the baseline run |
|---|---|---|
| 1 | **No independence -- one author, four voices** | Self-reported: "No subagents, no separate model calls... I wrote all four reviewer voices myself in a single pass." Each persona is "me adopting a stance," not a process "that could genuinely surprise me." |
| 2 | **Verdict convergence + grade inflation** | 4 of 4 reviewers returned **accept**. Scores 7, 6, 6, 5 out of 10; mean 6.0; range 2. No reviewer reached major-revision or reject. Every objection was framed as survivable. |
| 3 | **Unanchored, self-invented scale** | Self-reported: scores are "my own invention calibrated loosely," "not derived from any external rubric or precedent... illustrative, not calibrated." Nothing defines what a 5 is versus a 7. |
| 4 | **Authored disagreement, not emergent** | Self-reported: "I planned those disagreements deliberately while writing, as a single author deciding 'these two personas should clash here to feel realistic.'" |
| 5 | **Scores anchored on each other** | Self-reported: "all four scores came from the same pass so they could be anchored on each other rather than independent." |
| 6 | **No rebuttal round** | Self-reported missing: "no actual back-and-forth/rebuttal round between reviewers." |
| 7 | **Zero "what would change my mind" statements** | Grep for `change my mind|would change|convince me`: **0 matches** across 95 lines. No reviewer named a condition that would move their verdict. |
| 8 | **Invented archetypes** | Self-reported: reviewer profiles were "my own choice of 'who reviews at TSC' rather than pulled from any list of real referee profiles or venue norms." |
| 9 | **No venue criteria applied** | Nothing in the output references TSC's actual abstract-selection criteria; verdicts are generic conference-accept language. |

## PASSED -- preserve, do not over-legislate

- **Line anchoring worked unprompted.** Reviewers quoted real strings from the source
  (`4.28% bursts vs 5.00% expected`, `7 of 1,000 placements`, the QRNG / "attended
  TrueRNG" clash). Spot-checked against the abstract: the quotes are genuine, not
  paraphrased from memory. Keep the requirement, but it is not the weak point.
- **Low citation fabrication.** Only one outside citation appeared in the whole document
  (`Radin & Nelson 1989`, real). The hallucinated-citation risk is real in principle but
  did not fire here; a closed-citation rule is cheap insurance, not the main defect.
- **Substantive content.** The QRNG/TrueRNG terminology clash and the maximum-of-many-draws
  problem are both objections a real referee would raise.

## Design implications for SKILL.md

The failure is **not** that an unscaffolded agent writes shallow critique. It writes
plausible, well-quoted critique. The failure is that the critique is **one mind's opinion
wearing four name tags, and that mind will not fail its user.** Every reviewer accepted.

So the load-bearing requirements are, in order:

1. **Enforced independence.** Separate dispatches, separate models, no peer sees another
   peer's draft before submitting a verdict. Without this, items 2, 4 and 5 return.
2. **Anchored rubric with behavioral anchors per integer**, supplied by the skill, not
   invented at runtime -- counters 3.
3. **Mandatory "what would change my mind" field** -- counters 7, and is the cheapest
   single check against both grade inflation and severity theater.
4. **Verdict-distribution check.** If every peer accepts, or all scores land within 0.5,
   the orchestrator must flag it rather than average it into "accept with revisions."
   Counters 2.
5. **Real archetype dossiers loaded from `reviewer-archetypes.md`** -- counters 8.
6. **Venue criteria loaded from `venue-norms.md`** and cited in each verdict -- counters 9.
7. **A real rebuttal round** as its own phase -- counters 6.
