_Last updated: 2026-08-07_
Experiment 5 ablation -- which part of "structured prompting" is actually doing the work
Why this ran
Experiment 5 reported that structured beat prompts beat raw paragraph piping by 20-27 element-recall points. That number was real but confounded: arm B changed four things at once relative to arm A. It injected each character's physical attributes, imposed labeled segment structure, stated counts and spatial relations explicitly, and appended an exclusion clause.
That distinction decides an engineering workstream. Production already injects character attributes on every page -- the [A] / [B] verbatim-copy block puts a 9th grader Caucasian boy wearing crimson jacket into every prompt. If attribute injection is where the win lives, production has already banked it and there is nothing to build. If the labeled structure and explicit counts are doing the work, production needs a real structured-beat assembler.
Four rungs, each adding one component to the one before, on the tier-4 binding-trap beats:
- attrs -- the paragraph + character attribute injection only
- structure -- + labeled segment structure
- counts -- + explicit counts and spatial relations
- exclusion -- + the exclusion clause (byte-identical to experiment 5's arm B, pinned by test)
2 models x 4 rungs x 4 beats x 2 replicates = 64 renders, opus-judged.
Results: element recall
| Model | Rung | Recall | Misbound | Composition | Joint-correct |
|---|
| gpt-image-2 | attrs | 100% | 0% | 5.00 | 8/8 |
| gpt-image-2 | structure | 90% | 6% | 4.38 | 5/8 |
| gpt-image-2 | counts | 100% | 0% | 5.00 | 8/8 |
| gpt-image-2 | exclusion | 98% | 2% | 4.88 | 7/8 |
| nano-banana | attrs | 87% | 10% | 4.12 | 3/8 |
| nano-banana | structure | 81% | 15% | 3.75 | 2/8 |
| nano-banana | counts | 92% | 6% | 4.75 | 6/8 |
| nano-banana | exclusion | 88% | 12% | 4.50 | 4/8 |
The first rung is the whole effect. Attribute injection alone puts gpt-image-2 at 100% recall, zero misbinding, and 8/8 joint-correct -- matching what experiment 5's fully structured arm B achieved, with none of arm B's other three components present. nano-banana's attrs rung lands at 87%, inside the range its arm B reached.
Rungs 2 through 4 move within a few points in both directions and never above rung 1's ceiling. The sequence is not monotonic on either model -- recall dips at structure and recovers at counts -- which is itself the tell: increments that were doing real work would accumulate rather than wobble. At n=8 cells per rung these movements are noise around a flat line, not a ranking.
Results: unprompted lettering
The exclusion clause looked worthless above. It is not -- it is doing a different job.
| Model | attrs | structure | counts | exclusion |
|---|
| gpt-image-2 | 6/8 | 8/8 | 6/8 | 0/8 |
| nano-banana | 1/8 | 3/8 | 0/8 | 0/8 |
gpt-image-2 volunteers lettering in 75-100% of cells on every rung that lacks the exclusion clause, and in zero cells once it is present. One sentence is a complete off-switch, replicating experiment 5's finding on a different beat set.
The read
The two components that matter each fix a different failure, and neither of the other two fixes anything measured here:
- Attribute injection carries element recall and character correctness. It is the entire 20-27 point gap.
- The exclusion clause carries lettering suppression. It contributes nothing to recall and everything to keeping invented text off the page.
- Labeled structure and explicit counts did not earn their place on these beats. See the 2026-08-07 correction below before generalizing that to harder scenes.
What this changes for production
Do not build a structured-beat prompt assembler on this evidence alone. Production already injects character attributes on every page, which means it has already captured the component that produced the entire measured win on these beats. The workstream that experiment 5's headline number appeared to justify does not survive the ablation as stated.
Correction added 2026-08-07, after the tier-5 beats were judged. The sentence above is true for tier-4 beats and should not be generalized past them. On the tier-5 compound traps -- several constraints that must all hold in one image -- the full structured prompt still beats the bare paragraph 91% to 65% on gpt-image-2. That tier was never put through this ladder, so we cannot say whether its 26-point gap is also just attribute injection or whether labeled structure finally starts earning its keep once a scene gets dense enough. Decomposing tier 5 is roughly 64 renders and it is the measurement that settles whether the assembler is inert or merely unnecessary at today's page complexity. Until that runs, treat this as "not justified at the complexity we ship today" rather than "never worth building."
Do add a text-suppression clause. Production's image-prompt builder has no exclusion clause of any kind -- a search of generateImagePromptsForParagraphs.ts for lettering, text, signage, or caption directives returns nothing. On the evidence above that is a 75-100% lettering rate on the lead model, fixable by appending one sentence to a prompt template. No other correctness win in this program costs so little: no pipeline change, no new component, no added inference cost.
What is worth measuring next is whether production's existing attribute injection is as good as the harness's. The harness injects a compact per-beat descriptor (Brad (tan jacket, black trousers, short dark wavy hair)); production copies a longer descriptor built from database fields and currently ships two known defects into it (a missing null guard that can emit a null-year-old ... human, and label markup leaking into the final prompt). Those are filed and patched. Until they land, an A-to-B comparison against live production prompts measures the defects as much as the design.
Standing caveats
n=8 cells per rung, one style, tier-4 beats only -- and that last limit turned out to matter more than expected once tier 5 was judged. The rungs are cumulative rather than factorial, so this isolates where the gain first appears but cannot detect an interaction between structure and counts. The attrs rung minus its injected attribute text reproduces experiment 5's arm A exactly, and the exclusion rung is byte-identical to its arm B -- both anchored by committed fixtures, so this ladder is directly comparable to the experiment 5 numbers rather than merely similar to them.
Cost ~$2.50.
_Last updated: 2026-08-04_
Harder beats for a saturated rubric (tier 5)
Experiment 5 measured scene accuracy and produced a real finding: replacing the story
paragraph with a structured beat prompt takes element recall from 74% to 100% on
gpt-image-2 and from 72% to 90% on nano-banana, with actor recall going from 24%
and 18% to 100% on both. It also produced a problem that blocks every follow-up.
**All 32 of gpt-image-2's structured cells scored exactly 100% element recall and
exactly 5.00 composition.** Tier 1 and tier 4 alike, binding traps included. A
metric pinned at the top of its range cannot separate good from excellent, cannot
catch a regression, and cannot say how much headroom is left. Until the beats got
harder, no further number about the lead model meant anything.
This document is the fix: a fifth tier of compound traps, plus the probe that
validated it. It also reports a second finding that came out of the probe and
matters more than the tier itself, because it constrains what the program can
measure at all.
What tier 5 is
Tier 4 scored atoms that mostly stood alone: a red scarf on Maria, three pots on
a shelf, a cat under a table. Each element could be satisfied without reference
to the others, which is why a strong model could sweep them. Tier 5 scores atoms
that constrain each other, so a model can hit every individual noun and still
fail the beat.
The prior being tested is the GenEval 2 prompt-level cliff: 84.4% correctness
when atoms are scored individually against 31.0% when the whole prompt has to be
right at once. Tier 4 measured the first number. Tier 5 is built to measure the
second.
| Beat | Elements | Failure family it targets |
|---|
t5-seven-acorns | 8 | exact count above the reliable shelf (seven) crossed with ordinal position ("the fourth from the left") |
t5-empty-hands | 6 | required absence, the documented worst case, since these models have no mechanism for "no" |
t5-crossed-hats-trio | 11 | three-way crossed attribute permutation, anti-correlated on purpose so nobody's book matches their own hat |
t5-size-order | 9 | comparative size crossed with left-to-right ordering and color, all three of which must agree |
t5-left-hand-lantern | 9 | handedness, disambiguated against the viewer, plus deliberate partial-frame cropping |
Two mechanical additions came with the tier.
A new absence checklist category. Its entries carry a positive claim about
what the emptiness should look like plus the thing that must not appear. The GPT
family gets both, because exclusion clauses are endorsed there; Gemini gets the
claim only, because its playbook is explicit that negative prompting hurts on
that family. That split is a deliberate per-family difference, not an oversight.
A prompt-level metric, jointCorrect. A cell passes only if every element
landed. One wrong atom takes the cell from 1 to 0, which is why it stays
sensitive after per-element recall has already pinned at the top. It is computed
from the element list rather than asked of the judge, so it costs nothing and
cannot drift.
The tier-5 composition anchor was also tightened: a 5 now has to survive an art
director with no notes, and an image where every element is correct but the
anatomy is awkward or the focal point is muddled takes a 4. Tiers 1 and 4 keep
the original wording so their experiment 5 scores stay comparable.
Adding tier 5 changed no tier-1 or tier-4 prompt. All 133 prompts recorded in the
experiment 5 run directories rebuild byte-identically from the current code, so round
5 remains reproducible.
The probe
Five cells: every tier-5 beat once, on gpt-image-2, arm B (the structured prompt),
one replicate. This was validation of the instrument, not a measurement round.
The test-account token was expired, so the judge ran through the Anthropic key rather
than Bedrock. Same model, claude-opus-4-7, different route.
| Beat | Recall | Joint | Composition |
|---|
t5-empty-hands | 6/6 (100%) | pass | 5 |
t5-crossed-hats-trio | 11/11 (100%) | pass | 5 |
t5-size-order | 9/9 (100%) | pass | 5 |
t5-left-hand-lantern | 8/9 (89%) | fail | 4 |
t5-seven-acorns | 4/8 (50%) | fail | 2 |
| pooled | 38/43 (88%) | 3/5 (60%) | 4.2, spanning 2 to 5 |
Against experiment 5's 100% / 100% / 5.00 on the same model and the same arm. The
ceiling is broken and the scale has usable room under the top again.
What actually broke, and what did not
Deliberate cropping failed. Asked for Mr Penguin half out of frame at the
right edge, gpt-image-2 cropped a sliver instead. The model resists composing a
subject partly outside the picture, which is a real production constraint: story
pages often want a character entering or exiting a scene.
Counting failed, by one. Asked for seven acorns, the model rendered eight.
The ordinal it got right, which is the more interesting half: the gold acorn
landed fourth from the left exactly as asked, and the count around it drifted.
Three-way attribute permutation passed cleanly. Nine bindings across three
characters, deliberately anti-correlated, all correct. Tier 4's crossed-attribute
family is solved on this model, and scaling it from two-by-two to
three-by-three does not reopen it. Report that as a capability, not a gap.
Comparative size with ordering passed. Biggest blue on the left through
smallest red on the right, correct on every axis. The one composition point it
lost was an anatomy problem, not a binding problem, which is exactly the
distinction the tightened anchor was added to make.
So per-element recall alone nearly re-saturates: strip out the counting beat and
it sits at 34/35. The discrimination in tier 5 comes mostly from joint
correctness (3 of 4 rather than 4 of 4) and from the composition anchor having
somewhere below 5 to land. Know that before quoting the pooled recall
figure as the headline.
The finding that matters more: the judge cannot count either
On the counting beat the judge reported nine acorns with the gold one fifth from
the left. It reported the same thing twice, in independent passes, once before a
counting discipline was added to the rubric and once after. A pixel count of that
image gives eight acorns, gold fourth from the left.
The judge landed the right pass or fail on every element of that beat, but only
by luck: the generator was also wrong, so "misbound" was the correct verdict for
the wrong reason. Had the model rendered exactly seven, the grader would most
likely have failed a correct render. That is a false negative on the lead model,
produced by the instrument rather than the subject.
The counting discipline added in response is still worth keeping, because of what
it did fix. It requires the judge to walk any countable group strictly left to
right, list every item with its color or size, and score counts and ordinals
against its own enumeration rather than against the checklist's claim. On the
handedness beat that discipline moved the verdict from 5/9 to 8/9 on the same
image: forced to enumerate, the judge correctly resolved that the raised hand on
the viewer's right side of the body is the character's own left hand, where
before it had contradicted itself and marked four elements misbound. Enumeration
repaired relational reasoning outright. It did not repair perception. Counting
from a glance is what vision models are worst at, and asking more carefully did
not change what the model saw.
t5-seven-acorns therefore carries a judgeCaveat in its definition, the runner
prints it before every render, and it is written into every result record. **Its
count and ordinal elements must not be quoted as judged data** until a
deterministic checker exists. The other four tier-5 beats carry no caveat; they
were judge-verifiable.
This generalises past this program. Any element the program scores by counting, on any
tier, inherits this error. The three-pots trap in tier 4 is a count of three,
inside the range where the judge is reliable, so experiment 5's numbers stand. Counts
above roughly five need a different instrument.
What this does not settle
- n is 1. One replicate, one model, one arm. The probe shows the tier
discriminates; it does not measure anything.
- One model. nano-banana has not seen tier 5. Experiment 5's degradation curve
predicts it should fall further and faster here, but that is a prediction.
- Arm B only. No paragraph control, no reference arm.
- Judge variance is visible.
t5-size-order drew composition 4 on one pass
and 5 on the next, same image, the 4 attributed to an anatomy note that the
second pass did not raise. Roughly a point of noise on composition, which
argues for replicates before any composition delta under a point gets quoted.
- The counting beat is on hold as described above.
Reproduction
# Full tier-5 battery, both live candidates, all arms.
node the beat-run script --tier=5 --model=gpt-image-2,nano-banana
# The validation probe, as run.
node the beat-run script --tier=5 --model=gpt-image-2 --arm=B \
--replicates=1 --judge-via=claude
# Experiment 5's original slice is still the default and is unchanged.
node the beat-run script
Tier 5 is deliberately not in the default slice. An unqualified invocation still
reproduces experiment 5 rather than silently launching a larger, more expensive run.
Code
In the evaluation harness:
lib/beat-eval-set.mjs -- five tier-5 beats, the absence category, spelled
number words through twelve, BEAT_TIERS, beatKeysForTiers, and the
judgeCaveat field.
lib/beat-judge.mjs -- absence scoring semantics, the tier-5 counting
discipline and its enumeration output field, the tightened tier-5 composition
anchor, jointCorrect in recallStats, and renderBeatChecklist exported so
tests can assert every element reaches the judge.
the beat-run script -- --tier= selection defaulting to 1,4,
tier-derived run directory names, joint correctness in the console line and the
summary table, and the judge caveat printed and recorded.
tests/beat-eval-set.test.mjs -- 19 new tests, including one that pins every
tier-1 and tier-4 arm-B prompt against absence-driven drift.
Suite is green at 419 tests.
Next
- Run the full tier-5 battery on gpt-image-2 and nano-banana, arms A, B, and C,
two replicates, once the test-account token is refreshed so the judge goes back
through Bedrock. Roughly 60 renders.
- Build a deterministic counting checker, or accept human adjudication on count
elements, and unblock t5-seven-acorns.
- Take the cropping failure to the prompt-design side. It is the one tier-5
failure that looks like a prompting problem rather than a measurement problem.