A QA engineer sent me a tidy bug report. Creating a story fails when you pick grade 6 and the character "Unicorn," every time, first attempt or second. She included the story URL and the login code.

Both details in it were wrong. And the thing that was broken was code we had added to make the output more correct.

What the report looked like from the database

Ten stories had failed on our development environment that day. Every one of them died at the same step with the same error:

ZodError: characters_description -- Array must contain exactly 1 element(s) (code: too_big)

Not grade 6: grades 1, 5 and 6. Not Unicorn: Unicorn, Cow and Dragon. All ten had one thing in common: the learner had selected exactly one character.

1 character selected 10 of 70 failed · 13.9%
2 characters selected 0 of 9 failed
One day on our development environment. Nine stories is too few to call the second row a rate, so read it as no failures observed. The first row is the finding: the constraint is tightest exactly where users are most likely to sit.

The constraint

The pipeline asks a model for a short illustrated story plus a structured description of its characters, then validates the response against a Zod schema. The schema built the character array like this:

characters_description: z
  .array(z.union([humanCharacterOutputSchema, animalCharacterOutputSchema]))
  .length(characters.length)
  .describe("one entry per selected character")

.length() is an exact match, not a floor. Select one character and the model must return exactly one description or the entire story is rejected.

That reads reasonable until you think about what the model is being asked to write. A four-paragraph story needs someone for the protagonist to talk to. So the model invents a supporting character, because that is what writing a story requires, and then it describes that character, because the field is called characters_description and describing characters is what the field is for. Two entries. Validation fails. Story gone.

The model did nothing wrong. It satisfied the narrative instruction and the field's own name, and those two things put it in conflict with an arithmetic constraint nobody had tested against a one-character selection.

Retries cannot fix this

The pipeline retries three times on a parse failure. All three attempts on the story she reported failed identically, because the model was not being flaky. It was making the same defensible choice each time, and a retry loop wrapped around a deterministic disagreement just spends the money three times.

That distinction is worth pulling out. Retries protect against variance. They do nothing against a systematic mismatch between what you asked for and what you will accept, and a retry count in your logs makes a structural problem look like a transient one.

The part that made it a regression

Before this pipeline used schema validation, it used a hand-written salvage parser. That parser also looked at characters_description. It threw in exactly one case:

if (characters_description.length === 0) {
  throw new Error("Failed to extract character description in fallback");
}

Empty was fatal. Too many was fine. Surplus entries were tolerated and ignored downstream.

So story creation did not break because the model changed, or the prompt changed, or a learner did something unusual. It broke because we replaced a tolerant parser with a stricter one, and the new strictness covered a case the old code had quietly absorbed for as long as it had existed.

0 entries nothing describes the cast
1 entry exactly what was selected
2 entries the story needed a second character
old parser: reject length === 0 throws
old parser: accept  
old parser: accept surplus ignored downstream
new schema: reject too_small
new schema: accept  
new schema: reject too_big -- the story is lost
The tolerance surface for one selected character. One column moved, in one direction. The migration was reviewed as a correctness improvement, which it was; the column that moved is the one nobody diffed.

Nobody made a bad decision here. Validating model output against a schema is better engineering than regex salvage, and the person who wrote the schema was making the codebase more correct, not less. The failure is that the migration was reviewed as a correctness improvement rather than as a behaviour change, and nobody diffed the two implementations for what the old one had forgiven.

The question I should have been asking

When you replace tolerant parsing with schema validation, you are not only fixing a bug class. You are also making a product decision about what to reject, and the two arrive in the same commit wearing the same clothes.

The useful question is not "is the new schema correct?" It is: what did the old path let through, and is each of those cases now a user-visible failure?

For an array field, that decomposes into things you can check:

Our fix relaxes the floor and drops the surplus: .min(characters.length), then filter the array down to the selected character names before anything downstream reads it. An invented sidekick's description is not something we want to keep. It is also harmless once discarded, and discarding it beats losing the story.

When strict is right

This is not an argument for loose schemas. Exact lengths are correct when the count is a real invariant and a mismatch means the data is missing. Our paragraph array is .length(4) and that one should probably stay strict, because each paragraph becomes an illustrated page and a fifth paragraph has nowhere to go. The right handling there is to trim or regenerate, not to accept five.

The distinction is whether a violation means the response is unusable or the response contains more than you asked for. The first deserves a hard failure. The second deserves a filter. Zod will enforce either one with equal enthusiasm and will not tell you which you meant.

What actually found it

Not the error message, which named a schema path and an array length and pointed at neither the cause nor the pattern. Grouping the day's failures by a variable nobody thought was relevant is what found it. The report said grade 6 and Unicorn, so those were the two columns I checked first, and both were dead ends. Selected-character count was the third guess.

There is a habit worth forming in that. A bug report that arrives with a confident cause is carrying two different things: evidence about symptoms, which the reporter watched happen, and a hypothesis about cause, which they inferred from a sample of one or two. Those deserve different amounts of trust. Grouping every instance by several candidate variables costs one query, and it is the fastest way to learn that the obvious variable is not the one.

Her report was good. It had a URL, a login code, and a reproduction. It was also 13.9% of the story, and finding the rest took a GROUP BY.