The validator was the bug
A bug report blamed a grade and a character. The cause was a one-line schema constraint we had added to make the output more correct
A QA engineer sent me a tidy bug report. Creating a story fails when you pick grade 6 and the character "Unicorn," every time, first attempt or second. She included the story URL and the login code.
Both details in it were wrong. And the thing that was broken was code we had added to make the output more correct.
What the report looked like from the database
Ten stories had failed on our development environment that day. Every one of them died at the same step with the same error:
ZodError: characters_description -- Array must contain exactly 1 element(s) (code: too_big)
Not grade 6: grades 1, 5 and 6. Not Unicorn: Unicorn, Cow and Dragon. All ten had one thing in common: the learner had selected exactly one character.
The constraint
The pipeline asks a model for a short illustrated story plus a structured description of its characters, then validates the response against a Zod schema. The schema built the character array like this:
characters_description: z
.array(z.union([humanCharacterOutputSchema, animalCharacterOutputSchema]))
.length(characters.length)
.describe("one entry per selected character")
.length() is an exact match, not a floor. Select one character and the model must return exactly one description or the entire story is rejected.
That reads reasonable until you think about what the model is being asked to write. A four-paragraph story needs someone for the protagonist to talk to. So the model invents a supporting character, because that is what writing a story requires, and then it describes that character, because the field is called characters_description and describing characters is what the field is for. Two entries. Validation fails. Story gone.
The model did nothing wrong. It satisfied the narrative instruction and the field's own name, and those two things put it in conflict with an arithmetic constraint nobody had tested against a one-character selection.
Retries cannot fix this
The pipeline retries three times on a parse failure. All three attempts on the story she reported failed identically, because the model was not being flaky. It was making the same defensible choice each time, and a retry loop wrapped around a deterministic disagreement just spends the money three times.
That distinction is worth pulling out. Retries protect against variance. They do nothing against a systematic mismatch between what you asked for and what you will accept, and a retry count in your logs makes a structural problem look like a transient one.
The part that made it a regression
Before this pipeline used schema validation, it used a hand-written salvage parser. That parser also looked at characters_description. It threw in exactly one case:
if (characters_description.length === 0) {
throw new Error("Failed to extract character description in fallback");
}
Empty was fatal. Too many was fine. Surplus entries were tolerated and ignored downstream.
So story creation did not break because the model changed, or the prompt changed, or a learner did something unusual. It broke because we replaced a tolerant parser with a stricter one, and the new strictness covered a case the old code had quietly absorbed for as long as it had existed.
Nobody made a bad decision here. Validating model output against a schema is better engineering than regex salvage, and the person who wrote the schema was making the codebase more correct, not less. The failure is that the migration was reviewed as a correctness improvement rather than as a behaviour change, and nobody diffed the two implementations for what the old one had forgiven.
The question I should have been asking
When you replace tolerant parsing with schema validation, you are not only fixing a bug class. You are also making a product decision about what to reject, and the two arrive in the same commit wearing the same clothes.
The useful question is not "is the new schema correct?" It is: what did the old path let through, and is each of those cases now a user-visible failure?
For an array field, that decomposes into things you can check:
- Does the old code care about length at all, or only about emptiness?
- If the new schema says exact, what happens on one too many? Is the extra data harmful, or merely unwanted?
- Is the count derived from user input? A constraint of
.length(n)wherencomes from a selection is a constraint whose tightness varies by user, andn = 1is both the worst case and the most common one. - Can the model satisfy the constraint and the instruction at the same time? If the prompt asks for a story and the schema caps the cast, those can conflict.
Our fix relaxes the floor and drops the surplus: .min(characters.length), then filter the array down to the selected character names before anything downstream reads it. An invented sidekick's description is not something we want to keep. It is also harmless once discarded, and discarding it beats losing the story.
When strict is right
This is not an argument for loose schemas. Exact lengths are correct when the count is a real invariant and a mismatch means the data is missing. Our paragraph array is .length(4) and that one should probably stay strict, because each paragraph becomes an illustrated page and a fifth paragraph has nowhere to go. The right handling there is to trim or regenerate, not to accept five.
The distinction is whether a violation means the response is unusable or the response contains more than you asked for. The first deserves a hard failure. The second deserves a filter. Zod will enforce either one with equal enthusiasm and will not tell you which you meant.
What actually found it
Not the error message, which named a schema path and an array length and pointed at neither the cause nor the pattern. Grouping the day's failures by a variable nobody thought was relevant is what found it. The report said grade 6 and Unicorn, so those were the two columns I checked first, and both were dead ends. Selected-character count was the third guess.
There is a habit worth forming in that. A bug report that arrives with a confident cause is carrying two different things: evidence about symptoms, which the reporter watched happen, and a hypothesis about cause, which they inferred from a sample of one or two. Those deserve different amounts of trust. Grouping every instance by several candidate variables costs one query, and it is the fastest way to learn that the obvious variable is not the one.
Her report was good. It had a URL, a login code, and a reproduction. It was also 13.9% of the story, and finding the rest took a GROUP BY.