Wire a Fallback Model Before Your Credits Run Out
Quota exhaustion never happens at a convenient time -- mine hit at 1 a.m., mid-batch, on a workflow with one paid model and no backup. The boring fix you should wire before you need it, and the three ways I've seen it done wrong.
Quota exhaustion never happens at a convenient time. Mine happened at 1 a.m., in the middle of an overnight batch that was supposed to be finished by morning. The workflow leaned on one paid model, that model returned a hard 429, and everything downstream of the call just stopped. No retry, no fallback, no output -- a dead pipeline waiting for someone to wake up and notice.
The fix is boring and you should do it before you need it: give the workflow a second model to fall to, wire it to trigger on the actual failure, and label every result with the model that produced it. Here's the whole thing, and the three ways I've seen it done wrong.
Fall back on the right error, not every error
The instinct is to wrap the call in a try/except and fall back on anything. Don't. A malformed prompt or a schema violation is a bug; a timeout or a throttle is usually transient. Neither is a reason to switch models -- the first you fix, the second you retry -- and if you silently fall back on them you've hidden a bug or wasted the fallback on a request a second model wasn't going to save.
Fall back on one thing only: the model telling you the budget is gone -- insufficient quota, billing limit reached. That's different from a plain rate-limit 429, which is transient throttling: the right answer there is retry with backoff, not failover. Switch models only when retrying won't help because the budget is actually spent.
# generate.py -- try the primary model; on a budget error, fall to a second one.
# Every result carries the model that actually produced it.
PRIMARY = "primary-model"
FALLBACK = "fallback-model"
def call_model(prompt, model):
# your provider's client call; raises on error
...
def generate(prompt):
try:
return {"text": call_model(prompt, PRIMARY), "model": PRIMARY}
except QuotaError as e: # ONLY budget errors land here
log.warning("primary out of budget (%s); switching to %s", e.code, FALLBACK)
return {"text": call_model(prompt, FALLBACK), "model": FALLBACK}
# anything else propagates -- a bad request is a bug, not a budget problem
Two things carry this. QuotaError is narrow on purpose -- map only your provider's quota and billing-exhaustion codes to it, keep transient throttles and timeouts on a separate retry-with-backoff path, and let real bugs raise. And if the fallback also returns a quota error, the inner call raises and nothing catches it, so the run fails loudly instead of looping. Both models out of budget is a real problem; you want to hear about it, not paper over it.
schema error, bad request -> a bug raise -- fixing it is the job
timeout, plain 429 -> transient retry with backoff, same model
insufficient quota, billing -> budget gone switch to the fallback model
fallback raises it too -> both dry nothing catches it -- the run dies loudly
Label every result with the model that produced it
This is the line people skip, and it's the one that saves you: "model": PRIMARY on the way out.
The moment a fallback exists, your output is a mix of two models, and you usually can't tell which produced what after the fact. If you're generating content, that's a quality inconsistency you can't trace. If you're running an experiment, it's worse -- you've silently changed a variable mid-run, and unless the records are labeled you can't segment it back out. Stamping the model on each record -- one field, maybe a schema migration -- turns "the output is a mystery blend" into "filter by model and see."
So the fallback isn't just "keep running." It's "keep running, and keep the results honest about how they were produced."
Exercise the failure path before you trust it
You can't trust a fallback path you've never exercised. The failure mode is specific -- a particular error code, arriving mid-stream -- and the only way to know your except catches it is to make it fire once, in a test. Force the quota error (a stub that raises your QuotaError, or a throwaway key with no credit) and watch the run switch models and keep going. Do this before the overnight batch, not during it.
The parts that bit me
The failure arrives mid-run. Credits don't run out politely between jobs; they run out on call number 4,000 of 6,000. That's why the fallback has to be per-call, not a restart-the-whole-thing step. The wrapper above switches on the failing call and the batch keeps moving.
The fallback isn't free either. You've moved spend, not eliminated it -- the second model bills too, sometimes at a different rate, and may behave differently enough to matter. That's fine; it's the difference between a finished run and a dead one. Just don't treat the mixed output as a clean single-model dataset. The label is what keeps you honest about that.
"Newer" isn't automatically "compatible." When you pick the fallback, check that your prompts and parsing survive on it -- response format, token limits, stop behavior. A fallback that returns something your downstream code can't parse is just a slower way to fail.
The takeaway
If a workflow depends on one paid model and runs unattended, it has a single point of failure with a clock on it. Three lines of defense, none of them clever:
- Catch the budget error specifically, and fall to a second model -- let every other error raise, so bugs stay visible.
- Stamp the model on every result -- a fallback without labels quietly corrupts your output and your experiments.
- Fire the failure path once on purpose -- an untested fallback is decoration.
When this isn't worth it
If you're sitting at the keyboard while it runs, you can often skip the fallback -- you'll see the 429 and swap the model yourself. This is for the unattended stuff: overnight batches, schedulers, agent loops, anything that has to survive a limit it hits when no one's watching. And if a mid-run model switch would invalidate the whole point of the job -- a strict single-model benchmark, say -- then don't fall back silently at all; fail loudly and rerun clean. The fallback keeps the pipeline alive; it's on you to know when a live-but-mixed result is worth more than a clean failure.