LLM Failover in Python: Never Go Down

LLM failover in Python means routing each request through a chain of providers and keys so that a rate limit, dead key, or server error on one automatically falls through to the next. The pattern needs four parts: a key pool, retry classification, a circuit breaker per key, and cross-provider routing — all of which you can get from the open-source freelm library or build yourself.

The four parts of real failover

A retry loop alone is not failover. Robust failover distinguishes what failed and reacts differently. A 429 means rotate keys and cool the throttled one. A 5xx or timeout means back off and try elsewhere. A 401 means the key is dead — disable it, don't retry it. A 404 on a model means try a different model. Lumping these into one "retry 3 times" loop wastes attempts on errors that will never succeed and hammers providers that just told you to stop.

Classify, don't just retry

Map each HTTP status to an action before deciding what to do next:

| Failure | Meaning | Action | |---|---|---| | 429 | Rate limited | Cool this key, rotate to next | | 401 / 403 | Bad key | Disable key, fail over | | 5xx / timeout | Transient | Backoff, circuit breaker | | 404 model | Unknown model | Try next model | | 400 | Bad request | Surface — it's your bug |

This classification is the difference between a system that recovers and one that retries itself into a ban.

Circuit breakers stop you hammering dead keys

When a key fails repeatedly, keep calling it and you add latency and risk a longer ban. A circuit breaker tracks consecutive failures per key; after a threshold it "opens" and the key is skipped entirely, then "half-opens" after a cooldown to test if it recovered. This keeps a flaky provider from dragging down every request while still letting it back in once healthy.

Failover with freelm

freelm implements all four parts. You list providers and keys; it interleaves candidates across providers — best model of each provider first — so failover reaches every provider quickly rather than burning all attempts on one provider's many models.

from freelm import FreeLLM, OpenRouter, Groq, GoogleAIStudio

llm = FreeLLM(
    providers=[
        OpenRouter("sk-or-...", priority=0),   # try first
        Groq("gsk_...",        priority=1),     # then this
        GoogleAIStudio("AIza...", priority=2),  # last resort
    ],
    strategy="priority",
    max_attempts=12,
)

print(llm.text("hi"))   # transparently fails over if a provider is down

On a 429, freelm cools that key and rotates; if a model is throttled upstream it tries another model on the same key before benching it; if a key returns 401 it disables that key and keeps serving from the others. You get a single answer or a clear NoProvidersAvailable error — never a half-broken call.

Failover that survives streaming

Streaming complicates failover: you cannot switch providers after tokens have started, or you splice two answers together. freelm fails over only before the first token; once a stream produces output it commits to that provider.

for chunk in llm.stream("Write a limerick about uptime."):
    print(chunk, end="", flush=True)

Tune the failover budget

Two knobs control how hard freelm tries. max_attempts caps total tries across all providers/keys/models; timeout is both the per-request timeout and the overall deadline for one call. For latency-sensitive paths, lower both; for batch jobs where success matters more than speed, raise max_attempts and enable wait=True to pause until a cooling key recovers.

Frequently Asked Questions

What's the difference between retries and failover? Retries re-send to the same endpoint; failover routes to a different key or provider. Real resilience needs failover — retrying a rate-limited key just extends the rate limit.

How many providers do I need for reliable failover? Two or more. With one provider you can rotate keys; with several you survive an entire provider outage. freelm supports six free providers out of the box.

Does failover add latency? Only on failure. A healthy first attempt returns immediately; failover only kicks in when a provider errors, and circuit breakers keep known-bad keys out of the path.

Can I build this myself? Yes — the parts are a key pool, status classification, a circuit breaker, and a retry/backoff loop. freelm packages them so you don't re-implement and re-test the edge cases per project.

What happens if everything is down? freelm raises NoProvidersAvailable with the list of attempts, or waits for recovery if wait=True. You always get a definite result to handle.

---

Written by Shihab Shahriar Antor — AI Engineer & Founder of Shahriar Labs. The failover library shown here is freelm — open source on GitHub.