Failover That Loses State Isn't Failover

In December 2025, IsDown tracked 47 incidents across major AI providers in a single month. Anthropic logged 20 of them, totalling 184.5 hours of impact. OpenAI logged 22, totalling 182.7 hours. In April 2026, a ten-hour Claude outage stalled enterprise workloads worldwide, and OpenAI’s API went down for hours two weeks later.

If you run anything on a single provider, you have already had this conversation with someone. The fix is well understood, the arithmetic is genuinely compelling, and most teams implement it in an afternoon.

Then it fires in production and users tell you the assistant lost its mind.

The arithmetic that makes failover look solved

Two providers, each at 99.53% availability, combine to above 99.99% with automatic failover. Expected annual downtime falls from about 41 hours to about 12 minutes.

That number is real. It is also the reason failover gets marked done prematurely, because it measures exactly one thing: did a request come back. It says nothing about whether the response made sense given the fourteen turns that preceded it.

Availability is a transport metric. Conversations are stateful. The gap between those two facts is where the actual engineering lives.

What a naive switch throws away

Research on multi-provider routing found that stateless failover successfully maintains uptime while silently discarding conversation history. Silently is the operative word — nothing errors, the request succeeds, and the degradation surfaces as a confused user rather than a page.

Four things do not survive a naive switch.

Conversation state that lived on the provider side. If you used a provider-managed thread, assistant, or session abstraction, that state does not exist on the other provider. You are not failing over; you are starting a new conversation wearing the old one’s UI.

In-flight tool calls. The model asked to call get_account_balance, you executed it, and you are mid-way through returning the result when the provider drops. The second provider has no record of having asked. Replay it naively and you either double-execute a side effect or return a tool result for a call that, from the new model’s perspective, never happened.

Streaming position. You have already sent 400 tokens to the client over SSE. The failover produces a different completion for the same prompt, because it is a different model. Do you discard what the user already read, or splice two incompatible halves together? Both are bad. You have to decide which is less bad before it happens at 3am.

The output contract. Two models rarely produce byte-identical structured output for the same schema. If a downstream parser expects a particular shape, your failover path has to satisfy the same contract as your primary — which means validating output at the boundary, not trusting the provider.

What stateful failover actually requires

The principle is simple to state and annoying to implement: your application must own enough state to reconstruct the request against any provider, and must treat provider responses as untrusted until validated.

Concretely, four properties.

Own the transcript. Keep the full message history in your own store, in a provider-neutral format, and rebuild the request from it on every call. This costs you the convenience of provider-side thread management. It buys you the ability to move.

Make tool execution idempotent and logged. Every tool call gets an ID that you generate, an execution record, and a result you can replay. On failover you reconcile against that log rather than re-asking the model what it was doing.

Buffer the stream until commitment. Hold a short window of tokens before flushing to the client. If the provider dies inside the window, you can switch cleanly and the user sees one coherent response. Past the window you have committed, and the correct behaviour is to finish degraded rather than contradict yourself — a visible seam beats a confident non-sequitur.

Validate output at the boundary. Parse and schema-check every response regardless of which provider produced it. A failover that returns malformed JSON your primary would never have produced is a failover that moved the outage downstream.

Where to put the seam

There is a real architectural decision here, and the common answer is wrong.

Gateways — the routing layer in front of your providers — are genuinely good at transport. Retries, timeouts, key rotation, load balancing, cost-based routing to a cheaper model for simple requests. Put all of that in the gateway. It is undifferentiated work and someone else has already done it well.

What does not belong there is conversation state, tool reconciliation, and the output contract. A gateway sees a request and a response. It cannot know that turn eleven referenced a document retrieved in turn four, or that the pending tool call has a side effect that must not run twice. Only your application knows what a coherent continuation looks like for your workflow.

The split that works:

gateway        →  transport   →  retries, timeouts, keys, cost routing
application    →  semantics   →  transcript, tool log, stream buffer, schema

Teams that push semantics into the gateway end up with excellent uptime numbers and users who do not trust the product.

Routing is not only for outages

Once the seam exists, it pays for itself in ways that have nothing to do with reliability.

You can route by request shape — a classification call does not need your most expensive model. You can route by tenant, so one customer’s burst does not exhaust the key another customer depends on. You can A/B a new model against production traffic without a deploy. You can absorb a provider’s pricing change in an afternoon.

None of that is available to a team wired directly into one SDK. The failover work is the entry fee for the optionality, and the optionality is usually worth more than the uptime.

What to measure

Error rate will tell you the failover fired. It will not tell you whether it worked.

Track failover events as a first-class metric, with the reason. Track semantic divergence: sample conversations that crossed a provider boundary and compare them against ones that did not, on whatever quality signal you already trust. Track tool call reconciliation failures separately from request errors, because they indicate correctness bugs rather than availability ones. And track how often the buffer window saved you, because that number tells you whether the window is sized right.

Then force it. Deliberately fail a small percentage of production traffic to the secondary on a schedule. A failover path you have never exercised is not a failover path — it is an untested branch that will run for the first time during an incident, which is the worst possible time to discover that the tool log schema drifted three releases ago.

---

Sources: IsDown AI status report, ContinuityBench: stateful failover in multi-provider LLM routing, Redis on LLM router architecture.