Case 01 · Site redundancy

Corvin Active-Active

“We will never become dependent on the kindness of strangers. … we will always arrange our affairs so that any requirements for cash we may conceivably have will be dwarfed by our own liquidity.”

Warren Buffett · Berkshire Hathaway letter to shareholders, 2009

Two sites, one production. Both take traffic and share every GPU. Lose one and inference retries at once, web moves in seconds, and the database promotes itself. A third site breaks ties. This is the plan; every time on this page is a target the drills will measure.

0:00 / 1:24

Status · the planThe plan, issued 8 Oct 2026. Nothing here is a measurement yet: the timings are targets, and four quarterly drills will replace them with real numbers.

01 · The claim

The claim, in four numbers

2 sites

Both serve production. Both run every app. One database writes; the other holds a live replica.

3 to 4

GPUs in the production pool, day to day. Five in a failover. A failed call retries on another card.

15–30 s

Target for the database to promote itself. Web moves in seconds. Up to ~1 s of the newest writes at risk.

2 of 3

Votes pick the primary. A witness at a third location means a cut link can't split the data in two.

02 · Transcript

Transcript

0:01One production. Two sites. Both take traffic.

0:13Today one building holds all of production: apps, data, every GPU.

0:18A power cut, an ISP outage or a hardware fault stops it all at once.

0:28The plan: both sites run every app, and both take traffic.

0:34One database writes. The other site holds a live replica.

0:39Every GPU joins one pool: four cards, up from three.

0:44Lose a site. Inference retries on another GPU at once.

0:50Health checks move web traffic within seconds.

0:55The database promotes itself. Target: 15 to 30 seconds.

1:00Two sites can't tell a dead peer from a cut link.

1:05A third site breaks the tie. Only the majority writes.

1:10Every number here is a target from the plan.

1:14Four drills each quarter will measure the real ones.

03 · Sources

Sources and honest limits

DECK Corvin active-active architecture decision, 8 Oct 2026 (internal deck; names, hosts and tickets withheld)
BRK Warren Buffett, Berkshire Hathaway letter to shareholders, 2009: https://www.berkshirehathaway.com/letters/2009ltr.pdf
PG Patroni + etcd: leader key with a time-to-live; a majority of three members elects the primary
CF Cloudflare Load Balancing: two pools, health checks, prefers the site holding the database primary

This is an architecture plan, written before any failover has happened. The 15–30 s promotion, the seconds-long web failover and the ~1 s of writes at risk are targets from the plan; nothing has been drilled yet. Replication is asynchronous on purpose: writes don't wait on the link between sites, and the price is that the newest fraction of a second may be missing after a promotion. Synchronous replication would remove that and make every commit pay a round trip. Requests in flight on the lost site fail and are retried. Host names, people and internal tickets are left out.

How we solved it.