Multi-region failover monitoring has an awkward problem: the better your failover works, the less likely your monitoring is to notice that anything broke. When traffic quietly shifts from a failed primary region to a standby, the public URL stays green, the uptime percentage stays perfect, and nobody learns that the site is running without a safety net. This article covers how to monitor a multi-region setup so failover protects users without hiding real outages from the people responsible for fixing them.
Why Failover Hides Problems From Standard Checks
A typical uptime check requests one public hostname, gets an HTTP 200, and records the site as up. In a multi-region architecture, that hostname sits behind AWS Route 53 failover routing, Cloudflare Load Balancing, Azure Front Door or Google Cloud’s global external load balancer. The check only sees the result after routing has already done its job.
That is the point of failover. It also means a dead primary region can look exactly like a healthy one from outside. The external monitor sees a working site and stays quiet, while the internal health checks that triggered the failover often report only to a dashboard nobody watches at 3 a.m.
Here is a familiar pattern. The primary region in us-east-1 loses its database writer at 02:10 on a Saturday. Route 53 health checks, running at the standard 30-second interval with a failure threshold of 3, mark the primary unhealthy and DNS starts pointing to eu-west-1. Clients that respect a 60-second TTL move over within about two to three minutes. The external uptime check records one slow response, maybe one failure, and nothing more.
On Tuesday someone notices that US users have seen response times go from 180 ms to 450 ms for three days. The secondary region has been running at 85% CPU since it was sized for 40% of normal traffic. If eu-west-1 had failed during those three days, there would have been nowhere left to fail over to.
The Myth That Failover Means Zero Downtime
The most common misconception is that a multi-region setup removes the need for careful monitoring because “the other region takes over.” In practice, failover turns a visible outage into a hidden degradation, and hidden degradations last longer.
Failover is never instant. DNS-based failover depends on resolver behavior, and some ISP resolvers and corporate proxies ignore low TTLs and keep cached records for 5-15 minutes. Long-lived connections, mobile apps holding pinned endpoints and server-to-server integrations with their own DNS caching can keep hitting the dead region well after the switch. A partial outage like this affects some users and not others, and a single global check from one location will usually miss it. For more on that pattern, see detecting partial outages that only affect some users.
Data is a separate problem. With asynchronous replication, such as Aurora Global Database or cross-region PostgreSQL streaming replicas, the secondary may be seconds or minutes behind the primary. The site is up, but recent orders, password changes or uploads may be missing. An uptime check cannot see that.
Building a Monitoring Layout That Sees Through Failover
An experienced SRE starts by separating two questions: “Can users reach the service?” and “Is each region healthy on its own?” Each question needs its own checks, and both sets need to alert.
A setup that holds up in production usually has five layers:
1. A global endpoint check against the public hostname, run from several geographic locations, to measure what users actually experience.
2. Direct per-region checks against hostnames that skip the failover layer, such as us-east.origin.example.com and eu-west.origin.example.com, so each region reports its own status.
3. A region-identity check using a response header such as X-Served-By or a keyword in a health page, so you can tell which region is serving traffic right now.
4. Per-region response time baselines, so you notice when the secondary is struggling under load it was not sized for.
5. A failover-state alert that fires when traffic moves, even if the global endpoint never returns an error.
The per-region checks are the core of the setup. If us-east-1 goes down while the global URL stays up, that should still page someone, though perhaps at a lower severity than a full outage. “Primary region down, failover active” is an incident. It is not a non-event.
The region-identity check is cheap and often skipped. Adding a small header at the load balancer or application layer takes about ten minutes. A keyword check that expects “region=us-east-1” on the health page will fail as soon as traffic moves, and that failure tells you failover has happened.
Common Mistakes With Multi-Region Monitoring
The first mistake is monitoring only the global hostname. Teams set up one check against www.example.com, see 100% uptime for the quarter and report it upward. Meanwhile, the primary region may have been down for 14 hours spread across several silent failovers. The uptime number is accurate, but it says nothing about how close the site came to a full outage.
The second mistake is letting health check endpoints lie. A /health route that returns 200 as long as the web process is running will keep a region marked healthy while its database connection pool is exhausted. Route 53 or Cloudflare then keeps sending traffic to a region that returns 500 errors on every real page. A useful health endpoint checks its critical dependencies with short timeouts, around 2-3 seconds, and does not cascade into expensive queries.
The third mistake is never testing the failover path while monitoring is watching. Many teams discover during a real incident that the secondary region’s SSL certificate expired two months ago, or that its monitor was paused after a migration and never turned back on. Scheduled game days, where you deliberately fail the primary and confirm every expected alert fires, catch this. A structured approach is covered in testing your monitoring setup before disaster strikes.
Edge Cases Where the Standard Advice Changes
Active-active architectures need a different approach from active-passive ones. When both regions serve traffic all the time, there is no single failover moment to alert on. Instead, track each region’s share of traffic and error rate. If one region suddenly carries 100% of requests instead of its usual 50%, treat that as the failover signal.
Regulated workloads add more constraints. Under GDPR data residency requirements, or in healthcare and financial setups bound to specific jurisdictions, failing EU traffic over to a US region may be a compliance incident even when it is technically a success. In those cases the region-identity check is a compliance control, not just an operational one.
Small teams, roughly fewer than 10 engineers, should be careful about alert volume. Five checks per region across three regions is fifteen monitors, and paging on every one of them leads to burnout quickly. Page on the global check and the failover-state alert. Send per-region degradations to email or chat during business hours.
FAQ
Should a region failure alert if failover worked correctly?
Yes, but at an appropriate severity. Users may not be affected, but you have lost redundancy, and the next failure becomes a full outage. Treat it as a high-priority incident to be resolved within hours, not something to ignore until Monday.
How often should per-region checks run?
One-minute intervals are a sensible default for both global and per-region checks. Route 53 also offers fast 10-second health checks for routing decisions, but external monitoring at one-minute intervals from several locations is usually enough to catch failover events and confirm recovery.
Can I rely on my cloud provider’s health checks instead of external monitoring?
Not on their own. Provider health checks decide where traffic goes, and they run inside the same infrastructure that may be failing. External checks confirm what users actually see, and they still work when the provider’s own control plane is having a bad day.
Summary
Failover is meant to protect users. It should not hide incidents from the team. Monitor the global endpoint for user experience, monitor each region directly for its own health, and add a check that tells you which region is serving traffic. Then fail the primary on purpose once a quarter and confirm that every alert you expect actually arrives.