On this page · 11 sections
- What happened on 20 October, in order
- The root cause
- How a zone failure became a regional problem
- What Google says it will fix, and by when
- Not the first: europe-west4-a in July
- Design lessons for cloud teams
- Worked example: what surviving a zone loss costs
- UK, EU and India notes
- Lessons for teams that run their own facilities
- How eCorpIT can help
- References
Summary. At 06:53 BST on 20 October 2026, an underground cable fault on the utility network cut feed A to the facility that hosts europe-west2-b, one of the three zones in Google Cloud's London region. Feed B's breaker was already open for repairs after problems found in a quarterly generator load test on 14 October, so the site was running on one feed. The generators should have started and did not. Google's incident report, last updated on 4 November, blames a voltage-sensing design flaw and a PLC misconfiguration: the controllers read live voltage upstream of feed B's open breaker and treated it as a healthy supply. The UPS batteries, rated for about five minutes, ran down and the data halls lost power at 07:00. Power was back by 08:20, but storage recovery kept the main impact going for 6 hours 8 minutes, to 13:08 BST, across 40 listed services. Google names two reasons the damage spread past one zone: demand surges in zones a and c, and control-plane dependencies across zone boundaries. The return to normal feeds is due on 17 November. Our advice: size regional deployments to survive a zone loss without autoscaling, check where stateful replicas sit, and run tier-1 services in a second region.
About this article. Written for CTOs, SRE and platform teams that run workloads on Google Cloud, in London or elsewhere. On 11 November 2026 we checked every time, number and action date against Google's incident report on the Google Cloud Service Health page (version of 4 November 2026), Google's July 2026 europe-west4 report, Google's public incident history and Google Cloud product documentation. Customer names come from press reports and are marked as such. eCorpIT has no inside knowledge of the facility, Google may revise its report, and no vendor paid for inclusion. Researched with AI assistance and reviewed by our editorial team.
What happened on 20 October, in order
Google's report gives every time in US/Pacific with BST alongside. The table uses BST, the local time in London on that date, and every row comes from the 4 November version of the report.
| Time (BST, 20 Oct 2026) | Event, per Google's report |
|---|---|
| 06:53 | Utility feed A lost to an external underground cable fault; generators get no start signal; UPS batteries carry the load |
| 07:00 | Impact starts: batteries exhausted; hosts, local storage and network aggregation in europe-west2-b lose power |
| 07:04 | Engineers alerted by automated monitoring and probers |
| 07:09 | Regional traffic routing and client applications start failing over to other zones |
| 07:54 | Standby utility supply restored; data hall restoration begins |
| 08:20 | Utility power re-established to the data halls |
| 08:40 | Clusters begin to recover |
| 08:45 | Network devices back online |
| 08:50 | Storage devices that shut down uncleanly begin recovery; responders hold back services that would make it worse |
| 11:13 | Power redundancy verified; storage restored enough to prepare traffic re-admission |
| 11:45 | Gradual, controlled traffic re-admission to europe-west2-b |
| 12:11 | Services and APIs available; some Compute Engine VMs still down |
| 12:29 | Regional services and compute management fully undrained |
| 13:08 | Compute Engine VMs 99.5% healthy; new VM creation fully supported |
| 15:03 | Closure notice posted after a small tail of workloads recovered |
Two gaps stand out. Only seven minutes passed between losing feed A and the start of impact, close to the roughly five minutes of battery runtime the report gives. And power was back by 08:20, yet most services returned only after 12:00. The longest phase was storage recovery after the unclean shutdown, which Google chose to finish before re-admitting traffic so that it would not lose data.
Public communication lagged the internal alert. The first post on the incident page is stamped 01:35 US/Pacific, or 09:35 BST, about two and a half hours after impact began, and it offered no workaround. Later posts suggested that affected customers move workloads to another zone or region where feasible. The Register reported service interruptions at organisations including Lloyds, HSBC, Royal Mail and Revolut.
The root cause
The design of the site is conventional. Google describes a 2N utility architecture, with a complete duplicate power path for every critical load, backed by a central high-voltage generator bank under a dual-redundant programmable logic controller (PLC). Two independent utility feeders, A and B, supply the site, with a standby utility connection for manual campus-wide transfer during maintenance or emergencies. UPS batteries give about five minutes of runtime, meant to cover generator start, synchronisation and load transfer.
The site was not in its normal state that morning. Feed B's breaker had been opened for corrective repairs after issues found in the 14 October load test, leaving the facility in a temporary single-feed (1N) maintenance posture with feed A carrying the whole critical load. At 06:53 BST, a cable fault on the utility network outside the site took out feed A.
The generators never received the signal to start. Google's investigation found that the PLC logic did not check both an undervoltage condition and the physical position of the breaker on each feeder. Its undervoltage sensors were wired on the line side of the feeder breakers, upstream of them, and the logic only checked for voltage there before commanding a generator start. Feed B was still live upstream of its open breaker. So the active and standby master PLCs, running identical logic and reading shared upstream sensors, both concluded that a healthy utility source was available although the distribution feeds were dead, and neither issued a start command.
The onsite team tried a remote load transfer through the PLC, but the system blocked it because of the same live reading on feed B. With batteries running out, the team moved the site to the standby utility connection, which restored power after the data halls had already gone dark. Once power was back, the facility team and the equipment makers reconfigured the PLC to check both undervoltage and breaker status.
Why two controllers did not help
Three lessons sit inside that account, and they apply to any facility.
Redundancy without diversity is one failure, twice. A second PLC protects against hardware failure of the first. It does nothing against a logic error both share, especially when both read the same sensors.
The sensor measured the wrong thing. Voltage on the line side of an open breaker shows that the utility is alive, not that the load is being fed. Google's fix adds breaker position to the check.
Maintenance states are risk windows. The flaw mattered only because feed B was open. Routine testing in the normal 2N configuration would not have exposed it. That is why Google froze planned maintenance at sites with similar topologies and added elevated review for configuration changes outside standard designs.
How a zone failure became a regional problem
Losing zonal resources in the failed zone is expected. Google's documentation tells customers to treat a zone as a single failure domain and to spread applications across zones. The report accepts that losing zone-scoped resources in europe-west2-b was within design. The surprise was impact in europe-west2-a and europe-west2-c, which Google puts down to two causes: instantaneous demand surges on the neighbouring zones, and control-plane dependencies across zone boundaries, where regional services timed out calling metadata and catalogue components in the offline zone.
The product sections of the report show how those causes played out. Every row below comes from the 4 November report.
| Service | What happened | Google's stated next step |
|---|---|---|
| Compute Engine and Persistent Disk | Some customer failovers failed on capacity constraints and storage replication difficulty; over 99% of VMs healthy by 13:08 BST | Wider disaster recovery testing for full cluster power loss; new failure signals for Regional Persistent Disk |
| GKE | Incident lasted 7 hours 34 minutes; zonal clusters in -b fully unavailable 07:05 to 12:30 BST; regional scale-ups limited by capacity and API failures | No product-specific step listed |
| Cloud SQL | HA instances failed over; those hit by Regional Persistent Disk failures mostly recovered by 09:00 BST, the last by 19:00 BST | Faster recovery for HA instances on Regional Persistent Disk |
| AlloyDB | Regional primaries waited up to 95 minutes for automated failover to start; about 4% stalled and needed manual recovery | Monitoring that reliably detects zonal outages |
| BigQuery | Job latency, queuing and 503 errors 07:03 to 12:12 BST, from an autoscaling dependency in a critical data plane job; about 1% Job API errors for customers not assigned to the zone | Autoscaling dependency removed; stronger job server isolation |
| Apigee X | Elevated 5xx errors for 7 hours 19 minutes; a scheduling misconfiguration had put two of three datastore replicas in the failed zone, so quorum was lost | Enforce three replicas in three zones; alert on replica drift |
| Looker | About 45% of instances unresponsive at the peak; baseline restored by 13:20 BST | Better zonal outage telemetry and middleware failover |
| Memorystore for Redis | Basic Tier down; Standard Tier failed over; some instances stuck in a repairing state, all fixed within 24 hours | Detect and reconcile control plane and data plane drift |
| Cloud Bigtable | Behaved as expected; multi-cluster instances with HA routing failed over | None needed |
| Dataflow | New batch jobs routed away within about one minute; no data loss | Fix a zone-selection issue that sent a few new jobs to the dead zone |
The pattern is clear. Services that already held spare capacity and replicas in other zones did what they were designed to do: Bigtable's multi-cluster routing, for example, sends traffic to the nearest available cluster when one fails. Services that had to create capacity, call a control plane in the dead zone, or recognise an unusual failure sequence struggled. Google says the sequence in europe-west2-b was atypical of earlier zonal outages, and it is adding signals so that Regional Persistent Disk can tell this case apart from a typical one.
What Google says it will fix, and by when
The 4 November report lists data centre actions with dates. Several moved: the 27 October version expected the cable repair by 30 October, and the return to normal feeds and the PLC code review by 31 October.
| Action | Status in the 4 November report |
|---|---|
| Utility locates the damaged cable | Completed 26 October |
| Utility repairs the cable and restores feed A | Estimated 13 November |
| Site transfers from the standby utility back to feeds A and B | Estimated 17 November |
| Generator load transfer tests with automatic start through the PLC | Completed 20 and 26 October |
| Full generator load transfer tests on restored feeds A and B | Estimated 17 November |
| PLC reconfigured and validated | Completed 26 October |
| Full PLC code review with the facility vendor | Estimated 6 November |
| PLC telemetry validated once feeds A and B are restored | Estimated 17 November |
| PLC configurations checked at sites with similar topology and PLC maker | Estimated 15 November |
| Emergency operating procedure updated and practised twice per shift | Completed 24 and 26 October |
| PLC code reviews during commissioning, before Facility Ready, re-checked within a quarter | Estimated 18 December |
| Planned maintenance frozen at similar sites; elevated review of non-standard changes | Completed 30 October |
On 11 November the report had not been updated since 4 November, so it does not say whether the 6 November code review finished. On the cloud side, Google commits to an automated link from internal alerts to customer communication for incidents that affect a location, and to wider platform testing for infrastructure-level events. Our reading of the dates is that the London facility stays on its standby utility connection until the planned 17 November transfer, with the PLC fix already tested. If you run critical workloads in europe-west2-b, the days from 13 to 17 November are worth watching.
Not the first: europe-west4-a in July
Google's report on 15 July 2026 describes a similar chain in the Netherlands. A 3-millisecond voltage drop on the utility supply tripped the breakers on both feeds at a site serving europe-west4-a. The diesel rotary UPS on side A failed to take the load because of electrical component failures, one row then failed to transfer to feed B, and a chiller controller dropped offline. The redundant cooling source was out of service because of known construction work at the site. Data hall temperatures reached 44°C, and Google Cloud VMware Engine, Bare Metal Solution and NetApp Volumes were disrupted for a total of 14 hours 55 minutes. Google's public incident history lists no other power-related European zone failure between the two events.
The two events share a shape. Both began with a utility fault outside Google's control. Both turned serious because a control system, a PLC in London and a chiller controller in the Netherlands, did not do its job. And both happened while planned work had already removed a layer of redundancy. The practical reading for customers: treat a zone as a unit that can fail for hours, and expect the risk to be higher while the provider is doing maintenance.
Design lessons for cloud teams
These are our recommendations, drawn from the report. None would have stopped the power failure. All of them limit what a zone failure does to you.
- Size for zone loss without scaling. Failovers failed on capacity constraints, and GKE scale-ups were limited by capacity. Autoscaling into surviving zones competes with every other customer doing the same at the same moment. Pre-provision enough capacity in the remaining zones, or hold Compute Engine reservations, which Google says give a high level of assurance of capacity and are billed at on-demand rates while they exist.
- Remove control-plane calls from the failover path. If recovery needs an API call, a new VM or a fresh node, it depends on the busiest systems in the region at the worst time. Prefer designs where the surviving zones are already serving traffic.
- Check replica placement yourself. Apigee X was built to survive a zone loss and still lost quorum because two of three replicas sat in one zone. In your own clusters, use Kubernetes topology spread constraints and verify where etcd, Kafka, Elasticsearch or database replicas actually run.
- Use regional managed services. Regional GKE clusters replicate the control plane across zones, so a single zone loss does not take the control plane down. Zonal GKE clusters in -b were unavailable for more than five hours. Cloud SQL HA writes every transaction synchronously to disks in two zones before confirming the commit. Keep basic, zonal and single-node tiers for development.
- Do not assume synchronous storage replication fails over cleanly. Regional Persistent Disk replicates between two zones and is designed to survive one zonal failure, yet the unusual failure sequence delayed Cloud SQL and AlloyDB recovery. For tier-1 data, add a cross-region replica and a tested promotion runbook.
- Run tier-1 services in two regions. Google found no impact outside europe-west2, so a second region was the one place this incident did not reach. For serverless workloads, our Cloud Run cross-region failover guide shows the setup, and our multi-region resilience and DR service covers recovery time and recovery point choices.
- Detect outages yourself. The first public post came about two and a half hours after impact began. External probes and synthetic checks from outside the region should page your on-call team within minutes.
- Rehearse a zone loss. Google's disaster recovery guidance frames requirements around three scopes of loss: a zone, a region, or several regions. Fail a zone in a game day, including its control-plane endpoints, and measure recovery against your target.
The table below is our estimate of what each tier would have seen on 20 October, not Google's data.
| Workload tier | Minimum design after this incident | What it would have meant on 20 October |
|---|---|---|
| Dev, test, batch that can wait | Single zone, backups in another region | Down for up to about 6 hours |
| Internal business apps | Regional, three zones, autoscaling allowed | Degraded while surviving zones scaled |
| Customer-facing, revenue | Regional, three zones, N+1 capacity pre-provisioned, regional data services | Brief failover, some latency from regional control-plane issues |
| Payments, health, critical APIs | Two regions, active-active or warm standby, cross-region data replica | Traffic shifted away from europe-west2 |
Worked example: what surviving a zone loss costs
Take a service that needs 60 vCPU at peak in a three-zone region. The arithmetic uses capacity units, so it holds for any instance price.
| Design | Provisioned vCPU | Capacity premium | Left after losing one zone | Share of peak served |
|---|---|---|---|---|
| A. Single zone | 60 | 0% | 0 | 0% |
| B. Three zones of 20, rely on autoscaling | 60 | 0% | 40 | 67% until scale-up succeeds |
| C. Three zones of 25 | 75 | 25% | 50 | 83% |
| D. Three zones of 30 (N+1) | 90 | 50% | 60 | 100% |
| E. Two regions, three zones of 15 each | 90 | 50% | 75 after a zone loss, 45 after a region loss | 125% or 75% |
Design B looks efficient until the surviving zones run short of capacity, which is what happened in London. Design D costs 50% more compute (90 / 60 = 1.5) and serves full peak with no scaling. Design E costs the same 50%, keeps 125% of peak through a zone loss (75 / 60) and 75% through a full region loss (45 / 60), at the price of cross-region replication and more operational work.
Now set that against an availability target. A 30-day month has 43,200 minutes. A 99.9% target allows 43.2 minutes of downtime, 99.95% allows 21.6 minutes and 99.99% allows 4.32 minutes. The London impact ran 368 minutes (6 hours 8 minutes), which is 8.5 times a 99.9% monthly budget and 85 times a 99.99% one. If you promise customers 99.95% or better, designs A and B cannot meet it through an event like this one.
UK, EU and India notes
For UK teams, the incident tests data residency plans. Google Cloud's published zone list shows one UK region, europe-west2 in London, with three zones. If your rules require data to stay in the UK, a second Google Cloud region is not available in the country, so the realistic options are a second provider or a colocation site for disaster recovery, each with its own cost and skills.
For EU teams, the July Netherlands event and this one both started with utility faults. Neither was a cloud software failure. Ask providers which of your services depend on a single facility inside a zone; in July, Google listed only three services in one Netherlands zone as affected.
For Indian teams serving UK or EU customers from europe-west2, the same lessons apply. For India-hosted workloads, Google Cloud's Mumbai (asia-south1) and Delhi (asia-south2) regions, each with three zones, allow an in-country two-region design. Our data residency architecture service explains how to keep regulated data in India while still protecting against a single-region event.
Lessons for teams that run their own facilities
If you operate a server room, colocation suite or edge site, Google's root cause is a checklist. Confirm that generator start logic checks breaker position as well as undervoltage. Confirm that redundant controllers do not share sensors and identical logic without an independent check. Test generator start in each maintenance configuration, not only the normal one, and treat any 1N posture as a change that needs extra approval and a written emergency procedure. Google's revised procedure covers manual generator restart and manual transfer to another utility source within the UPS battery runtime, and the onsite team practises it twice per shift. If your UPS gives five minutes, your manual procedure has to fit inside five minutes.
How eCorpIT can help
eCorpIT is a Gurugram-based technology organisation, founded in 2021, assessed at CMMI Level 5 and MSME certified, with senior-led engineering teams working across AWS, Microsoft and Google platforms. Our cloud computing services team reviews Google Cloud estates for zone and region resilience, checks replica placement and capacity headroom, and runs zone-loss game days that measure real recovery times. Talk to us at /contact-us/.
Last updated: 11 November 2026.