Google Cloud europe-west2 outage: the PLC flaw behind London's 6-hour disruption

A PLC misread kept generators off and caused 6h 8m of impact in europe-west2-b; how to design so a zone loss stays a zone loss.

Read time
19 min
Word count
2.6K
Sections
11
FAQs
8
Share
Two engineers in a data centre power room, one in a hard hat at an electrical panel, under a Google Cloud London outage title
On this page · 11 sections
  1. What happened on 20 October, in order
  2. The root cause
  3. How a zone failure became a regional problem
  4. What Google says it will fix, and by when
  5. Not the first: europe-west4-a in July
  6. Design lessons for cloud teams
  7. Worked example: what surviving a zone loss costs
  8. UK, EU and India notes
  9. Lessons for teams that run their own facilities
  10. How eCorpIT can help
  11. References

Summary. At 06:53 BST on 20 October 2026, an underground cable fault on the utility network cut feed A to the facility that hosts europe-west2-b, one of the three zones in Google Cloud's London region. Feed B's breaker was already open for repairs after problems found in a quarterly generator load test on 14 October, so the site was running on one feed. The generators should have started and did not. Google's incident report, last updated on 4 November, blames a voltage-sensing design flaw and a PLC misconfiguration: the controllers read live voltage upstream of feed B's open breaker and treated it as a healthy supply. The UPS batteries, rated for about five minutes, ran down and the data halls lost power at 07:00. Power was back by 08:20, but storage recovery kept the main impact going for 6 hours 8 minutes, to 13:08 BST, across 40 listed services. Google names two reasons the damage spread past one zone: demand surges in zones a and c, and control-plane dependencies across zone boundaries. The return to normal feeds is due on 17 November. Our advice: size regional deployments to survive a zone loss without autoscaling, check where stateful replicas sit, and run tier-1 services in a second region.

About this article. Written for CTOs, SRE and platform teams that run workloads on Google Cloud, in London or elsewhere. On 11 November 2026 we checked every time, number and action date against Google's incident report on the Google Cloud Service Health page (version of 4 November 2026), Google's July 2026 europe-west4 report, Google's public incident history and Google Cloud product documentation. Customer names come from press reports and are marked as such. eCorpIT has no inside knowledge of the facility, Google may revise its report, and no vendor paid for inclusion. Researched with AI assistance and reviewed by our editorial team.

What happened on 20 October, in order

Google's report gives every time in US/Pacific with BST alongside. The table uses BST, the local time in London on that date, and every row comes from the 4 November version of the report.

Time (BST, 20 Oct 2026) Event, per Google's report
06:53 Utility feed A lost to an external underground cable fault; generators get no start signal; UPS batteries carry the load
07:00 Impact starts: batteries exhausted; hosts, local storage and network aggregation in europe-west2-b lose power
07:04 Engineers alerted by automated monitoring and probers
07:09 Regional traffic routing and client applications start failing over to other zones
07:54 Standby utility supply restored; data hall restoration begins
08:20 Utility power re-established to the data halls
08:40 Clusters begin to recover
08:45 Network devices back online
08:50 Storage devices that shut down uncleanly begin recovery; responders hold back services that would make it worse
11:13 Power redundancy verified; storage restored enough to prepare traffic re-admission
11:45 Gradual, controlled traffic re-admission to europe-west2-b
12:11 Services and APIs available; some Compute Engine VMs still down
12:29 Regional services and compute management fully undrained
13:08 Compute Engine VMs 99.5% healthy; new VM creation fully supported
15:03 Closure notice posted after a small tail of workloads recovered

Two gaps stand out. Only seven minutes passed between losing feed A and the start of impact, close to the roughly five minutes of battery runtime the report gives. And power was back by 08:20, yet most services returned only after 12:00. The longest phase was storage recovery after the unclean shutdown, which Google chose to finish before re-admitting traffic so that it would not lose data.

Public communication lagged the internal alert. The first post on the incident page is stamped 01:35 US/Pacific, or 09:35 BST, about two and a half hours after impact began, and it offered no workaround. Later posts suggested that affected customers move workloads to another zone or region where feasible. The Register reported service interruptions at organisations including Lloyds, HSBC, Royal Mail and Revolut.

The root cause

The design of the site is conventional. Google describes a 2N utility architecture, with a complete duplicate power path for every critical load, backed by a central high-voltage generator bank under a dual-redundant programmable logic controller (PLC). Two independent utility feeders, A and B, supply the site, with a standby utility connection for manual campus-wide transfer during maintenance or emergencies. UPS batteries give about five minutes of runtime, meant to cover generator start, synchronisation and load transfer.

The site was not in its normal state that morning. Feed B's breaker had been opened for corrective repairs after issues found in the 14 October load test, leaving the facility in a temporary single-feed (1N) maintenance posture with feed A carrying the whole critical load. At 06:53 BST, a cable fault on the utility network outside the site took out feed A.

The generators never received the signal to start. Google's investigation found that the PLC logic did not check both an undervoltage condition and the physical position of the breaker on each feeder. Its undervoltage sensors were wired on the line side of the feeder breakers, upstream of them, and the logic only checked for voltage there before commanding a generator start. Feed B was still live upstream of its open breaker. So the active and standby master PLCs, running identical logic and reading shared upstream sensors, both concluded that a healthy utility source was available although the distribution feeds were dead, and neither issued a start command.

The onsite team tried a remote load transfer through the PLC, but the system blocked it because of the same live reading on feed B. With batteries running out, the team moved the site to the standby utility connection, which restored power after the data halls had already gone dark. Once power was back, the facility team and the equipment makers reconfigured the PLC to check both undervoltage and breaker status.

Why two controllers did not help

Three lessons sit inside that account, and they apply to any facility.

Redundancy without diversity is one failure, twice. A second PLC protects against hardware failure of the first. It does nothing against a logic error both share, especially when both read the same sensors.

The sensor measured the wrong thing. Voltage on the line side of an open breaker shows that the utility is alive, not that the load is being fed. Google's fix adds breaker position to the check.

Maintenance states are risk windows. The flaw mattered only because feed B was open. Routine testing in the normal 2N configuration would not have exposed it. That is why Google froze planned maintenance at sites with similar topologies and added elevated review for configuration changes outside standard designs.

How a zone failure became a regional problem

Losing zonal resources in the failed zone is expected. Google's documentation tells customers to treat a zone as a single failure domain and to spread applications across zones. The report accepts that losing zone-scoped resources in europe-west2-b was within design. The surprise was impact in europe-west2-a and europe-west2-c, which Google puts down to two causes: instantaneous demand surges on the neighbouring zones, and control-plane dependencies across zone boundaries, where regional services timed out calling metadata and catalogue components in the offline zone.

The product sections of the report show how those causes played out. Every row below comes from the 4 November report.

Service What happened Google's stated next step
Compute Engine and Persistent Disk Some customer failovers failed on capacity constraints and storage replication difficulty; over 99% of VMs healthy by 13:08 BST Wider disaster recovery testing for full cluster power loss; new failure signals for Regional Persistent Disk
GKE Incident lasted 7 hours 34 minutes; zonal clusters in -b fully unavailable 07:05 to 12:30 BST; regional scale-ups limited by capacity and API failures No product-specific step listed
Cloud SQL HA instances failed over; those hit by Regional Persistent Disk failures mostly recovered by 09:00 BST, the last by 19:00 BST Faster recovery for HA instances on Regional Persistent Disk
AlloyDB Regional primaries waited up to 95 minutes for automated failover to start; about 4% stalled and needed manual recovery Monitoring that reliably detects zonal outages
BigQuery Job latency, queuing and 503 errors 07:03 to 12:12 BST, from an autoscaling dependency in a critical data plane job; about 1% Job API errors for customers not assigned to the zone Autoscaling dependency removed; stronger job server isolation
Apigee X Elevated 5xx errors for 7 hours 19 minutes; a scheduling misconfiguration had put two of three datastore replicas in the failed zone, so quorum was lost Enforce three replicas in three zones; alert on replica drift
Looker About 45% of instances unresponsive at the peak; baseline restored by 13:20 BST Better zonal outage telemetry and middleware failover
Memorystore for Redis Basic Tier down; Standard Tier failed over; some instances stuck in a repairing state, all fixed within 24 hours Detect and reconcile control plane and data plane drift
Cloud Bigtable Behaved as expected; multi-cluster instances with HA routing failed over None needed
Dataflow New batch jobs routed away within about one minute; no data loss Fix a zone-selection issue that sent a few new jobs to the dead zone

The pattern is clear. Services that already held spare capacity and replicas in other zones did what they were designed to do: Bigtable's multi-cluster routing, for example, sends traffic to the nearest available cluster when one fails. Services that had to create capacity, call a control plane in the dead zone, or recognise an unusual failure sequence struggled. Google says the sequence in europe-west2-b was atypical of earlier zonal outages, and it is adding signals so that Regional Persistent Disk can tell this case apart from a typical one.

What Google says it will fix, and by when

The 4 November report lists data centre actions with dates. Several moved: the 27 October version expected the cable repair by 30 October, and the return to normal feeds and the PLC code review by 31 October.

Action Status in the 4 November report
Utility locates the damaged cable Completed 26 October
Utility repairs the cable and restores feed A Estimated 13 November
Site transfers from the standby utility back to feeds A and B Estimated 17 November
Generator load transfer tests with automatic start through the PLC Completed 20 and 26 October
Full generator load transfer tests on restored feeds A and B Estimated 17 November
PLC reconfigured and validated Completed 26 October
Full PLC code review with the facility vendor Estimated 6 November
PLC telemetry validated once feeds A and B are restored Estimated 17 November
PLC configurations checked at sites with similar topology and PLC maker Estimated 15 November
Emergency operating procedure updated and practised twice per shift Completed 24 and 26 October
PLC code reviews during commissioning, before Facility Ready, re-checked within a quarter Estimated 18 December
Planned maintenance frozen at similar sites; elevated review of non-standard changes Completed 30 October

On 11 November the report had not been updated since 4 November, so it does not say whether the 6 November code review finished. On the cloud side, Google commits to an automated link from internal alerts to customer communication for incidents that affect a location, and to wider platform testing for infrastructure-level events. Our reading of the dates is that the London facility stays on its standby utility connection until the planned 17 November transfer, with the PLC fix already tested. If you run critical workloads in europe-west2-b, the days from 13 to 17 November are worth watching.

Not the first: europe-west4-a in July

Google's report on 15 July 2026 describes a similar chain in the Netherlands. A 3-millisecond voltage drop on the utility supply tripped the breakers on both feeds at a site serving europe-west4-a. The diesel rotary UPS on side A failed to take the load because of electrical component failures, one row then failed to transfer to feed B, and a chiller controller dropped offline. The redundant cooling source was out of service because of known construction work at the site. Data hall temperatures reached 44°C, and Google Cloud VMware Engine, Bare Metal Solution and NetApp Volumes were disrupted for a total of 14 hours 55 minutes. Google's public incident history lists no other power-related European zone failure between the two events.

The two events share a shape. Both began with a utility fault outside Google's control. Both turned serious because a control system, a PLC in London and a chiller controller in the Netherlands, did not do its job. And both happened while planned work had already removed a layer of redundancy. The practical reading for customers: treat a zone as a unit that can fail for hours, and expect the risk to be higher while the provider is doing maintenance.

Design lessons for cloud teams

These are our recommendations, drawn from the report. None would have stopped the power failure. All of them limit what a zone failure does to you.

  1. Size for zone loss without scaling. Failovers failed on capacity constraints, and GKE scale-ups were limited by capacity. Autoscaling into surviving zones competes with every other customer doing the same at the same moment. Pre-provision enough capacity in the remaining zones, or hold Compute Engine reservations, which Google says give a high level of assurance of capacity and are billed at on-demand rates while they exist.
  1. Remove control-plane calls from the failover path. If recovery needs an API call, a new VM or a fresh node, it depends on the busiest systems in the region at the worst time. Prefer designs where the surviving zones are already serving traffic.
  1. Check replica placement yourself. Apigee X was built to survive a zone loss and still lost quorum because two of three replicas sat in one zone. In your own clusters, use Kubernetes topology spread constraints and verify where etcd, Kafka, Elasticsearch or database replicas actually run.
  1. Use regional managed services. Regional GKE clusters replicate the control plane across zones, so a single zone loss does not take the control plane down. Zonal GKE clusters in -b were unavailable for more than five hours. Cloud SQL HA writes every transaction synchronously to disks in two zones before confirming the commit. Keep basic, zonal and single-node tiers for development.
  1. Do not assume synchronous storage replication fails over cleanly. Regional Persistent Disk replicates between two zones and is designed to survive one zonal failure, yet the unusual failure sequence delayed Cloud SQL and AlloyDB recovery. For tier-1 data, add a cross-region replica and a tested promotion runbook.
  1. Run tier-1 services in two regions. Google found no impact outside europe-west2, so a second region was the one place this incident did not reach. For serverless workloads, our Cloud Run cross-region failover guide shows the setup, and our multi-region resilience and DR service covers recovery time and recovery point choices.
  1. Detect outages yourself. The first public post came about two and a half hours after impact began. External probes and synthetic checks from outside the region should page your on-call team within minutes.
  1. Rehearse a zone loss. Google's disaster recovery guidance frames requirements around three scopes of loss: a zone, a region, or several regions. Fail a zone in a game day, including its control-plane endpoints, and measure recovery against your target.

The table below is our estimate of what each tier would have seen on 20 October, not Google's data.

Workload tier Minimum design after this incident What it would have meant on 20 October
Dev, test, batch that can wait Single zone, backups in another region Down for up to about 6 hours
Internal business apps Regional, three zones, autoscaling allowed Degraded while surviving zones scaled
Customer-facing, revenue Regional, three zones, N+1 capacity pre-provisioned, regional data services Brief failover, some latency from regional control-plane issues
Payments, health, critical APIs Two regions, active-active or warm standby, cross-region data replica Traffic shifted away from europe-west2

Worked example: what surviving a zone loss costs

Take a service that needs 60 vCPU at peak in a three-zone region. The arithmetic uses capacity units, so it holds for any instance price.

Design Provisioned vCPU Capacity premium Left after losing one zone Share of peak served
A. Single zone 60 0% 0 0%
B. Three zones of 20, rely on autoscaling 60 0% 40 67% until scale-up succeeds
C. Three zones of 25 75 25% 50 83%
D. Three zones of 30 (N+1) 90 50% 60 100%
E. Two regions, three zones of 15 each 90 50% 75 after a zone loss, 45 after a region loss 125% or 75%

Design B looks efficient until the surviving zones run short of capacity, which is what happened in London. Design D costs 50% more compute (90 / 60 = 1.5) and serves full peak with no scaling. Design E costs the same 50%, keeps 125% of peak through a zone loss (75 / 60) and 75% through a full region loss (45 / 60), at the price of cross-region replication and more operational work.

Now set that against an availability target. A 30-day month has 43,200 minutes. A 99.9% target allows 43.2 minutes of downtime, 99.95% allows 21.6 minutes and 99.99% allows 4.32 minutes. The London impact ran 368 minutes (6 hours 8 minutes), which is 8.5 times a 99.9% monthly budget and 85 times a 99.99% one. If you promise customers 99.95% or better, designs A and B cannot meet it through an event like this one.

UK, EU and India notes

For UK teams, the incident tests data residency plans. Google Cloud's published zone list shows one UK region, europe-west2 in London, with three zones. If your rules require data to stay in the UK, a second Google Cloud region is not available in the country, so the realistic options are a second provider or a colocation site for disaster recovery, each with its own cost and skills.

For EU teams, the July Netherlands event and this one both started with utility faults. Neither was a cloud software failure. Ask providers which of your services depend on a single facility inside a zone; in July, Google listed only three services in one Netherlands zone as affected.

For Indian teams serving UK or EU customers from europe-west2, the same lessons apply. For India-hosted workloads, Google Cloud's Mumbai (asia-south1) and Delhi (asia-south2) regions, each with three zones, allow an in-country two-region design. Our data residency architecture service explains how to keep regulated data in India while still protecting against a single-region event.

Lessons for teams that run their own facilities

If you operate a server room, colocation suite or edge site, Google's root cause is a checklist. Confirm that generator start logic checks breaker position as well as undervoltage. Confirm that redundant controllers do not share sensors and identical logic without an independent check. Test generator start in each maintenance configuration, not only the normal one, and treat any 1N posture as a change that needs extra approval and a written emergency procedure. Google's revised procedure covers manual generator restart and manual transfer to another utility source within the UPS battery runtime, and the onsite team practises it twice per shift. If your UPS gives five minutes, your manual procedure has to fit inside five minutes.

How eCorpIT can help

eCorpIT is a Gurugram-based technology organisation, founded in 2021, assessed at CMMI Level 5 and MSME certified, with senior-led engineering teams working across AWS, Microsoft and Google platforms. Our cloud computing services team reviews Google Cloud estates for zone and region resilience, checks replica placement and capacity headroom, and runs zone-loss game days that measure real recovery times. Talk to us at /contact-us/.

Last updated: 11 November 2026.

References

  1. Multiple services were experiencing issues in the europe-west2 region, incident report of 4 November 2026 — Google Cloud Service Health
  1. When datacenters lose power, generators should kick in. That didn't happen for Google in London, 5 November 2026 — The Register
  1. Geography and regions — Google Cloud documentation
  1. Routing options, Bigtable — Google Cloud documentation
  1. Incident history in JSON — Google Cloud Service Health
  1. VMware Engine, NetApp Volumes and Bare Metal Solution outage in europe-west4-a due to a cooling failure, incident report of 25 July 2026 — Google Cloud Service Health
  1. About reservations, Compute Engine — Google Cloud documentation
  1. Pod Topology Spread Constraints — Kubernetes documentation
  1. Regional clusters, Google Kubernetes Engine — Google Cloud documentation
  1. About high availability, Cloud SQL for MySQL — Google Cloud documentation
  1. Build HA services using synchronously replicated disks, Compute Engine — Google Cloud documentation
  1. Architecting disaster recovery for cloud infrastructure outages — Google Cloud Architecture Center
  1. Regions and zones, Compute Engine — Google Cloud documentation

Frequently asked

Quick answers.

01 What caused the Google Cloud europe-west2 outage on 20 October 2026?
An external underground cable fault cut utility feed A while feed B was switched out for repairs. A PLC misconfiguration then stopped the generators from starting, because the controllers read live voltage upstream of feed B's open breaker. UPS batteries lasted about five minutes, and europe-west2-b lost power at 07:00 BST.
02 How long did the Google Cloud London outage last?
Google's report gives 6 hours 8 minutes of main impact, from 07:00 to 13:08 BST on 20 October 2026, although power itself was back by 08:20. Some services took longer: the GKE incident lasted 7 hours 34 minutes, Apigee X returned errors for 7 hours 19 minutes, and the last Cloud SQL instances recovered at 19:00 BST.
03 Which Google Cloud services were affected?
The report lists 40 services in europe-west2, including Compute Engine, Persistent Disk, GKE, Cloud SQL, AlloyDB, BigQuery, Cloud Run, Spanner, Pub/Sub, Cloud Storage, Apigee, Looker and Memorystore. Zonal resources in europe-west2-b were unavailable, while regional services saw latency, 500 and 503 errors, and queuing. Google found no impact outside the region.
04 Why were zones europe-west2-a and europe-west2-c affected?
Google names two causes: instantaneous demand surges as failover traffic and workloads moved into zones a and c, and control-plane dependencies across zone boundaries, where regional systems waited on components in the offline zone. Capacity shortages also made some customer failovers fail and slowed GKE autoscaling in the surviving zones.
05 What is a PLC and why did it matter here?
A programmable logic controller is an industrial computer that automates equipment, here the start of the generator plant and the transfer of load to it. Both redundant PLCs ran identical logic and read shared sensors wired upstream of an open breaker, so both concluded utility power was healthy and neither started the generators.
06 Would a multi-zone deployment have protected my workload?
Partly. Multi-zone designs avoided total loss, but many still saw degraded performance because the surviving zones ran short of capacity and some control planes stalled. Designs with spare capacity already running in each zone fared best. Only a second region fully avoided the incident, since Google found no impact outside europe-west2.
07 Has Google fixed the problem?
Google says it reconfigured and tested the PLC by 26 October, retested automatic generator start, updated emergency procedures and froze maintenance at similar sites. Its report gives 13 November for the cable repair, 15 November for checking similar sites and 17 November for the return to normal feeds. No update had been posted after 4 November.
08 Is this the first Google Cloud power outage in Europe this year?
No. On 15 July 2026 a utility voltage drop, a backup power failure and a chiller controller fault hit a site serving europe-west4-a in the Netherlands, disrupting VMware Engine, Bare Metal Solution and NetApp Volumes for a total of 14 hours 55 minutes. Both incidents happened while planned work had reduced redundancy.

About the author

Manu Shukla

Founder & Director

Founder of eCorpIT. Hands-on engineer leading senior-only delivery for AI apps, custom software, and cloud systems for global clients.

Subscribe

One engineering note a week. No fluff, no spam.

Senior-architect playbooks on AI agents, mobile apps, cloud, security, data, and marketing — delivered every Wednesday.

Past the reading

Read enough. Let's build something.

A senior architect responds in 24 working hours with scope, indicative cost, and a timeline. NDA before any technical conversation.

Loading search…

Browse all guides