Skip to content

Google own-goaled its zone-resilient VMware clusters across three regions — and took 12 hours to recover

Google sells stretched private clouds as protection against a complete failure of any one zone. But it messed up the network config. 

Google own-goaled its zone-resilient VMware clusters across three regions — and took 12 hours to recover
Photo by David Farkas on Unsplash

Google Cloud on Tuesday recorded an outage of 11 hours and 46 minutes for regions in Australia and Europe on a product specifically marketed as protection against zonal outages.

Customers on Google Cloud VMware Engine (GCVE) Stretched Cluster saw cluster zones fail to synchronise state after what turned out to be a faulty network configuration update.

In the end, Google said, its engineers rolled back the configuration to the last-known good value.

It did not immediately respond to questions from The Stack, including why recovery took nearly half a day.

Nor did it say how many customers were actually affected.

When the witness fails

This content is for members only

Subscribe
Add The Stack on Google