The big August 25 internet outage in Japan that Google apologized for was always a mystery. Nobody in the media explained what caused it. Yesterday I stumbled on the answer: a massive BGP (Border Gateway Protocol) leak. BGPMon has the full story and concludes:
BGP leaks continue to be a great risk to the Internet’s stability. It’s easy to make configuration mistakes that can lead to incidents like this. In this case it appears a configuration error or software problem in Google’s network led to inadvertently announcing thousands of prefixes to Verizon, who in turn propagated the leak to many of its peers… In this case it appears Verizon had little or no filters, and accepted most if not all BGP announcements from Google which lead to widespread service disruptions.
The leak was an 8 minute ‘oops’ for Google, a 30 minute ‘oh no’ for Verizon and an 8 hour ‘oh shit’ for all of Japan.
It is weird, and scary, that Google and Verizon can quickly halt major and important web services in Japan. But that is the internet and the world we live in now.