A warning sign for Japanese open loop transit operators: when the EMV payment network goes down, FeliCa networks keep working

Mobile Suica has had a rough 2 weeks. On June 24 a construction error during server center power supply expansion work left JR East Mobile Suica and Eki-Net online reservation services offline for 12 hours (0:00~12:00). It was an embarrassing mishap but the actual damage was small, limited to refunding Eki-Net ticket holders who couldn’t change ticket reservations. Mobile Suica was offline so no refunding was necessary because nobody could use the Mobile Suica credit card recharge service. No need to refund what people can’t buy.

A shorter but much more problematic outage happened on June 27. Media mistakenly reported that Mobile Suica was down but this was not the case as Mobile Suica on Android was working just fine. It was an Apple Pay problem: Apple Pay servers went down from heavy demand on Apple Pay ICOCA launch day, taking down not only Apple Pay Suica recharge but also PASMO, ICOCA, nanaco, WAON, Octopus, China T-Union, adding credit cards and other Wallet services worldwide. As the outage took place during the Japanese business day, JR East had to refund iOS Suica App users who attempted to buy or use Suica Green Car tickets during the Apple Pay outage.

Just as things were settling down, another even shorter 40 minute period of trouble occurred on July 8 at 12:00~12:40 JST. Again the media reported that Mobile Suica was down, again they were mistaken, and again it wasn’t an JR East or Mobile Suica problem, it was a much wider, and unreported, EMV credit card payment network outage. EMV transactions on readers everywhere were not responding, and they were not working for Apple Pay or Google Pay. However FeliCa payment network cards were working.

And finally there was, yet another, Apple Pay and Wallet outage on July 14 from 17:45 to 18:30 JST, with another round of Japanese media bashing poor old Mobile Suica without checking for the wider Apple Pay outage.

Mobile Suica caught the media blame because they were the only company duly reporting the problems on Mobile Suica support SNS services. JR East never lays an outage blame on Apple Pay, or any other service partner because they know Mobile Suica users don’t care, they only want to know when things are not working and when they will be fixed. This is the way it should be done because they are giving their users fast, accurate, service information…even if that means they have to take the media and SNS blame that comes with it.

But despite all the Mobile Suica outages including the EMV payment network one, the Suica card itself always remained working, both digital or plastic versions. As long as there is money on the card it works for transit and payments, and cash recharge is available 24/7. This is an under appreciated but very important aspect of the Transit IC system: there is always a non-network fail safe cash backup. Japanese never put all their household finances in one basket, cash is always the one thing that works after an earthquake, typhoon, natural or manmade infrastructure damaging disaster strikes.

In the EMV credit card payment network outage there was, without doubt, unreported trouble with open loop system test deployments on Nankai, Fukuoka Metro and other QUADRACstera transit operated systems, which all open loop systems in Japan use: it’s the only open loop player in town.

Unlike Mobile Suica however, when the credit card payment processing network goes down, open loop doesn’t have a fail safe cash backup. And while that’s not a problem now with small installation test sites and a tiny user base, it will be when open loop goes big time. The transit companies deploying open loop have an obligation to take care of their customers, but will they take JR East-like responsibility when QUADRAC goes down, or stera goes down, or NTT Data CAFIS, payment processing centers, or mobile carrier networks? Because believe me they will. All highly connected, interdependent networks do. That’s why we always need alternative methods and networks. Too bad that VISA is working to remove the non-EMV transit gate competition in Japan.