A warning sign for Japanese open loop transit operators: when the EMV payment network goes down, FeliCa networks keep working

Mobile Suica has had a rough 2 weeks. On June 24 a construction error during server center power supply expansion work left JR East Mobile Suica and Eki-Net online reservation services offline for 12 hours (0:00~12:00). It was an embarrassing mishap but the actual damage was small, limited to refunding Eki-Net ticket holders who couldn’t change ticket reservations. Mobile Suica was offline so no refunding was necessary because nobody could use the Mobile Suica credit card recharge service. No need to refund what people can’t buy.

A shorter but much more problematic outage happened on June 27. Media mistakenly reported that Mobile Suica was down but this was not the case as Mobile Suica on Android was working just fine. It was an Apple Pay problem: Apple Pay servers went down from heavy demand on Apple Pay ICOCA launch day, taking down not only Apple Pay Suica recharge but also PASMO, ICOCA, nanaco, WAON, Octopus, China T-Union, adding credit cards and other Wallet services worldwide. As the outage took place during the Japanese business day, JR East had to refund iOS Suica App users who attempted to buy or use Suica Green Car tickets during the Apple Pay outage.

Just as things were settling down, another even shorter 40 minute period of trouble occurred on July 8 at 12:00~12:40 JST. Again the media reported that Mobile Suica was down, again they were mistaken, and again it wasn’t an JR East or Mobile Suica problem, it was a much wider, and unreported, EMV credit card payment network outage. EMV transactions on readers everywhere were not responding, and they were not working for Apple Pay or Google Pay. However FeliCa payment network cards were working.

And finally there was, yet another, Apple Pay and Wallet outage on July 14 from 17:45 to 18:30 JST, with another round of Japanese media bashing poor old Mobile Suica without checking for the wider Apple Pay outage.

Mobile Suica caught the media blame because they were the only company duly reporting the problems on Mobile Suica support SNS services. JR East never lays an outage blame on Apple Pay, or any other service partner because they know Mobile Suica users don’t care, they only want to know when things are not working and when they will be fixed. This is the way it should be done because they are giving their users fast, accurate, service information…even if that means they have to take the media and SNS blame that comes with it.

But despite all the Mobile Suica outages including the EMV payment network one, the Suica card itself always remained working, both digital or plastic versions. As long as there is money on the card it works for transit and payments, and cash recharge is available 24/7. This is an under appreciated but very important aspect of the Transit IC system: there is always a non-network fail safe cash backup. Japanese never put all their household finances in one basket, cash is always the one thing that works after an earthquake, typhoon, natural or manmade infrastructure damaging disaster strikes.

In the EMV credit card payment network outage there was, without doubt, unreported trouble with open loop system test deployments on Nankai, Fukuoka Metro and other QUADRACstera transit operated systems, which all open loop systems in Japan use: it’s the only open loop player in town.

Unlike Mobile Suica however, when the credit card payment processing network goes down, open loop doesn’t have a fail safe cash backup. And while that’s not a problem now with small installation test sites and a tiny user base, it will be when open loop goes big time. The transit companies deploying open loop have an obligation to take care of their customers, but will they take JR East-like responsibility when QUADRAC goes down, or stera goes down, or NTT Data CAFIS, payment processing centers, or mobile carrier networks? Because believe me they will. All highly connected, interdependent networks do. That’s why we always need alternative methods and networks. Too bad that VISA is working to remove the non-EMV transit gate competition in Japan.

The Wireless Android POS/NFC Reader Suck Index

You too can have the whole transaction world in your hands with the Android based Square Terminal for just ¥46,980

Now that contactless is everywhere, wireless contactless readers have become very fashionable and popular. Nobody wants wires or checkout lines. All of these systems are built around an Android based reading device connected to the internet payment service via Bluetooth, WiFi or 4G with a main terminal, an iPad or a laptop running POS software. Convenient though they may be, compared with hard wired NFC reader performance they all suck with different levels of suckiness:

  1. stera: this lovely little ‘NFC antenna under the screen’ piece of shit from SMBC, GMO and Visa Japan is so slow that checkout staff put their hand over the stera screen/reader to keep customers waiting until the device is ready to go. This is followed by the instruction ‘don’t move your device until the reader beeps.’ It’s a 2~4 second wait until it beeps. This is 2014 era ‘you’re holding it wrong’ garbage nonsense. I teased one store manager about the hard wired JREM FeliCa readers that were swapped out with stera, “Those were too fast,” he said. Too fast?!
  2. PAYGATE: Another payment provider associated with GMO, slightly faster than stera but still slow, PAYGATE does’t like Apple Pay Suica•PASMO Express Transit very much. Have of the time it ignores it altogether forcing customers into the 2016 era ‘manually bring up Apple Pay Suica’ authenticate and pay maneuver. Another ‘you’re holding/doing it wrong,’ when the fault is on the checkout system side. Passé and totally unnecessary.
  3. AirPay: It’s weird that the cheap AirPay hardware performs better than PAYGATE or stera, it’s even weirder that AirPay performs better than Rakuten Pay which uses the very same reader but is stera shitshow slow.
  4. Square Terminal has gotten lots of media attention in Japan. Too early to experience it in the field yet but I’m not hopeful. Square Terminal is Android based after all and the NCF antenna under the screen design is the worst performing reader design out there. As one Brazilian reader wrote: “I just don’t like the ones running Android because at least here the software is less reliable and I managed to crash a few one by just taping my phone.”

Yep, that observation matches my experience. Payment network providers need better Android readers, the current crop is too slow getting the payment transaction ready to tap. In this era of endless subcontractor layers in the development process, creating a fast reliable Android based NFC wireless reader might be a tall order, if not impossible. The all over the place wireless NFC reader experience certainly doesn’t boast well for open loop advocates.

UPDATE
I ran across another crappy reader experience (above) and retweeted it. A reader had some questions about it, answered here by an anonymous expert. It basically comes down to poorly executed reader polling or not following Sony polling recommendations for FeliCa cards. This is what is happening in the above retweet. It is also what is going on with PAYGATE Station readers, half of the time the proper code hasn’t loaded correctly although this issue seems to be fixed in new PAYGATE Station checkout installations. Which brings us to the point I was trying to make: these performance issues can be fixed with reader firmware updates or transaction system software updates, but never are.

Wildcard polling involves the reader making a request for system code 0xFFFF and expecting the card/device to list all the system codes that it supports. Wildcard polling won’t work on an Apple Pay device in Express Transit mode – instead, the system code must be explicitly polled for (0x0003 for CJRC, 0x8008 for Octopus). You can cause Suica/Octopus to be automatically selected by sending SENSF_REQ (Polling command, 06) for those services explicitly.

I have verified that doing so with Apple Pay will cause the emulated card to be switched out as appropriate – the IDm value will also change, since Apple Pay emulates each card separately, instead of with a common IDm as with Osaifu Keitai. If you read the Sony documentation, you will see that developers are cautioned to also poll for the specific service codes they want to access if there’s no response to a wildcard poll.

Perhaps your reader doesn’t do this, but it’s fairly big omission…it should be doing explicit polling. Simply polling for service code 0x0003 should wake up Suica if selected as an Express Transit candidate, even if you don’t send any other commands. I’ve verified this with an RC-S380 reader and NFCPay.

VISA Touch Transit Boutiques

VISA Japan announced VISA Touch support on Kyoto Tango Railway, most media outlets simply ran edited versions of the VISA Japan press release. The infrastructure uses a slightly modified stera store reader platform, which VISA has a hand in. Willer Inc. is hailing this as the first ‘VISA Touch’ solution on Japanese train systems but they ignored the fact that stera VISA Touch was already announced for Ibaraki Transit highway bus service back in July.

Japanese journalist comments on Twitter were fun to read with the ‘let’s just dump FeliCa and Suica already and go all in with EMV’ supporters club checking in as usual. Nobi Hayashi asked good questions regarding real user convenience. Junya Suzuki said he plans a trip to investigate the new service, his next ‘Pay Attention’ column promises to be a good read.

Just what kind of end user are these VISA Touch transit installations targeting anyway? Let’s do a quick profile:

  1. VISA Touch JP plastic cards are being issued in Japan but they are new and few and dwarfed by the number of Transit IC cards (Suica, PASMO, ICOCA, etc.) that can be bought by anybody at any station kiosk machine with cash. Apple Pay Japan users cannot use it because VISA JP refuses to support Apple Pay JP FeliCa/EMV dual mode NFC switching. This service is not targeted for domestic transit users.
  2. Both of these VISA Touch installation transit areas market heavily to inbound tourists, neither of them support Transit IC cards.
  3. VISA Touch is not compatible with PBOC Union Pay cards technology, the installations also support QR Code AliPay and WeChat Pay for inbound Chinese tourists

The short summary is these installations are for inbound tourists with VISA Touch contactless credit cards, a transit boutique for marketing purposes more than real use.

Japanese media is quick to dismiss FeliCa as a technical failure in the face of EMV but I think that is the wrong analysis. Looking back it’s easy to see a huge mistake was that the big push for Mobile FeliCa credit cards on smartphones was not matched with an equally big push for plastic credit cards with FeliCa support.

And the big EMV push instead of FeliCa has not worked out so great either. Instead of making a technology agnostic unified push for NFC contactless, EMV bank card interests pushed their own agenda. All that did was provide a big opening for domestic QR Code payment players like Line Pay and Pay Pay which they took and continue to take.

What I find fascinating is that the mainstream Japanese IT media has not written much about the Super Suica 2 in 1 card strategy or rollout plans. Low cost transit IC card infrastructure sharing that delivers consistent and seamless transit service on mobile and legacy plastic while offering local area branding and services is a compelling vision that I don’t see bank card companies matching.

The challenge for JR group companies (JR West, JR Central, JR Kyushu, etc.) is working with JR East to offer Super Suica 2 in 1 card solutions in their own regions, because if they do not we’ll see more VISA Touch transit boutiques.