A very strange Apple Pay Suica outage

The October 5 Apple Pay & Wallet outage that went completely unnoticed outside of Japan and Hong Kong was a very strange one. That’s because it was a very specific outage: ‘truth on the card’ transit cards with a stored value balance. But not all transit cards, only Suica, PASMO and Octopus. Similar transit cards such as Clipper, SmarTrip and TAP were unaffected. The former are FeliCa cards, the latter are MIFARE cards.

Let’s translate the Apple speak from the iCloud System Status page. Some users = FeliCa stored value card users, specifically: Suica, PASMO and Octopus. In addition to the not being able to add, suspend or remove existing cards, users could not recharge them (reload, top up, add money, etc.) but this issue was not listed. The add, suspend or remove existing card label gives us a big clue.

It soon became clear from the cacophony of Japanese tweet complaints that FeliCa credit/debit cards on the iD and QUICPay side were not a problem, only the stored value (SV) recharge, and practically unnoticed, Toyota Wallet iD prepaid card recharge. Meanwhile, Android Mobile Suica and Mobile PASMO users didn’t have any problems at all, everything worked fine. Bingo, the outage was an Apple Pay server problem, not caused by Mobile Suica or Mobile PASMO which was mistakenly reported by the Japanese media.

From the get-go Mobile Suica and PASMO support said it only an Apple Pay problem. Octopus Cards Limited, confoundedly coy as usual, didn’t say which ‘top up’ service wasn’t working but we damn well know it was Apple Pay. Overworked Suica, PASMO and Octopus support folk were left to clean up a recharge mess because the Apple Pay outage hit exactly at peak Tokyo morning commute time in the first real commuting week since the State of Emergency was lifted.

There was also another fascinating issue going on that most people didn’t pick up on. Suica and PASMO Apps offer 2 kinds of recharge: Apple Pay and their own direct credit card recharge system. Because Suica App recharge is independent of Apple Pay and deals directly with the Suica SV on the device, similar to adding cash at a station kiosk, it should have worked during the Apple Pay outage…but it did not. Why?

The FeliCa on Apple Pay story is a deep one. To date Apple Pay is the only digital wallet platform that offers all flavors of NFC (A-B-F) with the major transaction protocols that go with them, (EMV, FeliCa and MIFARE) as standard on all Apple NFC (and now also NFC + UWB) devices. The industry scuttlebutt in Japan is that to make this happen, Apple licensed a Mobile FeliCa key server from FeliCa Networks to manage their own Apple Pay devices. This indicates Apple’s commitment but I also believe was setup this way so that Apple Pay can safely backup the FeliCa card SV balance in Wallet if something goes wrong with the device or if some glitch happens during the recharge process from a credit/debit card, either Apple Pay (Wallet) or independent app (Suica, PASMO, Toyota Wallet).

Here is my take of what happened. Apple is preparing to launch 2 new FeliCa SV eMoney cards on Apple Pay very soon, WAON and nanaco. That means they need to reconfigure their FeliCa key server and Apple Pay server so that WAON and nanaco SV card balances are well protected because WAON and nanaco card balances are much bigger than Suica or PASMO, up to ¥40,000. If anything happens to the device, WAON and nanaco must always appear in the iOS 15 Wallet Previous Cards screen, otherwise users will freakout and won’t touch Apple Pay again. Hence the add, suspend, remove Apple Pay FeliCa SV card outage. It was a server configuration mistake on the Apple Pay side related to WAON and nanaco preparations to make sure those bigger card balances are ironclad protected.

In other words, WAON and nanaco are coming with iOS 15.1.

UWB Touchless Express Transit and Apple Pay for iOS 15?

A recent sudden surge of hits from Hong Kong accessing my December 2019 UWB Touchless Mobile FeliCa post seemed odd. I dug around and it appears that Hong Kong MTR, like JR East, is making noises about incorporating UWB technology in next generation transit gates.

iOS 14.5 added a new PassKit call for Bluetooth and the U1 chip integration since iPhone 11 and Apple Watch 6, coupled with global FeliCa support certainly puts Apple ahead of the game. I have no idea what WWDC21 will deliver but more UWB integration is a given.

Apple only mentioned UWB Touchless at WWDC20 in connection with digital car key without showing anything because the Car Connectivity Consortium Digital Key 3.0 spec was a work in progress. Now that the spec is in-place with BMW said to deliver car models incorporating UWB Touchless this year, will Apple show it in action? I think it’s highly likely, but since Car Key is a ‘Wallet Card’, and Wallet app Express Cards come is 3 types: Transit, Student ID, and Car Key, the more interesting question is…will Apple also show Touchless Transit and Student ID Express Cards? And what about Apple Pay?

People think Touchless is a completely new thing for ‘keep smartphone in pocket’ transactions, and they worry about security. You can’t blame them because marketers are selling the in-pocket payment experience. However, Touchless is simply long distance NFC without NFC. All UWB Touchless does is describe the frequency to use Bluetooth instead of NFC. The background stuff, secure element and so on, is exactly the same. This means user interaction is the same. For walking through transit gates and security doors, or unlocking your car, the convenience of Touchless is easy to understand: no more NFC tapping, just keep moving.

What about Express Card payments? The current Apple Pay Suica payment checkout experience: the user taps Suica on a touchscreen, or tells the clerk “Suica” then holds the device to the reader. The user has to give consent before the transaction is activated by checkout staff or the self checkout reader. For Apple Pay EMV transactions users have the extra step of confirming a transaction by Face ID/Touch ID to complete it.

Realistically however, in what situations does Touchless make store checkout more convenient and faster? Drive thru certainly, supermarkets…maybe, but most stores will probably not want to invest in Touchless without a good reason when the NFC readers they already have installed get the job done. There is one more interesting role that Apple has planned for UWB however, one that promises to improve the entire Apple Pay and Wallet experience: communicating with the reader before transaction to select the right Wallet card for the job, at a distance, for a truly smart Wallet app. With national ID cards, passports and more coming to Wallet at some point, UWB could be the Wallet reboot we really need.

And then there is EMVCo. The problems with UWB Touchless for EMVCo are that: (1) Touchless only works with devices with batteries, á la AirTag, and doesn’t work with the current plastic card model, (2) UWB + Bluetooth level the digital playing field with FeliCa and MIFARE, no more ‘real’ vs ‘who cares’ NFC hardware flavors to split hairs over. The plastic card NFC limitation is probably a bitter pill for everybody but especially for EMVCo members and issuers as plastic card issue is big business, and many customers are more comfortable with plastic cards. For those reasons I think EMVCo will be the last to support UWB Touchless, if they do at all. On the plus side Touchless does give digital wallet platforms an edge to create smart aware wallets, digital does NFC and Touchless, plastic only does NFC. We’ll find out about Apple’s UWB Touchless roadmap at WWDC21.

Octopus 2.0

The Apple Pay Octopus launch in June 2020 marked the end of an era of Octopus as the exclusive Hong Kong MTR home grown transit platform, and the start of MTR integrating into China mainland transit fare standards. In August 2020 Octopus Cards Limited announced they would join China T-Union. My take about it and the eventual migration of Octopus from FeliCa to PBOC 2.0, struck a raw nerve and did not go down well with some Hong Kong folk:

Can someone tell the ill-informed, self-centred, attention-seeking blogger to stop spreading fake rumours about octopus ditching FeliCa? Not in this lifetime…The self-proclaimed expert blogger’s been wrong on so many levels I’m amazed people still follow him like religion and never question his fantasy stories. Utterly annoyed by him dropping quotes from people out of context and use them to his benefits.

In April 2021 new OCL CEO Angus Lee Chun-ming said in a South China Morning Post interview that OCL had applied for China T-Union membership as planned, and will launch a dual mode Octopus card for mainland transit use:

“We have applied to join the China T-Union, the nationwide one-card payment system led by the Ministry of Transport. That will enable Octopus physical-card holders to pay for public transport fares in mainland China,”…

The service can be upgraded to digital Octopus cards in the phase two development. “The card will be denominated in Hong Kong dollars. Octopus will arrange the currency settlement with the mainland partner,” said Lee.

A one-card nationwide payment system eh? Sounds like an plug for China T-Union instead of an Octopus presser. Phase 1 is a physical dual mode Octopus card that appears to be 2 separate chips (PBOC and FeliCa) in one card with a common HKD ePurse. This is novel as Greater Bay Area dual mode cards up to now used separate ePurses for each currency. It’s also complicated because mainland transit operators have to do the currency conversion. A digital wallet version is phase 2. The elimination of FeliCa on the Hong Kong side will be the final phase, though that depends on the Ministry of Transport removing the current PBOC restriction that limits it to transit use and T-Union branding issue, or Octopus coming up with something else. We shall see.

On the mobile side Hong Kong iPhone users already have a dual mode Wallet option to add China T-Union cards if they have a China UnionPay credit or debit card. It’s not dual mode on one card and there is an Express Transit issue when turning on a China T-Union card turns off Express Transit for Octopus, but it works.

Dual mode transit cards on Apple Pay don’t exist yet but they are technically possible. Apple Pay already uses dual mode NFC switching for Japanese issue payment cards, FeliCa for contactless use in Japan, EMV for contactless use abroad. Another option might be the multiple secure element domain/multiple NFC protocol support of Mobile FeliCa 4.1 outlined by FeliCa Dude for dual mode transactions using just Mobile FeliCa with NFC-A/NFC-F.

On the transit gate side it will be interesting to see what design MTR uses for multiple protocol open-loop. NFC requires the reader side to specify the NFC protocol used for the transaction. This is a not a problem at store checkout, but how does the user specify the transaction protocol on transit gates? Answer: by tapping different readers. Perhaps the new MTR gates will host a NFC-A reader (EMV and PBOC), a NFC-F reader (FeliCa) in addition to the already separate QR reader? And if those Touchless UWB rumors are true, UWB and Bluetooth could be joining the MTR next generation gate party. One thing for sure, transitions are messy, and expensive.

QR Code Transit on Hong Kong MTR starts January 23 (Updated)

After a very long preparation period QR Code transit on Hong Kong MTR finally starts on Saturday, January 23. The MTR Fan FaceBook page:

Only TWO WEEKS left before the launch of QR code payment on 23 January! For this new service, we have installed about 1,000 QR code scanners at stations and conducted a series of system and on-site tests. Prominent purple signage will also be on display to help passengers identify the gates providing the new service.

This is the debut of MTR ‘open-loop’ ticketing. Up until now MTR used the ubiquitous Octopus card, the trail blazing transit card that showed the world what smartcard ticketing can do when extended beyond transit to include eMoney payments, transforming a transit card into a transit payment platform. Unlike Japan however Octopus Card Limited (OCL) was late bringing Octopus to mobile. Part of the problem was that Hong Kong mobile carriers never had an Osaifu Keitai-like standard that bridged the Symbian and Android hardware eras. OCL also wasted time with SIM card mobile support before finally launching the mobile Smart Octopus service first on Samsung Pay in late 2018, followed by Apple Pay Octopus in June 2020 and Huawei Pay Octopus in December 2020.

But MTR still faces a problem that most Android devices don’t support FeliCa even though NFC-F is supported across all NFC capable devices. It’s the global NFC dilemma best illustrated in the Google Pay on Google Pixel situation: Mobile FeliCa is installed on all Pixel devices but Google only turns it on for Pixel models sold in Japan. There are many takes on the reasons why. My take is that Google doesn’t want to do the all the global NFC OS level support work that benefits all Android manufacturers. Google’s stance is, ‘don’t ask us, roll your own embedded Secure Element (eSE) solution.’ And so it’s a race of how many ‘Octopus on XX Pay’ digital wallet platforms OCL can line up for Android and wearables.

For MTR, QR Code open loop transit sidesteps this Android hardware mess, but will it be a success when users have to open a smartphone app with a face mask on at every gate? Apple Pay Octopus on Apple Watch sure beats that problem and then some. Long term I think NFC wearables and UWB Touchless will be the QR killer. Time will tell.

AliPay HK is the first payment provider, others QR players will be added as they are qualified. The transit gate layout is interesting, QR is limited to purple colored gate lanes shown in a nifty MTR video. This is similar to what JR East will do when they phase out mag strip paper ticketing and replace it QR Code paper tickets. It’s also the layout that Nankai will do when they implement VISA Touch after testing it this year.

The next MTR open loop addition is expected to be EMV+PBOC China T-Union compatibility though MTR has not announced when that will happen. OCL already committed to a new Octopus card that will be compatible with China T-Union.

UPDATE

AliPay mainland accounts can also be used for Hong Kong MTR QR transit.

Huawei Pay Octopus: global NFC = custom eSE

Huawei Pay Octopus launched on December 9, the second digital Octopus 2020 launch after Apple Pay Octopus on June 2. The device list is interesting: Huawei Mate 40 Pro, P40, P40 Pro, P40 Pro+, P30, P30 Pro, Mate 30, Mate 30 Pro, Mate 20, Mate 20 Pro, Mate RS Porsche, Mate Xs, Nova 7 5G. A wide range of Huawei models sold in Hong Kong have global NFC support, at least on the smartphone side but does not include the G2 Pro smartwatch.

In Japan the only Huawei models with global NFC/Mobile FeliCa/Osaifu Keitai support are the P20 and P30 and only with a Docomo sim. Huawei Pay is not supported in Japan. The Huawei Pay Octopus device list tells us what we suspected all along: Huawei Pay is global NFC capable but Huawei kneecaps it so they can sell devices via Docomo. Huawei Pay can turn things on and off because, just like Apple Pay, Huawei has a custom embedded Secure Element (eSE), so does OPPO and so will Xiaomi in 2021.

The gist of all this is that Google’s Android Pay era HCE and HCE-F technology lost out to eSE long ago. As FeliCa Dude said, ‘it’s all eSE or nothing now,’ to which I would add, ‘it’s all custom eSE + XX-Pay or nothing now.’

So where does Google Pay fit in this scheme? It doesn’t. And it could be one of the things Google hopes to address with their own SoC, but a custom eSE is probably a long shot. That is unless Google is serious about making global NFC Google Pay work seamlessly on Pixel and Fitbit devices by dumping Osaifu Keitai and going all in with a custom eSE. If Google persists in wanting it both ways, limited NFC Google Pay for everybody and global NFC Google Pay in Pixel for Japan only so they can sell via Docomo etc., expect more of the confusing Android global NFC support mess we have now.