A very strange Apple Pay Suica outage

The October 5 Apple Pay & Wallet outage that went completely unnoticed outside of Japan and Hong Kong was a very strange one. That’s because it was a very specific outage: ‘truth on the card’ transit cards with a stored value balance. But not all transit cards, only Suica, PASMO and Octopus. Similar transit cards such as Clipper, SmarTrip and TAP were unaffected. The former are FeliCa cards, the latter are MIFARE cards.

Let’s translate the Apple speak from the iCloud System Status page. Some users = FeliCa stored value card users, specifically: Suica, PASMO and Octopus. In addition to the not being able to add, suspend or remove existing cards, users could not recharge them (reload, top up, add money, etc.) but this issue was not listed. The add, suspend or remove existing card label gives us a big clue.

It soon became clear from the cacophony of Japanese tweet complaints that FeliCa credit/debit cards on the iD and QUICPay side were not a problem, only the stored value (SV) recharge, and practically unnoticed, Toyota Wallet iD prepaid card recharge. Meanwhile, Android Mobile Suica and Mobile PASMO users didn’t have any problems at all, everything worked fine. Bingo, the outage was an Apple Pay server problem, not caused by Mobile Suica or Mobile PASMO which was mistakenly reported by the Japanese media.

From the get-go Mobile Suica and PASMO support said it only an Apple Pay problem. Octopus Cards Limited, confoundedly coy as usual, didn’t say which ‘top up’ service wasn’t working but we damn well know it was Apple Pay. Overworked Suica, PASMO and Octopus support folk were left to clean up a recharge mess because the Apple Pay outage hit exactly at peak Tokyo morning commute time in the first real commuting week since the State of Emergency was lifted.

There was also another fascinating issue going on that most people didn’t pick up on. Suica and PASMO Apps offer 2 kinds of recharge: Apple Pay and their own direct credit card recharge system. Because Suica App recharge is independent of Apple Pay and deals directly with the Suica SV on the device, similar to adding cash at a station kiosk, it should have worked during the Apple Pay outage…but it did not. Why?

The FeliCa on Apple Pay story is a deep one. To date Apple Pay is the only digital wallet platform that offers all flavors of NFC (A-B-F) with the major transaction protocols that go with them, (EMV, FeliCa and MIFARE) as standard on all Apple NFC (and now also NFC + UWB) devices. The industry scuttlebutt in Japan is that to make this happen, Apple licensed a Mobile FeliCa key server from FeliCa Networks to manage their own Apple Pay devices. This indicates Apple’s commitment but I also believe was setup this way so that Apple Pay can safely backup the FeliCa card SV balance in Wallet if something goes wrong with the device or if some glitch happens during the recharge process from a credit/debit card, either Apple Pay (Wallet) or independent app (Suica, PASMO, Toyota Wallet).

Here is my take of what happened. Apple is preparing to launch 2 new FeliCa SV eMoney cards on Apple Pay very soon, WAON and nanaco. That means they need to reconfigure their FeliCa key server and Apple Pay server so that WAON and nanaco SV card balances are well protected because WAON and nanaco card balances are much bigger than Suica or PASMO, up to ¥40,000. If anything happens to the device, WAON and nanaco must always appear in the iOS 15 Wallet Previous Cards screen, otherwise users will freakout and won’t touch Apple Pay again. Hence the add, suspend, remove Apple Pay FeliCa SV card outage. It was a server configuration mistake on the Apple Pay side related to WAON and nanaco preparations to make sure those bigger card balances are ironclad protected.

In other words, WAON and nanaco are coming with iOS 15.1.

WAON and nanaco coming to Apple Pay

UPDATE: WAON and nanaco will launch on Apple Pay October 21 JST

Two of the last big three Apple Pay Japan holdouts are finally coming: AEON announced WAON and Seven & i Holdings announced nanaco for ‘later this year’. These popular prepaid eMoney FeliCa cards have been on Osaifu Keitai and Google Pay for some time. This leaves Rakuten Edy as the last, and largest, Wallet holdout although the iOS Rakuten Edy app recently received an update that supports Apple Pay for physical card recharge.

Despite the uptake of QR Code payment apps such as PayPay, prepaid eMoney cards remain popular and getting them on Apple Pay is an important development. The cards are also more secure: Seven & i Holdings experienced a huge embarrassment when they launched their 7pay QR Code payment service in 2019 that quickly failed due to a security meltdown. Since that disaster they have refocused on nanaco as their in-store payment + loyalty point reward strategy. Currently nanaco has issued 74 million cards, WAON has issued 87 million cards. For comparison Suica has issued 84 million plastic cards and over 14 million Mobile Suica digital cards that includes Apple Pay Suica.

Release details are sparse but it’s safe to assume they are coming after iOS 15 ships (probably 15.1). iOS 15 Wallet includes UI improvements that remove the confusing device region setting requirement and simplify adding transit cards like Suica and non-bank stored value (SV) prepaid cards like WAON and nanaco. As pointed out many times before, all iPhone 8 • Apple Watch 3 and later models support Apple Pay Japan cards thanks to Apple’s global NFC support. The big questions are: (1) Is direct Wallet add card supported that bypasses creating a WAON or nanaco account as part of the digital card issue process on Google Pay? (2) Can physical cards be transferred like Suica and PASMO? None of this is supported on Android.

These and other usability issues have kept these cards from joining Apple Pay. It will be interesting to see if Apple has solved them and persuaded AEON • Seven & i to simplify their digital card issue process to follow the great example set by Apple Pay Suica because that is the high bar: direct Wallet adding with no sign up and open ended Apple Pay recharge. The low bar is the Toyota Wallet app-like model of chaining card issue and recharge functions to a user account app. The cards should support Express Mode as they do for Mobile WAON and Mobile nanaco on Android. The press release Apple Pay WAON image suggests Express Mode, the Apple Pay nanaco image does not, however the dual press announcement does suggest a level of commitment and integration on the Apple Pay side. We’ll see.

Not many of new iOS 15 Wallet goodies announced at WWDC will come to Japan soon with the exception of digital car keys, adding WAON and nanaco now is a smart move that will keep users happy. With all the card possibilities coming to Japan this year, it’s a good thing that iOS 15 ups the Wallet card max limit to 16.

Instant digital issue apps for Apple Pay Edy, nanaco, WAON support?

A fun mockup of card options for direct iOS 14 Wallet issue for Edy, nanaco, WAON that will probably never happen. If they come at all they will be digital issue via apps.

Now that VISA JP finally signed with Apple Pay, what about the last holdouts: Edy, nanaco and WAON? These have been on Google Pay for some time but like all things Google Pay Japan, it is courtesy of Osaifu Keitai rather than native Google support. Apple was smart to go for Suica first, then PASMO (which has yet to appear on Google Pay) but it’s time to complete the Apple Pay Japan lineup.

Google Pay Japan has 2 basic categories for adding cards: EMV bank payment cards (AMEX, JCB, mastercard, VISA) and Japanese eMoney cards (Suica, Edy, nanaco, WAON, iD, QUICPay). In other words, Google Pay arranges cards by NFC flavor. This is because many Android devices sold outside of Japan don’t include FeliCa even though they have the hardware to do so. Google Pixel 5 for example has Mobile FeliCa installed on every single device it sells, but only activates it for Japanese models. Perhaps this will change with Pixel 6, we will see.

Apple Pay doesn’t make a distinction between NFC flavors, just one global NFC. No EMV or FeliCa bank payment cards, just payment cards, period. Apple also encourages Japanese bank card issuers to use the NFC switching and dual mode features of iOS and watchOS Wallet for seamless use on any payment reader in Japan or abroad. The same thing applies to Wallet transit cards. Wallet can have multiple Express Transit cards and juggle between FeliCa (Suica, Octopus, PASMO) MIFARE (SmarTrip, Clipper, TAP) and PBOC (China T-Union cards).

So what is the Wallet category for non-transit stored value prepaid payment cards? I have no idea but for this exercise I’ll use eMoney (電子マネー). Apple Pay has everything in place to flip the switch since 2016, what’s the holdup? There’s a big problem using the Suica add card Wallet process for eMoney cards. This problem is on full display with Google Pay WAON: the user has to create an WAON account in Google Pay to add it. Worse, if the user deletes the WAON card they loose the Google Pay created WAON ID and card balance.

I don’t think Apple wants this ‘create an account’ nightmare scenario in Apple Pay, that’s what apps are for. Fortunately we have a growing collection of ‘instant issue’ apps for adding cards to Wallet and digital issue only is quickly becoming standard for Apple Pay Japan debit/prepaid cards: kyash, Minna no Ginko, Toyota Wallet, etc.

The digital issue app model is perfect for Edy, nanaco and WAON who want to be collecting accounts instead of selling plastic prepaid cards. And they already have iOS apps. Leave the account creation and management drudgery in the app so users curse the app instead of Apple Pay. Once done the user taps ‘Add to Wallet’ and presto, instant WAON or nanaco all ready to go with direct Wallet recharge. Other bonuses: (1) instant issue apps eliminate ‘I wanna transfer my plastic card to Wallet’ overhead, (2) if anything goes wrong and the balance is lost, it’s the fault of the app, not Apple Pay. Keeping things simple and streamlined is key for a good Apple Pay user experience, one more Wallet reboot challenge for iOS 15.

UPDATE
WAON and nanaco for Apple Pay have been announced for later in 2021. AEON also updated their iAEON App for issuing Mobile WAON cards on Osaifu Keitai Android smartphones, almost certainly the scenario described above for Apple Pay support when it launches after the release of iOS 15.1.