The open loop mobile connectivity challenge

The recent additions of stera transit (Visa-SMBC-Nippon Signal-QUADRAC) open loop test systems in Kyushu covering Fukuoka metro, Kuamamoto city transit and JR Kyushu expand the VISA Touch transit boutique deeper into western Japan territory. Open loop based cloud processing advocates like to portray these developments as proof that local processing based FeliCa systems like Suica et al. are expensive bygones due for replacement.

There’s just one little problem that open loop advocates fail to mention: mobile connectivity, aka the Suica app problem, the QR Code payment problem, the Smart Navigo HCE problem, etc. Wide LTE and 5G deployment doesn’t always mean reliable mobile and internet connectivity mobile payment apps depend on, and carrier outages can bring down the transaction processing side of the equation. This was proven, yet again, on July 2 when major carrier KDDI suffered a massive nationwide outage that lasted for 80 hours. Let’s make a quick reference graph for examining local processing vs cloud processing in the mobile era.

Stera is a mobile based payments platform that does away with the NTT Data Cafis dedicated backbone and replaces it with the internet based GMO Payment Gateway. The weak point of course is that since mobile powers the gate reader side, when mobile service goes down, stera gate readers stop working. As everybody found out during the KDDI network meltdown, Mobile Suica kept right on working on the transit gate and the store checkout reader, while mobile app based code payments and point systems all stopped. Some vital services that depended on KDDI connectivity like ATM networks also stopped working.

Cloud based Suica will face these challenges too when it goes online in March 2023. The only difference being how much local processing stays intact and how much system buffering there is (how much it needs to talk with the cloud server to do the job), we shall see. Which brings me to the point I want to make. The media almost always portrays the open loop/cloud vs closed loop/local match as a winner takes all, one size fits all proposition. As the KDDI meltdown proves, this is stupid, and dangerous. Never put all the eggs in one technology basket. I don’t think the risk will go away, not as long as telecommunication company corporate structures don’t foster and promote their engineering talent (the people who actually make things work) deep into the executive decision making forums.

Open loop in Japan is geared for inbound tourists the supplements, but does not replace, the old reliable Transit IC infrastructure which is evolving and reducing costs too. They compliment each other, address different needs and uses. One size doesn’t fit all. If it did, Oyster card would have died years ago.

Recharge your recharge, the winner/loser debate doesn’t mean shit in the post-Apple Pay Japanese payments market

I love articles like this one. It’s fun examining how the writer, freelancer Meiko Homma, takes old news bits, worn-out arguments and weaves them into a ‘new’ narrative with a titillatingly hot title: “QR Code payments won the cashless race, Suica utterly defeated.”

Her article trots out some QR Code payment usage data from somewhere, the PASPY transit card death saga that illustrates the increasingly difficult challenge of keeping region limited transit IC cards going, the fact that Suica only covers 840 stations out of a total of 1630, all while conveniently ignoring recent important developments like the Suica 2 in 1 Regional Affiliate program, and big updates coming in early 2023: Cloud Suica extensions and the Mobile ICOCA launch.

It has the classic feel of ‘here’s a headline, now write the article’ hack piece passing as industry analysis we have too much of these days. The Yahoo Japan portal site picked it up and the comments section was soon full of wicked fun posts picking apart the weak arguments.

I’ve said it before and say it again: the winner/loser debate doesn’t mean shit in the post-Apple Pay Japanese payments market. PayPay for example, started out as a code payment app but has added FeliCA QUICPay and EMV contactless support along with their PayPay card offering. Just like I predicted, these companies don’t care about payment technology, they just want people to use their services. My partner and I actually see less PayPay use at checkout these days and more Mobile Suica. Why?

The great thing about prepaid eMoney ‘truth in the card’ Suica, PASMO, WAON, Edy, nanaco, is they are like micro bank accounts coupled with the backend recharge flexibility of mobile wallets (Apple Pay, Google Pay, Suica App, etc.). PayPay, au Pay, Line Pay and similar Toyota Wallet knock-off payment apps with Apple Pay Wallet cards, are deployed as mobile recharge conduits that smart users leverage to put money into different eMoney micro bank accounts and get the points or instant cashback rebates they want to get at any given campaign moment. This is where the action is.

And so we have recharge acrobats like Twitter user #1: step 1 recharge PayPay account from Seven Bank account, step 2 move recharge amount from PayPay Money to PayPay Bank, step 3 move recharge from PayPay Bank to Line Pay, in Wallet app recharge Suica with Line Pay card. Or like recharge acrobat Twitter user #2: Sony Bank Wallet to Kyash to Toyota Wallet to Suica.

Phew…none of this involves transfer fees so it’s up to user creativity to come up with the recharge scenario that works best for them. Does it count as PayPay use or Line Pay use or Mobile Suica use? Does it matter?

It’s not about winners or losers, it’s about moving money around. Mobile Suica is extremely useful because of it’s recharge backend flexibility, thanks to Apple Pay and Google Pay (which does not support PASMO yet). This is the case for US citizens working in Japan who get a great return of their Suica or PASMO recharge right now using US issue credit cards because of the exchange rate. This is something visitors to Hong Kong cannot do with Apple Pay Octopus as the OCL recharge backend is far more restrictive than JR East. The biggest gripe users have with Suica is ¥20,000 balance limit.

In the weeks to come we’ll be sure to see hand wringing articles debating the future of Suica, open-loop, etc.,etc., because let’s face it, IT media journalists need something to write about in these challenging times where everything has to be sold as winner/loser, black/white, 0 or 10, and nothing in-between, to get any traction at all. As for me, I think it’s far more interesting, and real, to observe how users are using all these nifty mobile payment tools.

UPDATE 2022-07-04: Thoughts on the KDDI network outage
That was fast. No sooner had the “QR Codes won the mobile payments race” article appeared when major Japanese carrier KDDI experienced a nationwide mobile network meltdown on July 2 JST, lasted a full day with a very slow, still in progress, recovery affecting more than 40 million customers. Suddenly social media channels were full of people complaining that QR Code payments didn’t work, assuming that Mobile Suica and other NFC mobile payments stopped too. Which was not the case though a few fake posts claimed, or just ‘assumed’ people were stranded inside stations. Fortunately there were numerous online articles setting the record straight.

It’s a lesson that people soon forget in our attention span challenged social media era. We saw plenty of QR Code payment downsides in the 2018 Hokkaido Eastern Iburi earthquake that knocked out power and mobile service across Hokkaido. At the time some fake Chinese social media posts claimed AliPay and WeChat pay ‘still worked’ in Hokkaido at the time, of course they did not.

Mobile payment disruptions happen with every natural disaster and war. Good and safe practices don’t come easy when smartphone apps lure us down the easy path without spelling out the risks. It’s a lesson we have to learn again and again, that while network dependent code payment apps have some benefits, they also have limits and security risks. One size does not fit all, NFC and code payments each have their place and role to play in the expanding mobile payments universe. The key is understanding their strengths and weaknesses.

Cashless is fast and convenient? Point app mania reality check

My partner wanted to pick up some cheap t-shirts on bargain sale at Uniqlo yesterday. The Asagaya station building Beans shopping mall has all the latest cashless options but very bad network service so Uniqlo checkout was a comedy routine. First he brought up the Uniqulo app to get Uniqlo points, then I brought up my JRE POINT app to earn JRE POINT, then he finally paid with QR Code dBarai (docomo). But for each app launch and load we had to run to the store entrance to capture enough network connection for the apps codes to load. The staff is very used to this and suggest customers do so when apps didn’t load, patiently folding clothes while they run back and forth. I asked the cashier if this happens all the time. She smiled and nodded. “Cash is probably faster isn’t it?” She smiled and nodded.

Gosh, just when we thought cashless was going to free us from the so called inconvenient drudgery of cash along came smartphone reward point apps that bog down the whole cashless checkout experience, neatly killing off the supposed time saving advantage. You stand in line while the checkout customer fiddles with smartphone, digging around in an app to find the right coupon code thing. You feel smug until it’s your turn and the networks sucks, the discount coupon doesn’t load and bam, you’re holding up the line too. It has gotten to the point where Nikkei XTECH has provided an Apple Pay help article for faster checkout that explains the benefits of using Apple Value Added Services. Will Apple Pay VAS dPoint and Apple Pay VAS PONTA really help us? Probably not as they only work at LAWSON.

There is another checkout trend I see recently. With price increases everywhere people are using cash a lot more, even at places like in-station Beck’s Coffee Shop. Every customer has a Suica but more young people are keeping it in their pocket and plucking down ¥10,000 yen notes for ¥300 ice coffee. Why? I think it’s Kakebo culture at play, it’s easier to budget with cash payments and the small slightly inconvenient physical routines that accompany it. It’s not about doing everything with cash, but good old tsukae-wake compartmentalization helps keep focus and tamps down the impulsiveness when doing everything cashless. Another way of spreading the risk in these uncertain times.

iOS 16 Wallet: expanding the Apple Pay experience, aka Suica auto-charge for the rest of us

iOS 15 added big new features to Wallet, expanding digital keys from cars to include home, office and hotels and ID in Wallet driver licenses for the first time. There were smaller but important UI changes too. A new add card screen offered new categories making is easy to add transit cards regardless of the device region and quickly re-add previous Wallet items from iCloud. iOS 15 was all about Wallet to the extent that Apple now advertises it as a separate thing from Apple Pay with a separate web page, and even referred to Apple Pay as “one of the most important areas of Wallet” in the WWDC keynote. Very interesting.

iOS 16 moves the focus back to Apple Pay and making digital payments more useful, practical and universal. The WWDC22 Keynote announced Apple Pay Later, in-app ID card verification and key sharing. Apple Pay Later is one aspect of several new Apple Pay functions unveiled in the What’s new in Apple Pay and Wallet session.

Multi-merchant payments: In our online world we can never be sure how many sub-merchants are involved when we order something and how our card information is shared. In multi-merchant Apple Pay, multiple payment tokens are issued for each merchant in the same transaction, preserving user privacy, with the iOS 16 Apple Pay paysheet showing a breakdown of each sub-merchant charge. This feature works mostly on the backend, but showcases how smartly the Apple Pay Wallet team design features to ‘just work’ securely for merchants and customers.

Automatic Payments
My favorite iOS 16 feature as it addresses a lot of interesting use cases, much more than just Apple Pay Later installments which fall under:

Reoccurring payments, which include things like installments and subscriptions, basically any regularly scheduled payment. With the recent Starbucks Japan price increases, I decided to sign up for the new JR East Beck’s Coffee Shop subscription plan. Up to 3 cups a day for ¥2,800 a month. A pretty good deal for commuters like me. The Beck’s subscription service is subcontracted out to an interesting online business venture company called Favy that uses Sign in with Apple to create an account. Payment however is manual credit card entry with the onerous, ubiquitous 3D Secure sign-in. Pass issue and serving size selection (M=¥50, L=¥100 extra) is done in Safari. It works well enough, but canceling or getting payment details is a real Safari expedition. It would be a much better, and faster, customer experience doing it all in Apple Pay.

Automatic Reload: this is the real money feature for me because it plays on the classic snag of using Apple Pay Suica…recharge. All pre-paid cards are a catch-22. Japanese users love them because they like the “I know how much money I’m adding to my card” aspect of manual recharge, but there’s the inevitable, you know you forgot about it, bing-bong ‘please recharge’ transit gate alarm when Suica balance is short.

JR East offers Suica Auto-Charge (auto-reload) as a feature of their VIEW card. The auto-charge option works great with Apple Pay Suica but like all transit card auto-charge, it is tethered to the transit gate NFC system. This means the users gets instant, seamless auto-charge but only on the operator’s transit gates. Suica auto-charge does not work outside of the Suica and PASMO transit gates, not at store terminals, not in other transit card regions like JR West ICOCA. This limitation is a big customer complaint, I and many others would love Apple Pay Suica auto-charge to work everywhere.

Apple Pay automatic reload takes care of this problem very nicely. Suica would recharge anywhere because the card balance ‘trigger’ and reload process is done via Apple Pay instead of JR East transit gates and the Suica system. JR East could keep auto-charge exclusive to their VIEW cards as they do now or easily, selectively expand it. Either way they would greatly increase the usefulness of VIEW and Suica by supporting the new Apple Pay automatic reload feature. The possibilities are are pretty exciting.

Order tracking
Another very useful feature I think people will love using. The addition of QR/barcodes in the Apple Pay sheet is a first and will greatly shorten the order pickup~delivery process. The best use case of Apple Pay and bar codes that I can think of.

ID verification in apps
This is where ID in Wallet gets real. Wallet app has TSA airport checkpoint verification built-in but that’s not going to help all the government issuing agencies, not to mention software developers, around the world who want to implement digital ID verification to unlock various digital services.

JR East for example has centered their whole Super Suica MaaS Cloud initiative around ID PORT and the ability to match various region or age based services (discounts, special fares, etc.). In other words JR East and their sub-merchant or local government agency want to know where I live and how old I am. This is all provided on the Japanese government My Number digital identity card launching later this year on Android, and Apple Wallet later on. But I don’t want my personal details going everywhere. If the MaaS campaign app or website only needs to know that I live in Tokyo and am over 60, that’s the only info I want to give them. This is what the new PassKit ID request APIs in iOS 16 do: give apps only the information they need to perform a verification for a service and nothing more.

Key sharing
Nothing big here, but it does address one iOS 15 Wallet shortcoming for home, hotel keys which that could not be shared and expanded share options beyond mail and messages. I’m doubtful Apple includes office keys in the bargain but the fine print reads: available on participating car brands and access properties. We’ll find out when iOS 16 ships.

And then there’s Tap to Pay on iPhone. It’s really not an Apple Pay function to me because it turns iPhone into a very handy and portable NFC payment terminal, but it makes sense branding wise. Just say Apple Pay for making…and accepting payments. Anywhere the merchant has their payment provider POS app and a network connection, they are ready to go. This is big. Apple has lined up an impressive number payment providers in a very short time who are happy to leave all the hardware certification and secure element management to Apple and focus on software. I can practically feel the intense interest from Japan where local payment providers would love to leverage the global NFC capable iPhone for seamless EMV and FeliCa payment services. It could be an interesting Apple Pay year.

Apple Platform Security May 2022: Tap to Pay on iPhone, Express Mode scare mongers and other fun

Ahh springtime, flowers and the annual Apple Platform Security (APS) update. This year’s version has many Apple Pay housekeeping changes. Previous versions put everything Apple Pay in a single section. In keeping with Apple spinning out iOS 15 Wallet app as a separate identity, Wallet has its own separate section now, covering all the things Jennifer Bailey unveiled at WWDC21: hotel-home-office keys and ID in Wallet. The Apple Pay section adds a new category for Tap to Pay on iPhone with some interesting bits.

The Tap to Pay on iPhone servers manage the setup and provisioning of the payment kernels in the device. The servers also monitor the security of the Tap to Pay on iPhone devices in a manner compatible with to the Contactless Payments on COTS (CPoC) standard from the Payment Card Industry Security Standards Council (PCI SSC) and are PCI DSS compliant.

The Tap to Pay on iPhone server emits decryption keys to the Payment Service Provider after validation of the integrity and authenticity of the data, and after verifying that the card read was within 60 seconds of the card read on the device.

What’s interesting to me is that Tap to Pay on iPhone servers are providing a seamless payment reader experience in the same way that Apple Pay servers provide a seamless pay experience. It just works, from setup to use, the same tight integration allows payment service providers to focus on POS app development and forget about the hardware because Apple Pay takes care of everything. As Junya Suzuki tweeted recently, a lot of payment reader hardware is suddenly junk compared to what iPhone is providing with tight mobile integration and Tap to Pay servers on the backend. Now with Tap to Pay apps on the horizon, good thing that iOS 15 Wallet expanded the secure element max to 16 ain’t it?

Speaking of Wallet, this separate section covers all things “access credential” related (hotel-corporate-home-car-student ID) with App Clips suggested for provisioning multifamily home keys. Transit now includes eMoney cards (or is it e-Money, Apple seems confused about it just like Express Mode vs Express Transit) and IDs in Wallet is covered in detail. There is also an intriguing iOS 15.4 Wallet security tweak:

In iOS 15.4 or later, when a user double-clicks the side button on an iPhone with Face ID or double-clicks the Home button on an iPhone with Touch ID, their passes and access key details aren’t displayed until they authenticate to the device. Either Face ID, Touch ID, or passcode authentication is required before pass specific information including hotel booking details are displayed in Apple Wallet.

It sounds almost exactly what we already do with regular Apple Pay cards. Perhaps keys and passes only show a generic icon and checkmark with Express Mode with the double-click + authentication required for show details…it’s not very clear.

Speaking of Express Mode, ‘security experts’ are still scare mongering the masses with the tired old Russian security expert/Apple Pay VISA Express Transit exploit story that made the rounds last November, regurgitated by Forbes in the over the top scary sounding, and sloppily written (this is Forbes after all), “How hackers can drain your bank account with Apple and Samsung tap and pay apps“.

The whole security expert thing reminds me of what my uncle the doctor (who ran a medical research lab at Columbia University) used to say about his disdain for pharmaceutical companies, “They don’t want to cure you, they just want to keep ‘treating’ you with their medicines.” Human nature never changes. The gist is that EMV Express Transit Mode will always be a thorn in Apple Pay’s side because the security is up to the card companies.

The document is worth your time is you have any interest in Apple Pay and Wallet.