The open loop mobile connectivity challenge

The recent additions of stera transit (Visa-SMBC-Nippon Signal-QUADRAC) open loop test systems in Kyushu covering Fukuoka metro, Kuamamoto city transit and JR Kyushu expand the VISA Touch transit boutique deeper into western Japan territory. Open loop based cloud processing advocates like to portray these developments as proof that local processing based FeliCa systems like Suica et al. are expensive bygones due for replacement.

There’s just one little problem that open loop advocates fail to mention: mobile connectivity, aka the Suica app problem, the QR Code payment problem, the Smart Navigo HCE problem, etc. Wide LTE and 5G deployment doesn’t always mean reliable mobile and internet connectivity mobile payment apps depend on, and carrier outages can bring down the transaction processing side of the equation. This was proven, yet again, on July 2 when major carrier KDDI suffered a massive nationwide outage that lasted for 80 hours. Let’s make a quick reference graph for examining local processing vs cloud processing in the mobile era.

Stera is a mobile based payments platform that does away with the NTT Data Cafis dedicated backbone and replaces it with the internet based GMO Payment Gateway. The weak point of course is that since mobile powers the gate reader side, when mobile service goes down, stera gate readers stop working. As everybody found out during the KDDI network meltdown, Mobile Suica kept right on working on the transit gate and the store checkout reader, while mobile app based code payments and point systems all stopped. Some vital services that depended on KDDI connectivity like ATM networks also stopped working.

Cloud based Suica will face these challenges too when it goes online in March 2023. The only difference being how much local processing stays intact and how much system buffering there is (how much it needs to talk with the cloud server to do the job), we shall see. Which brings me to the point I want to make. The media almost always portrays the open loop/cloud vs closed loop/local match as a winner takes all, one size fits all proposition. As the KDDI meltdown proves, this is stupid, and dangerous. Never put all the eggs in one technology basket. I don’t think the risk will go away, not as long as telecommunication company corporate structures don’t foster and promote their engineering talent (the people who actually make things work) deep into the executive decision making forums.

Open loop in Japan is geared for inbound tourists the supplements, but does not replace, the old reliable Transit IC infrastructure which is evolving and reducing costs too. They compliment each other, address different needs and uses. One size doesn’t fit all. If it did, Oyster card would have died years ago.

Recharge your recharge, the winner/loser debate doesn’t mean shit in the post-Apple Pay Japanese payments market

I love articles like this one. It’s fun examining how the writer, freelancer Meiko Homma, takes old news bits, worn-out arguments and weaves them into a ‘new’ narrative with a titillatingly hot title: “QR Code payments won the cashless race, Suica utterly defeated.”

Her article trots out some QR Code payment usage data from somewhere, the PASPY transit card death saga that illustrates the increasingly difficult challenge of keeping region limited transit IC cards going, the fact that Suica only covers 840 stations out of a total of 1630, all while conveniently ignoring recent important developments like the Suica 2 in 1 Regional Affiliate program, and big updates coming in early 2023: Cloud Suica extensions and the Mobile ICOCA launch.

It has the classic feel of ‘here’s a headline, now write the article’ hack piece passing as industry analysis we have too much of these days. The Yahoo Japan portal site picked it up and the comments section was soon full of wicked fun posts picking apart the weak arguments.

I’ve said it before and say it again: the winner/loser debate doesn’t mean shit in the post-Apple Pay Japanese payments market. PayPay for example, started out as a code payment app but has added FeliCA QUICPay and EMV contactless support along with their PayPay card offering. Just like I predicted, these companies don’t care about payment technology, they just want people to use their services. My partner and I actually see less PayPay use at checkout these days and more Mobile Suica. Why?

The great thing about prepaid eMoney ‘truth in the card’ Suica, PASMO, WAON, Edy, nanaco, is they are like micro bank accounts coupled with the backend recharge flexibility of mobile wallets (Apple Pay, Google Pay, Suica App, etc.). PayPay, au Pay, Line Pay and similar Toyota Wallet knock-off payment apps with Apple Pay Wallet cards, are deployed as mobile recharge conduits that smart users leverage to put money into different eMoney micro bank accounts and get the points or instant cashback rebates they want to get at any given campaign moment. This is where the action is.

And so we have recharge acrobats like Twitter user #1: step 1 recharge PayPay account from Seven Bank account, step 2 move recharge amount from PayPay Money to PayPay Bank, step 3 move recharge from PayPay Bank to Line Pay, in Wallet app recharge Suica with Line Pay card. Or like recharge acrobat Twitter user #2: Sony Bank Wallet to Kyash to Toyota Wallet to Suica.

Phew…none of this involves transfer fees so it’s up to user creativity to come up with the recharge scenario that works best for them. Does it count as PayPay use or Line Pay use or Mobile Suica use? Does it matter?

It’s not about winners or losers, it’s about moving money around. Mobile Suica is extremely useful because of it’s recharge backend flexibility, thanks to Apple Pay and Google Pay (which does not support PASMO yet). This is the case for US citizens working in Japan who get a great return of their Suica or PASMO recharge right now using US issue credit cards because of the exchange rate. This is something visitors to Hong Kong cannot do with Apple Pay Octopus as the OCL recharge backend is far more restrictive than JR East. The biggest gripe users have with Suica is ¥20,000 balance limit.

In the weeks to come we’ll be sure to see hand wringing articles debating the future of Suica, open-loop, etc.,etc., because let’s face it, IT media journalists need something to write about in these challenging times where everything has to be sold as winner/loser, black/white, 0 or 10, and nothing in-between, to get any traction at all. As for me, I think it’s far more interesting, and real, to observe how users are using all these nifty mobile payment tools.

UPDATE 2022-07-04: Thoughts on the KDDI network outage
That was fast. No sooner had the “QR Codes won the mobile payments race” article appeared when major Japanese carrier KDDI experienced a nationwide mobile network meltdown on July 2 JST, lasted a full day with a very slow, still in progress, recovery affecting more than 40 million customers. Suddenly social media channels were full of people complaining that QR Code payments didn’t work, assuming that Mobile Suica and other NFC mobile payments stopped too. Which was not the case though a few fake posts claimed, or just ‘assumed’ people were stranded inside stations. Fortunately there were numerous online articles setting the record straight.

It’s a lesson that people soon forget in our attention span challenged social media era. We saw plenty of QR Code payment downsides in the 2018 Hokkaido Eastern Iburi earthquake that knocked out power and mobile service across Hokkaido. At the time some fake Chinese social media posts claimed AliPay and WeChat pay ‘still worked’ in Hokkaido at the time, of course they did not.

Mobile payment disruptions happen with every natural disaster and war. Good and safe practices don’t come easy when smartphone apps lure us down the easy path without spelling out the risks. It’s a lesson we have to learn again and again, that while network dependent code payment apps have some benefits, they also have limits and security risks. One size does not fit all, NFC and code payments each have their place and role to play in the expanding mobile payments universe. The key is understanding their strengths and weaknesses.

WWDC22 Wish List

It is hard to be enthusiastic about this year’s WWDC when Apple’s entire integrated software/hardware business model is coming under attack. With so much distraction these days there’s not much of a wish list, just a few observations for Apple Pay, Apple Maps and Text Layout.

Apple Pay
First up of course, is Apple Pay. After Jennifer Bailey’s WWDC21 appearance where she announced home keys, hotel keys, office keys and ID for iOS 15 Wallet, and the separate Tap to Pay on iPhone PR announcement release in January, I don’t think Jennifer will be in the WWDC22 keynote. She’s not going to appear just to explain that Apple Pay is not a monopoly, that’s Tim’s job with CEO level pay grade, it’s unlikely she’s doing to appear to just recap details of what’s already been announced.

Bailey’s job is to announce new features, and I don’t think that after the big iOS 15 rollout of new Wallet features and Tap to Pay on iPhone there’s nothing really new. And it’s not her job to announce new frameworks, that’s what the sessions are for. Things that I have been wishing for these past few years such include easier, more open NFC Pass certification process and/or new frameworks for developers to access the secure element for payments or use Tap to Pay on iPhone. There needs to a clearer path for developers who want to use the secure element for payments (Wallet) or iPhone as payment terminal (Tap to Pay on iPhone).

Apple needs to open up the NFC/Secure Element Pass certification process and clarify the process

The only possible ‘new’ Apple Pay Wallet feature I can think of is the long in the works Code Payments. It has been lurking in the iOS shadows since iOS 13, so long that Apple legal inserted official mention in a recent Apple Pay & Privacy web page update: “When you make a payment using a QR code pass in Wallet, your device will present a unique code and share that code with the pass provider to prevent fraud.” If Apple Pay delivers native device generated QR code payments without a network connection, just like all Apple Pay cards to date, it would be quite a coup but by itself, but probably not worth a Jennifer Bailey appearance. Other future goodies like passport in Wallet or ID in Wallet for other countires are too far out to mention, at least in the iOS 16 time frame.


Apple Maps
The only new Apple Maps feature that suggests itself is AR enhanced ‘Look Around’ indoor maps for stations. That’s the conclusion after examining the current (February ~ May 2022) backpack image collection in Tokyo, Osaka, Kyoto and Nagoya. It is highly focused on stations, and stations such as Shinjuku, Tokyo, Shibuya, Ikebukuro, etc., are mostly underground, surrounded with densely packed extensive maze like malls.

This means Apple image collection in Japan is going indoors for the first time, likely at pre-arranged times when people are scarce. This is hard to do at a place like Shinjuku station as multiple companies collectively manage the entire site (JR East, Odakyu, Keio, Seibu, Tokyo Metropolitan Bureau of Transportation, Tokyo Metro, just to name a few).

Apple needs something new with indoor maps as the current incarnation is inadequate for stations. As Google Maps Live has shown in Tokyo station, AR walking guidance is a good fit for indoor maps that navigate users through intricate, information dense underground station mazes, though Google’s version has its problems. New and improved, AR enhanced “Look Around” style indoor station maps with walking directions that seamlessly guide users from transit gate to final destination would be far more useful than they are now.

Recent image collection suggests Indoor Station Maps might be coming in iOS 16

Overall, I am not optimistic that Apple Maps in Japan can become a top tier digital map service. The local 3rd party map and transit data suppliers that Apple depends on to make up the bulk of the Japanese service are decidedly not top tier. Old problems remain unfixed. In the case of the main Japanese map data supplier things have deteriorated.

Increment P (IPC) was 100% owned by Pioneer but was sold to Polaris Capital Group in June 2021 with a new CEO (ex Oracle Japan) who quickly changed the name to GeoTechnologies Inc. Under hedge fund Polaris Capital Group led management the company has been busy inflating the number of cushy company director positions, never a good sign, and pushing out shitty ad-ware apps like Torima. The focus is leveraging assets not building them.

Apple’s Japanese map problem can only be fixed by dumping low quality GeoTechnologies for a top quality digital map supplier like Zenrin (the amateurish UK backed Open Street Map effort in Japan is not worth serious consideration) or Apple aggressively mapping Japan themselves. Apple has not pursued either option: the image collection effort in Japan is leisurely and limited, its use remains restricted to Look Around. Until this changes, expect more of the same old fundamental Japanese map problems in iOS 16 and beyond. Apple Maps is a collection of many different service parts. Some evolve and improve, some do not. Let’s hope for a good outcome with the data Apple is collecting for indoor station maps.


Apple Typography TextKit 2 migration
WWDC21 saw the unveiling of TextKit 2, the next generation replacement for the 30 year old TextKit, older than QuickDraw GX even, but much less capable. TextKit 2 marked the start of a long term migration with most of TextKit 2 initially ‘opt in’ for compatibility. We’ll find out how much of TextKit 2 will evolve to default on with an ‘opt out’. There are holes to fill too: the iOS side didn’t get all the TextKit 2 features of macOS such as UITextView (multiline text), some of the planned features like NSTextContainer apparently didn’t make the final cut either. We should get a much more complete package at WWDC22. Once the TextKit 2 transition is complete, I wonder if a Core Text reboot is next.


watchOS 9 Express Cards with Power Reserve?
Mark Gurman reported that watchOS 9 will have “a new low-power mode that is designed to let its smartwatch run some apps and features without using as much battery life.” While this sounds like Express Cards with Power Reserve (transit cards, student ID, hotel-home-car-office keys) and it might even mimic the iPhone feature to some degree, it will not be the real thing. Power Reserve on iPhone is a special mode where iOS powers down itself down but leaves the lights on for direct secure element NFC transactions. iOS isn’t involved at all.

Real Power Reserve requires an Apple silicon design that supports the hardware feature on Apple Watch, it cannot be added with a simple software upgrade. Until that happens, a new watchOS 9 low-power mode means that watchOS still babysits Express Cards, but anything that gives us better battery life than what we have now is a good thing. We’ll find out later this year if Apple Watch series 8 is the real Power Reserve deal.

Enjoy the keynote and have a good WWDC.

Apple Pay Enhanced Fraud Prevention (updated)

Apple Wallet VISA card users report receiving ‘Enhanced Fraud Prevention’ notifications today that outline changes how Apple shares ‘fraud prevention assessments’ with payment card networks based on analyzed information from user Apple Pay transactions (purchase amount, currency, date, location, very likely more). The changes seem to apply to web and in-app purchases.

Apple has been doing most of this already. The new Apple Pay and Privacy text expands upon earlier iOS user guide text: If you have Location Services turned on, the location of your iPhone at the time you make a purchase may be sent to Apple and the card issuer to help prevent fraud. Perhaps Apple is changing ‘may be sent’ to ‘will be sent’.

Enhanced Fraud Prevention might cause problems for some Apple Pay users when people start traveling again as in-app purchase is used for adding money to transit cards. There have already been a few very recent and odd, ‘I can’t use my home issued Apple Pay card to recharge PASMO’ complaints on social media from inbound visitors. Until now this kind of thing has been unheard of for Apple Pay Suica•PASMO users. A new complication to keep an eye on going forward. So far Wallet Enhanced Fraud Protection notifications only seem to be going out to VISA card users. Why and why now?

Because it’s starting with VISA with the focus on web and in-app payments, my first thought was this is partly a response to bad publicity from the silly VISA-centric ‘Apple Pay Express Transit has been hacked!‘ story that make the rounds last October. The new Apple Pay and Privacy text outlines how the new policy applies to various Apple Pay operations: adding a card, paying with Apple Pay, using transit cards, etc.

QR Code payments in Wallet are also referenced. The official mention may indicate the long in development feature will finally see light of day, perhaps iOS 15.5, we shall see. The text says, “When you make a payment using a QR code pass in Wallet, your device will present a unique code and share that code with the pass provider to prevent fraud.” If Apple Pay delivers native device generated QR code payments without a network connection, just like all Apple Pay cards to date, it would be quite a coup.

The notification privacy text is worth reading. As of this posting the Apple Pay & Privacy web page has not been updated with Enhanced Fraud Protection information.

2022-04-22 Update
Some clarity on the reasons and timing of Enhanced Fraud Prevention: Wallet notifications went to VISA card users in various Apple Pay regions (US, Japan, Australia and more) the same day Apple switched the Apple Cash card brand from Discover to VISA debit. Kissing the Green Dot Bank/Discover backend goodbye for VISA is the smart thing to do as Apple can finally take Apple Cash international. Enhanced Fraud Prevention had to be in place first for that to happen.

The PASPY thing

HIroden, NEC and LECIP team up for the new system announcement that replaces IC smartcard PASPY in 2024 (Hiroshima Home TV)

PASPY announced today that PASPY transit IC card service ends March 2025. The official replacement has been announced, billed as the “the fist Account Based Ticketing system in Japan” (yeah right) and launches October 2024. Main PASPY operator Hiroshima Electric Railway Co.,Ltd. (Hiroden) has been thinking out loud since last May that they planned to go all in with a QR Code smartphone app. Twitter users complain, a lot, that QR will be an inconvenient pain in the butt over what they have now.

Here’s the thing, most people assume that killing PASPY card means Hiroden and Hiroshima region PASPY transit partners will rip out all the FeliCa readers and replace them with optical code readers. I don’t think so. FeliCa PASPY cards will disappear but not the transit IC readers. If you listen carefully to Hiroden’s bitching and moaning about having to shoulder PASPY system costs from the PASPY/FeliCa fare processing server side (that the PASPY partners don’t help us enough with…boo-hoo-hoo). Dump that and get out of the plastic card issue business, leave ICOCA / Transit IC readers where they are and let them handle their own fare processing, retrofit a QR scanner or install Denso Wave QR+NFC readers, toss out a QR PASPY app and the PASPY associates can call it a day.

PASPY had all the limitations of region transit cards: no e-money functions for store purchases to juice the recharge business side, slowly declining ridership, and the card could not be used on JR West ICOCA and larger Transit IC network…limitations that the Suica 2 in 1 Region Affiliate program resolves. Too bad JR West doesn’t have a similar program for the ICOCA region but it says something about JR West and local government relations that Hiroshima City and prefecture officials have kept quiet.

Nevertheless, there are way too many ICOCA and Mobile Suica users out there and Mobile ICOCA goes live 12 months from now. PASPY partners will want to keep those users riding no matter what Hiroden ends up doing. And local government transit subsidies will help keep the Transit IC readers in place. The whole point of transit is encouraging people to use it…right? And if it all works out, for QR based PASPY MaaS with Transit IC support, all the better.