The open loop mobile connectivity challenge

The recent additions of stera transit (Visa-SMBC-Nippon Signal-QUADRAC) open loop test systems in Kyushu covering Fukuoka metro, Kuamamoto city transit and JR Kyushu expand the VISA Touch transit boutique deeper into western Japan territory. Open loop based cloud processing advocates like to portray these developments as proof that local processing based FeliCa systems like Suica et al. are expensive bygones due for replacement.

There’s just one little problem that open loop advocates fail to mention: mobile connectivity, aka the Suica app problem, the QR Code payment problem, the Smart Navigo HCE problem, etc. Wide LTE and 5G deployment doesn’t always mean reliable mobile and internet connectivity mobile payment apps depend on, and carrier outages can bring down the transaction processing side of the equation. This was proven, yet again, on July 2 when major carrier KDDI suffered a massive nationwide outage that lasted for 80 hours. Let’s make a quick reference graph for examining local processing vs cloud processing in the mobile era.

Stera is a mobile based payments platform that does away with the NTT Data Cafis dedicated backbone and replaces it with the internet based GMO Payment Gateway. The weak point of course is that since mobile powers the gate reader side, when mobile service goes down, stera gate readers stop working. As everybody found out during the KDDI network meltdown, Mobile Suica kept right on working on the transit gate and the store checkout reader, while mobile app based code payments and point systems all stopped. Some vital services that depended on KDDI connectivity like ATM networks also stopped working.

Cloud based Suica will face these challenges too when it goes online in March 2023. The only difference being how much local processing stays intact and how much system buffering there is (how much it needs to talk with the cloud server to do the job), we shall see. Which brings me to the point I want to make. The media almost always portrays the open loop/cloud vs closed loop/local match as a winner takes all, one size fits all proposition. As the KDDI meltdown proves, this is stupid, and dangerous. Never put all the eggs in one technology basket. I don’t think the risk will go away, not as long as telecommunication company corporate structures don’t foster and promote their engineering talent (the people who actually make things work) deep into the executive decision making forums.

Open loop in Japan is geared for inbound tourists the supplements, but does not replace, the old reliable Transit IC infrastructure which is evolving and reducing costs too. They compliment each other, address different needs and uses. One size doesn’t fit all. If it did, Oyster card would have died years ago.

Cashless is fast and convenient? Point app mania reality check

My partner wanted to pick up some cheap t-shirts on bargain sale at Uniqlo yesterday. The Asagaya station building Beans shopping mall has all the latest cashless options but very bad network service so Uniqlo checkout was a comedy routine. First he brought up the Uniqulo app to get Uniqlo points, then I brought up my JRE POINT app to earn JRE POINT, then he finally paid with QR Code dBarai (docomo). But for each app launch and load we had to run to the store entrance to capture enough network connection for the apps codes to load. The staff is very used to this and suggest customers do so when apps didn’t load, patiently folding clothes while they run back and forth. I asked the cashier if this happens all the time. She smiled and nodded. “Cash is probably faster isn’t it?” She smiled and nodded.

Gosh, just when we thought cashless was going to free us from the so called inconvenient drudgery of cash along came smartphone reward point apps that bog down the whole cashless checkout experience, neatly killing off the supposed time saving advantage. You stand in line while the checkout customer fiddles with smartphone, digging around in an app to find the right coupon code thing. You feel smug until it’s your turn and the networks sucks, the discount coupon doesn’t load and bam, you’re holding up the line too. It has gotten to the point where Nikkei XTECH has provided an Apple Pay help article for faster checkout that explains the benefits of using Apple Value Added Services. Will Apple Pay VAS dPoint and Apple Pay VAS PONTA really help us? Probably not as they only work at LAWSON.

There is another checkout trend I see recently. With price increases everywhere people are using cash a lot more, even at places like in-station Beck’s Coffee Shop. Every customer has a Suica but more young people are keeping it in their pocket and plucking down ¥10,000 yen notes for ¥300 ice coffee. Why? I think it’s Kakebo culture at play, it’s easier to budget with cash payments and the small slightly inconvenient physical routines that accompany it. It’s not about doing everything with cash, but good old tsukae-wake compartmentalization helps keep focus and tamps down the impulsiveness when doing everything cashless. Another way of spreading the risk in these uncertain times.

VISA Japan and Apple still at odds over VISA Touch debit cards

Mastercard but no VISA

The April 19 launch of SBI Neobank Mastercard debit card support for Apple Pay was a bit unique: the first time that a plastic issue Japanese debit card came to Apple Pay and the first Apple Pay Japan debit card supporting the FeliCa iD payment network. Another interesting aspect is that only the Mastercard version supports Apple Pay, the VISA version is plastic only with VISA Touch (EMV contactless) support.

There are plenty of bank app issue digital only debit cards from JCB, Mizuho, MUFG and others on Apple Pay. These all work on JCB’s QUICPay (FeliCa) and J/Speedy(EMV) payment networks. Apple Pay Japan supports many different mobile payment network cards thanks to Mobile FeliCa support, by far the largest selection of Apple Pay payment networks in the world: EMV (VISA, Mastercard, AMEX, JCB), iD, QUICPay, Suica, PASMO, nanaco, WAON. But VISA issue debit cards are not supported even though there are many, not a single one on Apple Pay.

Wasn’t this taken care of by the May 2021 Apple and VISA JP agreement? For credit cards yes, one year later they are still at odds over FeliCa support in debit cards. VISA Japan brand debit cards are VISA Touch EMV contactless exclusive, single mode cards. VISA JP credit cards are dual mode EMV/FeliCa for plastic and smartphones, but not debit cards. We don’t know the reason but debit cards deifintely fit the budget customer category while credit cards come with credit checks, perks and card membership fees for upscale cards.

As an easily available budget card, VISA cuts costs by dumping the dual mode EMV/FeliCa IC chip and transaction fees for the convenience of using FeliCa iD/QUICPay payment networks. In other words VISA keeps all transaction fees for themselves while marketing the shit out of VISA Touch as the greatest thing since…whenever.

All of the other card brands in Japan have dual mode NFC as standard. Not VISA, they’re playing the long game of eliminating FeliCa payment network competition. This stupid polarizing single flavor NFC position only served to give QR Code payment networks (PayPay, Line Pay, etc.) a huge opportunity that they smartly played. End result: more payment network competition than ever before.

Apple on the other hand has a very simple rule for all Apple Pay Japanese issue cards: they must support FeliCa and all EMV cards are global NFC dual mode. Was this the price for adding FeliCa support to Apple Pay? Perhaps, I think it’s more to do with the Apple Pay vision of removing complex and confusing hardware choices, the Google Pay Japan mess, for standard ‘just works everywhere’ NFC. Has this been successful? Very...just ask Suica.

USA transit fare system evolution

Reece Martin posted an interesting video, So you built the wrong transit system, that examines the American penchant for building cheap light rail systems that don’t make long term sense. Public transit is a waste of money to Americans with money, so cheap is only way to fund and build public transit infrastructure. The problem is this cheap short term thinking costs more money in the long run. It’s a ‘one size fits all’ mentality.

But as Reece points out, systems can evolve from humble beginnings. Many private Japanese rail lines started out as street trams (that evolved from horse trams) but evolved into the heavy duty regional rail lines we have today. Fare system have evolved too, from paper, to mag strip, to IC smartcard and now mobile devices.

Transit fare systems in America suffer from the same short term cheap thinking, on full display on the MTA OMNY system, the world’s first EMV only open + closed loop fare system. When it’s completed in 2023, barring more delays, MTA will have farmed out every aspect of their fare collection and OMNY transit card issue to banks.

Not to rehash points I already made about OMNY, but Reece’s wrong transit system analogy struck a chord. And unlike rail system evolution, once the transit fare system in locked into the bank payment card infrastructure, from technology (EMV) to payment network processing (VISA, mastercard, AMEX, etc.), it will be extremely difficult, if not impossible to change anything later on.

But why is America so short sighted when it comes to public transit, never investing in a long term self-sustaining viable business model? I ran across an interesting take that explains it neatly. The USA will never have a transit platform business because public transit is a welfare and jobs program, not a self-sustaining business model:

Public transportation in the US is generally very bad and very heavily subsidized. It’s cheap because extremely little service is being run, and the government picks up most of the bill.

Public transportation in the US is less of a way normal people get around, and more of a welfare program and jobs program. Even in places where public transportation is a way normal people get around, e.g., NYC, it is run more like a jobs program than an essential public service.

Reddit user Sassywhat

Open loop fare systems are also vulnerable in new ways nobody predicted: imagine the mess if payment networks go down in a cyberwar, à la the Moscow metro when digital wallets and bank payment card networks were suddenly and omniously turned off. In the case of OMNY where, unlike Moscow metro, everything is EMV payment networked…there is no backup in-house payment settlement system, there is no plan b.

In other words not only is OMNY EMV one size fits all, it’s all or nothing.


Related post: Hidden Assumptions

The digital wallet service suspension front line

That was quick. When I made the above table for mobile wallet chokepoint, there was no indication we’d get EMV confirmation so quickly. Many were quick to applaud sanctions against Russia to stop the war with Ukraine, and while stopping war is always the right thing to do, hurting citizens is never the right thing to do. Turning off basic digital wallet services should give people pause. What is easily done in one place can be easily done anywhere.

It’s also not clear cut how it is being done. Is Apple turning off select Russian bank services in Wallet or turning off select payment applets in the Apple Pay secure element, or turning off Wallet for Russian Apple ID users? Most likely the first but there’s no way to be sure and there is no way that Apple or Google will ever tell us.

Long lines at Moscow Metro transit gates are not so clear cut either. Open loop isn’t standard on all transit gates, most them being Troika transit card only, and according to a Twitter follower, physical Troika card only, not Google Pay/Samsung Pay Troika which only rolled out recently. If so this suggests the (so far only one) picture of long lines could be due to Troika system issues instead of Apple Pay/Google Pay/Samsung Pay, hacking, or something else.

VISA and mastercard soon followed and cut their services in Russia. Many people in Japan noted how easily all this happened and expressed their distrust, saying they would think twice about using digital wallet services from Apple and Google. Many also noted the importance of Japan having it’s own FeliCa technology and FeliCa based e-Money payment network

The value of non-EMV native payment networks controlled and operated by native companies should be clear to everyone by this point. Always, always have a backup plan. One thing is certain, warfare that attacks basic public service infrastructure like transit and digital wallets, far and away from any front line, is the new ugly reality.