The first indication that something was up the FeliCa chip supply chain came on May 31 as an small announcement from Iwatekenkotsu Co., Ltd that the scheduled last leg of their Suica 2 in 1 Iwate Green Pass bus support rollout would be delayed due to new IC reader device procurement delays.
This was quickly followed by a large joint press release from JR East and PASMO on June 2 announcing that unregistered Suica and PASMO cards, the plastic ones that people can buy in Tokyo area station kiosks, would not be available starting June 8, ‘until further notice’.
The official reason for the Suica plastic card sales suspension is ‘the global chip shortage,’ but that doesn’t sound right when there are gluts out there. These things can be complex so I asked an old colleague who specializes in chip production analysis about the situation. He had this to say:
Domestic manufacturers of non-volitive memory of the type used in FeliCa chips, reduced manufacturing capacity (the NAND market segment is currently in a recession due to overproduction and excess inventory). This reduction came when there was an increase of inbound visitors to Japan buying Suica and PASMO cards. There was also increased demand for new Suica purchases due to the expansion of the Suica area to the Tohoku region, because the population is not as large as Kanto and Kansai, demand is expected to settle down soon.
However as production capacity will not return to previous levels, there is a high possibility that supply and demand will continue to be tight to some extent.
Other Japanese sources say another factor is that all transit IC manufacturing has been sub-contracted out to Taiwan. In short, buckle up folks, it’s going to be a long bumpy ride as in addition to manufacturers cutting production due to the non-volitive memory glut, Japanese IC card production (FeliCa chip, antenna, card, etc.) has been farmed out to Taiwan. As the saying goes, they’re always a great risk putting all ones eggs in one basket, especially with chip production. But no matter how many times companies learn this hard lesson, companies soon forget.
The May 27 Tohoku Suica launch and Suica 2 in 1 Region Affiliate card launches are certainly a short term factor in the FeliCa chip shortage as JR East is still selling both registered and unregistered Suica cards in the Tohoku region and Suica 2 in 1 Region Affiliate cards. Outside of the Tohoku area JR East and PASMO will continue to sell plastic commuter passes and inbound speciality cards like Welcome Suica and PASMO PASSPORT, however inventory is tight and JR East reduced the number of Welcome Suica sales outlets on August 2.
People assume that Transit IC cards from other regions like ICOCA or SUGOCA are readily available but this is not the case as operators are quietly limiting sales outlets. SUGOCA for example is not available at station kiosks but kept ‘under the counter’: buyers have to go the nearest JR Train Reservation office and ask for it. Expect this to start happening for all regular non-commuter pass Transit IC cards as well, partly to control inventory, partly to keep scalpers from cleaning out transit cards from kiosk station machines for resale.
Why not use Mobile Suica and Mobile PASMO then? Unfortunately the mobile situation for inbound visitors isn’t great: visitors with only VISA cards or Android are basically out of luck.
The Android problem: this is the well known problem of Android manufactures who have the hardware and software in place to use Mobile Suica but kneecap it on non-JP models. Wearable devices like Fitbit and Garmin support Mobile Suica on non-JP models but Suica recharge with Google Pay has the same foreign VISA block that Apple Pay does.
So unless VISA lifts their foreign card Mobile Suica blockade, and it has been in place for a year now, even Apple Pay Suica • PASMO • ICOCA users are limited to using Mastercard and Amex cards…using cash recharge. More on that situation in later post.
The easiest solution for JR East and PASMO is to encourage domestic Suica and PASMO users to go with Mobile Suica and Mobile PASMO instead of plastic cards. They are already doing that but expect more Mobile Suica and Mobile PASMO promotion campaigns and reward point enticements. If anything, the plastic Suica • PASMO card shortage is the best Mobile Suica • PASMO promotion ever.
When the going gets tough, station staff roll out the transfer protocol information board at station entrances
The Yamanote Line was not running this morning due to a signal malfunction. Yet everything looked like any normal Monday morning commute so I had no idea when I entered Asagaya station. Then I heard the announcement on the platform. The train came, I got on. Again everything looked normal, nothing out of place, no unusual crowding. Everybody seemed to know what they were doing. As the train pulled into Nakano station the train conductor gave a helpful rundown of all the train transfer options with transfer protocol in effect closing with, “Please use smartphone and tablet train information apps to find your best route.” All the major transit apps, Apple Maps, Google Maps, et al., include real time transit stoppages and re-route automatically, but don’t always give you the best train route for the situation.
I got off and took a break in Beck’s Coffee Shop, planning a new commute route while enjoying morning coffee. It can be fun to take a leisurely commute knowing I can download a delay certificate, take any route I want, get to the office late and still get paid for the whole day. Usually I take the Yamanote Line from Shinjuku to Gotanda but decided to go via the longest route: ride to Tokyo station, change to the Keihin Tohoku line, ride to Kamata, transfer to Tokyu Ikegami line and get off at my usual station. Here are a few simple pointers for using a Apple Pay Suica or PASMO commuter pass when transfer protocol is in effect.
When a major line goes out, connecting train line companies put transfer protocol into effect so that commuters can reach their final destination. With a Yamanote Line stoppage, that’s pretty much everyone.
What is it? The Tokyo train region transfer protocol is a visual inspection re-rerouting procedure that goes into effect for commuter pass and paper ticket holders when a train line stoppage prevents them from reaching their destination via the normal route. All the connecting train line companies cooperate and allow commuter pass or paper ticket holders to travel by train and go through gates for free with a quick glance of their commuter pass or paper ticket. Regular Transit IC cards, including non-commuter pass Apple Pay Suica and PASMO, cannot be used with transfer protocol and pay regular fare at the gate.
How to use it It’s very simple: do not tap in or out at transit gates, go through the manned gate and show your Apple Pay Suica or PASMO commuter pass to the station staff. This is easy to do with iPhone Wallet as the Suica • PASMO card displays the commute route and validity dates. Apple Watch is a little tricky: bring up the card via the Apple Pay double click, tap the card and slightly scroll down so that the commute route shows.
You are still able to use transfer protocol if you have tapped in, just make sure that you do not tap out. Once you tap out, the fare is deducted and there is no refund. Go to the manned station gate, show your commuter pass and go through. At your final destination tell the station staff you tapped in and they will reset your Suica or PASMO. If you do not do this you will get a gate error when you tap in on your next transit.
As for today’s ride without the Yamanote Line? It was fun taking completely different, and much more expensive, commute route. Thanks to transfer protocol, my commuter pass covered it all. I got to my final station with so much time to spare I didn’t even bother downloading a delay certificate.
When foreign issue VISA cards in Wallet stopped working for Apple Pay In-App Suica and PASMO recharge on August 5, the first people to howl in pain were Apple Pay PASMO users who suddenly couldn’t recharge with their Chase Sapphire VISA cards. Chase Sapphire users earned 3x travel points with a PASMO recharge, long time resident Suica users migrated to PASMO when JR East and VISA shut down 3x travel points in May 2021 when VISA finally signed with Apple Pay in Japan.
After confirming that my Wells Fargo Signature VISA stopped working for Apple Pay Suica recharge, I contacted Mobile Suica support. The official line: “There should be no problem with foreign issue cards, contact the card issuer.” My next stop was Wells Fargo card services support, official line: “There should be no problem with your VISA, contact the merchant.” Entirely expected of course but Wells Fargo confirm that Mobile Suica transaction attempts were not even showing on the Wells Fago system. They said it seems to be a ‘communications issue’ which meant something is not right on the VISA payment network merchant transaction authorization side. Everything was stopping there.
An Android Suica user confirmed the same non-JP VISA problem with Google Pay Suica recharge so it was larger issue than just Apple Pay. I contacted IT journalist Junya Suzuki who focuses on mobile payments. His first thought was something was going on with the VISA Japan payment network merchant acquirer side. For reference, the merchant acquirer handles transaction authorization from the merchant side, ‘this transaction is clear to send to the card issuer.’ The issuer then clears the transaction with the customer account, ‘this customer is good to pay for this charge.’
Merchant acquirer relations are very secretive, nobody knows who is the merchant acquirer is for Mobile Suica, Mobile PASMO and Mobile ICOCA though everybody is pretty sure it is the SMBC Group who are the banking group for all things VISA in Japan. Maybe they were tightening online transaction security…or something else. Suzuki san checked his sources and had this to say:
An acquirer made the decision stopping handling cards issued in other countries…In addition, that means JRE doesn’t know what’s happening on this problem.
In a his Japanese article he described JR East as a ‘victim’ of a situation forced by VISA, their hands are clearly tied. VISA payment network and their merchant acquirer are highly selective. For example: foreign issue VISA works fine for Apple Pay in-app purchases with the Starsbucks app, but not in-app purchase with JR East for Suica recharge. If foreign VISA cards were insecure, VISA would be stopping all In-App and online transactions, but they are not. This means the ‘security concerns’ excuse doesn’t wash, it’s a ruse for something else.
Security and Apple Pay Enhanced Fraud Prevention It’s helpful to examine the impact of phishing and other security attacks targeting NTT Docomo, Line Pay, PayPay and other QR code mobile payment service users in late 2020, and JR East online service users (Mobile Suica, JRE POINT, Eki-Net and VIEW card) in early 2022. Security responses were varied and vague. Companies like to say they value customer security but hardly provide details of what they’re doing about it. Security details hashed out between the card brands, merchant acquirers and merchants are secret non-disclosure territory.
Japanese credit card issuers responded by upgrading to EMV 3-D Secure v2 (3-D stands for three domains: the merchant acquirer domain, the issuer domain, and the interoperability domain), for non-digital wallet browser and mobile app payments. EMV 3-D Secure is the EMV e-commerce browser and app authentication tokenization specification with the card brands using their own naming and implementing merchant support in their respective payment networks.
In addition to adding 3-D Secure v2 in their Mobile Suica and Eki-Net apps, JR East has beefed up security to fight Mobile Suica phishing attacks with tighter monitoring of Suica App recharge with the app registered credit card…not Wallet In-App recharge. It’s important to understand this key point:
3-D Secure has nothing to do with Apple Pay and Google Pay, they and all other digital wallets like Samsung Pay, Huawei Pay, etc., do not use it. They have their own tokenization scheme. This is a common online misconception. Japanese issue VISA (and everything else), foreign issue Mastercard and Amex cards work for Apple Pay Suica • PASMO • ICOCA recharge without problems, without 3-D Secure.
Domestic security issues do not apply to inbound visitors adding and using Suica cards in Apple Wallet. They do not use Suica App or have a Mobile Suica account. And yet VISA seems to be using domestic security problems to block foreign issue cards for Apple Pay In-App recharge.
JR East is fighting phishing attacks with stricter Suica App registered credit card use monitoring. Suspicious can lead to Suica being locked by JR East for credit card recharge but cash recharge is always available.
The tokenization that Apple Pay, Google Pay, Samsung Pay and similar digital wallets use is highly secure, some say more secure than EMV 3-D Secure tokenization. Despite this, Apple has been making some changes to Apple Pay to enhance security for online and in-app purchases, at the behest of VISA. Apple Pay quietly launched Enhanced Fraud Protection in April 2022 when Apple Cash switched from Discover to VISA. The updated Apple Pay and Privacy text added a new section:
For cards with certain enhanced fraud prevention, when you attempt an online or in-app transaction, your device will evaluate information about your Apple ID, device, and location if you have enabled Location Services for Wallet, in order to develop on-device fraud prevention assessments. The output of the on-device fraud prevention assessments, but not the underlying data, will be sent to Apple and combined with information Apple knows about your device and account to develop Apple Pay transaction fraud prevention assessments. These transaction fraud prevention assessments may be shared with your payment network, together with a shipping address identifier and IP address if available, in order to prevent fraud at the time of transaction. The shipping address identifier differs per payment network and may be used to confirm whether shipping addresses for different transactions using a particular card on your device are the same in a way that does not reveal the underlying address. You can check whether a card has this enhanced fraud prevention at any time by going to the back of your payment credential in Wallet. To prevent the sharing of fraud prevention assessments with your payment network, you can select another card.
This means that Apple Pay ‘might’ share iPhone/Apple Watch location information when making online or in-app purchases. So far VISA cards are the only ones that have Enhanced Fraud Protection but it doesn’t seem to apply to all VISA issue cards and it’s hard to tell which VISA cards use it.
Does enhanced fraud prevention have anything to do with Apple Pay Suica and PASMO recharge not working for foreign issue VISA? The short answer is no, but it’s a background development to be aware of because: 1) it’s limited to online and in-app purchases, 2) VISA is pushing for ‘fraud prevention assessments’ so they could obtain device location information and more. Only after VISA started pushing this agenda did we start having recharge issues with Apple Pay In-App payments.
The VISA open loop power play So we circle back to foreign issue VISA use in Japan again. Why are cards cleared for Apple Pay, cards that worked fine up until August, not working? The timing is perfect when you also consider that VISA is heavily promoting ‘VISA Touch’ EMV contactless and open loop transit in Japan as a challenge to the home grown FeliCa based Transit IC card system. It’s very convenient for VISA Touch open loop marketing purposes when Apple Pay Suica and PASMO are kneecapped as easy payment and transit options for inbound visitors.
VISA has a history of not playing nice with Japanese stored value cards on mobile and not playing nice with Apple Pay. Japanese issue VISA cards didn’t work for Apple Pay in-app purchases and Suica recharge until May 2021, VISA waited 5 years to ‘resolve’ that issue. VISA cards still do not work with Mobile WAON and Mobile nanaco on Android and Apple Pay, they likely never will. My take is that VISA is not happy with people using VISA cards like an ATM to move money into stored value prepaid cards for making payments, earning points, etc., that are not VISA.
Examining VISA’s moves in the Japanese market proves one thing: payment network issues are never simple or solved quickly because they often come down to market politics. VISA has never played nice with Apple Pay in Japan since the very beginning, they continue to do so. At the very least we can mark this down as another skirmish in the ongoing digital payment turf wars.
This post was originally posted 2022-08-08 and has been updated to reflect a changing situation. The post date reflects the latest major update.
Non-JP VISA cards cannot be used for select Apple Pay In-App payments for Mobile Suica, Mobile PASMO, Mobile ICOCA and Apple Pay online payments for Universal Studio Japan advance tickets.
Mobile Suica has had a rough 2 weeks. On June 24 a construction error during server center power supply expansion work left JR East Mobile Suica and Eki-Net online reservation services offline for 12 hours (0:00~12:00). It was an embarrassing mishap but the actual damage was small, limited to refunding Eki-Net ticket holders who couldn’t change ticket reservations. Mobile Suica was offline so no refunding was necessary because nobody could use the Mobile Suica credit card recharge service. No need to refund what people can’t buy.
A shorter but much more problematic outage happened on June 27. Media mistakenly reported that Mobile Suica was down but this was not the case as Mobile Suica on Android was working just fine. It was an Apple Pay problem: Apple Pay servers went down from heavy demand on Apple Pay ICOCA launch day, taking down not only Apple Pay Suica recharge but also PASMO, ICOCA, nanaco, WAON, Octopus, China T-Union, adding credit cards and other Wallet services worldwide. As the outage took place during the Japanese business day, JR East had to refund iOS Suica App users who attempted to buy or use Suica Green Car tickets during the Apple Pay outage.
Just as things were settling down, another even shorter 40 minute period of trouble occurred on July 8 at 12:00~12:40 JST. Again the media reported that Mobile Suica was down, again they were mistaken, and again it wasn’t an JR East or Mobile Suica problem, it was a much wider, and unreported, EMV credit card payment network outage. EMV transactions on readers everywhere were not responding, and they were not working for Apple Pay or Google Pay. However FeliCa payment network cards were working.
And finally there was, yet another, Apple Pay and Wallet outage on July 14 from 17:45 to 18:30 JST, with another round of Japanese media bashing poor old Mobile Suica without checking for the wider Apple Pay outage.
Mobile Suica caught the media blame because they were the only company duly reporting the problems on Mobile Suica support SNS services. JR East never lays an outage blame on Apple Pay, or any other service partner because they know Mobile Suica users don’t care, they only want to know when things are not working and when they will be fixed. This is the way it should be done because they are giving their users fast, accurate, service information…even if that means they have to take the media and SNS blame that comes with it.
But despite all the Mobile Suica outages including the EMV payment network one, the Suica card itself always remained working, both digital or plastic versions. As long as there is money on the card it works for transit and payments, and cash recharge is available 24/7. This is an under appreciated but very important aspect of the Transit IC system: there is always a non-network fail safe cash backup. Japanese never put all their household finances in one basket, cash is always the one thing that works after an earthquake, typhoon, natural or manmade infrastructure damaging disaster strikes.
In the EMV credit card payment network outage there was, without doubt, unreported trouble with open loop system test deployments on Nankai, Fukuoka Metro and other QUADRAC • stera transit operated systems, which all open loop systems in Japan use: it’s the only open loop player in town.
Unlike Mobile Suica however, when the credit card payment processing network goes down, open loop doesn’t have a fail safe cash backup. And while that’s not a problem now with small installation test sites and a tiny user base, it will be when open loop goes big time. The transit companies deploying open loop have an obligation to take care of their customers, but will they take JR East-like responsibility when QUADRAC goes down, or stera goes down, or NTT Data CAFIS, payment processing centers, or mobile carrier networks? Because believe me they will. All highly connected, interdependent networks do. That’s why we always need alternative methods and networks. Too bad that VISA is working to remove the non-EMV transit gate competition in Japan.
On June 24 from 0:37 to 13:00 JST, JR East online services went offline due to a mishap during server center related power supply construction. The most affected services were the most used: Mobile Suica and the online Eki-Net ticketing service. While embarrassing and inconvenient, the damage was minimal due to the fact that the outage was like a very long nightly maintenance when all services are taken offline. There were no transactions taking place, all JR East had to do was refund Eki-Net ticket holders who couldn’t change their reservations.
A shorter, far more problematic and partial Mobile Suica outage happened on June 24 from 16:24 to 18:00. Despite media reports screaming ‘Mobile Suica is down again’ even though the system itself was running just fine on non-Apple devices. This one was not on JR East or the Mobile Suica system: Apple Pay servers crashed from overload on Apple Pay ICOCA launch day taking down Suica, PASMO, WAON, nanaco, Hong Kong Octopus and other Apple device Wallet services worldwide. JR East had to refund iOS Suica App iPhone users who could not use purchased Suica Green Car Tickets.
Media reports were sensationalistic and misleading, claiming the outage left “Suica App Users Stranded at Ticket Gates“, when Suica App has nothing to do with using Suica at transit gates. Naturally, social media sites and online media report comment sections were full of ‘I’m going back to plastic Suica’ comments. Yahoo News Japan even ran a brain dead hack piece from GIZMODO entitled, “Be prepared: carry an unregistered plastic Suica in case of Mobile Suica problems,” recommending unregistered Suica because, ‘anybody can share it.’ Both writer and editors failed to notice that unregistered plastic Suica and PASMO cards are currently unavailable due to chip shortages.
Here’s the thing, plastic Suica isn’t ‘safer’ than Mobile Suica. It’s a myth:
Myth: I can’t use Mobile Suica if the service is offline.
Reality: Offline Mobile Suica doesn’t affect card use, the situation is no different than nightly offline maintenance.
You can always use Mobile Suica for transit and payments, it is no different from plastic, a Mobile Suica outage doesn’t affect that. The only real impact is the mobile exclusive credit/debit card recharge function and this is no different than the nightly maintenance when Mobile Suica goes offline. It’s the same situation for Mobile PASMO and Mobile ICOCA. Cash recharge is always available 24/7 at any convenience store, Seven Bank ATMs which are everywhere, and mobile friendly station recharge kiosks which are also everywhere these days.
The only real outage inconvenience was for people who wanted to buy Mobile Suica extras: Suica Green Car Tickets, Suica Day Passes, or those who might need to renew their Mobile Suica commuter pass…on a Saturday.
Which brings us to the GIZMODO Japan hack piece that was full of outrageous misinformation, implying not only that Mobile Suica on smartphones automatically stops working in an outage, you can simply go out and buy a plastic unregistered Suica card and use it anytime you need to. This is exactly the same stupid shit that bloggers posted back when Apple Pay Suica launched in 2016: carry a plastic Suica in case your iPhone battery dies.
Even in the days before Power Reserve Express Mode iPhone, this was, and is, completely wrong ‘advice’ that will get you in trouble at the transit gate. Never forget the golden Suica rule: the same Suica card used to tap in must be used to tap out to complete the transit. If the Suica you started out with cannot be used at the exit gate, for whatever reason (lost card, dead device, etc.), the gate alarm will sound and you must pay full fare in cash.
I’m sure there will always be people who believe that plastic Suica is somehow ‘safer’, and that’s fine, but the reality is that with JR East reducing station staff wherever they can, finding real people to deal with plastic card problems will be more of a hassle than it already is. In the future JR East will tell plastic card users to use online assistants at station kiosks. In the end it will be faster to do it all on mobile.
The GIZMODO JP Suica hack piece is credited to ‘amino’ who doesn’t have clue about using Suica
You must be logged in to post a comment.