It took me a while to fully appreciate the issue that Twitter user Yoshimasa Niwa was describing. At first glance I and many others assumed that setting Japanese over English would solve his app library sorting issue.
Then I realized that wasn’t his point at all. The software app in the screenshot is the Yahoo Japan ‘Norikae Annai’ transit app, one of the most popular free stand alone transit apps in Japan. I use it all the time. It’s a Japanese app with a Japanese name but the basic iOS English sorting algorithm ignores this and assumes all Chinese characters used everywhere must follow modern mainland China’s Simplified Chinese rules for reading and sorting.
This is ridiculous as assuming that all Roman based character sets everywhere must follow modern Italian reading and sorting rules. I always find that westerners assume the Kanji culture flow was always one way from China which it is not, with different and unique readings, usages, and Japanese Kanji like shitsuke 躾 traveling the other way over the centuries. The same is true for other cultures that adapted the Chinese writing system for their languages.
It amounts to cultural destruction by neglect and ignorance by large western based technology companies who think things are ‘good enough’. Or are just bugs to fix in a later software update that usually never appears. Modern computer software has pretty much destroyed traditional kanji culture publishing this way, with many countries abandoning mainstream traditional vertical text layout for western style layout because ‘it’s easier’, i.e. western tech companies couldn’t be bothered getting Asian language typography right. All these years later web browsers still can’t do vertical text worth a damn.
A veteran Japanese font engineer whose entire career was devoted to preserving high end Japanese typography in the digital age recently told me, “I don’t think anybody cares anymore.” In the end it all too often comes down to this: I don’t care cultural death by I don’t care companies who have the money and power to care.
That’s bitter irony in our age that purports to champion cultural diversity.
2020 is the coming out party for Apple designed OpenType variable fonts, both the SF Pro and SF Compact system fonts and the all-new New York font shipping in iOS 14, watchOS 7 and macOS 11. The Apple created variable font technology is not new of course. It has been around since the QuickDraw GX days along with the TrueType GX enhanced Skia font. It was due to be standard in MacOS Copland system fonts including a Japanese variable font created by FontWorks. Then Steve Jobs returned to Apple and everything changed.
Yes, it has taken 25 years for an Apple created technology to make it into the basic system. It proves my long stated belief that font technology doesn’t matter unless it is a standard feature built into every nook and cranny of the OS foundation. The TrueType GX Skia variable font has been with us all this time, but only matters now because the SF Pro system font has gone variable.
Why is It Taking So Long? iOS 14 and macOS 11 variable font basics are covered in an excellent WWDC20 video, ‘The Details of UI Typography’. It’s important to remember that while OpenType variable font technology is ‘world ready’, at this stage they only apply to Roman based font sets. It’s going to be a long time before we see a Japanese language system font in variable format.
There are many reasons. In the WWDC20 video Loïc Sander of the Apple design team drops a big hint when he explains that while digital technology (PostScript fonts) “gave us a lot more flexibility in handling text,” it also “made typography a bit more crude than it used to be.” The statement shows how clueless designers and engineers outside of Japan can be about Japanese fonts and typography.
While a ‘bit more crude’ might be true for Roman based fonts and text layout, PostScript fonts completely broke traditional Japanese font design and composition models. Everything was thrown out because Adobe made no accommodation for Japanese Kanji based font metrics. Western created DTP layout is graphics-driven and calculated by margins and font baselines.
The western baseline typography model and font metrics is how PostScript and OpenType fonts, and all layout engines evolved. Adobe was well acquainted with the shortcomings of their own font technology and InDesign J got around the problems by adding proprietary Kanji virtual body font metrics and Japanese line break algorithms. None of this exists as an open standard that benefits everybody.
Because of this situation Japanese DTP forced users to adapt to limited font technology rather than technology solving their production problems. I know this because everyday at work I had to deal with the endless problems and limitations of Japanese PostScript fonts that could only reside on the output device.
Another big problem was that Adobe relations with Japanese PostScript licensees in the 1990’s was not healthy. Adobe stuck with closed print device font licensing for far too long and discouraged independent font production wherever they could. Because of this situation, digital font progress in Japan was slow and very expensive.
Here are some challenges facing Japanese variable fonts.
Once Upon a Time One basic flaw of OpenType outline font technology is that it’s extremely inefficient for kanji glyph production and storage. Every glyph has to be created and stored separately and doesn’t scale well. This is why OpenType CJK fonts on tiny devices like Apple Watch are a match made in hell. One solution to this problem is stroke fonts. Stroke fonts use a library of basic glyph parts to efficiently create complex glyphs.
Stroke fonts are a perfect fit for kanji font production and for small constrained devices like Apple Watch because reusable parts don’t take up precious resources. On the desktop, stroke fonts can do weight variations over the full range from Light through Ultra Bold without losing typographic details, all in a single 4 MB font while an equivalent OpenType variable font can weigh in around 18 MB.
The technology has been around for a long time and was supported up until macOS 9 but lost out when Apple quietly dropped the QuickDraw GX derived Open Font Scaler architecture in the migration from classic to macOS X.
While stroke fonts are not supported in the current Apple OS lineup, on the font tool side stroke font technology has appeared in software such as the classic MacOS Gaiji Master from FontWorks. The lead engineer of that effort is currently working independently on a similar gaiji glyph tool for Windows based on stroke font technology that is much more advanced than the old and long unavailable FontWorks software. I plan to cover developments in a future post.
The Japanese Font Production Challenge The Hiragino iOS/macOS Japanese system font was not created by Apple, it was licensed from Screen Holdings (SH), originally created by independent font design studio Jiyukobo in the early 1990’s. There is much more work involved creating a Japanese font compared to Roman based languages. Hand drawn glyphs are created, scanned and cleaned up for digital production.
The Adobe Japan 1-7 glyph collection requires 23,060 glyphs for a single weight, multiply this work by the different weights for one family and you get an idea how massive the undertaking is. From Osamu Torinoumi, one of the key designers of the Apple licensed Hiragino font on its creation:
On average, one person would (hand) draw 12 or 13 glyphs a day, which is not much change of pace from the days of creating block type…the whole process, from start to finish, took three years.
One might think that a single CJK (Chinese-Japanese-Korean) font sharing a common design can streamline the process but this is a huge misconception. Each culture has centuries worth of different design aesthetics that good design must incorporate: what looks good to a Chinese designer and works well in a Chinese text design, looks terrible in Japanese context. I have yet to see a decent digital ‘kana’ design from a Chinese font designer. Osamu Torinoumi on the differences in creating the Simplified Chinese Hiragino Sans GB:
“We worked with the Adobe GB 1-4 character set (29,064 glyphs) at 2 weights. Basically we had to finish one weight in 6 months. One year for the entire project. At first we only thought we would be there as backup, but Screen kept passing us all the questions from Beijing. It turned out to be a lot more work than we anticipated.”
Jiyukobo sent all the original Hiragino design data to Hanyi Keyin through Screen and they adapted the designs for China. Torinoumi said that one of the major differences is that Chinese design demands that Gothic (sans serif) characters mimic handwritten style. This means the character should be slightly off center within the virtual body. “Even after the project was over I still didn’t understand the difference between Japan and Chinese “Kokoro” glyph which the Chinese designers insisted were different.”
The Variable font UI Challenge Finally we get to a problem on the Apple OS platform side that has been around since the GX days: how to present advanced typography features in a useful and easy to understand system UI that works everywhere. What works on macOS obviously won’t work on iOS, but iPad OS will need some degree of advanced typography feature access. Sliders have their place but I agree with Adobe Type Senior Manager Dan Rhatigan who made a very good point in his TYPO Talk 2016 presentation: there has to be a better UI control concept out there.
Dear Apple, didn’t Adobe tell us not to use sliders?
This is because there are many more OpenType Japanese variable font features than just weights. There are gylph variations, vertical layout variations, horizontal and vertical compression for tatechuyoko instances. In macOS Catalina these are hidden away in the crusty old Font Pallet that is desperately in need of a major overhaul. Please tell me that macOS 11 fixes this or that Apple has a vision how to.
Oh where can my glyph variations be?
Japanese typography is unique in that it has preserved its own print ‘moji bunka’ cultural history and vision that China and Korea have largely abandoned in the face of western centric computer culture that often pretends to care about such things, when it really doesn’t. If western developers cared about good typography for everybody we would have vertical text in web browsers that actually works. I hope a rich text culture can be preserved and conveyed to future generations even in such small details as a well designed and executed Japanese variable font for computers and smart-devices.
Japanese Typography and Font Posts
This is a collection of long form Japanese typography posts. They were written as stand alone pieces, so there is some background explanation overlap, always a weak point of the blog format.
The last one is a small step with big implications that people are only beginning to see. Instead of a plastic card that is read into Wallet or added to Wallet with a 3rd party app, Suica creation is now a basic function of iOS 13 Wallet that works with other Apple Pay cards from anywhere to add money. Suica is part of iOS. Think about that.
All of these developments have been driven by Suica, this is why Suica is the Apple Pay bellweather. Transit card creation in Wallet and adding money with Apple Pay cards from anywhere will undoubtedly be part of Apple Pay Octopus and migrate to other Apple Pay Transit cards over time. If you want to see where the Apple Pay puck is going, keep an eye on Apple Pay Suica developments. What arrives on Suica first becomes standard later.
while QR codes have proven remarkably effective at meeting most people’s mobile payment needs, it seems ill-suited for public transit compared with NFC. Since NFC relies on radio waves, payment requires only a tap of the phone. There’s no need to wake it up or turn the screen on, making it as convenient as traditional transit cards.
The limitations of the <QR> technology are apparent even as cities race to install QR code scanners in turnstiles across the country. Over time, though, the inconvenience might be enough to nudge China away from its reliance on QR codes.
The long answer requires a quick look at transit gate technology evolution. The success of Suica can be found in its development process, a fascinating story by itself. The Suica card and transit gate were developed as one thing to replicate the ease of flashing a commuter pass to the gate attendant without stopping.
A video of old style paper ticket manned gates illustrates the start point. There is no physical barrier. People slow down to get their ticket punched but rarely stop. For a commuter pass the user flashed a wallet with a clear plastic window at the attendant and kept on going, shown at the 0:16 mark:
Shigeo Miki came up with an idea of using IC cards for tickets. The magnetic-type ticket automatic gates, which were in use since the 1980’s, had some inconvenient aspects. Old-style passes could be shown to attendants without being taken out of their cases. But to use automatic ticket gates, passengers had to take them out, pass them through the automatic gate, and then put them away again. He thought that was a decline in service quality…
This was the late 1980’s when IC cards were just coming into wider use, but not for transit. The Suica project had a large impact on Sony FeliCa development as did the Hong Kong Octopus project starting in 1992…
Furthermore, systems that read ID data from read-only cards and interact with the main computer each time someone goes through the ticket gate could not keep up with the enormous volume of data processing transactions in rush hour. So Miki and his fellow researchers perceived that the cards must be read/write types.
There we have it, the Suica project goals were: open gates, waving commuter passes, local processing. Magnetic strip paper ticket gates got faster, Omron states the speed is within 600 Milliseconds (MS), and better with the ability to handle and sort multiple tickets at a time. Suica is cool but nothing is cooler than watching the physical action of a well designed machine:
Despite development problems and a low research priority within JR East at the time, Suica success was achieved by moving the battery supply from the card to the gate and creating fast reliable performance with an illuminated target NFC ‘hit area’ tilted forward at 15 degrees, the same design you see today on the JREM EG-20 transit gate. The EG-20 already looks surprisingly similar to the open public transport gate concept. (Here’s a Japanese website that catalogs every JR East ticket device if you are interested)
Smartcard Transit Gates Compared Smart transit cards were an important development that revolutionized transit and launched successful systems such as Suica, Hong Kong MTR Octopus and TfL Oyster. However all smart transit gates are not equal. Compare the Malaysia Touch n’ Go gate speed with Suica on EG-20:
One of the commentators notes the crucial differences: FeliCa (used for Suica and Octopus) is the most efficient NFC protocol, 212 kbps minimum/847 kbps maximum, while Touch ‘n Go is mainly MIFARE Classic at 106 kbps an “early form of ISO 14443A, …the least efficient NFC protocol.”
There is another crucial difference: Japan transit gates are open by default and close only when needed, just like the old manned JR gates, while Malaysia and Hong Kong gates are closed on default or use old fashioned turnstiles. The combination of the Ferrari fast FeliCa combined with the well designed JREM EG-20 gate (and variants) that is default open, keeps people moving, best highlighted in a Pokemon Go event ‘Pikachu’ transit gate video:
Suica speed is part of what makes it fun but there is a serious reason behind it: major Japanese transit operators like JR East have to move a tremendous volume of people through a fixed station infrastructure space that cannot be enlarged. Bigger stations with more transit gates are not an option. So the system focus is using the fixed space infrastructure as efficiently as possible. That is why the Suica transaction speed is less than 200 MS, that is why a Suica transit gate must clear 60 people a minute.
Open Loop Multiple Protocol Transit Gates Compared Using EMV contactless with cards and smartphones, or QR Codes on smartphones for transit instead of native transit smartcards, is a step backwards from the fast read/write local processing model of Suica, and back towards read only centralized processing, one of the original system bottlenecks that Suica was designed to avoid. The QR Codes used for transit in China appear to be particularly slow and a poor match for high traffic stations. Poor gate design is certainly a factor here.
— 山谷剛史 新刊『中国のITは新型コロナウイルスにどのように反撃したのか?』 (@YamayaT) January 19, 2019
EMV has its own transit gate problems as well, as Singapore transit users found out in the recent rollout of EMV SimplyGo service there, things slow down:
It’s fascinating that Singapore’s Land Transport Authority (LTA) dumped the fast FeliCa (rated 200 millisecond transaction but Octopus clocks in at around 100ms) behind EZ-Link cards to roll their own faster CEPAS technology (rated 180ms transaction) but are now letting super slow EMV contactless (500ms plus and counting) on their transit reader infrastructure. It’s like ripping out all the cutting edge transit gate technology and replacing it with clunky old supermarket cash register technology.
The last comment in the first Twitter timeline is an important observation: most EMV transit is simply grafted onto the current transit gate infrastructure which was designed for something else, a factor contributing to unreliable performance, forcing users to adapt. Most of the multi-protocol transit gates in service are poor design. Central processing EMV supermarket checkout technology was never designed with transit in mind, so we get poorly designed technology bolted on poorly designed transit gates.
This leads to another EMV issue users have to adapt to: ‘card clash’. When EMV is bolted onto an existing system slapping a wallet on the transit reader doesn’t work anymore, the card has to come out of the wallet. This is still one of the nice things about plastic Suica cards. Young Japanese women in particular seem to enjoy slapping those cute little Hello Kitty wallets on the gate reader with a surprisingly hard thwack, stress relief perhaps? Chicago Ventra support offers insight on the current state of EMV transit:
Get your device ready, first, for fastest entry
“Card clash”: touch only your desired payment method
Multiple credit cards: always use the same card on the same device on transit readers
These are issues that Apple Pay EMV Express Transit is designed to fix by designating a single EMV bank card for transit but it cannot change the inherently slow EMV transaction speed or solve the limitations of EMV bank card architecture which is basically centrally processed read only. There are limits on how much the central processing read only model can achieve when fast, precise, local transactions are required. All EMV Express Transit does it designate a bank card that tells the transit gate reader: I am a real bank card, not a forged one, we’ll settle the bill later.
Japanese IT journalist Junya Suzuki tests OMNY transit gate speed…
and reliability
The China Situation
The Abacus article highlights multiple protocol Chinese transit gates: paper tickets, NFC, QR Codes and Face recognition. Oh, and closed by default gates. This is not a fast transit gate environment.
The Abacus article points out the slow uptake of NFC, blaming it on UnionPay, but it boils down to the PBOC flavored EMV spec itself:
Each card organization has formed its own specifications based on the EMV specification based on its own business refinement and expansion, such as China UnionPay’s PBOC 2.0 specification, VISA’s VSDC specification and MasterCard’s M/Chip specification. Each specification follows the EMV specification for basic transaction processes and security mechanisms, but differs in terms of data element definition and extended application…PBOC based on the EMV standard, combined with the needs of domestic banks, the People’s Bank of China promulgated the PBOC series of standards: 1 PBOC1.0: e-wallet / electronic passbook / magnetic stripe card function 2 PBOC 2.0: E-wallet extension application, debit/credit application, personalization guide, contactless IC card standard 3 PBOC 3.0: Cancel e-wallet and electronic passbook application, cancel downgrade transaction, multi-algorithm extension, multi-application extension, mobile payment standard
Beijing and Shanghai Transit cards were originally MIFARE but instead of examining what technologies would be best for next generation transit needs, China simply migrated them to the much slower PBOC 2.0/EMV specification implemented on the China T-union transit card architecture. The China T-union card is country wide transit prepaid card spec for interoperable transit cards that can work everywhere, similar to what Japan has with Suica, ICOCA, PASMO, etc.
Unfortunately, instead of letting transit cards be transit cards, China Union Bank demanded the transit card be a slightly different credit card. The EMV transit problem all over again. The rollout of new format card issuance has also been slow and piecemeal.
Chinese users familiar with Suica performance find China T-union cards slower and less reliable at the gate, no surprise there. Because PBOC is slow EMV NFC spec 500 MS transaction speed and tightly chained to UnionPay, the transit gate performance edge is not great enough to ween users away from QR Codes and the point benefits of sticking with AliPay and WeChat Pay.
If the performance gain was similar to the huge Suica over QR difference, coupled with an open flexible backend for using different payment methods to add money, China T-union would stand a better chance of nudging QR users to NFC for transit. As it stands now, there’s no real difference between a UnionPay card and a China T-union card at the transit gate. One is post pay, the other is prepaid, 2 versions of the same thing, marginally faster than QR, but not much.
The China T-union rollout is slow, not well promoted, and too tightly bound to Union Pay
Whatever the causes for the current situation, it’s a perfect gift to Chinese QR code players, I suspect that the arrangement is also a profitable one for the Chinese government on some level because if it was not, they wouldn’t be adding QR Code readers to transit gates.
QR Code reader on JR East ticket machine
QR Code reader closeup
QR Codes for Japan Transit Some Japanese tech journalists have fretted about JR East not embracing QR Codes on transit gates because JR Central plans to completely eliminate paper tickets for the next generation Chuo Shinkansen. It’s less about QR and more about eliminating magnetic strip paper tickets. JR East does have limited QR code use for ticket purchases at station kiosks, we’ll likely see wide support of many cashless payment options, QR included, with the new JR East eTicketing system due in April 2020.
QR Codes have seen some limited use on local monorail systems such as Okinawa’s Yui Rail but Suica compatilbilty is coming to the system in April 2020. The next generation Super Suica that does a lot more for much less, will arrive in April 2021. QR Codes for transit use in Japan will reamain a small side show far away from the main attraction.
In summary, the use of EMV bank cards and QR Codes for transit all comes down to transit company priorities for safe operation, better customer service and long term business goals. My position has been and continues to be is that it’s a better long term business opportunity for transit companies to:
Offer robust support of bank cards, QR and digital wallets on the backend for adding money to native transit cards on digital wallets and plastic, where they are really useful and add value without giving control away to outside companies.
Use closed loop transit gate value capture to focus on building better services tied to transit cards that benefit customers and businesses of the entire transit region, aka the transit platform business model.
It’s a simple choice really, moving people quickly and safely by transit, managed wisely, is a license to make money. A company can either use that license to build something of greater long term value for the users and businesses of the transit region, a win-win, or give it away to someone else.
I like writing but am no writer, so I prescribe to the ‘if you’re not a sharpshooter shoot lots of bullets’ school of wannabes. When the Financial Times, “The painful path of curing Japan of its cash addiction” (paywalled) piece came out, I had 2 hours to kill before going on a business trip and decided to post something while my reaction was fresh, figuring nobody would read it. The piece has not gotten many hits, but a few western journalists based in Tokyo tweeted about it recently, defending the FT piece and the overall ‘Japan failed’ game over narrative.
Here’s the thing. The cashless payments market landscape in Japan is the most messy and exciting one in the world right now. Nowhere else can you find such a concentrated investment in contactless payment infrastructure and different technologies (EMV, FeliCa, QR Codes, smartphones, etc.) competing and playing out in the market.
Japan is also the world’s great guinea pig test market. What works here first is adapted and deployed in other markets, like mobile payments. My take, covered in countless messy posts over the span of 2 years, is actually quite simple. The market revolution of mobile payments and smartphones is just getting started. The hot messy exciting payments situation you see happening in Japan right now will play out, in some other form, in other markets later.
That’s the story I think western journalists are missing. The ‘game over’ Japan narrative has been a stock western journalist in Japan ploy since the end of the Japan bubble, almost 30 years ago. A lot of journalists stick with it because it still sells. It’s entertaining for some people, but it doesn’t convey reality or educate.
Our smart devices are quickly evolving into ‘do everything’ devices that, unlike plastic, don’t care about any particular payment technology. They just work. That’s where the puck is going. If you sit around declaring that the game is over, you’re gonna miss the game. And the opportunity to tell people about it.
You must be logged in to post a comment.