It took me a while to fully appreciate the issue that Twitter user Yoshimasa Niwa was describing. At first glance I and many others assumed that setting Japanese over English would solve his app library sorting issue.
Then I realized that wasn’t his point at all. The software app in the screenshot is the Yahoo Japan ‘Norikae Annai’ transit app, one of the most popular free stand alone transit apps in Japan. I use it all the time. It’s a Japanese app with a Japanese name but the basic iOS English sorting algorithm ignores this and assumes all Chinese characters used everywhere must follow modern mainland China’s Simplified Chinese rules for reading and sorting.
This is ridiculous as assuming that all Roman based character sets everywhere must follow modern Italian reading and sorting rules. I always find that westerners assume the Kanji culture flow was always one way from China which it is not, with different and unique readings, usages, and Japanese Kanji like shitsuke 躾 traveling the other way over the centuries. The same is true for other cultures that adapted the Chinese writing system for their languages.
It amounts to cultural destruction by neglect and ignorance by large western based technology companies who think things are ‘good enough’. Or are just bugs to fix in a later software update that usually never appears. Modern computer software has pretty much destroyed traditional kanji culture publishing this way, with many countries abandoning mainstream traditional vertical text layout for western style layout because ‘it’s easier’, i.e. western tech companies couldn’t be bothered getting Asian language typography right. All these years later web browsers still can’t do vertical text worth a damn.
A veteran Japanese font engineer whose entire career was devoted to preserving high end Japanese typography in the digital age recently told me, “I don’t think anybody cares anymore.” In the end it all too often comes down to this: I don’t care cultural death by I don’t care companies who have the money and power to care.
That’s bitter irony in our age that purports to champion cultural diversity.
2020 is the coming out party for Apple designed OpenType variable fonts, both the SF Pro and SF Compact system fonts and the all-new New York font shipping in iOS 14, watchOS 7 and macOS 11. The Apple created variable font technology is not new of course. It has been around since the QuickDraw GX days along with the TrueType GX enhanced Skia font. It was due to be standard in MacOS Copland system fonts including a Japanese variable font created by FontWorks. Then Steve Jobs returned to Apple and everything changed.
Yes, it has taken 25 years for an Apple created technology to make it into the basic system. It proves my long stated belief that font technology doesn’t matter unless it is built into every nook and cranny of the OS foundation. The TrueType GX Skia variable font has been with us all this time, but only matters now because the SF Pro system font has gone variable.
Why is It Taking So Long? iOS 14 and macOS 11 variable font basics are covered in an excellent WWDC20 video, ‘The Details of UI Typography’. It’s important to remember that while OpenType variable font technology is ‘world ready’, at this stage they only apply to Roman based font sets. It’s going to be a long time before we see a Japanese language system font in variable format.
There are many reasons. In the WWDC20 video Loïc Sander of the Apple design team drops a big hint when he explains that while digital technology (PostScript fonts) “gave us a lot more flexibility in handling text,” it also “made typography a bit more crude than it used to be.” The statement shows how clueless designers and engineers outside of Japan can be about Japanese fonts and typography.
While a ‘bit more crude’ might be true for Roman based fonts and text layout, PostScript fonts completely broke traditional Japanese font design and composition models. Everything was thrown out because Adobe made no accommodation outside of western typography needs when creating the PostScript font DTP foundation.
Another big problem was that Adobe relations with Japanese PostScript licensees in the 1990’s was not healthy. Adobe stuck with closed print device font licensing for far too long and discouraged independent font production wherever they could. Because of this situation, digital font progress in Japan was slow and very expensive.
Here are some challenges facing Japanese variable fonts.
Once Upon a Time One basic flaw of OpenType font outline technology is that it’s extremely inefficient for kanji glyph production and storage. Every glyph has to be created and stored separately and doesn’t scale well. This is why OpenType CJK fonts on tiny devices like Apple Watch are a match made in hell. One solution to this problem is stroke fonts. Stroke fonts use a library of basic glyph parts to efficiently create complex glyphs.
Stroke fonts are a perfect fit for kanji font production and for small constrained devices like Apple Watch because reusable parts don’t take up precious resources. On the desktop, stroke fonts can do weight variations over the full range from Light through Ultra Bold without losing typographic details, all in a single 4 MB font while an equivalent OpenType variable font can weigh in around 18 MB.
The technology has been around for a long time and was supported up until macOS 9 but lost out when Apple quietly dropped the QuickDraw GX derived Open Font Scaler architecture in the migration from classic to macOS X.
While stroke fonts are not supported in the current Apple OS lineup, on the font tool side stroke font technology has appeared in software such as the classic MacOS Gaiji Master from FontWorks. The lead engineer of that effort is currently working independently on a similar gaiji glyph tool for Windows based on stroke font technology that is much more advanced than the old and long unavailable FontWorks software. I plan to cover developments in a future post.
The Japanese Font Production Challenge The Hiragino iOS/macOS Japanese system font was not created by Apple, it was licensed from Screen Holdings (SH), originally created by independent font design studio Jiyukobo in the early 1990’s. There is much more work involved creating a Japanese font compared to Roman based languages. Hand drawn glyphs are created, scanned and cleaned up for digital production.
The Adobe Japan 1-7 glyph collection requires 23,060 glyphs for a single weight, multiply this work by the different weights for one family and you get an idea how massive the undertaking is. From Osamu Torinoumi, one of the key designers of the Apple licensed Hiragino font on its creation:
On average, one person would (hand) draw 12 or 13 glyphs a day, which is not much change of pace from the days of creating block type…the whole process, from start to finish, took three years.
One might think that a single CJK (Chinese-Japanese-Korean) font sharing a common design can streamline the process but this is a huge misconception. Each culture has centuries worth of different design aesthetics that good design must incorporate: what looks good to a Chinese designer and works well in a Chinese text design, looks terrible in Japanese context. I have yet to see a decent digital ‘kana’ design from a Chinese font designer. Osamu Torinoumi on the differences in creating the Simplified Chinese Hiragino Sans GB:
“We worked with the Adobe GB 1-4 character set (29,064 glyphs) at 2 weights. Basically we had to finish one weight in 6 months. One year for the entire project. At first we only thought we would be there as backup, but Screen kept passing us all the questions from Beijing. It turned out to be a lot more work than we anticipated.”
Jiyukobo sent all the original Hiragino design data to Hanyi Keyin through Screen and they adapted the designs for China. Torinoumi said that one of the major differences is that Chinese design demands that Gothic (sans serif) characters mimic handwritten style. This means the character should be slightly off center within the virtual body. “Even after the project was over I still didn’t understand the difference between Japan and Chinese “Kokoro” glyph which the Chinese designers insisted were different.”
The Variable font UI Challenge Finally we get to a problem on the Apple OS platform side that has been around since the GX days: how to present advanced typography features in a useful and easy to understand system UI that works everywhere. What works on macOS obviously won’t work on iOS, but iPad OS will need some degree of advanced typography feature access. Sliders have their place but I agree with Adobe Type Senior Manager Dan Rhatigan who made a very good point in his TYPO Talk 2016 presentation: there has to be a better UI control concept out there.
Japanese typography is unique in that it has preserved its own print ‘moji bunka’ cultural history and vision that China and Korea have largely abandoned in the face of western centric computer culture that all too often pretends to care about such things, which it does not. If it did we’d have vertical text in web browsers by now that actually works. I hope a rich text culture can be preserved and conveyed to future generations even in such small details as a well designed and executed Japanese variable font for computers and smart-devices.
The once mighty Shaken KK, who’s founder co-created the modern Japanese typesetter with Morisawa in 1924 and a huge highly profitable font library that once dominated the entire Japanese print industry, is not quite dead but the last vestiges are quickly disappearing. Japanese blogger tkri notes that the Shaken Saitama production facility is being demolished, the same fate of the Kawagoe factory in 2017. The very last Shaken ‘digital’ typesetter machines serving the last 100 customers are due to go out of service by 2024.
Shaken had everything but never made it into the desktop publishing era by their own choice:
Hiragino started just about the time that Japanese PostScript arrived in the late 1980’s. At that time, the king of the typesetting market was Shaken KK. Anyone who knows Japanese typography knows Shaken. It had the most sought-after type library, the best designers, the biggest market share. Shaken made so much money that it became arrogant and absolutely refused any offers from Adobe and Apple to jump on the PostScript bandwagon.
But there was trouble in paradise. A few designers felt Shaken was becoming stagnant and left the company. Each sought his own vision, yet they came together to create the fonts that would eventually end up in MacOS X.
Shaken kept its proprietary typesetters and formats, losing 40 percent of its market share by 1999 along with lots of prestige and talent in the bargain. Ex-Shaken people filled the Japanese print industry. The low point came that year when government tax raiders found several safes in the company basement filled with 10,000-yen bills; president Hiroko Ishii, daughter of the founder, had stashed away some 15 million USD in the early ’80s and reported it later as profit. The directors of the company tried to oust her but failed. She died in the presidents chair at age 92 in 2018 leaving a ghost company as her legacy.
This sorry state of affairs had two huge repercussions. It denied a vital part of the Japanese font design legacy to designers working with DTP. Imagine a world with younger generations of designers denied the use of Helvetica and Times and you get the idea. It also left the only other competitor, Morisawa, which did license its library to Adobe, with a practical monopoly in the new DTP era.
Tomihisa Uchida who retired from Iwata Corporation a few years back knows more about Japanese font programming and typography than anybody else in this world. He was involved with Japanese digital font production from the start, working at its very heart: Shaken KK. I interviewed him in 2003.
Uchida: I was with Shaken for 23 years. I entered right out of college, where I was a chemistry major. My first job there was working with analog plates and mechanical processes. That involved high-resolution plates, similar to what is used for IC chip manufacture, to produce high-quality typography. I did that for 10 years; then digital fonts came along in the early ’70s. Shaken was the first Japanese vendor to have computerized layout. It also did the Japanese version of Ikarus. There wasn’t any real competition and it had the market to itself.
Question: So Shaken made the transfer to digitalized fonts and computer-based layout successfully?
Uchida: Yes, it purchased the Japanese rights to Autologic technology to produce a hybrid product where the software was a customized Autologic engine running on Shaken hardware. That didn’t last too long, as Shaken had been developing in-house technology and soon released its own original product.
We simply implemented Japanese typography and composition rules on the computer with outline fonts that output on an imagesetter using proprietary technology. At that time, Shaken systems were extensively used in newspaper production. However, Shaken lost that market because it didn’t have strong network capability, which newspaper production demands.
One of the ways Shaken was able to build up a strong type library in a fairly short time was by sponsoring a typeface competition. It would pay the winner and purchase his typeface. It raised lots of young designers that way (like Suzuki-san, who would later create the Hiragino font used in Apple’s MacOS X) and really expanded the market with new typefaces for comic books and such.
Later on, my job was creating different weights. The designer would create the basic design, then we’d use the Ikarus system to make the weights. That was the late ’70s. The systems we designed ran on hardware from the likes of DEC and Wang. I had a group of people who were basically a font production line. When Japanese PostScript first arrived, it wasn’t immediately apparent that things would change as they did, it took forever to print. Shaken systems always had excellent performance.
Shaken sent Uchida san to programming school then put him in charge of their digital font engineering group. He created the Shaken proprietary font format for their digital typesetter machines. He joked that if Shaken ever wanted to convert their font library to OpenType format, they would have to hire him back.
The very last time that Shaken made any kind of product announcement these past 20 years was at the July 2011 International eBook Expo Tokyo. At the time I blogged:
The once mighty Shaken finally waved the white flag and demonstrated OpenType versions of their fonts running in InDesign and on an iPad today at International eBook Expo Tokyo. Shaken’s Toshiro Ito said they don’t have a set release date, but hope to make an announcement soon; Japanese designers, who have been waiting for Shaken packaged software fonts since 1989, can finally look forward to using Shaken fonts on personal computers and mobile devices.
Blogger Danbo was there too and took some video of the Shaken demonstration. I remember most of the audience being near retirement age. That was the last word from Shaken. The OpenType product never appeared. Uchida san told me that Shaken probably didn’t have any designers and engineers left to build out the smaller legacy proprietary digital font collections into Adobe Japan 1-4 character sets for modern designers and devices.
Any time I talk with ex-Shaken people like Torinoumi san of Jiyukobo, or Toyoizumi san of Screen, or Uchida san of Iwata, there’s a wistful quality to their shared memories, even though they all went on to create new and better things. It’s bittersweet legacy only they can know: building something great then watching it rot away, destroyed by greedy deluded people in charge of protecting it. And now that it is almost gone, with most designers of that time in retirement, it is largely forgotten.
Some people will want to write the Shaken story off as Japanese culture unable to change but that’s just snobbery. What happened to Shaken can happen to any company with a monopoly. Steve Jobs definitively explained how monopolistic market power rots a company in his lost interview:
Sales and marketing people end up running the companies and the product people get driven out of decision making forums. And the companies forget what it means to make great products. The product sensibility and product genius that brought them to that monopolistic position gets rotted out by people running these companies who have no conception of a good product vs. a bad product. They have no conception of the craftsmanship that’s required to take a good idea and turn it into a good product. And they really have no feeling in their hearts usually about wanting to really help the customers.
When the last person turns out the lights at Shaken KK, I hope they open the vaults and set the Shaken font library free. Only by taking flight and having a life of its own can it ever hope to live on in the hearts and imaginations of future Japanese designers. Only then can the Shaken tragedy be reborn into something new.
Postscript A reader forwarded a Japanese public record document from 2008 regarding a Shaken company labor union dispute with management over mandatory retirement policy. It’s a pretty damning document. Company sales data from page 11 on clearly shows in ugly detail how management ruined the company business: focusing exclusively on milking customers with outrageously expensive font library rental royalties on proprietary hardware at the expense of everything else.
Shaken had the talent, technical expertise and opportunity to lead Japan into the DTP era with non-proprietary digital fonts and imagesetters. Instead of innovating, management clung to a do-nothing easy money strategy that destroyed the company. Meanwhile, the Japanese print industry moved on to better modern production infrastructure; a perfect example of Steve Jobs’s explanation.
One last bit: the Shaken KK company registration was updated in August 2019 and lists digital font production and sales as a main company objective even though real estate holdings will probably fund most company operations.
With Unicode adding more and more useless emoji, and seemly doing little else, it’s time to ask an important question: what the fuck is the Unicode Consortium supposed to be doing anyway?
It’s time to dust off Howard Oakley’s excellent blog post Why we can’t keep stringing along with Unicode, and think about the Normalization problem for file names and the Glyph Variation problem of CJK font sets. These problems fit together surprisingly well. My take is the problems must be tackled together as one thing to find a solution. Let’s take a look at the essential points that Oakley makes:
Unicode is one of the foundations of digital culture. Without it, the loss of world languages would have accelerated greatly, and humankind would have become the poorer. But if the effect of Unicode is to turn a tower of Babel into a confusion of encodings, it has surely failed to provide a sound encoding system for language.
Neither is normalisation an answer. To perform normalisation sufficient to ensure that users are extremely unlikely to confuse any characters with different codes, a great many string operations would need to go through an even more laborious normalisation process than is performed patchily at present.
Pretending that the problem isn’t significant, or will just quietly go away, is also not an answer, unless you work in a purely English linguistic environment. With increasing use of Unicode around the world, and increasing global use of electronic devices like computers, these problems can only grow in scale…
Having grown the Unicode standard from just over seven thousand characters in twenty-four scripts, in Unicode 1.0.0 of 1991, to more than an eighth of a million characters in 135 scripts now (Unicode 9.0), it is time for the Unicode Consortium to map indistiguishable characters to the same encodings, so that each visually distinguishable character is represented by one, and only one, encoding.
The Normalization Problem and the Gylph Variation Problem As Oakley explains earlier in the post: the problem for file system naming boils down to the fact that Unicode represents many visually-identical characters using different encodings. Older file systems like HFS+ used Normalization to resolve the problem, but it is incomplete and inefficient. Modern file systems like APFS avoid Normalization to improve performance.
Glyph variations are the other side of the coin. Instead of identical looking characters using different encodings, we have different looking characters that are variations of the same ‘glyph’. They have the same encoding but they have to be distinguished as variation 1, 2, 3, etc. of the parent glyph. Because this is CJK problem, western software developers traditionally see it as a separate problem for the OpenType partners to solve and not worth considering.
Put another way there needs to be an unambiguous 1-to-1 mapping and an unambiguous 1-1/1-2/1-3-to-1 mapping. I say the problems are two sides of the same coin and must be solved together. Unicode has done a good job of mapping things but it is way past time for Unicode to evolve beyond that and tackle bigger things: lose the western centric problem solving worldview (i.e. let’s fix western encoding issues first and deal with CJK issues later), and start solving problems from a truly globally viewpoint.
I finally had time to catch Adobe Nat McCully’s ATypl Tokyo 2019 presentation. He covers the topic that I have covered in depth many times before: the (sad) state of CJK typography. As Nat points out most software developers and system engineers talk about CJK support as typography without any idea of what it means. Throwing CJK glyphs on a screen is not typography, they are not the same thing at all.
The defining feature of CJK typography and layout in general and Japanese typography in particular is that space is an essential composition element equal with text and graphics, with fine space element control way beyond a baseline. Instead of thinking about how much space should be between text, flip it around and think about how much text should be between the space. Baseline font metrics will never deliver great CJK typography because there are too many limitations. So everybody implements the missing stuff on the fly and everybody does it different. Unfortunately the irony of it all is that Adobe played a huge role in how these limitations played out in the evolution of digital fonts, desktop publishing (DTP) and the situation we have today.
QuickDraw GX was probably the only time in computer history that fonts, layout engine and the basic OS came together to solve these limitations for all language systems, all language typography as equal from the bottom up. Parts of that effort survived, such as Apple’s San Francisco variable system font based on the TrueType GX model, and the inclusion of the TrueType GX model as the base technology for OpenType Variable fonts. Nice as this is, it’s only a tiny sliver of the GX vision pie that survived, all the other baseline font metric and CJK typography limitations still exist. Outside of a handful of people like Nat at Adobe, and the Adobe CJK typography ghetto approach of keeping all the good stuff corralled in InDesign J, very little is being done to address them.
Call me a pessimist but after 20 years of watching things slide sideways, I don’t see much hope for the future evolution of great CJK typography on digital devices. Most western software development people think that having CKJ glyphs on a screen is ‘good enough’ CJK typography, end of story.
Already I see the OpenType Variable Font effort devolving into a bauble for web developer geeks, always stuck in demo-hell, never going mainstream. It is the same story for quality CJK typography on digital devices. When the current Adobe CJK leaders like McCully and Ken Lunde reach retirement age, whom have devoted their careers to fixing these problems, I think it will be the end of an era. In many ways we are already there.