Unicode needs a new Mission

With Unicode adding more and more useless emoji, and seemly doing little else, it’s time to ask an important question: what the fuck is the Unicode Consortium supposed to be doing anyway?

It’s time to dust off Howard Oakley’s excellent blog post Why we can’t keep stringing along with Unicode, and think about the Normalization problem for file names and the Glyph Variation problem of CJK font sets. These problems fit together surprisingly well. My take is the problems must be tackled together as one thing to find a solution. Let’s take a look at the essential points that Oakley makes:

Unicode is one of the foundations of digital culture. Without it, the loss of world languages would have accelerated greatly, and humankind would have become the poorer. But if the effect of Unicode is to turn a tower of Babel into a confusion of encodings, it has surely failed to provide a sound encoding system for language.

Neither is normalisation an answer. To perform normalisation sufficient to ensure that users are extremely unlikely to confuse any characters with different codes, a great many string operations would need to go through an even more laborious normalisation process than is performed patchily at present.

Pretending that the problem isn’t significant, or will just quietly go away, is also not an answer, unless you work in a purely English linguistic environment. With increasing use of Unicode around the world, and increasing global use of electronic devices like computers, these problems can only grow in scale…

Having grown the Unicode standard from just over seven thousand characters in twenty-four scripts, in Unicode 1.0.0 of 1991, to more than an eighth of a million characters in 135 scripts now (Unicode 9.0), it is time for the Unicode Consortium to map indistiguishable characters to the same encodings, so that each visually distinguishable character is represented by one, and only one, encoding.

The Normalization Problem and the Gylph Variation Problem
As Oakley explains earlier in the post: the problem for file system naming boils down to the fact that Unicode represents many visually-identical characters using different encodings. Older file systems like HFS+ used Normalization to resolve the problem, but it is incomplete and inefficient. Modern file systems like APFS avoid Normalization to improve performance.

Glyph variations are the other side of the coin. Instead of identical looking characters using different encodings, we have different looking characters that are variations of the same ‘glyph’. They have the same encoding but they have to be distinguished as variation 1, 2, 3, etc. of the parent glyph. Because this is CJK problem, western software developers traditionally see it as a separate problem for the OpenType partners to solve and not worth considering.

Put another way there needs to be an unambiguous 1-to-1 mapping and an unambiguous 1-1/1-2/1-3-to-1 mapping. I say the problems are two sides of the same coin and must be solved together. Unicode has done a good job of mapping things but it is way past time for Unicode to evolve beyond that and tackle bigger things: lose the western centric problem solving worldview (i.e. let’s fix western encoding issues first and deal with CJK issues later), and start solving problems from a truly globally viewpoint.

Advertisements

Japanese Text Layout for the Future* (hint: there isn’t one)

I finally had time to catch Adobe Nat McCully’s ATypl Tokyo 2019 presentation. He covers the topic that I have covered in depth many times before: the (sad) state of CJK typography. As Nat points out most software developers and system engineers talk about CJK support as typography without any idea of what it means. Throwing CJK glyphs on a screen is not typography, they are not the same thing at all.

The defining feature of CJK typography and layout in general and Japanese typography in particular is that space is an essential composition element equal with text and graphics, with fine space element control way beyond a baseline. Instead of thinking about how much space should be between text, flip it around and think about how much text should be between the space. Baseline font metrics will never deliver great CJK typography because there are too many limitations. So everybody implements the missing stuff on the fly and everybody does it different. Unfortunately the irony of it all is that Adobe played a huge role in how these limitations played out in the evolution of digital fonts, desktop publishing (DTP) and the situation we have today.

QuickDraw GX was probably the only time in computer history that fonts, layout engine and the basic OS came together to solve these limitations for all language systems, all language typography as equal from the bottom up. Parts of that effort survived, such as Apple’s San Francisco variable system font based on the TrueType GX model, and the inclusion of the TrueType GX model as the base technology for OpenType Variable fonts. Nice as this is, it’s only a tiny sliver of the GX vision pie that survived, all the other baseline font metric and CJK typography limitations still exist. Outside of a handful of people like Nat at Adobe, and the Adobe CJK typography ghetto approach of keeping all the good stuff corralled in InDesign J, very little is being done to address them.

Call me a pessimist but after 20 years of watching things slide sideways, I don’t see much hope for the future evolution of great CJK typography on digital devices. Most western software development people think that having CKJ glyphs on a screen is ‘good enough’ CJK typography, end of story.

Already I see the OpenType Variable Font effort devolving into a bauble for web developer geeks, always stuck in demo-hell, never going mainstream. It is the same story for quality CJK typography on digital devices. When the current Adobe CJK leaders like McCully and Ken Lunde reach retirement age, whom have devoted their careers to fixing these problems, I think it will be the end of an era. In many ways we are already there.

Apple prides itself on having good typography but cannot be bothered with such Japanese typography basics as not mixing Gothic and Ryumin Japanese font styles seen here in the Photos app

UPDATE
Ken Lunde posted a wonderful overview of his Adobe career to date, also his ATypl Tokyo 2019 presentation.

Tokyo Olympics Apple Maps: Death by Point of Interest

Apple tells Engadget Japan reporter Masaichi Honda that Apple Maps Japan will be ready for the Tokyo Olympics in the summer of 2020. Apple Maps will have robust indoor maps for tall buildings and underground station malls in Tokyo, and Real-time transit for better transit route searching and transit updates. That is exactly one year from now, far into the iOS 13 life cycle. Honda san also reports that Apple is not ready to show Japanese reporters a demo yet, not an encouraging sign.

In addition to the Apple Maps image collection vans combing Japan right now, WWDC19 unveiled the Indoor Maps Program for registered developers and building owners to map indoor areas and encode the data using (Apple’s ?) Indoor Mapping Data Format (IMDF). Once the data is encoded in IMDF, surveyed and validated, developers and building owners can use the data in their apps and designate indoor areas to share on Apple Maps.

That’s great for building owners to indoor map their own building. What about shared public places like Shinjuku Station which is spread out and shared by 8 different owners? There is also the localization problem. It’s one thing to indoor map for Japanese users, but who’s going to localize all those Point of Interest (POI) icons and information sheets in English, Chinese, Spanish, etc. That costs serious time and money.

Let’s take a comparison look at indoor maps of the primary entrance gate for inbound visitors coming to the Tokyo Olympics next year: Tokyo Station, and compare Yahoo Japan Maps, Apple Maps and Google Maps.

Yahoo Japan Maps
Yahoo Japan Maps only offers Japanese language but it has best cartography and attention to small details that matter, like yellow station exit signage colors that exactly match what you find on the ground. Apple and Google don’t.

Yahoo Japan Maps Tokyo Station Indoor mapping

Apple Maps Japan
Apple Maps does not offer indoor station mapping in Japan. It does offer multilingual support but judging from the English Point of Interest information, it’s not robust. As usual Apple Maps Japan overwhelms users with Point of Interest icons. It’s map death by Point of Interest. There’s a lot of fixing Apple needs to do if they want to present a good map product in time for the Olympics.

Apple Maps Japan Tokyo Station (as yet no indoor station mapping)

Google Maps Japan
Google Maps offers indoor mapping for Tokyo Station in multiple languages. For all the detail Google offers here, it’s much less helpful than Yahoo Japan Maps. For high density areas like Tokyo, good cartography and smart editing makes all the difference between a good map and lousy one.

Google Maps Japan Tokyo Station

Apple Advanced Typography Request #999

oh where can my glyph variations be? oh where, oh where…

The new SF Symbols system font is a fantastic addition to iOS, macOS, etc. and it is an OpenType (née QuickDraw GX TrueType) variable font to boot. But please, Apple, the crusty old Font and Typography pallet has to go. Put the poor dead horse in its grave already. A fantastic new variable font from Apple screams for a nice new user friendly font UI to access those lovely glyph variations. If you don’t, nobody will ever find them. And that’s a shame.

WWDC19 Apple Maps 2.0 Scorecard

The WWDC19 keynote had some Apple Maps 2.0 updates that included a few of my wish list items. The big marquee feature is iOS 13 Apple Maps ‘Look Around’ that finally delivers a Google Maps Street View equivalent. There was no mention of region availability but I assume it matches what Craig Federighi said about Apple Maps 2.0 in the keynote to ‘cover the entire US by the end of 2019 and select countries in 2020’. With Apple Maps image collections vans now combing Japan, Apple Maps Japan is a good candidate to offer Look Around in the next rollout.

Look Around will get most of the attention but it’s a feature I rarely need or use. Real-time Transit sounds like something that might actually be more useful than transit is right now. Look closely at the keynote screen and you see station train times listed exactly like they are Apple Maps Japan iOS 12. However you can also see faint blue text “On Time” listed directly below each subway line.

That would be marginally better than current clunky iOS 12 transit notices but I need to see what, if any, updates iOS 13 has for the transit widget. Will it be a Nearby Transit Time Widget? I doubt it will match the very useful Google Maps location aware train/bus times widget but reserve judgement until the iOS 13 Public Beta.

Some of the other new features like Junction view (is it really only for the USA and China?) would be great for car navigation in Tokyo. Better Siri guidance is listed but still does not includes transit awareness. New MapKit features allow developers to add heat maps and weather overlays but I wish those were built in like they are in Yahoo Japan Maps. Dark Mode Point of Interest icons are attractively toned down, unfortunately the regular map view counterparts still sport the same garish candy colors. Apple Maps 2.0 is slowly delivering better and more accurate map details, but has yet to deliver better basic 2D cartography.

UPDATE
Tokyo Olympics Apple Maps