High Sierra, Sierra’s dispatching bug, and iTunes 12.7

Eclectic Light does it again going deep into High Sierra where no other tech writer seems to go. Lots of writers have been complaining about the removal of app syncing in iTunes 12.7 but nobody seems to know about the replacement:

Apple’s answer, I suspect, is in the new Content Caching feature in High Sierra, although that doesn’t appear to be offered as a retrofit for Sierra. Apple hasn’t yet provided full details, but what I think is intended is that those with multiple iOS devices use their Mac as a local app and update server…

In many ways, this is a much neater solution than poor, overburdened iTunes has been.

If you used iTunes app syncing at all do yourself a favor and read the Eclectic Light post.

Something Went Wrong in Siri’s and Apple Maps Development (U)

John Gruber made a great post today:

Siri, as it stands today, is at best a halfway product. Again, I’m pro-Siri in the voice assistant debate, but even so I think it’s generous to describe it as “halfway”. The whole category is garbage, Siri included. And frankly, it just doesn’t feel like Apple has made as much progress in six years as they should have.

Something went wrong in Siri’s development, and it wasn’t the voice quality.

I don’t think it is entirely coincidence that two of Apple’s most troubled products, Siri and Apple Maps both appeared at the same troubled time: iOS 6 and the departure of Scott Forstall. They are very different products but the lack of progress in Siri and Apple Maps development after six years, is very troubling indeed.

Update
John was very kind and tweeted a reply that Siri (2011) and Apple Maps (2012) don’t compare because Apple Maps has made far more progress in six years than Siri. That may be true for Apple Maps in America. In Japan Apple Maps is stuck in 2012 limbo

I don’t think there is a definitive answer. Yes, Siri has made excellent progress with natural sounding voices. Yes, Apple Maps has evolved into a good product for America. But these are parts, not a whole.

Again and again I think it all comes back to Steve Jobs WWDC 1997: is the total more or less than the sum of the parts? For John the answer might be more. For me, the answer is less which is why I see both Siri and Apple Maps facing the same problem; lack of progress.

Return of the Best macOS Japanese Word Processor: EG Word (U)

In 2007 I visited the office of Ergosoft near Yokohama for what turned out to be the last time. Ergosoft had just released a spectacular upgrade to their Japanese word processor “EG Word” which had been on the Japanese market from 1984. EG Word Universal 2 was the first OSX/macOS app built ground up using Apple’s then brand new Core Text framework. Clean, fast and rock solid, it was, and still is, the best Japanese work processor I ever used.

Unfortunately Ergosoft parent company Koei KK closed down the subsidiary in late 2007 and killed off the entire product line seeing no future in macOS. iPhone had not been released in Japan yet and the app economy had not materialized.

Norihito Hirose and Kenta Arano, the very talented lead programmers of Ergosoft set up with own company Monokaki-do in early 2008 and created some of the first hit Japanese iPhone apps such as the Japanese dictionary Dajirin.

I talked with Hirose san in 2010 just as the first iPad hit the market. At the time he wanted to continue programming for macOS but Koei KK would not sell him the product assets.  A few years later he negotiated a deal to release the venerable Japanese text input module EG Bridge which he relaunched as Kawasemi (Kingfisher) but EG Word assets remained out of reach…until this week.

The Return of EG Word Universal 2
Hirose san announced the complete transfer of all Ergosoft assets to Monokaki-do on September 4 for what Hirose san said was “a reasonable price.” The first order of business will be releasing a patch for EG Word Universal 2 to run on High Sierra. Believe it or not EG Word Universal 2 still runs fine on macOS Sierra.

There is lots of work to do and there will probably be a branding change but it’s great that Norihito Hirose and Kenta Arano have their baby back. I can’t wait to see what they do with EG Word on macOS and iOS.

Hirose san is tweeting progress in real time, fascinating to read: 64 bit compatibility…check, Emoji compatibility…check, next up retina display compatibility. The software is still super fast as ever.

DJDeoMJUMAESXl1
Emoji compatibility and vertical layout compatibility a one hour job for Hirose san

Fun fact: famous Japanese writer Haruki Murakami writes with EG Word. It’s nice to know he can keep using his favorite writing tool.

Welcome back!

A long Strange Journey

Apple Typography & Layout Graphic 2
The torturous journey of Apple’s Advanced Typography Frameworks over the years was a dangerous one for developers. Many dropped out. Core Text is the heavy-duty lower level text layout engine with the lighter higher level Kits for less complicated typography.

Here are EG Word related posts from 2010 and 2007 that explain EG Word and the advanced Japanese typography it brought to a wide audience.

2010

GX→ATSUI→Core Text→EPUB 3
aka So Long ATSUI, We Hardly Knew You

MacOS X 10.5 Leopard introduced a new text framework called CoreText to replace the ATSUI and MLTE Carbon text API. Core Text is the third MacOS Text API makeover for which the few loyal MacOS developers who invested first in QuickDraw GX advanced text and layout technology and then ATSUI, have had to program for.

One Japanese software developer Ergosoft dutifully adopted each and every Apple text technology, perhaps the only developer in the world to to do so. I spoke with Ergosoft Marketing Director Isamu Iwata back in 2002, when Ergo used ATSUI for the Japanese advanced layout features in its EGWord Version 12 word processor package, and in 2007 when Ergo released free Leopard updates to EG Word Universal 2, the first third-party program built on Core Text.

EGWord was the first commercial word processor software package to deliver AAT (Apple Advanced Typography) features such as extended Japanese glyph access, glyph variations and Japanese kerning and did so in a clean easy to understand UI.

EGWrod H&V text

EG Word Universal 2 was a very innovative Japanese word processor for macOS, notice the slightly different vertical and horizontal glyph designs of ”と”

EGWord pallett
EG Word made high end Japanese typography an easy palette choice with clean intelligent design.

At that time Iwata said speed was the primary benefit of the new API, “Core Text is much faster than ATSUI was. Other developers didn’t use many ATSUI features and I don’t think many developers will invest much in Core Text as it’s an Apple-only solution.”

Iwata had a point, as Apple never bothered to use Core Text to implement a basic Japanese text feature such as vertical layout in its own Cocoa-based word processor, Pages 09. Apple even dropped previously marketed Apple Advanced Typography glyph variant features from Hiragino Pro N without any explanation. Major developers such as Adobe, Microsoft and Quark already had their own text engines and did not use Apple’s text frameworks.

glyph variations

Hiragino Pro N fonts dropped Glyph Variations. They still exist in the older Hiragino Pro

The big change came when Core Text was rolled into iOS with the iPad only iOS 3.2 release and in iPhone with iOS 4. For advanced layout, it’s the only framework available. From a Japanese developer point of view CoreText on iOS is somewhat useless because it doesn’t support the vertical text layout features of the desktop version. Core Text also has a reputation of being somewhat buggy and the Core Text documentation on the Apple developer site is weak.

Though Ergo shut down in early 2008, lead programmers Norihito Hirose and Kenta Arano established their own company Monokaki-do just in time to develop for iOS 2. They have produced some of the biggest selling iOS App titles on the Japan App store: Daijirin and Wisdom English Japanese Dictionary. In early 2010 Hirose san told me “Compared to ATSUI, Core Text feels incomplete.” The ugly reality is that without full vertical text layout support in Core Text on iOS Japanese developers have to ‘roll their own’ text layout engine for vertical layout.

Rolling your own text layout engine is fine if you are a developer but Japanese publishers had to make difficult choices on iOS and Android:

Go with weak native text layout engines and force ugly horizontal Japanese text on users who want the traditional vertical typography they have in ‘bunkobon’ books.

Pay a royalty to use Adobe’s Digital Publishing Suit or Morisawa’s MCBook to use their text layout engines locked with ‘our way or the highway’ publishing methods.

Wait for EPUB 3 and future versions of iOS and Android to support vertical layout and see how the pricing shakes out as the market opens up more.

Many Japanese publishers are dabbling with the second choice but really waiting to see what EPUB 3 brings. EPUB 3 has support for vertical layout and about 50%, at best, of high end Japanese typography as outlined by the X4301 Japanese Industrial Standard specification. Iwata Font lead font programmer Tomahisa Uchida said, “EPUB 3 is better than what we have now but I’m worried that quality will decline.” Uchida san is right that despite all the power and potential that computers offer, they have yet to achieve the quality of traditional Japanese print methods for the mass market.

If Apple announces EPUB 3 and vertical layout support in Safari and iBook along with some Japanese publisher content deals at the expected WWDC iOS 5 announcement on June 6, Apple will get a big leg up on Google in the Japanese market. Such an announcement could kick start the Japanese digital publishing market into high gear.

(2017 update: iBooks does support EPUB 3 and Japanese vertical text but Pages and iBooks Author still do not support CJK vertical layout)

2002

EG Word Universal v1

egword univewrsal 1
Easy access to extended Japanese glyph sets was a big sales point of EG Word and EG Bridge

Ergosoft was one of the first Japanese Macintosh developers. In the 1980s, its EG Word text editing program and EG Bridge input module were staples for every Japanese Mac user. The arrival of Atok from JustSystem (for kanji input) and Microsoft Word put an end to Ergosoft’s market dominance in the mid-1990s. At the same time, a foray into cross-platform development turned out to be an expensive disappointment, and Ergosoft dropped Windows development in 1997. Mac OS X gave Ergosoft a new opportunity, and the company took it by adopting Apple innovations that many developers have ignored: ATSUI, the extended glyph set of Hiragino and the power of Apple’s AAT (Advanced Typography Tables).

Ergosoft product manager Isamu Iwata sat down to discuss the latest products that build on Hiragino.

JB: How hard was it doing EG Word and EG Bridge for Mac OS X?

Isamu Iwata: It was pretty hard. EG Word has a code base going back 15 years. EG
Word was Carbon. The input method for X was so different from 9 that we just decided to start from scratch and write it in Cocoa using the best parts of EG Bridge.

JB: How has the market reception been?

Iwata: Very good. The Mac OS X version of Atok has a reputation of being buggy. I’m pretty confident we are gaining market share. The changes under the hood for 10.1 were very significant, and most input modules didn’t work all of a sudden. Since we are a Mac-only developer, we are able to address issues much faster than the com- petition. JustSystem [the developer of Atok] was very late addressing 10.1 issues, which worked out well for us.

JB: One of the big features of EG Word 12 and EG Bridge 13 is the ability to use all of Hiragino’s extended glyph set. Was it difficult implementing these features?

Iwata: There was some engineering work using ATSUI, but making the decision to use ATSUI or not was a much more difficult decision. It was a little risky.

Apple made lots of promises with QuickDraw GX, then completely dropped it. It took us two years to finally decide to support ATSUI. We did surveys of other developers to see if anybody was using ATSUI, but didn’t find a single one. We were worried that if we were the only ones, it would disappear. I’m fairly certain we are the first developer to use ATSUI in a big way. Not even AppleWorks uses ATSUI.

We put a lot of effort into making a GX version of EG Word, but ended up throwing it away when Apple killed GX.

JB: Did you see any opportunities in using ATSUI?

Iwata: The deciding factor was the Hiragino fonts becoming part of Mac OS X. Even though Morisawa and the other font makers don’t have APGS [AJ 1-5] extended glyph sets, we still felt it was an opportunity, because it is part of the base system. The Hiragino fonts are high quality; they can satisfy the DTP market but also be of real value to other users as well.

JB: OpenType fonts are a cross-platform standard. How do you feel about that, as a Mac-only developer? Are you trying just to sell to the DTP market?

Iwata: We are selling to ordinary end users so that they can get the full benefit of Mac OS X’s Japanese environment, which of course includes the Hiragino fonts and the extended character sets.

JB: Does EG Word use any other ATSUI features besides the extended character sets?
Iwata: Not at this time. There was also a bug in ATSUI [prior to 10.2] that prevented us from accessing approximately 300 glyphs. We told Apple engineering when they visited here and they were very surprised. They promised to fix it in the next OS update. Such is the risk of being the only developer to use ATSUI and Hiragino.

JB: How do you maintain font-data compatibility with Adobe’s InDesign [which does not use ATSUI]?

Iwata: Hiragino has CID ID tags so the correct glyphs are imported [and displayed, printed, embedded in PDF documents, etc.] However, we cannot import some glyphs because they do not have CID tags. [This has been fixed in 10.2.]

JB: All of APGS does not have CID tags?

Iwata: Not all, only some…. EG Bridge can access all of the characters, but applications like Excel and Word cannot accept them. The only applications that can display all the characters are EG Word and EG Word Pure.

JB: And the reaction to EG Word?

Iwata: It has been very good. Users like being able to use the extra characters—especially business users, for it is considered polite to have the correct kanji for a person’s name in business correspondence. And it is no problem to print or save them as PDF.

JB: Why did you choose to become a Mac-only developer again?
Iwata: Going cross-platform requires a huge investment in money and development resources. We had Windows versions of EG Word and EG Bridge, and it was huge effort. Our costs were twice what they had been. And the Windows word-processing market is monopolized by Microsoft. In the end, the Macintosh market was more profitable than Windows. The profit margins for Windows are much slimmer and the support costs are higher. We quit the Windows market for word processing in 1997 and the Windows input market in 1999.

JB: Do you think the competition will offer these new features too?

Iwata: Probably not. Developers like JustSystem have to keep the feature set the same for each platform. If they can’t do it on Windows, they won’t do it on the Mac.

Update 9/11: Cleaned up broken links, sorry I did not catch those sooner
Update 9/7: Cleaned up misplaced hyphens, added Monokaki-do tweet of EG Word progress

APFS and Unicode Normalization

One key feature of macOS High Sierra is the arrival of Apple File System (APFS) as the default file system format. The iOS 10.3 update migrated the iOS file system from HFS+ to APFS, an amazingly smooth transition that was celebrated at WWDC last week.

The WWDC APFS developer session video is well worth your time if you have access. I am familiar with font encoding issues but was completely unaware of the Unicode Normalization file system issue that developers outside the ASCII bubble have been worried about. The best blog to read about APFS and Unicode issues is The Eclectic Light Company by Howard Oakley. His take on AI is great too.

I particularly enjoyed reading his explanation of Unicode file naming and the limits of having the file system handle normalization. There will be two different flavors of APFS, native normalization will be default for iOS 11, the default for macOS High Sierra is normalization-insensitive. This should work well. The basic encoding issue that affects all systems everywhere however, remains:

it is time for the Unicode Consortium to map indistiguishable characters to the same encodings, so that each visually distinguishable character is represented by one, and only one, encoding.

That is a stark challenge, and one that I am sure will never even be started. But until we do, today’s minor running sores will only fester and grow.

I have heard similar complaints about the Unicode Consortium from Japanese font developers over the years. Unicode has done many good things but like any human organization there are agendas and politics. For some, the Unicode Consortium working method is too top down for comfort. Sometimes grand plans don’t work out, like IVS.

As Oakley points out, getting a big new effort off the ground is too much to ask of the Unicode Consortium.