At

I recently appeared with John Gruber on The Talk Show. During the episode, the following exchange took place:

When it comes to naming characters, the Unicode standard is the bible. And code point U+0040 is named as “COMMERCIAL AT”.

So yeah, we’re “right.”

But then Twitter got ahold of this exchange and I quickly realized something important: we don’t all speak English:

It turns out “arroba” has a very interesting history that originated in Spanish commerce:

“Whatever the origin of the @ symbol, the history of its usage is more well-known: it has long been used in Spanish and Portuguese as an abbreviation of arroba, a unit of weight equivalent to 25 pounds, and derived from the Arabic expression of “a quarter” (الربع pronounced ar-rubʿ)”

As someone who loves iconography, it’s pretty amazing to see @ as a handwritten symbol in 1148:

I also realized that I knew the Italian word for the @ symbol: “chiocciola”. It’s one of the names for a snail (the other being “lumaca” which is commonly used when ordering them in a restaurant.)

And why is this name used?

(It’s fun to say, too. Something like “key-o-cho-la” but with more exotic hand gestures.)

This tweet led to many responses that show how varied the pronunciations are in different languages.

  • Dutch: “apenstaartje” = “monkey tail”
  • Hebrew: “strudel” = shape of the cake
  • Danish/Swedish: “snabel-a” = “with an (elephant) trunk”
  • German: “Klammeraffe” = “spider monkey”
  • Poland: “małpa” = “monkey”
  • Korean: “골뱅이” (gol-baeng-ee) = “a type of sea snail”

Wikipedia has a full list of how @ is used in other languages.

But do you notice the pattern with these pronunciations?

They’re being used as pictograms:

“A pictogram…, is an ideogram that conveys its meaning through its pictorial resemblance to a physical object. Pictographs are often used in writing and graphic systems in which the characters are to a considerable extent pictorial in appearance.”

While pictograms are fairly common in Asian languages, it’s rare to see this kind of usage in the West. Written Kanji characters, such as 木 for “tree”, have been in use since the first century AD. Indeed, these kinds pictures were man’s first form of expression and communication.

But in these writing systems, someone saw a thing with a trunk and leaves growing from the ground and put it on a piece of paper as an 木 symbol. What we’ve seen happen with the @ symbol is the opposite. Many different cultures have seen our “COMMERCIAL AT” symbol and given it a name based on its appearance.

So even though John and I are right about the pronunciation, this is certainly a case where English pales when compared with other languages. I envy my colleagues that get to play with snails and monkeys while coding in Objective-C!

Wearing Apple

Tim Cook has openly stated that Apple is working on “new product” categories. Many people, customers and competitors alike, assume that means some kind of wearable computing device. And of course that means it has to be some kind of “smartwatch”, right?

I don’t think so.

The Market

First, let’s look at the market for quality timepieces; ones that you’d be proud to wear on your wrist. It’s dominated by companies with centuries of experience. It’s also a high-end market: spending a few thousand dollars on a nice watch is chump change. You’re buying a work of art.

Apple certainly has great designers, but they’re going to be competing against craftsmen who’ve been refining their craft since the 15th century.

I realize that this sounds a lot like Ed Colligan talking about mobile phones:

“PC guys are not going to just figure this out. They’re not going to just walk in.”

There’s a big difference here: the companies that dominated the music player and mobile phone markets were making complete crap prior to Apple’s arrival. Granted, there are a lot of cheap and crappy watches on the market, but they’re not remotely interesting to the demographic that buys Apple products. And to many people, a fine timepiece is more about status than technology.

Taking all of this into consideration, watches don’t sound like product category that fits Apple well. One of the many reasons we love their products is because they are best of breed.

The Watch

Now, let’s look at the state of the “smartwatch” market. Smart companies, like Nike and FitBit, are creating products that aren’t really a “watch”. It’s like the “phone” being the least interesting thing on our “smartphone”.

In fact, Tim Cook is already wearing one of these devices:

Walt: Is wearables a thing — is it part of the post-PC era wearables that go beyond fitness devices?

Tim: Yes, I think so. I wear a Fuelband, I think Nike did a great job.

Note: This quote, and others in this piece, were made during Tim Cook’s interview at D11.

However, once manufacturers try to put a lot of smarts in a watch, they end up with a product that’s a bulky mess or ugly as sin. Probably both.

Again, Tim Cook has noticed this same thing:

“There are lots of gadgets in the space. I would say that the ones that are doing more than one thing, there’s nothing great out there that I’ve seen. Nothing that’s going to convince a kid that’s never worn glasses or a band or a watch or whatever to wear one. At least I haven’t seen it. So there’s lots of things to solve in this space.”

We all know that Apple’s designers and developers can solve these problems. But can they do it without making compromises?

“It’s an area that’s ripe for exploration, it’s ripe for us to get excited about. Lots of companies will play in this space.”

That exploration is all about saying no a thousand times. It’s easy to create a great looking concept, but turning that fantasy into a product is where it gets hard and compromises are often made.

It’s my view that the technology needed to make that great product that does everything just isn’t there yet. There are too many compromises.

The New Hub

Let’s assume for a second that you can make a compelling device for a wrist. I’d still be asking myself this question: “Why do I need a computer on my wrist?” I already have one in my pocket.

With the introduction of CarPlay, Apple’s answer to this question is becoming clearer. That device in your pocket is the new hub for your digital data. Ben Thompson calls it Digital Hub 2.0.

You don’t need a computer on your wrist. Apple’s wearable devices will become a part of this circle that surrounds your main device. And as with CarPlay, it’s going to be a two-way street where interactions and information in the player make their way back to the main device.

The Fashion

The development of wearable technology is going to enter a space where most companies in the Silicon Valley have no experience. As a geek, do you know who Anna Wintour is and why she would be important to the success of a wearable device?

Angela Arhendts certainly does…

Put aside your geek sensibilities for a second and think about this device as a piece of fashion instead of a cool gadget. What’s the primary function of fashion?

It’s about expressing a personal style. It’s an extension of your personality and way to show it in a public way.

Look at the huge variety of iPhone cases that are on the market. That’s only 16 examples of 4.7 million search results. Fashion is as diverse as the people who wear it.

An important part of this diversity is customer’s gender: men have different tastes than women. That concept watch mentioned above looks fantastic—if you’re a male. I can’t imagine my wife wearing it.

Then there’s the question of size: as someone who’s 6’7″ tall, I know firsthand that “one size fits all” is the greatest lie ever told by clothing manufacturers. Since everyone’s body is unique, wearable devices must also consider comfort and fit in the design.

Apple may be able to use something like an iPhone case to make a wearable device appeal to both sexes. Or adjustments that make a single device (SKU) adapt to multiple body shapes. But I suspect that it’s going to take more than that to be appealing as a piece of fashion.

The Customer

Now that we’ve looked at the challenges in developing these devices, let’s think about the most important question Apple is asking itself:

Who is going to buy this wearable technology?

Trends are always set by the younger generation. Especially with clothing, jewelry and other items that appeal to a demographic with a lot of expendable income. To me, this quote by Tim Cook is the most telling:

“To convince people they have to wear something, it has to be incredible. If we asked a room of 20-year olds to stand up if they’re wearing a watch, I don’t think anyone would stand up.”

This response to Kara Swisher’s question about Apple’s interests in wearable technology covers all the bases. It includes the target market (“20-year-olds”), product focus (“has to be incredible”), and most importantly, he’s seeing the same thing I am: people don’t need to wear watches because they already have that computer in their pocket.

Note also that in the response he doesn’t say “wear a watch”, it’s “wear something”. It’s implied, but not stated. Remember that he learned from the master of misdirection: Steve Jobs.

The Product

Given everything presented above, it’s pretty clear to me that a “smartwatch” isn’t in Apple’s immediate future. But they’re clearly interested in wearable technology. So what are the alternatives for a product that could be released this year?

The first step is to start looking at things from Apple’s point-of-view. I ask myself, “What problems can a wearable device solve?”

As I think about answers to that question, it leads me to the conclusion that Jony Ive and crew aren’t looking solely at the wrist. Wearable technology could take cues from other kinds of jewelry: rings and necklaces, for example.

What if Apple’s entry into this space is a ring?

  • Limited display — A discreet way to provide notifications—just an LED or two that indicate what’s happening on your iPhone. Maybe even a small, flexible E Ink display for high contrast text.
  • Tactile — A way for your finger to sense a notification. It’s easy to miss audio cues in a noisy room or a vibration in your pocket.
  • Small & light — A hallmark of Apple design. Simplicity in form and function.
  • Customer appeal — Those 20-year-olds don’t wear watches, but jewelry is already a part of their culture. “Bling” came from the world of hip hop.
  • Sensors — Your finger has a pulse and blood pressure for Healthbook.
  • Power — The Lightning connector is small for a reason.
  • Competition — While all its competitors are rushing watches to market, Apple catches them with their pants down.
  • Low cost — How many of these rings could Apple sell if they were priced at $99?

But the biggest feature of all would be that this wearable device could support iBeacon. This technology is based on Bluetooth LE which Apple describes as “a new class of low-powered, low-cost transmitters that can notify nearby iOS 7 devices of their presence.”

Let this sink in for a second: your wearable device is transmitting a signal with a unique identifier that can be picked up by an iOS 7 device. And the proximity detection is sensitive within a few inches. Presumably, this signal could be also be detected on your Mac as well, since they have supported Bluetooth 4.0 since mid-2011.

By wearing this ring on your finger, your devices can know how close you are to them.

This opens up a world of possibilities: imagine the joy we’d all feel when a notification only popped up on the device we’re closest to. Right now my ring finger is hovering over my MacBook Air’s keyboard by 2-3 inches, while the phone in my pocket is over a foot away. Notification Center needs this information.

Conclusion

Predicting Apple’s future is always fraught with difficulties. I may not be right about the end result being something other than a watch, but I’m certain that there are people at Apple thinking about the issues I’ve outlined above.

And I, like many others, am really looking forward to wearing Apple.

Font Points and the Web

When sizing fonts with CSS, there’s a rule of thumb that states:

1em = 12pt = 16px = 100%

There are even handy tools that help you calculate sizes based on this rule.

But this rule of thumb leaves out a very important piece of information. And without that information, you could be left wondering why the size of your type doesn’t look right.

Testing Points and Ems

To illustrate, let’s take a look at some work I’ve been doing recently to measure text accurately on the web. I have a simple test page that displays text in a single font at various sizes:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8" />
  <title>Em Test</title>
  <meta name="generator" content="BBEdit 10.5" />
  <style>
    body {
      font-family: "Helvetica Neue", sans-serif;
      font-size: 100%;
      line-height: 1;
    }
    p {
      padding: 0;
      margin: 16px 0 16px 0;
    }
  </style>
</head>
<body>

<p style="font-size: 2em; background-color: #eee;">MjṎ @ 24pt / 2em</p>

</body>
</html>

You can view a slightly more complex version of the page here.

The W3C recommends specifying fonts using either ems or pixels. Since high resolution displays are becoming ever more common, that really leaves you with just one choice: the em.

(Note: In the examples that follow, I’m using text set at 2em since it’s easier to see the problem I’m going to describe. The effect, however, will happen with any em setting.)

The body is set in Helvetica Neue at 100% so 1em will be the equivalent of 16px when the user’s browser is using a default zoom setting. The line-height is 1 so there’s no additional padding around the text: the default of “normal” would make the line approximately 20% taller.

When you display the test page, everything looks good because the sizes are all scaled correctly relative to each other. A quick check of the light gray background with xScope shows that the height of the paragraph element is 32 pixels (2 ems):

Browser

But then things got confusing.

Points aren’t Points

I had just been measuring some text in TextEdit and noticed something was amiss. Comparing the two “24 point” fonts looked like this:

TextEdit versus Browser

I’m no typography expert, but there’s something clearly wrong here: a 24pt font in TextEdit was noticeably smaller than the same size in my web browser.

I confirmed this behavior in three browsers running on my Mac: Safari/Chrome (rendering with WebKit) and Firefox (rendering with Gecko) displayed the larger version of 24 point text. Why were the sizes different?

After some experimentation, it appeared that rendered glyphs were from a 32pt font:

72 DPI

What the hell?

A Brief History of Type

When confronted with a problem like this, it’s always a good idea to question your assumptions. Was I measuring the right thing? So I started learning more about type…

In general, points are meaningless. It’s a relic of the days of metal type where the size specified the height of the metal, not the mark it made on the page. Many people mistake the size of a glyph (shown below with red bars) with the size of a point (green bars):

Point versus Glyph

(Photo by Daniel Ullrich. Modifications by yours truly.)

Even things like the word “Em” got lost in translation when moving to the web: it originally referred to the width of a typeface at a given point size. This worked in the early days of printing because the metal castings for the letter “M” were square. Thankfully, we’re no longer working in the 16th century.

In modern fonts, like Helvetica Neue, a glyph for a capital “M” may be square, but the flexibility of digital type means that the width and height of a point rarely match.

Thanks Microsoft!

After doing this research, I was sure I was measuring the right thing. The sizes of the glyphs should match, regardless of point size. Something else was clearly in play here.

Let’s just say I spent a lot of quality time with Google before eventually stumbling across a hint on Microsoft’s developer site. The document talks about using a default setting of 96 DPI. I’ve been spending a lot of time lately with the Mac’s text system, so I knew that TextEdit was using 72 DPI to render text.

I quickly opened a new Photoshop document and set the resolution to 96 pixels per inch (DPI). And guess what?

96 DPI

All the text got larger and it matched what I saw in my browser. Mystery solved!

72 DPI on the Mac is 75% smaller than 96 DPI used by the Web. So the 24pt font I was seeing in TextEdit was 75% smaller than the 32pt font used in the browser.

Further research about how 96 DPI is used on the web turned up this Surfin’ Safari post on CSS units written by Dave Hyatt back in 2006:

This is why browsers use the 96 dpi rule when deciding how all of the absolute units relate to the CSS pixel. When evaluating the size of absolute units like pt browsers simply assume that the device is running at 96 CSS pixels per inch. This means that a pt ends up being 1.33 CSS pixels, since 96/72 = 1.33. You can’t actually use the physical DPI of the device because it could make the Web site look horribly wrong.

That’s another way to think about this problem: a single point of text on your Mac will be 1.33 times larger in your browser.

Now What?

Now that we know what caused the problem, how can we avoid it?

The key piece of information that’s missing in the “1em = 12pt = 16px = 100%” rule is resolution. If you’re specifying point sizes without also specifying resolution, you can end up with widely varying results. For example, each of these is “24 points tall”:

Comparison

If you’re lucky enough to have a Retina Display on your Mac, you’ll be rendering text at 144 DPI (2 × 72 DPI). That’s 20% larger than the last example above (Windows’ 120 DPI setting.) Display resolution is increasing and so are the number of pixels needed to represent “a point”.

Note that you can’t “fix” this problem by specifying sizes in pixels. If you specify a font size of 16px, your browser will still treat that as 12pt and display it accordingly.

As a designer or developer, you’ll want to make sure that any layout you’re doing for the web takes these size differences into account. Some apps, like Photoshop, allow you to specify the resolution of a document (that’s how I performed my sizing tests.) Make sure it’s set to 96 DPI. Resolution may not matter for exporting images from Photoshop, but it will make a difference in the size of the text in your design comps.

Most other Mac apps will rely on Cocoa’s text system to render text, so if there’s no setting to change the document’s resolution, a default of 72 DPI should be assumed.

An alternative that would suck is to start doing your design and development work on Windows where the default is 96 DPI. That setting also explains why web browsers use this resolution: remember when Internet Explorer had huge market share?

And finally, expect some exciting new features for measuring, inspecting and testing text in your favorite tool for design and development. I’m doing all this precise measurement for a reason :-)

Updated February 25, 2014: It turns out Todd Fahrner first identified this problem back in the late 1990′s. His work with the W3C was instrumental in standardizing on 96 DPI. What’s depressing is that I knew about this work: Jeffrey Zeldman and I even developed the Photoshop filters mentioned on his site. Maybe I should send in my AARP membership form now.

Counting Beans with AppViz 3

In my last post about AppViz, I mentioned that I spent about six months creating an internal accounting tool called BeanCounter. Why the hell did I do that?

Because I don’t want to run my business on financial guesses based on iTunes Connect sales reports.

The Need For Financial Accuracy

For many app developers and iBook publishers, income from iTunes Connect is a significant part of the business’ financial statement. In many cases, the App Store is a developer’s only distribution channel, so it’s responsible for 100% of gross revenue.

When you have a financial statement, you want it to be exact. If your local tax collector comes calling and you show them guestimates, you’re in for a rude awakening. You’ve got a paper trail that’s fishy and that’s going to invite closer scrutiny. It’s the reason every piece of accounting software ever written has a reconciliation feature.

Publishers pay author royalties and many app developers split revenue with a designer. When it comes time to share the monthly Apple ACH deposit, financial guesses are also a problem. Not only will you be giving your partners the wrong amount of money, you’ll eventually be reporting it incorrectly on a 1099-MISC. Your estimated payments can be off by thousands of dollars:

Sales Reports Are Not Accurate

The root of the problem is that all services except AppViz 3 use fluctuating values like fiscal calendar ending dates, payment clearing dates, and currency exchange rates. If you do, you’re making financial guesses.

Don’t take my word for it, here’s what App Annie has to say:

Why do my sales reports in App Annie not exactly match my financial reports in iTunes Connect?

App Annie fetches the sales data from the iTunes Connect “Sales and Trends” section. There are a few reasons why the sales data doesn’t match Apple’s “Financial Reports” and the amount of money you receive from Apple.

Fiscal calendar. Apple uses fiscal months rather than calendar months for payouts. To see Apple’s fiscal calendar, go to “Payments and Financial Reports” and click the “Fiscal Calendar” link. For example, fiscal March 2011 starts on Feb. 27 and ends on Mar. 26.

Clearing of payments. What matters in the case of financial reports is what payments were made during a fiscal month, not how many downloads were made. Since most people use credit cards to buy apps, it can take up to 2-3 days for a payment to go through. Also, some payments never go through!

Currency conversions. Apple’s “Financial Reports” use Apple’s own currency rates, whereas App Annie always uses today’s exchange rate.

Remember that the last two points above have a certain degree of unpredictability, which makes the figures we get from Apple’s “Sales and Trends” slightly different from the figures in Apple’s “Financial Reports”.

That last item is particularly bad: currency fluctuations will give you wildly inaccurate accounting. For example, let’s say you sold €10,000 of product and used the rate on October 28th, 2013 (1.38) to convert that amount to $13,800. Then, Apple deposited $13,400 into your account on November 7th using that day’s 1.34 rate. Hope you didn’t pay your partners using that financial guestimate of $13,800!

The bottom line is this: you cannot rely on estimates in Sales Reports when doing financial reconciliation or partner payouts. Only Financial Reports can be used for this purpose.

Financial Calculations Are Hard

The problem is compounded when you have multiple products with different partners. Figuring out how to divvy up the money turns out to be a hard problem and it’s why everyone else has punted on doing it right. It’s also a competitive advantage for us, so I’m not going to go into any details about how we get the numbers to come out right.

I will, however, give you a peek into the first hurdle you’ll encounter: you can’t use floating point values for financial calculations. If you write code like this, you’ll be in for a world of hurt as errors accumulate:

double subtotal = balanceTotal + 
    (salesTotal + adjustments);
if (subtotal != 0.0) { // is 0.00000001 a zero value?
  rate = deposit / subtotal;
}

Cocoa has an awesome subclass of NSNumber called NSDecimalNumber. Classes like NSDecimalNumber are one of the reasons that “NeXTSTEP had a long history in the financial programming community.”

When using NSDecimalNumber your code will get more verbose than the snippet above. But more importantly, it will now be exact:

NSDecimalNumber *subtotal = [balanceTotal
    decimalNumberByAdding:[salesTotal 
    decimalNumberByAdding:adjustments]];
if ([subtotal compare:[NSDecimalNumber zero]]
    != NSOrderedSame) {
  rate = [deposit decimalNumberByDividingBy:subtotal];
}

It also forces you to think about problems like rounding: do you want to use NSRoundBankers when your code calls -decimalNumberByRoundingAccordingToBehavior:?

Conclusion

I hope you now see why I thought it was wise to spend a lot of time coming up with a system that ensures financial accuracy. I’m also thrilled that this system is now a part of AppViz 3, the app that’s been keeping track of our products on iTunes Connect since the App Store first opened in June 2008.

It also makes me happy that other developers are seeing the same benefits for accurate financial reconciliation. Helping other developers is what makes me tick.

Take a moment to think about how these issues affect your own business and then take a look at AppViz 3. I think your balance sheet, your partners and the tax man will be happy you did.

iPad Not Annoying

In Mavericks, there’s a new notification that reminds users an iPad isn’t charging:

NotCharging

As iOS developers, we spend a lot of time plugging and unplugging devices each day. After you’ve seen this reminder a few dozen times it becomes more annoying than helpful.

So I complained about it on Twitter. And thanks to a pointer by Paul Haddad, I had a hint on how to get what I wanted.

After checking that /usr/libexec/usbd contained the “NoiPadNotifications” string and looking over the usbd manual page, I gave it a shot:

$ sudo defaults write com.apple.usbd NoiPadNotifications \
  -bool YES
$ sudo killall usbd

The next time you plug the device in, the usbd daemon will be started by launchd, but you won’t see any notification. If you change your mind at a later date, just undo the change with:

$ sudo defaults delete com.apple.usbd NoiPadNotifications

Yay!

AppViz, WTF?

AppViz 3 is a major rewrite of a product loved by many developers. Most parts of the app are completely new, but we know that two new features will be particularly contentious: cloud storage and a subscription model.

We’re developers selling a tool used by other developers. We know we can’t bullshit you with marketing buzzwords and a bunch of hand-waving. This post will explain exactly why we’ve introduced these features and let you draw your own conclusions. At a minimum, it will provide insight into the challenges and how we approached their solutions. If we’re lucky, you’ll agree with our pragmatism and continue to support our efforts.

tl;dr We’re not trying to screw you.

Learning from our past…

A product like AppViz should be easy to build, right? It just downloads a bunch of numbers in tab-delimited format, crunches the data and then reports it in graphs and tables. If only that were true.

We all know that the pace of iOS development has been staggering. In just five years, we’ve gone from the first apps on iOS 2 to the radical redesign of iOS 7. The iPhone itself has gone from being available in just the US to selling in a total of 100 countries. We’ve seen a completely new iOS device called the iPad and spectacular hardware improvements across the board. Mac developers also got to join in on the fun at the end of 2010. Publishers started selling iBooks like we sell apps that same year.

All of this, and it’s pretty incredible that many developers are just now celebrating their wood anniversary.

And through it all, Apple is using a time-honored development process: they’re making it up as they go along. Throughout those five years there have been a huge number of changes to the data coming out of iTunes Connect:

  • Changes to the formats and columns in existing reports
  • Changes to the data values in the reports
  • New kinds of reports, like financial and earnings reports
  • New kinds of data, like new regions and sales types
  • New offerings like iAds, Newstand and iBooks

A lot of this data doesn’t come in a nicely formatted file; the only way to get it is by scraping a web page. If you’ve ever done this, you know how fragile it can be: a web developer that makes a simple change to a <div> can ruin your finely crafted parser.

We also all know that Apple never gives away details of its future plans. When things like new regions are added, you’re lucky to get a couple of days notice. And even when you do have some advance notice, like knowing that Mac apps will be added to the reports, you still don’t have any details until they show up on Apple’s server.

At which point, you have a huge problem: every customer needs a new version and they need it now. It doesn’t matter if it’s a holiday or you’re on vacation. You’re under the gun to write some new parsing code, get it tested, and deployed as soon as possible.

If you’re still thinking about how easy it would be to write your own app to track reports, let me share a little of my own experience. Much of the work in the new financial reporting and reconciliation module is based on an internal tool called BeanCounter. I thought this tool would be fairly easy to write: all I had to do was replicate the stuff I had in our Excel spreadsheets.

After six months, I had pretty much covered all the edge cases and weird report formatting issues, but still had to manually download the reports. In fact, this final hurdle of scraping web pages is what eventually led to the partnership between the Iconfactory and IdeaSwarm. So yeah, even developers with over 35 years of professional experience fall for the “this should be easy” naiveté.

Cloud Storage

I’ll be honest. I wasn’t wild about the idea of storing iTunes Connect credentials and our data on a remote server. We live in an online world where security breaches are just another piece of daily news. I knew it would be a lot of work to keep my data safe.

Unfortunately, there’s no API for iTunes Connect (even though we’ve been asking for years.) We’d love the chance to use something like OAuth instead of raw credentials. Unfortunately, that rapid pace of change I mentioned above pretty much precludes a stable API to access iTunes. As a pragmatic developer, you have to go with what you have, not what you want.

The reality for us and many other developers is that our product team keeps growing. Having sales and other financial data locked away on my laptop became more and more of a problem. The information coming from iTunes Connect helps run our business and I needed to share it with employees and partners. If you’re managing iTunes Connect information for some or all of your paying clients, you’ll have a similar problem.

Likewise, there are some kinds of data management that I don’t want to do. For example, adding events in the product details or reading reviews. Those tasks are better handled by a product manager and a person doing support. Again, the focus has shifted from the data itself to the people who manage it.

If you’re going to share data, you need to think long and hard about who’s going to have access and how it’s stored. In our case, there are two kinds of information that need to be protected: your iTunes Connect credentials and the report data that’s collected using those credentials.

We’re using 256-bit AES encryption for the iTunes Connect credentials. If you’re storing passwords with 1Password, you’re using the same encryption. We’re not going to divulge where the keys to decrypt this data are located, but we will say they’re not stored in source code or any other location on the server’s disk. They’re very hard for an attacker to access.

As far as our report data is concerned, we store it on Amazon S3 using Server Side Encryption (SSE). Access to this data is secured using SSL over a low latency connection to an Amazon server which uses AES-256 encryption. Amazon uses this same service for their own business-critical operations.

Passing data to the AppViz server before handing it off to the application on your Mac also has some big benefits for the user experience. Remember how frustrating it was when AppViz 2 couldn’t download your data because of some change that Apple had made on their site? With this new architecture, any changes to the data parser can be made directly on the server and minimize the downtime for all customers. There are monitors in place that let us know when reports aren’t importing correctly.

Finally, storing data in the cloud also allows us to offer new and exciting products. Personally, I’m dying to see things like:

  • The contents of the Dashboard module in an email as soon as the reports are ready to download.
  • Reports available on the web so that people don’t have to install an app to just read the basic information.
  • An iOS app that lets me keep an eye on the business while I’m on the go.

(Note: these are just ideas, not a promise that anything is going to get implemented!)

Private Storage

Still, with all of that said about cloud storage, there are still a lot of developers who are truly independent: one person with one set of reports. And these individuals love to be in complete control of their privacy and data. We want to keep these customers happy, too.

To do this, we’re planning on adding “private storage” to AppViz by the end of the year. This mode will offer the same resiliency to Apple’s changes since report collection and parsing remain on the server. The difference is that no reports or credentials will be stored after the data is exchanged. Now that our goal of implementing a robust download pipeline is accomplished, this secondary mechanism can be started.

When using private storage, iTunes Connect credentials will be passed to the AppViz server to initiate the connections at Apple. Data from those connections will then be collected and passed off to the application running on your Mac. After a successful transfer, the report data will be deleted from the server. No record of your credentials or data will be left behind.

Hopefully, you’ll agree that this is the best compromise between a constantly shifting data source and your own privacy.

Subscriptions

As a developer, you know that you spend a lot of time up front building a product and then amortize those costs over the years that it’s for sale. That initial hump can kill you, but long-term earnings make you do it over and over again :-)

Unfortunately, AppViz is not one of those products. It has huge ongoing costs based purely on maintenance. To give you an idea of the scale, there have been 32 releases for AppViz 2 over the past two years. That’s over one release per month and includes a lot of non-trivial work:

  • 67 new App Store countries
  • Financial regions going from 7 to 25
  • iAd support (with many releases to track changes on the web pages)
  • Report download changes (3 separate releases for changes in April 2012 alone)
  • New Newstand categories
  • New iBooks support
  • Rankings download changes
  • Rate limiting for rankings and reviews downloads
  • Adapting to changes on financial pages
  • New categories on iOS and Mac App Stores
  • Increased rankings from top 200 to 300

In the same time period, there was one release to support Mountain Lion, Gatekeeper and Retina displays.

The increase in the size of the development team is the best metric to show how maintenance has become such a huge cost. Initially there was one developer working on AppViz full-time: now there are three.

So, why don’t we just charge for more of the upgrades? AppViz doesn’t fit the “new features, paid upgrade” model. How would you feel if an upgrade fee was required to download and parse reports with a new region? Or if support was added for a new product category like Apple TV that you’re not using? My answer is that I’d feel like I was being held hostage: the app is broken until I upgrade or I’m forced to pay for something I don’t use.

For AppViz to be a viable venture in the future, the cost of the app must reflect the costs of this ongoing maintenance. In our view, subscriptions are the best fit.

Summary

There you have it, a couple thousand words to explain what used to be a huge “WTF?” Hopefully, this comprehensive essay shows that we’ve thought about the problem and have come up with a solution that’s both pragmatic and viable. If you still have questions or concerns, the folks at IdeaSwarm would love to hear from you.

A Quick Look plug-in for Provisioning

As every iOS developer knows, when your provisioning gets messed up, your life becomes a living hell. I don’t even want to think about how many millions of man hours have been wasted getting broken projects working again.

The root of the problem is always the .mobileprovision files that are kept in your Library > MobileDevice > Provisioning Profiles folder. The file either references a certificate that has expired or a device that no longer exists. There can also be issues with the entitlements that are contained in the profile when Team and App IDs don’t match.

A large part of the problem is that this file is not directly readable in a text editor like all the other parts of our Xcode projects. The file is encoded in the Cryptographic Message Syntax (CMS) described in RFC 3852.

After doing a bit of research, I found that decoding the payload of this file format is very simple thanks to some helpful functions in Apple’s Security framework. And once decoded, these .mobileprovision files contain nothing more than Property List (.plist) with a lot of useful debugging information.

I had previously been using a Quick Look plug-in from MacMation, but that site’s gone offline and the plug-in no longer worked in Mavericks. I had originally thought the problems on Mavericks were due to the new code signing requirements, but the root of the issue was that the content type UTI changed from com.apple.iphone.mobileprovision to just com.apple.mobileprovision so Quick Look ignored the old plug-in.

Eventually, I decided to write my own Quick Look plug-in and add a bunch of new stuff that I had been wanting to display:

  • Developer certificates: Making it easier to verify that your keychain items match what’s in the profile.
  • Provisioning Profile UUID: When someone on the project team checks in a new Provisioning Profile in the Build Settings, the only information you have is that UUID of that new file. Showing the UUID lets you find the right match.
  • Entitlements: Checking the Push Notification environment, ubiquity container identifiers, and keychain access groups is essential for any app that uses Apple’s services.
  • Links: Whenever the provisioning is broken you spend a lot of time in various sections of the Dev Center. Why not make it easy to get there?

The results of a few days work can be found on GitHub. If you’re lazy like I am, just download the .qlgenerator file and pop it in your Library > QuickLook folder. To get Quick Look to recognize the new plug-in you’ll need to either logout or poke Quick Look from the command line:

$ qlmanage -r

While you’re at the command line, do this to allow text to be copied from any Quick Look plug-in:

$ defaults write com.apple.finder QLEnableTextSelection -bool TRUE
$ killall Finder

(Macworld has more info about this nice hidden setting. Being able to copy text from a PDF preview is pretty damn handy!)

The next time you’re stuck in the Fifth Circle of Provisioning Hell, this simple plug-in may just bring you back to life. Viva!

Mac App Store Receipts and Mavericks

The storeagent and I aren’t getting along too well these days.

We’re in the process of getting a new release of xScope ready for release. As a developer tool, we’ve been compatible with Mavericks for several months now, but there are some minor bug fixes that we’d like to get out before the new version of OS X ships.

As you might be aware, this is the first time I’ve done a build on Mavericks itself. Things haven’t exactly been smooth sailing.

Today’s revelation is how storeagent creates the /Contents/_MASReceipt/receipt file in Mavericks. It’s subtly different, and will confuse the heck out of you until you understand what’s going on.

For the past few days, I’ve been testing a beta release of the .pkg using the standard command:

sudo installer -store -pkg /tmp/xScope.pkg -target /

This version had a CFBundleShortVersionString of “3.6.2b1″. The installer and receipt checking code was working great.

Until I did the final build and used the version string “3.6.2″. I got this message after I double-clicked the app and entered my Test User Apple ID:

After checking the code signing, bundle IDs and all other parts of the app, I finally fired up the debugger and discovered that the receipt validation code was failing when checking receipt attribute type 3, the Application version field (in Table 1-1).

After decrypting and checking the receipt payload, the value was “3.6.2b1″ not the version I just installed. Where did this old version number come from? Why did following the advice in the dialog and deleting the app not fix the problem. How come this old receipt kept showing up no matter what I did?

Receipts from older versions had never been a problem in previous versions of OS X, so there must be some new behavior in Mavericks. And it took me almost a whole day to figure out that new behavior.

It turns out that storeagent is doing some kind of in-memory cache of receipts that have been downloaded from iTunes. Since a network connection is needed to retrieve the receipt, keeping it around would prevent a little bit of network traffic. In previous versions, the receipt was presumably recreated each time it was requested, so you always had a fresh copy.

The workaround is fairly simple. It even gives me a bit of pleasure at this point:

$ killall -KILL storeagent

You’ll need to delete the app at this point and re-install it using:

$ sudo installer -store -pkg YourApp.pkg -target /

When you relaunch your app, you’ll see the Apple ID login dialog. Since storeagent is launched on demand by launchd, a new process will be started at this point. After entering your Test User credentials, a new, and valid, receipt will be written into the _MASReceipt folder.

One could imagine this caching of receipt data being a problem with apps that are downloaded from the App Store. If someone never reboots between two versions of the same app being “Ready for Sale”, it may trigger the same problem. I have no way to test this hypothesis.

For any Apple folks that might be reading, here you go: rdar://problem/15283740

Code Signing and Mavericks

The Change

Very simply put, you can no longer sign a bundle (like your .app) if any nested bundle in that package is unsigned. These nested bundles are things like helper executables, embedded frameworks, plug-ins and XPC services.

The result is that you’ll need to update your Xcode projects as soon as you start building on 10.9. It’s taken me several days to understand what these changes are, and with the help of Perry Kiehtreiber on the developer forums, I’d like to share what I’ve learned.

(Yes, I realize this essay is going to break the NDA, but since Apple is asking us to submit apps for Mavericks, I want as many developers as possible to avoid the utter confusion I faced earlier this week.)

The Effect

So what happens when you do your first app build on 10.9 using Xcode 5.0.1? If you embed a framework that’s unsigned, like the very popular Sparkle.framework, you’ll see a message during the final CodeSign build phase:

CodeSign build/Release/xScope.app
    cd /Users/craig/Projects/Mac/xScope
    setenv CODESIGN_ALLOCATE /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate
    Using code signing identity "Developer ID Application: The Iconfactory"
    /usr/bin/codesign --force --sign D2A3FE1814B0BA31B1924F1C3C3B5C89643FBED5 --requirements =designated\ =>\ anchor\ apple\ generic\ \ and\ identifier\ \"xScope\"\ and\ ((cert\ leaf[field.1.2.840.113635.100.6.1.9]\ exists)\ or\ (\ certificate\ 1[field.1.2.840.113635.100.6.2.6]\ exists\ and\ certificate\ leaf[field.1.2.840.113635.100.6.1.13]\ exists\ \ and\ certificate\ leaf[subject.OU]\ =\ \"RYQWBTQRPT\"\ )) /Users/craig/Projects/Mac/xScope/build/Release/xScope.app
/Users/craig/Projects/Mac/xScope/build/Release/xScope.app: code object is not signed at all
In subcomponent: /Users/craig/Projects/Mac/xScope/build/Release/xScope.app/Contents/Frameworks/Sparkle.framework
Command /usr/bin/codesign failed with exit code 1

** BUILD FAILED **

The codesign command is reporting that “code object is not signed at all” and Xcode is adding the “In subcomponent” to tell you which framework is at fault (it could just have easily been HockeyApp or any other third-party framework you use.)

So how do you go about fixing this?

The Wrong Way

In the past, many developers have relied on codesign‘s --deep option to make sure the entire bundle is signed. Specifying this option in “Other Code Signing Flags” will get rid of the error during the build, but all you’re doing is just postponing the pain.

The reason is that --deep recursively signs the nested bundles. As it does this, it applies the parameters for the top-level bundle to all the nested bundles. Things like your app’s entitlements will cause the resulting bundles to not be valid.

In fact, if you try to download and install the resulting app, Gatekeeper will notify your customers that your app is damaged and can’t be opened, with a default button to move it to the Trash:

You’ll see the same thing if you check the binary using the command line:

$ spctl --verbose=4 --assess --type execute build/Release/xScope.app
build/Release/xScope.app: a sealed resource is missing or invalid

The Right Way

What’s the right way to make sure the embedded framework is correctly signed? The answer is to add another Build Phase to your target.

If you’re embedding frameworks, you’ll have a “Copy Files” phase that moves things like Sparkle.framework into the Frameworks destination. Just after this Build Phase, add a Run Script with the following shell commands:

LOCATION="${BUILT_PRODUCTS_DIR}"/"${FRAMEWORKS_FOLDER_PATH}"
IDENTITY="Developer ID Application: The Iconfactory"
codesign --verbose --force --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A"

This short script tells Xcode to sign the framework that’s just been copied into the build product. In this case, it’s using the Developer ID for Gatekeeper. If you were doing a build for the Mac App Store you’d use your “3rd Party Mac Developer Application” identity. Add a codesign command for every framework you use.

If you have other embedded code, such as helper executables, plug-ins or XPC services, you’ll need to sign them appropriately after copying them into your app bundle.

Updated October 18th, 2013: Another alternative is to set the code signing identity in the Build Settings of the frameworks you’re building from source. The trick here is that the identity of the framework needs to match the identity of the app itself. You can’t have have an App Store distribution identity for the framework and a Developer ID for the app. I found it much easier to explicitly re-sign the frameworks than to pass configuration settings from MyApp.xcodeproj to MyFramework.xcodeproj. It’s also easier to manage because the project changes are the same for binary-only frameworks (like Sparkle) and frameworks we build from source (like Chameleon).

The Checks

You’ll want to do a quick check of the build product before uploading it to either your website or iTunes Connect. The first thing you’ll want to do is check the signed bundle meets its designated requirement:

$ codesign --verify --verbose=4 build/Release/xScope.app
--validated:/Users/craig/Projects/Mac/xScope/build/Release/xScope.app/Contents/Frameworks/Sparkle.framework/Versions/Current/.
build/Release/xScope.app: valid on disk
build/Release/xScope.app: satisfies its Designated Requirement

If there’s a problem, you’ll see a message that the app “does not satisfy its designated Requirement”. To view information about the signed code or the designated requirements, you can use these commands:

$ codesign --display --verbose=4 build/Release/xScope.app
$ codesign --display --verbose=4 build/Release/xScope.app/Contents/Frameworks/Sparkle.framework
$ codesign --display --requirements - --verbose=4 build/Release/xScope.app
$ codesign --display --requirements - --verbose=4 build/Release/xScope.app/Contents/Frameworks/Sparkle.framework

If this is a build you’ll be uploading to your website, you’ll want to make sure it will be accepted by Gatekeeper (and not display the “damaged” dialog.) Use spctl to do this:

$ spctl --verbose=4 --assess --type execute build/Release/xScope.app
build/Release/xScope.app: accepted
source=Developer ID

If this is an App Store build, you MUST check the .pkg file that gets uploaded to iTunes Connect (see the next section and you’ll see why I say MUST.) If you use productbuild to create the package manually, you’ll already have a .pkg file to test.

For those of you who submit archives directly from Xcode, you can generate the .pkg file using the command line:

$ xcodebuild -exportArchive -exportFormat PKG -archivePath /path/to/your.xcarchive -exportPath /tmp/CHOCKS -exportSigningIdentity "3rd Party Mac Developer Application: CHOCK LOCK INK” -exportInstallerIdentity "3rd Party Mac Developer Installer:  CHOCK LOCK INK"

You can find the path to your .xcarchive by selecting it in the Organizer and then using the Editor > Show in Finder menu item. The command above will create a /tmp/CHOCKS.pkg. Yes, you now have CHOCKS PACKAGE IF YOU KNOW WHAT I MEAN

(A quick side note, if you use xcodebuild, it got a lot of love in Mavericks. Make sure to check out the man page.)

To check out CHOCKS.pkg, run the installer with the -store option:

$ sudo installer -store -pkg /tmp/CHOCKS.pkg -target /
installer: Note: running installer as an admin user (instead of root) gives better Mac App Store fidelity
installer: CHOCKS.pkg has valid signature for submission: 3rd Party Mac Developer Installer: The Iconfactory
installer: Installation Check: Passed
installer: Volume Check: Passed
installer: Bundle com.artissoftware.mac.xScope will be installed to /Applications/xScope.app
installer: Starting install
installer: Install 0.0% complete
installer: Install 13.8% complete
installer: Install 22.2% complete
installer: Install 47.6% complete
installer: Install 88.3% complete
installer: Install 100.0% complete
installer: Finished install

Now, sign out of the App Store and launch the app that was just installed in your Applications folder. If everything is OK, you’ll see the prompt for your Apple ID and a receipt will be written in the app’s _MASReceipt folder.

But not always.

(For more information on testing installer packages, check out the Testing section in my Mac App Store Guide.)

The Suckage

After being installed for the first time, some apps never get a receipt when they are launched on Mavericks. The app starts up, sees that there’s no receipt in /Contents/_MASReceipt and signals that it’s missing by exiting with a 173 code. Normally, storeagent will recognize this and prompt for an Apple ID. After valid credentials are provided, the receipt is written and the app is launched again.

Several developers, myself included, have noticed that after exiting with a 173, only the following is logged in the console:

Oct 17 11:59:03 Myrtle.local storeagent[72031]: Unsigned app (/Applications/Twitterrific.app).

If this happens to you, it seems your only course of action is to not validate the receipt. Your code will launch fine if you never return a 173. Which, of course, sucks because it’s then trivial to pirate your the app.

For any Apple folks that might be reading this, check out the Radar: rdar://problem/15254213

Updated October 18th, 2013: Developers that are doing their builds on 10.8 also need to watch out for this problem. As a workaround, I tried building the product on 10.8.4 using Xcode 5.0. The resulting .pkg exhibited the same behavior at launch time as the one created with the Xcode GM on Mavericks.

The Workarounds

Xcode sometimes has problems generating a valid designated requirement. When you check the designated requirement, you might see this:

$ codesign --verify --verbose=1 build/Release/Twitterrific.app
build/Release/Twitterrific.app: valid on disk
build/Release/Twitterrific.app: does not satisfy its designated Requirement

$ codesign --display --requirements - build/Release/Twitterrific.app
Executable=/Users/craig/Desktop/BugReport/Ostrich/build/Release/Twitterrific.app/Contents/MacOS/Twitterrific
designated => anchor apple generic and identifier Twitterrific and (certificate leaf[field.1.2.840.113635.100.6.1.9] /* exists */ or certificate 1[field.1.2.840.113635.100.6.2.6] /* exists */ and certificate leaf[field.1.2.840.113635.100.6.1.13] /* exists */ and certificate leaf[subject.OU] = RYQWBTQRPT)

If you look closely, you’ll see that the identifier used for the requirement is incorrect: it should be com.iconfactory.Twitterrific, not just Twitterrific. The workaround for this bug is simply to set the identifier explicitly in your Build Settings. In our case, we added --identifier com.iconfactory.Twitterrific to Other Code Signing Flags.

Updated October 18th, 2013: It looks like this bug happens when you precompile your Info.plist. Thanks to Chris Liscio for verifying that problem. Make sure to dupe it, if you’re affected!

As to why you’d want to precompile your Info.plist, there are two good reasons.

The End

There you have it: a short summary of my last three days of confusion caused by new Gatekeeper requirements, issues with Xcode and bugs in Mavericks. Hopefully, this essay will save you some of that same agony.

Sonderklasse

Let’s talk about cars for a second.

If you’re driving a car manufactured in the past five years, it’s likely it has anti-lock brakes. A standard feature at this point in time, but who came up with it first?

Mercedes-Benz first introduced it on the S-Class line back in 1978. It was a revolutionary technology and the first hint of how digital electronics would change the course of the automotive industry.

The S-Class pioneered many other safety innovations: crash crumple zones, air bags and traction control are a few of the most notable. This line has also become synonymous with comfort and luxury: it was the first Mercedes to be available with an automatic transmission. There’s even an armored version with a customized crocodile, gold and birch interior!

It’s clear this car is in a class by itself: “Sonderklasse”

“S-Class” is an anglicisation of “”S-Klasse,” a German abbreviation of “Sonderklasse,” which means “special class” (in the sense of “a class of its own”). In automotive terms thus refers to “a specially outfitted car”.

The current S600 model sports a 5.5L twin turbo V12 motor with a 7-speed automatic transmission. Enough power to propel a 4,950 lb vehicle from 0 to 60 mph in just 4.5 seconds. And that’s before AMG gets their hands on it. Impressive automotive technology, to be sure.

More importantly, these special cars have been recognized as a driving force behind the company’s success.

And now…

Another premium technology brand has an S-Class product: iPhone 5s.

“Craftsmanship”, “Power”, “Safety & Security”, “Ahead of its time”. Are we talking about a car or a phone?

Look at the words that dominate the iPhone 5s features page: Forward thinking. That’s a strategy that’s worked superbly at Mercedes-Benz for almost 60 years. I suspect Apple will get a pretty good run out of it, too.