Brain farts

What happened?

In spite of plenty of advance warning from Twitter, we got caught by the Twitpocalypse bug.

For the 2.0.1 release, we had tested our software extensively. I actually wrote an emulation layer on top of the code that reads data from Twitter that added a large number to every ID read from Twitter. This testing uncovered several bugs which were fixed and incorporated into the MGTwitterEngine open source code.

Unfortunately, this testing didn’t take into account that the library we use to parse the data (YAJL) includes a range check that ensures signed values, which are allowed by the JSON specification, will fit into 32-bit storage. The “YAJL Error 3” is that library telling us that the range check failed.

In hindsight, I should have fed the parser some large unsigned integers. In the wonderful world of software development, we call this a brain fart.

How did we respond?

Unfortunately, the Twitpocalypse occurred on Friday evening, just as my wife and I were heading out for a high school graduation. I quickly shot off an email to Twitter asking a few questions (I had mistakenly thought that there were problems with data being returned by an authenticated connection.)

After a couple of hours of iPhone email and SMS, it was clear that we were going to need do a new release. We had a critical bug that affected thousands of users. And our fear was this submission would take longer than normal: many developers are submitting 3.0 updates.

Life officially sucked at this point, and the graduation ceremony was memorable for all the wrong reasons.

After getting a few hours of fitful rest, I opened up Xcode on Saturday morning and started looking for the problem. It took just a few minutes to find the bug and another few minutes to fix it. Our concern at this point was getting the update to users.

Apple saves the day

In spite of it being the middle of weekend after a busy week at WWDC, we were able to get in touch with Apple Developer Relations. Our contact was able to expedite the approval of the application.

To say that this was a relief would be the understatement of the century. The update for the free version started showing up in iTunes on Sunday evening. The paid version was updated on Tuesday morning.

What can we learn from this?

Developers are human. We make mistakes. It’s interesting to note that Loren Brichter and Buzz Anderson, fellow developers whose skills I hold in high regard, were also affected by the Twitpocalypse. It really does happen to the best of us.

It’s also interesting to see that Loren and Buzz reacted in the same way I did: by fixing the problem ASAP. In Loren’s case, that meant finding a hotel with WiFi in order to distribute his update. Buzz released a new beta in a matter of hours.

In my experience, these brain farts are problems with easy fixes. It’s something like checking for invalid bounds, getting a Boolean state wrong, or something else of that nature. It’s not a complex problem: it’s an oversight.

The problem, therefore, is not how fast we can react to fixing critical bugs, but how fast the App Store reviewers can react with an approval.

As a device that’s constantly connected to the Internet, the iPhone taps into a stream of data that is unpredictable. Data in “the cloud” can change at any time, and web applications naturally adapt to these changes with a quick deployment strategies. Those of us who are building client applications on top of this network infrastructure need the ability to adapt quickly, too.

Security issues are another area where a quick response is a requirement, not a luxury. If I discover something that puts a user’s private data at risk, it’s my responsibility to fix the problem as quickly as possible. Time spent in a review queue is time spent being exposed to a flaw.

At the same time, we shouldn’t blame Apple for these delays in reviewing applications. In the year that they have been selling our products, the App Store has been more successful than anyone imagined. It’s clear to me that reviewers and others involved in the approval process are overwhelmed by this success.

How can we fix it?

Fortunately, I think there’s a simple way to solve this problem for all developers selling products on the App Store. The inspiration for this solution will be obvious to anyone who’s used Apple’s Developer Technical Support (DTS.)

When you purchase an ADC membership, you are given a number of “incidents”. These DTS incidents can be used when you have a problem that can’t be solved through documentation, support forums or hours and hours of debugging. It’s for the hard stuff, and usually involves getting an engineer at Apple involved to understand and fix the issue.

As a developer, I’m very careful to use these incidents wisely: they are a last resort. There are some years where I don’t use them at all, those are the good years.

A similar system could be put in place for critical bug fixes on the App Store. If every developer was given one or two “prioritized reviews,” it would act as insurance for the brain farts. You’d have a way to raise a flag and say “I need special attention for a critical bug.”

If another developer has a critical bug, I have no problem with my review process for a feature release taking a little longer. And since prioritized reviews would be a scarce resource, they won’t be open for abuse because developers will think twice before using them.

Because it’s not a matter of if you have a brain fart that leads to a critical bug, it’s a matter of when.

A phone by any other name would smell as sweet…

The general consensus is that there will be a new iPhone announced next week. I, like others, think it’s going to have new features and capabilities. But how is Apple going to label this new device?

iPhone 3G?

It’s entirely possible that Apple will keep the same name as the previous version. There’s precedence in the Mac product line: we’ve had MacBook Pros for several years now with various ways of distinguishing different product iterations (“Late 2008”, “Unibody”, etc.)

Given Apple’s reluctance to publish any of the device’s hardware capabilities, this would seem to make sense.

Except for one small problem. It’s likely that the new iPhone will have a faster processor and more memory. Some applications will be written to take specific advantage of these improvements. And that, of course, means that you need a way to let iTunes customers know if those applications are compatible with their device.

I don’t see Apple putting “iPhone 3G (2008)” and “iPhone 3G (2009)” anywhere in the iTunes UI. It’s just too confusing.

iPhone 4G?

What about having artificial product designations like they did with PowerPC desktop Macs: “G3”, “G4” and “G5.”

The problem here is that 3G refers to a network service, not a product generation. If this naming convention is used, customers are bound to wonder if a “4G” product works on their 3G network.

iPhone 3G Plus?

So maybe they do something like we’ve seen with the iPod product line. Using terms like “Classic”, “mini”, “nano” and “shuffle” to differentiate the products.

The problem here, of course, is that there’s only one product on sale. Unless Apple plans to keep the current product on sale (at a reduced price,) this just doesn’t make sense.

I’m not going to make any predictions on the device’s name, but I will be paying close attention. The ultimate choice is likely to offer some subtle clues regarding Apple’s plans to evolve this product in future revisions.