It’s pretty clear that the App Store is a huge hit. We’re all loving the ability to customize our iPhones and iPod touches with cool new software!
But with any big new release, there are problems that didn’t pop up while beta testing. As iPhone developers, we’re finding ourselves in a position where we can’t help customers who are encountering these issues.
Let’s take a couple of examples with our app Twitterrific. My friend Jeffrey Zeldman reports a problem where the application would crash just after launching. Another developer friend, Alex King, is having a problem with an authentication alert appearing when it shouldn’t. As a conscientious developer, I want to help these people and fix these bugs. The problem is that I have no tools to do so.
Jeffrey’s problem appears to be something with data that is stored on the phone. Alex’s problem is likely to be an issue with how data is being loaded from the network (from Twitter.) Note that I say “appears” and “likely”; I don’t know for sure, and that’s what is bugging me.
The first problem is knowing where the crash is occurring. The iPhone generates a crash report that automatically gets synced with iTunes. The crash report is stored in ~/Library/Logs/CrashReporter/MobileDevice. Unfortunately, this crash report is “raw” and developers don’t have tools to make it easy to understand (e.g. “symbolicating” crash reports only happens when they are loaded through Xcode’s device organizer.)
And even if I could interpret these crash reports, I’d be faced with another problem. There’s no way to gather additional information about what’s happening on Jeffrey’s and Alex’s device. With a desktop application that’s acting up, developers will often add logging and other kinds of output that track what is occurring around the time of a crash or other misbehavior.
In Jeffrey’s case, I would want to dump out the internal database that’s in use at the time of the crash. For Alex’s bug, I’d want to track the network requests to Twitter and the corresponding response. It’s easy to add this logging to Twitterrific, but the only way to retrieve the results is if you are a registered developer. There is no way I’m going to ask Jeffrey and Alex to spend $99 and install Xcode just so I can collect some debug output.
Assuming that I could get some debugging output, the next step in resolving these problems would be to create a special build with a proposed fix. After having Jeffrey and Alex verify the fix, I’d distribute that same build to a larger group of testers so that we could test for regressions (e.g. we don’t break some functionality in the process of fixing something else.)
The big problem here is that the only way to install software on an iPhone or iPod touch is with the App Store. There are also no provisions for beta testing. Without the ability to sign code, there is no way for a user to get code onto a device: most users fall into this category.
The only way to “test” a fix is to release the changes to tens of thousands of users. It’s the developer equivalent of playing Russian roulette.
(Note: there may be workarounds to some or all of these problems, but with the NDA in place, it’s difficult for developers to share their experiences and solutions.)
Apple has done an fantastic job with the tools that let us develop iPhone software. That’s clearly evident from the fantastic work we’ve seen displayed in the App Store. Unfortunately, the tools that developers use to analyze and debug problems are sorely missing at this point in time.
It’s our hope that this essay will do two things:
- As a user, please be extra patient when developers tell you that they are working on a problem—it’s hard work at the moment and the time it takes to resolve an issue will be longer than with a desktop application.
- We hope that bringing these shortcomings to Apple’s attention will help them address the issues and improve the iPhone SDK.
Until then, these problems will be bugging us all.
Updated July 15th, 2008: If you’re a developer, please feel free to submit a duplicate (“me too”) bug on the following Radars:
- rdar://problem/6073498 Developers do not receive crash reports automatically
- rdar://problem/6073509 Crash reports from users are not easily symbolicated
- rdar://problem/6073517 No way to beta test an iPhone application
Updated July 16th, 2008: After fixing bugs for customers, Brent Simmons notes that there are frustrations with the final part of the development cycle: making public releases.
Updated July 23rd, 2008: Brandon Sneed has discovered some techniques for doing remote debugging of iPhone applications.
Updated August 6th, 2008: We don’t have to play Russian roulette anymore.
Updated August 8th, 2008: I discovered how to “symbolicate” crash logs.
Updated November 10th, 2008: Getting preferences and data from customers who are having problems just got a lot easier.