Dealing with memory loss: the crash

Now that we’re beta testing and able to symbolicate our crash logs, let’s look into one of the crashes that they helped me solve in version 1.0 of Twitterrific.

The crash was happening during the posting of a notification. The backtrace looked something like this:

Program received signal:  “EXC_BAD_ACCESS”.
#0	0x300c87ec in objc_msgSend
#1	0x30675b0e in _nsnote_callback
#2	0x3025380c in _CFXNotificationPostNotification
#3	0x30673f46 in -[NSNotificationCenter postNotificationName:object:userInfo:]
#4	0x3067aa00 in -[NSNotificationCenter postNotificationName:object:]
#5	0x000029fe in -[TwitterrificTouchAppDelegate startTweetRefreshWithMessagesAndFavorites:] at TwitterrificTouchAppDelegate.m:183

The method that initiated the crash was sending IFStartTweetsNotification. This notification allows several views in the UI to reconfigure themselves while the application is refreshing.

One of those views is the toolpad that you see at the bottom of the detailed tweet view. It is a subclass of UIView and is instantiated like this:

- (id)initWithFrame:(CGRect)frame;
{
  self = [super initWithFrame:frame];
  if (self)
  {
    ...
    [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(refreshStarted) name:IFStartTweetsNotification object:nil];
  }
}

When the view was being freed, this code was being used:

- (void)dealloc
{
  ...
  [super dealloc];
}

I had looked at all of my code and was so convinced that this was a UIKit problem that I filed a Radar. But it turns out that it really was my fault: I had overlooked this little bit of code that gets defined when you create a new view controller:

- (void)didReceiveMemoryWarning {
 [super didReceiveMemoryWarning]; // Releases the view if it doesn't have a superview
 // Release anything that's not essential, such as cached data
}

Simple enough to understand: whenever the system determines that it is running low on memory, it starts to call these methods. The default implementation is to free a view if it’s not a part of the current view hierarchy.

With a desktop application, views are never released without an explicit action. (And it’s particularly hard to free them if they are instantiated via a NIB.) As a result, some of them us have gotten a bit sloppy with our view cleanup code. And that sloppiness is what bit me.

When my toolpad was out of view, as it is when you are looking at the main list, and a memory warning was issued, the detail view was released. Because the toolpad was a subview of that view, it was also released.

So what happens when you send a notification to an object that no longer exists? That backtrace I showed you above.

The fix was painfully easy:

- (void)dealloc
{
  [[NSNotificationCenter defaultCenter] removeObserver:self name:IFStartTweetsNotification object:nil];
  ...
  [super dealloc];
}

The lesson to be learned here: if you’re doing any kind of registration or allocation in your init methods, make sure that you have corresponding action in your dealloc method. Even if you don’t really need them on the desktop.

In my next essay, I’ll show you how to clean up memory when getting one of these warnings. It’s not as straightforward as you might think.

Note: If Apple feels that sharing this information is outside of the bounds of the NDA, I’ll be forced to remove this post. If you want, pretend like it’s 1995 and print this out instead of bookmarking it.

Symbolicatifination

Now that we’re all beta testing, we’ll hopefully get some crash logs from testers. But you’ll quickly realize that these crash logs don’t look as good as they do when you pull them off the device with Xcode’s Organizer: there are no symbols and fricken’ useless because you can’t tell where the code is crashing.

Thanks to the free-flowing information regarding this wonderful SDK we’re all using, it took me awhile to figure out how Xcode “symbolicates” these crash logs. The secret is here:

/Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Plug-ins/iPhoneRemoteDevice.xcodeplugin/Contents/Resources/symbolicatecrash

I copied this script into my ~/bin folder so I have easy access to from the command line. You’ll probably want to put it somewhere in your $PATH, too. I chose not to use a symbolic link because I don’t want to lose this valuable script during an Xcode upgrade.

After you’ve gotten things setup how you’d like, it’s a simple matter of running the script like this:

% symbolicatecrash Twitterrific_2008-07-21-191314_Nero.crash

Note that the script uses Spotlight to locate the .dSYM files used by atos. (Check the top of the PERL source code for a summary of how this tool works.)

This has a very important implication for your iPhone app development. It is absolutely essential that you save your .dSYM files for any release you make public. You also need to make sure that Spotlight is able to index those copies. (Some of us like to keep Spotlight out of project folders.)

I’d also suggest taking a look at the source code for the script. You’ll see that architectures besides “armv6” are supported: if you have a need to symbolicate logs from “i386”, “x86_64”, “ppc” or “ppc64”, it should work fine. There are also some options for parsing the crash log that I don’t fully understand. Yet.

Again, if Apple feels this information is crossing over the NDA line, I’ll be removing this essay. It’s probably a good idea to save it for reference. In any case, enjoy your newfound ability to symbolicate!

Update August 14th, 2008: symbolicatecrash has a bug that prevents it from working correctly when there is a space name in the path where the .dSYM files are being archived. Your app symbols won’t be included in the output if they are stored in a folder like “~/Projects/dSYM Archive”: use “~/Projects/dSYM_Archive” instead. (rdar://problem/6150458).

Update August 14th, 2008: If you add this script to the end of your build, it will automate the process of creating the .dSYM and .app archive. You’ll need to update the configurations which will be checked: $CONFIGURATION = “Distribution-Free” is specific to my project. Note that the .app binary that’s saved is not the same one that ends up in your build folder. It hasn’t been compressed or code signed (rdar://problem/6150088), but it appears to have enough information for symbols to be resolved with atos.

Update September 25th, 2008: If you’re having problems locating the crash logs, take a look at this article. It explains where iTunes puts the files on Mac OS X, Windows XP and Vista.

Update December 11th, 2008: Bryan Henry has discovered and fixed a bug with symbol name lookups. If you’re experiencing this problem, duping the Radar wouldn’t be a bad idea either.

Beta testing on iPhone 2.0

NOTE: It’s not clear if the information in this essay is covered by the NDA or not. The instructions presented here are either referenced on Apple’s own site or referenced in the publicly available version of Xcode. If requested, I will remove this post.

During Steve Jobs’ WWDC Keynote the announcement of Ad Hoc distribution meant one thing to most of us iPhone developers: beta testing. Unfortunately, the instructions for setting up this beta testing environment are not well documented. I used the following steps to setup a beta test for Twitterrific on the iPhone.

Updating Xcode for beta builds

This is the part of the whole process that eluded me the most. There is a new requirement to add entitlements to your Xcode project. Luckily, someone pointed me to this discussion on Apple’s website. It provides the following steps:

  1. Open up your ad hoc provisioning profile in Text Edit and verify that it contains a “<key>get-task-allow</key>”. If it does not, generate a new provisioning profile using the Program Portal.
  2. In Xcode, select File > New File…
  3. In the dialog source list, under iPhone, select “Code Signing”. Select “Entitlements” then click the Next button.
  4. Name the file “dist.plist” and put it in the root of your Xcode project. Click Finish.
  5. Open dist.plist and uncheck the “get-task-allowed” checkbox. Save the file.
Now that you have setup the entitlements, you’ll want to create a new configuration for your beta builds. I found it easiest to make a copy of my existing App Store distribution configuration and change the code signing setup:
  1. Make sure that you new beta configuration is the Active Configuration. Xcode has some problems if you try to modify Code Signing parameters on a configuration that is not active.
  2. Use Project > Project Settings to open the build settings. Change the “Code Signing Provisioning Profile” to be “Any iPhone OS Device” and then select the name of the provisioning profile you chose in the Program Portal. If you don’t see the name listed, make sure that the .mobileprovision file is located in ~/Library/MobileDevice/Provisioning Profiles and that the “Code Signing Identity” is set to “iPhone Distribution”.
  3. Use Project > Edit Active Target to open the target settings. Change the “Code Signing Entitlements” to be “dist.plist”.
  4. Clean the build and you should then be able to build a beta release.
Xcode can get confused when changing provisioning and other code signing settings, so don’t be afraid to quit and restart to get things synced up.

Adding beta testers

Of course the next step is to sign people up for your beta test. The most important thing to get from them is their device ID. Each iPhone or iPod touch has a unique identifier. You’ll need this to add them to the Ad Hoc distribution.

There are several ways to get this information. iTunes will display the Identifier if you click on the Serial Number after selecting the device in the source list. Pressing Cmd-C (Mac) or Ctrl-C (Windows) will copy the 40 character hex string to the clipboard.

If your beta tester is on a Mac, another solution is to use the iPhone Configuration Utility, which can be downloaded from Apple’s site. After selecting the device, you’ll see the same Identifier field which can be copy and pasted as text.

Finally, you can also use Erica Sadun’s Ad Hoc Helper. Your beta tester can use this application to send you an email with the device’s information directly from the device.

Creating a beta build

After collecting device identifiers from your beta testers, you’ll need to go into the Program Portal and add the devices. I’ve found that the easiest way to manage this is by using the tester’s name as the device name.

Once you’ve finished entering up to 100 device identifiers, you’ll need to add these devices to the your Ad Hoc provisioning profile (Edit > Modify.)

After you’ve finished updating the profile, Download the .mobileprovision file and move it into ~/Library/MobileDevice/Provisioning Profiles.

Now quit and restart Xcode so that it recognizes the new provisioning and perform the following steps:

  1. Select the beta configuration you setup earlier.
  2. Open your project settings with Project > Edit Project Settings.
  3. Update the “Code Signing Provisioning Profile” to use the name of the provisioning you just installed in MobileDevice.
You’ll notice that it’s quite a bit of work to update the provisioning, and since you’re modifying the contents of the project, you’re going to need to checkin .xcodeproj changes to your version control system. My recommendation is to get all your beta testers lined up and do it all at once.

Distributing and installing the beta

At this point, you can do a beta build and it can be run by your beta testers. The only thing left to do is getting them the software.

To be honest, I don’t know how this part works for users that are on Windows. I’m sure it’s possible, but you’re on your own as far as the steps involved.

Updated August 9th, 2008: Several readers have informed me that you can drag both the provisioning profile and the .app folder to the Applications section in iTunes on WIndows. Once there, they will get moved over to the device with the next sync.

For beta testers on the Mac, you’ll need to send them two pieces of data: a ZIP file that contains the .app bundle created by Xcode and a copy of the Ad Hoc .mobileprovision file in the MobileDevice folder.

To install the .mobileprovision file, the beta tester can just drag it onto the iTunes icon in the Dock. After unzipping the .app bundle, your tester can drag the application into Application in the iTunes Library. The next time they sync, the beta version of your application should appear on the device.

If your testers are managing a lot of different applications, they may find that the iPhone Configuration Utility is easier to use than iTunes. It has facilities to manage multiple devices, provisioning profiles and applications.

Hopefully you’ll find this information useful and allows you to increase the overall quality and reliability of your iPhone application. Happy testing!

Listeners found this review helpful

A major feature of the App Store are the user reviews about the software being offered. There’s just one problem: software is not music. I’ve never had an MP3 crash or lack features. Applications also evolve and improve; I’m pretty sure the Jimi Hendrix track I’m listening to right now is the same one he recorded in 1969.

The App Store in iTunes fails to address these fundamental differences between their latest offering and what has been offered previously (media.) There is so much potential here: iTunes could be a great way for developers to collect feedback and find problems. Instead, we get gems like these:

The icon to this App scares me so much… That I’m too afraid to install the App. That bird looks angry like it wants to peck my eyes out for even concidering [sic] to install the application.

If you are gullible enough to watch FOX “News,” then you are gullible enough to download this app and work for them for FOX for free– you already are in a way, just by watching. This would be a great app for those of you that like to monitor “ethnic” types when the nation goes to “Code Orange,” or, God forbid, “Code Red!” Make sure you have this app when you’re digging your bomb shelter or spying on your neighbors’ subversive activities.

What makes this worse is that flagging reviews as inappropriate content seems to have no effect. I have flagged reviews of my own products, and those of other developers, and nothing has changed. If Apple wants developers and users alike to take this system seriously, they must address this problem immediately. Yes, it’s tedious and costly to do this review, but with continued neglect this system will end up being like YouTube for software.

If you have doubts that this will happen, take a look at the most helpful review for Band. Users are already learning how to game the system.

Some have suggested that buying the app should be a requirement before leaving a review. I agree, but this will not completely mitigate the need to vet content. A large percentage of applications are free: the trolls will just download before going on their merry way.

If all of this wasn’t depressing enough for developers, I’ll leave you with my biggest disappointment: reviews are a one way street. I’m not one to feed the trolls, but many of the reviews I’m seeing would benefit from a “Just try this…” or “We’re working on that…” type of response. There’s not even a link to our support on the reviews page.

I remain hopeful that someone at Apple will see what’s going on and have the power to fix it. My only advice would be to act quickly: the longer you wait, the harder it will be to clean things up.

Bugging

It’s pretty clear that the App Store is a huge hit. We’re all loving the ability to customize our iPhones and iPod touches with cool new software!

But with any big new release, there are problems that didn’t pop up while beta testing. As iPhone developers, we’re finding ourselves in a position where we can’t help customers who are encountering these issues.

Let’s take a couple of examples with our app Twitterrific. My friend Jeffrey Zeldman reports a problem where the application would crash just after launching. Another developer friend, Alex King, is having a problem with an authentication alert appearing when it shouldn’t. As a conscientious developer, I want to help these people and fix these bugs. The problem is that I have no tools to do so.

Jeffrey’s problem appears to be something with data that is stored on the phone. Alex’s problem is likely to be an issue with how data is being loaded from the network (from Twitter.) Note that I say “appears” and “likely”; I don’t know for sure, and that’s what is bugging me.

The first problem is knowing where the crash is occurring. The iPhone generates a crash report that automatically gets synced with iTunes. The crash report is stored in ~/Library/Logs/CrashReporter/MobileDevice. Unfortunately, this crash report is “raw” and developers don’t have tools to make it easy to understand (e.g. “symbolicating” crash reports only happens when they are loaded through Xcode’s device organizer.)

And even if I could interpret these crash reports, I’d be faced with another problem. There’s no way to gather additional information about what’s happening on Jeffrey’s and Alex’s device. With a desktop application that’s acting up, developers will often add logging and other kinds of output that track what is occurring around the time of a crash or other misbehavior.

In Jeffrey’s case, I would want to dump out the internal database that’s in use at the time of the crash. For Alex’s bug, I’d want to track the network requests to Twitter and the corresponding response. It’s easy to add this logging to Twitterrific, but the only way to retrieve the results is if you are a registered developer. There is no way I’m going to ask Jeffrey and Alex to spend $99 and install Xcode just so I can collect some debug output.

Assuming that I could get some debugging output, the next step in resolving these problems would be to create a special build with a proposed fix. After having Jeffrey and Alex verify the fix, I’d distribute that same build to a larger group of testers so that we could test for regressions (e.g. we don’t break some functionality in the process of fixing something else.)

The big problem here is that the only way to install software on an iPhone or iPod touch is with the App Store. There are also no provisions for beta testing. Without the ability to sign code, there is no way for a user to get code onto a device: most users fall into this category.

The only way to “test” a fix is to release the changes to tens of thousands of users. It’s the developer equivalent of playing Russian roulette.

(Note: there may be workarounds to some or all of these problems, but with the NDA in place, it’s difficult for developers to share their experiences and solutions.)

Apple has done an fantastic job with the tools that let us develop iPhone software. That’s clearly evident from the fantastic work we’ve seen displayed in the App Store. Unfortunately, the tools that developers use to analyze and debug problems are sorely missing at this point in time.

It’s our hope that this essay will do two things:

  • As a user, please be extra patient when developers tell you that they are working on a problem—it’s hard work at the moment and the time it takes to resolve an issue will be longer than with a desktop application.
  • We hope that bringing these shortcomings to Apple’s attention will help them address the issues and improve the iPhone SDK.

Until then, these problems will be bugging us all.

Updated July 15th, 2008: If you’re a developer, please feel free to submit a duplicate (“me too”) bug on the following Radars:

Updated July 16th, 2008: After fixing bugs for customers, Brent Simmons notes that there are frustrations with the final part of the development cycle: making public releases.

Updated July 23rd, 2008: Brandon Sneed has discovered some techniques for doing remote debugging of iPhone applications.

Updated August 6th, 2008: We don’t have to play Russian roulette anymore.

Updated August 8th, 2008: I discovered how to “symbolicate” crash logs.

Updated November 10th, 2008: Getting preferences and data from customers who are having problems just got a lot easier.