Benchmarking in your pants

Just how fast is the iPhone?

Let’s run some benchmarks comparing the iPhone to my iMac running Safari 3 on a 1.83 Ghz Intel Core Duo processor:

Test iMac iPhone Slower by
100,000 iterations 0.041 secs. 3.209 secs. 78x
10,000 divisions 0.005 0.413 82x
10,000 sin(x) calls 0.009 0.709 79x
10,000 string allocations 0.010 0.777 78x
10,000 function calls 0.010 0.904 90x

This means that Javascript on the iPhone will take about 80 times longer to run than it does on the desktop. Keep that in mind while you’re writing your Web 2.0 applications.

If you need any more proof of how much slower Javascript is on the phone, take a look at the plotting application I did for C4[1]. Make sure to run it on the phone and the desktop: the performance difference is palpable.

But our benchmarking story doesn’t end there. What kind of performance would we get from a native iPhone application running these same kind of tests?

Thanks to the hard work of many developers congregating at the iPhone Dev Wiki, there is now a toolchain that allows us to create applications that don’t rely on sweet technologies like Javascript.

(Note: use Google if you’re interested in these tools; I’m honoring the request to not link directly.)

That, combined with the talents of Lucas Newman (author of Lights Off and other goodness at Delicious Monster,) resulted in a series of tests written in Objective-C. Here’s the source code: Marbles.zip

Let’s see how Javascript compares to native code running on the iPhone:

Test Native Javascript Slower by
100,000 iterations 0.015 secs. 3.209 secs. 214x
10,000 divisions 0.004 0.413 103x
10,000 sin(x) calls 0.105 0.709 7x
10,000 string allocations 0.085 0.777 9x
10,000 function calls 0.004 0.904 226x

In looking at these numbers, it’s easy to see that there are huge performance wins with native applications: lower function call overhead along with faster iteration and calculation. Even transcendental functions and object allocation sees a 700-900% speed increase.

More reasons for developers to crave a real iPhone SDK.

One thing that’s interesting to note: in the process of writing the Marbles test, Lucas noticed that _objc_msgSend_fpret was an undefined symbol in the ARM runtime libraries (triggered when calling NSDate’s timeIntervalSinceDate: method.) I looked around the libraries on the iPhone and couldn’t find any definitions, nor could a workaround with NSDecimal or NSDecimalNumber be found. Maybe this is one reason why the iPhone SDK isn’t available yet: important stuff is missing.

My guess is that developers just need to be patient. Apple is smart enough to realize that there are major advantages to having third party applications running natively on the iPhone. Not only will these applications be easier to write and faster to run, but they will also add to an ecosystem that makes the iPhone even more attractive to consumers. Apple also realizes that releasing an SDK means having interfaces that don’t change, code that works correctly, and documentation that explains its use.

If you’re a Cocoa developer and haven’t started looking at UIKit, now would be a good time. You have what you need to start experimenting. And you’ll be ready whenever Apple is.