“Must Fix for Next Release”

In the current version of xScope, there is a memory leak caused by a change in OS X 10.10.2. While the Loupe is in the background grabbing the screen, something in the frameworks is leaving images in the autorelease pool. The fix is literally two lines of code that forces the pool to empty.

But that’s not why I’m writing now.

This fix was submitted two weeks ago on February 2nd. A week later it went into review and was quickly rejected. The problem was that a buy button was accessible from our Help window.

The bulk of the help is static and built into the app, but there is a part online that we can update easily. This makes it really easy easy for us to add tips and other useful information for our customers. But since it’s just a web browser, it’s possible to wander into a part of our site that shows a header with mentions the word buy which is not allowed per rule 7.15. (Yes, the buy button is for something the customer has already purchased and is actively in the process of using, but technically it’s still a violation.)

My issue is the way that we must fix these problems. In this particular case, the issue was resolved by editing some HTML on our server, not by changing anything in the app itself. But we still must submit a “new” binary and go through the lengthy review process again. This is a huge waste of time for both developers and app reviewers (who are clearly lagging behind these days.)

I think there’s an easy way to fix these minor transgressions that would benefit both parties: add a new kind of approval with strings attached. A “Must Fix for Next Release” state where the app can go into “Ready for Sale” but the issue remains in the Resolution Center. At that point, both the app reviewers and developer know that an issue has to be dealt with before it’s approved the next time.

It would be like getting pulled over for a broken taillight on your car. You don’t need to visit your mechanic immediately to get the problem fixed. But you’ll certainly have to get things in order the next time you register the vehicle.

Please be sure to dupe Radar #19921616 if you agree that this would be a good change for iTunes Connect.

Quick Thoughts on an Apple Car

  • Shipping a phone from China can done overnight with air freight. Shipping a car from China cannot.
  • Good luck finding an independent mechanic that can break FairPlay encryption.
  • CarPlaysForSure™
  • Like iOS 8.0.1, but for getting to work.
  • Software quality issues have a different meaning at 75 miles per hour.
  • And you thought the Genius Bar was crowded before!
  • Finally: a minivan? That sure fits well with Jony Ive’s design sensibilities.

So yeah, put me in the “no way” column for this.

What can be done?

I had the pleasure of speaking with Brianna Wu earlier this week. During our discussion, we touched on some of the bullshit going on in her life.

I asked a simple question, “How is your husband doing?”

Her reply: “He’s a wreck.”

Now put yourself in his situation: how would you feel if this abuse was happening to your partner?

It’s time to start looking for ways to change our status quo. I know I am.

Grass Mud Horse

In my first post about the attack from the Great Firewall of China, I stuck to the facts. There was a simple reason for this: you don’t want conjecture when your site is down. You want to understand the problem and see suggestions on how to fix it.

This post will be different: these are my opinions and they are pointed. I’ll first note some of the reactions I received, then examine some of the technical subtleties, and conclude with speculation on the motives behind the attack.

By time you finish, you’ll also understand the odd title for this post.

Reactions from China

Luckily, the server that provided the page you’re reading now was not attacked. I have purposefully not blocked any Chinese IP addresses at furbo.org. I wanted people there to see what their government is doing from a western perspective.

It’s hard to say how many people in China saw the post, but I do know that some have (at one point I saw twelve visitors from Beijing.) What I found most interesting was that every single person who contacted me used the same word to express their thoughts: “shame.”

Clearly, this attack was not an intentional act by the people of China. No one approves of what their government is doing. I can empathize with this shame: I’d feel the same way if a malicious third party made my browsing ruin the day of a random site owner. Grass Mud Horse!

Me Too!

What happened on my server was not an isolated incident. I have seen many other developers saying “me too!” in the past few days.

I suspected this was the case, but wondered why there wasn’t more discussion about what was happening. I’m guessing that when you’re fighting a fire, there’s little time to discuss the intricacies of how the fricken’ flamethrower is melting the fricken’ network interface.

Some of the discussion came from people who know a lot more about running servers than I do. The most telling was from John Adams:

Look at his bio: he was one of the engineers that designed Twitter’s infrastructure. If a professional like John is saying “WTF?”, amateurs like me are pretty much screwed. Grass Mud Horse!

Another notable post was from Jamie Zawinski, one of the first people to write a web browser. I was hopeful that his clever response to the Chinese BitTorrent traffic would eventually make it go away.


Unfortunately, it appears there is nothing you can do to quiet the BitTorrent clients. I don’t know enough about this technology to offer any guidance, but someone sure as hell needs to look at the problem and deploy fixes across millions of machines in China. Unless, of course, the Chinese government decides to block BitTorrent client downloads.

The TorrentFreak website has a great overview of BitTorrent’s role in these attacks.

And now would be a good time to pray to your favorite deity that your IP address doesn’t show up here. Note that test is just against our friend “thepiratebay.org”. Your server’s IP address could show up for any other popular site on the web.

That, folks, is DNS poisoning in action. Grass Mud Horse!

False Sense of Security

Even with packet filtering in place, I still feel vulnerable. Why?

I’m not sure the blocks will withstand another 52 Mbps flood. Remember that up to 65,535 filter rules can be matched by code in the kernel. Your ability to block packets is only as good as the CPU that’s running that code. When I hear that dedicated Cisco firewall hardware is failing, it give me no confidence that my box with 6,500 rules getting 13,000 packets per second will be able to keep up. A back of the envelope calculation shows 84.5 million comparisons per second is needed (or one every 11 nanoseconds.)

For this same reason, don’t assume that any routers or load balancing schemes upstream from your server will be able to keep up with China. There’s no guarantee that your hosting provider will be able to protect your servers or VM instances at rates like we experienced last week.

Still don’t believe me? Look at the first comment on this post at the Internet Storm Center:

I had the same problem starting last Friday, the 2nd. Took out a full load balanced cluster of servers.

Grass Mud Horse!

Why Us?

The biggest unanswered question is why did this happen to the Iconfactory? (Apologies to visitors from China: you won’t be able to look at that link. Grass Mud Horse!)

Our only connection to China is that one of the partners, Talos Tsui, was born and raised in Hong Kong (during the years it was a Crown colony.) It seems unlikely that we’ve done anything to piss off the Chinese government. At least until just now…

The traffic spikes earlier in the week lead me to think that we were being randomly tested for our ability to handle a large volume of traffic. We have fat pipes without automatic DDoS protection. The duration and volume of the probes could determine both of those attributes.

I think James Moore nailed it in his tweet. (And he’s acutely aware of the implications of that analysis: we share a server cabinet with our friends at Panic.)

Government Behavior

The Chinese government is not only being deceitful with IP addresses, they’ve also begun cracking down on a mechanism that lets its citizens avoid the bullshit: VPN. Grass Mud Horse!

This action, combined with the DDoS floods, is beneficial to a government that’s intent on isolating its citizens from the free and open Internet. They make it hard to get a packet out of China, but even if you succeed, it’s likely to be blocked by a server that’s been victim of their DDoS.

On the surface, this seems like a good strategy for creating your own private Internet: a network where no packets can enter the west or leave the east.

There is Hope

The Internet was designed to route around damage. While the ability to withstand a nuclear war is a myth, the protocols we use every day were created to be robust against infrastructure loss. Even when that section of the network is the size of China.

But even more important than the technology is the people who use the Internet.

The GreatFire.org website monitors the Great Firewall and provides information in both English and Chinese. An informed populace is a powerful one.

There are also efforts underway to redirect bogus traffic to mirror sites. Geeks have never had a problem staying one step ahead of those who attempt to control.

From a personal perspective, the DDoS attack from China made me acutely aware of how screwed up things are over there. The government’s actions have pissed me off and I’ll now do anything in my power to thwart their efforts. Like writing this piece.

And given the feedback I’ve received, I’m not alone with this point-of-view. People are fighting back. I’m hopeful that over the course of several years, we’ll find better ways to cope with the idiocy of the Chinese government than to tunnel under their firewall and block their IP addresses.

If you doubt people’s ingenuity in routing around roadblocks, take a moment to learn about the Grass Mud Horse:

(The whole video is informative, but be sure to watch the end.)