For hire

We’ve been making award-winning apps since the App Store opened. Now we’re doing it for clients.

Grass Mud Horse

In my first post about the attack from the Great Firewall of China, I stuck to the facts. There was a simple reason for this: you don’t want conjecture when your site is down. You want to understand the problem and see suggestions on how to fix it.

This post will be different: these are my opinions and they are pointed. I’ll first note some of the reactions I received, then examine some of the technical subtleties, and conclude with speculation on the motives behind the attack.

By time you finish, you’ll also understand the odd title for this post.

Reactions from China

Luckily, the server that provided the page you’re reading now was not attacked. I have purposefully not blocked any Chinese IP addresses at furbo.org. I wanted people there to see what their government is doing from a western perspective.

It’s hard to say how many people in China saw the post, but I do know that some have (at one point I saw twelve visitors from Beijing.) What I found most interesting was that every single person who contacted me used the same word to express their thoughts: “shame.”

Clearly, this attack was not an intentional act by the people of China. No one approves of what their government is doing. I can empathize with this shame: I’d feel the same way if a malicious third party made my browsing ruin the day of a random site owner. Grass Mud Horse!

Me Too!

What happened on my server was not an isolated incident. I have seen many other developers saying “me too!” in the past few days.

I suspected this was the case, but wondered why there wasn’t more discussion about what was happening. I’m guessing that when you’re fighting a fire, there’s little time to discuss the intricacies of how the fricken’ flamethrower is melting the fricken’ network interface.

Some of the discussion came from people who know a lot more about running servers than I do. The most telling was from John Adams:

Look at his bio: he was one of the engineers that designed Twitter’s infrastructure. If a professional like John is saying “WTF?”, amateurs like me are pretty much screwed. Grass Mud Horse!

Another notable post was from Jamie Zawinski, one of the first people to write a web browser. I was hopeful that his clever response to the Chinese BitTorrent traffic would eventually make it go away.

BitTorrent

Unfortunately, it appears there is nothing you can do to quiet the BitTorrent clients. I don’t know enough about this technology to offer any guidance, but someone sure as hell needs to look at the problem and deploy fixes across millions of machines in China. Unless, of course, the Chinese government decides to block BitTorrent client downloads.

The TorrentFreak website has a great overview of BitTorrent’s role in these attacks.

And now would be a good time to pray to your favorite deity that your IP address doesn’t show up here. Note that test is just against our friend “thepiratebay.org”. Your server’s IP address could show up for any other popular site on the web.

That, folks, is DNS poisoning in action. Grass Mud Horse!

False Sense of Security

Even with packet filtering in place, I still feel vulnerable. Why?

I’m not sure the blocks will withstand another 52 Mbps flood. Remember that up to 65,535 filter rules can be matched by code in the kernel. Your ability to block packets is only as good as the CPU that’s running that code. When I hear that dedicated Cisco firewall hardware is failing, it give me no confidence that my box with 6,500 rules getting 13,000 packets per second will be able to keep up. A back of the envelope calculation shows 84.5 million comparisons per second is needed (or one every 11 nanoseconds.)

For this same reason, don’t assume that any routers or load balancing schemes upstream from your server will be able to keep up with China. There’s no guarantee that your hosting provider will be able to protect your servers or VM instances at rates like we experienced last week.

Still don’t believe me? Look at the first comment on this post at the Internet Storm Center:

I had the same problem starting last Friday, the 2nd. Took out a full load balanced cluster of servers.

Grass Mud Horse!

Why Us?

The biggest unanswered question is why did this happen to the Iconfactory? (Apologies to visitors from China: you won’t be able to look at that link. Grass Mud Horse!)

Our only connection to China is that one of the partners, Talos Tsui, was born and raised in Hong Kong (during the years it was a Crown colony.) It seems unlikely that we’ve done anything to piss off the Chinese government. At least until just now…

The traffic spikes earlier in the week lead me to think that we were being randomly tested for our ability to handle a large volume of traffic. We have fat pipes without automatic DDoS protection. The duration and volume of the probes could determine both of those attributes.

I think James Moore nailed it in his tweet. (And he’s acutely aware of the implications of that analysis: we share a server cabinet with our friends at Panic.)

Government Behavior

The Chinese government is not only being deceitful with IP addresses, they’ve also begun cracking down on a mechanism that lets its citizens avoid the bullshit: VPN. Grass Mud Horse!

This action, combined with the DDoS floods, is beneficial to a government that’s intent on isolating its citizens from the free and open Internet. They make it hard to get a packet out of China, but even if you succeed, it’s likely to be blocked by a server that’s been victim of their DDoS.

On the surface, this seems like a good strategy for creating your own private Internet: a network where no packets can enter the west or leave the east.

There is Hope

The Internet was designed to route around damage. While the ability to withstand a nuclear war is a myth, the protocols we use every day were created to be robust against infrastructure loss. Even when that section of the network is the size of China.

But even more important than the technology is the people who use the Internet.

The GreatFire.org website monitors the Great Firewall and provides information in both English and Chinese. An informed populace is a powerful one.

There are also efforts underway to redirect bogus traffic to mirror sites. Geeks have never had a problem staying one step ahead of those who attempt to control.

From a personal perspective, the DDoS attack from China made me acutely aware of how screwed up things are over there. The government’s actions have pissed me off and I’ll now do anything in my power to thwart their efforts. Like writing this piece.

And given the feedback I’ve received, I’m not alone with this point-of-view. People are fighting back. I’m hopeful that over the course of several years, we’ll find better ways to cope with the idiocy of the Chinese government than to tunnel under their firewall and block their IP addresses.

If you doubt people’s ingenuity in routing around roadblocks, take a moment to learn about the Grass Mud Horse:

(The whole video is informative, but be sure to watch the end.)

Fear China

I’ve been using the Internet in one form or another since the mid-80’s. In that time, I’ve seen a lot of strange stuff happening on our global network. On Tuesday, I experienced something extraordinary.

It all started with a text message from my partner Ged at 8:30 AM:

Mail server down. Please take a look when you can, thx.

I verified that the mail server was down from the west coast as well as the east coast, then started poking around to see what was wrong. When I looked at the server traffic, there was only one thing I could say:

daily

“Holy shit.”

Unless you’re a network engineer, that graph won’t mean much. The data shown is the amount of bandwidth into the Iconfactory’s main server. The blue line is the number of megabits per second for requests and the green area is the amount for responses to those requests. Normally, the blue line is much smaller than the green area: a small HTTP request returns larger HTML, CSS and images.

The number of requests peaked out at 52 Mbps. Let’s put that number in perspective: Daring Fireball is notorious for taking down sites by sending them about 500 Kbps of traffic. What we had just experienced was roughly the equivalent of 100 fireballs.

If each of those requests were 500 bytes, that’s 13,000 requests per second. That’s about a third of Google’s global search traffic. Look at how much careful planning went into handling Kim Kardashian’s butt at 8,000 requests per second.

All of this traffic directed at one IP address backed by a single server with a four core CPU.

Like I said, “Holy shit.”

Regaining Control

The first course of business was to regain control of the server. Every service on the machine was unresponsive, including SSH. The only thing to do was perform a remote restart and wait for things to come back online.

As soon as I got a shell prompt, I disabled the web server since that was the most likely source of the traffic. I was right: things quieted down as soon as traffic on port 80 and 443 was rejected. It was 9:30 AM (and you can see it in the graph above.)

The first log I looked at showed a kernel panic at 3:03 AM in zalloc. This was right at the time of the biggest spike. The system.log showed similar problems: the high level of traffic was causing all kinds of memory issues caused by too many processes.

As a test, I turned the server back on for a minute and immediately maxed out Apache’s MaxClients. Our server simply isn’t capable of handling thousands of Apache child processes (it normally runs with less than a hundred.)

So where the hell was all this traffic coming from?

Triage

Since I knew the traffic was from the web, it was likely that Apache’s logs would tell me something. Given that our Apache logs are usually in the 10 MB range, the current 100 MB log file surely contained a lot of useful information.

The first thing I noticed was a lot of requests being returned with a 403 status code. The paths for those requests also made no sense at all: one of the most common began with “/announce”. But there was also a lot of requests that looked like they were intended for CDNs, YouTube, Facebook, Twitter and other places that were not the Iconfactory.

As a test, I updated Apache’s CustomLog configuration with %{Host}i so it would show me the host headers being sent with the requests. I then turned the web server back on for 30 seconds and collected data. Indeed, the traffic we were seeing on our server was destined for someplace else.

The CHOCK was pretty proud to be serving traffic for cdn.gayhotlove.com, but I sure wasn’t.

Clearly there was some kind problem with traffic being routed to the wrong place. The most likely candidate would, of course, be DNS. While looking at IP addresses in my logs, I noticed something interesting: all of this traffic was coming from China.

The pieces were starting to fall into place. I understood the problem:

(Note: The “GFW” in responses refers to China’s Great Firewall. I’m pretty good with acronyms, but this was a new one to me!)

Now all I had to do was find a way to deal with the traffic.

Apache Configuration

My first thought was to deal with the traffic was by handling the HTTP traffic more efficiently.

We host several sites on our server and use VirtualHost to route traffic on a single IP address to multiple websites. Virtual hosts rely on the “Host:” header in the HTTP request to determine where the traffic should head, and as we’ve seen above, the host information was totally bogus.

One thing I learned is that Apache can have problems figuring out which virtual host to use in some cases:

If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address.

Remember that millions of requests had a host name that would need to be looked up. After consulting the documentation, I setup a virtual host that would quickly return a 404 error for the request and display a special message at the root directory. Here’s what it looks like:

<VirtualHost _default_:80>
    ServerName default
    DocumentRoot "/Web/Sites/default"
    <Directory "/Web/Sites/default">
        Options None
        AllowOverride None
        DAV Off
    </Directory>
    LogLevel warn
</VirtualHost>

If you run a server, take a second right now to make sure that it’s doing the right thing when presented with a bad header:

$ curl -H "Host: facebook.com" http://199.192.241.217

All of this helped deal with the traffic, but it only slowed down the amount of time it took Apache to max out the child processes. A Twitter follower in China also reminded me that their day was just beginning and traffic would be picking up. At 8 PM, the trend for traffic didn’t look good, so I turned off the web services and had a very stiff drink.

Then something strange happened at 11:30 PM: the inbound requests started to die off. Someone in China had flipped a switch.

I was tempted to bring the web server back up, but experience told me to leave things as they were. Michter’s and bash don’t make a good pair.

This problem would have to wait for another day.

Hello BitTorrent

The next morning, I tried bringing up the web server. Things ran fine for awhile, but after 10 minutes or so, Apache processes started climbing again.

Most of the traffic was to the BitTorrent /announce URL. BitTorrent clients in China still thought my server was a tracker and were noticing that port 80 was alive again.

And it’s not like there are just a couple of people using BitTorrent in China.

The direct traffic from DNS may have gone away, but secondary traffic from cached information was still killing us. At this point, the only recourse was to block IP addresses.

Blocking China

I’m a big believer in the power of an open and freely accessible Internet: I don’t take blocking traffic from innocent people lightly. But in this case, it’s the only thing that worked. If you get a DDOS like what I’ve described above, this should be the first thing you do.

The first step is to get a list of all the IP address blocks in the country. At present that’s 5,244 separate zones. You’ll then need to feed them to your firewall.

In our case, we use ipfw. So I wrote a script to create a list of rules from the cn.zone file:

#!/bin/sh

# cn.zone comes from http://www.ipdeny.com/ipblocks/
#
# build the rules with:
#
# $ build_rules > /tmp/china_rules
#
# apply rules with:
#
# $ sudo ipfw /tmp/china_rules 

r=1100
while read line; do
	echo "add $r deny ip from " $line " to any in";
	r=$(( $r + 1 ))
done < cn.zone

You’ll want to adjust the starting rule number (1100 above) to one that’s before the allow on port 80.

After setting these new rules, traffic on our server immediately returned to normal.

Digging Deeper

Now that I had my server back, I could take some time to look at logs more closely and see if anyone else had seen similar issues.

First Hits

BitTorrent /announce traffic turned up a few clues. I had noticed a few 5 Mbps spikes in our request traffic late on Thursday, January 15th and on the following Saturday:

weekly

Initially, I just chalked it up to random bullshit traffic on the Internet, much like the packets from Romania looking for phpMyAdmin. In retrospect, that was dumb.

If you look at the origins of those first packets, you’ll see that it’s not a regional problem: the IP addresses are physically located all the way from densely populated Hong Kong to the remoteness of Xinjiang province (north of Tibet.)

Was this traffic a probe or an unintentional screwup? I don’t know.

(Note: I have archived all of the logs mentioned above. If you have legitimate reason to analyze these logs, please get in touch.)

We’re Not Alone

More concerning, is that other site owners are seeing similar behavior starting in early January. I took some comfort in knowing that we weren’t alone on the 20th.

But at the end of the day, every machine in China has the potential be a part of a massive DDOS attack on innocent sites. As my colleague Sean quipped, “They have weaponized their entire population.”

Conclusion

Will this happen again? For everyone’s sake, I hope not. The people of China will only end up being banned from more websites and site owners will waste many hours in total panic.

But if it does happen, I hope this document helps you deal with China’s formidable firehose.

Other Resources

If you’re using ngnix instead of Apache, here are some instructions for blocking BitTorrent requests from China.

For those of you using iptables on Linux, here’s a tutorial for blocking IPs on that platform. It’s also interesting to note that Matt’s site is running on Linode: don’t assume that big providers will offer any protection upstream.

This thread has a good discussion with other site owners experiencing the BitTorrent traffic.

Another option to consider is moving the server’s IP address. You’ll have to deal with the normal DNS propagation and reconfigure reverse DNS (especially if you’re running a mail server on the box), but this may be quick and effective way to avoid the firehose.

Updated January 24th, 2015: Added the section above with additional resources for those of you who are experiencing the problem. Good luck!

Updated January 28th, 2015: I’ve written some opinions on the incident described above.

A documentary

For the past few months, I’ve been helping a couple of talented filmmakers with a new project. I’ve been making introductions, reminding busy people to find some time for an interview, and that kind of thing. Today, the result was made public and it’s better than I ever imagined.

I think this project that looks at our past is important for two reasons: our present and future.

It’s no secret that the art and business of software development has changed radically since that day in January 2007. Listening to my colleagues talk about what that moment meant to them is incredibly helpful for putting today’s situation in perspective. You don’t give a shit how big the next iPhone’s screen is going to be when your peers are talking about how these devices have changed their lives.

By funding this project, you’re also giving your future self a rare gift. I was a young developer when the Mac was announced in 1984, and let me assure you that many of the details of that time have been lost over the course of thirty years. You’re going to look back at this time in your career fondly and wish you remembered more about it.

Something like what Andy Hertzfeld has done at Folklore.org has been a godsend to those of us who were around in the early days of the Mac. But I don’t think that approach would work now: the Internet made the iPhone a world-wide phenomenon, not a project limited to the Texaco Towers. (Did you know that the first person to make money from selling apps was a developer in New Zealand?)

This is your chance to create a record of something that has changed so many lives. Please take a moment to look at what Jake and Jed have done and join me in contributing to this Kickstarter.

Un-fucking-believable

An anonymous individual is spreading accusations that I’m a patent troll. Hard to believe, huh?

It makes me furious that I have to respond to these claims without knowing my accuser, but here goes:

Yes, we have a patent. Yes, we got a license fee for it. No, it wasn’t unreasonable.

We originally got the patent as protection against a large company, like Adobe, using our innovation. Patents are like trademarks, you have to protect them after you get them (by collecting licensing fees, even if it’s a small one.) The arrangement with Ricci Adams was amicable from the first email to the last.

From a monetary point-of-view, all I’m going to tell you is that the fee we collected from Ricci Adams was less than the amount of time and money spent getting the legal documents in place. We didn’t profit from it, nor did it “stifle competition.” (Pixel Winch is a great product, you should check out the beta. And no, I’m not making any money by saying that.)

We also acquired this patent before our encounter with Lodsys. Our view of patents has obviously changed since then. I can’t go into any specific details there, thanks to the lawyers.

And you want to know the real kicker? After talking about this stuff with my friend Marco Arment, we’re no longer sure that software patents have the same “use it or lose it” conditions as with a trademark. Of course, for a lawyer who’s collecting fees, mentioning this is not in their best interest.

Any further comment on this situation will have to wait until the coward who’s making these claims comes forward.

P.S. Marco, I could use that beer now.