Category: Miscellaneous

Grass Mud Horse

In my first post about the attack from the Great Firewall of China, I stuck to the facts. There was a simple reason for this: you don’t want conjecture when your site is down. You want to understand the problem and see suggestions on how to fix it.

This post will be different: these are my opinions and they are pointed. I’ll first note some of the reactions I received, then examine some of the technical subtleties, and conclude with speculation on the motives behind the attack.

By time you finish, you’ll also understand the odd title for this post.

Reactions from China

Luckily, the server that provided the page you’re reading now was not attacked. I have purposefully not blocked any Chinese IP addresses at furbo.org. I wanted people there to see what their government is doing from a western perspective.

It’s hard to say how many people in China saw the post, but I do know that some have (at one point I saw twelve visitors from Beijing.) What I found most interesting was that every single person who contacted me used the same word to express their thoughts: “shame.”

Clearly, this attack was not an intentional act by the people of China. No one approves of what their government is doing. I can empathize with this shame: I’d feel the same way if a malicious third party made my browsing ruin the day of a random site owner. Grass Mud Horse!

Me Too!

What happened on my server was not an isolated incident. I have seen many other developers saying “me too!” in the past few days.

I suspected this was the case, but wondered why there wasn’t more discussion about what was happening. I’m guessing that when you’re fighting a fire, there’s little time to discuss the intricacies of how the fricken’ flamethrower is melting the fricken’ network interface.

Some of the discussion came from people who know a lot more about running servers than I do. The most telling was from John Adams:

Look at his bio: he was one of the engineers that designed Twitter’s infrastructure. If a professional like John is saying “WTF?”, amateurs like me are pretty much screwed. Grass Mud Horse!

Another notable post was from Jamie Zawinski, one of the first people to write a web browser. I was hopeful that his clever response to the Chinese BitTorrent traffic would eventually make it go away.

BitTorrent

Unfortunately, it appears there is nothing you can do to quiet the BitTorrent clients. I don’t know enough about this technology to offer any guidance, but someone sure as hell needs to look at the problem and deploy fixes across millions of machines in China. Unless, of course, the Chinese government decides to block BitTorrent client downloads.

The TorrentFreak website has a great overview of BitTorrent’s role in these attacks.

And now would be a good time to pray to your favorite deity that your IP address doesn’t show up here. Note that test is just against our friend “thepiratebay.org”. Your server’s IP address could show up for any other popular site on the web.

That, folks, is DNS poisoning in action. Grass Mud Horse!

False Sense of Security

Even with packet filtering in place, I still feel vulnerable. Why?

I’m not sure the blocks will withstand another 52 Mbps flood. Remember that up to 65,535 filter rules can be matched by code in the kernel. Your ability to block packets is only as good as the CPU that’s running that code. When I hear that dedicated Cisco firewall hardware is failing, it give me no confidence that my box with 6,500 rules getting 13,000 packets per second will be able to keep up. A back of the envelope calculation shows 84.5 million comparisons per second is needed (or one every 11 nanoseconds.)

For this same reason, don’t assume that any routers or load balancing schemes upstream from your server will be able to keep up with China. There’s no guarantee that your hosting provider will be able to protect your servers or VM instances at rates like we experienced last week.

Still don’t believe me? Look at the first comment on this post at the Internet Storm Center:

I had the same problem starting last Friday, the 2nd. Took out a full load balanced cluster of servers.

Grass Mud Horse!

Why Us?

The biggest unanswered question is why did this happen to the Iconfactory? (Apologies to visitors from China: you won’t be able to look at that link. Grass Mud Horse!)

Our only connection to China is that one of the partners, Talos Tsui, was born and raised in Hong Kong (during the years it was a Crown colony.) It seems unlikely that we’ve done anything to piss off the Chinese government. At least until just now…

The traffic spikes earlier in the week lead me to think that we were being randomly tested for our ability to handle a large volume of traffic. We have fat pipes without automatic DDoS protection. The duration and volume of the probes could determine both of those attributes.

I think James Moore nailed it in his tweet. (And he’s acutely aware of the implications of that analysis: we share a server cabinet with our friends at Panic.)

Government Behavior

The Chinese government is not only being deceitful with IP addresses, they’ve also begun cracking down on a mechanism that lets its citizens avoid the bullshit: VPN. Grass Mud Horse!

This action, combined with the DDoS floods, is beneficial to a government that’s intent on isolating its citizens from the free and open Internet. They make it hard to get a packet out of China, but even if you succeed, it’s likely to be blocked by a server that’s been victim of their DDoS.

On the surface, this seems like a good strategy for creating your own private Internet: a network where no packets can enter the west or leave the east.

There is Hope

The Internet was designed to route around damage. While the ability to withstand a nuclear war is a myth, the protocols we use every day were created to be robust against infrastructure loss. Even when that section of the network is the size of China.

But even more important than the technology is the people who use the Internet.

The GreatFire.org website monitors the Great Firewall and provides information in both English and Chinese. An informed populace is a powerful one.

There are also efforts underway to redirect bogus traffic to mirror sites. Geeks have never had a problem staying one step ahead of those who attempt to control.

From a personal perspective, the DDoS attack from China made me acutely aware of how screwed up things are over there. The government’s actions have pissed me off and I’ll now do anything in my power to thwart their efforts. Like writing this piece.

And given the feedback I’ve received, I’m not alone with this point-of-view. People are fighting back. I’m hopeful that over the course of several years, we’ll find better ways to cope with the idiocy of the Chinese government than to tunnel under their firewall and block their IP addresses.

If you doubt people’s ingenuity in routing around roadblocks, take a moment to learn about the Grass Mud Horse:

(The whole video is informative, but be sure to watch the end.)

Written on January 28th, 2015 for Business, Miscellaneous, Opinion

Twitter Nostalgia

December 1st, 2006. Something important in my life began rather inauspiciously:

My first tweet.

Little did I realize that these tweets would become a log of important events in my life. And now thanks to Twitter’s new search capabilities, I can remember that past. Please indulge me as I sift through these moments and get nostalgic…

It turns out I was the sixth person to mention “iPhone” on Twitter. My colleague Corey beat me by a few hours and the guy who started Twitter was first. There must have been something in the water at the Iconfactory water that day. I wish all my predictions on Twitter were so prescient!

(Thanks to Dan Frommer for doing the legwork on this one.)

Interestingly, the very next tweet in my timeline was the start of the world’s first Twitter client:

These two tweets, separated by only a few hours, are an amazing summary of what was about to happen.

But first, another important event transpired: I started writing publicly. Twitter was clearly an inspiration here: I loved those 140 characters, but found that I needed another venue to expand upon my thoughts:

Note the date on that last tweet: the day before the original iPhone went on sale. My first post stated that I didn’t know where there I was going with the blog. A few days later, I had a pretty good idea:

I had just bought an iPhone.

And remember that “video iPhone nano gaming system”? Here I am being the first person to display a Twitter timeline on it:

Worlds were colliding: Twitter, iPhone, and a place to talk about both.

Twitter was always an outlet for my strange sense of humor. Depending on your point-of-view, April Fool’s in 2008 was either the best or worst day ever:

It’s now commonly known as the CHOCK LOCK, but it took almost five months for someone to christen it:

And amazingly, just six minutes later:

Both Seth and Michael were spurred on by Dan Wood, so I guess we can blame him!

The iPhone SDK was released in February 2008 and a lot of that early hacking I did on the iPhone was finally turning into a real product. It’s likely that this affection with capital letters was triggered by a shitload of coding.

But all that hard work paid off:

I tweeted that just after being handed an Apple Design Award. Those colliding worlds were good to me.

I’m a firm believer of looking forward in your work, but there’s also value in remembering how you got to where you are today.

And speaking of today, guess when the bulk of this post was written?

Some things never change.

Written on November 26th, 2014 for Miscellaneous

Un-fucking-believable

An anonymous individual is spreading accusations that I’m a patent troll. Hard to believe, huh?

It makes me furious that I have to respond to these claims without knowing my accuser, but here goes:

Yes, we have a patent. Yes, we got a license fee for it. No, it wasn’t unreasonable.

We originally got the patent as protection against a large company, like Adobe, using our innovation. Patents are like trademarks, you have to protect them after you get them (by collecting licensing fees, even if it’s a small one.) The arrangement with Ricci Adams was amicable from the first email to the last.

From a monetary point-of-view, all I’m going to tell you is that the fee we collected from Ricci Adams was less than the amount of time and money spent getting the legal documents in place. We didn’t profit from it, nor did it “stifle competition.” (Pixel Winch is a great product, you should check out the beta. And no, I’m not making any money by saying that.)

We also acquired this patent before our encounter with Lodsys. Our view of patents has obviously changed since then. I can’t go into any specific details there, thanks to the lawyers.

And you want to know the real kicker? After talking about this stuff with my friend Marco Arment, we’re no longer sure that software patents have the same “use it or lose it” conditions as with a trademark. Of course, for a lawyer who’s collecting fees, mentioning this is not in their best interest.

Any further comment on this situation will have to wait until the coward who’s making these claims comes forward.

P.S. Marco, I could use that beer now.

Written on June 5th, 2014 for Business, Miscellaneous

At

I recently appeared with John Gruber on The Talk Show. During the episode, the following exchange took place:

When it comes to naming characters, the Unicode standard is the bible. And code point U+0040 is named as “COMMERCIAL AT”.

So yeah, we’re “right.”

But then Twitter got ahold of this exchange and I quickly realized something important: we don’t all speak English:

It turns out “arroba” has a very interesting history that originated in Spanish commerce:

“Whatever the origin of the @ symbol, the history of its usage is more well-known: it has long been used in Spanish and Portuguese as an abbreviation of arroba, a unit of weight equivalent to 25 pounds, and derived from the Arabic expression of “a quarter” (الربع pronounced ar-rubʿ)”

As someone who loves iconography, it’s pretty amazing to see @ as a handwritten symbol in 1148:

I also realized that I knew the Italian word for the @ symbol: “chiocciola”. It’s one of the names for a snail (the other being “lumaca” which is commonly used when ordering them in a restaurant.)

And why is this name used?

(It’s fun to say, too. Something like “key-o-cho-la” but with more exotic hand gestures.)

This tweet led to many responses that show how varied the pronunciations are in different languages.

  • Dutch: “apenstaartje” = “monkey tail”
  • Hebrew: “strudel” = shape of the cake
  • Danish/Swedish: “snabel-a” = “with an (elephant) trunk”
  • German: “Klammeraffe” = “spider monkey”
  • Poland: “małpa” = “monkey”
  • Korean: “골뱅이” (gol-baeng-ee) = “a type of sea snail”

Wikipedia has a full list of how @ is used in other languages.

But do you notice the pattern with these pronunciations?

They’re being used as pictograms:

“A pictogram…, is an ideogram that conveys its meaning through its pictorial resemblance to a physical object. Pictographs are often used in writing and graphic systems in which the characters are to a considerable extent pictorial in appearance.”

While pictograms are fairly common in Asian languages, it’s rare to see this kind of usage in the West. Written Kanji characters, such as 木 for “tree”, have been in use since the first century AD. Indeed, these kinds pictures were man’s first form of expression and communication.

But in these writing systems, someone saw a thing with a trunk and leaves growing from the ground and put it on a piece of paper as an 木 symbol. What we’ve seen happen with the @ symbol is the opposite. Many different cultures have seen our “COMMERCIAL AT” symbol and given it a name based on its appearance.

So even though John and I are right about the pronunciation, this is certainly a case where English pales when compared with other languages. I envy my colleagues that get to play with snails and monkeys while coding in Objective-C!

Written on March 25th, 2014 for Miscellaneous, Observation

Mac App Store Receipts and Mavericks

The storeagent and I aren’t getting along too well these days.

We’re in the process of getting a new release of xScope ready for release. As a developer tool, we’ve been compatible with Mavericks for several months now, but there are some minor bug fixes that we’d like to get out before the new version of OS X ships.

As you might be aware, this is the first time I’ve done a build on Mavericks itself. Things haven’t exactly been smooth sailing.

Today’s revelation is how storeagent creates the /Contents/_MASReceipt/receipt file in Mavericks. It’s subtly different, and will confuse the heck out of you until you understand what’s going on.

For the past few days, I’ve been testing a beta release of the .pkg using the standard command:

sudo installer -store -pkg /tmp/xScope.pkg -target /

This version had a CFBundleShortVersionString of “3.6.2b1”. The installer and receipt checking code was working great.

Until I did the final build and used the version string “3.6.2”. I got this message after I double-clicked the app and entered my Test User Apple ID:

After checking the code signing, bundle IDs and all other parts of the app, I finally fired up the debugger and discovered that the receipt validation code was failing when checking receipt attribute type 3, the Application version field (in Table 1-1).

After decrypting and checking the receipt payload, the value was “3.6.2b1” not the version I just installed. Where did this old version number come from? Why did following the advice in the dialog and deleting the app not fix the problem. How come this old receipt kept showing up no matter what I did?

Receipts from older versions had never been a problem in previous versions of OS X, so there must be some new behavior in Mavericks. And it took me almost a whole day to figure out that new behavior.

It turns out that storeagent is doing some kind of in-memory cache of receipts that have been downloaded from iTunes. Since a network connection is needed to retrieve the receipt, keeping it around would prevent a little bit of network traffic. In previous versions, the receipt was presumably recreated each time it was requested, so you always had a fresh copy.

The workaround is fairly simple. It even gives me a bit of pleasure at this point:

$ killall -KILL storeagent

You’ll need to delete the app at this point and re-install it using:

$ sudo installer -store -pkg YourApp.pkg -target /

When you relaunch your app, you’ll see the Apple ID login dialog. Since storeagent is launched on demand by launchd, a new process will be started at this point. After entering your Test User credentials, a new, and valid, receipt will be written into the _MASReceipt folder.

One could imagine this caching of receipt data being a problem with apps that are downloaded from the App Store. If someone never reboots between two versions of the same app being “Ready for Sale”, it may trigger the same problem. I have no way to test this hypothesis.

For any Apple folks that might be reading, here you go: rdar://problem/15283740

Written on October 21st, 2013 for Development, Miscellaneous
Newer
Older