iCloud Clusterfuck

Luckily, I don’t have to use this kind of title often. But when I do, there’s a good reason: this year’s beta release cycle for all of Apple’s operating systems has been a mess. The months since WWDC in June have been a terrible experience for both customers and developers alike and the literal center of the chaos was Apple’s iCloud syncing service.

For us, it all started with customers reporting lost Linea sketches in their iCloud Drive. Initial investigations led to a common factor: all of the people affected had installed the iOS 13 beta release.

And when I say lost, I mean really lost. Entire folders were either gone or corrupted. Apple’s mechanism to recover deleted files was of no help. The customers with weird folder duplicates were the “lucky” ones.

We couldn’t find any problems in our code and there was no information from Apple about problems with iCloud, so we did the only thing possible: we blocked people from using Linea on iOS 13. If a customer absolutely needed the app, we got them onto TestFlight after a stern warning about the real possibility of losing data.

A few weeks later, Apple finally indicated that there were some issues with iCloud and the beta release. In the same week, they released a public beta and sent out an email to customers encouraging them to try out iOS 13.

We did our best to understand the situation and provide information to Apple, but it felt like we were tossing bug reports into a black hole. The most discouraging part was when we tried to open an incident with Apple Developer Technical Support (DTS). After writing up a detailed report, we were informed that they don’t support beta releases!

By the time beta 6 rolled around, things were improving, but there were still isolated reports that the service might not be completely back in working order. Confirming the fixes on our end was impossible: the folks who’d encountered the previous data loss had no desire to mess with iCloud again.

Now it appears that the entire stack is getting rolled back and there won’t be new iCloud features in iOS 13 (at least initially.) I honestly think that’s the wisest course of action at this point. My only wish is that Apple would make an official statement.

Now that we’re past the worst of it, it’s time to think about why this whole episode happened and how it can be prevented in the future.

Folks don’t understand beta

Apple’s biggest fuck up was a bad assumption about who is testing a beta release. In WWDC presentations and developer documentation, there’s always warnings about using the release on non-production devices and only with test accounts.

But there are many folks that are just looking to get the new and shiny features. In past iOS beta releases, Apple hasn’t suffered too much from this because the early software was relatively stable. Maybe you got some dropped calls or bad battery life, but it was nothing too serious.

These early adopters installed iOS 13 and expected a similar experience. They also weren’t using an iCloud test account, so any instability in the beta release propagated bad data to their other devices.

Developers have long known to unhook external drives when testing a new OS release. Shit happens, and that’s OK because it’s a beta and we expect a bumpy road. In fact, I currently have an external Photos Library that I can’t use in Mojave because it got upgraded when I booted into Catalina: it’s my dumb mistake and I have no complaints.

Anyone who’s not a developer, and hasn’t been burned by a bad OS, does not know the kind of trouble that lies ahead. It’s irresponsible for Apple to release a public beta with known issues in iCloud. It’s doubly egregious to then promote that release with an email campaign to customers. For a company that prides itself in presenting a unified front, it sure looks like the left hand doesn’t know what the right hand is doing.

Disconnect iCloud by default

The solution to this situation is relatively simple, but with painful consequences. Apple needs to help unwitting customers by automatically disconnecting that external hard drive called iCloud.

If a device is using an Apple ID that’s also being used on a non-beta device, then iCloud shouldn’t be allowed. If you install an iOS beta on your iPad, it doesn’t get to use any cloud services because it puts the data on your iPhone or Mac at risk.

Of course, once this restriction is put into place, Apple will quickly realize that no one is testing iCloud or anything that touches it. If a beta tester can’t have their contacts, synced notes, reminders, and appointments over the course of three months, they’re not going to be testing much. I’d also expect to see lots of guides on “How to Get Back on Normal iOS” around the web.

But there’s a better, and even more unlikely solution…

iCloud can’t be a beta

Because it’s a service, iCloud doesn’t get to go into beta. It needs to be reliable all the time, regardless of whether iOS or any other platform is in beta test.

As it is now, Apple is effectively telling you that your storage device will be unreliable for a few months. It’s like having a hard drive where the manufacturer tells you it won’t work well for ¼ of the year. Would you purchase storage with a caveat that “the drive mechanism may not work properly during the hot summer months”?

You don’t see these kinds of issues with Google Drive or Dropbox because they don’t get a pass while an OS is being tested. Dropbox moved your folders from AWS to their own servers without you noticing even the slightest hiccup. Compare this to the last three months of iCloud in beta.

And yes this is a hard task: akin to swapping out the engines of an airplane in mid-flight. But data services, like hard drives, must work all the time.

We’re well past that point with iCloud and it’s just going to get worse as Apple moves more of our data there. This time next year, will I suddenly not know which episodes I’ve watched on Apple TV? Or will I suddenly lose all my scores and achievements in Apple Arcade? Will I shoot myself in the foot with some cool new feature in iOS 14?

Unfortunately for Apple, breaking iCloud away from the OS isn’t a simple problem to solve. It’s not a technical challenge as much as it is an organizational one: a move from a functional form to a divisional one.

So while we all dream of an OS release roadmap to ease the transition to new features, Apple needs to give some very close to attention to the underlying services that would enable that new way of working.

The lack of substantive communication about iCloud’s problems this summer shows how far the company has to go in that regard. You can’t use a roadmap without letting people know about your successes and failures. There will be wrong turns, and you can’t continue on in silence when people depend on the service’s reliability.

iCloud gives data a bad name

As a developer whose job requires frequent beta installations, this year’s problems with Apple’s service has brought something clearly into focus: as much as I love the functionality that iCloud provides, I cannot put my valuable data at risk.

And you can bet your ass that I have a new backup strategy for anything that’s in iCloud Drive. (Hint: rsync and ~/Library/Mobile Documents.)

As an Apple shareholder, I also worry about how these failures will damage the iCloud brand. Apple’s growth hinges on services and when they get a bad name, the stock price can only go one way. There is only one brand that people think of when you mention “click of death”.

I sincerely hope that Apple can address these issues, because I love the idea of services that focus on simplicity and privacy. I know my last clusterfuck memo got some notice within the company, and I hope this one does as well.