iCloud Clusterfuck

Luckily, I don’t have to use this kind of title often. But when I do, there’s a good reason: this year’s beta release cycle for all of Apple’s operating systems has been a mess. The months since WWDC in June have been a terrible experience for both customers and developers alike and the literal center of the chaos was Apple’s iCloud syncing service.

For us, it all started with customers reporting lost Linea sketches in their iCloud Drive. Initial investigations led to a common factor: all of the people affected had installed the iOS 13 beta release.

And when I say lost, I mean really lost. Entire folders were either gone or corrupted. Apple’s mechanism to recover deleted files was of no help. The customers with weird folder duplicates were the “lucky” ones.

We couldn’t find any problems in our code and there was no information from Apple about problems with iCloud, so we did the only thing possible: we blocked people from using Linea on iOS 13. If a customer absolutely needed the app, we got them onto TestFlight after a stern warning about the real possibility of losing data.

A few weeks later, Apple finally indicated that there were some issues with iCloud and the beta release. In the same week, they released a public beta and sent out an email to customers encouraging them to try out iOS 13.

We did our best to understand the situation and provide information to Apple, but it felt like we were tossing bug reports into a black hole. The most discouraging part was when we tried to open an incident with Apple Developer Technical Support (DTS). After writing up a detailed report, we were informed that they don’t support beta releases!

By the time beta 6 rolled around, things were improving, but there were still isolated reports that the service might not be completely back in working order. Confirming the fixes on our end was impossible: the folks who’d encountered the previous data loss had no desire to mess with iCloud again.

Now it appears that the entire stack is getting rolled back and there won’t be new iCloud features in iOS 13 (at least initially.) I honestly think that’s the wisest course of action at this point. My only wish is that Apple would make an official statement.

Now that we’re past the worst of it, it’s time to think about why this whole episode happened and how it can be prevented in the future.

Folks don’t understand beta

Apple’s biggest fuck up was a bad assumption about who is testing a beta release. In WWDC presentations and developer documentation, there’s always warnings about using the release on non-production devices and only with test accounts.

But there are many folks that are just looking to get the new and shiny features. In past iOS beta releases, Apple hasn’t suffered too much from this because the early software was relatively stable. Maybe you got some dropped calls or bad battery life, but it was nothing too serious.

These early adopters installed iOS 13 and expected a similar experience. They also weren’t using an iCloud test account, so any instability in the beta release propagated bad data to their other devices.

Developers have long known to unhook external drives when testing a new OS release. Shit happens, and that’s OK because it’s a beta and we expect a bumpy road. In fact, I currently have an external Photos Library that I can’t use in Mojave because it got upgraded when I booted into Catalina: it’s my dumb mistake and I have no complaints.

Anyone who’s not a developer, and hasn’t been burned by a bad OS, does not know the kind of trouble that lies ahead. It’s irresponsible for Apple to release a public beta with known issues in iCloud. It’s doubly egregious to then promote that release with an email campaign to customers. For a company that prides itself in presenting a unified front, it sure looks like the left hand doesn’t know what the right hand is doing.

Disconnect iCloud by default

The solution to this situation is relatively simple, but with painful consequences. Apple needs to help unwitting customers by automatically disconnecting that external hard drive called iCloud.

If a device is using an Apple ID that’s also being used on a non-beta device, then iCloud shouldn’t be allowed. If you install an iOS beta on your iPad, it doesn’t get to use any cloud services because it puts the data on your iPhone or Mac at risk.

Of course, once this restriction is put into place, Apple will quickly realize that no one is testing iCloud or anything that touches it. If a beta tester can’t have their contacts, synced notes, reminders, and appointments over the course of three months, they’re not going to be testing much. I’d also expect to see lots of guides on “How to Get Back on Normal iOS” around the web.

But there’s a better, and even more unlikely solution…

iCloud can’t be a beta

Because it’s a service, iCloud doesn’t get to go into beta. It needs to be reliable all the time, regardless of whether iOS or any other platform is in beta test.

As it is now, Apple is effectively telling you that your storage device will be unreliable for a few months. It’s like having a hard drive where the manufacturer tells you it won’t work well for ¼ of the year. Would you purchase storage with a caveat that “the drive mechanism may not work properly during the hot summer months”?

You don’t see these kinds of issues with Google Drive or Dropbox because they don’t get a pass while an OS is being tested. Dropbox moved your folders from AWS to their own servers without you noticing even the slightest hiccup. Compare this to the last three months of iCloud in beta.

And yes this is a hard task: akin to swapping out the engines of an airplane in mid-flight. But data services, like hard drives, must work all the time.

We’re well past that point with iCloud and it’s just going to get worse as Apple moves more of our data there. This time next year, will I suddenly not know which episodes I’ve watched on Apple TV? Or will I suddenly lose all my scores and achievements in Apple Arcade? Will I shoot myself in the foot with some cool new feature in iOS 14?

Unfortunately for Apple, breaking iCloud away from the OS isn’t a simple problem to solve. It’s not a technical challenge as much as it is an organizational one: a move from a functional form to a divisional one.

So while we all dream of an OS release roadmap to ease the transition to new features, Apple needs to give some very close to attention to the underlying services that would enable that new way of working.

The lack of substantive communication about iCloud’s problems this summer shows how far the company has to go in that regard. You can’t use a roadmap without letting people know about your successes and failures. There will be wrong turns, and you can’t continue on in silence when people depend on the service’s reliability.

iCloud gives data a bad name

As a developer whose job requires frequent beta installations, this year’s problems with Apple’s service has brought something clearly into focus: as much as I love the functionality that iCloud provides, I cannot put my valuable data at risk.

And you can bet your ass that I have a new backup strategy for anything that’s in iCloud Drive. (Hint: rsync and ~/Library/Mobile Documents.)

As an Apple shareholder, I also worry about how these failures will damage the iCloud brand. Apple’s growth hinges on services and when they get a bad name, the stock price can only go one way. There is only one brand that people think of when you mention “click of death”.

I sincerely hope that Apple can address these issues, because I love the idea of services that focus on simplicity and privacy. I know my last clusterfuck memo got some notice within the company, and I hope this one does as well.

Catalina, App Notarization, and Sparkle

We recently started updating our macOS apps for Catalina: so far there have been very few issues with APIs and frameworks. The biggest hurdle has been the new notarization process that’s required for apps signed with a Developer ID: customers will be unable to download and launch your product easily until this step is completed.

Notarization involves an extra step in your build process: you upload an archived binary to Apple’s server with Xcode’s Organizer window and a short time later, you can export the binary. If you’ve automated your build process, you’ll need to make changes to your scripts to accommodate this new manual step. Apple’s documentation explains the process well.

Before you can notarize the app, you’ll need to enable the hardened runtime in the target’s Capabilities panel. After flipping the switch you’ll see a array of exceptions and access permissions. You’ll want to survey this list carefully: for one product we needed Apple Events, for another Location was required.

Things start to get tricky when you go to upload the binary: if you’re using Sparkle, it’s probably been codesigned without the hardened runtime, so you’ll immediately see an error.

Sparkle Without a Sandbox

How you deal with this error depends on which of the Sparkle versions you’re using. If your app isn’t sandboxed, your life will be a bit simpler because there are fewer things you’ll need to sign manually.

After the target’s Copy Files build phase where the Sparkle.framework is moved into the application package, you’ll need to create a new Run Script step: I called ours “Sign Frameworks”. The script looks like this:


codesign --verbose --force --deep -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A/Resources/AutoUpdate.app"
codesign --verbose --force -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A"

The key part in this step is the -o runtime. The codesign manual page describes this flag as:

On macOS versions >= 10.14.0, opts signed processes into a hardened runtime environment which includes runtime code signing enforcement, library validation, hard, kill, and debugging restrictions. These restrictions can be selectively relaxed via entitlements. Note: macOS versions older than 10.14.0 ignore the presence of this flag in the code signature.

The good news here is that the build changes we’re making won’t affect your app when it runs on an older version of macOS.

Sparkle in a Sandbox

If your macOS app is in a sandbox, you’ll be using the version that relies on XPC services to perform the update. Like everything else in your application package, these will need to be signed correctly before you can submit your app for notarization.

The “Sign Frameworks” build phase should look like this:


codesign --verbose --force -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A/Resources/AutoUpdate"
codesign --verbose --force --deep -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A/Resources/Updater.app"
codesign --verbose --force -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A"

You’ll also add a new Run Script build phase just before the XPC Services are embedded in your application package. Since you’ll only need to do this for release builds, the script looks like this:


if [ "${CONFIGURATION}" = "Release" ]; then
    $PROJECT_DIR/Sparkle/bin/codesign_xpc "$IDENTITY" $BUILT_PRODUCTS_DIR/*.xpc

But wait, we’re not done yet! You’ll also need to update the codesign_xpc Python script with the -o runtime flag. It looks like this when you’re done:

def _codesign_service(identity, path, entitlements_path=None):
    command = ["codesign", "-f", "-o", "runtime", "-s", identity, path] + ([] if entitlements_path is None else ["--entitlements", entitlements_path])
    log_message(" ".join(map(sanitize, command)))

You’re Not Done Yet

At this point, you should be able to do a build where everything in your app is using the hardened runtime. It’s more likely that you’ve had some kind of issue along the way: this Apple document helped get me over the rough patches. (Thankfully, I didn’t have to write it this time around.)

After the notarization upload completes, you’ll see “Uploaded to Apple” in the organizer, then after a few minutes you’ll get an email and Xcode notification that your app is “Ready to distribute”. In the righthand panel underneath “Distribute App”, you’ll see that the “Export Notarized App” button is enabled and can be used to place the signed package anywhere on your Mac for further processing.

In our case, we had to split up the build scripts into two parts: previously we had a single script that did the build, signed it with the Developer ID, and then created an appcast. Sparkle’s XML file is now created with a separate script that also prepares the release to be checked into our repositories.

One final note: these instructions are based on Xcode 10, which is currently the only development tool that can be used to submit an app for notarization or the Mac App Store. Before we figured that out, we found that Xcode 11 does a better job passing along the -o runtime flag during a framework’s Code Sign On Copy. It’s likely that all this work you just did will only be needed for a few months. Sigh.

The Future of Interaction, Part II

I am so fucking excited about SwiftUI.

I had strong suspicions that Apple was working on something like this, but what we’ve seen this week goes way beyond any of my expectations. My thoughts have now turned to how important this development will be for the future of all user interaction.

It all started while sitting at this year’s edition of The Talk Show and hearing about one of its sponsors: OmniFocus for the Web.

I’ve been good friends with the mad scientists at the Omni Group for a long time: we all have the gray hairs to prove it.

When you hear that they made a server with a macOS app that renders content for web, it sounds strange at first, but then makes a lot of sense when you consider how much code sharing you get. They’re not fools.

Then, while watching the SwiftUI Essentials talk, it occurred to me that the new domain specific language (DSL) looked like a HTML, CSS, and JavaScript hierarchy.

When you connect these two dots, you see a pretty exciting future for your code.


Apple has talked a lot about SwiftUI this week, but one thing has been conspicuous in its absence: implementation details. Some folks have discovered that there is some underlying UIKit code, but looking at this is a distraction.

The most important part of this announcement is the abstraction they’re working with, not the view surface being used for rendering.

This view abstraction is important for our apps: you don’t need to care how a button is rendered on tvOS, or macOS, or iOS. You don’t need to worry about which interactions are available on each platform, other than to provide alternatives when things like a digital crown aren’t available.

Interactions Above All

Earlier this week, I was talking to my friend Soroush Khanlou about the announcements. He expressed surprise that so many long-time developers who were vocal about Swift’s deficiencies were suddenly excited about SwiftUI.

There’s a simple explanation for this: we value great interaction, both at the user level and API level.

To me, it always felt like there was a parity mismatch between Swift and the existing UI frameworks. Sure, you can put a Ferrari engine in a Ford pickup and get it to move, but it’s not going to be a great experience. Awkward to build, awkward to drive around in.

The Web, Too

So my observation about the SwiftUI DSL and OmniFocus show us something important: a web browser is just another surface to render your views and respond to user actions or external events.

And it will be a way where interactions can be adapted for the platform being targeted.

Think about all the work that Apple has put into making iWork apps run in a web browser. Would they be looking for ways to make this easier? Yep.

I also think we’ll see the Swift community pick up on this DSL. Initially, it rubbed me the wrong way that Apple was using “Swift” in a framework that targeted platforms using proprietary features like Core Animation.

But again, when you look at the abstraction, you see the greatest value in how the DSL is processed, not in how it makes it into a CALayer. Could some enterprising group of developers take the SwiftUI DSL and produce something that worked on Android or Playdate? Certainly.

And what about compiling Swift code and running it directly in a web browser? You can do it right now.

The Most Capable Environment

While commenting about this on Twitter, Steve Streza made an important observation: SwiftUI is like a “reverse React”.

But I think there’s something important to add to his note: the SwiftUI DSL describes the most capable environment. It’s the maximum interaction surface: platforms will render and react to a subset of what’s declared.

Some devices, like a watch, will be capable of handling a physical rotation from a dial. Your only concern on other platforms, like iOS, are alternate interactions. When you target Playdate with your app’s SwiftUI DSL, you won’t be surprised to see the crank do the right thing.

I didn’t realize it at the time, but John Gruber’s quip at the beginning of The Talk Show hinted at our future:

For all of the consternation we have for developers who have web apps and put it in a shell and call it their Mac app, [laughter] the Omni Group has done the right thing and took their Mac app, and now it’s on the web.

It was certainly a funny moment, but it also gets directly to the point of working from a more-capable to a less-capable environment. And also how we’re at a point where our apps will head in directions that we can’t even begin to predict. I find that tremendously exciting.

In summary, I think that SwiftUI is an absolutely fundamental change that we’ll be living with for decades. You can certainly dismiss it, but doing so will be at your own peril.

The Future of Interaction

Shortly after finishing my treatise on Marzipan, I started thinking about what lies beyond. Some of those initial thoughts made it into a thread on Twitter.

This post can be considered an addendum or a hell of a long footnote: in either case, you’ll want to start by reading my thoughts on Marzipan. Because what’s happening this year is just the start of a major shift in how we’re going to build apps.

So while everyone else is making predictions about WWDC 2019, what you’ll find below are the ones I’m making for 2020 and beyond.

Update June 7th, 2019: It turns out I was making predictions for 2019, after all.

What is Apple’s Problem?

Before we get into thinking about the future, let’s look at a problem that Apple has today: too many products on too many platforms.

Historically, a person only had one computer to deal with. For several generations it was a mainframe; more recently it was a PC. One of the disruptive changes that started with the iPhone was the need to juggle two computers. Now we have watches generating and displaying data: another computer. Increasingly, the audio and video devices in our living rooms are added to the mix.

Syncing and cloud services help manage the data, but we all know the challenges of keeping a consistent view on so many machines.

If you’re an iMessage developer, you have to think about a product that works on iOS, macOS, and watchOS. You get a pass on tvOS, but that’s small consolation. The same situation exists in various combinations for all of Apple’s major apps: Music, Calendar, Reminders, Notes, Mail, etc.

It’s likely that all of these apps share a common data model, probably supported by an internal framework that can be shared amongst platforms. That leaves the views and the controllers as an area where code can’t be shared.

Marzipan is About Views

With this insight, it’s easy to see Marzipan as a way towards views that share code. A UIView can be used on your TV, on your desktop, on your wrist, and in your pocket. That’s a big win for developer productivity.

It’s also a win for designer productivity: you can share app design elements. We already see this in Apple’s cross-platform apps when colors in Calendar match, speech bubbles in Message have the same shape, and Notes shares a special glyph for the “A”.

Everyone’s excited to know what the Dark Mode on iOS is going to look like. My guess is that people who have been running a dark user interface on their Mac have already seen it. It’s hard to find a balance of readability and contrast with dark elements and I don’t see Apple’s designers making any major changes in next week’s announcement. There will certainly be refinements, but it makes no sense to throw out the huge amount of work that’s already been done.

I also see the Mac leading the way with techniques that provide a more vibrant interface and allow a customer to customize their device. The accent color in System Preferences would be a welcome addition in the iOS Settings app.

All of this leads to a common appearance across platforms. In the near future, we’ll be in a nice place where our architecture can be shared in models, and our designs shared in views.

That leaves us with one final problem to solve: how do we share our interactions and controllers?

The Arrival of New Interactions

Apple ties interactions to platforms and their associated hardware. The Mac has interactions that are different than iOS. And watchOS has ones that are different than iOS. On tvOS, you can be limited to four arrow keys and two buttons.

There are some interactions, such as a swipe, that appear on multiple platforms, but as a whole each platform is different. This approach lets the customer get the most out of the device they purchase.

As we start to think about how interactions are shared amongst platforms, it’s wise to consider new hardware might be arriving soon.

For the past few years, Apple has been putting a lot of effort into augmented reality (AR). And I have no doubt that this hard work is not for our current devices.

AR is a great demo on an iPhone or iPad, but the reality is that you can’t hold a device in front of your face for an extended period of time: your arm gets tired after just a few minutes. I made this argument over a decade ago when everyone was getting excited about multi-touch displays coming to their desktop. It still holds true because it’s a dumb idea, just like AR on a mobile phone.

Apple will solve this problem with new hardware. And these devices will run “headgearOS” with new and completely different interactions. If you’re that Messages developer, it means you’ll be writing new code for yet another platform. Yay.

There are other twists to this story: rumors about iPads with mice and Macs with touch screens. And let’s not forget about interactions with voice commands using Siri technologies.

It all adds up to a situation where the complexity of products is increasing exponentially as new devices and interactions are introduced. There has to be a better way.

How Not to Do It

Cross-platform frameworks have a long history of sucking. If you ever used a Java app during the early days of Mac OS X, you know immediately what I’m talking about: the interactions were from a different universe. The design of the system was a “least common denominator” where only a limited set of capabilities was exposed. It just felt wrong.

More recent attempts have had more success but they still fail to address the problem of ever-expanding interactivity.

Apple’s been down this road before and I don’t see them making the journey again. Instead, I see them taking a new and forward thinking direction. A bold and pragmatic change that Apple is famous for: they’d be setting themselves up for the next decade of user interaction.

So what could they do that no one else in the industry is doing?

Declarative Interactions

As Matt Gallagher notes, we’ve slowly been heading towards a declarative programming style.

Declarative programming is describing a system using set of rules and relationships. The rules and relationships cannot be changed during the lifetime of the system (they are invariant), so any dynamic behavior in the system must be part of the description from the beginning.

Syntactically, declarative programming is often about assembling a whole system of rules as either a single expression or domain-specific language whose structure reflects the relationship between the rules in the system.

Layout is an inherently declarative task. Layout is a set of rules (which Auto Layout calls “constraints”) that apply to the contents of a view, ideally for the entire lifetime of the contents. Constraint programming itself is sometimes considered a sub-discipline of declarative programming.

He pulls this together with his own experiences into a prediction about a Swift-only framework for “declarative views”.

Independently, John Gruber has made similar observations.

The general idea is that rather than writing classic procedural code to, say, make a button, then configure the button, then position the button inside a view, you instead declare the button and its attributes using some other form. HTML is probably the most easily understood example. In HTML you don’t procedurally create elements like paragraphs, images, and tables — you declare them with tags and attributes in markup.

In my opinion, limiting this thinking to just views and layout is short-sighted.

That’s because it doesn’t address the interaction problem with an ever increasing set of platforms. But what if this new framework not only let you declare views, but also the behaviors they enable?

The developer would describe the interactions an app supports. There would be relationships between those declared interactions. All this immutable information would then be processed by user interface frameworks. Your app’s behavior would be determined at runtime, not when it was compiled.

Think about how this would work using auto layout as a point of comparison. Until that declarative system came along, we all worried about frame placement. Now we just worry about the relationships between those frames and let the system pick what’s best.

With our interactions, we still have this tight coupling between a user action and app’s behavior. Tapping on a button or a swipe gesture invokes some code directly. A declarative interaction would be a layer of abstraction between what a customer does and how your app reacts.

Again, using auto layout to help form our thinking, what if there were “interaction classes” that functioned like size classes? Your iPad would behave as it always has until you plugged in a mouse. At that point, how you interact with the device adapts:

  • Controls could get smaller because of the increased pointing accuracy
  • Views could gain a hover state to display additional information
  • Drag & drop could change because it no longer depends on two fingers
  • A mechanical wheel could replace a finger for scrolling

This kind of adaptability would work across platforms – your app would behave differently when it was running in augmented reality or on a TV screen. As a developer you wouldn’t have to worry about what kind of hardware is available, you’d have to worry about what to do when a customer used it to perform a task.

I’m not going to predict how this would be accomplished. Yes, it could draw inspiration from React or other similar technologies. The only thing I’m confident of at this point is that Apple knows it has a problem and is working actively to solve it in a platform-independent fashion.

Marzipan is our first step. And what I’ve described above is Amber, the next step.

Software Survival

I just finished reading an oddly named book by someone you’ve probably never heard of – I loved it so much I’m writing this site’s first book review.

Ken Kocienda is a talented developer. He’s also a lucky one: he worked at Apple during the years when the Mac was reborn and the iPhone was created.

If you’ve ever opened the web browser in your pocket or typed a text message on a piece of glass, you’ve used his seminal work. Ken was directly responsible for two important parts of the original iPhone: a fast and efficient web framework and a keyboard that was both virtual and predictive.

Ken is also a great storyteller. His experiences during these formative years provide great material for this narrative. But more importantly, he extracts lessons from these tales that can be applied to our own efforts.

In short, Creative Selection is a study on how great products evolve that inspires the reader.

I first learned of Ken’s work during a presentation at WWDC 2014. A Strategy for Great Work showed that in addition to doing interesting things, he also took the time for introspection and learning from the work. I immediately followed him on Twitter and eventually discovered our mutual admiration! (That, in turn, led to the advanced copy I’m reviewing today.)

This talk, along with the ones that preceded it, feel like a formative process for the chapters of the book. The video linked above will give you a very good idea for the tone of his latest work, but there’s also an important difference.

Ken no longer works at Apple.

The end of presentation in 2014 alludes to this situation: not being able to talk about certain things is a fact of life in Cupertino.

I found Ken’s new freedom most intriguing when he focused on other people’s contributions to the secret projects. Every developer knows that a success involves many different talents and personalities. And with Apple’s penchant for secrecy, we never hear about these team interactions.

The collaboration between Steve Jobs and Jony Ive is not only well known, it’s well understood. I think this is because industrial design is much more concrete and easier to interpret in the formative stages.

Steve had another important collaborator, but that relationship isn’t as well understood. Creating software requires abstract decisions and a more opaque process. How do you make a button on a piece of glass?

When you look at the Wallabies from Ken’s desk, you can see where the hardware is headed:

iPhone Prototypes from Ken Kocienda's desk.

When you think about what was on those screens, things get more complicated.

This book changed my view of Scott Forstall because it gave context to his work. Ken’s account breaks down the approach and shows how important Scott’s leadership was to Apple’s success.

Having Steve Jobs as a boss for your entire professional career would not be easy, but Scott handled it with great success. Even when that powerful mentor was asking for skeuomorphism.

The hardware would be lesser without Jony, and Ken shows that the software would be lesser without Scott.

If you’re a developer, you’ll find some passages a little slow going. That’s because you know too much :-)

As a great storyteller, Ken endeavors to make the topic as widely understood as possible. To achieve this he turns to analogies in cooking, sports, and the movies while describing the process of creating software.

While we may not need this high-level view, there is an important group that will benefit: our families. If you’re like me, you’ve struggled to explain what you do at that keyboard all day.

We all love our iPads and Kindles for reading, but I’m really glad I have a printed copy of the book to share with the people I love.

I’ve avoided spoilers in this review, but I’ll end with one story that touched me. When asked about the most important thing in development, I always answer with one word: empathy.

Ken links this concept, along with the combination of good taste, the philosophy of Immanuel Kant, and a discussion of atoms by Richard Feynman, all while explaining the choice of a QWERTY keyboard layout.

These complex factors enable a personal connection between people. Imagine being the developer whose choices led to the ability to send a quick message saying “Just landed.” Informing a loved one of your safe arrival transcends the importance of any line of code.

That is work to be proud of and we’re lucky Ken has taken the time to share it with us all.