Introducing Waterscope

It’s common wisdom that you should release a software product when it is minimally viable: get an early version out in the world as soon as it can perform a useful task for a customer.

When that product is for someone who is a developer that’s been coding since the dawn of time, the equation gets flipped on its head. Waterscope is a Maximally Viable Product™ and the customer is me.

The app got its start when Swift 1.0 was announced back in 2014: I wanted to build something with the new language. About that same time, I had also started learning about tides and how they are predicted. It’s a complex problem that has been vexing scientists since the three-body problem was first proposed by Newton with the publication of the Principia in 1687. Like determining the time and place for a lunar eclipse, we rely on derived approximations.

For learning a new language, tide prediction provided a lot of interesting work: data collection, complex calculations, graphical presentation, and animatable data. It also let me know when it was a good time for a dog walk.

As an ocean swimmer, I also wanted my weather app to provide information about water conditions. It turns out the scientists at the National Oceanic and Atmospheric Administration (NOAA) have that all figured out. As do the meteorologists at the National Weather Service (NWS) with their API for weather observations and forecasts. There are even high resolution images from environmental satellites launched by NASA. The United States government provides a treasure trove of data; the challenge with Waterscope was to organize and present it in a consistent manner.

Screenshots of Waterscope showing the home screen graphs, a map view with sea and land weather conditions, and a satellite view of California.

Which leads to a secondary goal for this app: to make it completely by myself. I work with some incredibly talented designers, but I wanted Waterscope to be uniquely my own. All the design, for better or worse, was created by my own hand. (The only exception is the use of SF Symbols when showing weather conditions.)

So not only was I learning Swift, I was also learning Sketch and, of course, how much time it takes to get something to feel right. Like coding, it’s not as easy as we sometimes make it look. Don’t take your designers for granted :-)

Along the way, there were some interesting hurdles. Some visual, some simple, and some complex. Many of the things I wanted to do required learning about astronomy and orbital mechanics. My sketches from Linea will give you an idea of the breadth of the challenges. (And being the day after the solstice, that first sketch is particularly relevant.)

Sketches for a user interface design that shows seasons using the illumination of the Earth, the math for linear interpolation, and a spherical projection of a point on a satellite image.
Sketches for a user interface that shows seasons by illuminating the Earth, the math for linear interpolation, and a spherical projection for a location on a satellite image.

Here I am, six years later. I’ve learned a lot, but as with my first app in Objective-C, the most important thing about this exercise was how not to use Swift. It will probably take me another 18 years to come to terms with this new language, and feel like I’ve mastered it, but a journey can’t start without the first steps. Another insight is that a programming language is just a means to an end: the hard part is not the code, it’s understanding what needs to be done.

I’m releasing Waterscope today because there are certainly other folks who will benefit from my personal weather app. There may even be some educational value in seeing how I approached a data-rich user interface (hint: Edward Tufte’s books taught me). Information can be dynamic and beautiful.

At the same time, if you’re outside the U.S., it’s unlikely to be a satisfying experience: most of the data sources and their presentation are oriented towards North America. An example: in the southern hemisphere your view of the sun and moon’s orbit is in a counterclockwise direction as you look north. Waterscope displays a clockwise orbit.

But the good news is that Waterscope, like the data it uses, is FREE to download and use. Enjoy!

Codesign: The Saga Continues

I have a long history of writing about code signing in macOS. When Big Sur was released, I thought “Finally!”

I was wrong.

This time around I was tripped up by Safari, of all things. It doesn’t open app archives like other parts of macOS.

This story began with customer reports of xScope being a “damaged app” on Big Sur. This was surprising because I had been downloading and testing the app on Big Sur for several months without issue.

I was also doing all this work on Apple Silicon using the DTK. And since Google Chrome wasn’t yet working on this device, all my testing was limited to Safari. Safari’s default setting is to open “safe” files after download, so I left that alone (as most customers would).

This is where I shot myself in the foot. At no point did my downloads touch the Archive Utility. And I had no idea that Safari’s code is different than the system utility.

When I checked the signature of the app downloaded with Safari, everything looks good:

$ codesign -vvvv ~/Downloads/xScope.app
...
/Users/CHOCK/Downloads/xScope.app: valid on disk
/Users/CHOCK/Downloads/xScope.app: satisfies its Designated Requirement

Things were very different when using Google Chrome:

$ codesign -vvvv ~/Downloads/xScope.app
/Users/CHOCK/Downloads/xScope.app: unsealed contents present in the root directory of an embedded framework
In subcomponent: /Users/CHOCK/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework

$ codesign -vvvv ~/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework
/Users/CHOCK/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework: a sealed resource is missing or invalid
file added: /Users/CHOCK/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework/Versions/Current/Resources/._fr_CA.lproj
file added: /Users/CHOCK/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework/Versions/Current/Resources/Updater.app/Contents/Resources/._fr_CA.lproj
file added: /Users/CHOCK/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework/Versions/Current/Resources/Updater.app/Contents/Resources/._pt.lproj
file added: /Users/CHOCK/Downloads/xScope.app/Contents/Frameworks/Sparkle.framework/Versions/Current/Resources/._pt.lproj

It turns out all these folks complaining about a “damaged app” were either using Chrome or had Safari’s “safe” file handling turned off. The damaged archive wasn’t getting repaired automatically by Safari.

The root of the problem is localization in the Sparkle framework. There are two symlinks with extended attributes (the “._” is where macOS stores things like Finder information). The intent of the symlink was to say that French Canadian is the same as French, and Portuguese is the same as Brazilian Portuguese.

Since macOS automatically makes this inference, it’s safe to just delete the scripts that create the symlinks. In your Sparkle project, find any occurrences of “Run Script: Link fr_CA to fr” and “Run Script: Link pt to pt_BR” in your Target Build Phases and remove them. I had them in “Sparkle”, “SparkleCore”, and “Installer Progress”.

After you build and notarize, you’ll see that your app is “valid on disk” no matter how it’s unarchived.

I’ve also submitted this information to Apple’s Product Security group. As I said in my email, the biggest problem here is expectations:

The reason I’m writing is because Safari’s implementation for opening “safe” files is somehow bypassing a code signing check or repairing the downloaded package. The Archive Utility does not. Customer and developer expectations for unzipping archives is that they are not modified and behave the same way across all Apple products.

If you’re a Mac developer who’s using Sparkle and distributing your product via a web download, now’s a good time to check how things work in a variety of browsers. I’ve heard that we’re not the only ones affected.

And if you encounter a download that’s damaged because of these Sparkle symlinks, this quick fix in the Terminal will set things right:

$ ditto xScope.app xScopeFixed.app
$ rm -rf xScope.app
$ mv xScopeFixed.app xScope.app

The ditto command strips the extended attributes that are causing the issue. This may, in fact, be what Safari is doing for “safe” files.

All that’s left to do now is wonder what surprises codesign has in store for next year’s release of macOS…

Triode and Internet Radio

Today, the Iconfactory is happy to introduce a product that’s made with three different apps: Triode. Simply put, it’s the best way to enjoy all of your favorite Internet radio stations, wherever you go.

So why did I make these apps?

A Brief History of Internet Radio

Before I can answer this question, it helps to know a bit about where I live. Laguna Beach is surrounded on three sides by a coastal range that blocks any broadcast from the Los Angeles and San Diego metro areas. A signal’s only clear shot is from the Pacific Ocean, where there are no radio towers.

I moved to Laguna in the mid-1990’s — as you probably know, that was a time when the Internet was starting to take off.

Progressive and independent radio stations immediately saw the Internet as way to extend their reach: WXYC was the first station to broadcast a stream in 1994. Many others soon followed.

I saw it as a way to overcome the physical limitations of where I lived. And in the process, I discovered a lot of interesting sounds from all over the world.

In short, Internet radio is nothing new and I’ve been listening to it since the late 1990’s.

Old Meets New

What is new are the array of devices we can use to listen to radio streams. In the beginning, my only option was a desktop computer, but now I also carry a powerful device in my pocket and have another one attached to my TV.

These devices also have advanced capabilities like multi-room audio with AirPlay 2, a CarPlay interface while driving, automation with Siri shortcuts, and iCloud data syncing. They’re also highly accessible thanks to VoiceOver.

Yet many radio stations provide free apps that are cross-platform and can’t take advantage of the features that Apple provides its developers. In fact, many of these apps are basically a view that use the same JavaScript-based player on the station’s website.

I saw an opportunity to do something better.

Always With Me

Early in 2018, Triode got its start on tvOS as a way to listen to music while relaxing on the couch. I loved that so much, I then wanted it while working on my Mac. Then, while out and about with my iPhone and AirPods. A simple idea was blossoming.

The challenge quickly became creating a unified experience across three different devices and input mechanisms. Building a user interface that worked natively with a TV remote, a mouse, and your finger was a challenge!

Overcoming this hurdle had a real benefit: it’s pretty awesome to find a great station on your Mac while you work, and have it automatically show up on your iPhone during a commute, or on your TV as you relax in the evening.

Early in the development process it became apparent that great artwork was an important part of the experience. Seeing a beautiful new album cover has always been essential part of discovering an artist or track. It became a central element of our unification effort.

This effort also helped refine my thoughts on the future of interaction on Apple’s platforms. And, as it turns out, SwiftUI.

A Place of Discovery

With the help of my colleagues, the user interface and interaction started shaping up. The task then became how to introduce folks to Internet radio.

Most existing players have another flaw: they promote quantity over quality. It sounds impressive to tout tens of thousands of stations — until you realize that a lot of them aren’t very good. Clearly, some curation was needed.

Over the years, I had accumulated a collection of favorite stations. This formed the core of the “Our Picks” in Triode.

While thinking about how to extend this group of stations, it became clear that they all had one thing in common: they were independent and listener-supported.

These stations have have long been a place to discover new music. They’re often associated with a college or university that has a student body looking for the latest and greatest. The lack of commercial pressures also provides an environment where new acts can get their first bread and established artists can do something unexpected.

Every major artist used to be someone you had never heard of, and Internet radio played a major part in that transformation.

That spirit guided our curation of “Our Picks”, while “Find a Station” provides quick access to thousands of stations from around the world. Both will become “Your Favorites” and automatically get synced to all your devices.

Privacy First Design

Last, but certainly not least, Triode was designed with your privacy in mind.

Unfortunately, today’s Internet is full of trackers. If you’re using a web-based stream or app, its likely that your listening habits and other metrics are being collected.

In Triode, the only information provided to broadcasters is your IP address. And since we don’t display any ads in the app, that’s one less thing for marketeers to know about you.

And More

The Triode website has more details and this review at MacStories goes into great detail about how the app looks and works on the different platforms.

It’s been a blast making these apps — I hope you enjoy them!

iCloud Clusterfuck

Luckily, I don’t have to use this kind of title often. But when I do, there’s a good reason: this year’s beta release cycle for all of Apple’s operating systems has been a mess. The months since WWDC in June have been a terrible experience for both customers and developers alike and the literal center of the chaos was Apple’s iCloud syncing service.

For us, it all started with customers reporting lost Linea sketches in their iCloud Drive. Initial investigations led to a common factor: all of the people affected had installed the iOS 13 beta release.

And when I say lost, I mean really lost. Entire folders were either gone or corrupted. Apple’s mechanism to recover deleted files was of no help. The customers with weird folder duplicates were the “lucky” ones.

We couldn’t find any problems in our code and there was no information from Apple about problems with iCloud, so we did the only thing possible: we blocked people from using Linea on iOS 13. If a customer absolutely needed the app, we got them onto TestFlight after a stern warning about the real possibility of losing data.

A few weeks later, Apple finally indicated that there were some issues with iCloud and the beta release. In the same week, they released a public beta and sent out an email to customers encouraging them to try out iOS 13.

We did our best to understand the situation and provide information to Apple, but it felt like we were tossing bug reports into a black hole. The most discouraging part was when we tried to open an incident with Apple Developer Technical Support (DTS). After writing up a detailed report, we were informed that they don’t support beta releases!

By the time beta 6 rolled around, things were improving, but there were still isolated reports that the service might not be completely back in working order. Confirming the fixes on our end was impossible: the folks who’d encountered the previous data loss had no desire to mess with iCloud again.

Now it appears that the entire stack is getting rolled back and there won’t be new iCloud features in iOS 13 (at least initially.) I honestly think that’s the wisest course of action at this point. My only wish is that Apple would make an official statement.

Now that we’re past the worst of it, it’s time to think about why this whole episode happened and how it can be prevented in the future.

Folks don’t understand beta

Apple’s biggest fuck up was a bad assumption about who is testing a beta release. In WWDC presentations and developer documentation, there’s always warnings about using the release on non-production devices and only with test accounts.

But there are many folks that are just looking to get the new and shiny features. In past iOS beta releases, Apple hasn’t suffered too much from this because the early software was relatively stable. Maybe you got some dropped calls or bad battery life, but it was nothing too serious.

These early adopters installed iOS 13 and expected a similar experience. They also weren’t using an iCloud test account, so any instability in the beta release propagated bad data to their other devices.

Developers have long known to unhook external drives when testing a new OS release. Shit happens, and that’s OK because it’s a beta and we expect a bumpy road. In fact, I currently have an external Photos Library that I can’t use in Mojave because it got upgraded when I booted into Catalina: it’s my dumb mistake and I have no complaints.

Anyone who’s not a developer, and hasn’t been burned by a bad OS, does not know the kind of trouble that lies ahead. It’s irresponsible for Apple to release a public beta with known issues in iCloud. It’s doubly egregious to then promote that release with an email campaign to customers. For a company that prides itself in presenting a unified front, it sure looks like the left hand doesn’t know what the right hand is doing.

Disconnect iCloud by default

The solution to this situation is relatively simple, but with painful consequences. Apple needs to help unwitting customers by automatically disconnecting that external hard drive called iCloud.

If a device is using an Apple ID that’s also being used on a non-beta device, then iCloud shouldn’t be allowed. If you install an iOS beta on your iPad, it doesn’t get to use any cloud services because it puts the data on your iPhone or Mac at risk.

Of course, once this restriction is put into place, Apple will quickly realize that no one is testing iCloud or anything that touches it. If a beta tester can’t have their contacts, synced notes, reminders, and appointments over the course of three months, they’re not going to be testing much. I’d also expect to see lots of guides on “How to Get Back on Normal iOS” around the web.

But there’s a better, and even more unlikely solution…

iCloud can’t be a beta

Because it’s a service, iCloud doesn’t get to go into beta. It needs to be reliable all the time, regardless of whether iOS or any other platform is in beta test.

As it is now, Apple is effectively telling you that your storage device will be unreliable for a few months. It’s like having a hard drive where the manufacturer tells you it won’t work well for ¼ of the year. Would you purchase storage with a caveat that “the drive mechanism may not work properly during the hot summer months”?

You don’t see these kinds of issues with Google Drive or Dropbox because they don’t get a pass while an OS is being tested. Dropbox moved your folders from AWS to their own servers without you noticing even the slightest hiccup. Compare this to the last three months of iCloud in beta.

And yes this is a hard task: akin to swapping out the engines of an airplane in mid-flight. But data services, like hard drives, must work all the time.

We’re well past that point with iCloud and it’s just going to get worse as Apple moves more of our data there. This time next year, will I suddenly not know which episodes I’ve watched on Apple TV? Or will I suddenly lose all my scores and achievements in Apple Arcade? Will I shoot myself in the foot with some cool new feature in iOS 14?

Unfortunately for Apple, breaking iCloud away from the OS isn’t a simple problem to solve. It’s not a technical challenge as much as it is an organizational one: a move from a functional form to a divisional one.

So while we all dream of an OS release roadmap to ease the transition to new features, Apple needs to give some very close to attention to the underlying services that would enable that new way of working.

The lack of substantive communication about iCloud’s problems this summer shows how far the company has to go in that regard. You can’t use a roadmap without letting people know about your successes and failures. There will be wrong turns, and you can’t continue on in silence when people depend on the service’s reliability.

iCloud gives data a bad name

As a developer whose job requires frequent beta installations, this year’s problems with Apple’s service has brought something clearly into focus: as much as I love the functionality that iCloud provides, I cannot put my valuable data at risk.

And you can bet your ass that I have a new backup strategy for anything that’s in iCloud Drive. (Hint: rsync and ~/Library/Mobile Documents.)

As an Apple shareholder, I also worry about how these failures will damage the iCloud brand. Apple’s growth hinges on services and when they get a bad name, the stock price can only go one way. There is only one brand that people think of when you mention “click of death”.

I sincerely hope that Apple can address these issues, because I love the idea of services that focus on simplicity and privacy. I know my last clusterfuck memo got some notice within the company, and I hope this one does as well.

Catalina, App Notarization, and Sparkle

We recently started updating our macOS apps for Catalina: so far there have been very few issues with APIs and frameworks. The biggest hurdle has been the new notarization process that’s required for apps signed with a Developer ID: customers will be unable to download and launch your product easily until this step is completed.

Notarization involves an extra step in your build process: you upload an archived binary to Apple’s server with Xcode’s Organizer window and a short time later, you can export the binary. If you’ve automated your build process, you’ll need to make changes to your scripts to accommodate this new manual step. Apple’s documentation explains the process well.

Before you can notarize the app, you’ll need to enable the hardened runtime in the target’s Capabilities panel. After flipping the switch you’ll see a array of exceptions and access permissions. You’ll want to survey this list carefully: for one product we needed Apple Events, for another Location was required.

Things start to get tricky when you go to upload the binary: if you’re using Sparkle, it’s probably been codesigned without the hardened runtime, so you’ll immediately see an error.

Sparkle Without a Sandbox

How you deal with this error depends on which of the Sparkle versions you’re using. If your app isn’t sandboxed, your life will be a bit simpler because there are fewer things you’ll need to sign manually.

After the target’s Copy Files build phase where the Sparkle.framework is moved into the application package, you’ll need to create a new Run Script step: I called ours “Sign Frameworks”. The script looks like this:

LOCATION="${BUILT_PRODUCTS_DIR}"/"${FRAMEWORKS_FOLDER_PATH}"
IDENTITY=${EXPANDED_CODE_SIGN_IDENTITY_NAME}

codesign --verbose --force --deep -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A/Resources/AutoUpdate.app"
codesign --verbose --force -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A"

The key part in this step is the -o runtime. The codesign manual page describes this flag as:

On macOS versions >= 10.14.0, opts signed processes into a hardened runtime environment which includes runtime code signing enforcement, library validation, hard, kill, and debugging restrictions. These restrictions can be selectively relaxed via entitlements. Note: macOS versions older than 10.14.0 ignore the presence of this flag in the code signature.

The good news here is that the build changes we’re making won’t affect your app when it runs on an older version of macOS.

Sparkle in a Sandbox

If your macOS app is in a sandbox, you’ll be using the version that relies on XPC services to perform the update. Like everything else in your application package, these will need to be signed correctly before you can submit your app for notarization.

The “Sign Frameworks” build phase should look like this:

LOCATION="${BUILT_PRODUCTS_DIR}"/"${FRAMEWORKS_FOLDER_PATH}"
IDENTITY="${EXPANDED_CODE_SIGN_IDENTITY}"

codesign --verbose --force -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A/Resources/AutoUpdate"
codesign --verbose --force --deep -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A/Resources/Updater.app"
codesign --verbose --force -o runtime --sign "$IDENTITY" "$LOCATION/Sparkle.framework/Versions/A"

You’ll also add a new Run Script build phase just before the XPC Services are embedded in your application package. Since you’ll only need to do this for release builds, the script looks like this:

IDENTITY="${EXPANDED_CODE_SIGN_IDENTITY}"

if [ "${CONFIGURATION}" = "Release" ]; then
    $PROJECT_DIR/Sparkle/bin/codesign_xpc "$IDENTITY" $BUILT_PRODUCTS_DIR/*.xpc
fi

But wait, we’re not done yet! You’ll also need to update the codesign_xpc Python script with the -o runtime flag. It looks like this when you’re done:

def _codesign_service(identity, path, entitlements_path=None):
    command = ["codesign", "-f", "-o", "runtime", "-s", identity, path] + ([] if entitlements_path is None else ["--entitlements", entitlements_path])
    log_message(" ".join(map(sanitize, command)))
    ...

You’re Not Done Yet

At this point, you should be able to do a build where everything in your app is using the hardened runtime. It’s more likely that you’ve had some kind of issue along the way: this Apple document helped get me over the rough patches. (Thankfully, I didn’t have to write it this time around.)

After the notarization upload completes, you’ll see “Uploaded to Apple” in the organizer, then after a few minutes you’ll get an email and Xcode notification that your app is “Ready to distribute”. In the righthand panel underneath “Distribute App”, you’ll see that the “Export Notarized App” button is enabled and can be used to place the signed package anywhere on your Mac for further processing.

In our case, we had to split up the build scripts into two parts: previously we had a single script that did the build, signed it with the Developer ID, and then created an appcast. Sparkle’s XML file is now created with a separate script that also prepares the release to be checked into our repositories.

One final note: these instructions are based on Xcode 10, which is currently the only development tool that can be used to submit an app for notarization or the Mac App Store. Before we figured that out, we found that Xcode 11 does a better job passing along the -o runtime flag during a framework’s Code Sign On Copy. It’s likely that all this work you just did will only be needed for a few months. Sigh.