Apple cuts its AI image synthesis in half with a new Stable Diffusion patch

On Wednesday, Apple released optimizations that enable the Stable Diffusion AI image generator to run on Apple Silicon using Core ML, Apple’s proprietary framework for machine learning models. The optimizations will allow app developers to use Apple Neural Engine hardware to run Stable Diffusion approximately twice as fast as previous Mac methods.

Launched in August, Stable Diffusion (SD) is an open source AI image synthesis model that generates new images using text input. For example, typing “astronaut on a kite” in SD will usually create an image of exactly that.

By releasing the new SD optimizations, available as conversion scripts on GitHub, Apple wants to unlock the full potential of image synthesis on its devices, it notes on Apple Research’s announcement page. “With the growing number of apps from Stable Diffusion, it’s important to ensure that developers can effectively leverage this technology to create apps that creatives everywhere can use.”

Apple also cites privacy and avoiding cloud computing costs as benefits of running an AI build model locally on a Mac or Apple device.

“End-user privacy is protected because any data the user provides as input to the model remains on the user’s device,” Apple explains. “Second, after the initial download, users do not need an Internet connection to use the template. Finally, deploying this model locally allows developers to reduce or eliminate their server costs.”

Stable Diffusion currently hits faster on Nvidia’s high-end GPUs when run locally on a Windows or Linux PC. For example, it takes about 8.7 seconds to generate a 512×512 frame with 50 steps on an RTX 3060 on our machine.

By comparison, the conventional method of running Stable Diffusion on an Apple Silicon Mac is much slower, taking about 69.8 seconds to generate a 512×512 image in 50 steps using Diffusion Bee in our test on a Mac Mini M1 .

According to Apple’s benchmarks on GitHub, Apple’s new Core ML SD optimizations can generate a 512×512 image in 50 steps on an M1 chip in 35 seconds. An M2 does the job in 23 seconds, and Apple’s most powerful silicon chip, the M1 Ultra, can achieve the same result in just nine seconds. It’s a dramatic improvement, cutting build time almost in half in the case of the M1.

Apple’s GitHub release is a Python package that converts Stable Diffusion models from PyTorch to Core ML and includes a Swift package for model implementation. The optimizations work for Stable Diffusion 1.4, 1.5 and the new version 2.0.

Currently, the experience of setting up Stable Diffusion with Core ML locally on a Mac is aimed at developers and requires some basic command line skills, but Hugging Face has released a detailed guide for setting up Core ML optimizations. ‘Apple for those who want to experiment.

For the less technically inclined, the previously mentioned app called Diffusion Bee makes it easier to run Stable Diffusion on Apple Silicon, but it doesn’t yet incorporate Apple’s new optimizations. Additionally, you can run Stable Diffusion on an iPhone or iPad using the Draw Things app.

Leave a Comment