Optimizing Stable Diffusion Models for Core ML and Apple Silicon

Today, we are releasing optimizations to Core ML for Stable Diffusion in macOS 13.1 and iOS 16.2, as well as code to get started with deploying to Apple Silicon devices.

Core ML is a machine learning library that enables developers to create apps that use image generation from text prompts.

The community has built an expansive ecosystem of extensions and tools around this core technology in a matter of weeks.

Getting to a compelling result with Stable Diffusion can require a lot of time and iteration, so a core challenge with on-device deployment of the model is making sure it can generate results fast enough on device.

This requires executing a complex pipeline comprising 4 different neural networks totaling approximately 1.275 billion parameters.

To learn more about how we optimized a model of this size and complexity to run on the Apple Neural Engine, you can check out our previous article.

This release comprises a Python package for converting Stable Diffusion models from PyTorch to Core ML using diffusers and coremltools, as well as a Swift package to deploy the models.