All software

colibri-swift

A Swift and Metal port of colibri for Apple silicon: mixture-of-experts models larger than the machine's memory, with the experts read from disk as the router asks for them. A command-line tool and a Swift package.

2026SwiftMetalmacOS

colibri-swift runs large language models on a Mac, including mixture-of-experts models bigger than the machine's memory. It is a port of colibri (external link, opens in a new tab), a C engine whose idea is simple: the dense weights (attention, norms, embeddings, the router) stay in memory, and the routed experts are read from the model file on demand, through a cache sized to what the machine has. The port keeps that idea and upstream's GGUF path, and runs it on the CPU or on the GPU through Metal. The post Porting colibri to Swift tells how it was done and how it compares with llama.cpp.

It reads Qwen3 (dense, 0.6B to 32B), Qwen3-30B-A3B and Qwen3-Coder-30B-A3B, and OLMoE, in fifteen weight types, which cover the common Q4_K_M, Q5_K_M, Q6_K and Q8_0 files; split models open from any shard. On a 64 GB M4 Max it ran Qwen3-235B-A22B, a 79.8 GB file, entirely on the GPU: 19 tokens a second on the prompt and 3.1 generating, where llama.cpp with every layer on the GPU runs out of memory.

Getting it

It needs macOS 15 or later and the Swift 6 toolchain, and nothing else: no package dependencies, only Foundation and Metal. Metal needs Apple silicon; elsewhere the CPU backend runs alone.

git clone https://github.com/bkarak/colibri-swift.git
cd colibri-swift
swift build -c release          # .build/release/coli-swift
swift test

Using it

coli-swift info --model Qwen3-4B-Q4_K_M.gguf
coli-swift generate --model Qwen3-4B-Q4_K_M.gguf --prompt "What is the capital of France?"
coli-swift chat --model Qwen3-4B-Q4_K_M.gguf
coli-swift generate --model Qwen3-30B-A3B-Q4_K_M.gguf --prompt "..." --expert-cache-gb 8

generate answers once through the model's chat template, chat keeps a conversation and reuses its cache across turns, and bench measures the prompt and generation speed. For a mixture-of-experts model the experts stay in memory when they fit and are streamed when they do not; --expert-cache-gb sets the cache. The backend is Metal by default, falling back to the CPU when Metal cannot load the model.

As a library

The same engine is a Swift package, tagged 0.1.0. Depend on its Colibri product:

// Package.swift
.package(url: "https://github.com/bkarak/colibri-swift", from: "0.1.0"),
// in the target
.product(name: "Colibri", package: "colibri-swift"),
import Colibri

let engine = try await ColibriEngine.load(modelPath: "/path/to/Qwen3-4B-Q4_K_M.gguf")

for try await text in engine.respond(to: [ChatMessage(role: .user, content: "Hi!")]) {
    print(text, terminator: "")
}

let chat = ChatSession(engine: engine, system: "Answer briefly.")
print(try await chat.reply("What is a mixture of experts?"))

A reply arrives as an AsyncThrowingStream of text. Generation runs on the engine's own serial queue, never on the caller's thread, so an app can call it from any task; leaving the loop stops the generation at the next token. ChatSession keeps the history and drops the oldest turns when the conversation outgrows the context. The configuration has the command line's options and defaults. For tokens, logits or a loop of your own, the layers underneath (ColibriCore, ColibriMetal) are public too.

How it was checked

Upstream colibri is the oracle. Its C engines were built from source and run in their exact floating-point mode, and their outputs are fixtures in the repository: four real models and sixteen small synthetic ones, compared token by token. The CPU's exact path and Metal match upstream on every one of the 84 runs, and streamed experts give the same tokens as resident ones. The CPU's default path, with int8 activations as upstream's own default, is faster and matches on 70 of the 84; the README lists where it does not, and why.

Where it stops

Three model families, so far. The rarer i-quants (IQ1, IQ2, IQ3), the ternary and the FP4 types are not decoded yet, and such a file is refused at load time with the type named. On Metal, llama.cpp is still faster for every model that fits in memory, on the prompt and in generation; the post has the tables, measured on the same machine and the same files.

Source

Apache-2.0, like upstream: about 10,700 lines of Swift (the Metal kernels among them) and 550 of C, with 184 tests. Ported from colibri at commit 0a5504a. It is current software, so where the repository and this page differ, the repository is current.

From the post

Qwen3-235B-A22B, a 79.8 GB file, against the 64 GB of RAM in the machine: 3.2 GB of dense weights and a 22 GB expert cache stay in memory, and the rest of the experts are read from the SSD when the router asks.
Qwen3-235B-A22B, a 79.8 GB file, against the 64 GB of RAM in the machine: 3.2 GB of dense weights and a 22 GB expert cache stay in memory, and the rest of the experts are read from the SSD when the router asks.