Porting colibri to Swift: an oracle, two backends, and the price of waking a thread
November 16, 2026 · AI
Recently I bumped into colibri, Vincenzo Fornaro's inference engine for very large mixture-of-experts models. Its idea is simple, and I liked it immediately. A mixture-of-experts (MoE) model of hundreds of billions of parameters uses only a small part of itself for each token; so colibri keeps that part in RAM (the attention, the norms, the router) and reads the rest, the experts, from the disk when the router asks for them. Pure C, one file per model family, no GPU required.
I wanted the same thing as a native engine on my Mac, one I could embed in a Swift program and use to try other models, with the GPU doing the work through Metal. So I ported it; the result is colibri-swift, Apache-2.0 like upstream, tagged 0.1.0, and it has a page here on building it and using it as a library. This post is about what the port taught me, mostly about what C gives you for free and Swift does not, and about the first numbers on my machine.
What I took, and what I left
colibri today has thirteen engines and about 261,000 lines of C. I did not port thirteen engines. I took the GGUF path of two of them: qwen36 (which runs Qwen3 dense and Qwen3 MoE models) and olmoe (Allen AI's OLMoE), plus the headers they lean on (the GGUF reader, the block decoders of ggml_quants.h, the tokenizer, the sampler, the expert tiering). That is roughly 20,000 lines of C. The port came out at 10,700 lines of Swift, 545 lines of C (I will come back to those) and 5,100 lines of tests.
The families went in one at a time behind a small registry, one Swift file per family, so the next one is a file and not a refactoring. The CPU backend is the reference; the Metal backend is tested against it.
The oracle comes first
How do you know that a port of a numerical program is correct? You cannot read the output of a language model and decide; a model that is slightly wrong still writes fluent English. The answer is an old one. William McKeeman wrote about differential testing in 1998 (if I remember correctly, for C compilers at Digital): run two implementations of the same thing on the same input and treat any difference as a bug until proven otherwise.
So before porting a single forward pass, I built upstream from source and ran it in its exact float mode (GQ_EXACT=1: no int8 shortcuts, f32 KV cache, greedy decoding, never stopping at the end-of-sequence token). Its token ids, and the logits behind every step, became fixtures. Then two kinds of models; four real ones (Qwen3-0.6B, Qwen3-4B, Qwen3-30B-A3B and OLMoE-1B-7B, five prompts each) and sixteen tiny random GGUFs that I generate with the real tensor layout of each family and every weight type that matters (four prompts each). That is 84 runs, and a script that reports the first step where my engine and upstream disagree.
Today the result is 84 out of 84 on the CPU and 84 out of 84 on Metal (24 to 32 tokens a run), including the cases where the experts are streamed through a cache far smaller than the model.
One number looks wrong, so let me explain it before you find it. Upstream's default CPU path rounds the activations to int8, and against that path the port matches 70 of 84, not 84. The reason is the rounding itself. The floats that go into it differ from upstream's in the sixth decimal (a different summation order), and a value that sits on the edge of two int8 codes falls on the other side. After that the two runs are no longer the same computation, and some of them pick a different token twenty steps later. The integer arithmetic around it is identical; I checked it against upstream's own golden blocks.
What C gives you for free
Most of the port was translation. Five things were not, and each cost me more than everything else together.
Floating point is not the same arithmetic. Clang, by default, contracts a * b + c into a fused multiply-add, one rounding instead of two. Swift does not. Neither is wrong, but they are different numbers, and an exact oracle notices. So the float code stayed in Swift, and when some of it later moved to C, that file starts with #pragma clang fp contract(off) and spells every fused operation explicitly.
The vectoriser is not the same either. The int8 dot product of the quantised path is a loop of sixteen multiply-adds. In C, with NEON intrinsics, it is one sdot instruction. In Swift, LLVM unrolled the sixteen-trip loop first and then had nothing left to vectorise, so it became sixteen scalar multiply-adds. Those 545 lines of C are mostly this (plus a small thread pool, below); kernels that decode the packed quantised blocks in registers and feed sdot, four weight rows at a time, bit-identical to the Swift version they replaced.
Threads have a price, and Dispatch charges it on every call. Upstream uses OpenMP teams pinned to the performance cores (on macOS its build has no OpenMP, so it runs on one core, which is why I compare against llama.cpp below and not against upstream). I used DispatchQueue.concurrentPerform, the idiomatic choice. Generating one token of Qwen3-4B is about 250 parallel loops of 10 to 150 microseconds each, and Dispatch parks its workers between them and wakes them up for the next. I replayed one token's matrix multiplications in C to measure it; with Dispatch the same kernels reached about 20 tokens a second, and with workers that spin between loops (and sleep after 3 ms of idleness) 31 to 34. Spinning has its own trap; with sixteen threads, the four efficiency cores slowed every loop to half. llama.cpp collapses the same way at -t 16, which made me feel a little better.
Memory mapping has a flag that costs seconds. The first prompt on Qwen3-30B-A3B waited 1.6 seconds before the GPU started. The file was mapped MAP_PRIVATE, and making 17 GB of private mapping resident for the GPU took 1.3 to 1.7 seconds, against 0.17 for MAP_SHARED. With the shared mapping, and one buffer per tensor instead of one for the whole file (so that streamed experts are never pulled in by the GPU), the first prompt went from 2.2 seconds to 0.27.
The build system has opinions. SwiftPM compiles C targets with -Os, which left my kernel helpers out of line and halved their speed. Telling it -O3 works, but only through unsafeFlags, and SwiftPM refuses a package with unsafe flags as a dependency pinned by version; so the engine could not be used as a library. The fix was to say in the source what -O3 would have done (forced inlining and explicit unrolling of the small fixed loops). With those, -Os matches -O3, and the package is now a normal dependency with a small async API on top.
Metal is a different port
Upstream has a Metal backend, but it decodes colibri's own weight formats, not the GGUF block types. So this part is new; 23 kernels that decode all fifteen weight types the port supports (from F32 to Q2_K and IQ4_XS) directly on the GPU. The first version ran prompts through the decode kernel, one token at a time, and it showed. Tiled matrix-multiply kernels (simdgroup matrices, 64 rows by 32 tokens) took prompt processing from 2,018 to 5,219 tokens a second on Qwen3-0.6B, and for the experts, a grouping step borrowed from llama.cpp's mul_mm_id (the GPU lists which tokens went to which expert, and each expert's tile runs over its own list) took Qwen3-30B-A3B from 227 to 861.
Preliminary numbers
All on my M4 Max (12 performance and 4 efficiency cores, 40-core GPU, 64 GB), against llama.cpp built from source on the same machine and the same GGUF files. A prompt of 512 tokens, then generation after it; median of three runs. These are preliminary; the CPU numbers in particular were measured on a busy machine, which I explain below the second table.
| Model | coli-swift prompt | llama.cpp prompt | coli-swift generation | llama.cpp generation |
|---|---|---|---|---|
| Qwen3-0.6B Q8_0 | 5,219 | 10,712 | 220 | 339 |
| Qwen3-4B Q4_K_M | 980 | 1,625 | 89 | 118 |
| OLMoE-1B-7B Q4_K_M | 2,840 | 4,080 | 219 | 322 |
| Qwen3-30B-A3B Q4_K_M | 861 | 1,660 | 84 | 128 |
On Metal, llama.cpp is still ahead; 1.4 to 2 times on prompts, 1.3 to 1.5 times on generation. Both are bound by reading the weights, and llama.cpp's kernels read them better and fuse more of each layer into fewer dispatches. That is the next piece of work.
| Model | coli-swift | llama.cpp, same session | llama.cpp, quiet machine |
|---|---|---|---|
| Qwen3-0.6B Q8_0 | 192 | 145 | 220 |
| Qwen3-4B Q4_K_M | 51 to 55 | 46 to 51 | 67 |
| OLMoE-1B-7B Q4_K_M | 114 | 102 | 199 |
| Qwen3-30B-A3B Q4_K_M | 45 | 30 | 69 |
The CPU was where I started furthest behind. The 4B generated 15.7 tokens a second, against llama.cpp's 67, and the sdot kernels, the spinning pool and a few smaller changes took it to 55. But look at the last two columns before you believe the middle one. Another job was running on the machine for that whole afternoon (load average between 15 and 40), and I ran the two engines interleaved, one after the other, to give them the same conditions. Under load, llama.cpp suffers more than my engine does (a spinning worker that gets preempted holds up its whole loop, and llama.cpp splits the rows statically, while mine hand them out on demand). On a quiet machine the day before, llama.cpp did 67 on the 4B; I have not rerun mine on a quiet machine yet. So the honest reading is "about level", not "faster".
And then the model the whole idea is for. Qwen3-235B-A22B, Unsloth's Q2_K, is 79.8 GB in two files; my machine has 64 GB of RAM.
| Engine, settings | Prompt | Generation |
|---|---|---|
| coli-swift, Metal, 22 GB expert cache | 19.0 | 3.1 |
| coli-swift, Metal, the 43 GB cache I started with | 19.8 | 0.9 |
| llama.cpp, Metal, every layer on the GPU | out of memory | |
| llama.cpp, Metal, 40 of 94 layers on the GPU | 8.3 | 1.8 |
| llama.cpp, CPU only | 6.9 | 5.2 |
This is the point of the whole design, so let me say it plainly. colibri does not need the model to fit in memory; it needs the dense part (3.2 GB here) and a cache of experts to fit, and it reads the rest from the SSD. So the same engine, with the same defaults, loads the 0.6B and the 235B, on the CPU or on the GPU. llama.cpp maps the file and leaves the paging to the operating system, which works on the CPU (the last row); but its Metal backend wants every layer it offloads in the GPU's working set, and with all 94 layers it stopped with an out-of-memory error. To use the GPU at all, I had to find by hand how many layers fit (40 and 56 did).
The second row is the one I learned from. Upstream gives the expert cache close to 90% of the free memory, which here was 43 GB, and the machine went into swap; the GPU spent most of its time waiting for the cache pages to come back. With half of that, the operating system's file cache holds much of what my cache does not, and generation went from 0.9 to 3.1 tokens a second. llama.cpp, which simply maps the file and lets the operating system page it, generates faster on the CPU (5.2); my engine processes the prompt about twice as fast as llama.cpp's best. And it answered the question I asked it (why the sky is blue), with the same 48 tokens on Metal and on the CPU.
Ahead in two places, behind in two
It is easy to read three tables of llama.cpp's figures as a defeat, so let me count. colibri-swift is ahead in two places. The first is the model that does not fit; it runs the 235B entirely on the GPU, where llama.cpp cannot, and processes its prompt about twice as fast as llama.cpp's best configuration. The second is the CPU under load, where it generated faster than llama.cpp on three of the four models, run for run, and level on the fourth (the 4B). On a quiet machine I do not know yet; I have not rerun it.
It is behind in two places. The first is Metal on the models that fit in memory, where llama.cpp processes prompts 1.4 to 2 times faster and generates 1.3 to 1.5 times faster. Part of that is maturity. llama.cpp's kernels read the weights at about 55% of the memory bandwidth on Qwen3-4B, mine at about 40%; it fuses more of each layer (the norms, the rotary positions, the attention) into fewer dispatches; and its prompt tiles are in half precision, mine in 32-bit floats. The second is generating with the 235B, where llama.cpp's CPU path (5.2 tokens a second) beats my best (3.1).
And part of the gap is a choice. My int8 path agrees with upstream colibri bit for bit in the integer arithmetic, which means a scale for every 32 activations and the float bookkeeping that goes with it; llama.cpp scales every 256 and folds the scales into the integer sums, which is several times less float work. I could do the same, and lose the exact agreement with upstream that the oracle depends on. For now I prefer the oracle.
Epilogue
Is it worth it? Ok, I know the obvious answer; llama.cpp exists, it is faster on Metal for every model that fits in memory, and thousands of people maintain it.
But a port is not a download. It is a fork that I now own, of a project with thirteen engines and its own pace, which I took two of. Every upstream fix to qwen36.c is now mine to notice, translate and verify, and nobody else will do it. The only reason I believe this is maintainable is the oracle; 84 runs that tell me in a few minutes whether my engine and upstream still agree. Without it, I would own 10,700 lines of plausible Swift and no way to know which ones were wrong.
And the small wish I started with? I wanted to ask a 235-billion-parameter model a question on the machine on my desk. It answered, at three tokens a second, slower than I read and faster than I expected.