The SSD as the poor man's RAM

Offline AI rarely fails because of the processor. It usually fails because of memory, and memory, as anyone who has spent an evening in Apple's configurator will know, is the most expensive ingredient in the recipe. If you want to run a serious language model on your own machine, without every question taking a round trip across the Atlantic, you end up sooner or later working out what your privacy is worth in gigabytes.

The open-source project edge0 suggests a different route, and although I have not tried it myself yet (I will, more on that below), the direction is interesting enough to write down.

The actual problem

At its core, a language model is a very large table of numbers, the so-called weights, and traditionally the whole table has to sit in memory before the model can answer anything. The larger edge0 variant still takes up around 23 GB even when heavily compressed, which simply does not work on a 16 GB Mac and only works in theory on a 24 GB one, provided you are not planning anything extravagant alongside it, such as a browser.

Many specialists, few at work

The models edge0 relies on are so-called Mixture-of-Experts models, MoE for short. Think of a crew of tradespeople, a carpenter, an electrician, a farmer, a scientist, of whom only a handful are needed for any given job: if you want a picture hung, you need the carpenter, not the farmer. (In reality the experts inside the model are less neatly sorted, they are learned sub-networks without a clearly defined trade, but the principle holds.) The Qwen model behind the larger variant has 35 billion parameters, of which only around 3 billion actually do any work at each step.

Until now, all of them still had to stand around in the living room, which is to say in memory, including the ones with nothing to do. edge0's idea is obvious once you have heard it: the idle ones wait outside in the van, which is to say on the SSD, and are called in when needed.

Three tricks

SSD streaming. The expert weights stay on the SSD and are read on demand. Memory use depends on what is currently working, not on the total size of the model.

Prerouter. The trouble with the van is the walk to the front door. So that the model does not have to wait at every word for the right tradesperson to come in, a small, purpose-trained add-on predicts one step ahead who will be needed next and calls them in early. According to the project, this adds up to 59 percent more speed when generating text.

Recover-LoRA. Compressing to 4 bits costs quality, which is known and unavoidable. edge0 therefore places small additional weights on top of the compressed model, trained to behave as much as possible like the uncompressed original. The base model itself stays untouched.

What the numbers say, and what they do not

On a Mac mini M4 Pro with 24 GB, the README reports just under 15 to just under 18 tokens (roughly: word fragments) per second for the 35B variant, with 2.9 GB of active memory. The smaller 8B variant manages around 24 to 25 tokens per second with about 1 GB. The quality loss compared to the uncompressed original averages 3.9 points (35B) and 2.8 points (8B), on a scale of 100.

That sounds good, and a few things are still worth reading alongside it:

• The quality measurements were run by the team itself. They may well be right, but nobody independent has confirmed them.
• The 2.9 GB is the memory the framework reserves for itself. The second run is faster because macOS keeps the experts it has read in its file cache, which lives, as you may have guessed, in memory. On an 8 GB Mac I would lower my expectations in advance.
• Longer conversations need extra memory; the README gives around 3.3 GB for the 8B variant at roughly 3,300 tokens of context.
• The project is very young, the models are explicitly labelled "preview", and for now it only runs on Apple Silicon. A CUDA version has been announced.

Why I like the direction anyway

Whether edge0 itself catches on, I do not know, and to be honest it does not matter much to me. What matters is that someone is seriously working on decoupling large models from memory, because that shifts the question of who can run AI locally from a RAM budget to SSD space, which is considerably cheaper (even if Apple is not exactly shy about SSD upgrades either). A model running on a Mac mini in the office sends no customer data abroad, charges nothing per request, and keeps working when the train's internet once again decides to spend the tunnel reflecting on first principles.

The fact that edge0 comes with an OpenAI-compatible interface is a detail that is easy to skim past and matters a great deal in practice: existing tools can be pointed at the local server without rewriting anything.

My own test

I'm on it.