I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?
pizza234 5 minutes ago [-]
> but are they functionally useful or helpful to anyone currently?
Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.
MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.
The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
xnorswap 13 minutes ago [-]
I wonder what this measures in J/token.
18 minutes ago [-]
jackb4040 39 minutes ago [-]
Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!
roger_ 46 minutes ago [-]
Seeing a lot of these “run 1TB models with 1GB RAM” projects recently. Most seem vibe coded and probably won’t be maintained.
Hoping a winner emerges with some real momentum behind it.
akie 3 minutes ago [-]
No no, don't just say "vibe coded", say "Fable and $500 of credits"
hiramwen 38 minutes ago [-]
You don't really need a maintainer when codex or claude code can set it up for you; thats how I got Trellis2 working on windows and tiny VRAM despite Microsoft recommending you have 24GB VRAM and Linux. Models are pretty disposable now.
classified 32 minutes ago [-]
How a patchwork of Python modules constitutes an API(?) or whatever(?) and how to use it (does anyone?), is beyond me. There is nothing that I would call documentation, let alone concrete usage examples. I know more amusing ways to waste my time. When I want to play with models, I use llama.cpp.
book_mike 8 minutes ago [-]
We will see if this project has legs. This is the kind of efficiency we desperately need. Now if we can address efficiency with llm training.
ilaksh 38 minutes ago [-]
I guess the use case is something like: you have a slightly obsolete Mac or PC or a whole bunch of them, and just need to compose one or more convincing spam emails, but it's fine if it takes a full week to do it?
speedgoose 37 minutes ago [-]
And you don’t pay for the electricity.
cpfohl 1 hours ago [-]
I’m still slightly confused on what this adds.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
dofm 50 minutes ago [-]
If you point it at a huggingface model identifier, it will download it, I assume. No way around that.
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
pvtmert 31 minutes ago [-]
Not an expert in this field, but the "expert" is consisting of multiple layers. To keep it small in terms of memory print, this project streams each layer (dividing even further).
It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity.
Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed.
I assume it's horribly slow, but can be put in a non-intrusive background task...
cpfohl 21 minutes ago [-]
Obviously needs downloading eventually :).
It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.
IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token
https://github.com/lyogavin/airllm/releases/tag/v3.1.0
Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.
MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.
The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
Hoping a winner emerges with some real momentum behind it.
Let’s say I wanted to run a full size open weight model. I have a 128GB m3 max laptop.
Does this basically load layers in and out on demand? So I still have to download the full model to disk, but the RAM requirements go way down? The readme calls out that one still needs to connect HuggingFace, which leads me to believe that maybe you don’t even need to download the full model?
It reads like it is keeping only the core and the active layer loaded at any one point, and streams layers from disk; there are several other solutions like this and if my understanding is right, this is probably better than an mmap implementation or just streaming experts in.
It also requires extra space because of decomposition of the layers. Normally the file format optimized for compute intense workloads. But here the bottleneck is the memory capacity.
Also guessing that you need to be able to hold at least 3-layers at once in the memory, given M x N = R operation, M is the previous layer, N is next, and R is the result. on the next "layer", the R (result) becomes M, gets computed against the next layer, N, yielding the further result R'. And so on, until all layers are processed.
I assume it's horribly slow, but can be put in a non-intrusive background task...
It seems like this tool saves on both disk space and RAM, then. Classic trade off: speed vs space.