Show HN: Getting GLM 5.2 running on my slow computer
github.comA few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me.
But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility.
I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context. How it responds in int4 and whether the quality is maintained or not. Until I got to the point, on my computer with 32GB of RAM, I was able to communicate with GLM 5.2 with times that, of course, aren't high in cold start, but even then, we're talking about 0.1 tok/s, but that wasn't important to me. The important thing was the journey to reach this goal. I just wanted it to work at all costs, even slowly.
So I created Colibrì, which was born from a very simple idea, to be honest, but tested in every way, where a 744B Mixture-of-Experts model activates only ~40B parameters per token—and only ~11 GB of those change from token to token (the routed experts). So:
The dense part (attention, shared experts, embeddings—~17B params) stays resident in RAM at int4 (~9.9 GB); The 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
The engine is a single C file (c/glm.c, ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU.No GPU or serious hardware because I don't have that hardware so I can't test it on hardware that is more powerful than my computer.Colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home.
Any feedback is welcome! (and if anyone wanted to participate in the project I would be delighted)
I have prototyped something similar with ollama some months ago.
Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?
I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.
EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so
My main question is whether when put into practical use, this can be measured in tokens/second, or more like 1 token per minute... I have seen locally hosted LLM that are as slow as 1 tok/second still be very useful if you give it a project to do something overnight and metaphorically walk away from it, check back with what it has done in 6 or 8 hours.
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
edit: I think this is a fantastic project in general concept, and look forward to seeing more efforts towards the general idea of being able to run a 350B to 900B size model locally, even if as slow as 1 tok/s, on hardware that ordinary people can afford. Anything along the general concept of "we have fast read NVME SSD storage, we have a big ass model on local disk, we'll read it at 11GB/tok as we need it, not try to load the whole thing".
In the readme you can see benchmark which everyone with different hardware is running Colibrì, and I have to say I've seen great times! I'm always doing more to improve!
I have a 16-core system with 256GB RAM here I could try it with but regretfully it's so old the CPUs aren't AVX2 capable. Otherwise it makes a fairly good llama-server test system for CPU only stuff. Oh well. Time to upgrade (painful to the wallet these days).
Maybe we can see some integration!
If you get good at extracting remarkable performance from the most lesser of instruments enough to pull their own weight regardless, just imagine what it can be like when such a practitioner gets behind the keyboard of a world-class Steinway. And just does what they do best. Without ever having touched such a capable instrument themself.
On a level playing field the expression of virtuosity can outshine those who have never known any instrumental limitations at all :)
When pulling way more than your own weight happens like for few others.
There should be an award for getting the most out of the electronics rather than trying to reach orbit by building the tallest pile of e-waste.
First Prize right before your eyes !
Grande praise !
And just starting to ascend toward an unconquered summit that others find forbidding ;) Or they find uninteresting since the limit naturally lies on firm earth somewhere below the stratosphere.
Thanks for kind words!
Agreed!
For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage.
But it's about the journey not the destination. My current running local LLMs train of thought...
> on hardware that ordinary people can afford
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
The very boring pair of two 16GB ddr5 6000 I had in my newegg shopping cart went from $399 to $475, so increasingly the answer will be "no".
Maybe that's a measure of the self-fulfilling dollar incentive toward "renting" someone else's RAM in the future rather than trying to actually own such an outlandishly luxury item :\
I bought whole Intel N100 mini pc with 16GB of DDR5 in it in 2023 for $AUD289 (so about $US200). I got a 16GB (DDR4) SODIMM in 2022 for $AUD88 ($US60).
Does it have to be DDR5? Is the limit RAM speed, or SSD speed?
I was just using that as an example of constant on going price rises, it was the most mundane and not particularly fast ddr5 6000 stuff. The 6400 is even more ridiculous.
Ideally this engineer's approach will yield better performance on lesser equipment in the future, if they keep up the good work after they get more-capable gear to experiment with as time goes by :)
Maybe not afford new, but they probably already had it from before the current crisis?
After 18y of thinkpads, this year I bouth a Lenovo yoga for... Cheap (1000€).
32G RAM, nvme 1TB, core ultra 258V.
Looking at the prices now... Wow, was I lucky.
Tried some of the 7b models locally, more than usable, around 30token/sec, not with the NPU, but using the ARC integrated GPU.
I am a noob for this, but I guess it's time to experiment more with this local setup
I’ve been wondering if chat is the wrong interface for slower local models (and some projects) and maybe something like a ticket system is a better fit. I just decided how I would test this idea on my available hardware before I go drop money on a Mac Studio or GPUs. I’ll probably have a POC this week. There is nothing novel here, just need to spend the time to get it working for me.
Having a thin python/ts orchestrator and workers that pick up tasks from the directories like events and decide whether to make deterministic calls and wait is pretty standard albeit custom way of doing things in this space where you're bottlenecked by the concurrent call your workers/agents can make.
The hard thing is always keeping complexity low and being ZeroOps.
Are there any frameworks/scaffolding/harnesses or general resources on this you can share? I’d love to learn more
Most any ticketing system can integrate with ordinary IMAP and smtp email flow, so you can really use any agent that can "do" inbound and outbound email to talk to a self hosted ticket queue.
I no longer use a harness directly, instead I use Github issues/Linear to work on multiple tickets in parallel while the agents are doing work:
https://github.com/skorokithakis/symphony
This is actually really smart. It would be like working with a team of humans.
I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration
Use the ticket system built into mininote.ink 's mcp server. Works perfect right out of the box. Also great notetaking app.
Docs:
https://mininote.ink/docs/mcp-docs
So I’ve been thinking about this problem a lot, specifically as it relates to running LLMs at home, and I’ve been using GLM-5.2 to make an SMTP/IMAP-to-LLM gateway.
https://tangled.org/clee.sh/posthorn
I’ve been wondering about something similar - a system that enforces (or does the heavy lifting) of dividing a large task into smaller sub-tasks so that it’s easy to run/check/test each one independently - even on a fresh model instance if needed.
This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.
The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
How was qwen3.6 launched?
The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples:
- Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec.
- Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue.
- Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture.
I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me.
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
> OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
> I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
What could be more valuable than outputting the exact thing you asked for?
Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028.
Expectations seem to be rising at a faster rate than models can improve.
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models?
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
I realized I didn't answer the CPU question, as a very quickly chosen example from eBay, there's a Dell R740XD with two Xeon Gold 6254 CPUs, 768GB RAM for sale for something like $5799 USD right now. I'm sure if I put some more time into it I could piece together something with a full terabyte for around the same price. Or faster/better CPUs, more core count CPUs by buying the system with no RAM, or minimal RAM (64GB) and then adding the DIMM kits from the more reputable refurb server part vendors on ebay.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Even though LLM benchmarks are very opinionated, I would really like to see some numbers for the setup parent suggested. From what I read elsewhere, anything below $40K in HW costs is not worth the effort for coding models locally.
Would be nice if you could somehow connect GPU-levels of parallel floating point cores to that amount of memory. I guess that's what the big AI datacenters are doing, but how can we do that on a budget?
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.
This is exactly the kind of technology that I expect Apple to ship anytime soon given the RAM prices and their HW/SW integration skills:
- ship super fast SSD (tbh they are already top notch)
- add a specific cache layer for tokens
- keep the amount of unified memory reasonable
Working on something similar targeting macOS on Apple Silicon, Unsloth split GGUF, compressed partial residency in unified memory (would make more sense on 128GB instead of my 64GB...), native Metal kernels, and RAM-only native compressed KV. Happy to put on GitHub when it's ready.
This is the way.
Followed you on GitHub to get notified when you are!
I will be delighted to try. I have a 128gb macbookpro m4 waiting for this.
I've taken a similar strategy w/ image/video gen at https://github.com/cretz/thinfer (see video branch for a ton of work).
Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
Wow, I see you managed to fit in so many models (krea, wan, hunyan, etc.). Did you get to build a common harness to run all of them? Which ones stay under your VRAM budget more consistently?
All stay under because I had Claude build the workflow to respect it (text encoding, denoising, vae, etc), there's just a tiny bit of untracked pieces. While there are common interfaces to invoke them (CLI and API/webpage) and they share ops and some pieces, lots of model logic is unique. This is all vibe coded and surely has inaccuracies.
I was actually just working on the same thing as this, but I went down the route of mmapping the entire model into memory to avoid the extra ram usage. I also had Claude implement Medusa[1] on the model to try and avoid loading an additional model into memory but still get the benefits of MTP. Currently at a stop light so I can't list everything and I didn't get to read your full post either yet.
To expand since I just got home, I'm making all of my modifications to llama.cpp, the goal was to eventually put this on a SBC of some kind with an nvme to handle the mmapped files. I think the theoretical limit of my current setup is about 1.8 tok/s based on prior testing but that is also with the additional medusa heads not fully trained (I honestly don't know if the counting it's generated tokens or not.)
In the end it seems like the idea we had is similar, I just don't know how to write an llm parser/runner from scratch yet and instead of specifying what needed to stay in memory I just let the linux kernel handle it.
Oh last note, I also capped llama.cpp usage to 16GB of my 32GB, so it might be possible to get it down even lower.
[1] https://arxiv.org/abs/2401.10774
if you like, colibrì always needs to improve so if you have ideas or anything else you are welcome for pull request issues and also benchmarks!
Yeah I'll see what I can transfer over from my llama.cpp work. As before I'm not too experienced with llm work, but I have a lot of experiments I'm trying out. So I'll make a PR if I get any interesting results.
This is the approach I was wondering about.
Pretty cool! I've also been playing around with GLM 5.2 this week and was equally impressed. At work we're running it locally on some crazy expensive hardware as a test before starting another project so it's great to see people taking this massive FOSS model release and running it on an average machine, even if it's not terribly practical at this point.
Nice work!
Really thanks!!
which hardware?
llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)
Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?
I am curious if it's possible to adjust this to use more RAM, as i've got a machine with 64GB RAM and 24GB VRAM. Or perhaps I could run Gemma/Qwen on the GPU and have GLM-5.2 delegate smaller tasks to it. It might take some retraining of GLM-5.2
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
That's possibly a good idea! We can work on it!
I also just edited my comment with more ideas in the beginning, sorry
> OS page cache can generate writes
Is this a hallucination? What am I missing? Why would heavy reads generate writes?
Spilling
Wouldn’t turning off swap fix this issue?
Better to just change swapiness?
https://askubuntu.com/questions/103915/how-do-i-configure-sw...
> Is this a hallucination? What am I missing? Why would heavy reads generate writes?
I take it heavy reads means more stuff goes into RAM, meaning other stuff has to be cached?
I've got same question as GP: e.g. is there a way to set moderately fast consumer NVMe SSDs (I've got both a Samsung 990 Pro and a WD SN850X) in a complete read-only mode to prevent "wear"?
Good catch! Disk reads do generate writes to cache. But the cache itself is in RAM, not on disk. So it shouldn’t cause additional wear of SSD.
I have a small laptop. If you have more disks available, you could really do some testing. When you have some benchmarks, submit a pull request or issue so we can maybe work on them. We are really happy for contribute!
I have epyc 9654 ES and a 7900 XTX. I was running the numbers, and even if I maxxed out the ram to like 12x32 gig sticks, it would cost me thousands more and I could only run GLM-5.2 at a couple tokens per second at q3. So this project is very promising because it suggests I could get pretty high speed and this CPU/motherboard combination suggests I have a lot of pci bandwidth that is unused.
I think another route might be looking at holding an even larger chunk of model weights in ram, and taking advantage of RAM<->GPU bandwidth, perhaps using a PCIe 5 GPU. This was my first thought since I have dedicated GPU.
If you are using Laptop, you're looking at shared memory between the iGPU and CPU. I've also tried that route, but I have always been skeptical of killing flash with too many reads, it essentially uses SSD like it's a consumable item.
I'm going to benchmark this right now with what I have and I'll get back to you on github.
Really thanks!!
At least for NVME, it is the write cycles that are limited. Read cycles are non-destructive and essentially unlimited.
The page has an SSD wear warning [0] I use desktop PCs that I build from components so I can replace the SSD, but what do users with soldered SSD do? Just avoid these applications or forge ahead disregarding the possible early burnout of their storage? They must use external storage as the burner SSD.
[0] https://github.com/JustVugg/colibri#ssd-wear-warning
Yes, avoid.
Laptops with soldered in SSDs should definitely monitor their usage and take care with this.
This project seems more of an experiment than something everyone should run, but pretty cool nonetheless
Thanks We're working on it!
AppleCare.
Even under AppleCare this is a $400 service which for an older macbook costs almost as much as the whole thing. And without Applecare it's not worth fixing at all.
Probably yes, use an external drive for that sort of thing
From what I understand, the warning is about swap-out during heavy memory use.
You don't need to be superstitious here: disk activity, including writes in particular, can be measured. E.g. `iostat` or `vmstat` on Linux.
Yes accurate!
It's a very conservative warning. The application does not perform writes, so the application doesn't actually wear your SSD at all. The rest is just application-independent general hygiene.
This is the hacker spirit
Thank you so much, it's true! It all started with this spirit!
I'm curious but don't know much about the internals of LLMs - could you use a similar architecture with other models that have "layers"? I mean, could you have one layer do its work, then remove that layer from RAM, load the next layer from disk, and have that layer activate on the result of the first layer?
The best ideas are the ones that seem obvious. This is one of those ideas.
So, could larger models work this way too?
“answering correctly on a machine that costs less than one H100 fan”
really love such comparison.
How much time is spent interfacing between userland and the kernel? Can you try to get it to run as a kernel module? :)
Also in case your CPU is old enough, did you try disabling CPU bug mitigations?
Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?
You could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.
Ahh, I remember hearing that before, and it makes general sense too.
I have a similar question and I’m inferring the answer is no - look at the cache hit rate of 23% for the 128GB M5 Max. I had previously assumed that the 40B active meant that a set of layers was chosen as THE expert for a given prompt and generation was then limited to those layers until complete. But in that case you’d have expected the expert caching to have a super high hit rate once you had enough RAM to hold an entire expert’s worth of layers.
I you look at https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
This sort of thing is a lot of fun.
I've been going smaller.. I have a custom-quantized Rust port of DiffusionGemma (26B) that seems to perform better (in responses) than benchmarks seemed to indicate and reasonably fast for its model size. Works really well on a 36GB mac as well for both prefill and generation.
It's been interesting learning about the balance of factors for performant metal kernels on unified memory.
Should have a repo up on github in the next few weeks.
I'd be interested in seeing this when you put it up! I've managed to get upwards of 300 tokens a second somewhat consistently on the desktop I already had gotten for gaming/personal project stuff several years ago (64 GB RAM, Ryzen 9 7900 X 12-core; I already had a fairly large m.2 SSD and a Radeon 6900 XT to plug into it), but only by severely compromising on the model (a custom quantized Qwen3.5-27B-Q2_K which someone published on huggingface) so that I can use a context size large enough that interfacing through opencode doesn't manage to run out of context while summarizing for compaction and then trigger a second "nested" compaction that instead just seems to cause it to lose all context and ask for a full new set of instructions from scratch. If yours ends up being anywhere as close to the one I've been using in quality, and it works on a 36 GB Mac, it sounds like it would be worth it for me to try out!
(For clarity, I have almost no idea what pretty much any of the terminology for models even means or how it translates to what the actual experience is. My strategy has mostly been using the free tier of OpenCode Zen to ask it stuff about what models and llama.cpp configurations to use in the hopes that I could maybe bootstrap something halfway decent to use locally, and so far what I've managed to get running is mediocre but at least passable)
I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
It’s a good question.
In theory MPI could distribute experts across nodes. In practice, for small clusters the added network latency usually hurts more than it helps.
Better suited for big clusters with fast interconnects. For now we're focusing on single-machine speed (caching, GPU hybrid, etc.).
I just learned about Gemma4.pas at the beginning of this week. Now this. This make me wonder how can inference engines could be built that easy. I'm not knowledgeable in this, but I thought it would take very deep Mathematic and system level knowledge, ... and a lot of patience.
If I understand correctly the basic structure hasn't changed since Transformers were invented. It's just gotten bigger and the models have had better training and also some optimizations. I don't think this one has optimizations.
I love it but where do you find that NVMe SSD for less than the price of an h100 fan let alone the memory
NVMe SSD prices had being gone down in price for a while, and the spikes are actually a lot more recent than you might think. From double checking my Amazon history, I bought my wife a 2 TB NVMe SSD for $160 back in November; it's now listed at three times that. I imagine that a lot of people just have them already from the past few years.
This is something that would benefit from Intel Optane memory. Too bad it was killed at the time.
Maybe some from intel can read and we can try? :)
This is technically impressive, but is it usable in practice?
I wonder if you could replicate this in a Colourful GeForce RTX 50-series GPU, they ship it with 2 NVMe drive slots.
I'd love to! Right now I only have a very consumer-grade computer that I've had fun with! We'll see!
Nice, looks good. This would mesh really well with the unified system memory on apple silicon
I love seeing that kind of tinkering
Really thanks!
Is this similar to fastllm?
https://github.com/ztxz16/fastllm
fastllm targets the GPU, while colibri uses CPU inference only
I'd be curious about an.option that would allow glm use with a low end GPU like a 2080 ti...
was lucky enough to snag the Olares One from Kickstarter just before this whole AI induced memory chip price gouging started.
specs are Intel Core Ultra 9 275HX (24 Cores, 5.4GHz),96GB of DDR5 5600MHz RAM, NVIDIA GeForce RTX 5090 Mobile GPU with 24GB of GDDR7 VRAM, 2TB NVMe PCIe 4.0 SSD.
going to see if I can wring at least 5 tok/s.
Let us know how it turns out.
Yeah this idea makes instant sense. Very well done, this deserves a github star on concept alone.
Curious for what an MTP only result would look like, both in terms of output quality & tk/s ?!
I wonder how would a RAID0 array of either disks or even nvme improve the performance of this.
Question to the OP, have you tested this on a machine where the entire model and context fit in RAM ?
No because I have only 32gb of ram too low
I think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
Is this inspired by antirez work on ds4?
Amazing job!
Antirez is the number one!thanks really thanks!
Antirez has a GLM 5.2 branch now in dwarfstar: https://github.com/antirez/ds4/tree/glm5.2
It heavily utilizes ssd streaming from my understanding and I think he mentioned getting some semi usable speeds on a 128gb m5 mbp.
Great job, it's unique!
Thanks really thanks!
Your coding style is halfway to IOCCC. I'm just jealous though :)
Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu].
Not hijacking anything as this project is amazing.
Would this cause issues with SSD lifespan?
What causes problems is the rewriting in this case are only read while writing is the cache! However, I'm working to improve more and more and make some parts lighter!
Is it possible to run this into an agent? pi, claude code, etc..? I've only tried it with LM studio, but i'm guessing this is a bit different
We're working on it right now with a pull request that will also arrive for opencode!
You can keep the KV cache in (possibly Unified) RAM to avoid SSD writes entirely. Not sure if it would fit on a 32GB laptop, though.
README covers that
https://github.com/JustVugg/colibri#ssd-wear-warning
This is great, well done! I love seeing people run things where they weren't meant to be run.
related and possibly more general purpose https://github.com/t8/hypura
With so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...
> slow computer > 25 GB of RAM
What?
Slow means CPU only in this context.