"The results showed that these GPUs can still deliver significant compute power at a fraction of the cost of newer models, making them attractive for budget-conscious users." -Mistral AI
16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.
This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length?
Do you have a set of models you'd like me to look at?
That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower looks like a reasonable target to me.
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
Similar interest here, possibly including if qwen 3.6, Gemma4 or DiffusionGemma (with the largest quants that will fit in a single card) will offer, say,
50 tokens-per-second (fast enough for interactive human-in-the-loop code research, print-f iterations on code to debug things, etc; or let the LLM churn on a problem for a minute while I step out to handle something else), context of up to 200k preferred.
Also if nothing else the below project lets you use an NVidia graphics card as low-latency swap, which has been nice as a buffer as RAM prices remain high and leaves me eyeing that 24GB card you mentioned as an alternative...
Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.
Darn, I was hoping to see bc-250's (aka PS5 chips) in there. They've recently become popular for inference and they are only about $200 on ebay. They hold a special place in my heart because I deployed 20k of them and I'm glad to see they are finding a purpose now and not just e-waste.
"The results showed that these GPUs can still deliver significant compute power at a fraction of the cost of newer models, making them attractive for budget-conscious users." -Mistral AI
Would it possible to stack up to 16x32GB VRAM, and test the performance of a MOE model such as Deepseek-v4-flash?
16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.
Have you tried 27B class models like qwen3.6?
This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length?
Do you have a set of models you'd like me to look at?
That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower looks like a reasonable target to me.
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
Similar interest here, possibly including if qwen 3.6, Gemma4 or DiffusionGemma (with the largest quants that will fit in a single card) will offer, say, 50 tokens-per-second (fast enough for interactive human-in-the-loop code research, print-f iterations on code to debug things, etc; or let the LLM churn on a problem for a minute while I step out to handle something else), context of up to 200k preferred.
Also if nothing else the below project lets you use an NVidia graphics card as low-latency swap, which has been nice as a buffer as RAM prices remain high and leaves me eyeing that 24GB card you mentioned as an alternative...
https://github.com/c0deJedi/nbd-vram
Great read. I'd love to know more about how power consumption changes as cards get newer too!
Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.
Darn, I was hoping to see bc-250's (aka PS5 chips) in there. They've recently become popular for inference and they are only about $200 on ebay. They hold a special place in my heart because I deployed 20k of them and I'm glad to see they are finding a purpose now and not just e-waste.
Wow holy crap this is news to me! I will have to consider picking some of these up for testing, what is it like working with them?
there is a discord server for fans of bc-250... lots of information there.
Interesting! I had only heard of them as cheap gaming boxes. Didn't know they were being used for cheap inference, too, but it makes sense.
> They hold a special place in my heart because I deployed 20k
Sounds like something I'd love to hear more about if you can share
ethereum mining, long shut down...