I wish we could stream active data to RAM, directly from the NVMe drive for the 2TB K3 model. Can't wait for prism-ml/ to release a ternary 2bit model, which would make this a bit more tenable.
You can, but the reported speeds for such big models were in the range from a token per a dozen seconds to slightly better than 1 or 2 tokens per second.
A 2 bit model is still 700gbs you need to stream through.
I know we say reading is free (vs writing). But on a 2tb drive, you're doing a full drive read for every three tokens. Thats 333,000 drive reads just to fill up the context. Well, this is at least an moe model so not that terrible. But i think the point remains
I am currently paying for multiple models that are open weight.
Choosing open models is more about long term availability and avoiding censorship than being free. For 90% of us it doesn't make sense to invest in hardware for $5-100.000 to run these models when API prices are this low
I buy DeepSeek and MiMo directly from them, on the assumption they're better at it, have a vested interest in me having a good experience and they'll get caching and other stuff right. They're also cheap enough that it's unlikely you'll find a much better deal on those models.
I'm likely to also add a small Kimi Code subscription, as the model looks very promising and I don't see any reason to support proprietary US models from companies I don't really trust overopen Chinese models. I've opted not to get a z.AI subscription, though, as the price/performance ends up not being great, because GLM chews a lot more on the problem so its actual price per task is roughly the same as the big guys. The same can't be said of DeepSeek. It's notably cheaper per task, especially when comparing API rates to Anthropic or OpenAI.
To anyone who is unfamiliar with LM Arena, this isn't a synthetic benchmark but basically an A/B test.
Human users submit the same prompt to two models and then pick with output they prefer.
Also very good for free image gen, for what it's worth. You just enter the same prompt 4 or 5 times and eventually it goes to a good model.
> same prompt 4 or 5 times
LM Arena permanently flagged / sent to the captcha hell my account after trying the same prompt twice. So, YMMV.
I wish we could stream active data to RAM, directly from the NVMe drive for the 2TB K3 model. Can't wait for prism-ml/ to release a ternary 2bit model, which would make this a bit more tenable.
You can, but the reported speeds for such big models were in the range from a token per a dozen seconds to slightly better than 1 or 2 tokens per second.
A 2 bit model is still 700gbs you need to stream through.
I know we say reading is free (vs writing). But on a 2tb drive, you're doing a full drive read for every three tokens. Thats 333,000 drive reads just to fill up the context. Well, this is at least an moe model so not that terrible. But i think the point remains
Welcome to the future. I think open weight models are our only hope for LLMs being net positive for society.
unless they figure out how to make money by themselves, closed models already capurd 99% of reaturly
I am currently paying for multiple models that are open weight.
Choosing open models is more about long term availability and avoiding censorship than being free. For 90% of us it doesn't make sense to invest in hardware for $5-100.000 to run these models when API prices are this low
I buy DeepSeek and MiMo directly from them, on the assumption they're better at it, have a vested interest in me having a good experience and they'll get caching and other stuff right. They're also cheap enough that it's unlikely you'll find a much better deal on those models.
I'm likely to also add a small Kimi Code subscription, as the model looks very promising and I don't see any reason to support proprietary US models from companies I don't really trust overopen Chinese models. I've opted not to get a z.AI subscription, though, as the price/performance ends up not being great, because GLM chews a lot more on the problem so its actual price per task is roughly the same as the big guys. The same can't be said of DeepSeek. It's notably cheaper per task, especially when comparing API rates to Anthropic or OpenAI.