The problem with open weight models is that they are still hard to deploy. For DeepSeek for example you require NVIDIA HGX B200, these things are expensive.
If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:
I can run a Qwen 3.8 35b Q_8 quant (BF16 is a tad too large) using CPU offloading, while still reaching acceptable speeds for private use.
Ternary Models are a game changer. Built to be used in edge computing, Bonsai-27B-Ternary from Prism ML runs completely on my GPU (using ca. 10GB of my 12GB VRAM incl. MMProj, 9 when disabling vision) and even leaves enough VRAM to run subagents (when killing my plasmashell lol -but i tend to offload these to the CPU in favor of increasing the context of the main model). currently you have to install a costum fork of llama.cpp to run ternary models, but the needed changes will be applied upstream soon. (There is a Bonsai quant which aims to run on iPhones with below 6GB VRAM footprint - the limit of what Ios currently allows)
If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.
These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.
Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.
Yes, unfortunately they are, and I do think efficiency is going to be a significant research front for open-weight models. The nature of this topic is highly speculative as our compute capabilities have only recently reached what is required to consider running generative AI models, what we have today are very crude first implementations of what I personally believe will become an everyday tool for developers, and more.
And we have seen this, there are models now capable of running on an individual’s hardware (and not particularly expensive hardware either) that can outperform what ChatGPT initially launched with.
The problem with open weight models is that they are still hard to deploy. For DeepSeek for example you require NVIDIA HGX B200, these things are expensive.
If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:
If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.
These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.
Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.
Yes, unfortunately they are, and I do think efficiency is going to be a significant research front for open-weight models. The nature of this topic is highly speculative as our compute capabilities have only recently reached what is required to consider running generative AI models, what we have today are very crude first implementations of what I personally believe will become an everyday tool for developers, and more.
And we have seen this, there are models now capable of running on an individual’s hardware (and not particularly expensive hardware either) that can outperform what ChatGPT initially launched with.