If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:
I can run a Qwen 3.8 35b Q_8 quant (BF16 is a tad too large) using CPU offloading, while still reaching acceptable speeds for private use.
Ternary Models are a game changer. Built to be used in edge computing, Bonsai-27B-Ternary from Prism ML runs completely on my GPU (using ca. 10GB of my 12GB VRAM incl. MMProj, 9 when disabling vision) and even leaves enough VRAM to run subagents (when killing my plasmashell lol -but i tend to offload these to the CPU in favor of increasing the context of the main model). currently you have to install a costum fork of llama.cpp to run ternary models, but the needed changes will be applied upstream soon. (There is a Bonsai quant which aims to run on iPhones with below 6GB VRAM footprint - the limit of what Ios currently allows)
If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.
These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.
Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.
If you are really using those things for work, a 32GB VRAM Blackwell currently costs around 3800-4000€. My own experience with local models on my 5070/12GB with 64GB DDR4 has shown me:
If this scales linear, it would mean that the 32 GB of a smallish Blackwell would be enough to run Ternary models with around 70B parameters plus a ton of context window without breaking a sweat.
These are strange times. I absolutely love the technology and the advances, and i fucking hate the companies pusting these technologies.
Altman, Amodei, Musk, Nadella, Huang and all of the othe AI evangelists can go choke on a bag of dicks in my opinion. This technology never should have been this divisive, it should have just become another tool in humanities toolbox. I fear that these people have tainted the general populations outlook on this tech for a very long time.