I've been running a weird quant of Gemma 4 26B-A4B it Heretic ARA: Q8 attention, Q4 experts. The one by AnubysIII
Intel Arc B580 (12GB) GPU, 16GB DDR5 RAM, regular desktop (so only 4/6 threads dedicated to it on CPU)
This was the benchmark with 16k context and 100 token gen from koboldcpp:
ProcessingTime: 19.392 s
ProcessingSpeed: 839.72 T/s
GenerationTime: 3.479 s
GenerationSpeed: 28.74 T/s
TotalTime: 22.871 s
It's absolutely nuts that this is possible on what a low-mid range PC can do. If I eventually stack another arc card in, it can go zoooom.
#aubreykunyap
Intel Arc B580 (12GB) GPU, 16GB DDR5 RAM, regular desktop (so only 4/6 threads dedicated to it on CPU)
This was the benchmark with 16k context and 100 token gen from koboldcpp:
ProcessingTime: 19.392 s
ProcessingSpeed: 839.72 T/s
GenerationTime: 3.479 s
GenerationSpeed: 28.74 T/s
TotalTime: 22.871 s
It's absolutely nuts that this is possible on what a low-mid range PC can do. If I eventually stack another arc card in, it can go zoooom.
#aubreykunyap
