Synapse Open Synapse
AI

At this point, just use the large quant your hardware fits

Reddit · r/LocalLLaMA · · 1 min read

Do anybody has the same feel? Going to lower quants, thinking models tend to think more and more. So much that the speed advantage is cancelled, or it becomes actually slower. The increase in prompt processing speed and token generation speed just doesn't make up for the longer thinking blocks.

Latest from Reddit · r/LocalLLaMA