Synapse Open Synapse
AI

"But wait... I need to check [thing that has already been reasoned though]" — an artifact of quantization?

Reddit · r/LocalLLaMA · · 1 min read

I saw this quite a fair bit with 3.8 27B Q4_K_M, which is reduced greatly by Swift. I tried a very low quant of Swift's QFN (IQ2-XXS) and the second guessing was much worse. I've read quite a bit that 27B is very thorough and that's how it achieves its high scores relative to its parameter count.

Latest from Reddit · r/LocalLLaMA