Synapse Open Synapse
AI

Massive B70 spec-decoding improvement - Intel XMX matrix engines refactored by budiony · Pull Request #29864 · ggml-org/llama.cpp

Reddit · r/LocalLLaMA · · 3 min read

This made a huge improvement to speculate decoding speeds on my B70 running SYCL. Below examples are for Muse Glimmer Q6_K (30B) running with DFlash2. NOTE: Due to the significantly reduced cost of token verification, the n-max was increased which is why the acceptance rate is lower.

Latest from Reddit · r/LocalLLaMA