Synapse Open Synapse
AI

2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU

Reddit · r/LocalLLaMA · · 2 min read

Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big, but maybe interesting approach that can be used in some of the cases.

Latest from Reddit · r/LocalLLaMA