Synapse Open Synapse
AI

Improve Qwen3.8-27b low quants by using a prefiller model that streams from SSD

Reddit · r/LocalLLaMA · · 1 min read

This demo from ISTA-DASLab recovers low quant accuracy while retaining low VRAM usage and fast decoding. It needs a Nvidia Blackwell GPU to run, so I have not tested it. For context longer than ~4k there should be no speed penalty compare to running without the prefiller model.

Latest from Reddit · r/LocalLLaMA