Synapse Open Synapse
AI

[Boole-32B-A2.5B-preview] 9GB 1.8-bit MoE, 565 token/s on 1 RTX 5090

Reddit · r/LocalLLaMA · · 1 min read

Hi r/LocalLLaMA , we're releasing Boole-32B-A2.5B-Preview, an early 1.8-bit version of Recursive's flint . With mixed-precision quantization, expert pruning, and custom kernels (kernel changes in attached figure), we made the model 9x smaller than BF16, and 2x faster than INT4.

Latest from Reddit · r/LocalLLaMA