Synapse Open Synapse
AI

I trained a 348M-active MoE on ~11B tokens. It scores 85.5% vs Qwen2.5-1.5B's 53.3% on shuffled fact chains. GGUF + eval set — please try to break it.

Reddit · r/LocalLLaMA · · 2 min read

I've been recently working a small sparse MoE, and I've got interesting results I'd like strangers to invalidate. Tldr; I train small models and got surprising results and seek invalidation. It's not a general assistant, and I'm not claiming it is. Edit: Trained on a single 5090.

Latest from Reddit · r/LocalLLaMA