I trained a 348M-active MoE on ~11B tokens. It scores 85.5% vs Qwen2.5-1.5B's 53.3% on shuffled fact chains. GGUF + eval set — please try to break it.
I've been recently working a small sparse MoE, and I've got interesting results I'd like strangers to invalidate. Tldr; I train small models and got surprising results and seek invalidation. It's not a general assistant, and I'm not claiming it is. Edit: Trained on a single 5090.