Synapse Open Synapse
AI

I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)

Reddit · r/LocalLLaMA · · 3 min read

Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.

Latest from Reddit · r/LocalLLaMA