I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)
Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it.