What NCM can do for local models and DiLoCo/DisTrO parallel training on home PCS
NCM is a fascinating study, the applications can be super promising. The paper main results revolve around how NCM has an edge over competitive models around 725m resulting in 21X faster response with 4-5x fewer training tokens per DLM models.