The Ultimate MoE Training Handbook: training Mixture-of-Experts models with Megatron-LM, setting by setting.
This handbook is for people who have trained a dense model and want to train an MoE next, or who are curious how MoE training works. Chapter 2 is a Megatron-LM runbook: test the hardware, build the image, train a dense baseline, then a first MoE on one node, compared against that baseline.