Synapse Open Synapse
AI

35B MoE running at 20.5 tok/s on an iPhone, without skipping experts

Reddit · r/LocalLLaMA ·

I came across OpenBMB’s engineering write-up on running large MoE models locally, and the systems work behind it is worth highlighting. Instead of shrinking the model or skipping experts, they tackled the real bottleneck: getting expert weights from flash storage to computation fast enough.

Latest from Reddit · r/LocalLLaMA