Synapse Open Synapse
AI

[Paper] Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading

Reddit · r/LocalLLaMA ·

Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path.

Latest from Reddit · r/LocalLLaMA