Synapse Open Synapse
AI

moe-cache : LRFU/MQ-like hybrid cache, ClassLRU by sanmai · Pull Request #31233 · ggml-org/llama.cpp

Reddit · r/LocalLLaMA · · 3 min read

Replace SLRU with a cache that prioritizes frequently used experts in classes, ranked using decaying usage history. With a warm cache, I measured +6% decode on QFN and +7% on GLM versus SLRU, with 8% fewer uploads.

Latest from Reddit · r/LocalLLaMA