Synapse Open Synapse
AI

Just update of ik_llama.cpp fork and seek of feedback

Reddit · r/LocalLLaMA · · 1 min read

Continuation of https://www.reddit.com/r/LocalLLaMA/comments/1wynomx/k_llamacpp_moe_optimizations_expert_residency/ This time I finally get speeds compared to Strata for ik_llama.cpp on 6gb vram rtx 2060 ( generates about 45–48 t/s on short prompts and about 37 t/s at 28K context.

Latest from Reddit · r/LocalLLaMA