Synapse Open Synapse
AI

Running 35B MoE models on a Tesla K80 (2014) at ~21 tok/s, in plain C

Reddit · r/LocalLLaMA · · 1 min read

I've spent about a year on an inference engine that doesn't need modern hardware. It's called Pantheon (LLM engine) on top of Foundry (ML library), both in C with no outside libraries. The CUDA backend targets Kepler ( sm_37 ), so a Tesla K80 you can buy used for very little does real work.

Latest from Reddit · r/LocalLLaMA