Synapse Open Synapse
AI

Int8-vllm updated with Nemotron 3.5 Bolt 74B

Reddit · r/LocalLLaMA · · 1 min read

Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.

Latest from Reddit · r/LocalLLaMA