Int8-vllm updated with Nemotron 3.5 Bolt 74B
Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.
Hi, I've been maintaining a comprehensive fork of vllm that rewrites almost everything in the serving stack for int8, which is not well supported in mainline vllm.