Synapse Open Synapse
AI

CUDA+ROCm dual-backends LLM inference for mixed-vendor GPUs (up to 48% faster)

Reddit · r/LocalLLaMA · · 1 min read

I have seen potential in combining Strix Halo's iGPU with RTX 5090 for LLM inferencing. Initially I found no inference engine that can run LLM inference on both these GPUs at the same time, inference engine such as vLLM support inference on all CUDA GPUs, or all AMD GPUs.

Latest from Reddit · r/LocalLLaMA