Synapse Open Synapse
Open source

Why we moved away from NVIDIA MIG and HAMi GPU fractions for multi-model inference on EKS

Reddit · r/selfhosted · · 1 min read

When you need to serve an LLM alongside smaller guardrails, NER extractors, and embedding models on a tight GPU budget, the standard platform advice is to slice up the GPU using NVIDIA MIG, HAMi, or Run:ai fractions. That works when you need strict hardware boundaries between untrusted tenants.

Latest from Reddit · r/selfhosted