Synapse Open Synapse
Open source

Embeddings, a reranker and an LLM on one home GPU - what changed when I stopped running one container per model

Reddit · r/selfhosted · · 1 min read

My home RAG setup ran one container per model. There was an embedding server, a cross-encoder for reranking, an LLM server, and a small FastAPI service gluing them together, so five services in one compose file. Whenever two jobs overlapped, one of them got OOM-killed.

Latest from Reddit · r/selfhosted