wici-ai/inferssd
InferSSD runs MoE models larger than your RAM or VRAM by loading experts from SSD.
Built on llama.cpp, it uses standard GGUF files and provides terminal chat, browser chat and an OpenAI-compatible API. The repository reports DeepSeek decode speeds of 2.874 tokens per second on a 48 GB M5 Max MacBook Pro and 7.270 on an RTX 5090—3.29× and 8.39× its llama.cpp results on the same hardware. Those benchmarks use a 4K prompt and measure decoding over output tokens 129–256. DeepSeek and Muse each require roughly 470 GiB of model downloads, making SSD capacity a substantial requirement.
GitHub · LLM repos' note
Built on llama.cpp, it uses standard GGUF files and provides terminal chat, browser chat and an OpenAI-compatible API. The repository reports DeepSeek decode speeds of 2.874 tokens per second on a 48 GB M5 Max MacBook Pro and 7.270 on an RTX 5090—3.29× and 8.39× its llama.cpp results on the same hardware. Those benchmarks use a 4K prompt and measure decoding over output tokens 129–256. DeepSeek and Muse each require roughly 470 GiB of model downloads, making SSD capacity a substantial requirement.
GitHub · LLM repos' note
score 5