drumih/turbo-fieldfare
TurboFieldfare runs Gemma 4 26B-A4B on 8 GB Apple Silicon Macs by keeping only about 2 GB resident and streaming experts from SSD.
The Swift and Metal runtime keeps a 1.35 GB shared core and FP16 KV cache in memory, then loads the needed MoE experts per token. The project includes a Mac app, CLI, streaming installer, decode service, and experimental loopback OpenAI-compatible server. Reported decode speeds are 5.1-6.3 tok/s on an 8 GB M2 MacBook Air and 31-35 tok/s on a 24 GB M5 Pro. Image input is optional through a separate 1.1 GB companion pack on M2 or newer Macs. Source: GitHub · LLM repos' note
The Swift and Metal runtime keeps a 1.35 GB shared core and FP16 KV cache in memory, then loads the needed MoE experts per token. The project includes a Mac app, CLI, streaming installer, decode service, and experimental loopback OpenAI-compatible server. Reported decode speeds are 5.1-6.3 tok/s on an 8 GB M2 MacBook Air and 31-35 tok/s on a 24 GB M5 Pro. Image input is optional through a separate 1.1 GB companion pack on M2 or newer Macs. Source: GitHub · LLM repos' note
score 5
Discussions
- hn · 282 points · 76 comments
- hn · 350 points · 97 comments
- hn · 392 points · 122 comments
- hn · 437 points · 147 comments
- hn · 476 points · 167 comments
- hn · 502 points · 174 comments
- hn · 528 points · 181 comments
- hn · 549 points · 186 comments
- hn · 566 points · 198 comments
- hn · 586 points · 206 comments
- hn · 606 points · 212 comments
- hn · 617 points · 215 comments
- hn · 629 points · 223 comments
- hn · 639 points · 223 comments
- hn · 653 points · 227 comments
- hn · 662 points · 230 comments
- hn · 668 points · 234 comments
- hn · 679 points · 236 comments
- hn · 693 points · 240 comments
- hn · 703 points · 246 comments
- hn · 708 points · 249 comments
- hn · 716 points · 250 comments
- hn · 728 points · 253 comments
- hn · 736 points · 258 comments
- hn · 754 points · 260 comments
- hn · 768 points · 268 comments
- hn · 779 points · 274 comments
- hn · 788 points · 277 comments
- hn · 800 points · 282 comments
- hn · 810 points · 284 comments
- hn · 822 points · 287 comments
- hn · 829 points · 292 comments
- hn · 843 points · 299 comments
- hn · 919 points · 345 comments