Breaking the 1.58-bit Barrier for Ternary LLMs
BITCOS gets ternary LLM weights below the usual 1.58-bit reference by exploiting how many weights are actually zero.
The authors measured 29 ternary LLMs and found zeros can make up 51.5% of weights, breaking the equal-symbol assumption behind five-trit packing. Their layout uses a presence bitmap plus a compact sign vector, reaching 1.485 bits per weight on the sparsest model. It beat five-trit packing in 26 of 29 tested models and showed up to 1.28x kernel gains. End-to-end decode throughput improved by up to 1.18x on CPUs and 1.27x on GPUs. HN · ArXiv's note
The authors measured 29 ternary LLMs and found zeros can make up 51.5% of weights, breaking the equal-symbol assumption behind five-trit packing. Their layout uses a presence bitmap plus a compact sign vector, reaching 1.485 bits per weight on the sparsest model. It beat five-trit packing in 26 of 29 tested models and showed up to 1.28x kernel gains. End-to-end decode throughput improved by up to 1.18x on CPUs and 1.27x on GPUs. HN · ArXiv's note
score 5
Discussions
- hn · 75 points · 1 comments
- hn · 85 points · 5 comments
- hn · 107 points · 10 comments
- hn · 107 points · 10 comments
- hn · 120 points · 12 comments
- hn · 120 points · 12 comments
- hn · 130 points · 15 comments
- hn · 130 points · 15 comments
- hn · 140 points · 16 comments
- hn · 140 points · 16 comments
- hn · 145 points · 18 comments
- hn · 145 points · 18 comments
- hn · 148 points · 20 comments
- hn · 148 points · 20 comments
- hn · 151 points · 21 comments
- hn · 151 points · 21 comments
- hn · 155 points · 21 comments
- hn · 155 points · 21 comments
- hn · 163 points · 22 comments
- hn · 163 points · 22 comments
- hn · 167 points · 22 comments
- hn · 167 points · 22 comments
- hn · 171 points · 22 comments
- hn · 171 points · 22 comments
- hn · 179 points · 25 comments
- hn · 179 points · 25 comments
- hn · 184 points · 26 comments
- hn · 184 points · 26 comments
- hn · 191 points · 29 comments
- hn · 191 points · 29 comments
- hn · 193 points · 29 comments
- hn · 193 points · 29 comments
- hn · 196 points · 31 comments
- hn · 196 points · 31 comments
- hn · 201 points · 31 comments
- hn · 201 points · 31 comments
- hn · 204 points · 32 comments
- hn · 204 points · 32 comments
- hn · 206 points · 32 comments
- hn · 206 points · 32 comments
- hn · 209 points · 33 comments
- hn · 209 points · 33 comments
- hn · 213 points · 34 comments
- hn · 213 points · 34 comments
- hn · 213 points · 34 comments
- hn · 213 points · 34 comments
- hn · 216 points · 34 comments
- hn · 216 points · 34 comments