PACE: Periodic and Channel-Aligned Encoding of Four-Bit Recurrent States
PACE’s mixed four- and eight-bit recurrent states were competitive with eight-bit DAMP on three models while using about two thirds of its memory.
The method rounds recurrent states once every few generated tokens, buffering the intervening updates to reduce accumulated rounding error. It assigns quantization scales by column and keeps the heads that retain rounding error longest at eight bits. Tests covered four hybrid models with 4B–35B parameters; the DAMP comparison held on the three larger models. The all-four-bit version also outscored Q-Mamba at equal memory, while periodic rounding preserved the buffer’s decoding speedup.
HF Daily Papers' note
The method rounds recurrent states once every few generated tokens, buffering the intervening updates to reduce accumulated rounding error. It assigns quantization scales by column and keeps the heads that retain rounding error longest at eight bits. Tests covered four hybrid models with 4B–35B parameters; the DAMP comparison held on the three larger models. The all-four-bit version also outscored Q-Mamba at equal memory, while periodic rounding preserved the buffer’s decoding speedup.
HF Daily Papers' note
score 5