ReRound: Reconstructive Rounding to Resolve Midpoint Ambiguity in Calibration-Free LLM Quantization
ReRound uses diffusion-reconstructed weights only where RTN rounding is most ambiguous.
The method targets low-bit, calibration-free quantization for pretrained LLMs. It trains a conditional diffusion model offline, then uses its continuous weight reconstructions to choose rounding directions near quantization interval midpoints. A tolerance sweep produces candidate quantized matrices, with selection based on how closely leading singular values match the original full-precision weights. The paper reports stronger 3-bit and 4-bit results than standard RTN on smaller LLMs, with no added inference overhead. HF Daily Papers' note
The method targets low-bit, calibration-free quantization for pretrained LLMs. It trains a conditional diffusion model offline, then uses its continuous weight reconstructions to choose rounding directions near quantization interval midpoints. A tolerance sweep produces candidate quantized matrices, with selection based on how closely leading singular values match the original full-precision weights. The paper reports stronger 3-bit and 4-bit results than standard RTN on smaller LLMs, with no added inference overhead. HF Daily Papers' note
score 4