FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding
FOSCAR cuts quantization overhead, with reported decode-attention speedups of up to 3.36× over FlashInfer BF16 on H100.
Packing the KV cache into fewer bits saves memory traffic, but converting values for Tensor Cores can consume the gains. FOSCAR moves scale-and-offset calculations to another matrix operand or the output, reducing those operations by up to 100× while preserving the dequantized attention result in exact arithmetic. On B200, its INT2 kernel averages 1.16× the speed of TensorRT-LLM NVFP4, while INT4 matches that baseline. Integrated into SGLang, it raises peak aggregate throughput on long-context workloads by up to 170% over a 2-bit baseline and 341% over a 16-bit baseline, the authors report.
ArXiv · AI/CL/LG's note
Packing the KV cache into fewer bits saves memory traffic, but converting values for Tensor Cores can consume the gains. FOSCAR moves scale-and-offset calculations to another matrix operand or the output, reducing those operations by up to 100× while preserving the dequantized attention result in exact arithmetic. On B200, its INT2 kernel averages 1.16× the speed of TensorRT-LLM NVFP4, while INT4 matches that baseline. Integrated into SGLang, it raises peak aggregate throughput on long-context workloads by up to 170% over a 2-bit baseline and 341% over a 16-bit baseline, the authors report.
ArXiv · AI/CL/LG's note
score 5