FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
FLAMIA reports up to 1.42× faster forward-and-backward attention, including quantization overhead.
Its MXFP8 path delivers 1.10–1.40× speedups on B200 and 1.06–1.42× on B300 against the fastest evaluated baseline for each tested causal packed-document workload. The kernels guard against causal information leakage through shared quantization scales and keep forward and backward computations consistent. Document-local scaling supports variable-length documents in packed layouts without repacking. The authors report stable training and preserved quality in Llama-style pretraining tests.
HF Daily Papers' note
Its MXFP8 path delivers 1.10–1.40× speedups on B200 and 1.06–1.42× on B300 against the fastest evaluated baseline for each tested causal packed-document workload. The kernels guard against causal information leakage through shared quantization scales and keep forward and backward computations consistent. Document-local scaling supports variable-length documents in packed layouts without repacking. The authors report stable training and preserved quality in Llama-style pretraining tests.
HF Daily Papers' note
score 5