FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
FVAttn targets the straggler problem created when adaptive sparse attention leaves some GPU ranks with heavier work than others.
The paper describes a training-free sparse-attention system for Video Diffusion Transformers. It combines Top-p routing, a Top-k safety floor, and video-aware block organization, then adjusts the attention mask at runtime. Its load balancer migrates a small number of heavy heads over P2P links, while unused rank slack is filled with extra high-value blocks. On step-distilled Wan2.2 I2V, the authors report load imbalance falling from 1.34 to 1.08, with a 4.41x attention speedup over FlashAttention and 2.02-2.11x DiT inference speedup. HF Daily Papers' note
The paper describes a training-free sparse-attention system for Video Diffusion Transformers. It combines Top-p routing, a Top-k safety floor, and video-aware block organization, then adjusts the attention mask at runtime. Its load balancer migrates a small number of heavy heads over P2P links, while unused rank slack is filled with extra high-value blocks. On step-distilled Wan2.2 I2V, the authors report load imbalance falling from 1.34 to 1.08, with a 4.41x attention speedup over FlashAttention and 2.02-2.11x DiT inference speedup. HF Daily Papers' note
score 4