MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion Transformers
MC-Sparse claims dense-attention fidelity with up to 2.32x denoising speedups and negligible quality loss.
The paper says sparse attention in diffusion transformers breaks down at high sparsity because of grouping constraints, poor interaction selection, and missing attention contributions. Its proposed MC-Sparse method selects individual KV tokens, groups similar queries for GPU efficiency, and reuses cached metadata across denoising steps. The authors report better fidelity than existing sparse-attention baselines on video and 3D generation models, without visible degradation. Source: ArXiv · AI/CL/LG's note.
The paper says sparse attention in diffusion transformers breaks down at high sparsity because of grouping constraints, poor interaction selection, and missing attention contributions. Its proposed MC-Sparse method selects individual KV tokens, groups similar queries for GPU efficiency, and reuses cached metadata across denoising steps. The authors report better fidelity than existing sparse-attention baselines on video and 3D generation models, without visible degradation. Source: ArXiv · AI/CL/LG's note.
score 6