BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
BitSparse reports 30–31% lower per-token VRAM use, with overhead of no more than 1% at practical sequence lengths.
The reported reductions are 31% on Modded-NanoGPT and 30% on Nemotron-H. The GPU-resident format losslessly compresses sparse ReLU and ReLU² activations, reconstructing them exactly for the backward pass through custom Triton kernels. It supports BF16 and FP8 activations, with optional sign-bit packing to increase compression. A preallocated buffer system also avoids host synchronization when allocating variable-size tensors during training.
ArXiv · AI/CL/LG's note
The reported reductions are 31% on Modded-NanoGPT and 30% on Nemotron-H. The GPU-resident format losslessly compresses sparse ReLU and ReLU² activations, reconstructing them exactly for the backward pass through custom Triton kernels. It supports BF16 and FP8 activations, with optional sign-bit packing to increase compression. A preallocated buffer system also avoids host synchronization when allocating variable-size tensors during training.
ArXiv · AI/CL/LG's note
score 5