Your Transformer Is Secretly Elastic
The researchers cut 427 million parameters from DINOv3 ViT-H+/16 with a reported 0.8% drop in ImageNet-1k linear-probing accuracy.
Their method jointly ranks individual MLP neurons and attention heads, allowing pretrained transformers to adjust to different computational budgets without retraining. Tests across 27 vision and text encoders, from 22 million to 8 billion parameters, showed smooth degradation up to 60% sparsity. At that sparsity, AugReg ViT-B/16 delivered up to 1.80× higher throughput and 1.26× lower single-query latency on an A100 GPU. The authors say constructing the elastic models takes minutes.
ArXiv · AI/CL/LG's note
Their method jointly ranks individual MLP neurons and attention heads, allowing pretrained transformers to adjust to different computational budgets without retraining. Tests across 27 vision and text encoders, from 22 million to 8 billion parameters, showed smooth degradation up to 60% sparsity. At that sparsity, AugReg ViT-B/16 delivered up to 1.80× higher throughput and 1.26× lower single-query latency on an A100 GPU. The authors say constructing the elastic models takes minutes.
ArXiv · AI/CL/LG's note
score 5