From Large Language Models to Efficient System One Models via Adaptive Computation
The paper reports that the final six model layers can be removed without reducing decision performance.
Decision information is concentrated in intermediate layers, while early layers build essential features. The proposed approach trains a full model alongside shorter versions that share layers and a decision head, using self-distillation. A lightweight gate decides at each exit whether more computation is needed. Across seven decision benchmarks, the approach matches or beats independently trained models at every tested depth and improves the accuracy-efficiency trade-off.
HF Daily Papers' note
Decision information is concentrated in intermediate layers, while early layers build essential features. The proposed approach trains a full model alongside shorter versions that share layers and a decision head, using self-distillation. A lightweight gate decides at each exit whether more computation is needed. Across seven decision benchmarks, the approach matches or beats independently trained models at every tested depth and improves the accuracy-efficiency trade-off.
HF Daily Papers' note
score 4