LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
LatentLoop’s fitted results predict 19% less compute than vanilla MoE at the same validation loss.
The model compresses the sequence on its first pass through a shared block, keeps the second pass at full resolution, and spends the saved compute on greater hidden width. This follows experiments showing that weakening the first pass preserved or improved performance, while weakening the second hurt it. Across 15 configurations spanning 1.3B–7B parameters, LatentLoop consistently reduced validation loss and improved average scores on nine benchmarks in both zero-shot and five-shot tests. At a fitted loss of 1.988, predicted compute savings were also 7.9% versus the looped baseline.
HF Daily Papers' note
The model compresses the sequence on its first pass through a shared block, keeps the second pass at full resolution, and spends the saved compute on greater hidden width. This follows experiments showing that weakening the first pass preserved or improved performance, while weakening the second hurt it. Across 15 configurations spanning 1.3B–7B parameters, LatentLoop consistently reduced validation loss and improved average scores on nine benchmarks in both zero-shot and five-shot tests. At a fitted loss of 1.988, predicted compute savings were also 7.9% versus the looped baseline.
HF Daily Papers' note
score 5