Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers
A 280-parameter transformer construction is claimed to evaluate fully parenthesized Boolean expressions at arbitrary depth and length.
The paper frames algorithmic reasoning as circuit computation embedded inside a transformer. Its model uses depth-aware positional encoding, masked hard attention, and an autonomous halt rule to reduce expressions over `d` iterations for depth `d`. The authors say shallow training on depth-1 and depth-2 cases recovers interpretable parameters that “snap” into place, producing exact length generalization. They also report 100% generalization on modular arithmetic and ListOps benchmarks. Source: ArXiv · AI/CL/LG's note.
The paper frames algorithmic reasoning as circuit computation embedded inside a transformer. Its model uses depth-aware positional encoding, masked hard attention, and an autonomous halt rule to reduce expressions over `d` iterations for depth `d`. The authors say shallow training on depth-1 and depth-2 cases recovers interpretable parameters that “snap” into place, producing exact length generalization. They also report 100% generalization on modular arithmetic and ListOps benchmarks. Source: ArXiv · AI/CL/LG's note.
score 5