Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
The paper says frontier LLMs can miss exact copying even when the string fits inside context.
The authors argue the failure comes from Transformer positional encodings favoring local-context shortcuts over precise position lookup. They propose 2D-RoPE, placing tokens on a grid with row and column IDs. In synthetic tests, shallow Transformers using it copied perfectly far beyond training lengths, while standard encodings lagged. The reported advantage also held during DCLM pretraining up to 1.4B parameters. ArXiv · AI/CL/LG's note
The authors argue the failure comes from Transformer positional encodings favoring local-context shortcuts over precise position lookup. They propose 2D-RoPE, placing tokens on a grid with row and column IDs. In synthetic tests, shallow Transformers using it copied perfectly far beyond training lengths, while standard encodings lagged. The reported advantage also held during DCLM pretraining up to 1.4B parameters. ArXiv · AI/CL/LG's note
score 6