I trained a small transformer in 1.5hrs and it beats many LLMs
A scratch-trained transformer hit about 44% on ARC-1 public eval in roughly 1.5 hours on an RTX 5090.
The author says the gain comes from architecture and training changes, including 3D RoPE, per-task embeddings, SwiGLU, RMSNorm, better shuffling, fewer augmentations, and NorMuon. They report that removing 3D RoPE or per-task embeddings drops performance to about 24-25%. The model no longer trains on input tokens, which the author says slightly improves score despite worse test loss. Code is open source, and the author argues the result points to sample efficiency as the central ARC problem. HN · LLMs' note
The author says the gain comes from architecture and training changes, including 3D RoPE, per-task embeddings, SwiGLU, RMSNorm, better shuffling, fewer augmentations, and NorMuon. They report that removing 3D RoPE or per-task embeddings drops performance to about 24-25%. The model no longer trains on input tokens, which the author says slightly improves score despite worse test loss. Code is open source, and the author argues the result points to sample efficiency as the central ARC problem. HN · LLMs' note
score 5