Weight-Space Programming Doesn't Need Transformers
An LLM-designed model beat the same LLM’s best released hand-written Transformer using about 1,800 times fewer parameters.
Given ten labeled examples and a sandbox, all three frontier LLMs chose sparse fact tables when allowed to design their own architectures, reaching up to 54.4% test accuracy on tasks focused mainly on multiple-choice questions. The tables made individual errors traceable to specific rows that could be rewritten. The authors also propose an associative-memory model up to 17 times faster on GPUs than a same-sized Transformer. This work-in-progress preprint concerns architectures whose weights LLMs write directly without gradient-based training; pretrained LLMs remain more accurate.
HF Daily Papers' note
Given ten labeled examples and a sandbox, all three frontier LLMs chose sparse fact tables when allowed to design their own architectures, reaching up to 54.4% test accuracy on tasks focused mainly on multiple-choice questions. The tables made individual errors traceable to specific rows that could be rewritten. The authors also propose an associative-memory model up to 17 times faster on GPUs than a same-sized Transformer. This work-in-progress preprint concerns architectures whose weights LLMs write directly without gradient-based training; pretrained LLMs remain more accurate.
HF Daily Papers' note
score 5