Megadose The AI curve, compressed.

Weight-Space Programming Doesn't Need Transformers

· HF Daily Papers ·
An LLM-designed model beat the same LLM’s best released hand-written Transformer using about 1,800 times fewer parameters.

Given ten labeled examples and a sandbox, all three frontier LLMs chose sparse fact tables when allowed to design their own architectures, reaching up to 54.4% test accuracy on tasks focused mainly on multiple-choice questions. The tables made individual errors traceable to specific rows that could be rewritten. The authors also propose an associative-memory model up to 17 times faster on GPUs than a same-sized Transformer. This work-in-progress preprint concerns architectures whose weights LLMs write directly without gradient-based training; pretrained LLMs remain more accurate.

HF Daily Papers' note

score 5

Categories: Research