Think Twice: The Metis-1.6 Technical Report
Logos placed seventh of 17 models while using roughly a twelfth of the median training data disclosed by its comparators.
Built from scratch by a single researcher, Metis-1.6’s two models run a hybrid Mamba-2 and attention mixture-of-experts stack twice per token. The report ranks Logos second of 30 models in mathematics, but identifies weaknesses in code, instruction following and long context. It also documents training failures: an intended optimizer never ran, a learned memory switched itself off, and five fine-tuning choices reduced capability. A length reward ended up penalizing long, correct solutions under certain conditions; the author derives a one-line fix.
HF Daily Papers' note
Built from scratch by a single researcher, Metis-1.6’s two models run a hybrid Mamba-2 and attention mixture-of-experts stack twice per token. The report ranks Logos second of 30 models in mathematics, but identifies weaknesses in code, instruction following and long context. It also documents training failures: an intended optimizer never ran, a learned memory switched itself off, and five fine-tuning choices reduced capability. A length reward ended up penalizing long, correct solutions under certain conditions; the author derives a one-line fix.
HF Daily Papers' note
score 4