onóna: Twin Embeddings with Tied Unembeddings
onóna improved validation loss and external perplexity in 11 of 12 comparisons, at the cost of an extra embedding table and vocabulary projection.
The method averages two embedding tables for input and mixes their output softmax distributions, preserving input-output weight sharing. A gate based only on the current input token sets the mixture weight; its values can be precomputed for inference, and the Transformer body stays unchanged. Tests covered four architectures at 142M, 330M and 1B parameters under matched token budgets, with the 1B mixture-of-experts model the sole exception on loss. Average accuracy across eight downstream tasks improved in 10 of 12 comparisons.
HF Daily Papers' note
The method averages two embedding tables for input and mixes their output softmax distributions, preserving input-output weight sharing. A gate based only on the current input token sets the mixture weight; its values can be precomputed for inference, and the Transformer body stays unchanged. Tests covered four architectures at 142M, 330M and 1B parameters under matched token budgets, with the 1B mixture-of-experts model the sole exception on loss. Average accuracy across eight downstream tasks improved in 10 of 12 comparisons.
HF Daily Papers' note
score 5