MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup
MoME lets a model route the same token to different memory slots depending on context.
The paper replaces a token’s single memory row with a mixture of slots, selected by a learned gate over the hidden state. In controlled pretraining tests on nanochat, Llama-3/MobileLLM, and Qwen3 backbones, it beats Value Embedding, Bigram, and STEM baselines under matched parameter and training-FLOP settings. The authors also report better memory-size scaling below the billion-parameter range, while keeping training and inference efficient. Routing examples on ambiguous tokens suggest the slots separate meanings, such as “python” as code versus animal. HF Daily Papers' note
The paper replaces a token’s single memory row with a mixture of slots, selected by a learned gate over the hidden state. In controlled pretraining tests on nanochat, Llama-3/MobileLLM, and Qwen3 backbones, it beats Value Embedding, Bigram, and STEM baselines under matched parameter and training-FLOP settings. The authors also report better memory-size scaling below the billion-parameter range, while keeping training and inference efficient. Routing examples on ambiguous tokens suggest the slots separate meanings, such as “python” as code versus animal. HF Daily Papers' note
score 5