OrBIT: Structure-Guided Embedding Compression
The paper claims learned local geometry can shrink LLM embedding tables by more than 23x on 7B models.
OrBIT compresses embedding tables by discovering a coding structure instead of fixing one in advance. It learns reusable local geometry from orbit dynamics, then uses a global residual to decide where the limited coding budget goes. The authors say the orbit machinery is removed after training, leaving a compact decoder. Reported results include 37.9x compression on GPT-2 and more than 23x compression on each tested 7B embedding table versus 16-bit storage.
ArXiv · AI/CL/LG's note
OrBIT compresses embedding tables by discovering a coding structure instead of fixing one in advance. It learns reusable local geometry from orbit dynamics, then uses a global residual to decide where the limited coding budget goes. The authors say the orbit machinery is removed after training, leaving a compact decoder. Reported results include 37.9x compression on GPT-2 and more than 23x compression on each tested 7B embedding table versus 16-bit storage.
ArXiv · AI/CL/LG's note
score 4