RiLM: Parameter-Efficient Language Modeling via Geodesic Decoding
RiLM removes the output matrix and decodes tokens by distance in embedding space.
Fang Li frames the method for sub-million-parameter language models, where the output layer can take a large share of capacity. The paper tests flat and hyperbolic versions with about 290k parameters on WikiText-2, reporting 54.2 validation perplexity for HypRiLM versus 87.6 for Flat RiLM and 113-147 for tied LSTM, Transformer, and SSM controls. It also reports transfer to Penn Treebank and a 10k-vocabulary stress test, with hyperbolic curvature helping selectively. The claims are limited to controlled small-model comparisons, not full-vocabulary state of the art. ArXiv · AI/CL/LG's note
Fang Li frames the method for sub-million-parameter language models, where the output layer can take a large share of capacity. The paper tests flat and hyperbolic versions with about 290k parameters on WikiText-2, reporting 54.2 validation perplexity for HypRiLM versus 87.6 for Flat RiLM and 113-147 for tied LSTM, Transformer, and SSM controls. It also reports transfer to Penn Treebank and a 10k-vocabulary stress test, with hyperbolic curvature helping selectively. The claims are limited to controlled small-model comparisons, not full-vocabulary state of the art. ArXiv · AI/CL/LG's note
score 5