The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
In 1B-parameter pretraining, COUNT improved validation loss by as much as giving a language model 57% more compute.
The method stores precomputed n-gram counts in host memory and uses them to build additional input embeddings without adding learned parameters. Corpus frequencies weight the model’s own output embeddings for tokens that commonly follow a given history. With GPU memory constrained, keeping backbone parameters on the GPU performed better than sharing that space with learned n-gram embeddings. Swapping in counts from a domain-specific corpus also enables adaptation at inference time without further training.
HF Daily Papers' note
The method stores precomputed n-gram counts in host memory and uses them to build additional input embeddings without adding learned parameters. Corpus frequencies weight the model’s own output embeddings for tokens that commonly follow a given history. With GPU memory constrained, keeping backbone parameters on the GPU performed better than sharing that space with learned n-gram embeddings. Swapping in counts from a domain-specific corpus also enables adaptation at inference time without further training.
HF Daily Papers' note
score 5