Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens
Trainable [DUMP] tokens improved retrieval benchmarks without changing the model architecture, pooling strategy or training loss.
The method adds learned tokens before the input text and trains them using the standard contrastive retrieval objective. These tokens help reduce the influence of residual, low-information signals on the final embedding. The authors report consistent gains over fine-tuned baselines across encoder and decoder models on standard retrieval and long-context benchmarks. Their analysis also finds changes in activation and attention patterns associated with low-information tokens.
HF Daily Papers' note
The method adds learned tokens before the input text and trains them using the standard contrastive retrieval objective. These tokens help reduce the influence of residual, low-information signals on the final embedding. The authors report consistent gains over fine-tuned baselines across encoder and decoder models on standard retrieval and long-context benchmarks. Their analysis also finds changes in activation and attention patterns associated with low-information tokens.
HF Daily Papers' note
score 4