AutoIndex: Learning Representation Programs for Retrieval
AutoIndex improves retrieval by learning how documents should be transformed before they are indexed.
The paper frames document representation as the optimization target, while keeping BM25 fixed. AutoIndex searches executable programs that slice, enrich, normalize, reweight, or reorganize documents, then keeps changes that improve validation retrieval quality. On CRUMB, it improves recall over a static full-document BM25 baseline on all 8 tasks, averaging +8.4% Recall@100 and +8.3% nDCG@10. The largest reported gains are +30.5% Recall@100 and +43.6% nDCG@10. HF Daily Papers' note
The paper frames document representation as the optimization target, while keeping BM25 fixed. AutoIndex searches executable programs that slice, enrich, normalize, reweight, or reorganize documents, then keeps changes that improve validation retrieval quality. On CRUMB, it improves recall over a static full-document BM25 baseline on all 8 tasks, averaging +8.4% Recall@100 and +8.3% nDCG@10. The largest reported gains are +30.5% Recall@100 and +43.6% nDCG@10. HF Daily Papers' note
score 5