ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
Near-identical buggy code routinely outranked the correct implementation in this benchmark.
ExecRetrieval tests code retrievers against 939 Python tasks with execution-verified correct solutions and single-edit buggy distractors. The best hosted system found the correct implementation somewhere in the top 10 every time, but put it first only 33.1% of the time. For the leading systems, rank-1 misses were paired buggy variants 91.5% to 99.4% of the time. The authors release the dataset, oracle, embeddings, environment snapshot, and statistical tests. HF Daily Papers' note
ExecRetrieval tests code retrievers against 939 Python tasks with execution-verified correct solutions and single-edit buggy distractors. The best hosted system found the correct implementation somewhere in the top 10 every time, but put it first only 33.1% of the time. For the leading systems, rank-1 misses were paired buggy variants 91.5% to 99.4% of the time. The authors release the dataset, oracle, embeddings, environment snapshot, and statistical tests. HF Daily Papers' note
score 5