Agent Retrieval Bench: Evaluating Repository Context Retrieval for Coding Agents
The benchmark isolates the file-finding step that coding agents must get right before they can write a patch.
Agent Retrieval Bench tests repository context retrieval across 427 samples from 25 repositories, using frozen base commits and relevance defined by what an agent would need next. Its tasks cover code-to-test, comment-to-context, trace-to-code, edit ripple effects, and selective retrieval cases where no gold file may exist. The paper reports no single retrieval approach winning across the board: Qwen3 embeddings, RepoMap, and task-specific methods trade off by metric and budget. It also finds selective thresholds calibrated on counterfactual controls fail to transfer to natural no-gold cases, and logged agent runs miss every gold file on 27-35% of samples. HF Daily Papers' note
Agent Retrieval Bench tests repository context retrieval across 427 samples from 25 repositories, using frozen base commits and relevance defined by what an agent would need next. Its tasks cover code-to-test, comment-to-context, trace-to-code, edit ripple effects, and selective retrieval cases where no gold file may exist. The paper reports no single retrieval approach winning across the board: Qwen3 embeddings, RepoMap, and task-specific methods trade off by metric and budget. It also finds selective thresholds calibrated on counterfactual controls fail to transfer to natural no-gold cases, and logged agent runs miss every gold file on 27-35% of samples. HF Daily Papers' note
score 6