Schrödinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?
The paper tests whether coding agents lose their edge when familiar repository cues are stripped away.
SchrodingerRepo keeps the repo’s executable behavior intact while changing names, layouts, file order, and code structure. On SWE-bench Verified and SWE-QA, popular LLMs performed worse and spent more interaction cost under those transformed repos. The authors trace much of that added cost to harder exploration and localization inside the codebase. HF Daily Papers' note
SchrodingerRepo keeps the repo’s executable behavior intact while changing names, layouts, file order, and code structure. On SWE-bench Verified and SWE-QA, popular LLMs performed worse and spent more interaction cost under those transformed repos. The authors trace much of that added cost to harder exploration and localization inside the codebase. HF Daily Papers' note
score 5