Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages
Longtail-SWE pairs 202 issue-resolution tasks across 13 languages with a pipeline designed to repair flawed tests and unclear requirements.
A solver attempts each task, then a judge reviews its trajectory and test results to guide repairs. The revisions aim to stop tests from rejecting valid implementations or accepting incorrect ones while preserving the intended requirements. The authors evaluate nine model configurations for solving performance, consistency across repeated attempts and resource use. They release the tasks, executable environments, tooling and agent trajectories, plus easier tasks for smaller models.
HF Daily Papers' note
A solver attempts each task, then a judge reviews its trajectory and test results to guide repairs. The revisions aim to stop tests from rejecting valid implementations or accepting incorrect ones while preserving the intended requirements. The authors evaluate nine model configurations for solving performance, consistency across repeated attempts and resource use. They release the tasks, executable environments, tooling and agent trajectories, plus easier tasks for smaller models.
HF Daily Papers' note
score 5