AgentSWE: Can Coding Agents Build the Agent You Actually Want?
AgentSWE tests whether coding agents can deliver an agent that actually satisfies a user’s stated requirement.
The benchmark covers creation, editing, and optimization tasks across the agent development lifecycle. Its 25 tasks give builders visible examples, then grade frozen submissions on held-out cases tied to requirement-derived rubrics. The authors report that evaluated coding agents consistently fell short, with failures including overfitting, incomplete runtimes, and unsupported claims of completion. HF Daily Papers' note
The benchmark covers creation, editing, and optimization tasks across the agent development lifecycle. Its 25 tasks give builders visible examples, then grade frozen submissions on held-out cases tied to requirement-derived rubrics. The authors report that evaluated coding agents consistently fell short, with failures including overfitting, incomplete runtimes, and unsupported claims of completion. HF Daily Papers' note
score 5