SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
SWE-DIVE tests whether coding agents can handle users who add tasks and requirements across a conversation.
The framework uses a simulated user with different behavioral patterns to interact with the agent. A conversation validator checks whether the simulator’s messages meet the initial requirements and guide the agent toward solving the task. Harbor coordinates those components and runs the test environment to check task completion. The abstract says results and source code are available on GitHub, but reports no performance figures.
ArXiv · AI/CL/LG's note
The framework uses a simulated user with different behavioral patterns to interact with the agent. A conversation validator checks whether the simulator’s messages meet the initial requirements and guide the agent toward solving the task. Harbor coordinates those components and runs the test environment to check task completion. The abstract says results and source code are available on GitHub, but reports no performance figures.
ArXiv · AI/CL/LG's note
score 5