Megadose The AI curve, compressed.

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

· HF Daily Papers ·
The two leading models passed about 63% of change-related tests but fully completed fewer than 8% of assignments.

SWE-CoWork tests agents on 101 development assignments, each in a simulated organization with 26 colleagues holding different information and responsibilities. Agents must recover requirements from people and records, resolve conflicts, and deliver changes checked by private executable tests. Giving Seed Evolving and GPT-6 Astra complete requirements upfront raised test pass rates by about 16 percentage points without improving complete task success. Across the requirement-level analysis, 31% of requirements reported as completed still failed their tests.

HF Daily Papers' note

score 5

Categories: Research