FDE-Gym: The Last Mile of AI Delivery
The best model configuration resolved just 14 of 95 simulated customer delivery tasks.
FDE-Gym tests agents on ambiguous requirements, scripted stakeholders and modifiable systems across 20 customer scenarios in 18 sectors. Across 13 model configurations from seven providers, the best averaged 0.56 on a scale from an unchanged system to a reference solution, with penalties for stakeholder interactions. Discovering unstated constraints was the main bottleneck. Even when that configuration obtained every key fact on 23 tasks, only 10 reached at least 0.8 in normalized business uplift.
HF Daily Papers' note
FDE-Gym tests agents on ambiguous requirements, scripted stakeholders and modifiable systems across 20 customer scenarios in 18 sectors. Across 13 model configurations from seven providers, the best averaged 0.56 on a scale from an unchanged system to a reference solution, with penalties for stakeholder interactions. Discovering unstated constraints was the main bottleneck. Even when that configuration obtained every key fact on 23 tasks, only 10 reached at least 0.8 in normalized business uplift.
HF Daily Papers' note
score 5