Is Your Long-Horizon Benchmark Really Deep? Measuring and Synthesizing Depth in Terminal Tasks
Of the solved synthetic terminal tasks audited, 94% passed within three turns.
The researchers examined 734 tasks across six datasets, separating total operations from steps that require reading earlier output before choosing the next command. They found that this sequential depth predicts difficulty beyond workload, while ordinary turn counts largely reflect model habits. Their DEPTH method builds tasks segment by segment in a live software environment, producing 105 tasks that exceed every audited dataset in both depth and work. None of the six models tested solved half of them.
HF Daily Papers' note
The researchers examined 734 tasks across six datasets, separating total operations from steps that require reading earlier output before choosing the next command. They found that this sequential depth predicts difficulty beyond workload, while ordinary turn counts largely reflect model habits. Their DEPTH method builds tasks segment by segment in a live software environment, producing 105 tasks that exceed every audited dataset in both depth and work. None of the six models tested solved half of them.
HF Daily Papers' note
score 5