Embodied First Exam
The strongest of eight coding agents solved just 34% of the benchmark’s 456 core robotics tasks under its Standard Setting.
The full benchmark spans 1,156 tasks across 35 suites and 47 robots, letting agents control robots directly, write control code, or train a policy. Tasks extend beyond pick-and-place to dexterous manipulation, multi-robot interaction and dynamic whole-body control. The Standard Setting provides only sensor observations and normally prohibits resets; its Realtime variant keeps the environment running while the agent reasons. A separate Privileged Setting grants full simulator state and unlimited resets, with success rates serving as an approximate upper bound.
HF Daily Papers' note
The full benchmark spans 1,156 tasks across 35 suites and 47 robots, letting agents control robots directly, write control code, or train a policy. Tasks extend beyond pick-and-place to dexterous manipulation, multi-robot interaction and dynamic whole-body control. The Standard Setting provides only sensor observations and normally prohibits resets; its Realtime variant keeps the environment running while the agent reasons. A separate Privileged Setting grants full simulator state and unlimited resets, with success rates serving as an approximate upper bound.
HF Daily Papers' note
score 5