SO-101 Bench: Accessible Spatial Manipulation
A code-writing LLM agent scored 83.2% on the benchmark; the best learned policies reached about 61%.
SO-101 Bench tests language-conditioned spatial manipulation on a simulated version of a low-cost robot arm, using 138 objects across 916 evaluation episodes. Learned policies were fine-tuned on 3,629 simulated demonstrations under an equal compute budget. Five tested policies retained their exact ranking when transferred to the physical arm with unseen objects, though simulation overstated success. Spatial placement, unfamiliar object classes and rotate-to-fit planning remained difficult; the LLM agent’s lead depended on both its harness and its language model.
HF Daily Papers' note
SO-101 Bench tests language-conditioned spatial manipulation on a simulated version of a low-cost robot arm, using 138 objects across 916 evaluation episodes. Learned policies were fine-tuned on 3,629 simulated demonstrations under an equal compute budget. Five tested policies retained their exact ranking when transferred to the physical arm with unseen objects, though simulation overstated success. Spatial placement, unfamiliar object classes and rotate-to-fit planning remained difficult; the LLM agent’s lead depended on both its harness and its language model.
HF Daily Papers' note
score 4