Paper Detail

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shijue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Zhuo Chen, Xierui Song, Juntao Dai, Boyuan Chen, Jiaming Ji, Fangneng Zhan, Mengkang Hu, Wei Xue, Yonggang Zhang, Han Hu, Tsung-Yi Ho, Yike Guo

arxiv Score 12.8

Published 2026-10-07 · First seen 2026-10-08

General AI

Abstract

General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world. To investigate this, we introduce RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and executable success checks. By analysing task outcomes alongside execution traces, we identify both the capabilities that transfer and the gaps that prevent reliable completion. Furthermore, we find that current agents can construct sophisticated perception and control workflows, including image segmentation, camera calibration, spatial estimation, and dynamics-based computation. These capabilities, however, do not consistently compose into successful behaviour: agents lose task-relevant object states despite reaching commanded poses, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion. This uneven transfer also differs across models: Astra succeeds more often on spatial and constrained-contact goals, whereas Opus 5.5 succeeds more often on continuous-balance and timed-interaction goals. By linking these outcomes to execution behaviour, RobotWorld provides both a rigorous proving ground and an empirical account of the remaining capability gaps, thereby establishing concrete targets for training and designing more reliable physical-world agents.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{yang2026robotworld,
  title = {RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments},
  author = {Zhiqin Yang and Chenxin Li and Xiaomeng Hu and Yibin Liu and Weidong Huang and Jiankai Sun and Haitao Li and Zijian Wu and Yuzhi Huang and Fanding Huang and Hanwen Sun and Jiashun Liu and Jingqi Tong and Mingxin Huang and Shaoli Hu and Shijue Huang and Tianyi Bai and Xinyuan Wang and Yunlong Lin and Zhengyang Tang and Zhexin Zhang and Zhuo Chen and Xierui Song and Juntao Dai and Boyuan Chen and Jiaming Ji and Fangneng Zhan and Mengkang Hu and Wei Xue and Yonggang Zhang and Han Hu and Tsung-Yi Ho and Yike Guo},
  year = {2026},
  abstract = {General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world. To investigate this, we introduce RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and exec},
  url = {https://arxiv.org/abs/2610.10409},
  keywords = {cs.RO, cs.LG},
  eprint = {2610.10409},
  archiveprefix = {arXiv},
}

Metadata

{}