收藏切换
Embodied intelligence model evaluation: from perception to execution
收藏切换
Chunyi Li1, 2, Jianbo Zhang1, 2, Jiahao Xiao1, 2, Bowen Yan1, 3, Shengyu Guo1, 4, Tongrui Ye1, 5, Weisi Lin6, Guangtao Zhai1, 2, *
Journal of Image and Graphics | 2026, 31(8) : 2705 - 2730
Less
收藏切换
Journal of Image and Graphics | 2026, 31(8): 2705-2730
• Embodied Intelligence and Humanoid Robotics Frontiers •
Embodied intelligence model evaluation: from perception to execution
Full
Chunyi Li1, 2, Jianbo Zhang1, 2, Jiahao Xiao1, 2, Bowen Yan1, 3, Shengyu Guo1, 4, Tongrui Ye1, 5, Weisi Lin6, Guangtao Zhai1, 2, *
Affiliations
  • 1 Center of AI Evaluation,Shanghai AI Lab,Shanghai 200032,China
  • 2 School of Information Science and Electronic Engineering,Shanghai Jiao Tong University,Shanghai 200240,China
  • 3 Institute of Data and Information,Tsinghua Shenzhen International Graduate School,Shenzhen 518055,China
  • 4 Yuanpei College,Peking University,Beijing 100871,China
  • 5 Department of Psychology,Tongji University,Shanghai 200292,China
  • 6 College of Computing and Data Science,Nanyang Technological University,Singapore 639798,Singapore
Published: 2026-08-16 doi: 10.11834/jig.250550
Outline
收藏切换

Embodied intelligence refers to agents that sense,reason,act,and learn through a tight “body-environment-task” feedback loop. It is widely regarded as the most promising route to general-purpose artificial intelligence (AI). Paradoxically,although model parameters,training data,and compute budgets have grown by four orders of magnitude in only five years,the evaluation ecosystem remains fragmented and largely irreproducible. The same functional capability (e.g.,“pick-and-place”) is formalized as object-detection mean average precision in one paper,as 3D intersection over union in a second,as task success rate in a third,and as human preference score in a fourth,with absolute gaps that exceed 10%—15% among “state-of-the-art” results. Environments,random seeds,physics parameters,sensor noise models,and success criteria are rarely open-sourced,and thus,the community cannot tell whether an apparent improvement originates from algorithmic innovation,data scale,evaluation cherry picking,or simple benchmark over-fitting. This uncertainty has become a critical bottleneck for scientific progress and industrial adoption. To address the gap,we conduct a systematic survey of 100+ papers published in IEEE/CVF Conference on Computer Vision and Pattern Recognition,IEEE International Conference on Robotics and Automation,Conference on Neural Information Processing Systems,Robotics: Science and Systems,International Conference on Learning Representations,International Journal of Robotic Research,and Journal of Machine Learning Research between 2020 and 2025. Then,we present the first holistic analysis of embodied AI evaluation. We organize the landscape into a three-tier pyramid: static datasets,simulation benchmarks,and real-robot trials. Each tier trades off cost,controllability,and ecological validity. Across all tiers,we decompose evaluation space into four tightly coupled stages: perception,cognition,decision,and execution. Then,we identify fine-grained capability dimensions along with 20+ quantitative metrics. For every dimension,we analyze the following: 1) the underlying methodological assumptions,2) the boundary conditions under which the metric is valid,3) the intrinsic limitations (e.g.,sim-to-real physics mismatch,dataset bias,human-label variance),and 4) the cost-credibility curve that determines when a researcher should ascend from static to simulation,and finally to the real world.All evaluations in this study are based on a theoretical “perception-cognition-decision-execution loop” framework. In embodied AI,perception is the process by which an agent converts raw,asynchronous,and often noisy signals from heterogeneous physical sensors,such as RGB-D cameras,LiDAR,tactile skins,joint encoders,force/torque,and inertial measurement unit units,into a registered,time-aligned digital twin that supplies metric 3D geometry and semantic labels. Cognition is then built on this torrent to distill structured,memory-enabled,and generalizable world models that encode object affordances,physical common sense,and relational and causal links,grounding free-form human language into the resulting latent space. Decision-making combines these world models with an explicit task goal and searches across symbolic,kinematic,and dynamic levels to output a coherent,risk-aware,and resource-efficient plan that is expressed as a hierarchy of subgoals,trajectories,and low-level set points. Finally,execution transforms this plan into real-world mechanical energy through high-rate feedback controllers that track desired motions or forces while compensating for actuator nonlinearities,shock,thermal limits,and human safety constraints,closing the perception-cognition-decision-execution loop that defines embodied AI. In contrast with humans,whose perception-cognition-decision-execution loop is seamlessly integrated,currently available embodied agents suffer structural flaws in every stage. Consequently,evaluation must decouple the four blocks and score accuracy,error,and latency separately to pinpoint bottlenecks. Static tests,i.e.,“written exams”,run on a GPU with a few megabytes of images or point clouds in minutes,but erase dynamics and physics. Simulation,i.e.,the “job interview”,adds interactive physics in tens of seconds but still drifts away from reality. Real robot trials,i.e.,“practical tests”,cost thousands,take days to deploy and minutes per run,but expose every sensing drift,prediction error,or control delay,offering the ultimate ground truth before field deployment. Therefore,by integrating the three-tier pyramid of static test-simulation-real robot evaluation into the closed perception-cognition-decision-execution loop,this work delivers the first systematic quantification of coverage,cost,and accuracy trade-offs across more than 100 embodied AI benchmarks,establishing a reproducible,traceable,and extensible evaluation baseline that not only exposes the exact stage at which state-of-the-art systems fail,but also provides a principled road map for allocating research budgets,standardizing protocols,and certifying safety before real-world deployment,marking a pivotal transition from empirical competition to scientific measurement,and accelerating the field’s development toward trustworthy,deployable,and general-purpose embodied intelligence. The detailed static-dynamic-real embodied intelligence evaluation can be accessed at https://opencompass.org.cn/embodied-intelligence.

embodied intelligence  /  benchmark evaluation  /  perception-cognition-decision-execution  /  simulation  /  real-world validation  /  sim2real consistency
李春一, 张建博, 肖嘉豪, 闫博闻, 郭晟毓, 叶桐瑞, 林维斯, 翟广涛. 具身智能模型评测:从感知到执行. 中国图象图形学报, 2026 , 31 (8) : 2705 -2730 . DOI: 10.11834/jig.250550
Chunyi Li, Jianbo Zhang, Jiahao Xiao, Bowen Yan, Shengyu Guo, Tongrui Ye, Weisi Lin, Guangtao Zhai. Embodied intelligence model evaluation: from perception to execution[J]. Journal of Image and Graphics, 2026 , 31 (8) : 2705 -2730 . DOI: 10.11834/jig.250550
  • National Natural Science Foundation of China(62225112,625B2118);Singapore MoE Foundation(MOE-T2EP20123-0006)
Year 2026 volume 31 Issue 8
17
8
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.250550
  • Receive Date:2025-11-05
  • Online Date:2026-10-30
  • Published:2026-08-16
Article Data
Affiliations
History
  • Received:2025-11-05
  • Revised:2026-01-29
Funding
National Natural Science Foundation of China(62225112,625B2118);Singapore MoE Foundation(MOE-T2EP20123-0006)
Affiliations
    1 Center of AI Evaluation,Shanghai AI Lab,Shanghai 200032,China
    2 School of Information Science and Electronic Engineering,Shanghai Jiao Tong University,Shanghai 200240,China
    3 Institute of Data and Information,Tsinghua Shenzhen International Graduate School,Shenzhen 518055,China
    4 Yuanpei College,Peking University,Beijing 100871,China
    5 Department of Psychology,Tongji University,Shanghai 200292,China
    6 College of Computing and Data Science,Nanyang Technological University,Singapore 639798,Singapore
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.250550
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT