In plain words: By watching short videos of its own movements, a robot learns to predict how its body will move, without needing motion-capture data or a hand-built model of its joints. The learned self-simulation planned motions accurately and spotted damage so the robot could adapt.
Abstract
The emergence of vision catalysed a pivotal evolutionary advancement, enabling organisms not only to perceive but also to interact intelligently with their environment. This transformation is mirrored by the evolution of robotic systems, where the ability to leverage vision to simulate and predict their own dynamics marks a leap towards autonomy and self-awareness. Humans utilize vision to record experiences and internally simulate potential actions. For example, we can imagine that, if we stand up and raise our arms, the body will form a T shape without physical movement. Similarly, simulation allows robots to plan and predict the outcomes of potential actions without execution. Here we introduce a self-supervised learning framework to enable robots to model and predict their morphology, kinematics and motor control using only brief raw video data, eliminating the need for extensive real-world data collection and kinematic priors. By observing their own movements, akin to humans watching their reflection in a mirror, robots learn an ability to simulate themselves and predict their spatial motion for various tasks. Our results demonstrate that this self-learned simulation not only enables accurate motion planning but also allows the robot to detect abnormalities and recover from damage.
Yuhang Hu, Jiong Lin, Hod Lipson
arXiv:2311.12151 · cs.RO, cs.AI, cs.CV · submitted Nov 20, 2023 · updated Mar 4, 2025
abstract · pdf · This paper was published on Nature Machine Intelligence