Human-Like Embodied Intelligence for Robot Learning

Mingyu Ding

UC Berkeley

Human-like embodied intelligence represents an important milestone in the realms of robotics and AI, offering practical applications such as household robots designed to streamline our daily tasks. Nevertheless, the current capabilities of robots still lag far behind humans, particularly in commonsense reasoning. Humans possess an array of sensory and reasoning abilities: our eyes perceive and comprehend the 3D world, our hands interact with objects to estimate their physical properties, and our brains plan action with inherent knowledge to navigate complex tasks.

In this talk, I will discuss human-oriented robot learning, with a focus on imbuing robots with human commonsense reasoning capabilities across three key dimensions. (i) Firstly, I will introduce vision world models capable of parsing visual scenes into objects and comprehending how humans interact with them, serving as the foundation for robots to extract knowledge from human videos and mimic human behaviors. (ii) Beyond visual concepts, I will delve into physical reasoning, which enables robots to infer dynamics and physical properties of objects, such as mass, friction, and fragility, thereby enhancing robotics manipulation with precision and generalizability. (iii) Lastly, I will present language-based task planners, functioning as the brain of robots, which decompose complex tasks, plan actions, and facilitate seamless communication with humans by bridging vision and physics through natural language.

The synergy of these components embodies robotic commonsense reasoning, highlighting the potential of this interdisciplinary field in shaping the future landscape. Despite challenges, I conclude optimistically: as more modalities and reasoning capabilities are integrated into robot learning, the generalization of robots will enhance across various everyday scenarios, delivering substantial social benefits and impact.

Mingyu Ding is currently a postdoctoral fellow at UC Berkeley, working with Prof. Masayoshi Tomizuka in the Department of Mechanical Engineering and Berkeley Artificial Intelligence Research (BAIR) Lab. Previously, he was a visiting scholar at MIT CSAIL and obtained his Ph.D. from the University of Hong Kong. Mingyu's research objective is to develop robots and embodied agents that can perceive, reason about, and interact effectively with the 3D physical world like humans, by integrating insights from interdisciplinary domains including robotics, vision & language, and brain science. He has published over 40 papers at leading AI and robotics conferences, including CoRL, CVPR, NeurIPS, ICLR, ICRA, etc, accumulating over 2000 citations on Google Scholar. His works have been recognized by multiple research awards, such as the Baidu Fellowship and Microsoft Fellowship Nomination, alongside Rising Star Awards from both ME and AI fields, for his contribution in improving the generalization of robotic manipulation toward achieving general robot intelligence.

https://columbiauniversity.zoom.us/j/93434954940?pwd=YlZ3aDRONTJEbVFkaStpUUxwamF6UT09
Meeting ID: 934 3495 4940
Passcode: 487143