Building Generalizable Robots through Compositional Generation

Yilun Du

 Generative AI has led to stunning successes in recent years but is fundamentally limited by the available data.  This is especially limiting in the robotics setting – where we wish to construct a robotic agent that can generalize in new environments. In this talk, I’ll introduce the idea of compositional generative modeling, which enables generalization beyond the training data that is available by building complex generative models from smaller constituents. I’ll first introduce the idea of energy-based models and illustrate how they enable compositional generative modeling. I’ll then illustrate how such compositional models enable us to synthesize complex plans in novel environments. Finally, I'll show how such compositionality can be applied to foundation models trained on Internet data to construct decision-making systems that can hierarchically plan and solve long-horizon problems in a zero-shot manner.

Yilun Du is a final-year PhD student at MIT. The NSF Graduate Research Fellowship supports him, and he was previously a research fellow at OpenAI, a visiting researcher at FAIR, and a student researcher at Google Deepmind.

https://columbiauniversity.zoom.us/j/91668632767?pwd=bXU2QXI0ZGtFcGRzdk55Q3NpVzR1QT09

Meeting ID: 916 6863 2767
Passcode: 109934