Exploring the Role of Foundation Models for Training Generalist Robot Learning Policies
Name
feng-eyf-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
28.89 MB
Format
Adobe PDF
Checksum (MD5)
c8e4f24c69c8b8ee515f9ab50b6ef619
Author(s)
Feng, Eugenia Y.
Advisor(s)
Kaelbling, Leslie
Agarwal, Aditya
Date Issued
February 2025
Publisher
Massachusetts Institute of Technology
Abstract
Numerous methodologies to solving goal-conditioned short-horizon tasks require hundreds of expert demonstrations, but these demonstrations are effort-intensive to collect, reducing the scalability of these approaches. Even with approaches that do work, they may have difficulty generalizing to slightly different settings. In this work, we explore two approaches to training generalist robot learning policies using large-scale foundation models.
The first approach aims to use a video foundation model to generate task-conditioned synthetic demonstrations at scale from a single expert demonstration. The objective is to leverage these synthetic demonstrations as proxy for expert demonstrations to train models that learn rewards from expert videos for solving complex visual RL problems.
The second approach seeks to improve upon the generalization ability of behavior cloning policies. Moving away from the use of videos for training, we explore using privileged representations such as keypoints or object-poses learned using open-set foundation models. By tracking pose or keypoint correspondences, the aim is to minimize the required number of demonstrations to achieve task completion and improve generalization within classes of objects.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
