Affordance-Based Navigation and Manipulation using Vision Language Models and Gaussian Processes
Name
abdulhai-rumaisa-meng-eecs-2026-thesis.pdf
Size
73.56 MB
Format
Adobe PDF
Checksum (MD5)
7ed50de4acc6da80d33c6e6656b2a429
Author(s)
Abdulhai, Rumaisa
Advisor(s)
Carlone, Luca
Date Issued
February 2026
Publisher
Massachusetts Institute of Technology
Abstract
Autonomous navigation and manipulation in indoor environments require robots to select appropriate body poses that enable both reliable perception and successful task execution, rather than relying solely on object detection or geometric localization. For instance, modern robots such as Boston Dynamics’ Spot provide prebuilt manipulation skills, but these behaviors typically require the robot to reach suitable poses from which the skills can be executed successfullly, and are highly sensitive to viewpoint and initial configuration. We propose a perception-driven, probabilistic framework for affordance-aware skill execution that combines semantic segmentation and Vision-Language Models (VLMs) with Gaussian Process (GP) regression. Our approach learns pose-conditioned feasibility regions for fixed manipulation skills, such as opening doors and pressing elevator buttons, by explicitly decomposing task success into two coupled factors: (i) affordance observability, capturing whether interaction regions (e.g., handles, hinges, buttons) can be reliably detected from a given pose, and (ii) skill execution success, capturing whether the platform’s built-in manipulation behavior is likely to succeed from that pose. Two GPs are trained from real robot data to model these factors and are queried at runtime to score candidate poses, accounting for predictive uncertainty and travel cost. We validate the method on real-world door opening with Spot, demonstrating fully autonomous pose selection, affordance grounding, and skill execution. We further discuss how the same formulation can be extended to elevator operation and other interaction-dependent navigation tasks.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
