Learning on 3D Scene Graphs
Name
hu-siyi-phd-aeroastro-2026-thesis.pdf
Size
19.37 MB
Format
Adobe PDF
Checksum (MD5)
d4bb8fcbbbda66c241e1fce7b4bb2e8b
Author(s)
Hu, Siyi
Advisor(s)
Carlone, Luca
Date Issued
May 2026
Publisher
Massachusetts Institute of Technology
Abstract
Autonomous robots operating in human-populated environments must interpret and execute complex natural language instructions such as “bring me the cup from the kitchen table” or “clean the living room.” To support such tasks, robots require rich spatial representations that integrate both geometric information for navigation and semantic understanding for reasoning about objects, rooms, and their relationships. 3D scene graphs address this need by providing hierarchical metric-semantic representations that organize spatial information across multiple levels of abstraction, from low-level geometric primitives to high-level structures like rooms and buildings. However, constructing complete and accurate scene graphs from realworld sensor data remains challenging, as the representations obtained in practice often suffer from missing semantic labels and incomplete observations of the environment. Specifically, two fundamental technical gaps limit the utility of 3D scene graphs for robotic applications. First, while perception systems can reliably detect and classify objects, higher-level semantic attributes—particularly room labels—are often missing or uncertain due to perceptual limitations, incomplete exploration, and computational constraints. Second, robots can only construct scene graphs for regions they have actively explored, while it is also desirable for them to reason about and predict the structure of unobserved spaces, such as adjacent rooms or occluded areas, to support informed exploration planning and long-horizon task execution. This thesis addresses these challenges through three contributions that leverage machine learning techniques tailored to the hierarchical structure of scene graphs. First, we introduce neural trees, a novel graph neural network architecture designed for learning on boundedtreewidth graphs. Unlike standard message-passing neural networks, neural trees operate on tree decompositions of the input graph, enabling them to exploit the structural properties of hierarchical representations. We prove that neural trees are universal approximators for functions defined on bounded-treewidth graphs and demonstrate their effectiveness for predicting missing room labels in 3D scene graphs by propagating information across hierarchical layers. Second, we develop a mixed discrete-continuous diffusion model for predicting realistic object configurations in indoor scenes. Our approach jointly generates object semantic categories and geometric attributes including locations, orientations, and sizes, enabling more coherent scene generation compared to methods that treat discrete and continuous domains separately. We introduce a corruption-and-masking strategy that allows flexible incorporation of partial constraints during generation without requiring model retraining, supporting various scene completion and editing scenarios through a single trained model. Third, we extend the mixed diffusion framework to general multi-domain graph generation by treating edges as a separate feature domain alongside node attributes. Building on the hierarchical nature of 3D scene graphs, we develop a top-down generation approach that sequentially predicts each layer from high-level room layouts down to detailed object arrangements. This exploits the fact that inter-layer connections are determined by geometric containment while intra-layer connections capture local spatial arrangements such as traversability between rooms. This hierarchical diffusion framework scales to complex multi-room environments and generates complete scene graphs with coherent global layouts and realistic local object arrangements. By predicting plausible configurations of unobserved spaces, this capability enables robots to reason about partially observed environments, supporting more informed exploration planning and long-horizon task execution. We validate our contributions on standard benchmarking datasets including Matterport3D and 3D-FRONT, demonstrating effectiveness on both clean ground-truth scene graphs and real-world scene graphs built from sensor data. Together, these contributions advance the state of the art in spatial AI and enable more capable autonomous systems operating in complex indoor environments.
MIT Department
Massachusetts Institute of Technology. Department of Aeronautics and Astronautics
Terms of Use
Attribution 4.0 International (CC BY 4.0)
Copyright retained by author(s)
Persistent DSpace Link
