2026-09-19T06:51:43Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1567432024-09-17T03:58:18Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Carlone, Luca
Morales, Joseph
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2024-09-16T13:46:25Z
2024-09-16T13:46:25Z
2024-05
2024-07-11T14:36:33.616Z
https://hdl.handle.net/1721.1/156743
3D Scene Graphs are expressive map representations for scene understanding in robotics and computer vision. Current approaches for automated zero-shot 3D Scene Graph generation rely on spatial ontologies that relate objects with the semantic locations they are found in (e.g., a fork is found in a kitchen). While conferring impressive zero-shot performance, these approaches are conditioned on the existence of disambiguating objects in a scene, the expressiveness of the generated spatial ontologies, and knowing during data collection that a robot needs to observe specific objects in the environment. This thesis proposes a method for zero-shot scene graph generation by leveraging Vision-Language Models (VLMs) to construct a layer of Viewpoints in the scene graph, which allow for after-the-fact open-vocabulary querying over the scene. Methods for utilizing different VLM features are explored, which result in improvement over the ontological approach on region segmentation tasks.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
Enhancing 3D Scene Graph Generation with Multimodal Embeddings
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="23cef366-24f1-4b73-98ec-74e80ce472f4">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Enhancing 3D Scene Graph Generation with Multimodal Embeddings</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2024-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Morales, Joseph</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>3D Scene Graphs are expressive map representations for scene understanding in robotics and computer vision. Current approaches for automated zero-shot 3D Scene Graph generation rely on spatial ontologies that relate objects with the semantic locations they are found in (e.g., a fork is found in a kitchen). While conferring impressive zero-shot performance, these approaches are conditioned on the existence of disambiguating objects in a scene, the expressiveness of the generated spatial ontologies, and knowing during data collection that a robot needs to observe specific objects in the environment. This thesis proposes a method for zero-shot scene graph generation by leveraging Vision-Language Models (VLMs) to construct a layer of Viewpoints in the scene graph, which allow for after-the-fact open-vocabulary querying over the scene. Methods for utilizing different VLM features are explored, which result in improvement over the ontological approach on region segmentation tasks.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>