2026-09-18T20:36:29Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1500822023-04-01T03:11:04Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Mansinghka, Vikash K.
Tenenbaum, Joshua B.
Gothoskar, Nishad
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2023-03-31T14:30:36Z
2023-03-31T14:30:36Z
2023-02
2023-02-28T14:36:03.121Z
https://hdl.handle.net/1721.1/150082
A central challenge in 3D scene perception via inverse graphics is robustly modeling the gap between 3D graphics and real-world data. We propose a novel 3D Neural Embedding Likelihood (3DNEL) over RGB-D images to address this gap. 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in a principled manner. 3DNEL is trained entirely from synthetic images and generalizes to real-world data. To showcase this capability, we develop a multi-stage inverse graphics pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images. Our method outperforms the previous state-of-the-art in sim-to-real pose estimation on the YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions. Unlike existing bottom-up, discriminative approaches that are specialized for pose estimation, 3DNEL adopts a probabilistic generative formulation that jointly models multi-object scenes. This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
RGB-D Likelihood for 3D Inverse Graphics
Thesis
application/pdf
Master
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="cb1cf97b-7234-401e-ae84-f23392e8eda7">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>RGB-D Likelihood for 3D Inverse Graphics</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2023-02</PublicationDate>
<Authors>
<Author>
<DisplayName>Gothoskar, Nishad</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>A central challenge in 3D scene perception via inverse graphics is robustly modeling the gap between 3D graphics and real-world data. We propose a novel 3D Neural Embedding Likelihood (3DNEL) over RGB-D images to address this gap. 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in a principled manner. 3DNEL is trained entirely from synthetic images and generalizes to real-world data. To showcase this capability, we develop a multi-stage inverse graphics pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images. Our method outperforms the previous state-of-the-art in sim-to-real pose estimation on the YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions. Unlike existing bottom-up, discriminative approaches that are specialized for pose estimation, 3DNEL adopts a probabilistic generative formulation that jointly models multi-object scenes. This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>