RGB-D Likelihood for 3D Inverse Graphics
Name
Gothoskar_nishadg_SM_EECS_2023_thesis.pdf
Description
Thesis PDF
Size
18.76 MB
Format
Adobe PDF
Checksum (MD5)
c388ca87233741eabb2eddd3876fcdf8
Author(s)
Gothoskar, Nishad
Advisor(s)
Mansinghka, Vikash K.
Tenenbaum, Joshua B.
Date Issued
February 2023
Publisher
Massachusetts Institute of Technology
Abstract
A central challenge in 3D scene perception via inverse graphics is robustly modeling the gap between 3D graphics and real-world data. We propose a novel 3D Neural Embedding Likelihood (3DNEL) over RGB-D images to address this gap. 3DNEL uses neural embeddings to predict 2D-3D correspondences from RGB and combines this with depth in a principled manner. 3DNEL is trained entirely from synthetic images and generalizes to real-world data. To showcase this capability, we develop a multi-stage inverse graphics pipeline that uses 3DNEL for 6D object pose estimation from real RGB-D images. Our method outperforms the previous state-of-the-art in sim-to-real pose estimation on the YCB-Video dataset, and improves robustness, with significantly fewer large-error predictions. Unlike existing bottom-up, discriminative approaches that are specialized for pose estimation, 3DNEL adopts a probabilistic generative formulation that jointly models multi-object scenes. This generative formulation enables easy extension of 3DNEL to additional tasks like object and camera tracking from video, using principled inference in the same probabilistic model without task specific retraining.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
