Mesh regularization for multi-view shape reconstruction via inverse graphics

Abstract

Inverse rendering uses 2D images to infer the 3D scene parameters that produced the images. Unfortunately, inverse rendering is an ill-posed problem that is difficult to optimize. In this paper, we explore using a differentiable renderer to solve inverse rendering problems that are constrained to a single object with two unknowns: shape and texture. The inferred 3D scene is thus a single 3D model. We iteratively optimize an input "guess" 3D model to fit a set of input target images of the desired 3D object. The constrained inverse rendering problem remains ill-posed. So, we adapt, develop, and evaluate a number of deformation, regularization, and training techniques that allow the optimization to converge to visually appealing output 3D models. The system consumes a 3D triangle mesh and target images as input. It outputs a 3D mesh and its corresponding 2D texture image that can easily be consumed by other programs. The best results are achieved using a deep-mesh prior neural network, an Image Pyramid coarse-to-fine loss function, a Silhouette Loss function that is robust to inaccuracies in texture, three separate mesh regularizing loss functions - Laplacian, Flatness, and Non-Uniformity, and periodic refinement operations where the output mesh is upsampled and its manifold is recomputed.