Neural Light Transport for Relighting and View Synthesis
Name
3446328.pdf
Size
33.77 MB
Format
Adobe PDF
Checksum (MD5)
dbbf531239778dcceebe900675b2428d
Author(s)
Zhang, Xiuming
Fanello, Sean
Tsai, Yun-Ta
Sun, Tiancheng
Xue, Tianfan
Pandey?, Rohit
Orts-Escolano, Sergio
Davidson?, Philip
Rhemann, Christoph
Debevec?, Paul
Date Issued
January 18, 2021
Journal
ACM Transactions on Graphics
Publisher
Association for Computing Machinery
Citation
Zhang, Xiuming, Fanello, Sean, Tsai, Yun-Ta, Sun, Tiancheng, Xue, Tianfan et al. 2021. "Neural Light Transport for Relighting and View Synthesis." ACM Transactions on Graphics, 40 (1).
Version
Final published version
Abstract
The light transport (LT) of a scene describes how it appears under different lighting conditions from different viewing directions, and complete knowledge of a scene?s LT enables the synthesis of novel views under arbitrary lighting. In this paper, we focus on image-based LT acquisition, primarily for human bodies within a light stage setup. We propose a semi-parametric approach for learning a neural representation of the LT that is embedded in a texture atlas of known but possibly rough geometry. We model all non-diffuse and global LT as residuals added to a physically-based diffuse base rendering. In particular, we show how to fuse previously seen observations of illuminants and views to synthesize a new image of the same scene under a desired lighting condition from a chosen viewpoint. This strategy allows the network to learn complex material effects (such as subsurface scattering) and global illumination (such as diffuse interreflection), while guaranteeing the physical correctness of the diffuse LT (such as hard shadows). With this learned LT, one can relight the scene photorealistically with a directional light or an HDRI map, synthesize novel views with view-dependent effects, or do both simultaneously, all in a unified framework using a set of sparse observations.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3446328
