Novel View Synthesis from Casually Recorded Videos

Abstract

Generating new, photorealistic views of a scene given only a single video is a difficult task that computer vision researchers have worked on for decades. This problem has recently seen a resurgence in interest due to its potential application in areas such as virtual reality. However, current novel view synthesis techniques are not suitable for the short, casual videos that people typically record. Such videos deviate from the setups that these approaches typically use, where there are dense, high-resolution images of the scene. In this paper, we propose a method for refining an initial, coarse scene geometry which we then use for novel view synthesis on short video sequences. The core of our method is a geometry refinement step where we project the geometry to source views to remove inconsistent points. This refined geometry provides important shape and appearance information in data poor regions that would otherwise be difficult to accurately render. We evaluate our approach on the RealEstate10K dataset and demonstrate that compared to prior work, we synthesize views that are more temporally consistent.