Towards Passing Visual Turing Test with Computational 3D Displays and Appearance Modeling

Abstract

In Neal Stephenson’s 1992 science-fiction novel "Snow Crash," the term "metaverse" was coined to describe a virtually persistent world in which humans, as avatars, interact with each other and software agents just as they would in the physical world. This is no longer science fiction with the rapid development of virtual and augmented reality devices in the past decade. To stimulate future development and research, Meta Inc. (previously Facebook) defined a grand challenge termed visual Turing test in 2022, asking if the virtual experience can be created as vivid and realistic as the real-world such that a human observer can no longer distinguish. Passing this test requires significant technological advancements in areas such as display, rendering, acquisition, sensing, modeling, and simulation. The focus of this thesis is on developing next-generation computational 3D near-eye displays to deliver a more realistic and natural 3D experience, as well as devising inverse procedural material (PM) modeling and capture frameworks that allow non-expert users to bring real-world appearances into a digital twin in a few simple clicks.

Existing near-eye 3D displays rely on the stereoscopic principle by showing parallax images to deceive the brain into perceiving depth from 2D images. This deviates from the experience of how humans observe the real world, causing visual discomfort and vision degradation after prolonged use. Computer-generated holography (CGH) is widely regarded as the next-generation technology capable of utilizing light interference and diffraction to produce true 3D graphical projections. However, its adoption was fundamentally limited by the low image quality and vast computational cost. Similarly, PMs have recently emerged as the industry standard for representing physically-based, spatially-varying appearance due to their unlimited resolution, editability, and memory efficiency. Nonetheless, the extensive domain knowledge and labor required to master and author PMs have made it difficult for non-experts to create and operate them.

This thesis exploits recent advances in machine learning, optimization, rendering, and differentiable software infrastructures to develop algorithms and user-friendly tools for synthesizing, compressing, encoding/decoding, and deploying photorealistic 3D holograms, as well as for authoring, editing, and manipulating procedural materials. The ultimate objective is to pass the visual turning test and democratize the creation of a metaverse for ordinary people.

At the very end of this thesis, an extrapolation of the Virtual Turing Test is applied for physical reproduction, and a novel 3D printing system that fabricates painting facsimiles faithfully to the originals will be discussed, as well as the potential of this technology to help culture preservation and education.