Computational visual reality

Abstract

It is not so far-fetched to envision a future student working through a difficult physics problem by using their hands to manipulate a 3D visualization that floats above the desk. A doctor preparing for heart surgery will rehearse on a photo-real replica of his patient's organ. A visitor to the British Museum in London will sketch a golden Pharaoh's headdress, illuminated by a ray of sunlight pouring in the window, never aware that the physical artifact is still in Egypt. Though such scenarios may seem cut from the pages of science fiction, this thesis illuminates a path to making them possible. To create more realistic and interactive visual information, displays must show high quality 3D images that respond to environmental lighting conditions and user input. The availability of displays capable of addressing the full range of visual experience will improve our ability to interact with computation, the world, and one another. Two of the many problems that have impeded previous efforts to design high-dimensional displays are the need to: 1. process large amounts of information in realtime; and 2. fabricate hardware capable of conveying that information. Light field capture and display is enormously data-intensive, but by applying compressive techniques that take advantage of multiple data redundancies in light transport, it is possible to overcome these challenges and make use of hardware available in the near-term. This thesis proposes display and capture frameworks that use non-negative tensor factorization and dictionary-based sparse reconstruction, respectively, in conjunction with the co-design of algorithms, optics, and electronics to allow compressive, simultaneous, light field display and capture.