A New Framework for Refraction-Based Image Verification

Abstract

We propose a novel approach to image verification that aims to unify optics, computer vision, computer graphics, and deep learning for active image protection. Our approach builds upon previous work involving placing spherical refractive objects in the scene that collectively act as a signature for authenticity, however we hypothesize that we can learn refraction models independent of scene and for arbitrary refractive objects. We develop a framework for learning such refraction models, where each model can be considered a key to authenticate an image or video. In this way, complex refraction models inherent to the physics of arbitrarily shaped objects can be used to increase security without requiring a closed form solution for their optical behavior. The approach involves scanning a laser over the scene and learning an image of its warping transformation by the refractive object. With a learned model, detecting and localizing manipulations in an image is accomplished by validating consistency between the primary, unverified image and a reconstruction based on the warped image in the object. This is demonstrated in simulation, using a photorealistic rendering engine to collect synthetic training data that captures real world behavior. We present both qualitative and quantitative results demonstrating the capabilities of our system, including computational speedups and practical improvements compared to prior work, as well as an analysis across different resolutions, model settings, and limiting factors. We demonstrate that with a sufficient sampling resolution, we can detect and localize content additions, content removals, and texture changes. Our key contribution is a novel integration of physical laws with deep learning in the context of image forensics. Further, the generalization introduced by our deep learning approach allows us to enhance image verification security.