Revealing and analyzing imperceptible deviations in images and videos

Abstract

The world is filled with objects that appear to follow some perfect model. A sleeping baby might look still and a house's roof .should be straight. However, both the baby and the roof can deviate subtly from their ideal models of perfect stillness and perfect straightness. These deviations can reveal important information like whether the baby is breathing normally or whether the house's roof is sagging. In this dissertation, we make the observation that these subtle deviations produce a visual signal that while invisible to the naked eye can be extracted from ordinary and ubiquitous images and videos. We propose new computational techniques to reveal these subtle deviations by producing new images and videos, in which the tiny deviations have been magnified. We focus on magnifying deviations from two ideal models: perfect stillness and perfect geometries in space. In the first case, we leverage the complex steerable pyramid, a localized version of the Fourier transform, whose notion of local phase can be used to process and manipulate small motions or changes from stillness in videos. In the second case, we find hidden geometric deformations in images by localizing edges to sub-pixel precision. In both cases, we experimentally validate that the tiny deviations we magnify are indeed real, comparing them to alternative ways of measuring tiny motions and subtle geometric deformations in the world. We also give a careful analysis of how noise in videos impacts our ability to see tiny motions. Additionally, we show the utility of revealing hidden deviations in a wide variety of fields, such as biology, physics and structural analysis.