Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
Author(s)
Xue, Tianfan; Wu, Jiajun; Bouman, Katherine L.; Freeman, William T.
DownloadPublished version (1.226Mb)
Terms of use
Metadata
Show full item recordAbstract
© 2016 NIPS Foundation - All Rights Reserved. We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, as well as on real-world video frames. We also show that our model can be applied to visual analogy-making, and present an analysis of the learned network representations.
Date issued
2016Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer ScienceCitation
Xue, Tianfan, Wu, Jiajun, Bouman, Katherine L. and Freeman, William T. 2016. "Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks."
Version: Final published version