Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks
Name
visualDynamics18.pdf
Description
Accepted version
Size
3.33 MB
Format
Unknown
Checksum (MD5)
366aed31a0ea0d083df2c2ac01d7f8e6
Author(s)
Xue, Tianfan
Wu, Jiajun
Bouman, Katherine L
Freeman, William T
Date Issued
2019
Journal
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Version
Author's final manuscript
Abstract
IEEE We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods that have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, and on real-world video frames. We present analyses of the learned network representations, showing it is implicitly learning a compact encoding of object appearance and motion. We also demonstrate a few of its applications, including visual analogy-making and video extrapolation.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/TPAMI.2018.2854726
