Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
Name
learning_to_zoom.pdf
Description
Published version
Size
1.12 MB
Format
Adobe PDF
Checksum (MD5)
4c08d7fe9e7909dca8587ce79bc3b859
Author(s)
Recasens, Adrià
Kellnhofer, Petr
Stent, Simon
Matusik, Wojciech
Torralba, Antonio
Date Issued
2018
Publisher
Springer International Publishing
Citation
Recasens, Adrià, Kellnhofer, Petr, Stent, Simon, Matusik, Wojciech and Torralba, Antonio. 2018. "Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks."
Version
Final published version
Abstract
© Springer Nature Switzerland AG 2018. We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much smaller, our layer learns how best to sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This has the effect of creating distorted, caricature-like intermediate images, in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such as spatial transformer networks, our proposed layer is inspired by image saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We apply our layer to improve existing networks for the tasks of human gaze estimation and fine-grained object classification. Code for our method is available in: http://github.com/recasens/Saliency-Sampler.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/978-3-030-01240-3_4
