This is not the latest version of this item. The latest version can be found here.
The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions
Name
fncom-16-760085.pdf
Size
1.73 MB
Format
Adobe PDF
Checksum (MD5)
070080904dbef641502b8d082f89c79c
Author(s)
D'Amario, Vanessa
Srivastava, Sanjana
Sasaki, Tomotake
Boix, Xavier
Date Issued
January 31, 2022
Journal
Frontiers in Computational Neuroscience
Publisher
Frontiers Media SA
Citation
D'Amario, Vanessa, Srivastava, Sanjana, Sasaki, Tomotake and Boix, Xavier. 2022. "The Data Efficiency of Deep Learning Is Degraded by Unnecessary Input Dimensions." Frontiers in Computational Neuroscience, 16.
Version
Final published version
Abstract
Biological learning systems are outstanding in their ability to learn from limited training data compared to the most successful learning machines, i.e., Deep Neural Networks (DNNs). What are the key aspects that underlie this data efficiency gap is an unresolved question at the core of biological and artificial intelligence. We hypothesize that one important aspect is that biological systems rely on mechanisms such as foveations in order to reduce unnecessary input dimensions for the task at hand, e.g., background in object recognition, while state-of-the-art DNNs do not. Datasets to train DNNs often contain such unnecessary input dimensions, and these lead to more trainable parameters. Yet, it is not clear whether this affects the DNNs' data efficiency because DNNs are robust to increasing the number of parameters in the hidden layers, and it is uncertain whether this holds true for the input layer. In this paper, we investigate the impact of unnecessary input dimensions on the DNNs data efficiency, namely, the amount of examples needed to achieve certain generalization performance. Our results show that unnecessary input dimensions that are task-unrelated substantially degrade data efficiency. This highlights the need for mechanisms that remove task-unrelated dimensions, such as foveation for image classification, in order to enable data efficiency gains.
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
10.3389/fncom.2022.760085
