An Empirical and Theoretical Analysis of the Role of Depth in Convolutional Neural Networks
Name
Nichani-eshnich-meng-eecs-2021-thesis.pdf
Description
Thesis PDF
Size
1.61 MB
Format
Adobe PDF
Checksum (MD5)
cb9181221a3b04764dcdcb59a7f7c4bb
Author(s)
Nichani, Eshaan
Advisor(s)
Uhler, Caroline
Date Issued
June 2021
Publisher
Massachusetts Institute of Technology
Abstract
While over-parameterized neural networks are capable of perfectly fitting (interpolating) training data, these networks often perform well on test data, thereby contradicting classical learning theory. Recent work provided an explanation for this phenomenon by introducing the double descent curve, showing that increasing model capacity past the interpolation threshold can lead to a decrease in test error. In line with this, it was recently shown empirically and theoretically that increasing neural network capacity through width leads to double descent. In this thesis, we analyze the effect of increasing depth on test performance. In contrast to what is observed for increasing width, we demonstrate through a variety of classification experiments on CIFAR10 and ImageNet32 using fully-convolutional nets, ResNets and the convolutional neural tangent kernel (CNTK) that test performance is U-shaped and in fact worsens beyond a critical depth. To better understand this phenomenon, we conduct a theoretical analysis on the impact of depth on generalization in linear convolutional networks of infinite width. In particular, we derive the feature map for the linear CNTK for arbitrary depths and identify the depth which minimizes the bias and variance terms of the excess risk. The findings of this thesis imply that increasing depth for interpolating convolutional networks can in fact lead to worse generalization.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
