Classical generalization bounds are surprisingly tight for Deep Networks
Name
CBMM-Memo-091.pdf
Size
1.43 MB
Format
Adobe PDF
Checksum (MD5)
607867655c0d8eb866279cd4770de66c
Author(s)
Liao, Qianli
Miranda, Brando
Hidary, Jack
Poggio, Tomaso
Date Issued
July 11, 2018
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo Series;091
Abstract
Deep networks are usually trained and tested in a regime in which the training classification error is not a good predictor of the test error. Thus the consensus has been that generalization, defined as convergence of the empirical to the expected error, does not hold for deep networks. Here we show that, when normalized appropriately after training, deep networks trained on exponential type losses show a good linear dependence of test loss on training loss. The observation, motivated by a previous theoretical analysis of overparameterization and overfitting, not only demonstrates the validity of classical generalization bounds for deep learning but suggests that they are tight. In addition, we also show that the bound of the classification error by the normalized cross entropy loss is empirically rather tight on the data sets we studied.
Persistent DSpace Link
