Cross-validation Stability of Deep Networks
Name
CBMM-Memo-115.pdf
Size
9.56 MB
Format
Adobe PDF
Checksum (MD5)
0f397cff8b1fd03238efe61ccc0dcd38
Author(s)
Banburski, Andrzej
De La Torre, Fernanda
Plant, Nishka
Shastri, Ishana
Poggio, Tomaso
Date Issued
February 9, 2021
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;115
Abstract
Recent theoretical results show that gradient descent on deep neural networks under exponential loss functions locally maximizes classification margin, which is equivalent to minimizing the norm of the weight matrices under margin constraints. This property of the solution however does not fully characterize the generalization performance. We motivate theoretically and show empirically that the area under the curve of the margin distribution on the training set is in fact a good measure of generalization. We then show that, after data separation is achieved, it is possible to dynamically reduce the training set by more than 99% without significant loss of performance. Interestingly, the resulting subset of “high capacity” features is not consistent across different training runs, which is consistent with the theoretical claim that all training points should converge to the same asymptotic margin under SGD and in the presence of both batch normalization and weight decay.
Persistent DSpace Link
