Feature learning in deep classifiers through Intermediate Neural Collapse
Name
CBMM-Memo-141.pdf
Size
2.16 MB
Format
Adobe PDF
Checksum (MD5)
f7220e7ae18e04663c325fdff41c5cc4
Author(s)
Rangamani, Akshay
Lindegaard, Marius
Galanti, Tomer
Poggio, Tomaso
Date Issued
February 27, 2023
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;141
Abstract
In this paper, we conduct an empirical study of the feature learning process in deep classifiers. Recent research has identified a training phenomenon called Neural Collapse (NC), in which the top-layer feature embeddings of samples from the same class tend to concentrate around their means, and the top layer’s weights align with those features. Our study aims to investigate if these properties extend to intermediate layers. We empirically study the evolution of the covariance and mean of representations across different layers and show that as we move deeper into a trained neural network, the within-class covariance decreases relative to the between-class covariance. Additionally, we find that in the top layers, where the between-class covariance is dominant, the subspace spanned by the class means aligns with the subspace spanned by the most significant singular vector components of the weight matrix in the corresponding layer. Finally, we discuss the relationship between NC and Associative Memories.
Persistent DSpace Link
