Beyond Memorization: Exploring the Dynamics of Grokking in Sparse Neural Networks
Name
fuangkawinsombut-taisf-meng-eecs-2024-thesis.pdf
Description
Thesis PDF
Size
1.6 MB
Format
Adobe PDF
Checksum (MD5)
1215280b853bb8d0d7fbeac6c5af6a64
Author(s)
Fuangkawinsombut, Siwakorn
Advisor(s)
Raghuraman, Srinivasan
Date Issued
May 2024
Publisher
Massachusetts Institute of Technology
Abstract
In the domain of machine learning, "grokking" is a phenomenon where neural network models demonstrate a sudden improvement in generalization, distinct from traditional learning phases, long after the initial training appears complete. This behavior was first identified by Power et al. (2022) [5]. This thesis explores grokking within the context of the (𝑛, 𝑘)-parity problem, aiming to uncover the mechanisms that trigger such transitions. Through extensive empirical research, we examine how different neural network configurations and training conditions influence the onset of grokking. Our methodology integrates advanced visualization techniques, such as t-SNE, and kernel density estimations to track the evolution from memorization to generalization phases. Furthermore, we investigate the roles of weight decay and network robustness against outliers, focusing on optimizing neural network architectures to achieve effective generalization with fewer computational resources. This study advances our understanding of grokking and proposes practical strategies for designing more efficient neural networks.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Copyright retained by author(s)
Persistent DSpace Link
