2026-09-20T00:23:54Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1567512024-09-17T03:49:37Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Raghuraman, Srinivasan
Fuangkawinsombut, Siwakorn
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2024-09-16T13:46:57Z
2024-09-16T13:46:57Z
2024-05
2024-07-11T14:36:39.212Z
https://hdl.handle.net/1721.1/156751
In the domain of machine learning, "grokking" is a phenomenon where neural network models demonstrate a sudden improvement in generalization, distinct from traditional learning phases, long after the initial training appears complete. This behavior was first identified by Power et al. (2022) [5]. This thesis explores grokking within the context of the (𝑛, 𝑘)-parity problem, aiming to uncover the mechanisms that trigger such transitions. Through extensive empirical research, we examine how different neural network configurations and training conditions influence the onset of grokking. Our methodology integrates advanced visualization techniques, such as t-SNE, and kernel density estimations to track the evolution from memorization to generalization phases. Furthermore, we investigate the roles of weight decay and network robustness against outliers, focusing on optimizing neural network architectures to achieve effective generalization with fewer computational resources. This study advances our understanding of grokking and proposes practical strategies for designing more efficient neural networks.
M.Eng.
Massachusetts Institute of Technology
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Copyright retained by author(s)
https://creativecommons.org/licenses/by-nc-nd/4.0/
Beyond Memorization: Exploring the Dynamics of Grokking in Sparse Neural Networks
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="420098a7-7129-492d-91f4-2746af40ed32">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Beyond Memorization: Exploring the Dynamics of Grokking in Sparse Neural Networks</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2024-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Fuangkawinsombut, Siwakorn</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://creativecommons.org/licenses/by-nc-nd/4.0/</License>
<Abstract>In the domain of machine learning, "grokking" is a phenomenon where neural network models demonstrate a sudden improvement in generalization, distinct from traditional learning phases, long after the initial training appears complete. This behavior was first identified by Power et al. (2022) [5]. This thesis explores grokking within the context of the (𝑛, 𝑘)-parity problem, aiming to uncover the mechanisms that trigger such transitions. Through extensive empirical research, we examine how different neural network configurations and training conditions influence the onset of grokking. Our methodology integrates advanced visualization techniques, such as t-SNE, and kernel density estimations to track the evolution from memorization to generalization phases. Furthermore, we investigate the roles of weight decay and network robustness against outliers, focusing on optimizing neural network architectures to achieve effective generalization with fewer computational resources. This study advances our understanding of grokking and proposes practical strategies for designing more efficient neural networks.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>