2026-09-20T11:38:50Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1298812026-06-06T00:49:26Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

SueYeon Chung. Del Río Fernández, Miguel Ángel. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2021-02-19T20:36:32Z 2021-02-19T20:36:32Z 2020 2020 https://hdl.handle.net/1721.1/129881 1237411406 Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, February, 2020 Cataloged from student-submitted PDF of thesis. Pages 166 and 167 are blank. Includes bibliographical references (pages 161-165). Recent success of state-of-the-art neural models on various natural language processing (NLP) tasks has spurred interest in understanding their representation space. In the following chapters we will use various techniques of representational analysis to understand the nature of neural-network based language modelling. To introduce the concept of linguistic probing, we explore how various language features affect model representations and long-term behavior through the use of linear probing techniques. To tease out the geometrical properties of BERT's internal representations, we task the model with 5 linguistic abstractions (word, part-of-speech, combinatory categorical grammar, dependency parse tree depth, and semantic tag). By using a Mean Field theory backed manifold capacity (MFT) metric, we show that BERT entangles linguistic information when contextualizing a normal sentence but detangles the same information when it must form a token prediction. To mend our findings to those of previous works that used linear probing, we reproduce the prior results and show that linear separation between classes follows the trends we present. To show that linguistic structure of a sentence is being geometrically embedded in BERT representations, we swap words in sentences such that the underlying tree structure becomes perturbed. By using canonical correlation analysis (CCA) to compare sentence representations, we find that the distance between swapped words is directly proportional to the decrease in geometric similarity of model representations. Miguel Ángel Del Río Fernández. M.Eng. M.Eng. Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science 167 pages eng Massachusetts Institute of Technology MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided. http://dspace.mit.edu/handle/1721.1/7582 Electrical Engineering and Computer Science. Structure and geometry in sequence-processing neural networks Thesis application/pdf 2021-02-19T20:36:02Z Publication Master EECS unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="46ed45b3-c1dd-45cf-8972-f4a908b1f80f"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>eng</Language> <Title>Structure and geometry in sequence-processing neural networks</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2020</PublicationDate> <Authors> <Author> <DisplayName>Del Río Fernández, Miguel Ángel.</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://dspace.mit.edu/handle/1721.1/7582</License> <Keyword>Electrical Engineering and Computer Science.</Keyword> <Abstract>Recent success of state-of-the-art neural models on various natural language processing (NLP) tasks has spurred interest in understanding their representation space. In the following chapters we will use various techniques of representational analysis to understand the nature of neural-network based language modelling. To introduce the concept of linguistic probing, we explore how various language features affect model representations and long-term behavior through the use of linear probing techniques. To tease out the geometrical properties of BERT&apos;s internal representations, we task the model with 5 linguistic abstractions (word, part-of-speech, combinatory categorical grammar, dependency parse tree depth, and semantic tag). By using a Mean Field theory backed manifold capacity (MFT) metric, we show that BERT entangles linguistic information when contextualizing a normal sentence but detangles the same information when it must form a token prediction. To mend our findings to those of previous works that used linear probing, we reproduce the prior results and show that linear separation between classes follows the trends we present. To show that linguistic structure of a sentence is being geometrically embedded in BERT representations, we swap words in sentences such that the underlying tree structure becomes perturbed. By using canonical correlation analysis (CCA) to compare sentence representations, we find that the distance between swapped words is directly proportional to the decrease in geometric similarity of model representations.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>