Interactions Between Syntax and Semantics in Language Models
Name
Bau-abau-meng-eecs-2022-thesis.pdf
Description
Thesis PDF
Size
1.59 MB
Format
Adobe PDF
Checksum (MD5)
add87786775a315f8d3db6e6c09718cf
Author(s)
Bau, Anthony
Advisor(s)
Andreas, Jacob
Date Issued
May 2022
Publisher
Massachusetts Institute of Technology
Abstract
How do syntax and semantics interact in neural language models? Is the intuitive dichotomy between semantics and syntax useful as a mental model of their behavior? In this thesis, I systematically investigate how models handle the interactions between sentence corruptions of each kind. I develop a setup for generating “syntactic” and “semantic” corruptions and combining them. Using this setup, I develop a general framework for understanding how classes of corruptions interact with each other, arguing that this can be understood in terms of a single “entanglement” parameter. This allows me to quantify entanglement between syntax and semantics in a variety of situations. The result is that semantics is more entangled with syntax than it is with itself, and that syntax is highly self-entangled.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link