Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Libraries
  3. MIT Theses
  4. Graduate Theses
  5. Long Sequence Transformer Variants on Varying Context Length

Long Sequence Transformer Variants on Varying Context Length

Thumbnail Image
Download
Name

sun-mmsun-meng-eecs-2023-thesis.pdf

Description
Thesis PDF
Size

498.06 KB

Format

Adobe PDF

Checksum (MD5)

35d9157ebc30b4642da3fb86cf7ac350

Author(s)
Sun, Melinda
Advisor(s)
Kim, Yoon
Date Issued
September 2023
Publisher
Massachusetts Institute of Technology
Abstract
Transformers are powerful and effective tools in natural language processing, but their scalability is limited by the quadratic complexity of attention. Several transformer variants that address this problem have recently been proposed, including Moving Average Equipped Gated Attention (Mega). In this thesis, we evaluate how effectively Mega uses past context, by comparing the perplexity trend as context length varies with the perplexity trend of a standard transformer. We find that Mega does not show greater benefit from longer context in a Wikipedia or book setting, though it does have a much better ability to extrapolate beyond training context lengths.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
https://rightsstatements.org/page/InC-EDU/1.0/
Copyright retained by author(s)
Persistent DSpace Link
https://hdl.handle.net/1721.1/152839
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.