Improving LLM Long Context Understanding via Synthetic Data and Adaptive Compression
Name
li-jerryli-meng-eecs-2024-thesis.pdf
Description
Thesis PDF
Size
1.56 MB
Format
Adobe PDF
Checksum (MD5)
098defa1fc755f3cf7df9e97b86cd698
Author(s)
Li, Jerry
Advisor(s)
Feris, Rogerio
Karlinsky, Leonid
Oliva, Aude
Date Issued
May 2024
Publisher
Massachusetts Institute of Technology
Abstract
Recent innovations in large language models (LLMs) have led to their widespread use, but the long context problem remains a fundamental challenge. Transformer-based LLMs are constrained by the quadratic scaling of the self-attention mechanism, which restricts most popular LLMs to a context length of several thousand tokens. Many methods have been introduced to extend the context of LLMs, including the Activation Beacon approach. In this work, we propose two key advancements to the existing methodology. First, we generate long context synthetic data across a variety of tasks for training context-extended models, which can supplement or even replace expensive human-annotated data. Second, we introduce a novel two-pass, adaptive compression technique for more intelligent compression of long contexts. We find that the two strategies lead to orthogonal performance improvements on real-world long context tasks, resulting in an overall 4.2% increase in accuracy compared to the previous benchmark.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
