2026-09-19T05:27:29Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1563322024-08-22T03:30:13Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Han, Song
Xiao, Guangxuan
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2024-08-21T18:57:28Z
2024-08-21T18:57:28Z
2024-05
2024-07-10T13:00:03.136Z
https://hdl.handle.net/1721.1/156332
https://orcid.org/0000-0002-7182-9284
Large language models (LLMs) have achieved impressive performance on various natural language tasks. However, their massive computational and memory requirements hinder widespread deployment. Additionally, deploying them on extensive inputs presents efficiency and accuracy challenges.
This proposal introduces two techniques to enable efficient and accurate quantization and streaming deployment of LLMs, facilitating their application in real-world systems with limited resources. First, we develop SmoothQuant, an accurate post-training 8-bit quantization method of both weights and activations in LLMs up to 530B parameters. By smoothing outliers in activations, SmoothQuant enables the use of efficient INT8 kernels on all matrix multiplications with negligible accuracy loss. Second, we present StreamingLLM, enabling LLMs to handle arbitrarily long text sequences using a fixed memory budget. It exploits ``attention sinks'' in LLMs to stably anchor attention computation on lengthy contexts. Experiments show StreamingLLM can model over 4 million tokens with up to 22x speedup compared to recomputation baselines.
Together, these two techniques can significantly reduce the computational and memory costs of large language models, increasing their accessibility for practical usage.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
Efficient Deployment Algorithms for Large Language Models
Thesis
application/pdf
Master
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="981bab68-7077-4493-8d6b-7948bbe04663">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Efficient Deployment Algorithms for Large Language Models</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2024-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Xiao, Guangxuan</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Large language models (LLMs) have achieved impressive performance on various natural language tasks. However, their massive computational and memory requirements hinder widespread deployment. Additionally, deploying them on extensive inputs presents efficiency and accuracy challenges.
This proposal introduces two techniques to enable efficient and accurate quantization and streaming deployment of LLMs, facilitating their application in real-world systems with limited resources. First, we develop SmoothQuant, an accurate post-training 8-bit quantization method of both weights and activations in LLMs up to 530B parameters. By smoothing outliers in activations, SmoothQuant enables the use of efficient INT8 kernels on all matrix multiplications with negligible accuracy loss. Second, we present StreamingLLM, enabling LLMs to handle arbitrarily long text sequences using a fixed memory budget. It exploits ``attention sinks'' in LLMs to stably anchor attention computation on lengthy contexts. Experiments show StreamingLLM can model over 4 million tokens with up to 22x speedup compared to recomputation baselines.
Together, these two techniques can significantly reduce the computational and memory costs of large language models, increasing their accessibility for practical usage.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>