MIT Libraries logoDSpace@MIT

MIT
View Item 
  • DSpace@MIT Home
  • MIT Libraries
  • MIT Theses
  • Graduate Theses
  • View Item
  • DSpace@MIT Home
  • MIT Libraries
  • MIT Theses
  • Graduate Theses
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

Comparing Parameter Efficient Finetuning Techniques (PEFT) using Datamodels

Author(s)
Chamdal, Harshal
Thumbnail
DownloadThesis PDF (669.0Kb)
Advisor
Mądry, Aleksander
Terms of use
In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/
Metadata
Show full item record
Abstract
Advances in machine learning, particularly through algorithmic innovations and large datasets, have led to models with hundreds of billions of parameters. Deploying these models is challenging and costly, especially due to the extensive finetuning required. Parameter-efficient finetuning techniques (PEFT) have been proposed to address this issue by significantly reducing the number of trainable parameters, achieving comparable results to full-parameter finetuning. Despite widespread adoption, PEFT methods are often used interchangeably without considering their qualitative differences and performance under various data distributions. This thesis extensively compares three PEFT methods: LoRA, BitFit, and (IA)³, using the ModelDiff framework to identify and apply data interventions. Our analysis reveals that the performance of these methods varies widely with different interventions, with BitFit showing the most variance, while LoRA and (IA)³ demonstrate greater resilience. This study informs the selection and optimization of PEFT techniques based on specific NLP task requirements, balancing performance, computational efficiency, and robustness to text variations.
Date issued
2024-05
URI
https://hdl.handle.net/1721.1/157345
Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Publisher
Massachusetts Institute of Technology

Collections
  • Graduate Theses

Browse

All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login

Statistics

OA StatisticsStatistics by CountryStatistics by Department
MIT Libraries
PrivacyPermissionsAccessibilityContact us
MIT
Content created by the MIT Libraries, CC BY-NC unless otherwise noted. Notify us about copyright concerns.