Evaluating Data Augmentation with Attention Masks for Context Aware Transformations
Name
marquez-sofiam-meng-bcs-2024-thesis.pdf
Description
Thesis PDF
Size
317.4 KB
Format
Adobe PDF
Checksum (MD5)
9cee83585f3514a1b84c7b7dff0cdc63
Author(s)
Marquez, Sofia M.
Advisor(s)
Murray, Fiona
Date Issued
February 2024
Publisher
Massachusetts Institute of Technology
Abstract
Transfer learning from large, pre-trained models and data augmentation are arguably the two most widespread solutions to the problem of data scarcity. However, both methods suffer from limitations that prevent more optimal solutions to natural language processing tasks. We consider that transfer learning benefits from fine-tuning on increased target dataset size, and that data augmentation benefits from applying transformations in a selective, rather than random, manner. Thus, this work evaluates a new augmentation paradigm that uses the attention masks of pre-trained transformers to more effectively apply text transformations in high-importance locations, creating augmentations which can be used for further finetuning. Our comprehensive analysis points to limited success of utilizing this context-aware augmentation method. By shedding light on its strengths and limitations, we offer insights that can guide the selection of optimal augmentation techniques for a variey of models, and lay groundwork for further research in the pursuit of effective solutions for natural language processing tasks under data constraints.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
