Transformer Pruning Relation and General Neural Network Augmentation
Name
Lim-yhlim-meng-eecs-2021-thesis.pdf
Description
Thesis PDF
Size
6.04 MB
Format
Adobe PDF
Checksum (MD5)
603dd21282e338565761e038cb60e38b
Author(s)
Lim, Yong Hui
Advisor(s)
Shavit, Nir
Date Issued
June 2021
Publisher
Massachusetts Institute of Technology
Abstract
In this thesis, a method of initializing neural networks with weights transferred from smaller trained neural network weights was investigated. We name this process augmentation and present a few versions of it, some of which involve pruning. Firstly, the pruning relation of testing loss against density was found for the GPT-2 transformer network on a causal language modeling task. An interesting double plateau of testing loss was found whenever the attention weights were pruned. Next, augmentation on low dimensional datasets and shallow networks was investigated. We found that performing a step of zeroing final layer initializations (ZFLI) results in better augmentation. With this insight, we proceeded to investigate a variety of datasets and networks. Two forms of augmentation were investigated: basic augmentation and pruned augmentation. However, both forms of augmentation were found to not produce any consistent improvement in testing accuracy/loss.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
