Scaling Laws for Deep Learning
Name
Rosenfeld-jonsr-PhD-EECS-2021-thesis.pdf
Description
Thesis PDF
Size
4.43 MB
Format
Adobe PDF
Checksum (MD5)
1e194f7d5aebf6fc84085f7d2dec4264
Author(s)
Rosenfeld, Jonathan S.
Advisor(s)
Shavit, Nir
Date Issued
September 2021
Publisher
Massachusetts Institute of Technology
Abstract
Running faster will only get you so far — it is generally advisable to first understand where the roads lead, then get a car ...
The renaissance of machine learning (ML) and deep learning (DL) over the last decade is accompanied by an unscalable computational cost, limiting its advancement and weighing on the field in practice. In this thesis we take a systematic approach to address the algorithmic and methodological limitations at the root of these costs. We first demonstrate that DL training and pruning are predictable and governed by scaling laws — for state of the art models and tasks, spanning image classification and language modeling, as well as for state of the art model compression via iterative pruning. Predictability, via the establishment of these scaling laws, provides the path for principled design and trade-off reasoning, currently largely lacking in the field. We then continue to analyze the sources of the scaling laws, offering an approximation-theoretic view and showing through the exploration of a noiseless realizable case that DL is in fact dominated by error sources very far from the lower error limit. We conclude by building on the gained theoretical understanding of the scaling laws’ origins. We present a conjectural path to eliminate one of the current dominant error sources — through a data bandwidth limiting hypothesis and the introduction of Nyquist learners — which can, in principle, reach the generalization error lower limit (e.g. 0 in the noiseless case), at finite dataset size.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
