Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Libraries
  3. MIT Theses
  4. Graduate Theses
  5. Reductions of ReLU neural networks to linear neural networks and their applications

Reductions of ReLU neural networks to linear neural networks and their applications

Thumbnail Image
Download
Name

Le-thienle-SM-EECS-2022-thesis.pdf

Description
Thesis PDF
Size

447.75 KB

Format

Adobe PDF

Checksum (MD5)

ec18118100103c40a126775e75c05a40

Author(s)
Le, Thien
Advisor(s)
Jegelka, Stefanie
Date Issued
February 2022
Publisher
Massachusetts Institute of Technology
Abstract
Deep neural networks are the main subject of interest in the study of theoretical deep learning, which aims to rigorously explain the incredible performance of these function classes in practice. Although a lot are understood about deep linear network (neural network with all linear activations), nonlinearities in the activation make it very challenging to extend existing techniques to more realistic types of neural networks. In this thesis, we describe reductions of ReLU neural networks to linear neural networks, under various general condition of network architectures, loss functions and datasets. When such conditions are met, one can adapt techniques used in the theory of linear neural networks to study ReLU neural networks. To this end, we provide two applications in which we put the reduction to use: the first characterizes an implicit regularization behavior of ReLU neural networks trained with gradient descent and the second characterizes their convergence under gradient-based algorithms.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
http://rightsstatements.org/page/InC-EDU/1.0/
Copyright MIT
Persistent DSpace Link
https://hdl.handle.net/1721.1/143196
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.