2026-09-18T17:37:48Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1431962022-06-16T03:07:46Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Jegelka, Stefanie
Le, Thien
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2022-06-15T13:02:50Z
2022-06-15T13:02:50Z
2022-02
2022-03-04T20:59:51.688Z
https://hdl.handle.net/1721.1/143196
Deep neural networks are the main subject of interest in the study of theoretical deep learning, which aims to rigorously explain the incredible performance of these function classes in practice. Although a lot are understood about deep linear network (neural network with all linear activations), nonlinearities in the activation make it very challenging to extend existing techniques to more realistic types of neural networks. In this thesis, we describe reductions of ReLU neural networks to linear neural networks, under various general condition of network architectures, loss functions and datasets. When such conditions are met, one can adapt techniques used in the theory of linear neural networks to study ReLU neural networks. To this end, we provide two applications in which we put the reduction to use: the first characterizes an implicit regularization behavior of ReLU neural networks trained with gradient descent and the second characterizes their convergence under gradient-based algorithms.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
Reductions of ReLU neural networks to linear neural networks and their applications
Thesis
application/pdf
Master
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="d502ce31-934b-4196-b1aa-1f25844d5852">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Reductions of ReLU neural networks to linear neural networks and their applications</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2022-02</PublicationDate>
<Authors>
<Author>
<DisplayName>Le, Thien</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Deep neural networks are the main subject of interest in the study of theoretical deep learning, which aims to rigorously explain the incredible performance of these function classes in practice. Although a lot are understood about deep linear network (neural network with all linear activations), nonlinearities in the activation make it very challenging to extend existing techniques to more realistic types of neural networks. In this thesis, we describe reductions of ReLU neural networks to linear neural networks, under various general condition of network architectures, loss functions and datasets. When such conditions are met, one can adapt techniques used in the theory of linear neural networks to study ReLU neural networks. To this end, we provide two applications in which we put the reduction to use: the first characterizes an implicit regularization behavior of ReLU neural networks trained with gradient descent and the second characterizes their convergence under gradient-based algorithms.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>