2026-09-19T13:32:50Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1277042026-06-06T00:49:11Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Robert C. Berwick.
Korsky, Samuel A.
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2020-09-25T20:03:17Z
2020-09-25T20:03:17Z
2019
2019
https://hdl.handle.net/1721.1/127704
1196234132
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2019
Cataloged from student-submitted PDF of thesis.
Includes bibliographical references (page 27).
Recent neural network architectures such as the basic recurrent neural network (RNN) and Gated Recurrent Unit (GRU) have gained prominence as end-to-end learning architectures for natural language processing tasks. But what is the computational power of such systems? We prove that finite precision RNNs with one hidden layer and ReLU activation and finite precision GRUs are exactly as computationally powerful as deterministic nice automata. Allowing arbitrary precision, we prove that RNNs with one hidden layer and ReLU activation are at least as computationally powerful as pushdown automata. If we also allow infinite precision, infinite edge weights, and nonlinear output activation functions, we prove that GRUs are at least as computationally powerful as pushdown automata. All results are shown constructively.
by Samuel A. Korsky.
M.Eng.
M.Eng. Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science
27 pages
eng
Massachusetts Institute of Technology
MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided.
http://dspace.mit.edu/handle/1721.1/7582
Electrical Engineering and Computer Science.
On the computational power of RNNs
Thesis
application/pdf
2020-09-25T20:03:16Z
Publication
Master
EECS
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="3536de00-afbf-434f-ab23-dfd5c45dddb5">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Language>eng</Language>
<Title>On the computational power of RNNs</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2019</PublicationDate>
<Authors>
<Author>
<DisplayName>Korsky, Samuel A.</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://dspace.mit.edu/handle/1721.1/7582</License>
<Keyword>Electrical Engineering and Computer Science.</Keyword>
<Abstract>Recent neural network architectures such as the basic recurrent neural network (RNN) and Gated Recurrent Unit (GRU) have gained prominence as end-to-end learning architectures for natural language processing tasks. But what is the computational power of such systems? We prove that finite precision RNNs with one hidden layer and ReLU activation and finite precision GRUs are exactly as computationally powerful as deterministic nice automata. Allowing arbitrary precision, we prove that RNNs with one hidden layer and ReLU activation are at least as computationally powerful as pushdown automata. If we also allow infinite precision, infinite edge weights, and nonlinear output activation functions, we prove that GRUs are at least as computationally powerful as pushdown automata. All results are shown constructively.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>