Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications
Name
tacl_a_00258.pdf
Description
Published version
Size
2.7 MB
Format
Unknown
Checksum (MD5)
686580857e385b6d5341164164905bfe
Author(s)
Dangovski, Rumen
Jing, Li
Nakov, Preslav
Tatalovic, Mico
Soljacic, Marin
Date Issued
2019
Journal
Transactions of the Association for Computational Linguistics
Publisher
MIT Press - Journals
Version
Final published version
Abstract
Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conventional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotational Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and memory recall much better than LSTMs/GRUs. We further demonstrate that by replacing LSTM/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art.
MIT Department
Massachusetts Institute of Technology. Research Laboratory of Electronics
Sloan School of Management
Massachusetts Institute of Technology. Graduate Program in Science Writing
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1162/TACL_A_00258
