Gated Orthogonal Recurrent Units: On Learning to Forget
Name
neco_a_01174.pdf
Description
Published version
Size
596.16 KB
Format
Adobe PDF
Checksum (MD5)
e76ac9ac5d171a57c0f2ae29d5ca4cd6
Author(s)
Jing, Li
Gulcehre, Caglar
Peurifoy, John
Shen, Yichen
Tegmark, Max
Soljacic, Marin
Bengio, Yoshua
Date Issued
2019
Journal
Neural Computation
Publisher
MIT Press - Journals
Version
Final published version
Abstract
© 2019 Massachusetts Institute of Technology. We present a novel recurrent neural network (RNN)based model that combines the remembering ability of unitary evolution RNNs with the ability of gated RNNs to effectively forget redundant or irrelevant information in its memory. We achieve this by extending restricted orthogonal evolution RNNs with a gating mechanism similar to gated recurrent unit RNNs with a reset gate and an update gate. Our model is able to outperform long short-term memory, gated recurrent units, and vanilla unitary or orthogonal RNNs on several long-term-dependency benchmark tasks. We empirically show that both orthogonal and unitary RNNs lack the ability to forget. This ability plays an important role in RNNs. We provide competitive results along with an analysis of our model on many natural sequential tasks, including question answering, speech spectrum prediction, character-level language modeling, and synthetic tasks that involve long-term dependencies such as algorithmic, denoising, and copying tasks.
MIT Department
Sloan School of Management
Massachusetts Institute of Technology. Department of Physics
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1162/neco_a_01174
