Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
Name
CBMM-Memo-057.pdf
Size
1.27 MB
Format
Adobe PDF
Checksum (MD5)
825ed0e068359cb71bf32f385c642103
Author(s)
Liao, Qianli
Kawaguchi, Kenji
Poggio, Tomaso
Date Issued
October 19, 2016
Publisher
Center for Brains, Minds and Machines (CBMM), arXiv
Citation
arXiv:1610.06160v1
Series/Report no.
CBMM Memo Series;057
Abstract
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major limitations of BN: (1) online learning and (2) recurrent learning. Our proposal is simpler and more biologically-plausible. Unlike previous approaches, our technique can be applied out of the box to all learning scenarios (e.g., online learning, batch learning, fully-connected, convolutional, feedforward, recurrent and mixed — recurrent and convolutional) and compare favorably with existing approaches. We also propose Lp Normalization for normalizing by different orders of statistical moments. In particular, L1 normalization is well-performing, simple to implement, fast to compute, more biologically-plausible and thus ideal for GPU or hardware implementations.
Subjects
Batch Normalization (BN)
recurrent learning
Lp Normalization
Terms of Use
Attribution-NonCommercial-ShareAlike 3.0 United States
Persistent DSpace Link
