dc.contributor.author | Liao, Qianli | |
dc.contributor.author | Leibo, Joel Z. | |
dc.contributor.author | Poggio, Tomaso | |
dc.date.accessioned | 2016-01-12T01:14:28Z | |
dc.date.available | 2016-01-12T01:14:28Z | |
dc.date.issued | 2015-11-29 | |
dc.identifier.uri | http://hdl.handle.net/1721.1/100797 | |
dc.description.abstract | Gradient backpropagation (BP) requires symmetric feedforward and feedback connections—the same weights must be used for forward and backward passes. This “weight transport problem” [1] is thought to be one of the main reasons of BP’s biological implausibility. Using 15 different classification datasets, we systematically study to what extent BP really depends on weight symmetry. In a study that turned out to be surprisingly similar in spirit to Lillicrap et al.’s demonstration [2] but orthogonal in its results, our experiments indicate that: (1) the magnitudes of feedback weights do not matter to performance (2) the signs of feedback weights do matter—the more concordant signs between feedforward and their corresponding feedback connections, the better (3) with feedback weights having random magnitudes and 100% concordant signs, we were able to achieve the same or even better performance than SGD. (4) some normalizations/stabilizations are indispensable for such asymmetric BP to work, namely Batch Normalization (BN) [3] and/or a “Batch Manhattan” (BM) update rule. | en_US |
dc.description.sponsorship | This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF - 1231216. | en_US |
dc.language.iso | en_US | en_US |
dc.publisher | Center for Brains, Minds and Machines (CBMM), arXiv | en_US |
dc.relation.ispartofseries | CBMM Memo Series;036 | |
dc.rights | Attribution-NonCommercial 3.0 United States | * |
dc.rights.uri | http://creativecommons.org/licenses/by-nc/3.0/us/ | * |
dc.subject | Gradient backpropagation (BP) | en_US |
dc.subject | Batch Normalization (BN) | en_US |
dc.title | How Important is Weight Symmetry in Backpropagation? | en_US |
dc.type | Technical Report | en_US |
dc.type | Working Paper | en_US |
dc.identifier.citation | arXiv:1510.05067v3 | en_US |