An Efficient Learning Procedure for Deep Boltzmann Machines
Name
MIT-CSAIL-TR-2010-037.pdf
Size
735.71 KB
Format
Adobe PDF
Checksum (MD5)
027d1a3ec4a81725b135bd2195d989f3
Author(s)
Salakhutdinov, Ruslan
Hinton, Geoffrey
Advisor(s)
Joshua Tenenbaum
Date Issued
August 4, 2010
Series/Report no.
MIT-CSAIL-TR-2010-037
Abstract
We present a new learning algorithm for Boltzmann Machines that contain many layers of hidden variables. Data-dependent statistics are estimated using a variational approximation that tends to focus on a single mode, and data-independent statistics are estimated using persistent Markov chains. The use of two quite different techniques for estimating the two types of statistic that enter into the gradient of the log likelihood makes it practical to learn Boltzmann Machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer "pre-training" phase that initializes the weights sensibly. The pre-training also allows the variational inference to be initialized sensibly with a single bottom-up pass. We present results on the MNIST and NORB datasets showing that Deep Boltzmann Machines learn very good generative models of hand-written digits and 3-D objects. We also show that the features discovered by Deep Boltzmann Machines are a very effective way to initialize the hidden layers of feed-forward neural nets which are then discriminatively fine-tuned.
Subjects
Deep learning
Graphical models
Boltzmann Machines
Persistent DSpace Link
