Bootstrapping EM via power EM and convergence in the naive bayes model
Name
daskalakis18a.pdf
Description
Published version
Size
714.6 KB
Format
Adobe PDF
Checksum (MD5)
e67dccea170a10bd5a65bf0e374c7df1
Author(s)
Daskalakis, C
Tzamos, C
Zampetakis, M
Date Issued
January 1, 2018
Journal
International Conference on Artificial Intelligence and Statistics, AISTATS 2018
Citation
Daskalakis, C, Tzamos, C and Zampetakis, M. 2018. "Bootstrapping EM via power EM and convergence in the naive bayes model." International Conference on Artificial Intelligence and Statistics, AISTATS 2018.
Version
Final published version
Abstract
Copyright 2018 by the author(s). We study the convergence properties of the Expectation-Maximization algorithm in the Naive Bayes model. We show that EM can get stuck in regions of slow convergence, even when the features are binary and i.i.d. conditioning on the class label, and even under random (i.e. non worst-case) initialization. In turn, we show that EM can be bootstrapped in a pre-training step that computes a good initialization. From this initialization we show theoretically and experimentally that EM converges exponentially fast to the true model parameters. Our bootstrapping method amounts to running the EM algorithm on appropriately centered iterates of small magnitude, which as we show corresponds to effectively performing power iteration on the covariance matrix of the mixture model, although power iteration is performed under the hood by EM itself. As such, we call our bootstrapping approach “power EM.” Specifically for the case of two binary features, we show global exponentially fast convergence of EM, even without bootstrapping. Finally, as the Naive Bayes model is quite expressive, we show as corollaries of our convergence results that the EM algorithm globally converges to the true model parameters for mixtures of two Gaussians, recovering recent results of [XHM16, DTZ17].
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://proceedings.mlr.press/v84/daskalakis18a.html
