A new perspective on boosting in linear regression via subgradient optimization and relatives
Name
1505.04243.pdf
Size
2.35 MB
Format
Adobe PDF
Checksum (MD5)
d7f257cd002c4f570e0b47923914f879
Author(s)
M. Freund, Robert
Grigas, Paul
Mazumder, Rahul
Freund, Robert Michael
Grigas, Paul Edward
Date Issued
December 2017
Journal
Annals of Statistics
Publisher
Institute of Mathematical Statistics
Citation
M. Freund, Robert et al. “A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives.” The Annals of Statistics 45, 6 (December 2017): 2328–2364 © 2017 Institute of Mathematical Statistics
Version
Original manuscript
Abstract
We analyze boosting algorithms [Ann. Statist. 29 (2001) 1189–1232; Ann. Statist. 28 (2000) 337–407; Ann. Statist. 32 (2004) 407–499] in linear regression from a new perspective: that of modern first-order methods in convex optimiz ation. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FS ? ) and least squares boosting [LS-BOOST(?)], can be viewed as subgradient descent to minimize the loss function defined as the maximum absolute correlation between the features and residuals. We also propose a minor modification of FS ? that yields an algorithm for the LASSO, and that may be easily extended to an algorithm that computes the LASSO path for different values of the regularization parameter. Furthermore, we show that these new algorithms for the LASSO may also be interpreted as the same master algorithm (subgradient descent), applied to a regularized version of the maximum absolute correlation loss function. We derive novel, comprehensive computational guarantees for several boosting algorithms in linear regression (including LS-BOOST(?) and FS ? ) by using techniques of first-order methods in convex optimization. Our computational guarantees inform us about the statistical properties of boosting algorithms. In particular, they provide, for the first time, a precise theoretical description of the amount of data-fidelity and regularization imparted by running a boosting algorithm with a prespecified learning rate for a fixed but arbitrary number of iterations, for any dataset.
MIT Department
Massachusetts Institute of Technology. Operations Research Center
Sloan School of Management
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1214/16-AOS1505
