dc.contributor.author | Mazumder, Rahul | |
dc.date.accessioned | 2021-03-17T22:25:11Z | |
dc.date.available | 2021-03-17T22:25:11Z | |
dc.date.issued | 2020-10 | |
dc.date.submitted | 2020-06 | |
dc.identifier.issn | 1052-6234 | |
dc.identifier.uri | https://hdl.handle.net/1721.1/130168 | |
dc.description.abstract | The Gradient Boosting Machine (GBM) introduced by Friedman [J. H. Friedman, Ann. Statist., 29 (2001), pp. 1189-1232] is a powerful supervised learning algorithm that is very widely used in practice-it routinely features as a leading algorithm in machine learning competitions such as Kaggle and the KDDCup. In spite of the usefulness of GBM in practice, our current theoretical understanding of this method is rather limited. In this work, we propose the Randomized Gradient Boosting Machine (RGBM), which leads to substantial computational gains compared to GBM by using a randomization scheme to reduce search in the space of weak learners. We derive novel computational guarantees for RGBM. We also provide a principled guideline towards better step-size selection in RGBM that does not require a line search. Our proposed framework is inspired by a special variant of coordinate descent that combines the benefits of randomized coordinate descent and greedy coordinate descent, and may be of independent interest as an optimization algorithm. As a special case, our results for RGBM lead to superior computational guarantees for GBM. Our computational guarantees depend upon a curious geometric quantity that we call the Minimal Cosine Angle, which relates to the density of weak learners in the prediction space. On a series of numerical experiments on real datasets, we demonstrate the effectiveness of RGBM over GBM in terms of obtaining a model with good training and/or testing data fidelity with a fraction of the computational cost. | en_US |
dc.description.sponsorship | United States. Office of Naval Research (Grants ONR-N000141512342,ONR-N000141812298) | en_US |
dc.description.sponsorship | National Science Foundation (U.S.) (Grant NSF-IIS-1718258) | en_US |
dc.language.iso | en | |
dc.publisher | Society for Industrial & Applied Mathematics (SIAM) | en_US |
dc.relation.isversionof | 10.1137/18M1223277 | en_US |
dc.rights | Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use. | en_US |
dc.source | SIAM | en_US |
dc.title | Randomized gradient boosting machine | en_US |
dc.type | Article | en_US |
dc.identifier.citation | Lu, Haihao and Rahul Mazumder. “Randomized gradient boosting machine.” SIAM Journal on Optimization, 30, 4 (October 2020): 780--2808 © 2020 The Author(s) | en_US |
dc.contributor.department | Sloan School of Management | en_US |
dc.relation.journal | SIAM Journal on Optimization | en_US |
dc.eprint.version | Final published version | en_US |
dc.type.uri | http://purl.org/eprint/type/JournalArticle | en_US |
eprint.status | http://purl.org/eprint/status/PeerReviewed | en_US |
dc.date.updated | 2021-03-12T15:50:40Z | |
dspace.orderedauthors | LU, H; MAZUMDER, R | en_US |
dspace.date.submission | 2021-03-12T15:50:45Z | |
mit.journal.volume | 30 | en_US |
mit.journal.issue | 4 | en_US |
mit.license | PUBLISHER_POLICY | |
mit.metadata.status | Complete | |