Grouped variable selection with discrete optimization: Computational and statistical perspectives
Name
Download (3).pdf
Description
Published version
Size
635.65 KB
Format
Adobe PDF
Checksum (MD5)
c1ec446e789f4295ee3a8891e4cbe61b
Author(s)
Hazimeh, Hussein
Mazumder, Rahul
Radchenko, Peter
Date Issued
February 2023
Journal
The Annals of Statistics
Publisher
Institute of Mathematical Statistics
Citation
Ann. Statist. 51(1): 1-32 (February 2023).
Version
Final published version
Abstract
Proofs, additional technical details and numerical experiments are presented in the Supplementary Material. We present a new algorithmic framework for grouped variable selection that is based on discrete mathematical optimization. While there exist several appealing approaches based on convex relaxations and nonconvex heuristics, we focus on optimal solutions for the ℓ0-regularized formulation, a problem that is relatively unexplored due to computational challenges. Our methodology covers both high-dimensional linear regression and nonparametric sparse additive modeling with smooth components. Our algorithmic framework consists of approximate and exact algorithms. The approximate algorithms are based on coordinate descent and local search, with runtimes comparable to popular sparse learning algorithms. Our exact algorithm is based on a standalone branch-and-bound (BnB) framework, which can solve the associated mixed integer programming (MIP) problem to certified optimality. By exploiting the problem structure, our custom BnB algorithm can solve to optimality problem instances with 5×106 features and 103 observations in minutes to hours—over 1000 times larger than what is currently possible using state-of-the-art commercial MIP solvers. We also explore statistical properties of the ℓ0-based estimators. We demonstrate, theoretically and empirically, that our proposed estimators have an edge over popular group-sparse estimators in terms of statistical performance in various regimes. We provide an open source implementation of our proposed framework.
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1214/21-aos2155
