Generalized stochastic Frank–Wolfe algorithm with stochastic “substitute” gradient for structured convex optimization
Name
10107_2020_1480_ReferencePDF.pdf
Size
689.17 KB
Format
Adobe PDF
Checksum (MD5)
13bb06b280a1ce3ae669bd078057d3be
Author(s)
Lu, Haihao
Freund, Robert M
Date Issued
March 4, 2020
Publisher
Springer Berlin Heidelberg
Version
Author's final manuscript
Abstract
Abstract
The stochastic Frank–Wolfe method has recently attracted much general interest in the context of optimization for statistical and machine learning due to its ability to work with a more general feasible region. However, there has been a complexity gap in the dependence on the optimality tolerance
$$\varepsilon $$
ε
in the guaranteed convergence rate for stochastic Frank–Wolfe compared to its deterministic counterpart. In this work, we present a new generalized stochastic Frank–Wolfe method which closes this gap for the class of structured optimization problems encountered in statistical and machine learning characterized by empirical loss minimization with a certain type of “linear prediction” property (formally defined in the paper), which is typically present in loss minimization problems in practice. Our method also introduces the notion of a “substitute gradient” that is a not-necessarily-unbiased estimate of the gradient. We show that our new method is equivalent to a particular randomized coordinate mirror descent algorithm applied to the dual problem, which in turn provides a new interpretation of randomized dual coordinate descent in the primal space. Also, in the special case of a strongly convex regularizer our generalized stochastic Frank–Wolfe method (as well as the randomized dual coordinate descent method) exhibits linear convergence. Furthermore, we present computational experiments that indicate that our method outperforms other stochastic Frank–Wolfe methods for a sufficiently small optimality tolerance, consistent with the theory developed herein.
MIT Department
Massachusetts Institute of Technology. Department of Mathematics
Sloan School of Management
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/s10107-020-01480-7
