Large-Scale Optimization Methods: Theory and Applications

Abstract

Large-scale optimization problems appear quite frequently in data science and machine learning applications. In this thesis, we show the efficiency of coordinate descent (CD) and mirror descent (MD) methods in solving large-scale optimization problems.

First, we investigate the convergence rate of the CD method with different coordinate selection rules. We present certain problem classes, for which deterministic rules provably outperform randomized rules. We quantify the amount of improvement and the corresponding deterministic order that achieves the maximum improvement. We then show that for a certain subclass of problems, using any fixed deterministic rule yields a superior performance than using random permutations. Then, we illustrate the efficiency of the CD method on a constrained non-convex optimization problem that arise from semidefinite programming with diagonal constraints. We show that the proposed CD methods can recover the optimal solution when the rank of the factorization is sufficiently large, and establish the rate of convergence. When the rank of the factorization is small, we provide tight approximation bounds as a function of the rank.

Next, we study convergence properties of the continuous-time and discrete-time MD methods. We present a unified convergence theory for mirror descent and related methods. Then, we establish the implicit bias of the MD method with non-differentiable distance generating functions. Finally, we introduce the continuous-time MD method with non-differentiable and non-strictly convex distance generating functions. We show the existence and convergence of the solutions generated by the MD method and establish their implicit bias. We illustrate that the combinatorial algorithms resulting from this approach can be used to solve sparse optimization problems.