A robust optimization approach to statistical estimation problems by Apostolos G. Fertis.

Abstract

There have long been intuitive connections between robustness and regularization in statistical estimation, for example, in lasso and support vector machines. In the first part of the thesis, we formalize these connections using robust optimization. Specifically (a) We show that in classical regression, regularized estimators like lasso can be derived by applying robust optimization to the classical least squares problem. We discover the explicit connection between the size and the structure of the uncertainty set used in the robust estimator, with the coefficient and the kind of norm used in regularization. We compare the out-of-sample performance of the nominal and the robust estimators in computer generated and real data. (b) We prove that the support vector machines estimator is also a robust estimator of some nominal classification estimator (this last fact was also observed independently and simultaneously by Xu, Caramanis, and Mannor [52]). We generalize the support vector machines estimator by considering several sizes and structures for the uncertainty sets, and proving that the respective max-min optimization problems can be expressed as regularization problems. In the second part of the thesis, we turn our attention to constructing robust maximum likelihood estimators. Specifically (a) We define robust estimators for the logistic regression model, taking into consideration uncertainty in the independent variables, in the response variable, and in both. We consider several structures for the uncertainty sets, and prove that, in all cases, they lead to convex optimization problems. We provide efficient algorithms to compute the estimates in all cases.

(cont.) We report on the out-of-sample performance of the robust, as well as the nominal estimators in both computer generated and real data sets, and conclude that the robust estimators achieve a higher success rate. (b) We develop a robust maximum likelihood estimator for the multivariate normal distribution by considering uncertainty sets for the data used to produce it. We develop an efficient first order gradient descent method to compute the estimate and compare the efficiency of the robust estimate to the respective nominal one in computer generated data.