Statistical Learning with Discrete Structures: Statistical and Computational Perspectives

Abstract

In various statistical tasks it is of interest to learn estimators with discrete structures (e.g., sparsity, low-rank, shared model parameters, etc)---they are appealing for their interpretability and compactness. However, learning with discrete structures can be computationally challenging. In this thesis, we explore statistical and computational aspects of statistics estimators (some classical and some new) that can be formulated as discrete optimization problems.

In Chapters 2 and 3, we study two well-known problems in high-dimensional statistics: sparse Principal Component Analysis (PCA) and Gaussian Graphical Models. These are notoriously hard optimization problems---we explore computationally friendlier estimators based on Mixed Integer Programming (MIP) under suitable statistical assumptions. We study the statistical and computational properties of our estimators.

In the fourth chapter, we study the multi-task learning problem with sparse linear estimators. Motivated by applications in biomedical sciences, we propose a new modeling framework to jointly learn sparse linear estimators for different tasks by sharing support information. Our theoretical results show that our joint estimation framework can lead to better statistical properties compared to independently fitting models for each task. We develop scalable approximate solvers for our MIP-based formulation.

In the fifth chapter, we study the problem of sparse Nonnegative Matrix Factorization (NMF) using Cutler and Breiman's archetypal regularization.

We explore the utility of our methods in the context of applications in the biomedical sciences, computer vision and computational finance.