Algorithms above the noise floor

Abstract

Many success stories in the data sciences share an intriguing computational phenomenon. While the core algorithmic problems might seem intractable at first, simple heuristics or approximation algorithms often perform surprisingly well in practice. Common examples include optimizing non-convex functions or optimizing over non-convex sets. In theory, such problems are usually NP-hard. But in practice, they are often solved sufficiently well for applications in machine learning and statistics. Even when a problem is convex, we often settle for sub-optimal solutions returned by inexact methods like stochastic gradient descent. And in nearest neighbor search, a variety of approximation algorithms works remarkably well despite the "curse of dimensionality". In this thesis, we study this phenomenon in the context of three fundamental algorithmic problems arising in the data sciences. * In constrained optimization, we show that it is possible to optimize over a wide range of non-convex sets up to the statistical noise floor. * In unconstrained optimization, we prove that important convex problems already require approximation if we want to find a solution quickly. * In nearest neighbor search, we show that approximation guarantees can explain much of the good performance observed in practice. The overarching theme is that the computational hardness of many problems emerges only below the inherent "noise floor" of real data. Hence computational hardness of these problems does not prevent us from finding answers that perform well from a statistical perspective. This offers an explanation for why algorithmic problems in the data sciences often turn out to be easier than expected.