From Data, to Models, and Back: Making Machine Learning Predictably Reliable
Name
ilyas-ailyas-phd-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
146.98 MB
Format
Adobe PDF
Checksum (MD5)
14d25de0f7525ad7ad1be4415535eba2
Author(s)
Ilyas, Andrew
Advisor(s)
Daskalakis, Constantinos
Mądry, Aleksander
Date Issued
February 2025
Publisher
Massachusetts Institute of Technology
Abstract
Machine learning systems exhibit impressive performance, but we currently lack scalable ways to anticipate their successes, failure modes, and biases. This position limits our ability to deploy these systems in the appropriate contexts, and to build systems which we can confidently deploy in high-risk settings. Motivated by this state of affairs, this thesis aims to develop design principles for predictably reliable machine learning. Our ultimate goal is to enable developers to know when their models will work, anticipate when they will fail, and understand “why” in both cases. In pursuit of this goal, this thesis combines large-scale experiments with theoretical analysis to form a precise understanding of the ML “pipeline,” from training data (and the way we collect it), to learning algorithms, to deployment. Fully realized, such an understanding would allow us to build ML systems the same way we build buildings or airplanes—safely, scalably, and with a robust grasp of the underlying principles. In this thesis, we focus on four design choices within this pipeline: model deployment (Part I), dataset creation (Part II), data collection (Part III), and algorithm selection (Part IV). For each of these design choices, we use targeted experiments to uncover the corresponding principles that actually underlie the behavior of ML systems. We distill these principles into concise conceptual models which allow us to both reason about existing systems and design improved ones. Along the way, we will revisit, challenge, and refine various aspects of conventional wisdom surrounding ML model development.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
