ModelDiff: A Framework for Comparing Learning Algorithms
Name
shah-harshay-sm-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
6.3 MB
Format
Adobe PDF
Checksum (MD5)
427d879a3cc0f57908c5c6e54b89cfd0
Author(s)
Shah, Harshay
Advisor(s)
Mądry, Aleksander
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
We study the problem of (learning) algorithm comparison, where the goal is to find differences between models trained with two different learning algorithms. We begin by formalizing this goal as one of finding distinguishing feature transformations, i.e., input transformations that change the predictions of models trained with one learning algorithm but not the other. We then present ModelDiff, a method that leverages the datamodels framework (Ilyas et al., 2022) to compare learning algorithms based on how they use their training data. We demonstrate ModelDiff through three case studies, comparing models trained with/without data augmentation, with/without pre-training, and with different SGD hyperparameters. Our code is available at https://github.com/MadryLab/modeldiff.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)
Copyright retained by author(s)
Persistent DSpace Link
