Dataset Deduplication with Datamodels
Name
Liao-yunxingl-meng-eecs-2022-thesis.pdf
Description
Thesis PDF
Size
9.27 MB
Format
Adobe PDF
Checksum (MD5)
a8ada9f01c188e92024f9b49f001126b
Author(s)
Liao, Yunxing
Advisor(s)
Mądry, Aleksander
Date Issued
May 2022
Publisher
Massachusetts Institute of Technology
Abstract
Large curated datasets have been essential to the development of deep learning models across many disciplines. Consequently, the properties of these datasets have a large impact on the behavior of these models. As machine learning pipelines increasingly leverage more unlabelled datasets—which tend to undergo less curation than labelled datasets—controlling data quality becomes even more important. We focus on a particular aspect of data quality: train-test leakage or duplicate examples. These can cause overestimation of models’ performance on benchmarks among other issues. In this work, we apply datamodels, a framework for analyzing the behavior of a model class as a function of its training data, to deduplicate unlabelled datasets. Inspired by the recent CLIP model, we focus on detecting duplicates between YFCC15M and the ImageNet validation dataset. Our results demonstrate how to adapt datamodels effectively for these filtering tasks in unsupervised, large-scale settings. We finish by discussing the challenges of our method and duplicate detection more broadly.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
