Gradient Subgroup Scanning for Distributionally and Outlier Robust Models
Name
Jung-luju-meng-eecs-2022-thesis.pdf
Description
Thesis PDF
Size
1.28 MB
Format
Adobe PDF
Checksum (MD5)
1ac6a43d3ad53ece8cfe15b67632b2f4
Author(s)
Jung, Luann
Advisor(s)
Solomon, Justin
Yurochkin, Mikhail
Greenewald, Kristjan
Date Issued
May 2022
Publisher
Massachusetts Institute of Technology
Abstract
Traditional machine learning methods such as empirical risk minimization (ERM) frequently encounter the issue of achieving high accuracy on average but low accuracy on certain subgroups, especially when there exist spurious correlations between the input data and label. Previous approaches for reducing the discrepancy between average and worst-group accuracies typically require expensive known subgroup annotations for either every training data point (as is the case in group distributionally robust optimization (DRO)), or every validation data point. Furthermore, these distributionally robust approaches tend to show reduced performance when outliers are also present in the data. Unfortunately, existing methods for improving subgroup performance cannot be simply combined with prior approaches for excluding outliers, as they often directly conflict. This work proposes a method for addressing both group robustness and outlier exclusion when training machine learning models that requires no previous knowledge about subgroups or outliers within the data. We focus on attempting to balance these two traditionally clashing goals by clustering the gradients of the losses with respect to the model parameters. In doing so, we find minority subgroups and exclude outliers under the assertion that gradients within groups behave similarly while outliers exhibit more randomized behavior. This work demonstrates an improvement to both average and worst-group accuracies compared to baselines and other previous methods when applied to the Waterbirds image dataset.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
