A comparison of approaches to improve worst-case predictive model performance over patient subpopulations
Name
s41598-022-07167-7.pdf
Description
Published version
Size
1.37 MB
Format
Adobe PDF
Checksum (MD5)
ad40d92e54063e8981c5139ee1db4749
Author(s)
Pfohl, Stephen R
Zhang, Haoran
Xu, Yizhe
Foryciarz, Agata
Ghassemi, Marzyeh
Shah, Nigam H
Date Issued
2022
Journal
Scientific Reports
Publisher
Springer Science and Business Media LLC
Citation
Pfohl, Stephen R, Zhang, Haoran, Xu, Yizhe, Foryciarz, Agata, Ghassemi, Marzyeh et al. 2022. "A comparison of approaches to improve worst-case predictive model performance over patient subpopulations." Scientific Reports, 12 (1).
Version
Final published version
Abstract
AbstractPredictive models for clinical outcomes that are accurate on average in a patient population may underperform drastically for some subpopulations, potentially introducing or reinforcing inequities in care access and quality. Model training approaches that aim to maximize worst-case model performance across subpopulations, such as distributionally robust optimization (DRO), attempt to address this problem without introducing additional harms. We conduct a large-scale empirical study of DRO and several variations of standard learning procedures to identify approaches for model development and selection that consistently improve disaggregated and worst-case performance over subpopulations compared to standard approaches for learning predictive models from electronic health records data. In the course of our evaluation, we introduce an extension to DRO approaches that allows for specification of the metric used to assess worst-case performance. We conduct the analysis for models that predict in-hospital mortality, prolonged length of stay, and 30-day readmission for inpatient admissions, and predict in-hospital mortality using intensive care data. We find that, with relatively few exceptions, no approach performs better, for each patient subpopulation examined, than standard learning procedures using the entire training dataset. These results imply that when it is of interest to improve model performance for patient subpopulations beyond what can be achieved with standard practices, it may be necessary to do so via data collection techniques that increase the effective sample size or reduce the level of noise in the prediction problem.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Institute for Medical Engineering & Science
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1038/S41598-022-07167-7
