Identifying unreliable predictions in clinical risk models
Name
s41746-019-0209-7.pdf
Description
Published version
Size
1.38 MB
Format
Adobe PDF
Checksum (MD5)
bce72b472d041be20a2fb790ddc90cee
Author(s)
Myers, Paul D
Ng, Kenney
Severson, Kristen
Kartoun, Uri
Dai, Wangzhi
Huang, Wei
Anderson, Frederick A
Stultz, Collin M
Date Issued
2020
Journal
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Version
Final published version
Abstract
© 2020, The Author(s). The ability to identify patients who are likely to have an adverse outcome is an essential component of good clinical care. Therefore, predictive risk stratification models play an important role in clinical decision making. Determining whether a given predictive model is suitable for clinical use usually involves evaluating the model’s performance on large patient datasets using standard statistical measures of success (e.g., accuracy, discriminatory ability). However, as these metrics correspond to averages over patients who have a range of different characteristics, it is difficult to discern whether an individual prediction on a given patient should be trusted using these measures alone. In this paper, we introduce a new method for identifying patient subgroups where a predictive model is expected to be poor, thereby highlighting when a given prediction is misleading and should not be trusted. The resulting “unreliability score” can be computed for any clinical risk model and is suitable in the setting of large class imbalance, a situation often encountered in healthcare settings. Using data from more than 40,000 patients in the Global Registry of Acute Coronary Events (GRACE), we demonstrate that patients with high unreliability scores form a subgroup in which the predictive model has both decreased accuracy and decreased discriminatory ability.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Research Laboratory of Electronics
Massachusetts Institute of Technology. Institute for Medical Engineering & Science
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
10.1038/S41746-019-0209-7
https://doi.org/10.1038/S41746-019-0209-7
