Developing clinically useful risk stratification models

Abstract

When a patient presents to a hospital with symptoms of cardiovascular disease, one of the first courses of action is to estimate the patient's risk of an adverse outcome. The process of categorizing patients by risk level, known as risk stratification, is an essential step in assigning appropriate therapy. Risk stratification models, which aid clinicians in this task, consist of feature sets that are combined by an algorithm to yield a score. In addition to the performance of the model, a key factor in model development is clinician acceptance of the score. One way to bolster clinician acceptance is to choose parsimonious feature sets to be used in risk scores that are convenient to integrate into the clinical workflow. A second consideration is establishing clinician trust in the model predictions. This is particularly important when using models that are difficult to explain to clinicians and when it is not straightforward to identify failure modes for the model.

Providing clinicians with a measure of how much to trust a given prediction from a model is one way to encourage the use of models that are difficult to interpret. In this thesis, we consider the problem of developing clinically useful risk models using real clinical data. We begin by discussing how to choose clinical variables in a data-driven fashion in the context of acute coronary syndrome. We present a risk score that can accommodate a variable number of inputs and demonstrate that it has superior performance to the Global Registry of Acute Coronary Events (GRACE) risk score, particularly on the difficult to risk stratify low-risk patients (AUC 0.754 vs. 0.688 for the GRACE score, p < 0.007). We then discuss the development of a risk score for aortic stenosis (AS) using both data-driven feature selection and expert opinion.

We show that the model performs well on patients with moderate to severe aortic stenosis (AUC 0.74), as well as on the difficult to risk stratify low gradient severe AS subgroup (2-5 year hazard ratios >/= 3.3, p < 0.05). Finally, we develop a method to identify unreliable predictions in clinical risk models and show, using the GRACE dataset, that we can identify subgroups of poor model performance to aid in bolstering clinician trust of risk models.