Dynamic risk adjustment of prediction models using statistical process control methods

Abstract

Introduction. Models that represent mathematical relationships between clinical outcomes and their predictors are useful to the decision making process in patient care. Many models, such as the score of neonatal physiology (SNAP II) that predicts in-hospital mortality, have been well validated on several large populations. However, the performance profile of such models in the midst of changing predictor-outcome relationships or newly appearing outcome predictors have not been well studied. We address this problem using statistical process control (SPC) techniques in a novel way. Although widely used in the manufacturing industry to maintain high quality in critical processes, SPC's value to healthcare has begun only recently to gain attention from decision makers. It has been used to construct risk-adjusted charts to track outcomes in the intensive care unit and the surgical arena, and to monitor hospital acquired infections. However, there are no reports of using SPC techniques to scrutinize the performance quality of a clinical model over time. The series of experiments in this manuscript show that the deterioration of a model's performance can be a useful indicator of unexpected changes in the environment that it represents; therefore, defining when a model is statistically not performing according to expectations is the first step towards determining the causes of clinical variations that might impact patient healthcare. Methods. We obtained a database of 3437 newborns admitted to 7 Neonatal Intensive Care Units in the New England area from October 1994 to January 1996. We chronologically arranged the patients by birthday and grouped them into 14 sequential periods; thereby establishing a time-sequenced database to be used in our SPC experiments.

(cont.) Each of the first thirteen periods contained 250 cases, while the last period had the remaining 187 cases. Several versions of the database were constructed by altering patient data in order to simulate various clinical scenarios--we either introduced graded changes in predictor values and mortality outcomes, or added new predictors. We analyzed the prediction performance pattern of the SNAP II model as applied to periods 1 to 14 in the original and modified versions of our database. The quality parameter tracked by our SPC charts is the C-index, which has been shown to be equivalent to the area under the Receiver Operating Characteristic curve and a well accepted indicator of a model's predictive performance. We introduced the 'deterioration index' as a quantitative measure of performance degradation that permitted us to compare results among experiments. Results. Applying the SNAP II model to the unaltered database, we showed that the c-indices remained well within statistically acceptable boundaries over time. This supported the generalizability of the SNAPII model as well as allowed us to use the mean and standard deviation of the c-indices as control values for our later experiments. In chapter 5, we showed that the model's performance can be degraded beyond acceptable limits by variations in the database (high deterioration index). The index depends on how much the changes in the database affect the existing predictor-outcome relationships. We also showed how the deterioration index can be used to assess and rank contributions of predictors to the model over time. In chapter 6, we showed that model performance ...