Reverse engineering of receiver operating characteristic curves

Abstract

Receiver operating characteristic (ROC) curves have played a crucial role in the design and evaluation of radar systems for many decades. More recently, their use has spread to a variety of fields including clinical decision-making and machine learning. The common thread in all of these fields is an interest in binary hypothesis testing problems, in which the objective is to use an observation of a random variable of interest, sometimes referred to as a score variable, to infer the answer to a yes-no question. The standard progression of a binary hypothesis testing system from an observation of a score variable, to a set of parameterized decision rules with a binary outputs, to an ROC curve that characterizes the performance of those decision rules is well-understood. Thus, it comes as no surprise that an ROC curve only contains partial information about the problem for which it was designed. In this thesis, a key objective is to find ways of "reverse engineering" ROC curves in order to infer as much information as possible about the underlying binary hypothesis testing problems. We focus specifically on ROC curves that were or could have been constructed using likelihood ratio tests on an actual score variable, which we refer to as LRT-consistent ROC curves. For example, a specific LRT-consistent ROC curve does not uniquely determine the conditional distributions of the score variable used to generate it. A main result is a method for starting with an LRT-consistent ROC curve and using it to construct the conditional distributions of an unlimited number of score variables that could have been used to produce it. One interpretation of the result is as a characterization of the family of score variables that lead to the same ROC curve. This approach is extended to the similar problem of characterizing the family of score variables that lead to the same a set of LRT decision rules.