Evaluating discrete choice prediction models when the evaluation data is corrupted: analytic results and bias corrections for the area under the ROC
Author(s)
Stein, Roger Mark
Download10618_2015_437_ReferencePDF.pdf (325.2Kb)
PUBLISHER_POLICY
Publisher Policy
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Terms of use
Metadata
Show full item recordAbstract
There has been a growing recognition that issues of data quality, which are routine in practice, can materially affect the assessment of learned model performance. In this paper, we develop some analytic results that are useful in sizing the biases associated with tests of discriminatory model power when these are performed using corrupt (“noisy”) data. As it is sometimes unavoidable to test models with data that are known to be corrupt, we also provide some guidance on interpreting results of such tests. In some cases, with appropriate knowledge of the corruption mechanism, the true values of the performance statistics such as the area under the ROC curve may be recovered (in expectation), even when the underlying data have been corrupted. We also provide estimators of the standard errors of such recovered performance statistics. An analysis of the estimators reveals interesting behavior including the observation that “noisy” data does not “cancel out” across models even when the same corrupt data set is used to test multiple candidate models. Because our results are analytic, they may be applied in a broad range of settings and this can be done without the need for simulation.
Date issued
2015-09Department
Sloan School of ManagementJournal
Data Mining and Knowledge Discovery
Publisher
Springer US
Citation
Stein, Roger M. “Evaluating Discrete Choice Prediction Models When the Evaluation Data Is Corrupted: Analytic Results and Bias Corrections for the Area under the ROC.” Data Mining and Knowledge Discovery 30.4 (2016): 763–796.
Version: Author's final manuscript
ISSN
1384-5810
1573-756X