Evaluating discrete choice prediction models when the evaluation data is corrupted: analytic results and bias corrections for the area under the ROC
Name
10618_2015_437_ReferencePDF.pdf
Size
325.22 KB
Format
Adobe PDF
Checksum (MD5)
180ff4102fdf6cc011ffe417c09280e1
Author(s)
Stein, Roger Mark
Date Issued
September 2015
Journal
Data Mining and Knowledge Discovery
Publisher
Springer US
Citation
Stein, Roger M. “Evaluating Discrete Choice Prediction Models When the Evaluation Data Is Corrupted: Analytic Results and Bias Corrections for the Area under the ROC.” Data Mining and Knowledge Discovery 30.4 (2016): 763–796.
Version
Author's final manuscript
Abstract
There has been a growing recognition that issues of data quality, which are routine in practice, can materially affect the assessment of learned model performance. In this paper, we develop some analytic results that are useful in sizing the biases associated with tests of discriminatory model power when these are performed using corrupt (“noisy”) data. As it is sometimes unavoidable to test models with data that are known to be corrupt, we also provide some guidance on interpreting results of such tests. In some cases, with appropriate knowledge of the corruption mechanism, the true values of the performance statistics such as the area under the ROC curve may be recovered (in expectation), even when the underlying data have been corrupted. We also provide estimators of the standard errors of such recovered performance statistics. An analysis of the estimators reveals interesting behavior including the observation that “noisy” data does not “cancel out” across models even when the same corrupt data set is used to test multiple candidate models. Because our results are analytic, they may be applied in a broad range of settings and this can be done without the need for simulation.
MIT Department
Sloan School of Management
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/s10618-015-0437-7
