Evaluating Scalable Uncertainty Estimation Methods for Deep Learning-Based Molecular Property Prediction
Name
acs.jcim.9b00975.pdf
Description
Published version
Size
2.88 MB
Format
Adobe PDF
Checksum (MD5)
d8719bc40345827423acc785bb39142b
Author(s)
Scalia, Gabriele
Grambow, Colin A
Pernici, Barbara
Li, Yi-Pei
Green, William H
Date Issued
2020
Journal
Journal of Chemical Information and Modeling
Publisher
American Chemical Society (ACS)
Version
Author's final manuscript
Abstract
© 2020 American Chemical Society. Advances in deep neural network (DNN)-based molecular property prediction have recently led to the development of models of remarkable accuracy and generalization ability, with graph convolutional neural networks (GCNNs) reporting state-of-the-art performance for this task. However, some challenges remain, and one of the most important that needs to be fully addressed concerns uncertainty quantification. DNN performance is affected by the volume and the quality of the training samples. Therefore, establishing when and to what extent a prediction can be considered reliable is just as important as outputting accurate predictions, especially when out-of-domain molecules are targeted. Recently, several methods to account for uncertainty in DNNs have been proposed, most of which are based on approximate Bayesian inference. Among these, only a few scale to the large data sets required in applications. Evaluating and comparing these methods has recently attracted great interest, but results are generally fragmented and absent for molecular property prediction. In this paper, we quantitatively compare scalable techniques for uncertainty estimation in GCNNs. We introduce a set of quantitative criteria to capture different uncertainty aspects and then use these criteria to compare MC-dropout, Deep Ensembles, and bootstrapping, both theoretically in a unified framework that separates aleatoric/epistemic uncertainty and experimentally on public data sets. Our experiments quantify the performance of the different uncertainty estimation methods and their impact on uncertainty-related error reduction. Our findings indicate that Deep Ensembles and bootstrapping consistently outperform MC-dropout, with different context-specific pros and cons. Our analysis leads to a better understanding of the role of aleatoric/epistemic uncertainty, also in relation to the target data set features, and highlights the challenge posed by out-of-domain uncertainty.
MIT Department
Massachusetts Institute of Technology. Department of Chemical Engineering
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1021/acs.jcim.9b00975
