Mathematical analysis of uncertainty in machine learning and deep learning

Abstract

In this paper, we study uncertainty in machine learning and deep learning from the mathematical point of view. Uncertainty is involved in many real-world situations. The Bayesian modelling can handle such uncertainty in machine learning community. However, the traditional deep learning model fails to show uncertainty for its outputs. Recently, at the intersection of the Bayesian modelling and deep learning, a new framework called the Bayesian deep learning (BDL) has been proposed and studied, which enables us to estimate uncertainty of deep learning models. As an example of it, we can review the results of Yarin Gal, in which the famous dropout method can be seen as a Bayesian modelling. We also see that overfitting problem of the framework due to the property of the KL divergence, and review the modified algorithm using o-divergence which generalizes the KL divergence. We also study a confidence band to assess uncertainty of a kernel ridge regression estimator. We propose the formulation to obtain a confidence band as the convex optimization, which enables us to use existing algorithms such as the primal-dual inner point method. The proposed method acquires a more accurate and fast confidence band than a bootstrap algorithm. We also see the effectiveness of our proposed method both in the case of function approximation and an estimate of an actual dataset.