On the Inductive Biases of Conditional Diffusion Models

Abstract

Diffusion models have achieved remarkable progress in recent years across various domains and applications, but how diffusion models generalize is still not well understood. While prior work predominantly focuses on unconditional diffusion models, in this thesis we focus on understanding generalization for conditional diffusion models, which is especially relevant for modern text- or observation- conditioned applications. In particular, we are interested in the inductive biases of conditional diffusion models which predispose them to certain forms of interpolation in regions outside the support of the training data. We observe that neural networks are capable of learning qualitatively different forms of interpolation, which may be influenced by the architecture and capacity of the network and other aspects of neural network training. We develop a potential framework to model the interpolation behavior of neural networks via nonparametric estimation, which happens to have the property of being schedule consistent, or truly denoising at every time step. We find that, assuming a neural network with sufficient capacity, conditional diffusion models are biased towards smoothing, which can lead to non-schedule consistent behavior away from the training data and has a number of interesting consequences.