On Efficient Training & Inference of Neural Differential Equations

Abstract

The democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations have emerged as a popular modeling framework, enabling ML practitioners to design neural networks that can adaptively modify their depth based on the input problem. Neural Differential Equations combine differential equations with neural networks and rely on adaptive differential equation solvers for the forward process.

The flexibility of automatically adapting the depths comes with the cost of expensive training and slower predictions. Several prior works have tried to accelerate training and inference. However, almost all of them have severe tradeoffs. Either these works rely on expensive training methods to accelerate predictions or use algorithms that are harder to integrate into existing workflows.

This thesis will discuss two methods to accelerate Neural Differential Equations. We propose an Infinite Time Neural ODE, which paradoxically can be trained faster than integrating a Neural ODE to a fixed time-point. We also build upon prior works on regularized Neural ODEs and propose a stochastic local regularization scheme that can be used as a drop-in replacement for Neural ODEs.