On Efficient Training & Inference of Neural Differential Equations
Name
pal-avikpal-sm-eecs-2023-thesis.pdf
Description
Thesis PDF
Size
1.74 MB
Format
Adobe PDF
Checksum (MD5)
228dae337a14b257aab04452b61cab14
Author(s)
Pal, Avik
Advisor(s)
Edelman, Alan
Date Issued
June 2023
Publisher
Massachusetts Institute of Technology
Abstract
The democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations have emerged as a popular modeling framework, enabling ML practitioners to design neural networks that can adaptively modify their depth based on the input problem. Neural Differential Equations combine differential equations with neural networks and rely on adaptive differential equation solvers for the forward process.
The flexibility of automatically adapting the depths comes with the cost of expensive training and slower predictions. Several prior works have tried to accelerate training and inference. However, almost all of them have severe tradeoffs. Either these works rely on expensive training methods to accelerate predictions or use algorithms that are harder to integrate into existing workflows.
This thesis will discuss two methods to accelerate Neural Differential Equations. We propose an Infinite Time Neural ODE, which paradoxically can be trained faster than integrating a Neural ODE to a fixed time-point. We also build upon prior works on regularized Neural ODEs and propose a stochastic local regularization scheme that can be used as a drop-in replacement for Neural ODEs.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
