2026-09-19T19:52:20Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1513792023-08-01T03:53:34Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Edelman, Alan
Pal, Avik
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2023-07-31T19:35:22Z
2023-07-31T19:35:22Z
2023-06
2023-07-13T14:26:19.721Z
https://hdl.handle.net/1721.1/151379
The democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations have emerged as a popular modeling framework, enabling ML practitioners to design neural networks that can adaptively modify their depth based on the input problem. Neural Differential Equations combine differential equations with neural networks and rely on adaptive differential equation solvers for the forward process.
The flexibility of automatically adapting the depths comes with the cost of expensive training and slower predictions. Several prior works have tried to accelerate training and inference. However, almost all of them have severe tradeoffs. Either these works rely on expensive training methods to accelerate predictions or use algorithms that are harder to integrate into existing workflows.
This thesis will discuss two methods to accelerate Neural Differential Equations. We propose an Infinite Time Neural ODE, which paradoxically can be trained faster than integrating a Neural ODE to a fixed time-point. We also build upon prior works on regularized Neural ODEs and propose a stochastic local regularization scheme that can be used as a drop-in replacement for Neural ODEs.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
On Efficient Training & Inference of Neural Differential Equations
Thesis
application/pdf
Master
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="652fa947-b45d-4f24-9a7a-530a3fdaa8d4">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>On Efficient Training & Inference of Neural Differential Equations</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2023-06</PublicationDate>
<Authors>
<Author>
<DisplayName>Pal, Avik</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>The democratization of machine learning requires architectures that automatically adapt to new problems. Neural Differential Equations have emerged as a popular modeling framework, enabling ML practitioners to design neural networks that can adaptively modify their depth based on the input problem. Neural Differential Equations combine differential equations with neural networks and rely on adaptive differential equation solvers for the forward process.
The flexibility of automatically adapting the depths comes with the cost of expensive training and slower predictions. Several prior works have tried to accelerate training and inference. However, almost all of them have severe tradeoffs. Either these works rely on expensive training methods to accelerate predictions or use algorithms that are harder to integrate into existing workflows.
This thesis will discuss two methods to accelerate Neural Differential Equations. We propose an Infinite Time Neural ODE, which paradoxically can be trained faster than integrating a Neural ODE to a fixed time-point. We also build upon prior works on regularized Neural ODEs and propose a stochastic local regularization scheme that can be used as a drop-in replacement for Neural ODEs.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>