Higher-Order Automatic Differentiation and Its Applications
Name
tan-songchen-csesm-ccse-2023-thesis Songchen Tan.pdf
Description
Thesis PDF
Size
242.58 KB
Format
Adobe PDF
Checksum (MD5)
bdb3a84d278601e71ad53efe1e7c8718
Author(s)
Tan, Songchen
Advisor(s)
Edelman, Alan
Date Issued
June 2023
Publisher
Massachusetts Institute of Technology
Abstract
Differentiable programming is a new paradigm for modeling and optimization in many fields of science and engineering, and automatic differentiation (AD) algorithms are at the heart of differentiable programming. Existing methods to achieve higher-order AD often suffer from one or more of the following problems: (1) exponential scaling with respect to order due to nesting first-order AD; (2) ad-hoc handwritten higher-order rules which are hard to maintain and do not utilize existing first-order AD infrastructures; (3) inefficient data representation and manipulation that causes significant overhead at lowered-order when compared to nesting highly-optimized first-order AD libraries. By combining advanced techniques in computational science, i.e., aggressive type specializing, metaprogramming, and symbolic computing, we introduce a new implementation of Taylor mode automatic differentiation in Julia that addresses these problems. The new implementation shows that it is possible to achieve higher-order AD with minimal overhead and without sacrificing the performance of lower-order AD and obtain significant speedup in real-world scenarios over the existing Julia AD library. In addition, this implementation automatically generates higher-order AD rules from first-order AD rules, which is a step towards a general framework for higher-order AD.
MIT Department
Massachusetts Institute of Technology. Center for Computational Science and Engineering
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
