2026-09-19T00:44:15Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1515012023-08-01T03:01:38Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

Edelman, Alan Tan, Songchen Massachusetts Institute of Technology. Center for Computational Science and Engineering 2023-07-31T19:44:39Z 2023-07-31T19:44:39Z 2023-06 2023-06-13T13:14:18.189Z https://hdl.handle.net/1721.1/151501 https://orcid.org/0009-0008-6168-3462 Differentiable programming is a new paradigm for modeling and optimization in many fields of science and engineering, and automatic differentiation (AD) algorithms are at the heart of differentiable programming. Existing methods to achieve higher-order AD often suffer from one or more of the following problems: (1) exponential scaling with respect to order due to nesting first-order AD; (2) ad-hoc handwritten higher-order rules which are hard to maintain and do not utilize existing first-order AD infrastructures; (3) inefficient data representation and manipulation that causes significant overhead at lowered-order when compared to nesting highly-optimized first-order AD libraries. By combining advanced techniques in computational science, i.e., aggressive type specializing, metaprogramming, and symbolic computing, we introduce a new implementation of Taylor mode automatic differentiation in Julia that addresses these problems. The new implementation shows that it is possible to achieve higher-order AD with minimal overhead and without sacrificing the performance of lower-order AD and obtain significant speedup in real-world scenarios over the existing Julia AD library. In addition, this implementation automatically generates higher-order AD rules from first-order AD rules, which is a step towards a general framework for higher-order AD. S.M. Massachusetts Institute of Technology In Copyright - Educational Use Permitted Copyright retained by author(s) https://rightsstatements.org/page/InC-EDU/1.0/ Higher-Order Automatic Differentiation and Its Applications Thesis application/pdf Master Master of Science in Computational Science and Engineering Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="fb104543-4661-4ecb-93c5-83d668397d79"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Title>Higher-Order Automatic Differentiation and Its Applications</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2023-06</PublicationDate> <Authors> <Author> <DisplayName>Tan, Songchen</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>https://rightsstatements.org/page/InC-EDU/1.0/</License> <Abstract>Differentiable programming is a new paradigm for modeling and optimization in many fields of science and engineering, and automatic differentiation (AD) algorithms are at the heart of differentiable programming. Existing methods to achieve higher-order AD often suffer from one or more of the following problems: (1) exponential scaling with respect to order due to nesting first-order AD; (2) ad-hoc handwritten higher-order rules which are hard to maintain and do not utilize existing first-order AD infrastructures; (3) inefficient data representation and manipulation that causes significant overhead at lowered-order when compared to nesting highly-optimized first-order AD libraries. By combining advanced techniques in computational science, i.e., aggressive type specializing, metaprogramming, and symbolic computing, we introduce a new implementation of Taylor mode automatic differentiation in Julia that addresses these problems. The new implementation shows that it is possible to achieve higher-order AD with minimal overhead and without sacrificing the performance of lower-order AD and obtain significant speedup in real-world scenarios over the existing Julia AD library. In addition, this implementation automatically generates higher-order AD rules from first-order AD rules, which is a step towards a general framework for higher-order AD.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>