Supercharging Programming through Compiler Technology

Abstract

The decline of Moore’s law and an increasing reliance on computation has led to an explosion of specialized software packages and hardware architectures. While this diversity enables unprecedented flexibility, it also requires domain-experts to learn how to customize programs to efficiently leverage the latest platform-specific API’s and data structures, instead of working on their intended problem. For example, a researcher hoping to use machine learning on climate code must write a corresponding derivative simulation, understand and implement linear algebra routines, and performance engineer their simulation to run on multiple cores and nodes. Rather than forcing each user to bear this burden, I propose building high-level abstractions within general-purpose compilers that enable fast, portable, and composable programs to be automatically generated.

This thesis will demonstrate this approach through several real-world and composable compilers that I built for a variety of domains including parallelism, automatic differentiation, scheduling, portability, program search, and tensor arithmetic. These domains are critical to both scientific computing and machine learning. Individually, integration of domain knowledge into each of these compilers enable (often asymptotic) performance and usability benefits. Operating on a common compiler representation, however, enables these benefits to compound and provide greater performance than any domain-specific optimization in isolation.

This research in this thesis contains joint work with Charles E. Leiserson, Tao B. Schardl, Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, Zachary, Sven Verdoolaege, Andrew Adams, Albert Cohen, Qijing (Jenny) Huang, Ameer Haj-Ali, John Xiang, Ion Stoica, Krste Asanovic, John Wawrzynek, Valentin Churavy, Lorenzo Chelini, Ruizhe Zhao, Ludger Paehler, Jan Hückelheim, Sri Hari Krishna Narayanan, Michel Schanen, Johannes Doerfert, Paul Hovland, Ivan R. Ivanov, Jens Domke, and Toshio Endo.