Weld: A Common Runtime for High Performance Data Analytics
Name
cidr_weld.pdf
Description
Published version
Size
229.41 KB
Format
Unknown
Checksum (MD5)
4d51324b4c55a09dc7b96657d1c9d559
Author(s)
Palkar, Shoumik
Thomas, James J.
Shanbhag, Anil Atmanand
Narayanan, Deepak
Pirk, Holger
Schwarzkopf, Malte
Amarasinghe, Saman P
Zaharia, Matei A
Date Issued
January 2017
Journal
CIDR 2017 - 8th Biennial Conference on Innovative Data Systems Research
Citation
Palkar, S, Thomas, JJ, Shanbhag, A, Narayanan, D, Pirk, H et al. 2017. "Weld: A common runtime for high performance data analytics." CIDR 2017 - 8th Biennial Conference on Innovative Data Systems Research.
Version
Final published version
Abstract
© 2017 Conference on Innovative Data Systems Research (CIDR). All rights reserved. Modern analytics applications combine multiple functions from different libraries and frameworks to build increasingly complex workflows. Even though each function may achieve high performance in isolation, the performance of the combined workflow is often an order of magnitude below hardware limits due to extensive data movement across the functions. To address this problem, we propose Weld, a runtime for data-intensive applications that optimizes across disjoint libraries and functions. Weld uses a common intermediate representation to capture the structure of diverse data-parallel workloads, including SQL, machine learning and graph analytics. It then performs key data movement optimizations and generates efficient parallel code for the whole workflow. Weld can be integrated incrementally into existing frameworks like TensorFlow, Apache Spark, NumPy and Pandas without changing their user-facing APIs. We show that Weld can speed up these frameworks, as well as applications that combine them, by up to 30×.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution 3.0 unported license
Persistent DSpace Link
