Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks
Name
Zaharia_Tachyon.pdf
Size
514.55 KB
Format
Adobe PDF
Checksum (MD5)
fcde207416c78f475a26b5db1022fde8
Author(s)
Li, Haoyuan
Ghodsi, Ali
Shenker, Scott
Stoica, Ion
Zaharia, Matei A.
Date Issued
November 2014
Journal
Proceedings of the ACM Symposium on Cloud Computing (SOCC '14)
Publisher
Association for Computing Machinery (ACM)
Citation
Haoyuan Li, Ali Ghodsi, Matei Zaharia, Scott Shenker, and Ion Stoica. 2014. Tachyon: Reliable, Memory Speed Storage for Cluster Computing Frameworks. In Proceedings of the ACM Symposium on Cloud Computing (SOCC '14). ACM, New York, NY, USA, Article 6 , 15 pages.
Version
Author's final manuscript
Abstract
Tachyon is a distributed file system enabling reliable data sharing at memory speed across cluster computing frameworks. While caching today improves read workloads, writes are either network or disk bound, as replication is used for fault-tolerance. Tachyon eliminates this bottleneck by pushing lineage, a well-known technique, into the storage layer. The key challenge in making a long-running lineage-based storage system is timely data recovery in case of failures. Tachyon addresses this issue by introducing a checkpointing algorithm that guarantees bounded recovery cost and resource allocation strategies for recomputation under commonly used resource schedulers. Our evaluation shows that Tachyon outperforms in-memory HDFS by 110x for writes. It also improves the end-to-end latency of a realistic workflow by 4x. Tachyon is open source and is deployed at multiple companies.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/2670979.2670985
