2026-09-20T04:05:17Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1059772026-06-06T00:55:23Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Matei Zaharia.
Mahajan, Rohan
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2016-12-22T15:17:11Z
2016-12-22T15:17:11Z
2016
2016
http://hdl.handle.net/1721.1/105977
965643791
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2016.
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Cataloged from student-submitted PDF version of thesis.
Includes bibliographical references (page 33).
Because most data processing systems are distributed in nature, data must be transferred between machines. Currently, Spark, a prominent such system, predetermines the strategies for shuffling this data, but in certain situations, different shuffle strategies would improve performance. We add functionality to track metrics about the data during the job and appropriately adapt the shuffle strategy. We show improvements in ShuffledRDD performance, joins using Spark's RDD interface, and joins in Spark SQL.
by Rohan Mahajan.
M.Eng.
33 pages
eng
Massachusetts Institute of Technology
M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.
http://dspace.mit.edu/handle/1721.1/7582
Electrical Engineering and Computer Science.
Adaptive scheduling in Spark
Thesis
application/pdf
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="a3cacf47-af8f-4e5a-855f-8836d1ffefa0">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Language>eng</Language>
<Title>Adaptive scheduling in Spark</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2016</PublicationDate>
<Authors>
<Author>
<DisplayName>Mahajan, Rohan</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://dspace.mit.edu/handle/1721.1/7582</License>
<Keyword>Electrical Engineering and Computer Science.</Keyword>
<Abstract>Because most data processing systems are distributed in nature, data must be transferred between machines. Currently, Spark, a prominent such system, predetermines the strategies for shuffling this data, but in certain situations, different shuffle strategies would improve performance. We add functionality to track metrics about the data during the job and appropriately adapt the shuffle strategy. We show improvements in ShuffledRDD performance, joins using Spark's RDD interface, and joins in Spark SQL.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>