2026-09-19T00:57:47Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/1230232026-06-06T00:49:08Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

Charles E. Leiserson. Wheatman, Brian. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2019-11-22T00:02:53Z 2019-11-22T00:02:53Z 2019 https://hdl.handle.net/1721.1/123023 1127649151 This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2019 "June 2019." Cataloged from student-submitted PDF version of thesis. Includes bibliographical references (pages 101-106). High performance applications are becoming increasingly resource hungry. We want to solve more complex problems and use more data to get higher quality results. However, the more data we store, the slower it is to access any piece. This effect is seen directly in the memory hierarchy. We can access our caches faster than our memory, which is faster than reading our disk, which is still faster than going across the network. This means that when processing large data sets, we can spend a large portion of our time simply in data movement. However, there is much we can do to optimize our programs to exploit our memory systems, so that we do not incur performance degradation as our datasets grow. I show how the careful design of data structures and algorithms allow us to scale to much larger datasets without impacting performance due to the cost of data movement. I demonstrate the impact of these designs with two case studies. The first examines large-scale image alignment, where I describe how to align a petabyte scale set of images in memory on a single machine and match the performance of current cluster solutions. I achieve .6 - .8 TB/hr on a medium-sized multicore and linear scalability on hundreds of nodes in a shared supercomputing cluster. The second case study explores dynamic graph analytics, where I describe the design of a new data structure for storing dynamic graphs that matches the performance of standard, static formats and enables high performance, dynamic operations achieving millions of updates per second. by Brian Wheatman. M.Eng. M.Eng. Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science 106 pages eng Massachusetts Institute of Technology MIT theses are protected by copyright. They may be viewed, downloaded, or printed from this source but further reproduction or distribution in any format is prohibited without written permission. http://dspace.mit.edu/handle/1721.1/7582 Electrical Engineering and Computer Science. Image alignment and dynamic graph analytics : two case studies of how managing data movement can make (parallel) code run fast 2 case studies of how managing data movement can make (parallel) code run fast Thesis application/pdf 2019-11-22T00:02:52Z Publication Master EECS unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="30bb3af8-bfc9-48d4-8b5b-d65ac0c5069a"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>eng</Language> <Title>Image alignment and dynamic graph analytics : two case studies of how managing data movement can make (parallel) code run fast</Title> <Subtitle>2 case studies of how managing data movement can make (parallel) code run fast</Subtitle> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2019</PublicationDate> <Authors> <Author> <DisplayName>Wheatman, Brian.</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://dspace.mit.edu/handle/1721.1/7582</License> <Keyword>Electrical Engineering and Computer Science.</Keyword> <Abstract>High performance applications are becoming increasingly resource hungry. We want to solve more complex problems and use more data to get higher quality results. However, the more data we store, the slower it is to access any piece. This effect is seen directly in the memory hierarchy. We can access our caches faster than our memory, which is faster than reading our disk, which is still faster than going across the network. This means that when processing large data sets, we can spend a large portion of our time simply in data movement. However, there is much we can do to optimize our programs to exploit our memory systems, so that we do not incur performance degradation as our datasets grow. I show how the careful design of data structures and algorithms allow us to scale to much larger datasets without impacting performance due to the cost of data movement. I demonstrate the impact of these designs with two case studies. The first examines large-scale image alignment, where I describe how to align a petabyte scale set of images in memory on a single machine and match the performance of current cluster solutions. I achieve .6 - .8 TB/hr on a medium-sized multicore and linear scalability on hundreds of nodes in a shared supercomputing cluster. The second case study explores dynamic graph analytics, where I describe the design of a new data structure for storing dynamic graphs that matches the performance of standard, static formats and enables high performance, dynamic operations achieving millions of updates per second.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>