Optimizing MapReduce for Multicore Architectures
Name
MIT-CSAIL-TR-2010-020.pdf
Size
222.64 KB
Format
Adobe PDF
Checksum (MD5)
b5382c3a035abd6f05161cad9a562f2d
Author(s)
Kaashoek, Frans
Morris, Robert
Mao, Yandong
Advisor(s)
Frans Kaashoek
Date Issued
May 2, 2010
Series/Report no.
MIT-CSAIL-TR-2010-020
Abstract
MapReduce is a programming model for data-parallel programs originally intended for data centers. MapReduce simplifies parallel programming, hiding synchronization and task management. These properties make it a promising programming model for future processors with many cores, and existing MapReduce libraries such as Phoenix have demonstrated that applications written with MapReduce perform competitively with those written with Pthreads. This paper explores the design of the MapReduce data structures for grouping intermediate key/value pairs, which is often a performance bottleneck on multicore processors. The paper finds the best choice depends on workload characteristics, such as the number of keys used by the application, the degree of repetition of keys, etc. This paper also introduces a new MapReduce library, Metis, with a compromise data structure designed to perform well for most workloads. Experiments with the Phoenix benchmarks on a 16-core AMD-based servershow that Metisâ data structure performs better than simpler alternatives, including Phoenix.
Persistent DSpace Link
