2026-09-19T03:47:09Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1447672022-08-30T03:37:23Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Sanchez, Daniel
Lee, Hyun Ryong
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2022-08-29T16:10:21Z
2022-08-29T16:10:21Z
2022-05
2022-06-21T19:25:45.947Z
https://hdl.handle.net/1721.1/144767
0000-0002-8627-2781
Benchmarks that closely match the behavior of production workloads are crucial to design and provision computer systems. However, current approaches fall short: First, open-source benchmarks use public datasets that cause different behavior from production workloads. Second, black-box workload cloning techniques generate synthetic code that imitates the target workload, but the resulting program fails to capture most workload characteristics, such as microarchitectural bottlenecks or time-varying behavior.
Generating code that mimics a complex application is an extremely hard problem. Instead, this thesis proposes a different and easier approach to benchmark synthesis. The key insight is that for many production workloads the program is publicly available, or there is a reasonably similar open-source program. In this case, generating the right dataset is sufficient to produce an accurate benchmark.
Based on this observation, this thesis presents Datamime, a profile-guided approach to generate representative benchmarks for production workloads. Datamime uses the performance profiles of a target workload to generate a dataset that, when used by a benchmark program, behaves very similarly to the target workload in terms of its microarchitectural characteristics.
We evaluate Datamime on several datacenter workloads. Datamime generates synthetic benchmarks that closely match the microarchitectural features of these workloads, with a mean absolute percentage error of 4% on IPC. Microarchitectural behavior stays close across processor types. Finally, time-varying behaviors are also replicated, making these benchmarks useful to e.g. characterize and optimize tail latency.
S.M.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
Generating Representative Benchmarks by Automatically Synthesizing Datasets
Thesis
application/pdf
Master
Master of Science in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="7491c4ce-90a5-4bbd-bfee-53ec03479545">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Generating Representative Benchmarks by Automatically Synthesizing Datasets</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2022-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Lee, Hyun Ryong</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Benchmarks that closely match the behavior of production workloads are crucial to design and provision computer systems. However, current approaches fall short: First, open-source benchmarks use public datasets that cause different behavior from production workloads. Second, black-box workload cloning techniques generate synthetic code that imitates the target workload, but the resulting program fails to capture most workload characteristics, such as microarchitectural bottlenecks or time-varying behavior.
Generating code that mimics a complex application is an extremely hard problem. Instead, this thesis proposes a different and easier approach to benchmark synthesis. The key insight is that for many production workloads the program is publicly available, or there is a reasonably similar open-source program. In this case, generating the right dataset is sufficient to produce an accurate benchmark.
Based on this observation, this thesis presents Datamime, a profile-guided approach to generate representative benchmarks for production workloads. Datamime uses the performance profiles of a target workload to generate a dataset that, when used by a benchmark program, behaves very similarly to the target workload in terms of its microarchitectural characteristics.
We evaluate Datamime on several datacenter workloads. Datamime generates synthetic benchmarks that closely match the microarchitectural features of these workloads, with a mean absolute percentage error of 4% on IPC. Microarchitectural behavior stays close across processor types. Finally, time-varying behaviors are also replicated, making these benchmarks useful to e.g. characterize and optimize tail latency.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>