An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems
Name
MIT-CSAIL-TR-2016-003.pdf
Size
447.55 KB
Format
Adobe PDF
Checksum (MD5)
14a68d1770b08301755f33580c20ac00
Download all files submitted through automated deposit
Author(s)
Long, Fan
Rinard, Martin
Advisor(s)
Martin Rinard
Date Issued
February 18, 2016
Series/Report no.
MIT-CSAIL-TR-2016-003
Abstract
We present the first systematic analysis of the characteristics of patch search spaces for automatic patch generation systems. We analyze the search spaces of two current state-of- the-art systems, SPR and Prophet, with 16 different search space configurations. Our results are derived from an analysis of 1104 different search spaces and 768 patch generation executions. Together these experiments consumed over 9000 hours of CPU time on Amazon EC2.The analysis shows that 1) correct patches are sparse in the search spaces (typically at most one correct patch per search space per defect), 2) incorrect patches that nevertheless pass all of the test cases in the validation test suite are typically orders of magnitude more abundant, and 3) leveraging information other than the test suite is therefore critical for enabling the system to successfully isolate correct patches.We also characterize a key tradeoff in the structure of the search spaces. Larger and richer search spaces that contain correct patches for more defects can actually cause systems to find fewer, not more, correct patches. We identify two reasons for this phenomenon: 1) increased validation times because of the presence of more candidate patches and 2) more incorrect patches that pass the test suite and block the discovery of correct patches. These fundamental properties, which are all characterized for the first time in this paper, help explain why past systems often fail to generate correct patches and help identify challenges, opportunities, and productive future directions for the field.
Subjects
Program repair
Persistent DSpace Link
