Many SWE-bench-Passing PRs Would Not Be Merged into Main
Name
whitfill-whitfill-sm-econ-2026-thesis.pdf
Size
946.98 KB
Format
Adobe PDF
Checksum (MD5)
2f7bd9b5a8a06fd0c2ffa46b4cb49417
Author(s)
Whitfill, Parker
Advisor(s)
Mullainathan, Sendhil
Date Issued
May 2026
Publisher
Massachusetts Institute of Technology
Abstract
We find that roughly half of test-passing SWE-bench Verified PRs written by mid-2024 to mid/late-2025 agents would not be merged into main by repo maintainers, even after adjusting for noise in maintainer merge decisions. Since the agents are not given a chance to iterate on their solution in response to feedback the way a human developer would, we do not claim that this represents a fundamental capability limitation. Rather, our results indicate that a naive interpretation of benchmark scores may lead one to overestimate how useful agents are without more elicitation or human feedback.
MIT Department
Massachusetts Institute of Technology. Department of Economics
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
