Design Rules for LLM-Generated Code: A RealWorld Case Study

Abstract

This thesis conducts a case study exploring the interaction between software design, extensibility, and LLM code generation. The central problem we investigate is whether LLMs violate software design principles in ways that introduce bugs and ultimately hinder extensibility. We examine several repositories belonging to the RealWorld collection, a project that demonstrates combinations of frameworks, database, and programming languages for building full stack web apps modeled on an existing social media application. We create a concept-based implementation of the RealWorld API. Concept Design defines software systems in terms of the abstract purposes and relationships of self-contained units of functionality. It enforces stringent design standards and aims to aid humans better understand complex software behavior. To test code extensibility, we develop three phases of new functionality to be added to the RealWorld API. Each phase is intended to mimic real-world software development, adding functionality that is commonly found in social media platforms while increasing nuance and complexity. The code for these extensions is generated by an AI agent, then reviewed by a human coder who classifies and fixes any bugs. In this study, we examine how LLMs interact with software paradigms like Concept Design, the kinds of design violations they produce, and whether these violations correlate with bugs that impede extensibility.