Examples out of Thin Air: AI-Generated Dynamic Context to Assist Program Comprehension by Example

Abstract

Programmers often benefit from the availability of concrete run-time data alongside abstract source code. However, programmers need to manually exercise the program to reach an interesting state or write code that reproducibly executes a functionality with concrete inputs to be able to observe concrete data. This work aims to automate this process by leveraging generative AI. We present a framework and a preliminary Smalltalk-based prototype allowing programmers to obtain and run examples for the currently viewed source code section from a large language model. Our approach demonstrates how locally hosted LLMs can be fine-tuned and used for such a task with reasonable computational effort while minimizing common problems like hallucinations and out-of-date knowledge. The framework has direct applications in example-based live programming, where it can suggest new examples, and in learning settings where novices need to know how to use certain functionality.