Utilization and Synthesis of Symbolic World Models for Safe, Generalizable, and Efficient Action

Abstract

Reinforcement learning with neural networks has proven incredibly flexible at learning to act in diverse environments. Model-based RL techniques have helped to ameliorate the dependence on large quantities of data that these models normally have. However, despite their flexibility, neural world models have several drawbacks. Symbolic world models, in comparison, are easier to verify (e.g. for safety concerns), more compatible with domain-independent planning techniques, and able to be learned or adapted with more limited data. In this thesis, I will demonstrate these advantages of symbolic world models in three projects. The first, VSRL, shows how we can use a symbolic world model to ensure that an RL policy is safe during both training and deployment and promote safe exploration. The second, SPARSER, presents a hybrid domain planner which uses world models in a planning domain description language. It showcases how we can exploit the event structure in the world model to enable more efficient planning. In the final project, PWM, I will explore learning a world model directly from observations and actions gathered from interacting with an environment. We combine symbolic and neural synthesis techniques to enable efficient world model synthesis even from visual observations. Together, these projects demonstrate the versatility and value of symbolic world models.