Building Intelligence that can Interact with the Physical World

Abstract

Recent advances in Artificial Intelligence (AI) have demonstrated remarkable success in parsing, reasoning, and generating digital content across modalities such as natural language, speech, images, videos, and 3D data. However, these breakthroughs have yet to extend meaningfully beyond the digital realm into the physical world. Developing AI for physical interaction poses challenges such as limited grounding, scarce physical data, and high reliability demands in safety-critical settings. This thesis takes a holistic approach to building intelligence that can interact with the physical world – through the lenses of data, brain, and body. Data is the fuel powering highly capable AI systems. We present methods for data-driven simulation that synthesize sensor measurements from physical processes, and knowledge-driven simulation that leverages large language models to generate actor behaviors and scenarios. By reverse engineering the generative processes behind physical data, we address data scarcity while enabling scalable and effective evaluation. The brain, driven by data, demands a deep understanding of the physical world and reliable interaction with it. We introduce methods to bridge the internet-scale knowledge of digital AI with the physical world to improve generalization and interpretability. For greater reliability, we integrate control-theoretic modules into AI models to enable certifiability. Beyond the behavioral intelligence, the body plays a crucial role in physical interaction. We demonstrate how morphological intelligence can emerge from computation and show how pre-trained generative AI models (brain), when augmented with physics-based simulation that provides feedback on generated data, can be applied to robot design. To this end, this thesis explores how digital AI can be extended into the physical world through a comprehensive investigation of data, brain, and body – laying the groundwork for building physical AI.