Designing Visual Intelligence from Photons to Action

Abstract

For embodied agents to perceive and effectively act within their environment, they must sense the world around them and translate this information into meaningful and safe actions; a process fundamental to both biological and human-engineered systems. Nature has evolved highly attuned visual systems, resulting in diverse and efficient eyes capable of facilitating complex behaviors. Conversely, roboticists have engineered sophisticated cameras and sensors, enabling robots to perform tasks beyond the capabilities of natural systems. This thesis explores the design of visual intelligence by integrating insights from both biology and engineering in two complementary parts. In Part I, we computationally recreate the evolution of vision within simulated embodied agents. By evolving the physical and neural aspects of vision in simulation - and training these visually-capable agents with deep reinforcement learning - we demonstrate that task-specific environmental pressures lead to distinct eye morphologies and behaviors, mirroring observations in biological evolution. This in silico approach enables us to investigate the fundamental principles underlying the emergence of animal eyes and provides a framework for exploring novel sensor designs subject to both biological (e.g., survival) and engineering constraints (e.g., manufacturability). In Part II, we leverage visual cues not typically used in nature (i.e., active illumination and multi-bounce light) to demonstrate enhanced robotic navigation via non-line-of-sight imaging. Using single-photon LiDARs, we capture the temporal propagation of individual photons, enabling the detection of objects around corners. This sensing capability allows us to develop robots that effectively anticipate and avoid hidden obstacles, reducing navigation time by 50% and overall trajectory length by 33%. Together, these works demonstrate how the synthesis of biologically-inspired design principles with advanced sensing modalities can enhance embodied agents' capabilities, while providing insights into both natural vision evolution and robotic perception.