Multimodal AI for Human Sensing and Interaction

Abstract

A significant body of HCI research today focuses on applying AI to sense, learn, and interact with humans through a wide range of wearable and ubiquitous sensors. These methods typically involve learning features from multimodal sensory data using AI methods. To aid HCI researchers who want to apply AI to their sensing problems, this course will cover the fundamental challenges and approaches in multimodal AI for human sensing and interaction. It is planned for 3 parts, one given by each organizer. The first covers the foundations of multimodal AI, studying how AI systems can represent, combine, and learn information from many interconnected sensory inputs. The second part discusses the practice of multimodal AI for human sensing, covering the latest methods for cross-modal learning across diverse sensors, human-centered application domains, and real-world concerns around their usage. The final part covers the hardware, fabrication, and data collection challenges that must be tackled to deploy these multimodal AI systems in the real world. By the end of this course, attendees should understand the fundamental principles and challenges of multimodal AI, identify the right AI approaches for their problems, prototype basic hardware systems for efficient and robust sensing, be aware of real-world concerns around ethics, interpretability, and privacy, and appreciate the range of human-centered applications enabled by multimodal AI and sensing.