MemPal: Leveraging Multimodal AI and LLMs for Voice-Activated Object Retrieval in Homes of Older Adults
Name
3708359.3712151.pdf
Size
21.51 MB
Format
Adobe PDF
Checksum (MD5)
8e13b7ab2c15e248af0f15435ee41856
Author(s)
Maniar, Natasha
Chan, Samantha
Zulfikar, Wazeer
Ren, Scott
Xu, Christine
Maes, Pattie
Date Issued
March 24, 2025
Publisher
ACM|30th International Conference on Intelligent User Interfaces
Citation
Maniar, Natasha, Chan, Samantha, Zulfikar, Wazeer, Ren, Scott, Xu, Christine et al. 2025. "MemPal: Leveraging Multimodal AI and LLMs for Voice-Activated Object Retrieval in Homes of Older Adults."
Version
Final published version
Abstract
Older adults have increasing difficulty with retrospective memory, hindering their abilities to perform daily activities and posing stress on caregivers to ensure their wellbeing. Recent developments in Artificial Intelligence (AI) and large context-aware multimodal models offer an opportunity to create memory support systems that assist older adults with common issues like object finding. This paper discusses the development of an AI-based, wearable memory assistant, MemPal, that helps older adults with a common problem, finding lost objects at home, and presents results from tests of the system in older adults’ own homes. Using visual context from a wearable camera, the multimodal LLM system creates a real-time automated text diary of the person’s activities for memory support purposes, offering object retrieval assistance using a voice-based interface. The system is designed to support additional use cases like context-based proactive safety reminders and recall of past actions. We report on a quantitative and qualitative study with N=15 older adults within their own homes that showed improved performance of object finding with audio-based assistance compared to no aid and positive overall user perceptions on the designed system. We discuss further applications of MemPal’s design as a multi-purpose memory aid and future design guidelines to adapt memory assistants to older adults’ unique needs.
Description
IUI ’25, Cagliari, Italy
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-ShareAlike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3708359.3712151
