A mobile phone GuiDE : a GUI Decoder and Enricher

Abstract

Much information can be obtained from knowing what tasks the user does on his or her cell phone. This paper describes GuiDE, an automated user-activity recognition system on the mobile phone. GuiDE's unique approach to activity recognition exploits GUI screenshots taken as the individual interacts with their cell phone. These screenshots are aggregated into a graph to help probabilistically determine whether or not a set of screenshots can be considered a user-level activity. A frequency count of different sets of screenshots is also kept to act as a sanity check against the probabilistic result. GuiDE is just a partial step towards a much more powerful tool that can correlate GUI information with other services to provide a better understanding of user activity.