A platform for reaching into the environment of a remote collaborator

Abstract

In this thesis we present ShowMe++, an immersive mobile collaboration system that allows a remote user to communicate with a peer using video, audio and hand gestures. We explore the use of a Head Mounted Display (HMD), depth camera and wearable haptic devices to create a system that (1) enables a remote user to be immersed in another first-person's point of view, (2) offers a new way for the remote expert to provide guidance through three dimensional, real-time hand gestures and voice, (3) allows natural interactions with interfaces of Internet of Things (IoT) devices and (4) provides haptic feedback when interacting with remote or virtual interfaces. Using our system, both users feel present in the same physical environment and can perceive realtime communication from one another in the form of 2-handed gestures and voice. We discuss the design and implementation of the system as well as applications scenarios such as remote maintenance, 3D exploration and remote ghost presence. The user study demonstrates that hand transmission, first person point of view and immersion improve the feeling of co-presence and make remote teaching more effective.