MagicHand: A Deep Learning Approach towards Manipulating IoT Devices in Augmented Reality Environment
Name
magichand.pdf
Description
Submitted version
Size
5.83 MB
Format
Adobe PDF
Checksum (MD5)
245b52c2d8bc0149d5237e9bad0aff50
Author(s)
Sun, Yongbin
Armengol Urpi, Alexandre
Kantareddy, Sai Nithin R.
Siegel, Joshua E
Sarma, Sanjay E
Alternative Title
MagicHand: Interact with IoT Devices in Augmented Reality Environment
Date Issued
August 2019
Journal
26th IEEE Conference on Virtual Reality and 3D User Interfaces
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Sun, Yongbin et al. "MagicHand: Interact with IoT Devices in Augmented Reality Environment." 26th IEEE Conference on Virtual Reality and 3D User Interfaces, March 2019, Osaka, Japan, Institute of Electrical and Electronics Engineers, August 2019. © 2019 IEEE
Version
Original manuscript
Abstract
We present an Augmented Reality (AR) visualization and interaction tool for users to control Internet of Things (IoT) devices with hand gestures. Today, smart IoT devices are becoming increasingly ubiquitous with diverse forms and functions, yet most user controls over them are still limited to mobile devices and web interfaces. Recently, AR has been developed rapidly, and provided immersive solutions to enhance user experience of applications in many fields. Its capability to create immersive interactions allows AR to improve the way smart devices are controlled via more direct visual feedback. In this paper, we create a functional prototype of one such system, enabling seamless interactions with sound and lighting systems through the use of augmented hand-controlled interaction panels. To interpret users' intentions, we implement a standard 2D convolution neural network (CNN) for real-time hand gesture recognition and deploy it within our system. Our prototype is also equipped with a simple but effective object detector which can identify target devices within a proper range by analyzing geometric features. We evaluate the performance of our system qualitatively and quantitatively and demonstrate it on two smart devices.
MIT Department
Massachusetts Institute of Technology. Department of Mechanical Engineering
Massachusetts Institute of Technology. Office of Digital Learning
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/vr.2019.8798053
