Show simple item record

dc.contributor.authorRichardson, Mark
dc.contributor.authorDurasoff, Matt
dc.contributor.authorWang, Robert
dc.date.accessioned2025-02-12T17:33:17Z
dc.date.available2025-02-12T17:33:17Z
dc.date.issued2020-10-20
dc.identifier.isbn978-1-4503-7514-6
dc.identifier.urihttps://hdl.handle.net/1721.1/158198
dc.descriptionUIST ’20, October 20–23, 2020, Virtual Event, USAen_US
dc.description.abstractWe propose a novel text decoding method that enables touch typing on an uninstrumented flat surface. Rather than relying on physical keyboards or capacitive touch, our method takes as input hand motion of the typist, obtained through hand-tracking, and decodes this motion directly into text. We use a temporal convolutional network to represent a motion model that maps the hand motion, represented as a sequence of hand pose features, into text characters. To enable touch typing without the haptic feedback of a physical keyboard, we had to address more erratic typing motion due to drift of the fingers. Thus, we incorporate a language model as a text prior and use beam search to efficiently combine our motion and language models to decode text from erratic or ambiguous hand motion. We collected a dataset of 20 touch typists and evaluated our model on several baselines, including contact-based text decoding and typing on a physical keyboard. Our proposed method is able to leverage continuous hand pose information to decode text more accurately than contact-based methods and an offline study shows parity (73 WPM, 2.38% UER) with typing on a physical keyboard. Our results show that hand-tracking has the potential to enable rapid text entry in mobile environments.en_US
dc.publisherACM|Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology CD-ROMen_US
dc.relation.isversionofhttps://doi.org/10.1145/3379337.3415816en_US
dc.rightsCreative Commons Attributionen_US
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/en_US
dc.sourceAssociation for Computing Machineryen_US
dc.titleDecoding Surface Touch Typing from Hand-Trackingen_US
dc.typeArticleen_US
dc.identifier.citationRichardson, Mark, Durasoff, Matt and Wang, Robert. 2020. "Decoding Surface Touch Typing from Hand-Tracking."
dc.identifier.mitlicensePUBLISHER_CC
dc.eprint.versionFinal published versionen_US
dc.type.urihttp://purl.org/eprint/type/JournalArticleen_US
eprint.statushttp://purl.org/eprint/status/NonPeerRevieweden_US
dc.date.updated2025-02-01T08:46:00Z
dc.language.rfc3066en
dc.rights.holderThe author(s)
dspace.date.submission2025-02-01T08:46:00Z
mit.licensePUBLISHER_CC
mit.metadata.statusAuthority Work and Publication Information Neededen_US


Files in this item

Thumbnail
Thumbnail

This item appears in the following Collection(s)

Show simple item record