A multimodal Ouija board for aircraft carrier deck operations

Abstract

In this thesis, we present improvements to DeckAssistant, a system that provides a traditional Ouija board interface by displaying a digital rendering of an aircraft carrier deck that assists deck handlers in planning deck operations. DeckAssistant has a large digital tabletop display that shows the status of the deck and has an understanding of certain deck actions for scenario planning. To preserve the conventional way of interacting with the old-school Ouija board where deck handlers move aircraft by hand, the system takes advantage of multiple modes of interaction. Deck handlers plan strategies by pointing at aircraft, gesturing and talking to the system. The system responds with its own speech and gestures, and it updates the display to show the consequences of the actions taken by the handlers. The system can also be used to simulate certain scenarios during the planning process. The multimodal interaction described here creates a communication of sorts between deck handlers and the system. Our contributions include improvements in hand-tracking, speech synthesis and speech recognition.