LEARNSTEM : an interactive and intuitive STEM accessibility system

Abstract

I present an intuitive and interactive platform to make complex STEM (Science Technology Engineering and Mathematics) educational materials accessible to blind and visually impaired people using a mini-hyper-Braille device with almost no loss of information as compared to printed materials. I have come up with a novel way to increase the effective resolution of the Braille device by adding a mechanical XY gantry. Using this XY gantry enables us to create an ultra-high resolution, larger surface Braille device without paying the prohibitive price charged for the bigger hyper Braille displays available in the market. In addition to that, to further augment usability and intuitiveness of our system, we have integrated a Nod-ring, which is a tiny finger-worn device for supporting additional hand gestures such as zoom in and out. Previous studies have shown that the use of zoom and pan can increase usability and improve the understanding of tactile graphics. Along with zooming and panning, our system uses vibrating patterns, rhythmic motions, synthetic voice and synchronized voiced-vibrations to convey information to blind users in an intuitive way. We also implemented a touch gesture recognition framework on our touch enabled Braille device. Using these touch gestures and a high quality synthetic voice, we have developed a highly responsive system for providing voice annotations of the graphics content. An important contribution of this work is the implementation of a high-quality system for automatic transcription of STEM (including difficult math Nemeth translations) books into Braille. Learning resources for blind people are quite sparse and transcription of STEM material is a very expensive and time consuming process. Using our automated transcription platform it is easy, fast and inexpensive for publishers to transcribe STEM books into Braille books. The scope of this automated transcription platform is not only limited to STEM books but it can be used for transcription of any content/book/web-page available online.