Multimedia for language learning

Abstract

Students studying foreign languages often wish to enjoy authentic foreign-language content - for example, foreign-language videos and comics. Existing means of presenting this content, however, are suboptimal from the perspective of language learning. We have developed a pair of tools that aim to help learners acquire the foreign language while enjoying authentic, foreign-language material. One tool is Smart Subtitles, which help learners learn vocabulary while watching videos. Our user evaluations have shown that Smart Subtitles help learners learn vocabulary more effectively than a popular approach for learning from video (dual subtitles). The other tool is a grammar visualization which illustrates the grammatical structure of sentences. This grammar visualization can be embedded into a number of language-learning applications. This includes a foreign-language manga reader we have built, which aims to help learners learn vocabulary, grammar and pronunciation while reading comics and manga. Our study for the grammar visualization shows that it helps monolinguals arrive at more accurate translations if we have an oracle taking the best translation of all, though there was no significant improvement in the average translation quality.