Language-independent methods for computer-assisted pronunciation training

Abstract

Computer-assisted pronunciation training (CAPT) systems help students practice speaking foreign languages by providing automatic pronunciation assessment and corrective feedback. Automatic speech recognition (ASR) technology is a natural component in CAPT systems. Since a nonnative speaker's native language (Li) background affects their pronunciation patterns in a target language (L2), typically not only native but also nonnative training data of specific Ls is needed to train a recognizer for CAPT systems. Given that there are around 7,000 languages in the world, the data collection process is costly and has scalability issues. In addition, expert knowledge on the target L2 is also often needed to design a large feature set describing the deviation of nonnative speech from native speech. In contrast to machines, it is relatively easy for native listeners to detect pronunciation errors without being exposed to nonnative speech or trained with linguistic knowledge beforehand. In this thesis, we are interested in this unsupervised capability and propose methods to overcome the language-dependent challenges. Inspired by the success of unsupervised acoustic pattern discovery, we propose to discover an individual learner's pronunciation error patterns in an unsupervised manner by analyzing the acoustic similarity between speech segments from the learner. Experimental results on nonnative English and nonnative Mandarin Chinese spoken by students from different Ls show that the proposed method is Li-independent and can be portable to different L2s. Moreover, the method is personalized such that it accommodates variations in pronunciation patterns across students. In addition, motivated by the success of deep learning models in unsupervised feature learning, we explore the use of convolutional neural networks (CNNs) for mispronunciation detection. A language-independent data augmentation method is developed to take advantage of native speech as training samples. Experimental results on nonnative Mandarin Chinese speech show the effectiveness of the model and the method. Moreover, both qualitative and quantitative analyses on the convolutional filters reveal that the CNN automatically learns a set of human-interpretable high-level features.