In plain words: They tested what makes pretraining a speech translator on speech-recognition data from a data-rich language help low-resource languages, varying data size and language closeness. How often the pretrained recognizer got words wrong best predicted translation quality, and pretraining added gains beyond data augmentation.
Abstract
Previous work has shown that for low-resource source languages, automatic speech-to-text translation (AST) can be improved by pretraining an end-to-end model on automatic speech recognition (ASR) data from a high-resource language. However, it is not clear what factors --e.g., language relatedness or size of the pretraining data-- yield the biggest improvements, or whether pretraining can be effectively combined with other methods such as data augmentation. Here, we experiment with pretraining on datasets of varying sizes, including languages related and unrelated to the AST source language. We find that the best predictor of final AST performance is the word error rate of the pretrained ASR model, and that differences in ASR/AST performance correlate with how phonetic information is encoded in the later RNN layers of our model. We also show that pretraining and data augmentation yield complementary benefits for AST.
Mihaela C. Stoian, Sameer Bansal, Sharon Goldwater
arXiv:1910.10762 · cs.CL, eess.AS · submitted Oct 23, 2019 · updated Feb 9, 2020
abstract · pdf · html · Accepted at ICASSP 2020