about
Large-scale multilingual audio visual dubbing (arxiv.org)
23 points by jonbaer on Nov 10, 2020 | hide | past | pdf | 2 comments on HN

In plain words: It translates a video by turning speech into text, translating it, and rebuilding the audio in the new language with the original speaker's voice. It also reshapes the speaker's lips to match the new words, so it looks and sounds like the same person speaking.

Abstract

We describe a system for large-scale audiovisual translation and dubbing, which translates videos from one language to another. The source language's speech content is transcribed to text, translated, and automatically synthesized into target language speech using the original speaker's voice. The visual content is translated by synthesizing lip movements for the speaker to match the translated audio, creating a seamless audiovisual experience in the target language. The audio and visual translation subsystems each contain a large-scale generic synthesis model trained on thousands of hours of data in the corresponding domain. These generic models are fine-tuned to a specific speaker before translation, either using an auxiliary corpus of data from the target speaker, or using the video to be translated itself as the input to the fine-tuning process. This report gives an architectural overview of the full system, as well as an in-depth discussion of the video dubbing component. The role of the audio and text components in relation to the full system is outlined, but their design is not discussed in detail. Translated and dubbed demo videos generated using our system can be viewed at https://www.youtube.com/playlist?list=PLSi232j2ZA6_1Exhof5vndzyfbxAhhEs5

Yi Yang, Brendan Shillingford, Yannis Assael, Miaosen Wang, Wendi Liu, Yutian Chen, Yu Zhang, Eren Sezener, Luis C. Cobo, Misha Denil, Yusuf Aytar, Nando de Freitas
arXiv:2011.03530 · cs.CV, cs.SD, eess.AS · submitted Nov 6, 2020
abstract · pdf · html · 26 pages, 8 figures

add comment on HN
Also discussed: Nov 2020 (1 point, 0 comments)

Definitely an ambitious project. Is there sample videos to evaluate?

I only skimmed this but I'm curious if this also kills the need for a voice actor or if it requires sample voice data from a native speaker to sound good (i.e. is it training from a corpus of attempts of the original actor trying to speak that language or does it just need some samples of different phonemes to train from?).