about
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio (arxiv.org)
2 points by graderjs on Mar 5, 2023 | hide | past | pdf | discuss on HN

In plain words: Long audio is first chopped into speech-only chunks, transcribed in batches, then matched to the sounds in each word to give exact word timings. This beats the usual sliding-window approach on long recordings and runs twelve times faster.

Abstract

Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.

Max Bain, Jaesung Huh, Tengda Han, Andrew Zisserman
arXiv:2303.00747 · cs.SD, eess.AS · submitted Mar 1, 2023 · updated Jul 11, 2023
abstract · pdf · html · Accepted to INTERSPEECH 2023

add comment on HN