In plain words: SUSing turns lyrics and a music score into a 2D map, then a U-shaped network predicts the sound's frequency pattern one step at a time, using long thin pooling to track how pitch changes over time. On a public singing dataset it produced more natural-sounding voices than the usual approach.
Abstract · SUSing: SU-net for Singing Voice Synthesis
Singing voice synthesis is a generative task that involves multi-dimensional control of the singing model, including lyrics, pitch, and duration, and includes the timbre of the singer and singing skills such as vibrato. In this paper, we proposed SU-net for singing voice synthesis named SUSing. Synthesizing singing voice is treated as a translation task between lyrics and music score and spectrum. The lyrics and music score information is encoded into a two-dimensional feature representation through the convolution layer. The two-dimensional feature and its frequency spectrum are mapped to the target spectrum in an autoregressive manner through a SU-net network. Within the SU-net the stripe pooling method is used to replace the alternate global pooling method to learn the vertical frequency relationship in the spectrum and the changes of frequency in the time domain. The experimental results on the public dataset Kiritan show that the proposed method can synthesize more natural singing voices.
Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao
arXiv:2205.11841 · cs.SD, eess.AS · submitted May 24, 2022
abstract · pdf · html · Accepted by IJCNN2022 (The 2022 International Joint Conference on Neural Networks)