Towards audio-visual deep learning methods for singing voice separation and lip synchronization
A typical music audio consists of different sounds spread across voices and musical instruments. We propose a low-cost deep learning (DL) method for simultaneously isolating multiple sound sources present in a music audio. The specific case of isolating a voice in a music audio containing more than...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis doctoral |
| Estado: | Versión publicada |
| Fecha de publicación: | 2022 |
| País: | España |
| Institución: | CBUC, CESCA |
| Repositorio: | TDR. Tesis Doctorales en Red |
| OAI Identifier: | oai:www.tdx.cat:10803/675490 |
| Acceso en línea: | http://hdl.handle.net/10803/675490 |
| Access Level: | acceso abierto |
| Palabra clave: | Music source separation Lip synchronization Audio-visual models Deep learning Multimodal learning Separació de fonts musicals Sincronització de llavis Models audiovisuals Aprenentatge profund Aprenentatge multimodal Separación de fuentes de música Sincronización de labios Maquetas audiovisuales Aprendizaje profundo Aprendizaje multimodal 62 |
| Sumario: | A typical music audio consists of different sounds spread across voices and musical instruments. We propose a low-cost deep learning (DL) method for simultaneously isolating multiple sound sources present in a music audio. The specific case of isolating a voice in a music audio containing more than one overlapping singing voice is particularly challenging. The existing source separation methods operating in audio modality alone do not perform well in isolating a particular singing voice in such multi-voice settings. To address this problem, we present an audio-visual DL method that makes use of motion features sourced from the visual modality to assist in the singing voice separation task. Finally, we develop a new DL model to detect the synchronization between lip motion and a voice audio. We show that the visual features learned by our lip synchronization detector serves as a useful means in guiding the singing voice separation in multi-voice music videos. |
|---|