Audio-visual deep learning methods for musical instrument classification and separation
In music perception, the information we receive from a visual system and audio system is often complementary. Moreover, visual perception plays an important role in the overall experience of being exposed to a music performance. This fact brings attention to machine learning methods that could combi...
| Autor: | |
|---|---|
| Tipo de documento: | tese |
| Estado: | Versão publicada |
| Data de publicação: | 2020 |
| País: | España |
| Recursos: | CBUC, CESCA |
| Repositório: | TDR. Tesis Doctorales en Red |
| OAI Identifier: | oai:www.tdx.cat:10803/669963 |
| Acesso em linha: | http://hdl.handle.net/10803/669963 |
| Access Level: | Acceso aberto |
| Palavra-chave: | Audio-visual deep learning Multimodal deep learning Music information retrieval Musical performance video Musical performance analysis Musical instrument classification Sound source separation Fusion techniques Conditioning techniques Aprendizaje profundo audiovisual Aprendizaje profundo multimodal Recuperación de información musical Video musical Análisis de interpretación musical Clasificación de instrumentos musicales Separación de fuentes de sonido Técnicas de fusión Técnicas de acondicionamiento 62 |
| Resumo: | In music perception, the information we receive from a visual system and audio system is often complementary. Moreover, visual perception plays an important role in the overall experience of being exposed to a music performance. This fact brings attention to machine learning methods that could combine audio and visual information for automatic music analysis. This thesis addresses two research problems: instrument classification and source separation in the context of music performance videos. A multimodal approach for each task is developed using deep learning techniques to train an encoded representation for each modality. For source separation, we also study two approaches conditioned on instrument labels and examine the influence that two extra sources of information have on separation performance compared with a conventional model. Another important aspect of this work is in the exploration of different fusion methods which allow for better multimodal integration of information sources from associated domains. |
|---|