A bimodal deep model to capture emotions from music tracks

This work aims to develop a deep model for automatically labeling music tracks in terms of induced emotions. The machine learning architecture consists of two components: one dedicated to lyric processing based on Natural Language Processing (NLP) and another devoted to music processing. These two c...

Descripción completa

Detalles Bibliográficos
Autores: Tobolewski, Jan, Sakowicz, Michal, Turmo Borras, Jorge|||0000-0002-7521-1115, Kostek, Bozena
Tipo de recurso: artículo
Fecha de publicación:2025
País:España
Institución:Universitat Politècnica de Catalunya (UPC)
Repositorio:UPCommons. Portal del coneixement obert de la UPC
Idioma:inglés
OAI Identifier:oai:upcommons.upc.edu:2117/428993
Acceso en línea:https://hdl.handle.net/2117/428993
https://dx.doi.org/10.2478/jaiscr-2025-0011
Access Level:acceso abierto
Palabra clave:Automatic labeling
Deep model
Emotion
Music
lyrics
Machine learning
Deep learning
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la parla i del senyal acústic
id ES_e8a45a3f1e50bdf7c9e864d3ef8c71e2
oai_identifier_str oai:upcommons.upc.edu:2117/428993
network_acronym_str ES
network_name_str España
repository_id_str
spelling A bimodal deep model to capture emotions from music tracksTobolewski, JanSakowicz, MichalTurmo Borras, Jorge|||0000-0002-7521-1115Kostek, BozenaAutomatic labelingDeep modelEmotionMusiclyricsMachine learningDeep learningÀrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàticÀrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la parla i del senyal acústicThis work aims to develop a deep model for automatically labeling music tracks in terms of induced emotions. The machine learning architecture consists of two components: one dedicated to lyric processing based on Natural Language Processing (NLP) and another devoted to music processing. These two components are combined at the decision-making level. To achieve this, a range of neural networks are explored for the task of emotion extraction from both lyrics and music. For lyric classification, three architectures are compared, i.e., a 4-layer neural network, FastText, and a transformer-based approach. For music classification, the architectures investigated include InceptionV3, a collection of models from the ResNet family, and a joint architecture combining Inception and ResNet. SVM serves as a baseline in both threads. The study explores three datasets of songs accompanied by lyrics, with MoodyLyrics4Q selected and preprocessed for model training. The bimodal approach, incorporating both lyrics and audio modules, achieves a classification accuracy of 60.7% in identifying emotions evoked by music pieces. The MoodyLyrics4Q dataset used in this study encompasses musical pieces spanning diverse genres, including rock, jazz, electronic, pop, blues, and country. The algorithms demonstrate reliable performance across the dataset, highlighting their robustness in handling a wide variety of musical styles.Peer Reviewed20252025-07-0120252025-05-08journal articlehttp://purl.org/coar/resource_type/c_6501VoRhttp://purl.org/coar/version/c_970fb48d4fbd8a85info:eu-repo/semantics/articleapplication/pdfhttps://hdl.handle.net/2117/428993https://dx.doi.org/10.2478/jaiscr-2025-0011reponame:UPCommons. Portal del coneixement obert de la UPCinstname:Universitat Politècnica de Catalunya (UPC)Inglésengopen accesshttp://purl.org/coar/access_right/c_abf2Attribution 4.0 Internationalhttp://creativecommons.org/licenses/by/4.0/info:eu-repo/semantics/openAccessoai:upcommons.upc.edu:2117/4289932026-05-27T15:37:01Z
dc.title.none.fl_str_mv A bimodal deep model to capture emotions from music tracks
title A bimodal deep model to capture emotions from music tracks
spellingShingle A bimodal deep model to capture emotions from music tracks
Tobolewski, Jan
Automatic labeling
Deep model
Emotion
Music
lyrics
Machine learning
Deep learning
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la parla i del senyal acústic
title_short A bimodal deep model to capture emotions from music tracks
title_full A bimodal deep model to capture emotions from music tracks
title_fullStr A bimodal deep model to capture emotions from music tracks
title_full_unstemmed A bimodal deep model to capture emotions from music tracks
title_sort A bimodal deep model to capture emotions from music tracks
dc.creator.none.fl_str_mv Tobolewski, Jan
Sakowicz, Michal
Turmo Borras, Jorge|||0000-0002-7521-1115
Kostek, Bozena
author Tobolewski, Jan
author_facet Tobolewski, Jan
Sakowicz, Michal
Turmo Borras, Jorge|||0000-0002-7521-1115
Kostek, Bozena
author_role author
author2 Sakowicz, Michal
Turmo Borras, Jorge|||0000-0002-7521-1115
Kostek, Bozena
author2_role author
author
author
dc.subject.none.fl_str_mv Automatic labeling
Deep model
Emotion
Music
lyrics
Machine learning
Deep learning
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la parla i del senyal acústic
topic Automatic labeling
Deep model
Emotion
Music
lyrics
Machine learning
Deep learning
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Aprenentatge automàtic
Àrees temàtiques de la UPC::Enginyeria de la telecomunicació::Processament del senyal::Processament de la parla i del senyal acústic
description This work aims to develop a deep model for automatically labeling music tracks in terms of induced emotions. The machine learning architecture consists of two components: one dedicated to lyric processing based on Natural Language Processing (NLP) and another devoted to music processing. These two components are combined at the decision-making level. To achieve this, a range of neural networks are explored for the task of emotion extraction from both lyrics and music. For lyric classification, three architectures are compared, i.e., a 4-layer neural network, FastText, and a transformer-based approach. For music classification, the architectures investigated include InceptionV3, a collection of models from the ResNet family, and a joint architecture combining Inception and ResNet. SVM serves as a baseline in both threads. The study explores three datasets of songs accompanied by lyrics, with MoodyLyrics4Q selected and preprocessed for model training. The bimodal approach, incorporating both lyrics and audio modules, achieves a classification accuracy of 60.7% in identifying emotions evoked by music pieces. The MoodyLyrics4Q dataset used in this study encompasses musical pieces spanning diverse genres, including rock, jazz, electronic, pop, blues, and country. The algorithms demonstrate reliable performance across the dataset, highlighting their robustness in handling a wide variety of musical styles.
publishDate 2025
dc.date.none.fl_str_mv 2025
2025-07-01
2025
2025-05-08
dc.type.none.fl_str_mv journal article
http://purl.org/coar/resource_type/c_6501
VoR
http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.openaire.fl_str_mv info:eu-repo/semantics/article
format article
dc.identifier.none.fl_str_mv https://hdl.handle.net/2117/428993
https://dx.doi.org/10.2478/jaiscr-2025-0011
url https://hdl.handle.net/2117/428993
https://dx.doi.org/10.2478/jaiscr-2025-0011
dc.language.none.fl_str_mv Inglés
eng
language_invalid_str_mv Inglés
language eng
dc.rights.none.fl_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 4.0 International
http://creativecommons.org/licenses/by/4.0/
dc.rights.openaire.fl_str_mv info:eu-repo/semantics/openAccess
rights_invalid_str_mv open access
http://purl.org/coar/access_right/c_abf2
Attribution 4.0 International
http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.source.none.fl_str_mv reponame:UPCommons. Portal del coneixement obert de la UPC
instname:Universitat Politècnica de Catalunya (UPC)
instname_str Universitat Politècnica de Catalunya (UPC)
reponame_str UPCommons. Portal del coneixement obert de la UPC
collection UPCommons. Portal del coneixement obert de la UPC
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869422959321415680
score 15.811543

Similares en LA Referencia