Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review

Producción Científica

Detalhes bibliográficos
Autores: Chaves Villota, Andrea, Jiménez Martín, Ana, Jojoa Acosta, Mario Fernando, Bahillo Martínez, Alfonso, García Domínguez, Juan Jesús
Formato: artículo
Estado:Versión publicada
Fecha de publicación:2026
País:España
Recursos:Universidad de Valladolid
Repositorio:UVaDOC. Repositorio Documental de la Universidad de Valladolid
OAI Identifier:oai:uvadoc.uva.es:10324/80150
Acesso em linha:https://doi.org/10.1016/j.csl.2025.101873
https://uvadoc.uva.es/handle/10324/80150
Access Level:acceso abierto
Palavra-chave:Emotion recognition
Speech
Linguistic
Acoustic
Fusion
Deep learning
Machine learning
Low and high-level features
id ES_3d3fd2e4741bdc64a48b086c077db70f
oai_identifier_str oai:uvadoc.uva.es:10324/80150
network_acronym_str ES
network_name_str España
repository_id_str
spelling Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic reviewChaves Villota, AndreaJiménez Martín, AnaJojoa Acosta, Mario FernandoBahillo Martínez, AlfonsoGarcía Domínguez, Juan JesúsEmotion recognitionSpeechLinguisticAcousticFusionDeep learningMachine learningLow and high-level featuresProducción CientíficaEmotion Recognition (ER) has gained significant attention due to its importance in advanced human-machine interaction and its widespread real-world applications. In recent years, research on ER systems has focused on multiple key aspects, including the development of high-quality emotional databases, the selection of robust feature representations, and the implementation of advanced classifiers leveraging AI-based techniques. Despite this progress in research, ER still faces significant challenges and gaps that must be addressed to develop accurate and reliable systems. To systematically assess these critical aspects, particularly those centered on AI-based techniques, we employed the PRISMA methodology. Thus, we include journal and conference papers that provide essential insights into key parameters required for dataset development, involving emotion modeling (categorical or dimensional), the type of speech data (natural, acted, or elicited), the most common modalities integrated with acoustic and linguistic data from speech and the technologies used. Similarly, following this methodology, we identified the key representative features that serve as critical emotional information sources in both modalities. For acoustic, this included those extracted from the time and frequency domains, while for linguistic, earlier embeddings and the most common transformer models were considered. In addition, Deep Learning (DL) and attention-based methods were analyzed for both. Given the importance of effectively combining these diverse features for improving ER, we then explore fusion techniques based on the level of abstraction. Specifically, we focus on traditional approaches, including feature-, decision-, DL-, and attention-based fusion methods. Next, we provide a comparative analysis to assess the performance of the approaches included in our study. Our findings indicate that for the most commonly used datasets in the literature: IEMOCAP and MELD, the integration of acoustic and linguistic features reached a weighted accuracy (WA) of 85.71% and 63.80%, respectively. Finally, we discuss the main challenges and propose future guidelines that could enhance the performance of ER systems using acoustic and linguistic features from speech.Proyecto FrailAlert SBPLY/21/180501/000216 cofinanciado por la Junta de Comunidades de Castilla-La Mancha y la Unión Europea a través del Fondo Europeo de Desarrollo RegionalActiTracker TED2021-130867B-I00 financiado por MCIN/AEI/10.13039/501100011033 y por European Union NextGenerationEU/PRTRINDRI (PID2021-122642OB-C41 /AEI/10.13039/501100011033/ FEDER, UE)Ministerio de Ciencia e Innovación bajo el proyecto PID2023-146254OB-C41Elsevier Ltd.2026info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfhttps://doi.org/10.1016/j.csl.2025.101873https://uvadoc.uva.es/handle/10324/80150reponame:UVaDOC. Repositorio Documental de la Universidad de Valladolidinstname:Universidad de ValladolidIngléshttps://www.sciencedirect.com/science/article/pii/S0885230825000981info:eu-repo/semantics/openAccesshttp://creativecommons.org/licenses/by/4.0/oai:uvadoc.uva.es:10324/801502026-06-13T12:44:47Z
dc.title.none.fl_str_mv Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
title Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
spellingShingle Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
Chaves Villota, Andrea
Emotion recognition
Speech
Linguistic
Acoustic
Fusion
Deep learning
Machine learning
Low and high-level features
title_short Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
title_full Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
title_fullStr Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
title_full_unstemmed Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
title_sort Deep feature representations and fusion strategies for speech emotion recognition from acoustic and linguistic modalities: A systematic review
dc.creator.none.fl_str_mv Chaves Villota, Andrea
Jiménez Martín, Ana
Jojoa Acosta, Mario Fernando
Bahillo Martínez, Alfonso
García Domínguez, Juan Jesús
author Chaves Villota, Andrea
author_facet Chaves Villota, Andrea
Jiménez Martín, Ana
Jojoa Acosta, Mario Fernando
Bahillo Martínez, Alfonso
García Domínguez, Juan Jesús
author_role author
author2 Jiménez Martín, Ana
Jojoa Acosta, Mario Fernando
Bahillo Martínez, Alfonso
García Domínguez, Juan Jesús
author2_role author
author
author
author
dc.subject.none.fl_str_mv Emotion recognition
Speech
Linguistic
Acoustic
Fusion
Deep learning
Machine learning
Low and high-level features
topic Emotion recognition
Speech
Linguistic
Acoustic
Fusion
Deep learning
Machine learning
Low and high-level features
description Producción Científica
publishDate 2026
dc.date.none.fl_str_mv 2026
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv https://doi.org/10.1016/j.csl.2025.101873
https://uvadoc.uva.es/handle/10324/80150
url https://doi.org/10.1016/j.csl.2025.101873
https://uvadoc.uva.es/handle/10324/80150
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv https://www.sciencedirect.com/science/article/pii/S0885230825000981
dc.rights.none.fl_str_mv info:eu-repo/semantics/openAccess
http://creativecommons.org/licenses/by/4.0/
eu_rights_str_mv openAccess
rights_invalid_str_mv http://creativecommons.org/licenses/by/4.0/
dc.format.none.fl_str_mv application/pdf
dc.publisher.none.fl_str_mv Elsevier Ltd.
publisher.none.fl_str_mv Elsevier Ltd.
dc.source.none.fl_str_mv reponame:UVaDOC. Repositorio Documental de la Universidad de Valladolid
instname:Universidad de Valladolid
instname_str Universidad de Valladolid
reponame_str UVaDOC. Repositorio Documental de la Universidad de Valladolid
collection UVaDOC. Repositorio Documental de la Universidad de Valladolid
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869406427960836096
score 15,812429

Registos relacionados