Multisensory integration of speech and gestures in a naturalistic paradigm

Speech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand...

Descripción completa

Detalles Bibliográficos
Autores: Matyjek, Magdalena, Kita, Sotaro, Torralba Cuello, Mireia, Soto-Faraco, Salvador, 1970-
Tipo de recurso: artículo
Estado:Versión publicada
Fecha de publicación:2024
País:España
Institución:Universitat Pompeu Fabra
Repositorio:Repositorio Digital de la UPF
OAI Identifier:oai:dnet:rdupf_______::09e53b417255169fff86869284b4e275
Acceso en línea:https://hdl.handle.net/10230/72943
http://dx.doi.org/10.1002/hbm.26797
Access Level:acceso abierto
Palabra clave:Audio-visual speech
EEG
Iconic gestures
Multisensory integration
id ES_4eecd28feb2748002696eafb4c863e65
oai_identifier_str oai:dnet:rdupf_______::09e53b417255169fff86869284b4e275
network_acronym_str ES
network_name_str España
repository_id_str
spelling Multisensory integration of speech and gestures in a naturalistic paradigmMatyjek, MagdalenaKita, SotaroTorralba Cuello, MireiaSoto-Faraco, Salvador, 1970-Audio-visual speechEEGIconic gesturesMultisensory integrationSpeech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand this process in response to complex, naturalistic stimuli encountered in everyday life. This study investigated behavioural and neural MSI in neurotypical adults experiencing audio-visual speech within a naturalistic, social context. Our novel paradigm incorporated a broader social situational context, complete words, and speech-supporting iconic gestures, allowing for context-based pragmatics and semantic priors. We investigated MSI in the presence of unimodal (auditory or visual) or complementary, bimodal speech signals. During audio-visual speech trials, compared to unimodal trials, participants more accurately recognised spoken words and showed a more pronounced suppression of alpha power'an indicator of heightened integration load. Importantly, on the neural level, these effects surpassed mere summation of unimodal responses, suggesting non-linear MSI mechanisms. Overall, our findings demonstrate that typically developing adults integrate audio-visual speech and gesture information to facilitate speech comprehension in noisy environments, highlighting the importance of studying MSI in ecologically valid contexts.This project has received funding from the European Union's Horizon 2020 Research and Innovation Programme under the Marie Skłodowska-Curie grant agreement No. 945380, Agencia Estatal de Investigación grant No. PID2019-108531GB-I00 AEI/FEDER, and AGAUR Generalitat de Catalunya grant 2021 SGR 00911.Wiley2026202620242026info:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionapplication/pdfapplication/pdfhttps://hdl.handle.net/10230/72943http://dx.doi.org/10.1002/hbm.26797reponame:Repositorio Digital de la UPFinstname:Universitat Pompeu FabraInglésHuman Brain Mapping. 2024 Aug 1;45(11):e26797info:eu-repo/grantAgreement/EC/H2020/945380info:eu-repo/grantAgreement/ES/2PE/PID2019-108531GB-I00© 2024 The Author(s). Human Brain Mapping published by Wiley Periodicals LLC. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.https://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessoai:dnet:rdupf_______::09e53b417255169fff86869284b4e2752026-06-12T07:21:37Z
dc.title.none.fl_str_mv Multisensory integration of speech and gestures in a naturalistic paradigm
title Multisensory integration of speech and gestures in a naturalistic paradigm
spellingShingle Multisensory integration of speech and gestures in a naturalistic paradigm
Matyjek, Magdalena
Audio-visual speech
EEG
Iconic gestures
Multisensory integration
title_short Multisensory integration of speech and gestures in a naturalistic paradigm
title_full Multisensory integration of speech and gestures in a naturalistic paradigm
title_fullStr Multisensory integration of speech and gestures in a naturalistic paradigm
title_full_unstemmed Multisensory integration of speech and gestures in a naturalistic paradigm
title_sort Multisensory integration of speech and gestures in a naturalistic paradigm
dc.creator.none.fl_str_mv Matyjek, Magdalena
Kita, Sotaro
Torralba Cuello, Mireia
Soto-Faraco, Salvador, 1970-
author Matyjek, Magdalena
author_facet Matyjek, Magdalena
Kita, Sotaro
Torralba Cuello, Mireia
Soto-Faraco, Salvador, 1970-
author_role author
author2 Kita, Sotaro
Torralba Cuello, Mireia
Soto-Faraco, Salvador, 1970-
author2_role author
author
author
dc.subject.none.fl_str_mv Audio-visual speech
EEG
Iconic gestures
Multisensory integration
topic Audio-visual speech
EEG
Iconic gestures
Multisensory integration
description Speech comprehension is crucial for human social interaction, relying on the integration of auditory and visual cues across various levels of representation. While research has extensively studied multisensory integration (MSI) using idealised, well-controlled stimuli, there is a need to understand this process in response to complex, naturalistic stimuli encountered in everyday life. This study investigated behavioural and neural MSI in neurotypical adults experiencing audio-visual speech within a naturalistic, social context. Our novel paradigm incorporated a broader social situational context, complete words, and speech-supporting iconic gestures, allowing for context-based pragmatics and semantic priors. We investigated MSI in the presence of unimodal (auditory or visual) or complementary, bimodal speech signals. During audio-visual speech trials, compared to unimodal trials, participants more accurately recognised spoken words and showed a more pronounced suppression of alpha power'an indicator of heightened integration load. Importantly, on the neural level, these effects surpassed mere summation of unimodal responses, suggesting non-linear MSI mechanisms. Overall, our findings demonstrate that typically developing adults integrate audio-visual speech and gesture information to facilitate speech comprehension in noisy environments, highlighting the importance of studying MSI in ecologically valid contexts.
publishDate 2024
dc.date.none.fl_str_mv 2024
2026
2026
2026
dc.type.none.fl_str_mv info:eu-repo/semantics/article
info:eu-repo/semantics/publishedVersion
format article
status_str publishedVersion
dc.identifier.none.fl_str_mv https://hdl.handle.net/10230/72943
http://dx.doi.org/10.1002/hbm.26797
url https://hdl.handle.net/10230/72943
http://dx.doi.org/10.1002/hbm.26797
dc.language.none.fl_str_mv Inglés
language_invalid_str_mv Inglés
dc.relation.none.fl_str_mv Human Brain Mapping. 2024 Aug 1;45(11):e26797
info:eu-repo/grantAgreement/EC/H2020/945380
info:eu-repo/grantAgreement/ES/2PE/PID2019-108531GB-I00
dc.rights.none.fl_str_mv https://creativecommons.org/licenses/by-nc-nd/4.0/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv https://creativecommons.org/licenses/by-nc-nd/4.0/
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
application/pdf
dc.publisher.none.fl_str_mv Wiley
publisher.none.fl_str_mv Wiley
dc.source.none.fl_str_mv reponame:Repositorio Digital de la UPF
instname:Universitat Pompeu Fabra
instname_str Universitat Pompeu Fabra
reponame_str Repositorio Digital de la UPF
collection Repositorio Digital de la UPF
repository.name.fl_str_mv
repository.mail.fl_str_mv
_version_ 1869407787520360448
score 15,811543

Similares en LA Referencia