A temporal and pragmatic analysis of gestures-speech association: a corpus-based approach using the novel Multimodal MultiDimensional (M3D) labeling system
Human language is essentially multimodal in that speakers use multiple channels to convey meaning, including speech prosody and gesture (e.g. Mondada, 2016; Perniss, 2018). In the last decades, studies within the field of gesture research have shown both the strong temporal relationship between manu...
| Autor: | |
|---|---|
| Tipo de documento: | tese |
| Estado: | Versão publicada |
| Data de publicação: | 2022 |
| País: | España |
| Recursos: | CBUC, CESCA |
| Repositório: | TDR. Tesis Doctorales en Red |
| OAI Identifier: | oai:www.tdx.cat:10803/687534 |
| Acesso em linha: | http://hdl.handle.net/10803/687534 |
| Access Level: | Acceso aberto |
| Palavra-chave: | Gestures-speech association M3D MultiModal MultiDimension (M3D) labeling system 81 |
| Resumo: | Human language is essentially multimodal in that speakers use multiple channels to convey meaning, including speech prosody and gesture (e.g. Mondada, 2016; Perniss, 2018). In the last decades, studies within the field of gesture research have shown both the strong temporal relationship between manual co-speech gestures and prosodic prominence, and have given initial evidence of the relevant pragmatic role of gestures. However, gesture studies have shown a tendency to focus on the role of prosodic prominence alone as the main attractor for gesture production, and little empirical research has systematically assessed the role of prosodic phrasal structure in the attraction of gesture, or the joint contribution of gestural and prosodic prominence for pragmatic effects, particularly in terms of signaling information structure (henceforth, IS). Furthermore, no studies have specifically accounted for potential difference in gesture type (i.e., referential vs. non-referential gestures). In our view, a multidimensional analysis of independent aspects of gesture is crucial to allow for a systematic assessment of their different prosodic and pragmatic characteristics. The two main goals of this thesis will be to develop a novel gesture labeling system (i.e., the MultiModal MultiDimensional (M3D) system) and to apply the system to better understand the prosodic and pragmatic characteristics of both referential and non-referential gestures. The present PhD thesis will consist of four independent studies plus introductory and conclusion sections that unite the four studies. The first study proposes M3D as a novel tool for multidimensional gesture annotation that is in line with the advancing theories in gesture. Through the application of M3D to a corpus of French TED Talks (5 TED Talks with over 37 minutes of multimodal speech), the second study shows how phrase-initial accents act as strong gestural attractors regardless of gesture type, and how the production of multiple subsequent gestures is largely guided by the temporal duration of prosodic phrases. To further examine the effects of phrasal position, a third study was carried out on English TED Talks (5 TED Talks with over 28 minutes of multimodal speech), assessing the temporal association of gestures with pitch accentuation while systematically taking into account the effects of nuclear status and degrees of relative prominence. The results highlight the role of prenuclear pitch accentuation as a strong attractor of gesture, independent of relative prominence. Finally, the fourth study assesses the joint role of prosody and gesture in the marking of IS (particularly, the information status of referents; henceforth, ISR) in the same corpus of English TED Talks. The results show how prominence (via pitch accentuation) and the production of gesture work together to mark newer information in speech, with pitch accent type and gesture type not playing key roles as cues to ISR. All in all, the four studies contained in this thesis offer a novel gesture annotation tool that can be used for the development of multimodal corpora accounting for a variety of aspects of speech, gesture, and prosody. The empirical studies further our knowledge about the temporal association of gesture and speech, showing that not only prosodic prominence, but also prosodic phrasing are key to understanding the relationship between the two channels. The studies also further our knowledge in terms of how these two channels interact to convey pragmatic meaning. Thus, this multidimensional analysis of gesture greatly contributes to the ongoing effort to elucidate the precise nature of the temporal and pragmatic properties of both referential and non-referential gestures in discursive speech. |
|---|