Researchers involved: Vinicius Macuch-Silva (ViCom Post-Doc-Fellow), Camila Antônio Barros (Freie Universität Berlin)
Research on the integration between speech and gesture has been concerned with how form-meaning associations can be constructed multimodally (Ebert et al., 2022; Esteve-Gibert & Prieto, 2013; Krahmer & Swerts, 2007). A key question in this line of work relates to how verbal and visual signals are integrated and temporally coordinated, normally centered in verbal landmarks (McNeill, 1992), stretches of speech that co-occur with gesture and enable a multimodal interpretation of the message conveyed in an utterance. This raises the question of what motivates the integration of gestures with different verbal landmarks. We tackle this issue by focusing on utterance interpretation in Portuguese and disentangling the integration of speech and gesture at the interface between pragmatics and phonology and pragmatics and semantics. More specifically, we investigate if modulating the integration of manual gestures with (i) phonological landmarks and (ii) pragmatic context surrounding the semantic landmark they are associated with impacts the meaning interpreters associate with those same gestures in context. We propose two experiments in which participants are asked to rate the adequacy of acted scenes containing multimodal utterances using a rating scale.
In the first experiment (2 x 2, between-participants), we investigate if the presence of a co-speech manual gesture indicating uncertainty (Palm Up Open Hand) can differentially impact the interpretation of functionally different nuclear accents, namely neutral declaratives and surprised declaratives. Based on previous work looking at the interplay between pitch accents and co-speech facial gestures in Portuguese (Carnaval et al., 2022), we hypothesize that when a manual gesture indicating uncertainty is produced with an accent associated with neutral declaratives, participants will rate the scenes as less adequate compared to when the same utterance is produced without any co-speech manual gesture, as the gesture is pragmatically incongruent with the meaning indexed by the co-occurring pitch accent. Conversely, we hypothesize that when a manual gesture indicating uncertainty is produced with a pitch accent associated with surprised declaratives, participants will rate the scenes as equally adequate or more adequate compared to when the same utterance is produced without any co-speech manual gesture, as the gesture is pragmatically congruent with the meaning indexed by the co-occurring pitch accent.
In the second experiment (single independent variable, between-participants), we investigate if the situational context in which a co-speech manual gesture is produced can shift the gesture’s perceived meaning contribution from a pragmatic to a semantic one. The target utterances contain a gesture with a recurrent form-meaning association which co-occurs with a semantic landmark indicating the speaker’s commitment to the content of their utterance, namely an epistemic adverbial phrase. Crucially, we manipulate whether the situational context favors a pragmatic or a semantic interpretation of the gesture, both of which are, in principle, possible, given the gesture’s meaning profile. In other words, we manipulate whether the discourse context provides the necessary condition for the gesture to co-index the speaker’s commitment alongside the verbal landmark. We hypothesize that in situations in which the context favors a pragmatic interpretation of the gesture, participants will rate the scenes as more adequate compared to situations in which the context favors a semantic interpretation of the gesture, as in the former the gesture’s meaning is congruent with the semantic landmark it is integrated with, while in the latter it is not.
Put together, the two experiments aim to test if synchrony to a specific verbal landmark (nuclear accent or adverbial phrase) can impact the meaning conveyed by a co-speech gesture. Our collaboration will take place in Berlin, where both collaborators currently reside. We will work on preparing the stimuli, recording experimental scenes that can be manipulated, dubbing the actors in the scene to modulate gestural placement relative to speech, and implementing the experiments. Ultimately, we will work towards preparing joint presentations and publications based on the study results.