Researchers involved: Susanne Fuchs (FLESH, ZAS Berlin) and Alina Gregori (MultIS, Goethe University Frankfurt)
With this collaboration, we aim to gain deeper insights into a) the importance of the coordination of prosody and gesture and b) the general influence that multimodal non-lexical cues have on the disambiguation of a sentence.
Prosody and gesture have been shown to coordinate with each other, forming an integrated system in communication. Several empirical studies have found evidence for the temporal coordination of prosodic and gestural cues, showing that gestural strokes associate with prominent syllables in speech. In this study, we want to assess how prosody-gesture (mis)alignment can influence the disambiguation of an ambiguous sentence using the “Next Wednesday Question” (NWQ) paradigm. The NWQ paradigm is a well-known method in cognitive science which can also be used to test linguistic phenomena. In such a study, participants have to answer the question “Next Wednesday’s meeting has been moved forward two days. On what day is it now?” (or similar) and based on their response “Monday” or “Friday”, conclusions can be drawn. Apart from linguistic insights, the NWQ paradigm has been used to test intuitions about time, for instance, whether a person perceived time from a “Moving Ego” or from a “Moving Time” perspective. The answer to the question thus depends on how the spatial adverbial “forward” is interpreted. While in general, responses to the NWQ are divided evenly into “Monday” and “Friday” responses, it has been found that gestures can influence the response: the gesture amplifies a direction in time that is expressed in speech with the adverbial. This has been tested with gestures moving on a horizontal axis, thus expressing a spatial relation matching the adverbial. However, whether and how prosody (accentuation in this case) of the utterance influences the perception of time in this paradigm has not been investigated. Thus, it is interesting to test the NWQ paradigm multimodally in a mismatch study, to assess the role of prosody in the interpretation of the spatial adverbial “forward” with regard to time and to explore the contribution of prosody, gesture and their synchronization to the disambiguation of ambiguous sentences.
Study design:
To test responses to a multimodally manipulated NWQ, we will conduct a perception study, applying the NWQ paradigm in the English language in a between-subject design. The study is split up into two sub-studies. The first study will investigate the gestural influence on the interpretation of the question when the spatial adverbial is varied – present, not present or replaced by a (pro-speech) gesture. A horizontal gesture is produced while the (above mentioned) NWQ is produced, with the speakers’ hand moving towards or away from their body. The gesture is produced on the adverbial when present, and when no adverbial is produced, there are two options: The gesture is produced earlier on the verb “moved” or a speech break is inserted replacing the adverbial, during which the gesture is produced as a pro-speech gesture or. Sentence accentuation is produced on the adverbial when present, or on the verb “moved” when there is no adverbial. The first study thus applies a 2 x 3 design with the factors Gesture (towards / away from speaker horizontally) and Adverbial (present, absent, replaced by pro-speech gesture).
The second study will address the question whether the placement of prosodic cues (pitch accents) and gestural cues (gestures from study 1) influences the responses to the NWQ. To test this, the NWQ is posed with the same horizontal gesture moving towards or away from the speaker’s body. This time, the spatial adverbial in speech is additionally varied in directionality (“forward” vs. “backward”) in order to create a mismatch component between gesture and speech. Finally, either the pitch accent or the gesture is moved away from the adverbial (where they would be placed by default) to the verb “moved” to create a mismatch between prosody and gesture. This allows to investigate whether these cues then still act together and whether one of them is the more important cue for disambiguation. The second study is thus a 2 x 2 x 2 design with the factors Gesture (towards / away from speaker horizontally), Language direction (forward / backwards) and Moved cue (prosody moved / gesture moved). We plan on collecting data from 250 participants per substudy, thus 500 participants in total.