Researchers involved: Pilar Prieto, Frank Kügler (MultIS), Ingrid Vilà-Giménez (University of Girona), Sandra Geladó (University of Pompeu), Sotaro Kita (Mercator Fellow)
Gesture-speech integration theories suggest that gestures and speech are part of the same integrated communication system (Alibali et al., 2000; Kendon, 1994; Kita et al., 2017; McNeill, 1992). The Gesture for Conceptualization Hypothesis (Kita et al., 2017) states that co-speech gestures serve self-oriented cognitive functions by activating, manipulating, packaging, and exploring visual and spatial information. In this way, gestures help to organize and process information, supporting the schematization of content and facilitating language processing.
However, existing research has mainly focused on referential gestures—those that visually represent semantic discourse content—while non-referential gestures, which emphasize important information and discourse structure without directly representing semantic content, remain relatively underexplored (e.g., Vilà-Giménez & Prieto, 2021; see also Rohrer et al., 2023 for detailed gesture definitions). As a result, non-referential gestures have not yet been fully integrated into established gesture-speech theoretical frameworks (e.g., Kita & Emmorey, 2023).
This collaboration addresses this critical gap by investigating how non-referential gestures contribute to discourse-pragmatic, self-oriented cognitive functions in human communication. Specifically, we aim to determine whether non-referential gestures activate, manipulate, package, and explore discourse information and structure in ways similar to referential gestures. Our main objective is to incorporate non-referential gestures into the Gesture for Conceptualization Hypothesis (Kita et al., 2017), thus expanding its theoretical scope.
To achieve this, we propose a within-subjects experimental study using a series of short oral narrative tasks. These tasks are designed to examine the four self-oriented cognitive functions (activation, manipulation, packaging, and exploration) in a narrative discourse context, under both a control condition and an experimental condition with increased cognitive demands. The study adopts a recent and inclusive view of non-referential gestures, accounting for their complex phasing structures and diverse hand shapes, following the Multimodal Multidimensional (M3D) system for gesture labeling (Rohrer et al., 2023).
One of the strengths of this collaboration lies in its interdisciplinary nature. Kita’s foundational contributions to gesture-speech integration theories provide a solid theoretical basis for expanding current frameworks during the preparation of the second phase of the ViCom project. The expertise of the MultIS group and Dr. Vilà-Giménez in non-referential gesture and oral communication research ensures a rigorous and innovative approach to integrating these gesture types into current gesture-speech theories.
This project represents a significant step forward in understanding human multimodal communication by addressing a critical gap in gesture-speech integration research. By better understanding the self-oriented cognitive and discourse-pragmatic functions of non-referential gestures, this collaboration will also open avenues for further investigation into the cognitive mechanisms underlying co-speech gesture production and deepen our understanding of the discourse functions of non-referential gestures.