The CoSGrIn-VR project addresses the computational semantic mechanisms underlying speech-gesture integration, a research gap in formal semantics and multimodal communication studies. Little is known in computational semantics about how gestures acquire meaning and how gesture meaning combines with speech meaning. The project challenges holistic and gestalt-based models by developing a computational semantics of speech-gesture integration that is (1) formally modelled, (2) computationally implementable, (3) experimentally testable in virtual reality (VR), and (4) cognitively interpretable.

CoSGrIn-VR focuses on two key research areas: (i) developing computational semantic models for recognizing acting gestures and (ii) extending the GeMDiS-Model from the first phase to handle graded exemplification, addressing cases where gestures only partially or indirectly affiliate with speech. The first research area examines how perceptual action classifiers can account for acting gestures, which simulate actions. The second research area investigates the informational uncertainty in gesture-speech integration by studying quantified noun phrases and references to atypical objects.

Methodologically, the project employs Interaction Verification Games within VR environments to test its models. These immersive experiments capture multimodal data, including body movements, gestures, spatial behavior, and gaze. Additionally, CoSGrIn-VR aims to develop an AI-based VR lab to enhance experimental control using avatars and automate the annotation of multimodal communication. A key component of this approach is the Multi-perspective Annotation Model (MAM), which enables the systematic and largely automatic annotation of multimodal experimental data.

This integration of linguistic theory, computational modeling, and VR-based empirical testing positions CoSGrIn-VR as a novel approach to understanding the graded nature of speech-gesture integration and its implications for visual communication.