Skip to main content Scroll Top
Head and Hand Movements During Turn Transitions: Data-Based Multimodal Analysis Using the Frankfurt VR Gesture–Speech Alignment Corpus (FraGA)

Authors:
Lücking, A., Voll, F., Rott, D., Henlein, A., Mehler, A.

Year / Status:
2025

Journal / Venue:
Proceedings of the 29th Workshop on The Semantics and Pragmatics of Dialogue

Abstract

We introduce FraGA, a VR-based corpus of direction giving dialogues following the model of the SaGA corpus. The tracking data of FraGA are used to carry out multimodal computing: we look at turn transitions and re-evaluate findings from the literature on real-world dialogues and compare them with avatar-mediated VR dialogues. Interestingly, the established temporal overlap patterns of turns could not be replicated, and no significant amount of partner-directed head orientation (approximating gaze) was observed around turn transition points. However, the special status of hesitations is evidenced by the co-occurring head movements, but not by hand movements. In addition, we apply pink noise distribution fitting to the dialogue data, in particular to the hand movements. Here we find that hand movements indeed follow 1/f fluctuations, a property of "meta-stable" dynamic systems.