How do people signal attention, understanding, and engagement through visual cues during conversation? This project investigates visual feedback in spontaneous spoken and signed dyadic interactions, exploring how head movements, facial expressions, and other signals vary across languages, modalities, and social contexts. Using diverse spoken and signed languages from around the world, the research examines the role of cognitive differences and familiarity between interlocutors. The findings will deepen our understanding of multimodal communication and highlight the importance of visual feedback in everyday interaction.
Project Members
Anastasia Bauer
Principal Investigator
University of Cologne, Germany
Project Description
This project aims to systematically investigate the dynamics of visual feedback in spontaneous spoken and signed dyadic interactions, identifying factors that account for variation in feedback production. While previous research has highlighted the critical role of feedback in human interaction, particularly visual feedback still remains understudied. Our preliminary work has shown that visual feedback occurs far more frequently in human interaction than previously assumed. Despite its essential role, we lack a comprehensive understanding of the factors influencing variation in feedback production.
This project will explore how cross-linguistic and cross-modal differences, social-cognitive abilities, and familiarity between interlocutors shape visual feedback production. By examining a diverse range of spoken and signed languages, including those that are underresearched, the study will provide insights into the linguistic, cognitive, and social factors that influence feedback dynamics in human interaction.
The research questions center on cross-linguistic differences in visual feedback, the impact of individual social-cognitive abilities, and the influence of familiarity between interlocutors on feedback patterns. The project incorporates existing corpora and collaborates with global researchers to examine a range of typologically diverse languages such as Russian, German, Bulgarian, Polish, Yurakarée, Datooga, Khoekhoe, Russian Sign Language, German Sign Language, Kata Kolok, Balinese Homesign and others.
The project examines whether speakers and signers of Indo-European languages predominantly use head movements as feedback signals, while those of non-Indo-European languages rely on alternative forms. Additionally, the study investigates how social-cognitive differences, such as Autism Spectrum Disorder, and conversational familiarity influence the frequency and type of visual feedback, with unfamiliar interlocutors using more visual cues to establish common ground.
By shifting the focus from unimodal to multimodal interactional phenomena, the project advocates for a paradigm shift in conceptualizing language, highlighting the significant role of visual cues in daily interactions.