Researchers involved: Anastasia Bauer (GeSi, University of Cologne), Alexander Mehler, Alexander Henlein and Andy Lücking (GeMDiS, Goethe University Frankfurt)
In this collaboration, we analyze mouthings in three sign languages. We compare authenticity judgments on the use of mouthing in German Sign Language (DGS), American Sign Language (ASL) and Russian Sign Language (RSL). The aim of the study is two-fold: (i) to compare authenticity judgments and the metalinguistic awareness of mouthings in three sign languages, and (ii) to examine the applicability of VR-applications for linguistic analysis of mouth movement in natural singing.
Recent studies show that ASL signers rate videos of ASL signing produced with lower mouthing more positively than those produced with higher mouthing (Bisnath 2024). Conversely, in DGS avatars producing signing without mouthing are perceived as highly unnatural by deaf signers (Kipp et al. 2011). Nothing is known about the perception of mouthing in RSL signing and there is no research to directly compare the frequency of mouthing between ASL and DGS. DGS is known to employ mouthings quite extensively (Ebbinghaus and Heßmann 1995), while RSL has been recently described to use mouthing significantly less in deaf natural signing (Bauer & Kyuseva 2020). Anecdotal evidence suggests that ASL might use even less mouthing than RSL. We therefore expect that DGS, ASL and RSL signers will differ in their authenticity judgments about the frequency of mouthing in their respective languages. To explore this, we apply the avatar technology of GeMDiS (Mehler et al. 2023; Henlein et al. 2023).
How the collaboration is designed to achieve these aims
The GeSi project is responsible for the acquisition of three deaf signers of DGS, ASL and RSL who each will produce the same 2–3 min narrative describing a museum exhibition while wearing Virtual Reality (VR) glasses (Meta Quest Pro, Va.Si.Li-Lab, Mehler et al. 2023, Henlein et al. 2023) at Goethe University Frankfurt. Each signed narrative will be recorded by tracking facilities of the VR glasses. A playback of the previously recorded signing is expected to reflect all multimodal cues relevant to our analysis, especially lip and mouth movements (see fig. 1 for stills of avatars). The avatars will be later manipulated resulting in 3 conditions per sign language that only differ in the rate of mouthing per manual sign: (i) high mouthing, mouthing on every manual sign, (ii) natural mouthing, no manipulation of original video, and (iii) zero mouthing, no mouthing on any manual sign, while mouth gestures remain present. Note that the difference in mouthing across conditions will differ by sign language because we expect that natural ASL signing will have the least mouthing, followed by RSL and then DGS. Stated differently, the difference in the quantity of mouthing between natural mouthing and high mouthing will be the greatest for ASL, least for DGS and middling for RSL. Note also that the visual appearance of the avatar and the genre of the signed text will be kept constant. Forty deaf participants per sign language (N=120) will be recruited online and asked to complete an authenticity judgment task and a metalinguistic awareness survey hosted on Qualtrics. Participants will watch each mouthing condition in their respective sign language, and rate them for naturalness using a continuous slider scale. Participants will afterwards also be asked questions about their use and knowledge of mouthings.
Results/Expectations
We distinguish the following two broad patterns in ratings, where a higher value indicates greater naturalness: 1. high < zero < natural mouthing, and 2. zero < high < natural mouthing. We expect that ASL and RSL follow the first pattern in ratings, while DGS conform to the second sequence. We expect that beliefs about the naturalness of different levels of mouthing will differ based on differences in the overall prevalence of mouthing in these sign languages. The question of naturalness gives an idea of how much signers perceive mouthing to be part of their sign languages, adding a much-needed evaluative dimension to the broader questioning of the status of mouthing found in sign language linguistics.
Results/Expectations
We distinguish the following two broad patterns in ratings, where a higher value indicates greater naturalness: 1. high < zero < natural mouthing, and 2. zero < high < natural mouthing. We expect that ASL and RSL follow the first pattern in ratings, while DGS conform to the second sequence. We expect that beliefs about the naturalness of different levels of mouthing will differ based on differences in the overall prevalence of mouthing in these sign languages. The question of naturalness gives an idea of how much signers perceive mouthing to be part of their sign languages, adding a much-needed evaluative dimension to the broader questioning of the status of mouthing found in sign language linguistics.
