Researchers involved: Šárka Kadavá (FLESH, ZAS Berlin) and Door Spruijt (The Gesture-to-Sign Trajectory: Phonological Parameters in Production and Real-Time Comprehension, University of Cologne)
The aim of the collaboration between GtST and FLESH is to bring our two datasets and methodologies together to perform two analyses on a subset of concepts and video recordings that are shared between the projects. Each analysis will be led by one of the applicants. The two stays are intended for interim discussions on progress in both analyses. Moreover, since the kinematic analysis (see below) requires the exchange of large amounts of video data and a local computer with GPU and installed motion-tracking software, we will use this time to run a pre-prepared pipeline with full tracking on all the data.
1. A kinematic journey from gesture to sign (led by Kadavá)
The first aim is to investigate the trajectory from gesture to sign using computer vision tools (e.g., OpenPose) that provide kinematic properties of the movement. We will perform motion tracking on videos that originate from different ‘time stamps’ of the gesture–sign continuum, namely:
- Unrestricted pantomime (data from FLESH)
- Single gesture, produced under time pressure (GtST)
- Repetition of a sign by naive sign learners (GtST)
- Repetition of a sign by learners after exposure to DGS (GtST)
- Sign by a native signer of DGS (GtST)
We will use features such as gesture size (amplitude) and velocity to explore how the kinematics of one’s movement change under different conditions, and how they possibly evolve when increasingly constrained by the normative, conventionalized form.
The second aim is to explore possible relations between the different ratings each project has obtained for the subset of concepts that overlap. The FLESH project collected data on the imagined expressibility of these concepts (i.e. how easily do you think you could express this concept through gesture/pantomime), and GtST has obtained ratings on iconicity and transparency of the specific gestures and signs. Furthermore, GtST has analyzed the silent gesture productions of participants for the occurrence of predominant gestures (i.e., gestures that are produced by >50% of participants). We want to investigate whether (a) higher imagined expressibility and the occurrence of a predominant gesture are correlated, and (b) more transparent signs/gestures are found in concepts that are rated as more expressible.