Co-organized by the MA program in Theoretical and Applied Linguistics at UPF and MultIS (ViCom)
When? January 16th – February 3rd, 2023
Where? Online and in-person at the Poblenou Campus of UPF, Barcelona
The aim of this 30-hour course was to give students the tools and knowledge on how to annotate manual co-speech gesture for the elaboration of multimodal corpora using the MultiModal MultiDimensional (M3D) gesture labeling system. Specifically, students learned how to use ELAN and Praat for annotation following M3D standards, as well as some post-annotation data handling techniques.
Course report
Interest in research on multimodal communication, and specifically, manual co-speech gestures has grown in recent years. However, there is not a standard approach to the annotation of manual co-speech gestures, which makes it quite difficult to compare results across studies. Given that many of the projects across the ViCom enterprise will indeed require gesture annotation, as well as the interest generated by the M3D labeling system at international gesture conferences as well as within the department for translation and language sciences at Universitat Pompeu Fabra, the leaders of the MultIS project organized a 30-hour training course on the M3D gesture labelling system. Over three weeks from January 16th to February 3rd, the hybrid course was held every morning from 10h30 to 12h30, with the aim to teach Master’s students as well as researchers participating in the ViCom priority program about how to annotate manual co-speech gestures following standard M3D procedures. The objective of such a course is to ultimately foster comparable annotation procedures both across the various projects within ViCom, and more generally in the research community.
The course was divided into two sections. The first week was devoted to a basic overview of the M3D gesture labelling system. The first day was dedicated to familiarization of ELAN, the software used in the course to annotate gesture. Each subsequent day was devoted to a particular aspect of gesture, namely the identification and annotation of Gesture units (day 2), gesture form (day 3), gesture phasing and referentiality (day 4), and finally, gesture pragmatics and basic data exportation and management (day 5). The two following weeks corresponded to the advanced portion of the course. In this section of the course, the different aspects of gesture were described in much more thorough detail, and the course presented more ambiguous and difficult cases for practice. Specifically, these two weeks covered:
- more specific aspects of form (handedness, handshape, palm orientation, trajectory shape, and trajectory direction)
- the prosodic dimension of gestures (gesture phasing, rhythmic properties of gesture, interactions between gesture and speech prosody)
- the dimensional nature of gesture referentiality
- more thorough assessment of pragmatic functions of gestures
- More complex data management, including calculating inter-annotator reliability and data pre-processing and analysis in R.
Importantly, each individual session was divided into three parts. The first hour involved a theoretical lecture on the topic for the day. After the theoretical lecture, the instructor did a live annotation demonstration for approximately 20-30 minutes. During this time, the instructor annotated novel material while describing the workflow, procedures, and thought processes involved. After the live annotation demonstration, students were then invited to practice annotating on their own, using either their own data or the M3D_TED corpus openly available online. All throughout the session students were free to ask questions to resolve any doubts they may have had. This was particularly crucial during the last part of the course while students were independently annotating. Often, students raised particularly difficult or ambiguous sequences, in which cases the instructor would reconvene the class to discuss such cases and encourage the students to jointly find an appropriate solution. A survey at the end of the course showed that the students really enjoyed the dynamics of the class and felt that they had learned a lot about gesture annotation and the M3D system. The course has not only been beneficial for those who participated as students, but for the M3D system as well. The development of the course material as well as the feedback from the students in the course has served as the basis for the development of online training materials that will be published by the M3D team in the Spring of 2023. We look forward to sharing these online training materials with ViCom and the broader research community in the near future, as we believe this will be a crucial resource for gesture researchers at large to create reproducible and comparable multimodal corpora.